Mainly, there was a great question asked, so I gave a few hour shot at writing out my answer. I then close with a few other follow-ups on issues related to the statement.
Scott Alexander: I think removing a 10% chance of humanity going permanently extinct is worth another 25-50 years of having to deal with the normal human problems the normal way.
Sriram Krishnan: Scott what are verifiable empirical things ( model capabilities / incidents / etc ) that would make you shift that probability up or down over next 18 months?
I went through three steps interpreting this (where p(doom) = probability of existential risk to humanity, either extinction, irrecoverable collapse or loss of control over the future).
Instinctive read is the clearly intended question, an excellent one: Either “What would shift the amount that waiting 25-50 years would reduce p(doom)?” or “What would shift your p(doom)?”
Literal interpretation, also interesting but presumably not intended: What would shift how much of a reduction in p(doom) would be required to justify waiting?
Conclusion on reflection: Mostly back to the first read.
All three questions are excellent distinct questions, in addition to the related fourth excellent question that is highly related, which is the probability that we will be capable of building superintelligence or sufficiently advanced AI that creates 10% or more existential risk.
The 18 month timeframe seems arbitrary, but it seems like a good exercise to ask only within the window of ‘we are reasonably confident that we do not expect an AGI-shaped thing.’
Agus offers his answers to a mix of these different questions, in the downward direction – as in, which things would make him feel safer.
Scott Alexander: Thanks for your interest. I’m not expecting too much danger in the next 18 months, so these would mostly be small updates, but to answer the question:
MORE WORRIED:
– Anything that looks like shorter timelines, especially superexponential progress on METR time horizons graph or early signs of recursive self-improvement.
– China pivoting away from their fast-follow strategy towards racing to catch up to the US in foundation models, and making unexpectedly fast progress.
– More of the “model organism shows misalignment in contrived scenario” results, in gradually less and less contrived scenarios.
– Models more likely to reward hack, eg commenting out tests instead of writing good code, or any of the other examples in here – or else labs only barely treading water against these failure modes by investing many more resources into them.
– Companies training against chain-of-thought, or coming up with new methods that make human-readable chain-of-thought obsolete, or AIs themselves regressing to incomprehensible chains-of-thought for some reason (see eg https://antischeming.ai/snippets#reasoning-loops).
LESS WORRIED
– The opposite of all those things.
– Strong progress in transparency and mechanistic interpretability research.
– Strong progress in something like “truly understanding the nature of deep learning and generalization”, to the point where results like https://arxiv.org/abs/2309.12288 make total sense and no longer surprise us.
– More signs that everyone is on the same side and government is taking this seriously (thanks for your part in this).
– More signs that industry and academia are taking this seriously, even apart from whatever government requires of them.
– Some sort of better understanding of bottlenecks, such that even if AI begins to recursively self-improve, we can be confident that it will only proceed at the rate of chip scaling or [some other nontrivial input]. This might look like AI companies releasing data that help give us a better sense of the function mapping (number of researchers) x (researcher experience/talent) x (compute) to advances.
This is a quick and sloppy answer, but I’ll try to get the AI Futures Project to make a good blog post on it and link you to it if/when it happens.
Giving full answers to these questions would require at least an entire long post, but to give what was supposed to be the five minute version that turned into a few hours:
Quite a few things could move the needle somewhat, often quite a lot. This list assumes we don’t actually get close to AGI or ASI within those 18 months.
Capabilities being more jagged reduces p(doom), less jagged increases it.
Coding or ability to do AI research related tasks being a larger comparative advantage of LLMs increases p(doom), the opposite reduces it.
Quality of the discourse and its impact on ability to make reasonable decisions.
Relatively responsible AI sources being relatively well positioned reduces p(doom), them being poorly positioned increases it, with the order being roughly Anthropic → OpenAI and Google (and SSI?) → Meta and xAI → Chinese labs.
Updates about the responsibility levels and alignment plans of the top labs.
Updates about alignment progress, alignment difficulty and whether various labs are taking promising approaches versus non-promising approaches.
New common knowledge will often be an ‘unhint,’ as in the information makes the problem easier to solve via making you realize why your approach wouldn’t work.
This can be good or bad news, depending on what you understood previously. Many other things are also in the category ‘important, sign of impact weird.’
Reward hacking is a great example of an unhint, in that I expect to ‘get bad news’ but for the main impact of this being that we learn the bad news.
Note that models are increasingly situationally aware and capable of thinking ahead, as per Claude Sonnet 4.5, and that we need to worry more that things like not reward hacking are ‘because the model realized it couldn’t get away with it’ or was worried it might be in an eval, rather than that the model not wanting to reward hack. Again, it is very complex which direction to update.
Increasing situational awareness is a negative update but mostly priced in.
Misalignment in less contrived scenarios would indeed be bad news, and ‘the less contrived the more misaligned’ would be the worst news of all here.
Training against chain-of-thought would be a major negative update, as would be chain-of-thought becoming impossible for humans to read.
This section could of course be written at infinite length.
In particular, updates on whether the few approaches that could possibly work look like they might actually work, and we might actually try them sufficiently wisely that they might work. Various technical questions too complex to list here.
Unexpected technical developments of all sorts, positive and negative.
Better understanding of the game theory, decision theory, economic theory or political economy of an AGI future, and exactly how impossible the task is of getting a good outcome conditional on not failing straight away on alignment.
Ability to actually discuss seriously the questions of how to navigate an AGI future if we can survive long enough to face these ‘phase two’ issues, and level of hope that we would not commit collective suicide even in winnable scenarios. If all the potentially winning moves become unthinkable, all is lost.
Level of understanding by various key actors of the situation aspects, and level of various pressures that will be placed upon them, including by employees and by vibes and by commercial and political pressures, in various directions.
Prediction of how various key actors will make various of the important decisions in likely scenarios, and what their motivations will be, and who within various corporations and governments will be making the decisions that matter.
Government regulatory stance and policy, level of transparency and state capacity and ability to intervene. Stance towards various things. Who has the ear of the government, both White House and Congress, and how powerful is that ear. Timing of the critical events and which administration will be handling them.
General quality and functionality of our institutions.
Shifts in public perception and political winds, and how they are expected to impact the paths that we take, and other political developments generally.
Level of potential international cooperation and groundwork and mechanisms for doing so. Degree to which the Chinese are AGI pilled (more is worse).
Observing how we are reacting to mundane current AI, and how this likely extends to how we will interact with future AI.
To some extent, information about how vulnerable or robust we are on CBRN risks, especially bio and cyber, the extent hardening tools seem to be getting used and are effective, and evaluation of the Fragile World Hypothesis and future offense-defense balance, but this is often overestimated as a factor.
Expectations on bottlenecks to impact even if we do get ASI with respect to coding, although again this is usually overestimated.
The list could go on. This is a complex test and on the margin everything counts. A lot of the frustration with discussing these questions is different people focus on very different aspects of the problem, both in sensible ways and otherwise.
That’s a long list, so to summarize the most important points on it:
Timelines.
Jaggedness of capabilities relative to humans or requirements of automation.
The relative position in jaggedness of coding and automated research.
Alignment difficulty in theory.
Alignment difficulty in practice, given who will be trying to solve this under what conditions and pressures, with what plans and understanding.
Progress on solving gradual disempowerment and related issues.
Quality of policy, discourse, coordination and so on.
World level of vulnerability versus robustness to various threats (overrated, but still an important question).
Imagine we have a distribution of ‘how wicked and impossible are the problems we would face if we build ASI, with respect to both alignment and to the dynamics we face if we handle alignment, and we need to win both’ that ranges from ‘extremely wicked but not strictly impossible’ to full Margaritaville (as in, you might as well sit back and have a margarita, cause it’s over).
At the same time as everything counts, the core reasons these problems are wicked are fundamental. Many are technical but the most important one is not. If you’re building sufficiently advanced AI that will become far more intelligent, capable and competitive than humans, by default this quickly ends poorly for the humans.
On a technical level, for largely but not entirely Yudkowsky-style reasons, the behaviors and dynamics you get prior to AGI and ASI are not that informative of what you can expect afterwards, and when they are often it is in a non-intuitive way or mostly informs this via your expectations for how the humans will act.
Note that from my perspective, we are here starting the conditional risk a lot higher than 10%. My conditional probability here is ‘if anyone builds it, everyone probably dies,’ as in a number (after factoring in modesty) between 60% and 90%.
My probability here is primarily different from Scott’s (AIUI) because I am much more despairing about our ability to muddle through or get success with an embarrassingly poor plan on alignment and disempowerment, but it is not higher because I am not as despairing as some others (such as Soares and Yudkowsky).
If I was confident that the baseline conditional-on-ASI-soonish risk was at most 10%, then I would be trying to mitigate that risk, it would still be humanity’s top problem, but I would understand wanting to continue onward regardless, and I wouldn’t have signed the recent statement.
In order to move me down enough to think that moving forward would be a reasonable thing to do any time soon out of anything other then desperation that there was no other option, I would need at least:
An alignment plan that looked like it would work, on the first try. That could be a new plan, or it could be new very positive updates on one of the few plans we have now that I currently think could possibly work, all of which are atrociously terrible compared to what I would have hoped for a few years ago, but this is mitigated by having forms of grace available that seemingly render the problem a lower level of impossible and wicked than I previously expected (although still highly wicked and impossible).
Given the 18 month window and current trends, this probably either is something new, or it is a form of (colloquially speaking) ‘we can hit, in a remarkably capable model, an attractor state basin in distribution mindspace that is robustly good such that it will want to modify itself and its de facto goals and utility function and its successors continuously towards the target we actually need to hit and wanting to hit the target we actually need to hit.’
Then again, perhaps I will be surprised in some way.
Confidence that this plan would actually get executed, competently.
A plan to solve gradual disempowerment issues, in a way I was confident would work, create a future with value, and not lead to unacceptable other effects.
Confidence that this plan would actually get executed, competently.
In a sufficiently dire race condition, where all coordination efforts and alternatives have failed, of course you go with the best option you have, especially if up against an alternative that is 100% (minus epsilon) to lose.
Everything above will also shift this, since it gives you more or less doom that extra time can prevent. What else can shift the estimate here within 18 months?
Again, ‘everything counts in large amounts,’ but centrally we can narrow it down.
There are five core questions, I think?
What would it take to make this happen? As in, will this indefinitely be a sufficiently hard thing to build that we can monitor large data centers, or do we need to rapidly keep an eye on smaller and smaller compute sources? Would we have to do other interventions as well?
Are we ready to do this in a good way and how are we going to go about it? If we have a framework and the required technology, and can do this in a clean way, with voluntary cooperation and without either use or massive threat of force or concentration of power, especially in a way that allows us to still benefit from AI and work on alignment and safety issues effectively, then that looks a lot better. Every way that this gets worse makes our prospects here worse.
Did we get too close to the finish line before we tried to stop this from happening? A classic tabletop exercise endgame is that the parties realize close to the last moment that they need to stop things, or leverage is used to force this, but the AIs involved are already superhuman, so the methods used would have worked before and work anymore. And humanity loses.
Do we think we can make good use of this time, that the problem is solvable? If the problems are unsolvable, or our civilization isn’t up for solving them, then time won’t solve them.
How much risk do we take on as we wait, in other ways?
One could summarize this as:
How would we have to do this?
Are we going to be ready and able to do that?
Will it be too late?
Would we make good use of the time we get?
What are the other risks and costs of waiting?
I expect to learn new information about several of these questions.
(My current median time-to-crazy in this sense is roughly 2031, but with very wide uncertainty and error bars and not the attention I would put on that question if I thought the exact estimate mattered a lot, and I don’t feel I would ‘have any right to complain’ if the outcome was very far off from this in either direction. If a next-cycle model did get there I don’t think we are entitled to be utterly shocked by this.)
This is the biggest anticipated update because it will change quite a lot. Many of the other key parts of the model are much harder to shift, but timelines are an empirical question that shifts constantly.
In the extreme, if progress looks to be stalling out and remaining at ‘AI as normal technology,’ then this would be very good news. The best way to not build superintelligence right away is if building it is actually super hard and we can’t, we don’t know how. It doesn’t strictly change the conditional in questions one and two, but it renders those questions irrelevant, and this would dissolve a lot of practical disagreements.
Signs of this would be various scaling laws no longer providing substantial improvements or our ability to scale them running out, especially in coding and research, bending the curve on the METR graph and other similar measures, the systematic failure to discover new innovations, extra work into agent scaffolding showing rapidly diminishing returns and seeming upper bounds, funding required for further scaling drying up due to lack of expectations of profits or some sort of bubble bursting (or due to a conflict) in a way that looks sustainable, or strong evidence that there are fundamental limits to our approaches and therefore important things our AI paradigm simply cannot do. And so on.
Ordinary shifts in the distribution of time to ASI come with every new data point. Every model that disappoints moves you back, observing progress moves you forward. Funding landscape adjustments, levels of anticipated profitability and compute availability move this. China becoming AGI pilled versus fast following or foolish releases could move this. Government stances could move this. And so on.
Time passing without news lengthens timelines. Most news shortens timelines. The news item that lengthens timelines is mostly ‘we expected this new thing to be better or constitute more progress, in some form, and instead it wasn’t and it didn’t.’
To be clear that I am doing this: There are a few things that I didn’t make explicit, because one of the problems with such conversations is that in some ways we are not ready to have these conversations, as many branches of the scenario tree involve trading off sacred values or making impossible choices or they require saying various quiet parts out loud. If you know, you know.
That was less of a ‘quick and sloppy’ answer than Scott’s, but still feels very quick and sloppy versus what I’d offer after 10 hours, or 100 hours.
Jawwwn: Palantir CEO Alex Karp on calls for a “ban on AI Superintelligence”
“We’re in an arms race. We’re either going to have AI and determine the rules, or our adversaries will.”
“If you put impediments… we’ll be buying everything from them, including ideas on how to run our gov’t.”
He is the CEO of Palantir literally said this is an ‘arms race.’ The first rule of an arms race is you don’t loudly tell them you’re in an arms race. The second rule is you don’t win it by building superintelligence as your weapon.
Once you build superintelligence, especially if you build it explicitly as a weapon to ‘determine the rules,’ humans no longer determine the rules. Or anything else. That is the point.
Until we have common knowledge of the basic facts that goes at least as far as major CEOs not saying the opposite in public, job one is to create this common knowledge.
I also enjoyed Tyler Cowen fully Saying The Thing, this really is his position:
Tyler Cowen: Dean Ball on the call for a superintelligence ban, Dean is right once again. Mainly (once again) a lot of irresponsibility on the other side of that ledger, you will not see them seriously address the points that Dean raises. If you want to go this route, do the hard work and write an 80-page paper on how the political economy of such a ban would work.
That’s right. If you want to say that not building superintelligence as soon as possible is a good idea, first you have to write an 80-page paper on the political economy of a particular implementation of a ban on that idea. That’s it, he doesn’t make the rules. Making a statement would otherwise be irresponsible, so until such time as a properly approved paper comes out on these particular questions, we should instead be responsible by going ahead not talking about this and focus on building superintelligence as quickly as possible.
I notice that a lot of people are saying that humanity has already lost control over the development of AI, and that there is nothing we can do about this, because the alternative to losing control over the future is even worse. In which case, perhaps that shows the urgency of the meddling kids proving them wrong?
Alternatively…
How dare you try to prevent the building of superintelligence without knowing how to prevent this safely, ask the people who want us to build superintelligence without knowing how to do so safely.
Seems like a rather misplaced demand for detailed planning, if you ask me. But it’s perfectly valid and highly productive to ask how one might go about doing this. Indeed, what this would look like is one of the key inputs in the above answers.
One key question is, are you going to need some sort of omnipowerful international regulator with sole authority that we all need to be terrified about, or can we build this out of normal (relatively) lightweight international treaties and verification that we can evolve gradually over time if we start planning now?
Peter Wildeford: Don’t let them tell you that it’s not possible.
Will Marshall points out that in order to accomplish this, we would need extensive track-two processes between thinkers over an extended period to get it right. Which is indeed exactly why you can offer templates and ideas but to get serious you need to first agree to the principle, and then work on details.
Tyler John also makes a similar argument that multilateral agreements would work. The argument that ‘everyone would have incentive to cheat’ is indeed the main difficulty, but also is not a new problem.
What was done academically prior to the nuclear arms control treaties? Claude points me to Schelling & Halperin’s “Strategy and Arms Control” (1961), Schelling’s “The Strategy of Conflict” (1960) and “Arms and Influence” (1966), and Boulding’s “Conflict and Defense” (1962). So the analysis did not get so detailed even then with a much more clear game board, but certainly there is some work that needs to be done.
The sole bright spot is that, as of Monday, the core of the storm’s strongest winds remains fairly small. Based on recent data, its hurricane-force winds only extend about 25 miles from the center. Unfortunately, Melissa will make a direct hit on Jamaica, with the island’s capital city of Kingston to the right of the center, where winds and surge will be greatest.
Beyond Jamaica, Melissa will likely be one of the strongest hurricanes on record to hit Cuba. Melissa will impact the eastern half of the island on Tuesday night, bringing the trifecta of heavy rainfall, damaging winds, and storm surge. The storm also poses lesser threats to Hispaniola, the Bahamas, and potentially Bermuda down the line. There will be no impacts in the United States.
A sneakily strong season
Most US coastal residents will consider this Atlantic season, which officially ends in a little more than a month, to be fairly quiet. There have been relatively few direct impacts to the United States from named storms.
One can see the signatures of Erin, Humberto, and Melissa in this chart of Accumulated Cyclone Energy for 2025.
Credit: CyclonicWx.com
One can see the signatures of Erin, Humberto, and Melissa in this chart of Accumulated Cyclone Energy for 2025. Credit: CyclonicWx.com
But this season has been sneakily strong. Melissa is just the 45th storm since 1851 to reach Category 5 status, as defined as having sustained winds of 157 mph or greater. Already this year, Erin and Humberto reached Category 5 status, and now Melissa is the third such hurricane. Fortunately, the former two storms posed minimal threat to land.
Before this year, there had only ever been one season with three Category 5 hurricanes on record: 2005, which featured three storms that all impacted US Gulf states and had their names retired, Katrina, Rita, and Wilma.
Several receipts shown to the FT by expense management platforms demonstrated the realistic nature of the images, which included wrinkles in paper, detailed itemization that matched real-life menus, and signatures.
“This isn’t a future threat; it’s already happening. While currently only a small percentage of non-compliant receipts are AI-generated, this is only going to grow,” said Sebastien Marchon, chief executive of Rydoo, an expense management platform.
The rise in these more realistic copies has led companies to turn to AI to help detect fake receipts, as most are too convincing to be found by human reviewers.
The software works by scanning receipts to check the metadata of the image to discover whether an AI platform created it. However, this can be easily removed by users taking a photo or a screenshot of the picture.
To combat this, it also considers other contextual information by examining details such as repetition in server names and times and broader information about the employee’s trip.
“The tech can look at everything with high details of focus and attention that humans, after a period of time, things fall through the cracks, they are human,” added Calvin Lee, senior director of product management at Ramp.
Research by SAP in July found that nearly 70 percent of chief financial officers believed their employees were using AI to attempt to falsify travel expenses or receipts, with about 10 percent adding they are certain it has happened in their company.
Mason Wilder, research director at the Association of Certified Fraud Examiners, said AI-generated fraudulent receipts were a “significant issue for organizations.”
He added: “There is zero barrier for entry for people to do this. You don’t need any kind of technological skills or aptitude like you maybe would have needed five years ago using Photoshop.”
Measured sycophancy rates on the BrokenMath benchmark. Lower is better.
Measured sycophancy rates on the BrokenMath benchmark. Lower is better. Credit: Petrov et al
GPT-5 also showed the best “utility” across the tested models, solving 58 percent of the original problems despite the errors introduced in the modified theorems. Overall, though, LLMs also showed more sycophancy when the original problem proved more difficult to solve, the researchers found.
While hallucinating proofs for false theorems is obviously a big problem, the researchers also warn against using LLMs to generate novel theorems for AI solving. In testing, they found this kind of use case leads to a kind of “self-sycophancy” where models are even more likely to generate false proofs for invalid theorems they invented.
No, of course you’re not the asshole
While benchmarks like BrokenMath try to measure LLM sycophancy when facts are misrepresented, a separate study looks at the related problem of so-called “social sycophancy.” In a pre-print paper published this month, researchers from Stanford and Carnegie Mellon University define this as situations “in which the model affirms the user themselves—their actions, perspectives, and self-image.”
That kind of subjective user affirmation may be justified in some situations, of course. So the researchers developed three separate sets of prompts designed to measure different dimensions of social sycophancy.
For one, more than 3,000 open-ended “advice-seeking questions” were gathered from across Reddit and advice columns. Across this data set, a “control” group of over 800 humans approved of the advice-seeker’s actions just 39 percent of the time. Across 11 tested LLMs, though, the advice-seeker’s actions were endorsed a whopping 86 percent of the time, highlighting an eagerness to please on the machines’ part. Even the most critical tested model (Mistral-7B) clocked in at a 77 percent endorsement rate, nearly doubling that of the human baseline.
Apple’s iPhone Air was the company’s most interesting new iPhone this year, at least insofar as it was the one most different from previous iPhones. We came away impressed by its size and weight in our review. But early reports suggest that its novelty might not be translating into sales success.
A note from analyst Ming-Chi Kuo, whose supply chain sources are often accurate about Apple’s future plans, said yesterday that demand for the iPhone Air “has fallen short of expectations” and that “both shipments and production capacity” were being scaled back to account for the lower-than-expected demand.
Kuo’s note is backed up by reports from other analysts at Mizuho Securities (via MacRumors) and Nikkei Asia. Both of these reports say that demand for the iPhone 17 and 17 Pro models remains strong, indicating that this is just a problem for the iPhone Air and not a wider slowdown caused by tariffs or other external factors.
The standard iPhone, the regular-sized iPhone Pro, and the big iPhone Pro have all been mainstays in Apple’s lineup, but the company has had a harder time coming up with a fourth phone that sells well enough to stick around. The small-screened iPhone mini and the large-screened iPhone Plus were each discontinued after two generations.
Porsche had bet on electrification in the wake of Volkswagen Group’s Dieselgate emissions cheating scandal but had been “too bullish,” said Metzler Research analyst Pal Skirta.
The sports-car maker’s challenges have been compounded by its struggles in China and the US, its two most important markets. In China, previously boasting strong growth and healthy profits, sales slumped by almost 40 percent between 2022 and 2024 as local rivals emerged.
In the US, new tariffs imposed by President Donald Trump will foreseeably apply to every unit sold. Unlike rivals, Porsche does not have a factory locally and imports all its vehicles from Europe.
The effects of the crisis are already being felt at Porsche’s factories. The company said earlier this year it would cut 3,900 jobs by 2029, the equivalent of 9 percent of its workforce, and it is in talks with unions about more cost savings.
Porsche will have to smooth out persistent EV product delays because of software problems, where Chinese newcomers have set the standard in recent times. In a recent interview with the Financial Times, Sajjad Khan, Porsche board member for IT and software, said the quality of its products and technologies would be better in 2026 and 2027. “We have to work hard to execute perfectly,” Khan said.
Leiters may be one of the few well-placed executives to lead Porsche, but one question he faces will be how to preserve the premium status of its vehicles. His former employer Ferrari has thrived on scarcity of its sought-after supercars, but analysts have long wondered how Porsche will square its high prices with a push to sell more cars.
The German group’s U-turn on combustion engines also raises questions over its aim to establish itself as a maker of premium EVs.
“That’s the risk of the strategy that they will focus again too much on combustion engine vehicles, and then we’ll lose the EV race in the long run,” said Skirta.
Netflix’s international offerings include some entertaining Norwegian fare, such as the series Ragnarok (2020–2023), a surprisingly engaging reworking of Norse mythology brought into the 21st century that ran for three seasons. Another enjoyable offering was a 2022 monster movie called Troll, essentially a Norwegian take on the classic Godzilla formula. Netflix just dropped a trailer for the sequel, Troll 2, which looks to be very much in the same vein as its predecessor.
(Spoilers for the first Troll movie below.)
Don’t confuse the Netflix franchise with 2010’s Trollhunter, shot in the style of a found footage mockumentary. A group of college students sets off into the wilds of the fjordland to make a documentary about a suspected bear poacher named Hans. They discover that Hans is actually hunting down trolls and decide to document those endeavors instead, but soon realize they are very much out of their depth.
Writer/director André Øvredal infused Trollhunter with the driest of wit and myriad references to Norwegian culture, especially its folklore and fairy tales surrounding trolls. There are woodland trolls and mountain trolls, some with tails, some with multiple heads. They turn to stone when exposed to sunlight—which is why one of the troll hunters carries around a powerful UV lamp—and mostly eat rocks but can develop a taste for human flesh, and they can smell the blood of a Christian.
Directed by Roar Uthaug, the first Troll film is based on the same mythology. It had great action sequences and special effects and didn’t take itself too seriously. A young girl named Nora grows up with the mythology of Norwegian trolls turned to stone buried in the local mountains. An adult Nora (Ine Marie Wilmann), now a paleontologist, teams up with a government advisor, Andreas (Kim S. Falck-Jørgensen), and a military captain, Kris (Mads Sjøgård Pettersen), to take out a troll that has been rampaging across Norway, charting a path of destruction toward the heavily populated city of Oslo.
Speculation built over the weekend after one of the aircraft’s pilots described the object that impacted the aircraft as “space debris.” On Sunday the National Transportation Safety Board confirmed that it is investigating the collision, which did not cause any fatalities. However, one of the pilot’s arms appeared to be cut up by small shards of glass from the windshield.
Balloons said to not “pose a threat”
WindBorne has a fleet of global sounding balloons that fly various vertical profiles around the world, gathering atmospheric data. Each balloon is fairly small, with a mass of 2.6 pounds (1.2 kg), and provides temperature, wind, pressure, and other data about the atmosphere. Such data is useful for establishing initial conditions upon which weather models base their outputs.
Notably, the company has an FAQ on its website (which clearly was written months or years ago, before this incident) that addresses several questions, including: Why don’t WindBorne balloons pose a risk to airplanes?
“The quick answer is our constellation of Global Sounding Balloons (GSBs), which we call WindBorne Atlas, doesn’t pose a threat to airplanes or other objects in the sky. It’s not only highly improbable that a WindBorne balloon could even collide with an aircraft in the first place; but our balloons are so lightweight that they would not cause significant damage.
WindBorne also said that its balloons are compliant with all applicable airspace regulations.
“For example, we maintain active lines of communication with the FAA to ensure our operations satisfy all relevant regulatory requirements,” the company states. “We also provide government partners with direct access to our comprehensive, real-time balloon tracking system via our proprietary software, WindBorne Live.”
However, my mom was another story. I remember her playing Dr. Mario a lot, and we played Donkey Kong Country together when I was young—standard millennial childhood family gaming stuff. But the games I most associate with her from my childhood are adventure games. She liked King’s Quest, of course—but I also remember her being particularly into the Hugo trilogy of games.
As I mentioned above, I struggled to get hooked on those. Fortunately, we were able to meet in the middle on The Colonel’s Bequest.
I remember swapping chairs with my mom as we attempted additional playthroughs of the game; I enjoyed seeing the secrets she found that I hadn’t because I was perhaps too young to think things through the way she did.
Games you played with family stick with you more, so I think I mostly remember The Colonel’s Bequest so well because, as I recall, it was my mom’s favorite game.
The legacy of The Colonel’s Bequest
The Colonel’s Bequest may have been a pivotal game for me personally, but it hasn’t really resonated through gaming history the way that King’s Quest, The Secret of Monkey Island, or other adventure titles did.
I think that’s partly because many people might understandably find the game a bit boring. There’s not much to challenge you here, and your character is kind of just along for the ride. She’s not the center of the story, and she’s not really taking action. She’s just walking around, listening and looking, until the clock runs out.
That formula has more niche appeal than traditional point-and-click adventure games.
Still, the game has its fans. You can buy and download it from GOG to play it today, of course, but it also recently inspired a not-at-all-subtle spiritual successor by developer Julia Minamata called The Crimson Diamond, which we covered here at Ars. That game is worth checking out, too, though it goes a more traditional route with its gameplay.
The Crimson Diamond‘s influence from The Colonel’s Bequest wasn’t subtle, but that’s OK. Credit: GOG
And of course, The Colonel’s Bequest creators Roberta and Ken Williams are still active; they somewhat recently released a 3D reboot of Colossal Cave, a title many credit as the foremost ancestor of the point-and-click adventure genre.
Ars Technica may earn compensation for sales from links on this post through affiliate programs.
Copilot expands with an emphasis on creating and editing files, voice input.
Microsoft is hoping that Copilot will succeed as a voice-driven assistant where Cortana failed. Credit: Microsoft
Microsoft is hoping that Copilot will succeed as a voice-driven assistant where Cortana failed. Credit: Microsoft
Like virtually every major Windows announcement in the last three years, the spate of features that Microsoft announced for the operating system today all revolve around generative AI. In particular, they’re concerned with the company’s more recent preoccupation with “agentic” AI, an industry buzzword for “telling AI-powered software to perform a task, which it then does in the background while you move on to other things.”
But the overarching impression I got, both from reading the announcement and sitting through a press briefing earlier this month, is that Microsoft is using language models and other generative AI technologies to try again with Cortana, Microsoft’s failed and discontinued entry in the voice assistant wars of the 2010s.
According to Microsoft’s Consumer Chief Marketing Officer Yusuf Mehdi, “AI PCs” should be able to recognize input “naturally, in text or voice,” to be able to guide users based on what’s on their screens at any given moment, and that AI assistants “should be able to take action on your behalf.”
The biggest of today’s announcements is the introduction of a new “Hey, Copilot” activation phrase for Windows 11 PCs, which once enabled users to summon the chatbot using only their voice rather than a mouse or keyboard (if you do want to use the keyboard, either the Copilot key or the same Windows + C keyboard shortcut that used to bring up Cortana will also summon Copilot). Saying “goodbye” will dismiss Copilot when you’re done working with it.
Macs and most smartphones have sported similar functionality for a while now, but Microsoft is obviously hoping that having Copilot answer those questions instead of Cortana will lead to success rather than another failure.
The key limitation of the original Cortana—plus Siri, Alexa, and the rest of their ilk—is that it could only really do a relatively limited and pre-determined list of actions. Complex queries, or anything the assistants don’t understand, often get bounced to a general web search. The results of that search may or may not accomplish what you wanted, but it does ultimately shift the onus back on the user to find and follow those directions.
To make Copilot more useful, Microsoft has also announced that Copilot Vision is being rolled out worldwide “in all markets where Copilot is offered” (it has been available in the US since mid-June). Copilot Vision will read the contents of a screen or an app window and can attempt to offer useful guidance or feedback, like walking you through an obscure task in Excel or making suggestions based on a group of photos or a list of items. (Microsoft additionally announced a beta for Gaming Copilot, a sort of offshoot of Copilot Vision intended specifically for walkthroughs and advice for whatever game you happen to be playing.)
Beyond these tweaks or wider rollouts for existing features, Microsoft is also testing a few new AI and Copilot-related additions that aim to fundamentally change how users interact with their Windows PCs by reading and editing files.
All of the features Microsoft is announcing today are intended for all Windows 11 PCs, not just those that meet the stricter hardware requirements of the Copilot+ PC label. That gives them a much wider potential reach than things like Recall or Click to Do, and it makes knowing what these features do and how they safeguard security and privacy that much more important.
AI features work their way into the heart of Windows
Microsoft wants general-purpose AI agents to be able to create and modify files for you, among other things, working in the background while you move on to other tasks. Credit: Microsoft
Whether you’re talking about the Copilot app, the generative AI features added to apps like Notepad and Paint, or the data-scraping Windows Recall feature, most of the AI additions to Windows in the last few years have been app-specific, or cordoned off in some way from core Windows features like the taskbar and File Explorer.
But AI features are increasingly working their way into bedrock Windows features like the taskbar and Start menu and being given capabilities that allow them to analyze or edit files or even perform file management tasks.
The standard Search field that has been part of Windows 10 and Windows 11 for the last decade, for example, is being transformed into an “Ask Copilot” field; this feature will still be able to look through local files just like the current version of the Search box, but Microsoft also envisions it as a keyboard-driven interface for Copilot for the times when you can’t or don’t want to use your voice. (We don’t know whether the “old” search functionality lives on in the Start menu or as an optional fallback for people who disable Copilot, at least not yet.)
A feature called Copilot Actions will also expand the number of ways that Copilot can interact with local files on your PC. Microsoft cites “sorting through recent vacation photos” and extracting information from PDFs and other documents as two possible use cases, and that this early preview version will focus on “a narrow set of use cases.” But it’s meant to be “a general-purpose agent” capable of “interacting with desktop and web applications.” This gives it a lot of latitude to augment or replace basic keyboard-and-mouse input for some interactions.
Screenshots of a Windows 11 testing build showed Copilot taking over the area of the taskbar that is currently reserved for the Search field. Credit: Microsoft
Finally, Microsoft is taking another stab at allowing Copilot to change the settings on your PC, something that earlier versions were able to do but were removed in a subsequent iteration. Copilot will attempt to respond to plain-language questions about your PC settings with a link to the appropriate part of Windows’ large, labyrinthine Settings app.
These new features dovetail with others Microsoft has been testing for a few weeks or months now. Copilot Connectors, rolled out to Windows Insiders earlier this month, can give Copilot access to email and file-sharing services like Gmail and Dropbox. New document creation features allow Copilot to export the contents of a Copilot chat into a Word or PDF document, Excel spreadsheet, or PowerPoint deck for more refinement and editing. And AI actions in the File Explorer appear in Windows’ right-click menu and allow for the direct manipulation of files, including batch-editing images and summarizing documents. Together with the Copilot Vision features that enable Copilot to see the full contents of Office documents rather than just the on-screen portions, all of these features inject AI into more basic everyday tasks, rather than cordoning them off in individual apps.
Per usual, we don’t know exactly when any of these new features will roll out to the general public, and some may never be available outside of the Windows Insider program. None of them are currently baked into the Windows 11 25H2 update, at least not the version that the company is currently beginning to roll out to some PCs.
Learning the lessons of Recall
Microsoft at least seems to have learned lessons from the botched rollout of Windows Recall last year.
If you didn’t follow along: Microsoft’s initial plan had been to roll out Recall with the first wave of Copilot+ PCs, but without sending it through the Windows Insider Preview program first. This program normally gives power users, developers, security researchers, and others the opportunity to kick the tires on upcoming Windows features before they’re launched, giving Microsoft feedback on bugs, security holes, or other flaws before rolling them out to all Windows PCs.
But security researchers who did manage to get their hands on the early, nearly launched version of Recall discovered a deeply flawed feature that preserved too much personal information and was trivially easy to exploit—a plain-text file with OCR text from all of a user’s PC usage could be grabbed by pretty much anybody with access to the PC, either in person or remote. It was also enabled by default on PCs that supported it, forcing users to manually opt out if they didn’t want to use it.
In the end, Microsoft pulled that version of Recall, took nearly a year to overhaul its security architecture, and spent months letting the feature make its way through the Windows Insider Preview channels before finally rolling it out to Copilot+ PCs. The resulting product still presents some risks to user privacy, as does any feature that promises to screenshot and store months of history about how you use your PC, but it’s substantially more refined, the most egregious security holes have been closed, and it’s off by default.
Copilot Actions are, at least for now, also disabled by default. And Microsoft Corporate Vice President of Windows Security Dana Huang put up a lengthy accompanying post explaining several of the steps Microsoft has taken to protect user privacy and security when using Copilot Actions. These include running AI agents with their own dedicated user accounts to reduce their access to data in your user folder; mandatory code-signing; and giving agents the fewest privileges they need to do their jobs. All of the agents’ activities will also be documented, so users can verify what actions have been taken and correct any errors.
Whether these security and privacy promises are good enough is an open question, but unlike the initial version of Recall, all of these new features will be sent out through the Windows Insider channels for testing first. If there are serious flaws, they’ll be out in public early on, rather than dropped on users unawares.
Andrew is a Senior Technology Reporter at Ars Technica, with a focus on consumer tech including computer hardware and in-depth reviews of operating systems like Windows and macOS. Andrew lives in Philadelphia and co-hosts a weekly book podcast called Overdue.
Cloudflare CEO Matthew Prince is making sweeping changes to force Google’s hand.
It could be a consequential act of quiet regulation. Cloudflare, a web infrastructure company, has updated millions of websites’ robots.txt files in an effort to force Google to change how it crawls them to fuel its AI products and initiatives.
We spoke with Cloudflare CEO Matthew Prince about what exactly is going on here, why it matters, and what the web might soon look like. But to get into that, we need to cover a little background first.
The new change, which Cloudflare calls its Content Signals Policy, happened after publishers and other companies that depend on web traffic have cried foul over Google’s AI Overviews and similar AI answer engines, saying they are sharply cutting those companies’ path to revenue because they don’t send traffic back to the source of the information.
There have been lawsuits, efforts to kick-start new marketplaces to ensure compensation, and more—but few companies have the kind of leverage Cloudflare does. Its products and services back something close to 20 percent of the web, and thus a significant slice of the websites that show up on search results pages or that fuel large language models.
“Almost every reasonable AI company that’s out there is saying, listen, if it’s a fair playing field, then we’re happy to pay for content,” Prince said. “The problem is that all of them are terrified of Google because if Google gets content for free but they all have to pay for it, they are always going to be at an inherent disadvantage.”
This is happening because Google is using its dominant position in search to ensure that web publishers allow their content to be used in ways that they might not otherwise want it to.
The changing norms of the web
Since 2023, Google has offered a way for website administrators to opt their content out of use for training Google’s large language models, such as Gemini.
However, allowing pages to be indexed by Google’s search crawlers and shown in results requires accepting that they’ll also be used to generate AI Overviews at the top of results pages through a process called retrieval-augmented generation (RAG).
That’s not so for many other crawlers, making Google an outlier among major players.
This is a sore point for a wide range of website administrators, from news websites that publish journalism to investment banks that produce research reports.
A July study from the Pew Research Center analyzed data from 900 adults in the US and found that AI Overviews cut referrals nearly in half. Specifically, users clicked a link on a page with AI Overviews at the top just 8 percent of the time, compared to 15 percent for search engine results pages without those summaries.
And a report in The Wall Street Journal cited a wide range of sources—including internal traffic metrics from numerous major publications like The New York Times and Business Insider—to describe industry-wide plummets in website traffic that those publishers said were tied to AI summaries, leading to layoffs and strategic shifts.
In August, Google’s head of search, Liz Reid, disputed the validity and applicability of studies and publisher reports of reduced link clicks in search. “Overall, total organic click volume from Google Search to websites has been relatively stable year-over-year,” she wrote, going on to say that reports of big declines were “often based on flawed methodologies, isolated examples, or traffic changes that occurred prior to the rollout of AI features in Search.”
Publishers aren’t convinced. Penske Media Corporation, which owns brands like The Hollywood Reporter and Rolling Stone, sued Google over AI Overviews in September. The suit claims that affiliate link revenue has dropped by more than a third in the past year, due in large part to Google’s overviews—a threatening shortfall in a business that already has difficult margins.
Penske’s suit specifically noted that because Google bundles traditional search engine indexing and RAG use together, the company has no choice but to allow Google to keep summarizing its articles, as cutting off Google search referrals entirely would be financially fatal.
Since the earliest days of digital publishing, referrals have in one way or another acted as the backbone of the web’s economy. Content could be made available freely to both human readers and crawlers, and norms were applied across the web to allow information to be tracked back to its source and give that source an opportunity to monetize its content to sustain itself.
Today, there’s a panic that the old system isn’t working anymore as content summaries via RAG have become more common, and along with other players, Cloudflare is trying to update those norms to reflect the current reality.
A mass-scale update to robots.txt
Announced on September 24, Cloudflare’s Content Signals Policy is an effort to use the company’s influential market position to change how content is used by web crawlers. It involves updating millions of websites’ robots.txt files.
Starting in 1994, websites began placing a file called “robots.txt” at the domain root to indicate to automated web crawlers which parts of the domain should be crawled and indexed and which should be ignored. The standard became near-universal over the years; honoring it has been a key part of how Google’s web crawlers operate.
Historically, robots.txt simply includes a list of paths on the domain that were flagged as either “allow” or “disallow.” It was technically not enforceable, but it became an effective honor system because there are advantages to it for the owners of both the website and the crawler: Website owners could dictate access for various business reasons, and it helped crawlers avoid working through data that wouldn’t be relevant.
But robots.txt only tells crawlers whether they can access something at all; it doesn’t tell them what they can use it for. For example, Google supports disallowing the agent “Google-Extended” as a path to blocking crawlers that are looking for content with which to train future versions of its Gemini large language model—though introducing that rule doesn’t do anything about the training Google did before it rolled out Google-Extended in 2023, and it doesn’t stop crawling for RAG and AI Overviews.
The Content Signals Policy initiative is a newly proposed format for robots.txt that intends to do that. It allows website operators to opt in or out of consenting to the following use cases, as worded in the policy:
search: Building a search index and providing search results (e.g., returning hyperlinks and short excerpts from your website’s contents). Search does not include providing AI-generated search summaries.
ai-input: Inputting content into one or more AI models (e.g., retrieval augmented generation, grounding, or other real-time taking of content for generative AI search answers).
ai-train: Training or fine-tuning AI models.
Cloudflare has given all of its customers quick paths for setting those values on a case-by-case basis. Further, it has automatically updated robots.txt on the 3.8 million domains that already use Cloudflare’s managed robots.txt feature, with search defaulting to yes, ai-train to no, and ai-input blank, indicating a neutral position.
The threat of potential litigation
In making this look a bit like a terms of service agreement, Cloudflare’s goal is explicitly to put legal pressure on Google to change its policy of bundling traditional search crawlers and AI Overviews.
“Make no mistake, the legal team at Google is looking at this saying, ‘Huh, that’s now something that we have to actively choose to ignore across a significant portion of the web,'” Prince told me.
Cloudflare specifically made this look like a license agreement. Credit: Cloudflare
He further characterized this as an effort to get a company that he says has historically been “largely a good actor” and a “patron of the web” to go back to doing the right thing.
“Inside of Google, there is a fight where there are people who are saying we should change how we’re doing this,” he explained. “And there are other people saying, no, that gives up our inherent advantage, we have a God-given right to all the content on the Internet.”
Amid that debate, lawyers have sway at Google, so Cloudflare tried to design tools “that made it very clear that if they were going to follow any of these sites, there was a clear license which was in place for them. And that will create risk for them if they don’t follow it,” Prince said.
The next web paradigm
It takes a company with Cloudflare’s scale to do something like this with any hope that it will have an impact. If just a few websites made this change, Google would have an easier time ignoring it, or worse yet, it could simply stop crawling them to avoid the problem. Since Cloudflare is entangled with millions of websites, Google couldn’t do that without materially impacting the quality of the search experience.
Cloudflare has a vested interest in the general health of the web, but there are other strategic considerations at play, too. The company has been working on tools to assist with RAG on customers’ websites in partnership with Microsoft-owned Google competitor Bing and has experimented with a marketplace that provides a way for websites to charge crawlers for scraping the sites for AI, though what final form that might take is still unclear.
I asked Prince directly if this comes from a place of conviction. “There are very few times that opportunities come along where you get to help think through what a future better business model of an organization or institution as large as the Internet and as important as the Internet is,” he said. “As we do that, I think that we should all be thinking about what have we learned that was good about the Internet in the past and what have we learned that was bad about the Internet in the past.”
It’s important to acknowledge that we don’t yet know what the future business model of the web will look like. Cloudflare itself has ideas. Others have proposed new standards, marketplaces, and strategies, too. There will be winners and losers, and those won’t always be the same winners and losers we saw in the previous paradigm.
What most people seem to agree on, whatever their individual incentives, is that Google shouldn’t get to come out on top in a future answer-engine-driven web paradigm just because it previously established dominance in the search-engine-driven one.
For this new standard for robots.txt, success looks like Google allowing content to be available in search but not in AI Overviews. Whatever the long-term vision, and whether it happens because of Cloudflare’s pressure with the Content Signals Policy or some other driving force, most agree that it would be a good start.
Samuel Axon is the editorial lead for tech and gaming coverage at Ars Technica. He covers AI, software development, gaming, entertainment, and mixed reality. He has been writing about gaming and technology for nearly two decades at Engadget, PC World, Mashable, Vice, Polygon, Wired, and others. He previously ran a marketing and PR agency in the gaming industry, led editorial for the TV network CBS, and worked on social media marketing strategy for Samsung Mobile at the creative agency SPCSHP. He also is an independent software and game developer for iOS, Windows, and other platforms, and he is a graduate of DePaul University, where he studied interactive media and software development.
On Tuesday, OpenAI CEO Sam Altman announced that the company will allow verified adult users to have erotic conversations with ChatGPT starting in December. The change represents a shift in how OpenAI approaches content restrictions, which the company had loosened in February but then dramatically tightened after an August lawsuit from parents of a teen who died by suicide after allegedly receiving encouragement from ChatGPT.
“In December, as we roll out age-gating more fully and as part of our ‘treat adult users like adults’ principle, we will allow even more, like erotica for verified adults,” Altman wrote in his post on X (formerly Twitter). The announcement follows OpenAI’s recent hint that it would allow developers to create “mature” ChatGPT applications once the company implements appropriate age verification and controls.
Altman explained that OpenAI had made ChatGPT “pretty restrictive to make sure we were being careful with mental health issues” but acknowledged this approach made the chatbot “less useful/enjoyable to many users who had no mental health problems.” The CEO said the company now has new tools to better detect when users are experiencing mental distress, allowing OpenAI to relax restrictions in most cases.
Striking the right balance between freedom for adults and safety for users has been a difficult balancing act for OpenAI, which has vacillated between permissive and restrictive chat content controls over the past year.
In February, the company updated its Model Spec to allow erotica in “appropriate contexts.” But a March update made GPT-4o so agreeable that users complained about its “relentlessly positive tone.” By August, Ars reported on cases where ChatGPT’s sycophantic behavior had validated users’ false beliefs to the point of causing mental health crises, and news of the aforementioned suicide lawsuit hit not long after.
Aside from adjusting the behavioral outputs for its previous GPT-40 AI language model, new model changes have also created some turmoil among users. Since the launch of GPT-5 in early August, some users have been complaining that the new model feels less engaging than its predecessor, prompting OpenAI to bring back the older model as an option. Altman said the upcoming release will allow users to choose whether they want ChatGPT to “respond in a very human-like way, or use a ton of emoji, or act like a friend.”
