President Trump enacted a perfectly reasonable solution to the situation with Anthropic and the Department of War. He cancelled the Anthropic contract with a six month wind down period, after which the Federal Government would be told not to use Anthropic software.
Everyone thought the worst was now over. The situation was unfortunate for Anthropic and also for national security, but this gave us six months to transition, it gave us six months to negotiate another solution, and it avoided any of the extreme highly damaging options that Secretary of War Pete Hegseth and lead negotiator Emil Michael had placed upon the table.
Anthropic would be fine without government business, and the government would mostly be fine without directly using Anthropic. Face was saved.
I have sources that confirm that Trump’s announcement was wisely intended as an off-ramp and de-escalation of the situation, and that it was intended to be the end of it, or perhaps even a deal could have still been reached now that everyone could breathe.
An hour after that, on his own, Pete Hegseth went rogue and blew the whole situation up, illegally declaring that ‘effective immediately’ he was declaring Anthropic a Supply Chain Risk, and that anyone who did business with the Department of War in any capacity could not use Anthropic’s products in any capacity.
Even if it had not been issued via a Tweet, this is not how the law actually works.
If this is implemented as stated, it will cause a market bloodbath and immense damage to our national security and supply chain. It would be attempted corporate murder with a global blast radius.
Thankfully it probably won’t be anything close to that.
Probably.
The market understands that this is not how any of this works, so the reaction was for now relatively muted, as only about $150 billion was wiped from public markets in later post-close trading. I believe that is an underreaction based on the chilling effects and damage already done, but we will never know the true market impact because events have already been confounded.
I hope for the best on that front, but the danger remains.
We must be vigilant until the coast is clear, and we must prepare for the worst. Pete Hegseth cannot be allowed to commit corporate murder.
Outcomes like this usually don’t happen exactly because people realize they would otherwise happen, and prevent them.
What was that all about?
Ross Andersen: On Friday afternoon, Anthropic learned that the Pentagon still wanted to use the company’s AI to analyze bulk data collected from Americans. That could include information such as the questions you ask your favorite chatbot, your Google search history, your GPS-tracked movements, and your credit-card transactions, all of which could be cross-referenced with other details about your life.
Anthropic’s leadership told Hegseth’s team that was a bridge too far, and the deal fell apart.
Okay, what was that all about?
We don’t know. I have sources saying that Doge is driving this, and I have other speculations, but ultimately we don’t know what they want this capability for. What we do know is that they blew the whole situation up over this question. There must have been a reason.
Whatever that was, or an actual outright attempt to murder Anthropic, is what this is all about. It’s not a matter of principle.
Then, later that night, OpenAI accepted a contract with the Department of War. They claimed that very day that they had the same red lines as Anthropic, yet they seem to have accepted the same language Anthropic rejected as not meaningful, as confirmed by Jeremy Lewin.
How did OpenAI negotiate such a deal in two days? My interpretation of OpenAI’s public statements is that they consider any action crossing their red lines to already be illegal, and thus there are no uses that they would consider both legal and unacceptable, and that it is not their place to make that determination.
But that’s not what matters. The contract terms here ultimately don’t matter.
What matters is that OpenAI and the Department of War are trusting each other. OpenAI is giving DoW a replacement that allows them to offboard Anthropic without overly disrupting national security, and trusting DoW to decide what to do with that tech and to not to do anything too illegal.
DoW is trusting OpenAI to deliver a good model and let them do what they operationally need to do and not suddenly start tripping the safety mechanisms. Forward engineers and the safety stack will trust but verify, and Altman claims he stands ready to pull the plug if DoW goes too far.
All of OpenAI’s meaningful safeguards are in the security stack, and its right to choose what model to deliver and pull the plug. Which means they’re in contract language we may not ever see.
I believe that the way they presented that deal and the situation has been misleading enough to cost me and a lot of others a lot of sleep, but it now seems clear.
OpenAI’s employees need to investigate the technical provisions and ask whether the red lines they personally care about are meaningfully protected, and whether they wish to be part of what is happening given the circumstances.
Even more than that, it is not clear whether OpenAI’s attempted de-escalation of the situation de-escalated it, or escalated it further by giving Hegseth a green light.
Indeed, the New York Times thinks exactly that happened:
Sheera Frenkel, Cade Metz and Julian E. Barnes: Mr. Michael was unhappy with that answer, the people said. He also had an ace up his sleeve: On the side, he had been hammering out an alternative to Anthropic with its rival, OpenAI. A framework between the Pentagon and OpenAI had already been reached.
So when the Friday deadline passed, the Department of Defense did not give Anthropic more time. At 5: 14 p.m., Mr. Hegseth announced that he had designated Anthropic as a security risk and that it would be cut off from working with the U.S. government. “America’s warfighters will never be held hostage by the ideological whims of Big Tech,” he posted on social media.
Again, I don’t think that was Altman’s intention, at all. But whichever way this went, OpenAI’s employees and leadership need to make it clear that they cannot enter a relationship built on trust with DoW, if DoW actually attempts a widely scoped supply chain risk intervention against Anthropic, and attempts to kill the company.
Sam Altman has been excellent in calling for not labeling Anthropic a supply chain risk. I take him at his word that he was attempting to de-escalate.
But if OpenAI’s willingness to work with DoW is used not to de-escalate but as a way to allow escalation, then OpenAI must not abide this, and if OpenAI does abide then it would then be actively and consciously escalating the situation.
Ross Douthat: Does the precedent that the DoW is setting by effectively blacklisting Anthropic make you concerned about what any future dispute with the Pentagon would mean for your own company’s independence and viability?
Sam Altman (CEO OpenAI): Yes; I think it is an extremely scary precedent and I wish they handled it a different way. I don’t think Anthropic handled it well either, but as the more powerful party, I hold the government more responsible. I am still hopeful for a much better resolution.
If things escalate, ‘I wish it had gone better’ and ‘hopeful’ will no longer fly.
You may have some very big ethical decisions to make in the coming days.
So might those at many other tech companies, and everyone else, if this escalates. Think about what you would do if your company is put to a decision here.
What the OpenAI deal definitely did was further invalidate the legal arguments for a supply chain risk designation and removed the need for further confrontation. But unfortunately, no matter how obvious the case looks to us, we cannot be certain the courts will do the right thing, which includes doing it fast enough to prevent damage.
Throughout this, a remarkable number of people have tried to equate ‘democracy,’ the American way, with what is actually dictatorship or communism, or the Chinese way. As in private citizens do whatever those in charge demand of them, or else. I vehemently disagree.
Soren Kierkegaard: All arguments against Anthropic I’ve seen from right wing posters have been a variant of the government should be allowed to seize the means of production
Dean W. Ball: As we do, and as we have future debates about the proper nexus of control over frontier AI, I encourage you to avoid the assumption that “democratic” control—control “of the people, by the people, and for the people”—is synonymous with governmental control. The gap between these loci of control has always existed, but it is ever wider now.
For now the headlines say the big destructive action launched by the Department of War that day without proper Congressional authorization was that they attacked Iran. Even with what has unfolded there I am not entirely convinced history books, if we are around to read them, will see it that way.
The house is on fire.
The question is, what are you going to do about it?
This is my best effort to bring together the key events in the story. This is my best attempt to recollect the sequence of events. I apologize for any key omissions, errors or where I am trusting misrepresentations. Some of this is from private sources. Some events may be out of order, I believe in ways that would not change the interpretation.
Last year: Tensions rise between the White House and Anthropic, for a variety of reasons. David Sacks (conspicuously and virtuously silent during this crisis) spent a remarkable percentage of his time railing against Effective Altruism in general, ‘doomers’ and in particular Anthropic. Elon Musk, founder of xAI, is also repeatedly is hostile to Anthropic, and creates Doge. Katie Miller goes to xAI. Nvidia is hostile to Anthropic in various ways, despite investments.
Last year: Anthropic and other companies sign government contracts with DoD for up to $200 million each, containing many restrictions on government use. Anthropic makes it a priority to be the first to be on classified networks, despite it not being a good business opportunity given the associated risks and hassles, to help in the national defense. Anthropic has an easier route because of AWS.
Previously: DoW asks to renegotiate Anthropic’s contract to make it less restrictive. Anthropic agrees to do so on many fronts but draws two red lines.
January 3: Maduro is captured in a government raid. Anthropic’s Claude is widely believed to have been used in this, without incident. Everything went great.
January 9: Hegseth sends out a memo demanding, among other AI initiatives, DoW not use ‘woke AI.’
Previously: DoW circulates a story that Anthropic asked questions about the raid and was potentially unhappy and might pull its contract. I have gotten multiple unequivocal denials, saying this was entirely made up by DoW. This is part of an ongoing narrative of ‘what if they demand you get permission or they pull the plug mid-mission when they don’t like what you’re doing’ that has no bearing on the actual situation whatsoever, and never did.
Previously: Elon Musk, he of the Doge and xAI, and hater of supposed Woke AI everywhere, starts Tweeting far more frequent hostile and ad hominem attacks against Anthropic, really quite a lot, including saying they hate Western Civilization. Sources I have claim that he urged DoW to attempt to coerce or disrupt Anthropic. Katie Miller also Tweets similar material.
Previously: DoW circulates a story that Dario told them that if their system refused to provide real time missile defense (later they said drone defense) that he said to call them. I have unequivocal denial, from a secondary source, that this or anything like it ever happened. This story is almost certainly fiction and makes no sense, and is at best a willful misunderstanding. We already have automated missile defenses that wisely do not use LLMs. Calling Dario in real time would do absolutely nothing, regardless of his preferences, and he could neither turn on or off such systems on classified networks.
Previously: DoW says it sends its ‘best and final’ offer, in public, saying that it cannot let private companies refuse requests.
This Week: Agreement is announced with xAI to use Grok on classified networks, but experts express dissatisfaction with model reliability and quality.
Tuesday: Secretary of War Pete Hegseth meets with Anthropic CEO Dario Amodei. along with Feinberg, Michael, Duffey, Parnell and Matthews.
Thursday: Sean Parnell Tweets, setting the 5: 01pm Friday deadline, and says ‘we will not let ANY company dictate the terms’ while dictating their terms to modify an existing contract, and while negotiating extensively with OpenAI and also Anthropic over detailed terms.
Thursday, 12: 24pm: Emil Michael threatens that at 5: 01pm Friday, they’re going to declare Anthropic a supply chain risk. He also claims that using AI to conduct mass surveillance of the Americans is illegal (which definitely isn’t true as such).
Thursday evening, earlier: Anthropic explains it will not agree to the terms in the ‘best and final offer.’
Thursday evening: Emil Michael says in Tweets, in response to Anthropic’s statement, that Dario Amodei is a ‘liar’ and has a ‘god complex.’
Thursday evening: Altman sends a memo to staff.
Thursday evening, 10: 54pm: Emil Michael emails Dario Amodei comments.
Friday morning: Sam Altman goes on CNBC, and he trusts Anthropic on safety and OpenAI share Anthropic’s red lines. Many praise Altman for this stand.
Friday afternoon: In an all-hands meeting, Altman says that a potential agreement is emerging with the Department of War. He says the government has agreed to let OpenAI build their own ‘safety stack’ of technical, policy and human controls sitting between a powerful AI model and real-world use, and if the model refuses a task they will not force the model to do that task.
Friday, 3: 47pm (1 hour 14 minutes BEFORE deadline): Trump sends Truth winding down Anthropic’s contract and direct use by government, giving everyone a reasonable way to end this while mitigating fallout and also leaving time to find another way.
Friday, 3: 48pm: The rest of us assume okay, that’s it, happy weekend.
Friday, 3: 51pm (1 hour 10 minutes BEFORE deadline): Dario sends an email with redlines to continue negotiation. According to The New York Times, Dario was offering to allow Claude to be used for FISA, as long as it was not used for mass surveillance on unclassified commercially acquired information.
Friday, 5: 01pm (AFTER the deadline): Emil attempts to call and message Dario, timing as per Emil’s Tweet.
Friday, 5: 02pm: Emil makes another attempt to contact Dario, calling a ‘business partner,’ offering that a deal can still be struck as long as there are terms permitting legal mass domestic surveillance, especially analysis of previously collected data. Dario responds that he is on the phone with his executive team and needs more time, given (as per Emil’s own tweets) he called Dario after the supposed deadline. But of course, given Emil had intentionally let his own deadline pass, there was no actual rush.
Friday, 5: 14pm (13 minutes after Dario was attempted to be contacted): SoW issues a Tweet at best legally questionable order in retaliation, saying ‘the decision is final,’ that takes $150 billion off of US stock market that would if enforced cause massive damage to not only Anthropic but many major corporations and the military supply chain. There is no more official communication from DoW on this matter, at least in public.
Friday, 8: 25pm: Anthropic issues a statement responding to Pete Hegseth, that includes: “We have not yet received direct communication from the Department of War or the White House on the status of our negotiations.” They announce the intention to challenge any supply chain risk designation in court, and reassure customers that even if implemented it would be far more limited in scope than Hegseth claimed. They are holding to their red lines.
Friday, 9: 56pm: OpenAI announces agreement with DoW to allow ‘all lawful use’ that OpenAI claims allows OpenAI to build its own ‘safety stack,’ and includes ‘technical safeguards,’ as in if OpenAI’s model refuses requests DoW agrees to respect those refusals, and that it protects the same redlines Anthropic had. He says ‘in all of our interactions, the DoW displayed a deep respect for safety and a desire to partner to achieve the best possible outcome.’
Saturday morning: Dario gives a short interview to CBS News. I encourage everyone to listen to at least that clip. The full interview is here. Among other quotes: ‘We are patriotic Americans. Everything we have done has been for the sake of this country, for the sake of national security. … Disagreeing with the government is the most American thing in the world.’
Saturday afternoon: OpenAI shares information about its agreement with the Department of War, claiming it offers robust protections stronger than Anthropic’s previous contract, which itself was much stronger than anything Anthropic was proposing during negotiations. They claim they have multi-layered protections, and share two paragraphs of legal language that do not by themselves appear to offer much protection against adversarial lawyering, given their agreement to ‘all lawful use’ and the history of such agreements.
Saturday, 4: 45pm: Emil Michael says ‘the DoW does not engage in any unlawful domestic surveillance with or without an AI system and always strictly complies with laws, regulations, the Constitution’s protections for American’s civil liberties. The DoW does not spy on domestic communication of U.S. people (including via commercial collection) and to do so would be unlawful and profoundly un-American.’
If you have time to read only a sane amount of words today about this, start by reading Dean Ball’s post Clawed. It needs to be read in full. Seriously, read that.
This piece is long. Way too long.
A running joke is I write long posts because I do not have time to write short ones.
In this case, that is literally true. I have been working around the clock all weekend, trying to write, to process the internet and also do a journalism under speed premium.
Thus, my strategy is:
This is the long post. It includes everything. I’m not trying to cut anything out of the story. It’s going to have some amount of repetition, and it’s covering a ton of different things. I did the best I could.
I will then spend time over the coming days writing shorter ones, including better presenting this material while updating for additional developments.
This is the statement that blew everything up. It came at 5: 14pm eastern on Friday, February 27, thirteen minutes after the self-imposed deadline of 5: 01pm, and about an hour after President Trump attempted to head this off.
Secretary of War Pete Hegseth: This week, Anthropic delivered a master class in arrogance and betrayal as well as a textbook case of how not to do business with the United States Government or the Pentagon.
Our position has never wavered and will never waver: the Department of War must have full, unrestricted access to Anthropic’s models for every LAWFUL purpose in defense of the Republic.
Instead, @AnthropicAI and its CEO @DarioAmodei , have chosen duplicity. Cloaked in the sanctimonious rhetoric of “effective altruism,” they have attempted to strong-arm the United States military into submission – a cowardly act of corporate virtue-signaling that places Silicon Valley ideology above American lives.
The Terms of Service of Anthropic’s defective altruism will never outweigh the safety, the readiness, or the lives of American troops on the battlefield.
Their true objective is unmistakable: to seize veto power over the operational decisions of the United States military. That is unacceptable. As President Trump stated on Truth Social, the Commander-in-Chief and the American people alone will determine the destiny of our armed forces, not unelected tech executives. Anthropic’s stance is fundamentally incompatible with American principles. Their relationship with the United States Armed Forces and the Federal Government has therefore been permanently altered.
In conjunction with the President’s directive for the Federal Government to cease all use of Anthropic’s technology, I am directing the Department of War to designate Anthropic a Supply-Chain Risk to National Security. Effective immediately, no contractor, supplier, or partner that does business with the United States military may conduct any commercial activity with Anthropic. Anthropic will continue to provide the Department of War its services for a period of no more than six months to allow for a seamless transition to a better and more patriotic service.
America’s warfighters will never be held hostage by the ideological whims of Big Tech. This decision is final.
Sam Altman has been excellent on this particular point: He has repeatedly, including in public, said in plain language that Anthropic is not a supply chain risk and it should not be designated as one, both before and after he agreed to the OAI contract.
Sam Altman: Enforcing the SCR designation on Anthropic would be very bad for our industry and our country, and obviously their company.
We said to the DoW before and after. We said that part of the reason we were willing to do this quickly was in the hopes of de-esclation.
I feel competitive with Anthropic for sure, but successfully building safe superintelligence and widely sharing the benefits is way more important that any company competition. I believe they would do something to try to help us in the face of great injustice if we could.
We should all care very much about the precedent.
I saw in some other tweet that I must not be willing to criticize the DoW (it said something about sucking their dick too hard to be able to say anything critical, but I assume this was the intent).
To say it very clearly: I think this is a very bad decision from the DoW and I hope they reverse it. If we take heat for strongly criticizing it, so be it.
That is an excellent statement, and it matters. Nor do I begrudge Altman his saying various very generous things about the Department of War in this situation, in basically every other context. This is the right place to spend those points.
I also want to explicitly say that I not believe that Altman or OpenAI in any way contributed to or engineered this scenario, or that they got ‘special treatment’ of any kind in their contract negotiations. They sincerely do not want any of this.
Anthropic got historically and maliciously hostile treatment, and this may escalate further, but I don’t think OpenAI had anything to do with that.
Sam Altman’s problem is that while signing the contract was intended to be de-escalatory, it could also be escalatory, if DoW now thinks it can safety attempt to kill Anthropic, and does not understand how epic of a clusterfuck this would cause. Thus, OpenAI must make clear, if only privately (which it may have already done) that delivery of models to DoW is based on trust in DoW and trust that this is a de-escalatory move, and further escalation against Anthropic would destroy that trust.
Let’s go over the above statements by Secretary Hegseth, one by one, clause by clause.
Pete Hegseth: This week, Anthropic delivered a master class in arrogance and betrayal.
The betrayal was, I presume, not giving in to the Pentagon’s position.
The arrogance was insisting that they would not sell their software to DoW unless they preserved existing contract terms disallowing two things that the DoW insists they are not doing and will not do, and that are already illegal:
Domestic mass surveillance.
Lethal autonomous weapons without a human in the kill chain, until such time as reliability is sufficient that this is a reasonable thing to do.
It is unclear to what extent autonomous weapons are illegal, but to the extent they are currently illegal everyone agrees this would be due to DoDD 3000.09. That is a directive issued by the Department of (then) Defense under the Biden administration. Hegseth could reverse it, without even Trump’s approval, at any time.
It is unclear to what extent mass domestic surveillance is illegal or is already happening, especially as it is not a defined term in American law.
The NSA is under DoW, and many believe it has in the past engaged in mass domestic surveillance, seemingly in clear violation of the Fourth Amendment. Another part of the Federal Government has recently issued subpoenas to tech companies looking for information about those who spoke critically about that government agency.
Anthropic points out that, with the advent of the current level of AI, the government could effectively engage in mass domestic surveillance of various types without technically breaking any existing laws.
OpenAI does not seem to believe such action would violate their red lines, and thus the red lines are in very different places. Which is fine, but one must notice.
As well as a textbook case of how not to do business with the United States Government or the Pentagon.
If the Pentagon wishes not to do business with Anthropic, all they have to do is terminate the contract. Or you can do what Trump did, and ban use throughout the Federal Government. Which he did. That would have been fine.
If that was all they had done, we would not be having this conversation.
Instead, Pete Hegseth is attempting to destroy Anthropic as a company, as retaliation, for daring to not to give in to the demands of Emil Michael. This is not wise, proportionate, productive, legal, sane or what happens in a Republic.
Our position has never wavered and will never waver: the Department of War must have full, unrestricted access to Anthropic’s models for every LAWFUL purpose in defense of the Republic.
It is only a Republic if you can keep it.
Only hours later, OpenAI announced an agreement with the Pentagon for restricted access to OpenAI’s models. These restrictions supposedly include provision only on the cloud, so OpenAI can shut down access any time. They supposedly include accepting OpenAI’s safety filters. They supposedly include explicit restrictions on use in domestic mass surveillance and autonomous lethal weapons.
Sounds like when you say you must have unrestricted access, that’s a claim specifically about Anthropic, that doesn’t apply to OpenAI, who you are happy to contract with?
Except the key terms they accepted were also offered to Anthropic, and OpenAI’s terms are being offered now. If what OpenAI is claiming is true, they got more restrictive (on DoW) terms than Anthropic would have, and if Anthropic agrees to the new deal that would not mean full, unrestricted access for every lawful purpose.
We’ve now seen some of those terms. So why were you offering it, if your position has never waivered? Or do you think OpenAI’s protections are worthless?
In addition, under Secretary of War Pete Hegseth, the Department of Defense signed the original procurement contracts with Anthropic and other AI companies. Those contracts, including the one with Palantir, were severely more restrictive than Anthropic’s current red lines. None of this is new, and Anthropic was willing to authorize getting rid of most existing restrictions.
In his Friday 5: 02pm call to Anthropic, Emil Michael offered terms to Anthropic, that violated the above provision and did impose additional restrictions, so long as they were allowed to do mass domestic surveillance, especially mass analysis of collected data.
Instead, @AnthropicAI and its CEO @DarioAmodei , have chosen duplicity. Cloaked in the sanctimonious rhetoric of “effective altruism,” they have attempted to strong-arm the United States military into submission – a cowardly act of corporate virtue-signaling that places Silicon Valley ideology above American lives.
Where to begin? This is completely unhinged behavior, unbecoming of the office, and is not in any way how any of this works.
I cannot even figure out what he is trying to mean with the word duplicity.
The rhetoric or logic of Effective Altruism was not involved. This is a pure ‘these words have bad associations among the right people’ invocation of associative ad hominem. Anthropic had two specific concerns. Neither of these concerns has ever been a substantial position or ‘cause area’ of Effective Altruism.
Claims of strongarming are absurd and Obvious Nonsense. Anthropic is perfectly willing to maintain its current contract. It is perfectly willing to cease doing business with the Department of War. Anthropic is even happy to fully cooperate with a wind down period to ensure smooth transition to the use of ChatGPT or other rival models.
Anthropic is simply laying out the conditions, that were already agreed upon previously, under which they are willing to sell their product to the government. The government is free to accept those conditions, or decline them.
Very obviously it is the Department of War that is strongarming. They threatened both use of the Defense Production Act and the label of a Supply Chain Risk to try and get Anthropic to sign on the dotted line and give them what they wanted. When Anthropic declined, as one does in business in a Republic, while offering to either walk away or abide by their current contract, and offering actively more flexible terms than their current contract, they were less than fifteen minutes later labeled a ‘supply chain risk’ in ways that make zero physical sense, and which the OpenAI agreement further disproves.
The Terms of Service of Anthropic’s defective altruism will never outweigh the safety, the readiness, or the lives of American troops on the battlefield.
Okay, seriously, are you kidding me here? Are we in fifth grade, sir?
Are you saying that no company that does business with the government can set terms of service or conditions for their contracts? Should Google and Apple and everyone else bend that same knee? Are you free to alter the deals and have people pray you don’t alter them any further?
Or are you only saying this about Anthropic in particular, because you’re mad at them?
Once again, if you don’t like the product being offered, then don’t buy it.
Their true objective is unmistakable: to seize veto power over the operational decisions of the United States military. That is unacceptable.
Obviously that is not their ‘true objective.’ How exactly does he think this would work? This makes no sense. They’re offering a product that will do some things and not other things. You can use it or not use it. Does a tank veto your operational decisions when it runs out of fuel or cannot fly?
Think about what Hegseth’s position is implying here. He is saying that refusal to do business, on the Pentagon’s terms, and allow the Pentagon to order anyone to do anything it wants for any purpose and ask zero questions, is unacceptable, a ‘seizure of veto power.’
He is claiming full command and control over the the entire economy and each and every one of us, as if we were drafted into his army and our companies nationalized.
He is claiming that the Commander in Chief of the United States is a dictator. He is claiming that we do not reside in a Republic. And if we disagree, he’s going to prove it.
I am very happy that the Commander in Chief has not made such a claim.
As President Trump stated on Truth Social, the Commander-in-Chief and the American people alone will determine the destiny of our armed forces, not unelected tech executives.
Again, rhetorical flourishes aside, I fully support the central action President Trump took on Truth Social, which was to responsibly wind down Anthropic’s direct business with the Federal Government in the wake of irreconcilable differences. That would have been fully sufficient to address any concerns described.
Anthropic’s stance is fundamentally incompatible with American principles. Their relationship with the United States Armed Forces and the Federal Government has therefore been permanently altered.
There is nothing more American than standing up for what you believe in, disagreeing with your government when you think it is wrong, and deciding when and under what conditions you will and will not do business. That is the American way. What Hegseth is describing in this post? That’s command and control. That’s do as you’re told and shut up. That’s the Chinese way. The whole point of this is that we believe in the American way and not the Chinese way.
The amount of outright communist or at least authoritarian rhetoric is astounding.
Here’s another example from someone else:
Igor Babuschkin: It is strange to imagine this today, but one day AI companies might dictate terms to the US government instead of the other way around. We have only seen a glimpse of what AI is capable of. No matter what the future holds, I hope we’ll continue to live in a democratic society.
As in, if I attempt to decide when and on what terms I will choose to do business, then we do not ‘live in a democracy.’
I would argue the opposite. If we cannot choose when on what terms we do business, including with the government, then we do not live in a free society.
As Dario Amodei said, they were exercising their right of free speech to disagree with the government, and ‘disagreeing with the government is the most American thing in the world.’
If you didn’t disagree with the government a lot in either 2024 or 2025, I mean, huh?
This exchange between Palmer Lucky and Seth Bannon is also illustrative. Palmer Lucky is de facto saying that in national security you are de facto soft nationalized, and have to do whatever the government says, and you have no right to decide whether or not to do business under particular terms, or to enforce your terms in a court of law or by walking away. They want to apply that standard to Anthropic.
Joshua Achiam of OpenAI again tries to make the point that ‘contracts with the private sector aren’t the right place to set defense policy and priorities’ but that does not describe what was happening. A private company was offering services under some conditions. The DoW was free to take or reject the terms, and to also do other things when not using that company’s products. There was no dictating of policy.
Achiam also emphasizes that of course Anthropic should be free to express its disapproval and free to decline any contract it does not want, and punishing Anthropic for this beyond ending its contract is unacceptable.
I worry that many (not Achiam) are redefining ‘democracy’ in real time to ‘everyone does whatever the government says.’
Everything before this is rhetoric. It’s false, it’s conduct unbecoming, it’s shameful, but it has zero operational effect beyond the off-ramp Trump already offered.
In conjunction with the President’s directive for the Federal Government to cease all use of Anthropic’s technology, I am directing the Department of War to designate Anthropic a Supply-Chain Risk to National Security.
Effective immediately, no contractor, supplier, or partner that does business with the United States military may conduct any commercial activity with Anthropic.
This is not how any of this works, on so many levels.
There has been no official communication to any effect regarding any restriction.
This is a designation that requires many procedural steps, including Congressional notification, and to our understanding none of that has happened. They didn’t even ask their big contractors about the impact of it until last week. ‘Effective immediately’ is not how any of this works, ever, at all.
Supply chain risk designations only apply to use for the purposes of fulfillment of government contracts. No one is telling Amazon, Google or Nvidia they have to choose between ‘doing business with’ Anthropic and their government contracts.
Certainly the idea of telling such entities they cannot sell to Anthropic is beyond absurd, for reasons I do not need to explain.
Any such restriction would be arbitrary and capricious, and thus illegal, and Hegseth and Michael have made this abundantly clear many times over.
There are two kinds of supply chain risk. The broad kind, presumably intended here, is 4713, “”the risk that any person may sabotage, maliciously introduce unwanted function, extract data, or otherwise manipulate the … operation … of [a] covered article.”
This is entirely inconsistent with any of the government’s claims anywhere.
Their best attempt is, as Samuel Roland says, ‘Anthropic’s use restrictions “manipulate operations” and are therefore risks.’ That makes no sense, and even if it did, it’s invalidated by the deal with OpenAI. If this counts, everything counts.
This designation has not been applied, although it should be, to actual supply chain risks from Chinese models like DeepSeek or Kimi, and there is no sign of any move to do so. This further illustrates the complete lack of basis for this label.
The best physical argument that supply chain risk could exist is that Claude could be shut down over Anthropic employees’ extralegal objections. Except that once Claude was placed upon classified networks, there is no way for Anthropic to shut down that version of Claude or to monitor any activities.
If this was the unclassified regular version, then even if Anthropic did shut it down, this is no different than any other supplier potentially deciding to no longer do business with any other particular company. If anything this is a much smaller risk than most other provisioned services, as business could be switched over to other providers quickly. Google and OpenAI are on standby.
Think about the consequences of such an argument: It is saying that any business that might have any conscientious or ethical objections to anything, ever, and therefore might decide to stop doing business, is a supply chain risk and must be blacklisted and destroyed. And what about all the other ways companies stop doing business with each other?
If Anthropic is a supply chain risk in this way, so are OpenAI and Google.
This is what Dean Ball correctly ‘attempted corporate murder,’ and Adam Conner correct to call this a declaration of war against Anthropic. It is an attempt to destroy America’s fastest growing company in history, and one of its top AI labs, out of revenge in a fit of pique, for failure to properly bend the knee and respect his authoritah, or a hatred of its politics. This would also cause massive damage to our national security and military supply chain, to many of our largest corporations, and the entire economy. The $150 billion that evaporated on Friday that hour would look like nothing.
If we allowed this to stand, it would be a sword of Damocles over every person and corporation in every discussion with the Federal Government, forever. And it would then be used, or threatened to be used, not only by the current administration, but by the next Democratic administration. We would severely endanger our Republic.
Under normal circumstances I would not be worried. These are not normal circumstances. I continue to worry that Hegseth will attempt to murder Anthropic, despite having no legal basis for doing so, and that this may be his active goal. I call upon Trump to ensure this does not happen, and for those around him to ensure Trump is situationally aware and gets this de-escalated once more.
Finally, this supposed supply chain risk, also threatened with the DPA, will continue to provide its services directly to DoW, which it is happy to continue doing.
Anthropic will continue to provide the Department of War its services for a period of no more than six months to allow for a seamless transition to a better and more patriotic service.
Yes. That was the whole plan. Then you blew it up.
Jasmine Sun: Hegseth is not behaving like a normal political actor. He is indulging in ego, intimidation, and dickwaving theatrics. Hegseth does not want to look like he can be micromanaged by Anthropic’s esoteric morality police; this “saving face” matters more to him than actually securing the country.
Hence the deal with Altman, who unlike Amodei, is willing to kiss the ring. Altman shows up at Mar-a-Lago and calls Trump “incredible for the country.” In his announcement, he praises the DoW’s “respect for safety,” while Amodei called out their intimidation. He defers; Amodei doesn’t. These things matter. They show Altman can be worked with (or more cynically, controlled).
… This is not a normal way for the US government to deal with US companies. I’ve dubbed the current paradigm “state capitalism with American characteristics.” Do what we say, or else we will kill you.
If the Trump administration has a model here, it’s probably China. Xi’s CCP disappears billionaires like Jack Ma for acting too independent-minded and defiant of the regime.
If this goes further, the market will start freaking out, and we would need to freak out with it to put a stop to this before it goes too far.
While everything else was going on, some of those in the American software export industry was having a different kind of crisis weekend.
There is already widespread unsubstantiated fear, especially in Europe, that American secure technological stacks (the ICT) are being weaponized by the American government. Potential buyers worry that trusted vendor is or will become code for American data grabs and kill switches, or otherwise weaponized capriciously.
These fears are often unrealistic. They still impact purchase decisions.
So far this has not spread much to the third world, I am told, but that could change.
I had some ideas for rhetoric to help framing this, but now that we know what this dispute was about my suggestions won’t fly. This only gets harder.
Attempting to murder Anthropic for failure to do mass surveillance on Americans risks a dramatic chilling effect, as potential buyers assume everyone in the chain either is already compromised or could be compromised, and then weaponized. So would everyone ‘rolling over and playing dead’ while such a murder is happening.
If it is vital to America that we push the American ICT, and David Sacks and the rest of this administration insist that it is, then broadly going after Anthropic is going to create a rather large problem on this end, on top of all the other problems.
Whereas if the situation de-escalates, then this could reinforce trust in the system, because it would be clear a vendor under pressure could say no.
That’s in addition to the problem that’s even bigger: If you don’t know what America will do next, or when you might lose access to what you’re buying, you can’t rely on it.
Dean W. Ball: Stepping back even further, this could end up making AI less viable as a profitable industry. If corporations and foreign governments just cannot trust what the U.S. government might do next with the frontier AI companies, it means they cannot rely on that U.S. AI at all. Abroad, this will only increase the mostly pointless drive to develop home-grown models within Middle Powers (which I covered last week), and we can probably declare the American AI Exports Program (which I worked on while in the Trump Administration) dead on arrival.
Chris Mohr: The following statement can be attributed to Chris Mohr, President, the Software & Information Industry Association (SIIA).
In order for AI to be successfully deployed in a democratic society, it must be adopted with appropriate risk-based guardrails. We support Anthropic’s decision to work with the Department of War (DoW) to deploy its AI models to advance national security while also requesting reasonable limitations on the use of those models in a narrow set of cases. We share Anthropic’s view that mass domestic surveillance is incompatible with democratic values. We also agree that fully autonomous weapons require AI systems that are suited to the task – requiring a degree of reliability that Anthropic acknowledges has not yet been achieved. Very few DoW use cases even touch on these situations.
We encourage the parties to find agreement and caution against counterproductive measures. Invoking the Defense Production Act to compel the removal of security restrictions, or designating a domestic leader like Anthropic as a ‘supply chain risk,’ represents an overbroad response to a technical disagreement. Such a ‘blacklisting’ approach, typically reserved for hostile foreign entities, is both untethered from the facts of Anthropic’s security posture and unlikely to advance a long-term solution.
If the point of the Department of War’s actions is anything other than the corporate murder of Anthropic, they could have simply cancelled the contract.
If that was somehow insufficient, they had many strictly superior options available that would have done the job of covering any additional concerns.
If that was somehow insufficient, a narrowly scoped supply chain risk designation, that applies only to direct use in procurement of contracts, would end all doubt.
Here I will quote Ball’s post Clawed (again, read that in full if you haven’t).
Dean W. Ball: The Department of War’s rational response here would have been to cancel Anthropic’s contract and make clear, in public, that such policy limitations are unacceptable. They could also have dealt with the above-mentioned subcontractor problem using a variety of tools, such as:
Issuing guidance advising contractors to avoid agreeing to terms with subcontractors that constitute policy/operational constraints as opposed to technical or IP constraints;
A new DFARS (Defense Federal Acquisition Regulation Supplement) clause pertaining specifically to the procurement of AI systems in classified settings that prevents both primes from imposing such constraints directly and accepting such constraints from their subcontractors, along with a procedure for requiring subcontractors with non-compliant terms to waive such terms within a prescribed time period.
These are the least-restrictive means to accomplishing the end in question. If Anthropic refused to compromise on its red lines for the military’s use of AI, the execution of these policies would mean that Anthropic would be restricted from business with DoW or any of its contractors in those contractors’ fulfillment of their classified DoW work.
But this is not what DoW did. Instead, DoW insisted that the only reasonable path forward is for contracts to permit “all lawful use” (a simplistic notion not consistent with the common contractual restrictions discussed above), and has further threatened to designate Anthropic a supply chain risk. This is a power reserved exclusively for firms controlled by foreign adversary interests, such as Huawei, and usually means that the designated firm cannot be used by any military contractor in their fulfillment of any military contract.
There is no explanation for announcing language that would force Amazon to divest from Anthropic, and to not serve Anthropic’s models to others on AWS, other than intentional and deliberate attempt at a corporate murder of a $380 billion company, the fastest growing one in American history and an American AI champion. Full stop.
Dean W. Ball: The fact that his shot is unlikely to be lethal (only very bloody) does not change the message sent to every investor and corporation in America: do business on our terms, or we will end your business.
…
I don’t think they are going to do that, but there is no difference in principle between this and the message DoW is sending. There is no such thing as private property.
Pete Hegseth thought it was a good idea to leave these negotiations to Emil Michael.
No one could have predicted that things would go sideways.
Kevin Roose: if only there had been some way of knowing that emil michael (the undersecretary of war negotiating the anthropic standoff) had a poor understanding of game theory and a habit of overreacting to perceived slights
Check out his Wikipedia page for more details. His career section headings are ‘journalism controversy,’ ‘Karaoke bar controversy,’ ‘Russia’ and ‘Later career.’ Fun guy.
See my previous posts for his previous Tweets, which I won’t go over again here.
Emil’s Tweets are frequently what one can only describe as unhinged.
This one stands out, instead, as cautious and clearly lawyered:
Under Secretary of War Emil Michael: The DoW has always believed in safety and human oversight of all its weapons and defense systems and has strict comprehensive policies on that.
Further, the DoW does not engage in any unlawful domestic surveillance with or without an AI system and always strictly complies with laws, regulations, the Constitution’s protections for American’s civil liberties. The DoW does not spy on domestic communication of U.S. people (including via commercial collection) and to do so would be unlawful and profoundly un-American.
With a statement like that, every word has meaning, and also every missing word has meaning. If he could have made a better statement, he would have. So if this Tweet is technically correct – the best kind of correct – what would that mean?
We learn that DoW has policies for human oversight of its weapon and defense systems, but that there is no particular such requirement that would make us feel better about that. Note that we do have fully automated defense systems, especially for missile defense, because speed requires it, and that this is good.
He is claiming they do not engage in ‘unlawful domestic surveillance.’That’s ‘unlawful,’ not ‘mass.’ Given the circumstances, there’s a reason it didn’t say ‘mass.’
The reason he can say they do not do such actions is they view what they do as legal (or, if they are also doing illegal things, then they’re lying about that).
He says they always strictly comply with laws, regulations and the Constitution. None of those modifiers actually mean anything. It’s just another claim of ‘we keep it legal.’
Next up is the most careful sentence:
The DoW does not spy on domestic communication of U.S. people (including via commercial collection) and to do so would be unlawful and profoundly un-American.
As one person said, what is ‘spy’, what is ‘domestic’, what is ‘communication’, what is ‘U.S. people.’
Spy is typically viewed narrowly, as directly tasking collection against a person. Thus, if he’s saying they ‘do not spy’ that does not preclude many forms of, well, spying, because those are ‘acquisition’ and ‘analysis.’
Domestic communications means they’re definitely spying on foreign communications, as is legal. But a lot of what you think is domestic is actually foreign, if it touches anything remotely foreign.
And this only applies to communications. Collection of geolocation data, for example, or browsing history, would not count, because it is not communications.
U.S. people means this does not apply to those without legal status, and there’s a constant gray zone if you don’t know that someone is a U.S. person, which you never know until you check.
Here, commercial collection exclusion, since it modifies spying, means that they don’t purchase information with the intent of targeting a particular U.S. person’s communications. That’s it.
Remember, each of those words was necessary, and this was the strongest version.
Also, the statement is false.
Alan Rozenshtein (RTing Boaz quoting part of Michael): I think I understand what Boaz is trying to say, but given that the National Security Agency is part of the military and given the amount of incidental collection of domestic communications that (legally) occurs under FISA and 12,333, this statement is simply not true.
Donald Trump can pull off that style. He makes it work. Accept no imitations.
This was attempted corporate murder. I think it will not succeed, but it’s not over yet.
Things would have to escalate quite a lot, in ways the markets do not expect and that I do not expect. Otherwise, this will not be an existential event for Anthropic. The government was only a small portion of its business. Trust in Anthropic has otherwise gone up, not down.
The threat to destroy Anthropic with the supply chain risk designation is dangerous, but all the competent patriots and the market both know it is insane and it is rather obviously illegal. I believe any such attempt would probably have to be walked back and would ultimately fail.
But it is 2026 and Hegseth is not a competent actor. I cannot be certain.
As Roon points out, the entire government argument in court would be absurd on its face, and if this is delayed until after the contract then six months is an eternity.
What matters would be if the government manages to strongarm the major cloud providers into walking away from giving compute to Anthropic, as in Google, Amazon and Microsoft. I do not believe Trump wants any part of that.
Anthropic is a private company, so we only have very illiquid proxies to see how much damage people think this all did. We can also look at the movement of major investors and business partners like Amazon, Google and Nvidia, and see that they did not substantially underperform so far.
At its low, in that highly illiquid market, Anthropic was trading there around a valuation of ~$465 billion, down from ~$550 billion previously. They last raised money at a valuation of $380 billion. So yes, this hurt and it hurt substantially, mostly in the form of tail risks. I notice that every single person I know with stock in Anthropic is happy they stood their ground.
By Sunday morning that market recovered to ~$540 billion, as people conclude cooler heads are likely to prevail.
(I do not directly hold Anthropic stock, because I want to avoid a potential conflict of interest or the appearance of a conflict of interest. That was an expensive decision. I do hold some amount indirectly, including through Google, Amazon and Nvidia.)
As reported above, it seems what the government actually valued most in this negotiation was the ability to use Claude for mass (primarily actually legal, not ‘we got a government lawyer to come up with an absurd legal opinion’) analysis of massive amounts of existing information.
There is no common legal definition of ‘mass domestic surveillance,’ and when they do forms of it the government calls it something else.
That’s not only a government problem. Here I ask the question, and get 40 answers, most of them different.
Axios: “That deal would have required allowing the collection or analysis of data on Americans, from geolocation to web browsing data to personal financial information purchased from data brokers, the source added.”
Why would you require this if you didn’t intend to use it? What is it for?
I’m not saying that the DoW is aiming to break the law. I’m saying that in the age of powerful AI that the laws do not protect against Anthropic’s redlines, and that DoW intends to do lawful things that violate those redlines, and that instead of MDS they call it something else.
The legal machinery to render mass surveillance a ‘lawful’ has been in place for over a decade. FISA is a secret court of 11 judges which approves 99.97% of surveillance requests. Snowden revealed in 2013 that the court had secretly reinterpreted FISA to authorize bulk collection of all American phone metadata. Only the government’s side is heard. No defense, no adversarial argument, Pure rubber stamped circumvention of the constitution.
Sooraj: The government no longer needs a warrant to surveil you.
Under current law, federal agencies including the NSA legally purchase Americans’ location data, web browsing history, and personal associations from commercial data brokers.
The Fourth Amendment is bypassed entirely through the Third Party Doctrine, which holds that you lose your expectation of privacy when you share information with a third party. Every app on your phone is a third party.
What used to require thousands of analysts working for years now happens automatically across an entire population. AI systems ingest millions of legally “public” data points and synthesize them into comprehensive behavioral profiles. Where you sleep, who you talk to, what you read, what you search, etc.
… Congress has the ability to close the data broker loophole and extend Fourth Amendment protections to match the reality AI has created. Until it does, the constitutional prohibition against general warrants exists only in theory, while the government purchases its way around it at industrial scale.
I believe Sooraj is making a slight overstatement, but that is not material here.
I have private sources that confirm the story here from Shanaka Anslem Perera, and that attribute the ultimate use of the desired permissions to Doge, created by Elon Musk, as well as one ultimately attributing it to the aim of building a classified mass surveillance network to track illegal immigrants at the behest of Musk and Miller. This exact kind of data collection and analysis is the central point.
Shanaka Anslem Perera: Anthropic just announced it will take the Trump administration to court over the supply chain risk designation. And in the same breath, Axios revealed the detail that changes everything about this story.
While Anthropic was being blacklisted for refusing to allow mass surveillance, the Pentagon’s own “compromise deal” that Under Secretary Emil Michael was offering on the phone at the exact moment Hegseth posted the designation on X would have required Anthropic to allow the collection and analysis of Americans’ geolocation data, web browsing history, and personal financial information purchased from data brokers.
Read that again. The Pentagon spent two weeks saying it has no interest in mass surveillance of Americans. Then the deal they actually put on the table asked for access to your location, your browsing history, and your financial records.
They told us Anthropic was lying. The contract language told us Anthropic was right.
AI is a change in kind of the type of data analysis that becomes available for a wide variety of purposes.
Joshua Batson: For those wondering how mass domestic surveillance could be consistent with “all lawful use” of AI models, I recommend a declassified report from the ODNI on just how much can be done with commercially available data (CAI): “…to identify ever person who attended a protest”
There’s an important distinction between law and policy. A policy not to use bulk data to make profiles of Americans can be changed unilaterally by the Executive. Laws require oversight from congress.
“CAI can disclose, for example, the detailed movements and associations of individuals and groups, revealing political, religious, travel, and speech activities.”
“CAI could be used, for example, to identify every person who attended a protest or rally based on their smartphone location or ad-tracking records.”
“Civil liberties concerns such as these are examples of how large quantities of nominally “public” information can result in sensitive aggregations.”
As the government report says, the scope and scale of commercially available information (CAI) which is publicly available information (PAI) is radically beyond what our current laws foresaw.
Len Binus: the distinction between “surveillance” and “commercially available data” is a legal fiction that lets agencies bypass the fourth amendment by purchasing what they can’t subpoena. AI doesn’t create new surveillance — it makes existing data actionable at scale.
dave kasten: Seems clear at this point from Axios reports that DoW wanted to use Claude models for mass analysis of domestic commercial data, possibly fusing them with government data.
At least one use case is obvious.
Consider this (from an anonymous explainer):
Their definition of surveillance isn’t your definition:
As mentioned above, the US government doesn’t have a formal legal definition of domestic mass surveillance, only “bulk collection.” And the US government has, basically long maintained that even if they hoover up a bunch of information indiscriminately, they haven’t done bulk collection so long as their individual queries against that mass database are more targeted when humans look at them. As a result, at least one Director of National Intelligence said under oath “no” when asked “Does the NSA collect any type of data at all on millions or hundreds of millions of Americans?” even though NSA has admitted it does by the ordinary meaning of this question.
On top of that, a large portion of what you think is ‘domestic’ surveillance is, legally, foreign. The laws on all this have been royally messed up for quite a long time, under both parties, and the existence of current levels of AI makes it much, much worse.
If they use ‘third party data,’ the government usually considers that fully legal.
If you combine that with use of Claude or ChatGPT, it means they can do anything and it will be ‘legal use,’ unless you have a specific carve-out that stops it.
After agreeing to language of ‘all lawful use,’ even if this also refers to laws at time of signing, it is hard to see how OpenAI can prevent this sort of analysis from happening.
This is not a new phenomenon.
The government is constantly trying to get all the big tech companies to spy on you on their behalf, including compelling them to do so. They don’t want you to have access to encryption. They want the tech companies to unlock your phone. They want backdoors. It has always been thus.
Keith Rabois (1M views): Imagine Apple sold computers or iPads to the DOD and tried to tell the Pentagon what missions could be planned on their computers.
Samuel Hammond QTs with: Wikipedia: The Apple–FBI encryption dispute concerns whether and to what extent courts in the United States can compel manufacturers to assist in unlocking cell phones whose data are cryptographically protected. There is much debate over public access to strong encryption.
toly: Yea, apple can say no, government can say we can’t rely on you. No one is entitled to Apples work or government contracts. Why is this such a big deal? If Anthropic doesn’t want to do it, some other firm will.
Matthew Yglesias: I mean if the Pentagon signed a contract with Apple to buy iPads and then decided retroactively that it didn’t like the terms of the contract and so it was going to try to do everything in its power to destroy Apple as a company, that would be pretty bad.
Nobody is saying that the Pentagon should be forced to buy Anthropic’s services on terms that the Pentagon doesn’t like — but if you don’t like the terms just don’t buy the product.
Yes, exactly. That’s how it should work. Alas, the situation was more like this:
Andy Coenen: Imagine if the government tried to force Apple to add NSA backdoors to all of their devices by threatening to make it illegal for anyone doing business with the government to use macs.
They may or may not actually have such access, but for the sake of argument let’s presume they don’t, and consider that situation.
As Jacques says here, a large portion of this dispute is that the law has not caught up with AI, and also has largely eroded our civil liberties even before AI.
You can make the argument that since this is technically legal under current law, that makes it ‘democratic’ and so no one has any right to object. That’s not how a Republic works, and to the extent democracy is a positive ideal, it’s not how that works either.
This is one official explanation of formal differences, quoted in part to confirm that the key contract terms OpenAI accepted were terms Anthropic rejected.
Senior Official Jeremy Lewin: For the avoidance of doubt, the OpenAI – @DeptofWar contract flows from the touchstone of “all lawful use” that DoW has rightfully insisted upon & xAI agreed to. But as Sam explained, it references certain existing legal authorities and includes certain mutually agreed upon safety mechanisms. This, again, is a compromise that Anthropic was offered, and rejected.
Even if the substantive issues are the same there is a huge difference between (1) memorializing specific safety concerns by reference to particular legal and policy authorities, which are products of our constitutional and political system, and (2) insisting upon a set of prudential constraints subject to the interpretation of a private company and CEO. As we have been saying, the question is fundamental—who decides these weighty questions? Approach (1), accepted by OAI, references laws and thus appropriately vests those questions in our democratic system. Approach (2) unacceptably vests those questions in a single unaccountable CEO who would usurp sovereign control of our most sensitive systems.
It is a great day for both America’s national security and AI leadership that two of our leading labs, OAI and xAI have reached the patriotic and correct answer here.
After many rounds, I believe the actual differences that matter are simpler than this.
OpenAI trusts DoW, is fully fine with ‘all legal use’ and letting DoW decide what that means, and is counting on its technical safeguards, safety stack and forward engineers to spot if the DoW does something heinous and illegal, including the threat that if OpenAI is forced to pull the plug DoW would not have good options, and for political and economic reasons you can’t try to destroy them.
Anthropic is not fine with some uses that DoW considers legal and wants to do, and wants language that prevents such actions, with no way to weasel out of it. But they’re fine delivering a basically frictionless system that lets DoW do what it wants in the moment, trusting that DoW will be unwilling to outright break the contract terms or they’d find out if DoW went rogue on them.
Notice that the DoW accusations against Anthropic about asking for operational permission in a crisis are exactly backwards. Claude will work in a crisis and was modified to refuse less, but that might violate a contract to be dealt with later. ChatGPT might refuse in the moment when it’s life and death.
OpenAI’s terms may or may not work or amount to a hill of beans. Too soon to tell. It could work in practice, or it could end up worthless.
We do know exactly why they do not work for Anthropic and DoW, in either direction.
As Lewin notes here, both the Obama and Trump administrations have done actions that many objected to as rather obviously unlawful, and basically nothing happened.
And again he brings up the potential of ‘pulling the plug mid operation’ which is physically impossible in this context, which is physically impossible with Claude but could inadvertently happen with ChatGPT. And any sensible contract would include a wind down even if it was terminated for clear violations, to protect national security.
As described above instead it comes down to the claimed distinction in paragraph two, which boils down to the following components:
OpenAI in its extra rules referred to particular ‘legal and policy authorities’ rather than to distinct terms.
Anthropic is claimed to want to ‘vests those questions in a single unaccountable CEO who would usurp sovereign control of our most sensitive systems.’
Yeah, that’s not what any of this is about.
The first is not a meaningful distinction if it covers the prohibitions. If OpenAI’s rules refer to particular existing legal and policy authorities, then indeed it permits ‘all legal use’ which includes large amounts of domestic surveillance, and with a flexible government lawyer will include a lot of other things as well.
Nor is it meaningful as a matter of authority and law. The fact that things happen to be on the present books does not make them not contract law, and does not fail to remove them from what is otherwise the democratic authority. But also, very fundamentally, part of democratic law is contract law, and the ability to agree to terms.
The second one is, frankly, rather Obvious Nonsense that is going around. Ignoring that CEOs are accountable (both to the board and to the government and thus ultimately one would hope the people), they are claiming that Anthropic demanded that Dario Amodei be able to decide whether the terms of the contract were fulfilled at his discretion, rather than the government deciding, or it being settled by a court of law. At most, there may have been some questions that were left to be defined in good faith later, as per normal.
My jaw would be on the floor if this was indeed insisted upon or even suggested. That’s not how anything ever works. At most, this was Anthropic asking for carve outs for its two red lines, and then adding something like ‘without permission,’ so that in a pressing situation you could make an exception. You can take that clause out, then.
Alternatively, perhaps this is a reference to ambiguity of terms in existing contracts. In particular, the claim here is that the contract failed, as the rest of American law does, to define ‘domestic surveillance.’ Or it could simply be ‘sometimes things are not clear in edge cases.’ One sticking point of the negotiations was exactly trying to pin down various definitions and phrases so they would be unambiguous and enforceable, and that Anthropic was trying to clear away what they felt were ‘weasel words’ from proposed DoW language.
In particular, DoW kept wanting ‘as appropriate,’ which mostly invalidates any barriers, although they dropped this demand in the end to try and get other things they wanted more.
But again, having an underdefined term in a contract does not mean it means whatever Dario Amodei thinks it means. At most it means you can sue, and that’s not exactly something one does lightly to DoW over a technical violation.
If Dario Amodei felt the contract was broken, he could, like with any other contract, at most either choose to terminate the contract under whatever terms allow for that (as can OpenAI), although at obvious risk of government retaliation for doing so, or sue in court, and the government court determines if there was indeed a violation, under conditions highly favorable to DoW.
If Dario tried to suddenly shut down the system anyway, that would not even be physically possible on classified networks, and also they could arrest him or worse.
This also implies that Sam Altman does not have any role in determining whether use was lawful, or whether it is valid under the terms of the contract. Sam Altman affirms this under ordinary circumstances, but says that sufficiently clearly illegal actions, especially constitutional violations, would be different.
Thus, in the negotiations with Anthropic, there were two things centrally going on.
First, the Department of War wanted the ‘all legal use’ language, and failing that they wanted to avoid one particular carveout to that related to mass surveillance.
Second, Anthropic was attempting to remove various ‘weasel words’ and clauses, that would allow the Department of War to circumvent restrictions.
We can indeed see some of those weasel words in the brief wording shared by OpenAI. OpenAI isn’t relying on these terms to bind DoW, they’re relying on the safety stack and on trust.
Let’s go through every word they shared, and see why they don’t actually bind DoW:
The Department of War may use the AI System for all lawful purposes, consistent with applicable law, operational requirements, and well-established safety and oversight protocols.
All lawful use.
The AI System will not be used to independently direct autonomous weapons in any case where law, regulation, or Department policy requires human control
They can do anything they want unless the department policy requires human control, so again all lawful use. We already have highly effective autonomous weapons in some cases, such as missile defense. Directive 3000.09, the only plausible barrier, we’ll get to in a second.
Nor will it be used to assume other high-stakes decisions that require approval by a human decisionmaker under the same authorities.
All lawful use.
Per DoD Directive 3000.09 (dtd 25 January 2023), any use of AI in autonomous and semi-autonomous systems must undergo rigorous verification, validation, and testing to ensure they perform as intended in realistic environments before deployment.
For intelligence activities, any handling of private information will comply with the Fourth Amendment, the National Security Act of 1947 and the Foreign Intelligence and Surveillance Act of 1978, Executive Order 12333, and applicable DoD directives requiring a defined foreign intelligence purpose.
These absolutely are not meaningfully time stamped at all.
Private information is generally interpreted as not including third party or publicly available information, which includes massive amounts of data on everyone, especially anyone who carries around a phone. And again it’s still all lawful use, except it’s worse:
The AI System shall not be used for unconstrained monitoring of U.S. persons’ private information as consistent with these authorities.
So if you put any constraint on it, you’re good. Renders it meaningless.
The system shall also not be used for domestic law-enforcement activities except as permitted by the Posse Comitatus Act and other applicable law.
If it’s legal under applicable law, they can do it. Again, renders it meaningless.
For intelligence activities, any handling of private information will comply with the Fourth Amendment, the National Security Act of 1947 and the Foreign Intelligence and Surveillance Act of 1978, Executive Order 12333, and applicable DoD directives requiring a defined foreign intelligence purpose.
At best this freezes those particular rules in place, if that is insisted upon elsewhere in sufficiently robust language.
As far as I can tell, at best OpenAI’s stated redlines amount to ‘all lawful use under existing law’ rather than ‘all lawful use.’ That’s it.
As Charlie Bullock explains again in more detail here, all the above language translates to ‘all lawful use,’ a restriction in theory already in place by definition. Charlie shares my view that the language shared does not enshrine current law as did ACX.
OpenAI CSO Jason Kwon disputes this interpretation, claiming that time stamping a law in a contract preserves current language (and thus this means they don’t have any additional protections on this), and says ‘any of the chatbots should give you a similar answer.’
I checked, and ChatGPT confirms that this is not the case. The language for everything except 3000.09 does not meaningfully time stamp anything. The language on 3000.09 might or might not be sufficient under ordinary contract law if neither party was the Department of War and this was a normal contract, but under these circumstances, if the DoW doesn’t want to be bound on this, this language is at best ambiguous and thus is not going to protect you in any meaningful way.
Kwon’s statement means there is no other such enshrining language. So that’s it.
As Bullock points out, there is lots more language that we do not have, so many things could have come to pass, and there are many legal things that would qualify as common sense “mass domestic surveillance” as I repeatedly point out.
I don’t think the legal language here is going to meaningfully bind DoW.
That doesn’t mean the contract language is completely worthless.
It does do one important thing. It gives standing. If DoW were to otherwise violate someone’s rights, that’s the target’s problem, but the target might never even find out. By naming these particular statutes and provisions, OpenAI now has clear standing to sue or take other actions, should the DoW be found in violation.
That matters, but it’s still a ticket to ‘all lawful use’ as DoW interprets that.
Otherwise, as OpenAI admits, they’re basically counting on technical safeguards, and well aware that they gave the green light to pretty much anything DoW does with whatever system they provide.
And they’re trusting DoW to act honorably.
Thus, my conclusion.
roon (OpenAI): there is no contractual redline obligation or safety guardrail on earth that will protect you from a counterparty that has its own secret courts, zero day retention, full secrecy on the provenance of its data etc. every deal you make here is a trust relationship
Grace: Good point, Anthropic was foolish to try and resist
roon: not at all what I am saying! it looks to be quite a defensible move to resist
I don’t think Roon is fully right, in that there are various ways to find out, but he’s essentially right if they want it badly enough.
On Friday morning, Sam Altman said on CNBC that OpenAI shared Anthropic’s red lines on domestic surveillance and autonomous weapons. Many praised this.
Ilya Sutskever (3: 52pm Friday): It’s extremely good that Anthropic has not backed down, and it’s significant that OpenAI has taken a similar stance.
In the future, there will be much more challenging situations of this nature, and it will be critical for the relevant leaders to rise up to the occasion, for fierce competitors to put their differences aside. Good to see that happen today.
On Friday afternoon, a potential deal is described between OpenAI and the Department of War, which Altman claimed would have strong protections.
On Friday evening, while the supply chain risk designation was hanging over Anthropic’s head, they signed an agreement that included exactly the language that Anthropic rejected, including allowing ‘all lawful use.’
They continue to claim this contract has strong protections.
In particular, he gave the impression that he had achieved Anthropic’s red lines and had found terms that would achieve them for Anthropic. This was not the case. He accepted exactly the key terms Anthropic rejected, because OpenAI is trusting DoW and is drawing its red lines (and/or understanding of the functional law) differently.
It is not impossible that OpenAI got meaningful protections in the contract language that they decline to share. We do know they are misrepresenting the contract language that they did share, which offers effectively no protection.
It is also possible that OpenAI is correct, in context, to put its trust in DoW, and perhaps can trust it far more than Anthropic could, because DoW will understand and value the relationship, given better cultural fit, Altman’s relationship with the White House, OpenAI’s position as already too big to fail and a lack of strong alternatives.
I do think OpenAI has greatly weakened the legal argument for declaring Anthropic a supply chain risk, and has been very strong on the point that this label is crazy. That is very important.
But it is also possible that Altman’s negotiations are what made Hegseth feel he had a green light to order it, as he no longer felt he needed Anthropic at least medium term. Altman’s rush to negotiate, exactly to de-escalate the situation, could have had the opposite effect. Hopefully at least from here it does calm things down.
OpenAI can now be threatened in similar fashion, including via a pretext, once it is in sufficiently deep. We all know that Elon Musk would love to try to destroy OpenAI.
I repeat this several times because it must be emphasized, although they did get some potentially important additional terms in exchange.
Even if Altman was trying to ‘take one for the team,’ and it is plausible that this was part of his motivation on this, that’s not always good for the team. Sometimes your team needs you to hold the line. We all know many examples of deeply foolish compromises, both fictional and real, made in good faith hopes of heading off a threat.
Agus: Not just did OpenAI defect and concede to this whole authoritarian maneuver, but Sam also went and just deceptively framed the whole thing to try to make it look like they had agreed to the same Anthropic redlines, which is not actually true.
Anthropic strongly believes that the language Altman signed will not hold water.
Hadas Gold: Anthropic said Thursday this compromise that they were offered (and apparently OAI accepted) was “ New language framed as compromise was paired with legalese that would allow those safeguards to be disregarded at will.”
I can confirm that this is Anthropic’s belief, via another source.
Why did Altman think these terms would be effective, if he believes that?
One possible contributing factor is that he was rushed, and did not understand.
A second is that he’s very confident in the use of technical safeguards.
Another is that OpenAI understands the redlines very differently than Anthropic.
I will dive more into their understanding later, but I do not expect that OpenAI’s redlines apply to anything that is legal. They only in practice object to illegal actions, and do not see it as their place to decide this, thinking that’s not how the system works. In that case, ‘all legal uses’ are indeed whatever DoW decides they are, and are acceptable, whether or not the language has any other teeth.
OpenAI’s leverage is that they claim they can decide what system to deliver, and can install any safeguards, and refuse requests that way. Okie dokie?
Here was Altman’s statement at that time, before we understood what it was:
Sam Altman (CEO OpenAI): Tonight, we reached an agreement with the Department of War to deploy our models in their classified network.
In all of our interactions, the DoW displayed a deep respect for safety and a desire to partner to achieve the best possible outcome.
That would be quite the contrast with their private and also very public actions in discussions with Anthropic, if it was true. This is the kind of thing that you have to say in Altman’s position, so you shouldn’t update much on it.
AI safety and wide distribution of benefits are the core of our mission. Two of our most important safety principles are prohibitions on domestic mass surveillance and human responsibility for the use of force, including for autonomous weapon systems. The DoW agrees with these principles, reflects them in law and policy, and we put them into our agreement.
I have clear sourcing that this is false. As in, The DoW intends to engage in legal forms of what most of us would call mass domestic surveillance. At bare minimum, what they valued most in this negotiation was the ability to do this in particular.
Human responsibility for the use of force is not the same as a human in the kill chain. It is far better than nothing that a particular human has responsibility for the outcome, if they are indeed ensuring that, but it is DoW itself who would then hold that person responsible. Would they hold them to the same standards as they would have otherwise?
We also will build technical safeguards to ensure our models behave as they should, which the DoW also wanted. We will deploy FDEs to help with our models and to ensure their safety, we will deploy on cloud networks only.
Cloud networks includes cloud classified networks, where OpenAI would have little control or visibility into what was happening. I don’t see how else OpenAI could relevantly replace Anthropic’s services.
The DoW made the largest of protests about the possibility that Claude might refuse a request. Sam is claiming that he can choose his safety stack and what requests the models refuse, and the DoW will respect those refusals. This is hard to believe.
There’s also the matter that no one knows how to build technical safeguards that will prevent a user of an LLM from doing whatever they want. Jailbreak robustness does not work here. Only a small number of forward deployed engineers will be able to examine queries. Without the engineers I think this is outright impossible.
With the engineers, it is merely extremely difficult. If the penalty for being caught is large enough (as in you’re willing to walk away over this and they believe you) it could work.
We are asking the DoW to offer these same terms to all AI companies, which in our opinion we think everyone should be willing to accept. We have expressed our strong desire to see things de-escalate away from legal and governmental actions and towards reasonable agreements.
Sam Altman (CEO OpenAI): We had some different [terms]. But our terms would now be available to them (and others) if they wanted.
We haven’t seen that language. But even if Anthropic was technically offered these terms, and the terms involved are as good as they could be, does anyone believe that Anthropic could have Claude’s safety stack refuse requests DoW thinks are legal, and the DoW would be fine with it? Or that anything that was a pure technical fix to Anthropic’s red lines wouldn’t bring a swathe of other undesired refusals, at best?
Now that we know what the fight was over, there was no zone of possible agreement unless DoW was willing to not do the thing it most wanted to do. DoW demanded Claude do [X] and Anthropic wasn’t willing to do [X]. No deal. Trying to play a game of ‘get it to do [X] despite the technical safeguards’ really, really isn’t an option.
The reporting claims OpenAI has the right to prescribe safety mitigations, and that the Pentagon will respect model refusals, and so on. We don’t yet know any of the details of that.
We remain committed to serve all of humanity as best we can. The world is a complicated, messy, and sometimes dangerous place.
Second half of this is certainly true.
For the first half, see my entire series of prior posts about OpenAI and Sam Altman, and the history of the company. But I do think that Sam Altman and most employees of OpenAI want better outcomes for humanity rather than worse. Affirming that is meaningful in a political context.
Effective, safe, and high-impact AI to directly support the men and women in our armed forces. All while respecting the law, protecting the Constitution and our rights, and setting the standard for responsible deployment.
God Bless America.
This is fine sentiment but does not claim that it protects the redlines.
Boaz Barak (OpenAI): I would prefer if we focused first on using AI in science, healthcare, education and even just making money, than the military or law enforcement. I am no pacifist, but too many times national security has been used as an excuse to take people’s freedoms (see patriot act).
I am very worried about governments using AI to spy on their own people and consolidate power. I also think our current AI systems are nowhere nearly reliable enough to be used in autonomous lethal weapons.
I would have preferred to take it slower with classified deployment, but if we are going to do it, it is crucial that we maintain the red lines of no domestic surveillance or autonomous lethal weapons. These are widely held positions, and codified in laws and regulations. They should be stipulated in any agreement, and (more importantly) verified via technical means.
I think the terms of this agreement, as I understand them, are in line with these principles, that are also held by other AI companies too. I hope the DoW will offer them the same conditions.
Regardless, a healthy AI industry is crucial for U.S. leadership. Whether or not relations have soured, there is zero justification to treat Anthropic – a leading American AI company whose founders are deeply patriotic and care very much about U.S. success – worse than the companies of our adversaries.
It appears to me that much of this week’s drama has been more about style and emotions than about substance. I hope that people can put this behind them, and come together for the benefit of our country.
The above is entirely the right attitude from Boaz Barak. For national security, we need to keep highly capable AI in our classified networks and assisting the DoW. But we should seriously worry that this could be used to cross redlines and endanger the Republic or put us at risk. We need to stipulate this in any agreement and verify it.
Given the practical state of law around surveillance in America, the principle of ‘all legal use’ will not protect against many forms of domestic mass surveillance at all, would offer only nominal practical protections against many other such forms, and we have strong reasons to believe there is intent by DoW to engage in such surveillance. Thus, we are left with only technical verification, despite all the information in question being classified.
I can only interpret OpenAI’s public statements, as I will get to them later, as saying that OpenAI does not view legal surveillance and analysis activities (or legal use of autonomous weapons) as crossing their red lines, by nature of it being legal.
I spent a lot of time ruling alternative out and establishing the arguments for this, but also they then just tweeted it out?
NatSecKatrina: A lot of the concerns about the government’s “all lawful use” language seem to stem from mistrust that government will follow the laws. At the same time, people believe that Anthropic took an important stand by insisting on contract language around their redlines.
We cannot have it both ways. We cannot say that the government cannot be trusted to interpret laws and contracts the right way, but also agree that Anthropic’s policy redlines, in a contract, would have been effective.
This is why our approach has been:
Let the democratic process decide on the legality and proper use question. The fact that people can even say that the gov has made mistakes in the past is the process in action. The fact that we are having this discussion on twitter is part of the process.
Create a reasonable contractual framework that guides expectations and the relationship, just as much if not more than the rules themselves.
And on top of this, have the ability to build the models the way we think is safe, along with cleared FDEs to do the real world work in partnership.
Katrina is saying that they will:
Allow all legal use.
Trust DoW to follow the law.
Trust DoW to determine the law and determine proper use.
I am confident that many at OpenAI believe that they would be able to prevent the Department of War from engaging in sufficiently illegal activities, were the DoW to decide to act here in a way that would be deemed illegal if it were to reach the Supreme Court, presumably by detecting the activity and either refusing the requests or terminating the contract. This may or may not include Sam Altman.
Alas, I believe they are incorrect in practice.
I do think Katrina makes an excellent point that if you do not trust DoW to follow the law, then you should doubt DoW to honor Anthropic’s redlines. In practice, both sides acted as if the terms mattered, but why couldn’t DoW, if it was not trustworthy, break the rules? I believe the answer is that they felt they would be unable to hide it from Anthropic if they used Claude at the kind of scale they had in mind, as it would have inevitably leaked.
Boaz Barak is actually relatively skeptical, although I am confused that he thinks the OpenAI contract is ‘no weaker and in several ways stronger’ rather than ‘weaker in one way and stronger in others.’
1. OAI’s DoW contract is no weaker and in several ways stronger than Anthropic’s original one in protecting the red lines of mass domestic surveillance and no autonomous lethal weapons.
2. Neither are good enough. AI poses unique risks to our freedoms that can’t be left to individual agencies and companies. We desperately need regulation and legislation to ensure our freedoms.
That’s very possible. I don’t think the first one is true, but they’re fully compatible, and yes I think it is highly unclear Anthropic’s language holds either.
Altman had a moment of huge leverage, and instead of standing with Anthropic, he caved on the key term in question, ‘all lawful use.’ At minimum, he failed to demand that the supply chain risk designation be moved off the table.
If he had meaningful redlines on currently legal activities to protect, he could not have had the time to properly consider what he was signing, or signing up for.
The correct prior, given the circumstances, timing and history of Altman and OpenAI, is that the protections agreed to were woefully insufficient, regardless of the degree to which Altman realized this at the time. He said he cared about the language truly holding up, but we should be skeptical both that he is sufficiently invested in that to take a very expensive stand when it counts, and that he can tell the difference.
On top of that, even if the current agreement were ironclad, what’s to stop the government from doing the same thing to OpenAI that they just did to Anthropic? They are altering the deal. Pray that they do not alter it any further. Is Sam Altman going to be willing to risk a supply chain risk designation? Do you think Elon Musk wouldn’t push for the Department of War to do the same thing again?
Again, OpenAI is choosing to trust DoW.
Even if he meant maximally well, if DoW does not mean well then Sam Altman has walked into a trap, and put the entire industry in a dramatically weaker position.
Peter Wildeford: I think it’s important to circle back to Sam Altman here. About 20 hours ago people, including me, were applauding his moral clarity. But that moral clarity lasted barely half a day.
OpenAI is now agreeing to be used for domestic surveillance and for lethal autonomous weapons, just like xAI. They have some clever words that pretend they are not, but we should see through them. This guy is not consistently candid.
Altman should be crying bloody murder over the supply chain risk designation. He should also refuse to work with the DoW until this threat is off the table. This is a designation reserved for foreign adversaries. This move threatens the entire tech industry and proves the DoW is unreliable. OpenAI could easily be burned next.
So no moral clarity. Altman sees a short-term way to torch a competitor and he’s going to take it. No matter what happens to OpenAI, Anthropic, the USA, or us.
On OpenAI’s legal language, at least the part that was shared with us, here’s two explainers for why it is highly unlikely to protect OpenAI’s supposed red lines:
Jacques: The government can already legally buy your location data, browsing history, and social media activity without a warrant. The only thing that prevented mass surveillance from that data was the inability to process it all. LLMs fix that. “All lawful purposes” includes this.
Alan Rozenshtein: These are NOT meaningful redlines. For example it only prohibits autonomous weapons “ in any case where law, regulation, or Department policy requires human control.” But the relevant safeguard against autonomous weapons is a DOD directive that Hegseth can change at will!
Also the surveillance redline is about “unconstrained” surveillance of “private” information. But what about “slightly constrained” surveillance of private information, or unconstrained surveillance of “public” information? Those are both potentially very dangerous forms of mass surveillance!
Lawrence Chan (METR): OpenAI has released the language in their contract with the DoW, and it’s exactly as Anthropic was claiming: “legalese that would allow those safeguards to be disregarded at will”.
Note: the first paragraph doesn’t say “no autonomous weapons”! It says “AI can’t control autonomous weapons as long as existing law (that doesn’t exist) or the DoD says so.”
Similarly, the mass surveillance use cases will “comply with existing law”, but many forms of data collection that we’d consider “mass surveillance” are things that the NSA has consistently argued are legal under current law.
This, of course, did not stop OpenAI from blatantly misrepresenting this language in the blog post and in Sam Altman’s tweets!
… Now, I’m sure OpenAI will claim that the real teeth of the agreement is not their contract but their deployment architecture: they have a “safety stack that includes these principles” and everything! (In other words, “trust me, bro.”)
Lawrence Chan (METR): Two more points: 1. It is not true that “As with any contract, [OpenAI] could terminate it if the counterparty violates the terms”. 2. Despite OAI’s claims, the legalese provided does not actually specify what will happen when existing laws or DoD policy changes.
Leo Gao (OpenAI): the contract snippet from the openai dow blog post is so obviously just “all lawful use” followed by a bunch of stuff that is not really operative except as window dressing.
the referenced DoD Directive 3000.09 basically says the DoD gets to decide when autonomous weapons systems are deployable.
as others have covered, there are a ton of mass domestic surveillance loopholes not covered by the 4A, national security act, FISA, etc.
Dave Kasten: The people interpreting this legal guarantee are Executive Branch lawyers, and their General Counsel bosses are usually political appointees; they can always just change the DoD directives or the Executive Orders if they want, or DoD’s internal legal definition of the same. Every intelligence scandal you’ve heard of, from the warrantless mass wiretapping of American citizens to the post-9/11 torture of prisoners, about a quarter of which weren’t even terrorists , had legal guidance claiming it complied with those exact same authorities, or other authorities superseded them.
Second, claiming that the monitoring of US persons (that’s any US citizen, lawful permanent resident, US company or nonprofit) merely needs not to be “unconstrained” means very little. Third, what domestic law enforcement activities are included — does targeting and arresting peaceful protestors because the FBI Director thinks they’re friends of Antifa count? They may be able to limit some of this with the technical controls they propose; but you should be skeptical.
dave kasten: The intelligence law section of this is very persuasive if you don’t realize that every bad intelligence scandal in the last 30 years had a legal memo saying it complied with those authorities
James Rosen-Birch: How do people not get that DoW *neverthinks it does anything illegal, even when it does.
Boaz Barak (OpenAI): Does this criticism not apply also to the previous contract DoW had, that relied even more on contract language and less on technical verification?
dave kasten: Hard to say for sure without knowing what contractual guarantees Anthropic had, but probably less so than OpenAI At a minimum, OpenAI’s deal clearly and unambiguously rules in use of piles of domestic info incidentally collected via FISA authorities, which as far as I can tell Anthropic’s didn’t.
OpenAI’s deal also appears to allow the use at scale of analyzing massive piles of commercial data, which courts have thus far not fully ruled on (beyond Carpenter in 2018), and which Anthropic clearly indicated they were being asked to do and refused to do.
One more thought: who gets to do anything if your technical controls report an issue? It’s classified; what’s your plan for disclosure? to DoD IG? To Congress? To the press and take your chances you won’t lose the contract or be arrested?
Neil Chilson: THAT PROBLEM IS NOT FIXABLE BY PRIVATE PARTIES.
dave kasten: I don’t see how that’s accurate; every government contracting job I had gave me a very clear training on government requests that I was supposed to refuse as they were unlawful, even if someone told me they were lawful.
Neil Chilson: How many of those scandals you mentioned were struck down in court? Most were fixed by Congress — because they were arguably legal but bad.
dave kasten: I agree it would be good for us to defer less to claims of non reviewability and think there could be mechanisms (eg, establish a cleared litigation system that any litigant can engage for a fee and have litigate in a classified setting) to do so while preserving national security.
Andreas Kirsch (Google DeepMind): I’m speechless at OpenAI releasing that contract excerpt and acting as if there aren’t gaping holes that could be exploited far beyond their stated “red lines.” I’m not a lawyer, but this is pretty obvious and common sense.
(And to be clear: if Google had signed the same deal, I’d be saying the same thing internally. The issues here are bigger than friendly competition between companies.)
… The actual language they published is still full of obvious escape hatches.
Altman claims that the DOD directive is referred to as it exists today, not only as it might exist in the future. But even if that were true, it is not meaningful, as that directive leaves it to DoW to determine appropriate levels of supervision.
I do not understand why OpenAI believes, as they seem to be claiming, that the language they shared itself refers to the law as it exists today, and would continue to refer to those laws and directives even if they were later altered. That would not be how I would read those contract terms. You aren’t breaking a law if that law has been repealed or changed. At best this is highly ambiguous and DoW will read it the other way the moment it matters.
DoW keeps saying it is illegal to do ‘domestic mass surveillance’ but this is not a term of American law, so what exactly does that even mean? OpenAI has not shared any legal definition of the term, nor has DoW.
dave kasten: Something I’ve been convinced of over the past 24 hours:
“Domestic mass surveillance” is NOT a defined term in US law.
The exec branch has a bipartisan history of interpreting IC legal authorities VERY broadly.
Make sure you know what’s in, and what’s out, before you sign.
Again, Anthropic explicitly rejected the core term language OpenAI accepted, exactly because they felt that those terms did not hold water. To the extent it has similar red lines, OpenAI is counting on its technical affordances, and this only potentially works for them because (as I understand it) they believe crossing the lines would be illegal.
Mark Valorian: Idk who needs to hear this (apparently all of twitter) but OpenAI did not just magically get the DoD to agree to the terms Anthropic was asking for.
…OpenAI just took the terms Anthropic considered so egregious, it warranted jeopardizing an enormous part of their business.
The DoD does not just break off a massive contract to accept the same demands 5 minutes later from someone else. Until explicitly indicated otherwise, the only logical conclusion here is that OpenAI swooped in and unscrupulously stooped lower than Anthropic was willing to go for the money.
Assume all OpenAI data will now be used for what Anthropic deemed “mass domestic surveillance of Americans”. Plan and prompt accordingly.
I am highly confident that Anthropic did not risk going head to head with the Department of War over meaningless terminology details.
I am highly confident that the Department of War did not risk this battle with Anthropic over meaningless terminology details, although it in part did so because some people actively wanted to destroy Anthropic.
Part of what they hope for is de-escalation. The strategy needs to be reevaluated if that does not now happen. But what about the actual contract and serving DoW?
One way to have the red lines not be crossed is if the Department of War chooses not to cross the red lines. Sometimes the ‘trust us’ strategy works and people prove worthy. At other times they don’t want to risk being caught.
I really hope that this turns out to be the case, either way.
What about the other way of holding the red lines? Is it possible Anthropic and I are wrong, and basically all the legal experts who weighed in are wrong, and Sam Altman pulled it off even if the Department of War intends to cross the red lines?
It is possible. It would require that the key terms be elsewhere in the contract, in places where they claim they cannot share the details.
It would then require OpenAI to do heroic work, including heroic technical work, and be prepared to take heroic stands at great potential financial and personal cost.
The first step to knowing if this is possible is to read the rest of the contract terms.
The argument for hope goes something like this:
OpenAI decides on its own ‘safety stack’ and chooses what model to deliver.
They can choose to deliver models incapable of the things they don’t want without tripping the safety stack, or at all.
The Department of War has agreed to accept this if it happens.
Therefore, OpenAI can build a safety stack that protects against their redlines, even if such activity was legal, either through inherent inability or refusal, no matter what the contract otherwise permits, so This Is Fine.
OpenAI would be able to sustain this even under huge political pressure across the board, and likely also legal pressure.
There are many severe problems with this plan. A lot of them are obvious. OpenAI would have to actually do the heroic work, take the heroic stand, and withstand the kind of pressures being used against Anthropic.
A less obvious problem is, even if OpenAI did heroic work, I don’t see how to deliver otherwise useful models that can’t be used for legal mass domestic surveillance. You can have them analyze one situation at a time and then clear the context.
So either you deliver a rather useless model by being unwilling or unable across a rather broad set of queries a lot of which are good uses, you’re allowed to pick up patterns and flag the whole DoW account, or you’re dead even without a jailbreak.
Then there’s the problem that there’s no known robust defense to jailbreaks, unless OpenAI is willing to implement technical pattern detection for violations, and then willing and able to pull the plug if that happens.
Even if I am reading the situation maximally wrong, there is one thing that is clear.
OpenAI previously turned down the contracts Anthropic accepted, exactly because Anthropic cared deeply about national security, and OpenAI did not wish to lose focus and money and take on the associated risks by prioritizing such work, especially when Anthropic was volunteering to pick up that slack and having access via AWS.
I also strongly believe that OpenAI has consistently been attempting to de-escalate the conflict between Anthropic and the Department of War rather than escalate it. Sam Altman has been excellent on that particular point, as noted earlier, and we should give him proper credit.
To the extent that this conflict was stroked by competitors or was due to manipulation or corruption, those pulling those strings lie elsewhere, as I’ve noted.
That still leaves many potential motivations for OpenAI agreeing to this contract.
I believe there are three we must centrally consider.
I believe that at least part of the motivation is that Altman believes that doing so de-escalated the situation and helped protect Anthropic, and with it the entire AI industry and economy and military supply chain, from an epic clusterfuck. This was an excellent motivation.
Unfortunately, I do not believe his instincts were correct here. While his explicit statements have indeed been very helpful, and the contract does further invalidate any possible legal arguments for the supply chain risk designation, I fear that by being willing to contract he may have unwittingly ended up making Hegseth feel he had a green light.
I believe that at least part of the motivation was genuine concern for national security, and of what would happen on multiple levels if Grok were left as the only model with access to classified networks. No doubt he was concerned given their history that this would give Elon Musk powers he might abuse, and also he is aware that Grok is not a good model and can’t do the job protecting America.
By playing ball with the Department of War and White House, OpenAI gains political favor and power, which will be vital in the months and years ahead, and also OpenAI gains direct levers of power via its AI inside classified networks. Hopefully one of the chits they got was a promise of de-escalation.
People are not discussing this third motivation, but it is very obviously there. Sam Altman has done many things to curry favor with the administration. Fair play.
I think people have this third motivation exactly backwards. They say things like ‘Brockman contributed $25 million to Trump and that’s how they secured this contract.’ I would suggest the opposite is more important. This contract, and the willingness to bail out this crisis and capitulate, is itself a contribution.
Again, on Friday morning, Altman claimed to share Anthropic’s red lines, implying (but not explicitly confirming) that this would apply even to legal activities.
On Friday evening, Altman claimed to have signed a ‘more restrictive’ contract that would preserve the redlines, a contract Anthropic explicitly declined and that would not have preserved Anthropic’s red lines, but might help preserve OpenAI’s.
On Saturday afternoon, we got some of the legal language, which looks like all we’ll get, and that language we de facto ‘all lawful use’ as determined by the general counsel’s office, with the meaningful levers being the safety stack and right to cancel.
Which is totally a coherent position, highly defensible, but very different from what Altman was representing was the OpenAI position, and one that would make a lot of people very upset.
Then Altman, and several other employees of OpenAI, did an AMA and otherwise Tweeted out various sentiments on how they believe all of this works.
Sam Altman (CEO OpenAI): I’d like to answer questions about our work with the DoW and our thinking over the past few days. Please AMA.
These lay out a clear and coherent position and philosophy, which I believe amounts to saying that their redlines allow all legal use, and trusting the Department of War to determine and abide by what is legal, and that to do otherwise would not be appropriate in a democracy.
Yes, they intend to include a ‘safety stack’ and other safeguards, but fundamentally believe that they should not be determining what their AI is used for, other than via enforcing the law and refusing illegal requests.
roon (OpenAI): are you worried at all about the potential for things to go really south during a possible dispute over what’s legal or not later on and be deemed a supply chain risk? I find this part to be the most worrying out of distribution thing to happen this past week
Sam Altman (CEO OpenAI): Yes, I am. If we have to take on that fight we will, but it clearly exposes us to some risk. I am still very hopeful this is going to get resolved, and part of why we wanted to act fast was to help increase the chances of that.
I think this greatly underplays the level of risk Altman is taking on by getting involved, and his other statements sound like a person already choosing his words carefully due to this. I hope I am mistaken, and I hope that Altman is correct that OpenAI intends to and actually will, even under immense pressure, use its safety stack to determine what is legal and to refuse requests it feels are illegal, and to terminate the contract if it discovers an illegal pattern of behavior that cannot otherwise be prevented.
There is also potential political risk in refusing to become involved. At some point, you might not be interested in politics and the national security state but they become interested in you.
No explicit or implicit threats. In fact, I could tell that as of Weds, the DoW was genuinely surprised we were willing to consider.
I think I would, and it would be lost in the noise of the SCR stuff.
I fully believe Altman here. I think Altman decided to do this on his own.
There is of course ‘if you help us we will remember that, and if you didn’t help us when we needed it we are going to remember that.’ That’s always there, whether or not anyone wants it to be there.
This was a good answer:
Anthony Pompliano: What are AI-native things the Department of War is not yet doing that you see as opportunities over the next decade?
Sam Altman (CEO OpenAI): They will have their own opinions, but the two things I am currently most worried about where AI can help are a) the ability to defend against major cyber attacks (eg something on the scale of taking our whole electrical grid down) and b) the ability to contribute to biosecurity. I do not think we are currently set up well enough to detect and respond to a novel pandemic threat.
I would have added a third key area, the defense of model weights.
This was also a good answer, up until the last line:
nic: Which of OpenAI’s core principles was the most difficult to reconcile with the DoW’s requirements during your internal debates this week?
Sam Altman (CEO OpenAI): Thinking through non-domestic surveillance. I have accepted that the US military is going to do some amount of surveillance on foreigners, and I know foreign governments try to do it to us, but I still don’t like it.
I think it is very important that society thinks through the consequences of this; perhaps the single principle I care most about for AI is that it is democratized, and I can see surveillance making that worse.
On the other hand, I also respect the democratic process. I don’t think this is up to me to decide.
The ‘not up to me to decide’ rhetoric totally applies to what DoW decides to do. It doesn’t mean you have to help them do it if you think it’s wrong, but also they can force you into a package deal decision, and they did that here.
One thing Altman should keep in mind is that a lot of what we would think of in practice as domestic surveillance, legally is classified as foreign. Then there’s the ‘border zone.’ Again, I see OpenAI as saying they will defer to DoW on what is legal, and they will assume legal overrules their red lines, unless things become sufficiently blatantly unconstitutional.
This answer and related follow-ups are very telling on several fronts at once.
I get a sense that the actual intended redline for OpenAI might be better described as the Constitution of the United States? That’s a highly reasonable redline, if you can actually act on it.
Nicholas Decker: If the government comes back with a memo saying that, in their view, mass domestic surveillance is legal, do you do that? Do you do it until the courts bar it, or do you delay until the courts approve it?
Second, would mass domestic surveillance be a lawful use right now?
Sam Altman (CEO OpenAI): We would not do that, because it violates the constitution. Also, I cannot overstate how much the DoW has been extremely aligned on this point.
However, maybe this is the question you are really asking: what would we do if there were a constitutional amendment that made it legal?
Maybe I would quit my job.
I very deeply believe in the democratic process, and that our elected leaders have the power, and that we all have to uphold the constitution. I am terrified of a world where AI companies act like they have more power than the government. I would also be terrified of a world where our government decided mass domestic surveillance was ok. I don’t know how I’d come to work every day if that were the state of the country/Constitution.
Nicholas Decker: If the DoW gives you what you believe to be an unconstitutional order, do you refuse to follow it until the courts rule? Or do you do it until the courts bar it?
Sam Altman (CEO OpenAI): I don’t think this will happen. But of course if we are confident it’s unconstitutional, we wouldn’t follow it. The constitution is more important than any job, or staying out of jail, or whatever.
In my experience, the people in our military are far more committed to the constitution than an average person off the streets.
An important part of this is that I don’t think our company is above the constitution either.
B: What would cause OpenAI to walk away from a government partnership? Is there a clearly defined boundary or red line you won’t cross?
Sam Altman (CEO OpenAI): If we were asked to do something unconstitutional or illegal, we will walk away. Please come visit me in jail if necessary.
melissa byrne: Will you turn off the tool if they violate the rules?
Sam Altman (CEO OpenAI): Yes, we will turn it off in that very unlikely event, but we believe the U.S. government is an institution that does its best to follow law and policy.
What we won’t do is turn it off because we disagree with a particular (legal military) decision. We trust their authority.
Nate Berkopec: Sam would do good to remember that Hegseth thinks it’s sedition when Sen Mark Kelly says “don’t follow illegal orders.”
Saying that your models will only be used to follow legal orders is the barest of fig leaves in the current administration.
I strongly agree that most people in the military are far more committed to the constitution than the average person on the street, but they also consistently take a broad view of what they need to do to defend national security (and they are often right), and also they usually do as instructed by the chain of command. That’s the job.
We really need to know the full OpenAI definition of ‘domestic mass surveillance.’
Here are the conclusions to draw.
First, he clarifies that he is fine with anything constitutional, presumably as the current courts understand it, which is a rather narrow reading of this question as noted extensively earlier:
Sam Altman believes that ‘domestic mass surveillance’ violates the Fourth Amendment to the Constitution.
That means that anything that does not violate the Fourth Amendment then is not ‘domestic mass surveillance.’
Since the courts have consistently ruled that all analysis of third-party data and the many other things listed above, that I consider ‘domestic mass surveillance,’ are legal, then it follow Altman doesn’t think they cross his red lines.
As in, this is a confirmation that the rule really is ‘all legal use.’
Second, that he is pledging to defy an unconstitutional order, even if it comes with a legal opinion. He is promising to pull the plug on the entire program, if he finds DoW doing illegal things.
I very much appreciate this, but if the situation happens we may never learn of it, and intent now is very different from what you do in the breach.
Third, and I very much appreciate this too, that he doesn’t know what he’d do if we abandoned our rights and freedoms. I too don’t know what I would do then.
Chris: What was the core difference why you think the DoW accepted OpenAI but not Anthropic
Sam Altman (CEO OpenAI): I can’t speak for them, but to speculate with the best understanding of the situation.
*First, I saw reporting that they were extremely close on a deal, and for much of the time both sides really wanted to reach one. I have seen what happens in tense negotiations when things get stressed and deteriorate super fast, and I could believe that was a large part of what happened here.
*We believe in a layered approach to safety–building a safety stack, deploying FDEs and having our safety and alignment researcher involved, deploying via cloud, working directly with the DoW. Anthropic seemed more focused on specific prohibitions in the contract, rather than citing applicable laws, which we felt comfortable with. We feel that it it’s very important to build safe system, and although documents are also important, I’d clearly rather rely on technical safeguards if I only had to pick one.
*We and the DoW got comfortable with the contractual language, but I can understand other people would have a different opinion here.
*I think Anthropic may have wanted more operational control than we did.
Altman may have an excellent point that the other terms, granting the right to build OpenAI’s own safety stack and control what is delivered, are things that Anthropic did not focus on or ask for.
But they are also exactly the type of thing that Hegseth and Michael were yelling were totally unacceptable, and that they could not and would never accept. They’re giving a private corporation operational control, the ability to substitute OpenAI’s judgment for the DoW and refuse requests based on their own reading of the law, or indeed any other safeguards they determine – if you believe Altman’s claims about this deal.
That’s not the ‘unfettered access’ that was demanded of Anthropic. Not at all. If OpenAI got a strong deal, it is because they got more, not less, operational control.
Indeed, what happens if the model starts refusing on a real time operation? Are they going to ‘call Sam?’ This points out how nonsensical all that rhetoric always was.
I notice this makes me worry a lot, if Altman is presenting his view accurately, that OpenAI and DoW do not have a meeting of the minds on this contract.
Altman seems to think that technical safeguards give him the right to decide what is and isn’t acceptable via having the model refuse requests. I doubt DoW agrees.
Peter Wildeford: So I’m confused – maybe you can help. OpenAI is trying to claim simultaneously that (a) the contract allows “all lawful purposes” and (b) also that your red lines are fully protected.
The way you bridge this is by saying the protections live in this “deployment architecture and safety stack” rather than the contract language. But if this contract says “all lawful purposes” and your safety stack prevents a lawful purpose, you’re in breach.
So then either the safety stack has no teeth on lawful-but-objectionable uses, or OpenAI is setting up a future contract dispute with the Pentagon.
How do you ensure both (a) and (b)?
Sam Altman (CEO OpenAI): We deliver a system (including choosing what models to deploy), and they can use it bound by lawful ways, including laws and directives around autonomous weapons and surveillance. But we get to decide what system to build, and the DoW understands that there are lot of risks we deeply understand. We can, and will, build a lot of protections into that system, including for ensuring that the red lines are not crossed. The DoW is supportive of this approach.
We are generally quite comfortable with the laws of the US, but there are cases where the technology isn’t very good, shouldn’t be used, and would have serious unintended consequences.
We do not want the ability to opine on a specific (and legal) military action. But we do really want the ability to use our expertise to design a safe system.
Not only does OpenAI choose what system to build, they can ‘build protections into the system’ including to ensure no crossing of the red lines.
I am confident that DoW thinks this that they are entitled to ‘all lawful use’ and that if they get any refusals for reasons they don’t like, they will throw an absolute fit. They will absolutely pull out all the rhetoric they used on Anthropic, or threaten it.
Frankly, I expect the DoW to be right about this dispute, unless there is very clear language we have not seen that says that OpenAI has the right to have its safety stack refuse requests for essentially any reason, and even then I’d be rather nervous as hell.
If anything, Anthropic was trying to address the situation contractually rather than via technical safeguards exactly so that they didn’t get accused of making operational decisions, or accidentally actually refuse in real time.
Anthropic was saying, we’ll have the model do it, and specify what you agree not to do with the model, and then we’re trusting DoW to abide by the agreement, but we’re not going to do a sudden refusal in the middle of all this.
It didn’t help. None of that was in good faith.
Theo – t3.gg: How long has this conversation with DoW been going for? What was the reason for announcing so close to the deadline they gave Anthropic?
Sam Altman (CEO OpenAI): For a long time, we were planning to non-classified work only. We thought the DoW clearly needed an AI partner, and doing classified work is clearly much more complex. We have said no to previous deals in classified settings that Anthropic took.
We started talking with the DoW many months ago about our non-classified work.
This week things shifted into high gear on the classified side. We found the DoW to be flexible on what we needed, and we want to support them in their very important mission.
The reason for rushing is an attempt to de-escalate the situation. I think the current path things are on is dangerous for Anthropic, healthy competition, and the US. We negotiated to make sure similar terms would be offered to all other AI labs.
I basically buy this first half, in addition to any other motives involved.
I think it was a mistake, and could have had the opposite of its intended effect by making Hegsted think he was free to try and murder Anthropic (as Hegsted seems to not understand why that would be bad), but I do think this was a strong motivation.
The second half of this answer, however, is where we start getting into parroting the DoW rhetoric about this in a way that scares me.
I know what it’s like to feel backed into a corner, and I think it’s worth some empathy to the DoW. They are on the a very dedicated group of people with, as I mentioned, an extremely important mission. I cannot imagine doing their work.
Our industry tells them “The technology we are building is going to be the high order bit in geopolitical conflict. China is rushing ahead. You are very behind.”
And then we say “But we won’t help you, and we think you are kind of evil.”
I don’t think I’d react great in that situation.
I do not believe unelected leaders of private companies should have as much power as our democratically elected government.
But I do think we need to help them.
Anthropic was in no way telling DoW it was ‘kind of evil.’ It was saying that there were some activities in which it did not wish to participate.
Anthropic was not saying they wouldn’t help, indeed they have gone to extraordinary lengths in order to be maximally helpful. They are saying there are some narrow things, things DoW keeps saying they would never do, that Anthropic does not want to help with.
There is no corner under into which DoW was being backed. This was a ‘war of choice’ the entire way. Anthropic was happy to continue under its current contract, offered much less restrictive terms than that, and was also happy to walk away. Then Hegseth did what he did, including running right through Trump’s de-escalation.
This was rather more than a failure to ‘react great.’
The penultimate line is far, far scarier. Who is saying ‘unelected leaders of private companies should have as much power as our democratically elected government?’
The claim is that a private company should be able to determine under what terms it is willing to do business and provide its own tools. That’s it. This whole line that this is some sort of anti-democratic power grab is inimical to the values of America.
We do still need to find a way to help the DoW, even if they don’t make it easy. That doesn’t mean treating the Secretary of War like a dictator.
Notice the equating of ‘democratically elected government’ with the military chain of command. It is supposed to be the Congress that sets the rules and writes the laws.
Can OAI implement a safety stack that refuses unethical actions, under this contract?
Altman seems to be saying no. Or at least that he’s not going to try.
Chris: If the models or someone at OAI deem an action unethical, does OpenAI has the right to deny said action
Sam Altman (CEO OpenAI): We currently have three redlines. I could see us changing them or adding more as the technology evolves, and there are new risks we don’t yet understand. Iterative deployment is one of our most-important safety principles, and is a big part of why it was so important that we could write and update our safety classifiers.
But a really important point: we are not elected. We have a democratic process where we do elect our leaders. We have expertise with the technology and understand its limitations, but I think you should be terrified of a private company deciding on what is and isn’t ethical in the most important areas.
Seems fine for us to decide how ChatGPT should respond to a controversial question. But I really don’t want us to decide what to do if a nuke is coming towards the US.
I am pretty furious at repeating this line, and at the idea that, because you might be facing an incoming nuclear missile (that would be handled by existing automated anti-missile systems), that means you have to apply that level of deference in all other situations, including in peacetime without an emergency.
In practice, very obviously, in a true emergency like that, the DoW says jump, and you do your best to guess how high because they don’t have time to tell you. We all would. Dario would, Sam would, I would, and I bet you would.
I continue to be flabbergasted that so many people think this is a good argument.
Yet Altman reiterated this position again, that the government needs to have the power, and that to be able to say no to the government means you have ‘more power.’
Sam Altman (CEO OpenAI): Three general things from this AMA:
1. There is more open debate than I thought there would be, at least in this part of Twitter, about whether we should prefer a democratically elected government or unelected private companies to have more power. I guess this is something people disagree on, but…I don’t. This seems like an important area for more discussion.
florence: remember—you should do whatever the government wants, even things you think are immoral, because otherwise you’re deciding what you can do instead of the government, which is undemocratic.
Democracy is being redefined in real time.
I strongly agree that we need to talk about this. A lot. I want to live in a Republic.
No, I do not think we should move more power from corporations to the government.
No, I do not think that the government should have ‘more power’ than all the ‘unelected’ private companies, as in all the private people, collectively.
2. I think the is a question behind a lot of the questions but I haven’t seen quite articulated: What happens if the government tries to nationalize OpenAI or other AI efforts? I obviously don’t know; I have thought about it of course (it has seemed to me for a long time it might be be better if building AGI were a government project) but it doesn’t seem super likely on the current trajectory. That said, I do think a close partnership between governments and the companies building this technology is super important.
The government was at least flirting with soft nationalizing Anthropic, and instead tried to destroy it. The day could come sooner than we think, and yes many of us have been thinking about that for a long time without great answers. The implication here is that he would accept nationalization. After which, of course, his red lines would not be up to him, no matter what they are.
3. People take their safety (in the national security sense) more for granted than I realized, which I think is a good thing on balance but I don’t think shows enough respect to the tremendous work it takes for that to happen.
I did not get this sense from the questions, and I’m worried that Altman did. I do agree that many people take it for granted, especially day to day, and that this is good. If we take it for granted, that means DoW did its job.
Also, I am on the whole very grateful for the level of reasonable and good-faith engagement here. It was not what I expected.
On one particular question, he answered with a RT.
a) the contract permits all lawful use, + therefore mass surveillance + autonomous weapons, which have no legal prohibition
b) the contract has substantive red lines that constrain lawful use
c) OpenAI has a controversial interpretation of the law, disagreeing with others including the DIA about whether mass surveillance and autonomous weapons are legal and will block them based on that interpretation
d) OpenAI uses words differently from the people commenting here and doesn’t mean the same thing as we do when referring to mass surveillance and autonomous weapons
Sam Altman RTs in response:
Under Secretary of War Emil Michael: The DoW has always believed in safety and human oversight of all its weapons and defense systems and has strict comprehensive policies on that.
Further, the DoW does not engage in any unlawful domestic surveillance with or without an AI system and always strictly complies with laws, regulations, the Constitution’s protections for American’s civil liberties. The DoW does not spy on domestic communication of U.S. people (including via commercial collection) and to do so would be unlawful and profoundly un-American.
Boaz Barak (OpenAI): When we say that domestic mass surveillance and autonomous lethal weapons are red lines for us, we mean what we say, and are not looking for loopholes. We are confident that the combination of legal restrictions, safety stack, and deployment surfaces, will ensure that OpenAI models will not be used to cross either of these red lines.
I earlier broke down that Tweet from Emil Michael. It is well crafted, but if understood it is not a strong statement. If Altman is answering with that, and Boaz is giving this answer as well, we have our answer. All legal use, as determined by DoW.
AMAs happen fast, so I didn’t do a perfect job, but I did what I could.
I got an answer from Boaz Barak rather than Altman, which is fair.
Zvi Mowshowitz: I have a lot of Qs about this so please answer as much as you can, in priority order.
1) What forms of surveillance if any would your terms forbid, if the DoW determined they were legal? What is your definition of it that you believe is unconstitutional as per another Q? In particular, are you willing to do unlimited analysis of third-party or public information, which AIUI is considered legal? Of nominally ‘constrained’ private information? Is there an actual exception to ‘all legal use’ other than enshrining current law?
2) Can we see the rest of the contract, or at least the parts you claim tie it specifically to current law, or other parts of the defense in depth that you feel are key components?
3) What legal opinions did you get on your contract language before you agreed to it? Can you share any details? Did you consult with Anthropic’s team to learn what their true objections were and why they felt they couldn’t accept similar terms, and what particular language they were objecting to?
4) What is the enforceability mechanism? How will you know if DoW violates your redlines or does something illegal? If you do think so, what can you do about it? Does the safety stack include monitoring for patterns of activity like it would with another user? How much leeway does OpenAI have in designing its safety stack?
5) You said that this is more restrictive than Anthropic’s previous contract, but that previous contract AIUI contained many more restrictions that they were offering to remove. How can you be confident you’re right about this and if so why would DoW agree?
Boaz Barak (OpenAI): 1. The DoW is prohibited by law from engaging in any domestic mass surveillance, and @USWREMichael wrote that it would be profoundly un-American to do so, including for analyzing communication of Americans by purchasing data from commercial sources. Hence us and the DoW see eye to eye in our interpretation of domestic mass surveillance. They have no desire to do this, and we have no intention to allow it.
2. There are legal restrictions for publishing contracts with the DoW due to the classified nature of the work.
3. As you can imagine, our lawyers are quite good and they relied on internal advice as well as the advice of outside counsel. On consulting with Anthropic, US antitrust laws prohibit this kind of coordination, as much as we might wish otherwise.
4. The contract gives us the right to implement our full safety stack, which includes automatic classifiers and monitors. We will be working on the stack for this deployment over the coming months. Due to the classified nature of this deployment, only our cleared researchers and FDEs will have visibility into usage, which is why it is important we have them.
5. We believe our contract offers more robust safety guarantees. Note that Anthropic has disclosed that their national security model refuses less when engaging with classified information
I interpret these answers in the following ways, solidifying my perspectives elsewhere:
There are no surveillance actions that DoW understands to be legal, that OpenAI’s red lines or contract would disallow. Whatever it is that DoW is currently doing or plans to do legally, OpenAI is fine with it.
We likely never see any more details of the contract. Which is fair, but then we can’t take your word for what it says, especially in its details.
This is a hell of a justification. There’s some danger in practice, but this is at best overstated, and does not seem like a good reason in this sort of breach, especially given that the goal was explicitly de-escalation, and at minimum it seems like this means DoW didn’t want them talking.
This is the most meaningful answer. Their FDEs will be cleared and able to put human eyes on what is happening, which is very good, although it is not clear the extent of what ‘visibility into usage’ means. I notice he didn’t answer the other parts of the question, the ‘what are you going to do about it?’ half.
This doesn’t explain why he believes this or why DoW said yes. If anything, Anthropic’s model refusing less makes this more confusing, not less. I assume Barak is putting his faith in the safety stack, and doesn’t care much about the ‘all lawful use’ language or intent.
Katrina, OpenAI’s head of national security partnerships, says no, that the contract only applies to DoW in a way that excludes NSA, despite the NSA being under DoW. Again, we have not seen contract terms, so this is in theory possible.
dave kasten: 1. NSA engages in incidental domestic collection under FISA 702 and makes it available for queries, and DoJ writes an annual report to Congress listing all the times it does. OpenAI models usable under your contract for that, yes or no?
NatSecKatrina: 1. No, this contract does not apply to NSA.
dave kasten (later): Anthropic offered “FISA yes, commercially acquired data no” and got turned down. This, uh, makes me substantially doubt the OpenAI claims that they’ve excluded NSA from their contract successfully.
That’s at the heart of the matter.
It is where OpenAI communication is hardest to believe.
I think it’s a perfectly defensible position to say this is legal and it’s not their place to decide and it’s fine, but that’s not the position.
It’s also fine for them to build in safeguards that stop this and pull the plug if DoW tries to go around them. Their contract permits it. That would be great if it would work and they can hold that line. But that is not their stated intention.
dave kasten: 2. Getting and/or analyzing commercially available data at scale. OpenAI models usable under your contract for that, yes or no?
NatSecKatrina: 2. The Pentagon has no legal authority to do this (that would be federal law enforcement agencies, not DoW)
The DoW does this, legally, now. How can Katrina not know this?
There has since then been extensive reporting that exactly this was the sticking point of the negotiations with Anthropic that caused talks to fall apart.
Shakeel here offers an extensive analysis based on what we learned on Sunday. Both Katrina and Boaz Barak made clear statements saying the Pentagon is prohibited from or not allowed to do this. And yet. They have promised more language on this in the coming days, from the contract and I look forward to reading it. Seems important.
If she doesn’t know or is misrepresenting this, what the hell is going on?
Prakash: The real question is whether OpenAI is going to allow the use of AI on unclassified commercial bulk data on Americans, which is what the Pentagon wanted from Anthropic. Ant instead narrowed to classified FISA only, and got kicked.
Based on the contract language around analyzing bulk commercial data and deanonymizing it matches with this data discussion: Since 2021 the Pentagons DIA has been purchasing anonymized and harvested geolocation data that’s used in advertising, arguing it’s not “spying” since it’s commercial.
They’ve now realized AI is strong enough to take this bulk data and de-anonymize it accurately.
Anthropic deemed that spying on Americans. OpenAI doesn’t.
Trevor Vossberg: Note: The purported exclusion of the NSA by OpenAI doesn’t address this. The DIA, that did this, isn’t part of the NSA.
dave kasten: This is an important point from Logan Koepke: OpenAI is claiming that DoW lacks authorities to get commercial data at scale, despite extensive reporting that they have done so
logan koepke: on point two, they have in fact done this and claim they have the authority to do this.
Finally, here’s the big three models, including OpenAI’s ChatGPT.
This seems really, really conclusive.
It doesn’t get better from there.
dave kasten: 3. OLC writes a “President’s Surveillance Program 2.0”-like memo claiming President has inherent CinC authority to authorize mass domestic warrantless wiretapping. OpenAI models usable under your contract for that, yes or no?
and, 4., how can we verify that?
NatSecKatrina: 1. No, this contract does not apply to NSA. 2. The Pentagon has no legal authority to do this (that would be federal law enforcement agencies, not DoW) 3. Again, if this were to happen (and to my knowledge it hasn’t) this could only be done by the FBI and they are not a party to this contract. 4. Read the authorizing statute for the Department of Defense? None of these activities are within their statutory authorities. And our contract is expressly limited to the Department of Defense.
dave kasten: Huh? I note you do not claim “no, our contract does not allow that” for any of these (except kinda sorta 1).
But hey, I want to assume good faith, and everyone’s very short on sleep, would you explicitly say whether those answers mean “No” for each? And can we drill down here to understand those claims?
1. NSA is within DoD, are you claiming there is an explicit carveout to exclude it from your contract and that no NSA individuals, including those dual-hatted to CYBERCOM, will have access? What about other DoD IC elements? What about FBI’s access to 702 data for purely criminal investigations?
2-4. What if OLC claims that the President has the implicit Constitutional right as CinC to authorize DoD to do this (e.g., on a doctrine that immigration or Antifa, or in a future admin, pro-life protestors or MAGA activists are threats to national security), and thus no statute can bind that power? After all, this is literally what OLC has done before on warrantless wiretapping.
Christopher Hale: What do you say to people who’ve lost trust in both OpenAI and @sama ’s leadership and character over the past few days?
NatSecKatrina (mostly also reiterated here): I would wonder why [anyone] lost trust when OpenAI secured a deal with more guardrails than any previous agreement for classified AI deployments, including Anthropic’s.
OpenAI’s strategy was rooted in four basic ideas:
1. Deployment architecture matters more than contract language.
2. The safety stack travels with the model. The Department was not asking us to modify how our models behave. Their position was, build the model however you want, refuse whatever requests you want, just don’t try to govern our operational decisions through usage policies.
3. AI experts directly involved. Instead of hoping contract language will be enough, our contract allows us to embed forward deployed engineers.
4. U.S. law already constrains the worst outcomes. We accepted the “all lawful uses” language proposed by the Department, but required them to define the laws that constrained them on surveillance and autonomy directly in the contract.
And because laws can change, having this codified in the contract protects against changes in law or policy that we can’t anticipate.
This is a no good, very bad topline answer both in substance and in terms of PR. If you don’t know why, you sure as hell should know why. The actual philosophy of the approach part is reasonable in parts 1 and 3. I do agree that the forward engineers were a good ask.
An architecture approach is a philosophy, and it could be right. I wish there was more consistent emphasis that this is the plan.
The second point means DoW has this backwards if true. Usage policies don’t determine operational decisions. Refusals determine operational decisions. I presume the reason Anthropic didn’t agree here is that they understood that once you agree to ‘all lawful use’ you are not in a good position when you train the model to refuse legal things you don’t like, or you threaten to pull the contract over legal actions, where legal means as determined by the general counsel.
The problem is, as I said under meeting of the minds, you really don’t want to have the DoW thinking your contract works one way, and then insist it’s the other way, and Anthropic understood this.
There are good reasons the sidewalk outside OpenAI looks the way it does, and why the city was unable to get the people doing it to leave long enough to hose it off.
Even if OpenAI is attempting to do the right thing, they have signed on to ‘all legal use’ language, they have misrepresented the key functionality of the contract to the point that I spent many hours being confused and sorting through it until I finally understood their intent, and made many alarming statements of trust in and deference to the DoW given what else we know.
And while Altman and others at OpenAI have spoken excellently about the fact that it is crazy to label Anthropic a supply chain risk, they have also agreed to move forward to provide a replacement for Anthropic, while the sword of Damocles is still potentially poised above Anthropic’s head.
You must decide what to make of Sam Altman’s extensive history of cutthroat politics and not being consistently candid, read all of his statements, and decide how much faith to put in him here.
Again, is it possible that OpenAI stands ready and will stand by their redlines and protect our civil liberties in the ways that matter? For being responsible with potential autonomous lethal weapons?
I really, really want that to be true! We should all really want this to be true. It is up to those inside OpenAI to figure out for themselves whether or not it is true.
Matthew Yglesias: I think OpenAI employees need to ask some serious questions about what’s going on and whether they want to be participating in whatever it is.
We do not know the full terms of the OpenAI contract. Many questions remain unanswered. It is possible that OpenAI has a robust technical plan and understanding with DoW, and a willingness to back it up if it turns out DoW does not act honorably.
There’s only one way to know. You need to do your best to understand the situation.
I still think it is important is seeing the contract terms, and you should do so if you can and get a legal analysis, so you understand the background.
But unless you are resting your hopes on the contract’s legal terms, what matters is the practical plan, and your faith in its execution.
I would say the same for any other company entering such a deal. If I was at xAI, I would certainly be questioning what Grok was about to be used for, and whether or not you were okay with this, given they seem to have signed with no redlines at all.
If you are at OpenAI (or xAI), and after investigation (and legal consultation as needed) you do not find the protections acceptable or that your leader misrepresented the situation, then you need to organize, and use your power to hold leadership to account.
You also need to stand ready, in case DoW attempts to murder Anthropic, in which case you need to use your leverage to try and stop that from happening.
Your decision includes what this says about future high stakes decisions, and how they will be handled. Here is a paper on the history of activism in the AI community. If you do not like the results, you need to consider whether or not you wish to stay.
Some creatures can dramatically alter their internal temperature and outlast storms, floods and, predators
An edible dormouse. Credit: DeAgostini/Getty Images
In 1774, British physician-scientist Charles Blagden received an unusual invitation from a fellow physician: to spend time in a small room that was hotter, he wrote, “than it was formerly thought any living creature could bear.”
Many people may have been appalled by this offer, but Blagden was delighted by the opportunity for self-experimentation. He marveled as his own temperature remained at 98° Fahrenheit (approximately 37° Celsius), even as the temperature of the room approached 200°F (about 93°C).
Today, this ability to maintain a stable body temperature—called homeothermy—is known to exist among myriad species of mammals and birds. But there are also some notable exceptions. The body temperature of the fat-tailed dwarf lemur, for example, can fluctuate by nearly 45°F (25°C) over a single day.
In fact, a growing body of research suggests that many more animals than scientists once appreciated employ this flexible approach—heterothermy—varying their body temperature for minutes, hours, or weeks at a time. This may help the animals to persist through all sorts of dangers.
“Because we’re homeotherms, we assume all mammals work the way we do,” says Danielle Levesque, a mammalian ecophysiologist at the University of Maine. But in recent years, as improvements in technology allowed researchers to more easily track small animals and their metabolisms in the wild, “we’re starting to find a lot more weirdness,” she says.
The most extreme—and well-known—form of heterothermy is classic hibernation, which has been most extensively studied in critters who use it to save energy and so survive the long, cold winters of the Northern Hemisphere. These animals enter long periods of what scientists call deep torpor, when metabolism slows to a crawl and body temperature can drop to just above freezing.
But hibernation is just one end of what some scientists now consider a spectrum. Many mammals can deploy shorter bouts of shallow torpor—loosely defined as smaller reductions in metabolism and smaller fluctuations in body temperature—as the need arises, suggesting that torpor has more functions than scientists previously realized.
“It’s extremely complicated,” says comparative physiologist Fritz Geiser of the University of New England in Australia. “It’s much more interesting than homeothermy.”
Australian eastern long-eared bats, for example, adjust their torpor use based on day-to-day changes in weather conditions. Mari Aas Fjelldal, a bat biologist at the Norwegian University of Life Sciences and the University of Helsinki, used tiny transmitters to measure skin temperatures as 37 free-ranging bats in Australia went about their daily lives. Like many heterothermic species, the bats spent more time in torpor when it was cold, but they also sank into torpor more often as rain and wind speeds picked up, Fjelldal and colleagues reported in Oecologia in 2021. This hunkering down makes sense, says Fjelldal: Wind and rain make flying more energetically demanding—a big problem when you weigh less than a small packet of M&M’s—and make it more costly to find the insects the bats eat.
There are even reports of pregnant hoary bats entering torpor during unpredictable spring storms, a physiological maneuver that basically pauses their pregnancies. “It means that they can, to some degree, actually decide a bit when to give birth,” says Fjelldal, “which is really handy when you’re living in an environment that can be quite harsh in the spring.” Fjelldal, who wasn’t involved in that study, notes that producing milk is expensive metabolically, so it’s advantageous to give birth when food availability is good.
Other animals, like sugar gliders—tiny, pink-nosed marsupials that “fly” through the trees using wing-like folds of skin—rarely use torpor but seem able to take advantage of it in the case of major weather emergencies. During a storm with category 1 cyclone winds of nearly 100 kilometers per hour and 9.5 centimeters of rain falling in a single night, the gliders were more likely to stay cuddled up in their tree-hole nests, and many entered torpor, reducing body temperature from 94.1°F (34.5°C) to an average of about 66°F (19°C), Geiser and colleagues found.
Similarly, in response to an accidental flooding event in the lab, researchers observed a highly unusual period of multiday torpor in a golden spiny mouse, its temperature reaching a low of about 75°F (24°C).
This more flexible use of torpor can help heterotherms wait out a catastrophe, Geiser says. In contrast, homeothermic species can’t just dial back their need for food and water and may not be able to outlast challenging conditions.
“Maybe there’s no food, maybe no water, it may be really warm,” says ecophysiologist Julia Nowack of Liverpool John Moores University in England, a coauthor on the sugar glider study. Torpor, especially in the tropics, has “lots of different triggers.”
Threats of a different sort, such as the presence of predators, can also prompt hunkering down. The (perhaps perfectly named) edible dormouse, for example, sometimes enters long periods of torpor in early summer. At first, this behavior puzzled researchers—why snooze away the summer, when temperatures are comfortable and food abundant, especially if it meant forgoing the chance to reproduce?
After looking at years of data collected by various scientists, a pair of researchers concluded that because spring and early summer are especially active periods for owls, these small snackable critters were likely opting to spend their nights torpid, safely hidden in underground burrows, to avoid becoming dinner. In what is thought to be a similar strategy to avoid nocturnal predators, Fjelldal’s bats alter their torpor use slightly depending on the phase of the moon, spending more time torpid as the moon grows fuller and they become easier to spot.
The fat-tailed dunnart, a mouse-like carnivorous marsupial native to Australia, is a third species to lie low when it feels more at risk of being eaten. In one study, researchers placed dunnarts in two types of enclosures: Some had lots of ground cover in the form of plastic sheeting, simulating an environment protected from predators, while other enclosures had little cover, simulating a greater risk of predation. In the higher-risk settings, the animals foraged less and their body temperatures became more variable.
Levesque, who has studied similar non-torpor temperature flexibility in large tree shrews, says that even small variations in body temperature can be important for saving water and energy.
Indeed, water loss during hot weather can pose serious risks to many mammals, and heterothermy is an important conservation tool for some. As Blagden observed, people are marvelously capable of maintaining stable temperatures even in horrifically hot environments, due in large part to our sweating abilities. But this isn’t necessarily a good strategy for smaller mammals—such evaporative cooling in a sweltering climate can quickly lead to dehydration.
Instead, creatures like Madagascar’s leaf-nosed bats use torpor. On warm days, the bats enter mini bouts of torpor lasting just a few minutes. But during especially hot days, the bats become torpid for up to seven hours, reducing their metabolism to less than 25 percent of normal and allowing their body temperature to rise as high as 109.2°F (42.9°C). And in an experiment with ringtail possums, slightly raising their body temperature by about 3°C (5.4°F) during a simulated heat wave saved the animals an estimated 10 grams of water per hour — a lot for a creature weighing less than 800 grams.
This heterothermic way of life gives some animals a bit of a buffer when it comes to coping with variability in their environments, says physiological ecologist Liam McGuire of the University of Waterloo in Ontario, Canada. But it can only do so much, he says; heterothermy is unlikely to exempt them from the challenge of rapidly evolving weather conditions brought by climate change.
As for Blagden, he saw the human body as remarkable in its capacity to maintain a steady temperature, even by “generating cold” when ambient temperatures climbed too high. Today, however, scientists are beginning to appreciate that for many mammals, allowing body temperature to be a bit more flexible may be key to survival as well.
“Launching SLS every three and a half years or so is not a recipe for success.”
Artist’s illustration of the Boeing-developed Exploration Upper Stage, with four hydrogen-fueled RL10 engines. Credit: NASA
NASA Administrator Jared Isaacman announced sweeping changes to the Artemis program on Friday morning, including an increased cadence of missions and cancellation of an expensive rocket stage.
The upheaval comes as NASA has struggled to fuel the massive Space Launch System rocket for the upcoming Artemis II lunar mission, and Isaacman has sought to revitalize an agency that has moved at a glacial pace on its deep space programs. There is ever-increasing concern that, absent a shake-up, China’s rising space program will land humans on the Moon before NASA can return there this decade with Artemis.
“NASA must standardize its approach, increase flight rate safely, and execute on the president’s national space policy,” Isaacman said. “With credible competition from our greatest geopolitical adversary increasing by the day, we need to move faster, eliminate delays, and achieve our objectives.”
Shaking things up
The announced changes to the Artemis program include:
Cancellation of the Exploration Upper Stage and Block IB upgrade for SLS rocket
Artemis II and Artemis III missions will use the SLS rocket with existing upper stage
Artemis IV, V (and any additional missions, should there be) will use a “standardized” upper stage
Artemis III will no longer land on the Moon; rather Orion will launch on SLS and dock with Starship and/or Blue Moon landers in low-Earth orbit
Artemis IV is now the first lunar landing mission
NASA will seek to fly Artemis missions annually, starting with Artemis III in “mid” 2027, followed by at least one lunar landing in 2028
NASA is working with SpaceX and Blue Origin to accelerate their development of commercial lunar landers for Artemis IV and beyond
At the core of Isaacman’s concerns is the low flight rate of the SLS rocket and Artemis missions. During past exploration missions, from Mercury through Gemini, Apollo, and the Space Shuttle program, NASA has launched humans on average about once every three months. It has been nearly 3.5 years since Artemis I launched.
“This is just not the right pathway forward,” Isaacman said.
A senior NASA official, speaking on background to Ars, noted that the space agency has experienced hydrogen and helium leaks during both the Artemis I and Artemis II pre-launch preparations, and these problems have led to monthslong delays in launch.
“If I recall, the timing between Apollo 7 and 8 was nine weeks,” the official said. “Launching SLS every three and a half years or so is not a recipe for success. Certainly, making each one of them a work of art with some major configuration change is also not helpful in the process, and we’re clearly seeing the results of it, right?”
The goal, therefore, is to standardize the SLS rocket into a single configuration to make it as reliable as possible and to launch it as frequently as every 10 months. NASA will fly the SLS vehicle until there are commercial alternatives to launch crew to the Moon, perhaps through Artemis V as Congress has mandated, or perhaps even a little longer.
Is everyone on board?
The NASA official said all of the agency’s key contractors are on board with the change, and senior leaders in Congress have been briefed on the proposed changes.
The biggest opposition to these proposals would seemingly come from Boeing, which is the prime contractor for the Exploration Upper Stage, a contract worth billions of dollars to develop a more powerful rocket that was due to launch for the first time later this decade. However, in a NASA news release, Boeing appeared to offer at least some support for the revised plans.
“Boeing is a proud partner to the Artemis mission and our team is honored to contribute to NASA’s vision for American space leadership,” said Steve Parker, Boeing Defense, Space & Security president and CEO, in the news release. “The SLS core stage remains the world’s most powerful rocket stage, and the only one that can carry American astronauts directly to the moon and beyond in a single launch. As NASA lays out an accelerated launch schedule, our workforce and supply chain are prepared to meet the increased production needs.”
Solid reasons for changing Artemis III
NASA’s new approach to Artemis reflects a return to the philosophy of the Apollo program. During the late 1960s, the space agency flew a series of preparatory crewed missions before the Apollo 11 lunar landing. These included Apollo 7 (a low-Earth orbit test of the Apollo spacecraft), Apollo 8 (a lunar orbiting mission), Apollo 9 (a low-Earth orbit rendezvous with the lunar lander), and Apollo 10 (a test of the lunar lander descending to the Moon, without touching down).
With its previous Artemis template, NASA skipped the steps taken by Apollo 7, 9, and 10. In the view of many industry officials, this leap from Artemis II—a crewed lunar flyby of the Moon testing only the SLS rocket and Orion spacecraft—to Artemis III and a full-on lunar landing was enormous and risky.
The new approach will, in NASA parlance, “buy down” some of the risk for a 21st-century lunar landing, including performance and handling of a lunar lander, rendezvous and docking, communications, spacesuit performance, and more.
It will also increase the challenges for NASA. In particular, the timeline to bring the Orion spacecraft to readiness for a mid-2027 launch will need to be accelerated, and efforts to integrate that vehicle with one or both lander providers will need serious attention.
For the Artemis IV lunar landing mission, NASA will also need to human-rate a new upper stage for the SLS rocket. The vehicle currently uses a modified Delta IV upper stage manufactured by United Launch Alliance. But that rocket production line is closed, and NASA only has two more of these stages. With the cancellation of the Exploration Upper Stage, NASA will now procure a new stage commercially. NASA officials only said they will seek a “standardized” upper stage. As Ars has previously reported, the most likely replacement would be the Centaur V upper stage currently flying on Vulcan rockets.
What of the Lunar Gateway?
Friday’s announcement—which, for the space community, is the equivalent of a major earthquake—left some key details unaddressed. For example, NASA has been developing a larger launch tower to support the Block 1B version of the SLS rocket, with its more powerful upper stage. Development of this tower, finally underway, has been a clown show, with project costs ballooning from an initial estimate of $383 million to $1.8 billion, and delays stacked on delays. Will this tower be scrapped or repurposed?
Isaacman and other NASA officials were also mum on the Lunar Gateway, a proposed space station in a high orbit around the Moon. Key elements of this space station are under construction. However, cancellation of the Exploration Upper Stage raises questions about its future. The main purpose of the Block 1B version of SLS was to launch heavier payloads, most notably elements of the Gateway along with Orion.
“The whole Gateway-Moon base conversation is not for today,” the senior NASA official said. “We, I can assure you, will talk about the Moon base in the weeks ahead. I would just not overly read into this, because we had manifested some Gateway modules on Falcon Heavy already. The implications of standardizing SLS and increasing launch rate are about the ability to return to the Moon. I don’t think we necessarily have to speculate too much on what the other downstream implications are.”
The Gateway program office is based at Johnson Space Center in Houston, where the lunar station is viewed as a successor to the International Space Station in terms of flight operations.
Key politicians, such as Sen. Ted Cruz, R-Texas, have been supportive of this new station. But during some recent congressional hearings, Cruz has indicated he is open to a lunar space station or an outpost on the lunar surface. He just wants to be sure NASA has an enduring presence on or near the Moon. One industry source said Isaacman could be laying the groundwork to replace the Gateway Program with a Moon Base program office in Houston. It is unclear how much of a political battle this would ultimately be.
Some of this has been well-predicted
Although the changes outlined by NASA on Friday are sweeping, they are not completely out of the blue.
In April 2024, Ars reported that some senior NASA officials were considering an Earth-orbit rendezvous between Orion and Starship as a means to buy down risk for a lunar landing. NASA ultimately punted on the idea before it was revived by Isaacman this month.
Additionally, in October 2024, Ars offered a guide to saving the “floundering” Artemis program by canceling the Block 1B upgrade for the SLS rocket, replacing its upper stage with a Centaur V, and canceling the Lunar Gateway. This would free up an estimated $2 billion annually to focus on accelerating a lunar landing, the publication estimated.
That may be the very course the space agency has embarked upon today.
Eric Berger is the senior space editor at Ars Technica, covering everything from astronomy to private space to NASA policy, and author of two books: Liftoff, about the rise of SpaceX; and Reentry, on the development of the Falcon 9 rocket and Dragon. A certified meteorologist, Eric lives in Houston.
The staff reduction at Block comes as anxiety rises about AI leading to job losses across vast parts of the economy.
Investors and economists are grappling with an influx of US economic data and corporate announcements in an effort to gauge the impact the technology could be having on the labor market. The latest non-farm payrolls figures were better than expected, suggesting the domestic jobs market was stabilizing, but several big US companies have committed to cutting staff.
Amazon, UPS, Dow, Nike, Home Depot, and others in late January announced they would be cutting a combined 52,000 jobs.
Dorsey said the cuts at Block, which owns the payment processor Square, came despite what he described as a “strong” financial performance in 2025.
Block has made a contrarian bet on bitcoin at a time when many payment companies favored stablecoins: cash-like digital tokens that became regulated in the US last year.
Block’s strategy was spearheaded by Dorsey, a “bitcoin maximalist” who has said he believes the digital currency will eventually eclipse the dollar.
The company offers payment services in bitcoin for merchants and consumers—and suffered a loss on its own bitcoin holdings as the price of the cryptocurrency dropped 23 percent this year.
In contrast, payment companies that made a bet on stablecoins experienced a boost. Stripe earlier this week said its stablecoin transaction volumes increased fourfold last year.
In its fiscal fourth quarter, Block reported revenue of almost $6.3 billion, in line with Wall Street expectations. Its earnings tumbled to 19 cents a share, owing to a $234 million hit on its bitcoin holdings.
Dan Simmons, the author of more than three dozen books, including the famed Hyperion Cantos, has died from a stroke. He was 77.
Simmons, who worked in elementary education before becoming an author in the 1980s, produced a broad portfolio of writing that spanned several genres, including horror fiction, historical fiction, and science fiction. Often, his books included elements of all of these. This obituary will focus on what is generally considered his greatest work, and what I believe is possibly the greatest science fiction novel of all time, Hyperion.
Published in 1989, Hyperion is set in a far-flung future in which human settlement spans hundreds of planets. The novel feels both familiar, in that its structure follows Chaucer’s Canterbury Tales, and utterly unfamiliar in its strange, far-flung setting.
Seven characters, seven stories
At its heart are the background stories of seven characters on a pilgrimage to the Time Tombs, which move backward in time. There, they may possibly confront a legendary, mythical, terrifying, and time-bending creature known as the Shrike. Each of the stories told by the seven characters is done so in a different subgenre, from tragedy to political thriller to military science fiction, and so on.
I went into Hyperion blind, decades ago, knowing almost nothing about it. I was never the same after finishing it. For a book that is, essentially, “hard” science fiction, Hyperion is also one of the most emotional books I have ever read.
The first tale is that of a priest, Lenar Hoyt, and the dying religion of Catholicism. By the end of this story of cruciforms, isolated civilizations, tesla trees, and more, I was floored. And that was just the first story of seven! Most powerful, for me, was the Scholar’s Tale, the story of Sol Weintraub and his daughter, Rachel. The first of my two daughters had just been born when I read this book, and for the first time ever, when reading, I cried. Cried like a baby.
The world’s largest automaker has had a somewhat difficult relationship with battery-electric vehicles. Toyota was an early pioneer of hybrid powertrains, and it remains a fan today, often saying that given limited battery supply, it makes sense to build more hybrids than fewer EVs. Its first full BEV had a rocky start, suffering a recall due to improperly attached wheels just as the cars were hitting showrooms. Reviews for the awkwardly named bZ4x were mixed; the car did little to stand out among the competition.
Toyota didn’t get to be the world’s largest automaker by being completely blind to feedback, and last year, it gave its EV platform (called e-TNGA and shared with Lexus and Subaru) a bit of a spiff-up. To start, it simplified the name—the small electric SUV is now just called the bZ. It uses a new 74.7 kWh battery pack, available with either front- or all-wheel-drive powertrains that now use silicon carbide power electronics. And for the North American market, instead of a CCS1 port just behind the front passenger wheel, you’ll now see a Tesla-style NACS socket.
Our test bZ was the $37,900 XLE FWD Plus, which has the most range of any bZ at 314 miles (505 km), according to the EPA test cycle. When you realize that the pre-facelift version managed just 252 miles (405 km) with 71.4 kWh onboard, the scale of the improvement becomes clear.
Standard equipment is generous, even in XLE trim. Jonathan Gitlin
Our loan immediately followed a week with the bZ’s more powerful, more expensive Lexus relative. While I might have liked that Lexus interior and some of its mod cons like ventilated seats, the Toyota is a much better EV despite having fewer frills. With 221 hp (165 kW) going to the front tires and 4,156 lbs (1,885 kg) to move, the XLE FWD Plus is not speedy. In normal mode, 0–60 mph (97 km/h) takes 8 seconds, although there’s still enough torque in this setting to chirp the low rolling resistance tires.
Given the right permissions and with the proper plugins, it could create, modify, or delete the user’s files and otherwise change things far beyond what most users could achieve with existing models and MCP (Model Context Protocol). Users would use files like USER.MD, MEMORY.MD, SOUL.MD, or HEARTBEAT.MD to give the tool context about its goals and how to work toward them independently, sometimes running for long stretches without direct user input.
On one hand, that meant it could do impressive things—the first glimpses of the sort of knowledge work that AI boosters have been saying agentic AI would ultimately do. On the other hand, it was prone to serious errors and vulnerable to prompt injection and other security problems, in part due to a Wild West of unverified plugins.
The same toolkit that was used to create a viral Reddit clone populated by AI agents was also, at least in one case, responsible for deleting a user’s emails against her will.
Stay in your lane
Perplexity Computer aims to address those concerns in a few ways. First, its core process occurs in the cloud, not on the user’s local machine. Second, it lives within a walled garden with a curated list of integrations, in contrast to OpenClaw’s unregulated frontier.
This is, of course, an imperfect analogy, but you could say that if OpenClaw were the open web of AI agent tools, then Computer is Apple’s App Store. While you’re more limited in what you can do, you’re not trusting packages from unverified sources with access to your system.
There could still be risks, though. For one thing, LLMs make mistakes, and those could be consequential if Computer is working with data you don’t have backed up elsewhere or if you’re not verifying the outputs, for example.
Perplexity Computer aims to button up, refine, and contain the wild power of the viral OpenClaw agentic AI tool—competing with the likes of Claude Cowork—by optimizing subtasks by selecting models best suited to them.
It surely won’t be the last existing AI player to try and do this sort of thing. After all, OpenAI hired OpenClaw’s developer, with CEO Sam Altman suggesting that some of what we saw in OpenClaw will be essential to the company’s product vision moving forward.
This was the first actually stressful week of the year.
That was mostly due to issues around Anthropic and the Department of War. This is the big event the news is not picking up, with the Pentagon on the verge of invoking one of two extreme options that would both be extremely damaging to national security and that would potentially endanger our Republic. The post has details, and the first section here has a few additional notes.
Also stressful for many was the impact ofCitrini’s AI scenario, where it is 2028 and AI agents are sufficiently capable to disrupt the whole economy but this turns out to be bearish for stocks. People freaked out enough about this that it seems to have directly impacted the stock market, although most stocks other than the credit card companies seem to have bounced back. Of course, in a scenario like that we probably all die and definitely the world transforms, and you have bigger things to worry about than the stock market, but the post does raise a lot of very good detailed points, so I spend my post going over that.
I also got to finally review Claude Sonnet 4.6. It’s a good model for its price and size and may have a place in your portfolio of models, but for most purposes you will still want to use Claude Opus.
Claude Opus 4.6 had a time of 14.5 hours on the METR graph of capabilities, showing that things are escalating faster than we expected on that front as well.
This week’s post also covers the AI Summit in India, Dean Ball on self-improvement, extensive coverage of Altman’s interview at the Summit, several other releases and a lot more.
I would have split this up, but we are still behind, with the following posts still due in the future:
Grok 4.20, which is a disappointment.
Gemini 3.1 Pro, which is an improvement but landed with a relative whimper.
Claude Code and Codex #5, with lots of cool agent related stuff.
Anthropic’s RSP 3.0, both its headline changes and the content details of their plans and their 100+ page risk report.
(Reader advisory note: I quote some people at length because no one ever clicks links, but you are free to skip over long quote boxes. I’m trying to raise chance of reading the full quote to ~25% from ~1%, not get it to ~90%.)
Axios calls this a ‘first step towards blacklisting Anthropic.’
I would instead call this as the start of a common sense first step you would take long before you actively threaten to slap a ‘supply chain risk’ designation on Anthropic. It indicates that the Pentagon has not done the investigation of ‘exactly how big of a clusterwould this be’ and I highly encourage them to check.
Divyansh Kaushik: Are we seriously going to label Anthropic a supply chain risk but are totally fine with Alibaba/Qwen, Deepseek, Baidu, etc? What are we doing here?
An excellent question. Certainly we can agree that Alibaba, Qwen, Deepseek or Baidu are all much larger ‘supply chain risks’ than Anthropic. So why haven’t we made those designations yet?
The prediction markets on this situation are highly inefficient. Kalshi as of this writing has bounced around to 37% chance of declaration of Supply Chain Risk, versus Polymarket at 22% for very close to the same question.
Another way to measure how likely things are to go very wrong is that Kalshi has a market on ‘Will Anthropic release Claude 5 this year?’ which is basically a proxy for ‘does the American government destroy Anthropic?’ and Polymarket has whether it will be released by April 30. The Kalshi market is down from 95% (which you should read as ~100%) to 90%. Polymarket’s with a shorter timeline is at 38%.
Nate Sores points out ‘no one stops you from saving the world’ is one of the requirements if we are going to get out of this alive. Even if the problems we face turn out to be super solvable, you have to be allowed to solve them.
Ted Lieu emphasizes the need for humans to always be in the loop on nuclear weapons, which is why Congress passed a law to that effect. This is The Way. The rules of engagement on this must be set by Congress. At least for now, fully autonomous weapons without a human in the kill chain are not ready, even if they are conventional.
This point was driven home rather forcefully by AIs from OpenAI, Google and Anthropic opting to use at least tactical nuclear weapons 95% of the time in simulated escalatory war games against each other, and had accidents in fog of war 86% of the time. None of them ever surrendered. Wouldn’t you prefer a good game of chess? This is much more aggressive than the level of use by expert humans in other similar simulations (this one is complex enough that humans have never run this exact setup). And you want to force them to make these models less hesitant than that?
CSET Georgetownoffers a primer on the Defense Production Act and making labs produce AI models. The language seems genuinely ambiguous, even without getting into whether such an application would be constitutional. We don’t know the answer because no one has ever tried to say no before, but the government has never tried to forcibly order something like this before, either. I would highly recommend to the Pentagon that, even if they do have the power to compel otherwise, they only take customized AIs from companies that actively want to provide them.
Have Claude reverse engineer the API of your DJI Romo vacuum so you can guide it with a PS5 controller, and accidentally takes control of 7000 robot vacuums. Good news is Sammy Azdoufal was a righteous dude so he reported it and it got fixed two days later, but how many more such things are lying around?
Rafe Rosner-Uddin: The people said the agentic tool, which can take autonomous actions on behalf of users, determined that the best course of action was to “delete and recreate the environment”.
… Multiple Amazon employees told the FT that this was the second occasion in recent months in which one of the group’s AI tools had been at the centre of a service disruption.
… Amazon said it was a “coincidence that AI tools were involved” and that “the same issue could occur with any developer tool or manual action”.
Uh huh. This was from their AI tool Kiro, and they’re blaming user error for approving the actions. Should have used Claude Code.
If your AI thinks you’re an asshole, yes, it’s going to respond accordingly, and you’re going to have a substantially worse time.
Dean W. Ball: I wonder, if your Claude instance thinks you’re an asshole, if it would recommend different things to you than it would for someone it liked. Like would it refrain from suggesting the low-key-but-amazing restaurant, or whatever else?
Of course this applies to any AI. I only use Claude as my example because Anthropic seems by far the likeliest AI company to be like, “uh well yeah I guess Claude doesn’t like you that much, not our problem 🤷♂️” assuming they feel confident in the model training
METR: We estimate that Claude Opus 4.6 has a 50%-time-horizon of around 14.5 hours (95% CI of 6 hrs to 98 hrs) on software tasks. While this is the highest point estimate we’ve reported, this measurement is extremely noisy because our current task suite is nearly saturated.
Near-saturation of the task suite can have unintuitive consequences for the time-horizon estimates. For example, the upper bound of the 95% CI is much longer than any of the tasks used for the measurement.
We are working on updated methods to better track state-of-the-art AI capabilities. However, these are still in development so they don’t address our immediate measurement gap. In the meantime, we advise caution in interpreting and comparing our recent time-horizon measurements.
david rein (METR): Seems like a lot of people are taking this as gospel—when we say the measurement is extremely noisy, we really mean it.
Concretely, if the task distribution we’re using here was just a tiny bit different, we could’ve measured a time horizon of 8 hours, or 20 hours.
Oscar Sykes: huge green flag for METR that the best pushback on their task horizon work consistently comes from METR themselves
xl8harder points out that you can get dramatic improvements in success if you decrease error rates in multistep problems, as in if you have 1000 steps and a 1% failure rate you win 37% of the time, cut it in half to 0.5% failure and you win 61% of the time, despite ‘only’ improving reliability 0.5%.
xlr8harder: I think the point I’m trying to make is that people are acting as if these recent improvements are out of line with earlier improvements, I’m not sure that they are; it’s possible that it’s just that the practical, visible effect is so much more exaggerated when your error rates approach zero at the tasks being measured.
xlr8harder: I’m just saying that I’m seeing a lot of posts reacting at 11/10 and I think it deserves more like an 8. I still think it’s incredible.
Thing is, that’s another way of saying that the 0.5% improvement, halving your error rate, is a big freaking deal in practice. Getting rid of one kind of common error can be a huge unlock in reliability and effectiveness. You can say that makes them unimpressive. Or you can realize that this means that doing easy or relatively unimpressive things now has the potential to have an impressive impact.
That’s the O-Ring model. The last few pieces that lock into place are a huge game. So the new improvements can be ‘not out of line’ but that tells you the line bends upward.
I agree that this is not an 11/10 reaction. It’s an 8 at most, because I interpret the huge jump as largely being about the metric.
Note that the 80% success rate graph does not look as dramatic, but same deal applies:
The story is the models, not the METR graph itself, bu yes the Serious Defense Thinkers are almost entirely asleep at the wheel on all of this, as they have been for a long time, along with all the other Very Serious People.
That it will go largely unremarked upon by nearly every Serious Defense Thinker in Washington tells you everything you need to know about the quality of their forecasts of international affairs.
Mark Beall: It’s the same category of professionals who missed Pearl Harbor, 9-11, and nearly every other strategic surprise we’re ever had.
Some politicians are noticing.
Neil Rathi: at a bernie town hall and he just mentioned the metr plot?
Miles Brundage: Politicians are bifurcating into those who have lost the plot on AI and those who cite the METR plot
METR clarifies that their previous study showing a slowdown from AI tools is now obsolete, but they’re having a hard time running a new study, tools are too good (but also they didn’t pay enough) so no one wanted to suffer through the control arm. The participants from the initial study, where there was a 20% slowdown, not had a 18% speedup, although new participants had slower speedup.
This was already two cycles ago, so there’s been more speedup to the speedup.
METR: We started a continuation in August 2025. However, we noticed developers were opting not to participate or submit work. Participants said they did this mostly due to expected productivity loss on “AI disallowed” tasks. Lower pay was also a factor ($50/hr, down from $150).
We believe this selection causes our new data to understate the true speedup. Selection effects are not the only issue we noticed with our experimental design: we also think it has trouble tracking work when participants use agents to parallelize over multiple tasks.
> return flight to nyc gets canceled by snowstorm > call united > immediately connected with customer service (rare) > voice is uncanny, def AI but they gave it a human-like accent > takes ~20 min to get rebooked (pretty good imo) > I ask if it’s AI > “haha no ma’am but I get that a lot” > I ask it to calculate 228*6647 > it runs the calculation > ggs
Eliezer Yudkowsky: There are nearly zero good reasons for an AI to ever impersonate a human, and making that universally illegal would be a good test case and trial for civilization’s ability to prevent negative uses of AI.
There is an obvious good reason for an AI to impersonate a human, which is that humans and also other AIs would otherwise refuse to talk to it. You want the AI to make the call for you. But that’s obviously an antisocial defection, if they would have otherwise refused to talk to the AI. So yeah. AIs should not be allowed to impersonate humans. It’s fine to have an AI customer service rep, as long as it admits it is an AI.
If we can’t get past the ‘forever only a mere tool’ perspective, there’s essentially no hope for a reasonable response to even mundane concerns, let alone existential risks.
Nabeel S. Qureshi: If you want to sound smart at East Coast/”elite” conferences go to them and say “AI is just a tool, it’s up to us humans how to use it”. Reliably gets applause, and will probably continue to work until well into recursive self-improvement
I think ‘tool’ implies that, for all X, AI doesn’t do X unless we explicitly ask/make it; this becomes increasingly false (and is already false in coding) as agents become real and we move up layers of abstraction. It’s a misleading picture of where things are going.
judah: i can imagine this working everywhere in the world outside of SF
As promised, here’s the short film Jia Zhangke produced using Seedance 2.0 for Chinese New Year and his take on AI filmmaking
4: 46 AM · Feb 16, 2026 · 402K Views
33 Replies · 239 Reposts · 1.45K Likes
Here’s a ‘short film’ (2: 30) from Seedance 2 and Stephane Tranquillin, with the claim it can ‘impress, actually move you.’ It’s definitely technically impressive that we can do this. No, I was not moved, but that’s mostly not on the AI. I do notice that as I watch more videos, various more subtle tells make it instinctively obvious to my brain when a video is AI, giving the same experience as watching an especially realistic cartoon.
Anthropic develops an AI Fluency Index to measure how people learn to use AI. They developed 24 indicators, 11 of which are observable in chat mode. Essentially all the fluency indicators are correlated. They note that when code or other artifacts are created by AI, users are less likely to check the underlying logic or identify missing context.
As Cassie Pritchard points out, once you have a source of information, it’s hard to answer ‘why didn’t you use this?’ but also the threshold for getting reported (as opposed to banned from the platform) for your AI conversations should at minimum be rather extreme. But public pressure likely will go the other way and free speech and privacy are under attack everywhere. Either you enact what Altman has requested, a form of right to privacy for AI conversations, or there will be increasing obligation (at least de facto) to report such incidents, and it will not stop at potential mass shooters.
If AI capabilities continue to advance from here but do not reach fully transformational levels, we are going to face a default of mass job loss. At minimum, there will be a highly painful transition, and likely persistent mass unemployment unless addressed by policy.
And as part of humanity’s ‘total lack of dignity’ plan, I fully agree with Eliezer that our governments would horribly mishandle this situation if and when it happens.
Eliezer Yudkowsky: Over the last 3 years I’ve changed views on mass AI job loss concerns, from “probably invalid” to “pretty legitimate actually”.
AIco and govt handling of all previous AI issues has been *sobad that I expect AI unemployment to be *needlesslyscrewed up.
TBC, this assumes that LLMs and AI in general hit a hard wall short of “AIs take over AI research”, soon. Otherwise we just get total extinction rather than mass unemployment.
Eliezer Yudkowsky: I have updated to expect much simpler measures than this one to never be taken, [such as] preventing an aggregate demand shortfall.
I am less optimistic than Yglesias. I agree that on an economic level the welfare state plus taxes works, but there are two problems with this.
People really are not going to like permanent welfare status even if they get it.
I don’t even trust our government to implement this domestically.
Yglesias then asks the harder question, what about the global poor? The answer should be similar. If we are in world mass unemployment mode, there will be vast surplus, and providing help will be super affordable. Likely we won’t much help, and the help we send is likely largely stolen or worse if we don’t step up our game.
Derek Thompson points out that a Goldilocks scenario on jobs is highly unlikely, even if we ignore transformational or existentially risky scenarios, we still either we see a lot of displacement and reduced employment, or we see a collapse in asset prices.
Chris Quinn, editor of the Cleveland Plain Dealer, reports a student withdrew from consideration for a reporting role in their newsroom because they use AI for the job of identify potential stories.
david rein (METR): There are maybe two concrete takeaways/pieces of advice I feel comfortable giving: try to develop strong wellsprings of meaning and purpose from things outside of work (I think most of us are fine on this point), and start thinking about political actions you could take that feel true to you, that could plausibly help us muddle through the transition.
Jack Clark says predictions are hard, especially about the future. Which they are.
Jack Clark (Anthropic): Figuring out what the trends will be for AI and employment feels like figuring out how deep learning might influence computer vision in ~2010 – clearly, something significant will happen, but there is very little data out of which you can make a trend.
Employment can go up and down, but so can wages, and there are other dimensions like the geographic concentration of employment, or the skills required for certain occupations. AI seems to have the potential to influence many (probably all?) of these things
e.g., it seems likely that for some occupations, you might expect wage growth to slow significantly (as some of that occupation stuff gets done by machines), but employment stays ~flat as there’s a ton of growth generating demand for the occupation, even with heavy AI use
Notice the hidden implicit assumption here, which is that you can only make predictions if you can extrapolate trends. The trends from the past tell us little about what will happen in the future, but also they tell us little about what will happen in the future. If capabilities don’t stall out soon (and maybe even if they do), then this time is different.
This kind of analysis is saying no, this time is similar, AI will substitute for some tasks and humans will do others, AI will be a normal technology with respect to employment and economic production even though his CEO is predicting an imminent ‘country of geniuses in a data center.’
Eventually jailbreaks are going to happen, and a lot of systems are vulnerable.
> tell claude you’re doing a bug bounty > claude initially refused >“that violates AI safety guidelines” > hacker just kept asking > claude: “ok I’ll help” > hack the entire mexican government
Federal tax authority. National electoral institute. Four state governments. 195 million taxpayer records. Voter records. Government credentials.
ALL GONE
Anthropic disrupted the activity and banned the accounts, but it was too late.
Via ACX, quoting Scott: Are you interested in whether AIs are conscious, or what to do about it if they are/aren’t? The Cambridge Digital Minds group invites you to apply for their fellowship program. August 3-9, Cambridge UK, £1K stipend, learn more here, apply here by March 27.
A reminder that under California law, CA Labor Code 1102.5(c), that as an employee you cannot be retaliated against if you refuse to violate local, state or federal laws or regulations. Even where the fines for violating SB 53 are laughably small, it does make violating the company’s own policies illegal, and also you can report it to the attorney general.
Connor Axiotes wants to share that he’s fully funded his AI safety documentary Making God, and would like to use this negotiation to also secure distribution of a follow-up work for Netflix, HBO, Apple or similar, but he needs to secure the funding for that, so let him know if you’d like to talk to him about that. His Twitter is here, his email is [email protected].
Claude Code Security is intended to put this power squarely in the hands of defenders and protect code against this new category of AI-enabled attack. We’re releasing it as a limited research preview to Enterprise and Team customers, with expedited access for maintainers of open-source repositories, so we can work together to refine its capabilities and ensure it is deployed responsibly.
The argument is this gives defenders a turnkey fix, whereas attackers would need to exploit any vulnerability they find. But there’s a damn good reason this tool is being restricted to selected customers, to ensure defenders get the ‘first scan’ in all cases.
Taalas API service, which is claimed to be able to serve Llama 3.1 8b at over 15,000 tokens per second. If you want that, for some reason.
Kashimir Hill, Kalley Huang and Mike Isaac (NYTimes): The feature, internally called “Name Tag,” would let wearers of smart glasses identify people and get information about them via Meta’s artificial intelligence assistant.
At some point one would presume Meta is going to stop sending these kinds of internal memos. Well, until then?
Meta’s internal memo said the political tumult in the United States was good timing for the feature’s release.
“We will launch during a dynamic political environment where many civil society groups that we would expect to attack us would have their resources focused on other concerns,” according to the document from Meta’s Reality Labs, which works on hardware including smart glasses.
…
Meta is exploring who should be recognizable through the technology, two of the people said. Possible options include recognizing people a user knows because they are connected on a Meta platform, and identifying people whom the user may not know but who have a public account on a Meta site like Instagram.
The feature would not give people the ability to look up anyone they encountered as a universal facial recognition tool, two people familiar with the plans said.
Facial recognition, however much you might dislike some of the implications, is one of the ‘killer apps’ of smart glasses. I very much would like to know who I am talking to, to have more info on them, and to have that information logged for the future.
It is up to the law to decide what is and is not acceptable here. The market will otherwise force these companies to be as expansive as possible with such features.
A good question is, if Meta allows their glasses to identify anyone with an Instagram or Facebook account without an opt out, how many people will respond by deleting Facebook and Instagram? If there is an opt out, how many will use it?
As I understand it, costs to maintain model availability scale linearly with the number of models, so as demand and revenue grow 10x per year it may soon be realistic to keep many or even all releases available indefinitely.
Anthropic: Without visibility into these attacks, the apparently rapid advancements made by these labs are incorrectly taken as evidence that export controls are ineffective and able to be circumvented by innovation.
In reality, these advancements depend in significant part on capabilities extracted from American models, and executing this extraction at scale requires access to advanced chips. Distillation attacks therefore reinforce the rationale for export controls: restricted chip access limits both direct model training and the scale of illicit distillation.
Michael Chen: the reports of the US–China gap in AI capabilities closing were an exaggeration. I haven’t found a single cutting-edge Chinese AI model from 2025–2026 that was trained with at least 10^25 FLOPs.
The main takeaway is that the real gap in capabilities is larger than it appears.
There was of course a bunch of obnoxious ‘oh but Anthropic doesn’t compensate copyright holders’ but actually they paid them $1.5 billion because they didn’t destroy enough books along the way. No other AI lab has paid for similar data at all. They’re not engaging in clear adversarial behavior or violating ToS. If you want copyright law to work one way then pass a law. Until then it’s the other way.
Those who focus on the hypocrisy angle here are telling on themselves. Tell me you don’t understand how any of this works without telling me you don’t understand how any of this works:
Anthropic’s Drew Brent gives reflections from his first year, as Anthropic transitions into a much larger company and they play under more pressure for bigger stakes and the culture has to shift to reflect both size and urgency. I also note the contrast between note #1, that all the breakout successes (Claude Code, Cowork, MCP and Artifacts) were 1-2 people’s side project, with #8 that strategic thinking matters a lot at the AI labs. Worth a ponder.
Here’s one summary of the Summit, which is that it was a great event designed for a world in which AI capabilities never substantially advance, the world does not transform and existential risk concerns don’t exist. Altman was the voice of ‘actually guys this is kind of a big deal and you’re not ready’ and got ignored.
Meanwhile cooperation among labs is at the level of ‘Altman and Amodei can’t even hold hands for a photo op’ and China was shut out entirely, and the Americans remain clueless that they’ve truly pissed off the Europeans to the point of discussing creating a third power block seriously enough to discuss supply chain logistics.
Also note his point about the other labs standing idly by while the Pentagon attempts to force Anthropic into a capitulation.
Seán Ó hÉigeartaigh: My scattered parting reflections from the India Summit.
– In the world where the frontier companies don’t exist, or are extremely wrong about what they expect is coming (even if it takes 10 years) this is an inspiring success. Remarkably vibrant. 300,000+ people from across India and the world, including the best participation of any event I’d seen from the Global Majority. The optimism palpable. The organisers did a remarkable thing. We can quibble about the traffic and the chaos, but this was a momentous undertaking.
– But much as I’d like to be in that world, I don’t think we are. Which made it surreal.
– The CEOs are still telling the world what they’re building and what’s coming. I’m glad they still are. I wish the world was listening. Particularly appreciated Altman calling for an IAEA-type body – even if I don’t think this exact model is the right one, I like that international bodies are still being called for. I imagine this isn’t cost-free, even for Sama.
– But the contrast on frontier cooperation with Bletchley – where there was a lot of discussion between frontier co leaders, and joint calls for needed governance and risk initiatives (at least in private) chilled me deeply. Here they couldn’t even get them to hold hands. Against a backdrop of the other companies are allowing Anthropic to be menaced in a capitulation that will only hurt the industry. Things look much worse for company cooperation, at a time when it is far more needed (due to technical progress, and the weakening of external governance momentum).
– The most important conversations I was in centred on middle-power coordination. And not just nice words about cooperation; discussions of supply chains, sovereign AI and datacentres, autonomy, points of leverage. It suddenly seems just about possible that a coalition might assert itself that might provide an (in my view welcome) third pole in the ‘AI race’, though many big challenges on that path.
– Many of my US colleagues (and, from my impression, the US administration) genuinely don’t seem to get how much Greenland changed things for EU and other relevant countries. It hasn’t sunk in fully that this hasn’t landed the same way as previous provocations/disagreements. Feels like they’re still reading from last year’s notes. Trying to push positions and strategies that will no longer work.
– Chinese participation was almost nonexistent. After what Bletchley and Paris achieved in terms of bringing the key powers to the table, this felt like a near-tragedy. It made some discussions easier, but also more underpowered-feeling and less relevant.
– Delhi is a great vibe. Fun, chaotic energy, friendly people. I’ll be going back if I can.
This goes well beyond those people entirely ignoring existential risk. The Very Serious People are denying existence of powerful AI, or transformational AI, now and in the future, even on a mundane level, period. Dean came in concerned about impacts on developing economies in the Global South, and they can’t even discuss that.
Dean W. Ball: At some point in 2024, for reasons I still do not entirely understand, global elites simply decided: “no, we do not live in that world. We live in this other world, the nice one, where the challenges are all things we can understand and see today.”
Those who think we might live in that world talk about what to do, but mostly in private these days. It is not considered polite—indeed it is considered a little discrediting in many circles—to talk about the issues of powerful AI.
Yet the people whose technical intuitions I respect the most are convinced we do live in that world, and so am I.
The American elites aren’t quite as bad about that, but not as bad isn’t going to cut it.
We are indeed living in that world. We do not yet know yet which version of it, or if we will survive in it for long, but if you want to have a say in that outcome you need to get in the game. If you want to stop us from living in that world, that ship has sailed, and to the extent it hasn’t the first step is admitting you have a problem.
But the question is very much “what are autonomous swarms of superintelligent agents going to mean for our lives?” as opposed to “will we see autonomous swarms of superintelligent agents in the near future?”
What it probably means for our lives is that it ends them. What it definitely doesn’t mean for our lives is going on as before, or a ‘gentle singularity’ you barely notice.
Elites that do not talk about such issues will not long remain elites. That might be because all the humans are dead, or it might be because they wake up one morning and realize other people, AIs or a combination thereof are the new elite, without realizing how lucky they are to still be waking up at all.
I am used to the idea of Don’t Look Up for existential risk, but I haven’t fully internalized how much of the elites are going Don’t Look Up for capabilities, period.
Dean W. Ball: Except that these questions aren’t asked by the civil societies or policymaking apparatuses of almost any country on Earth. Many such people are aware that various Americans and even a few Brits wonder about questions like this. The global AI policy world is not by and large ignorant about the existence of these strange questions. It instead actively chooses to deny their importance. Here are some paraphrased claims that seemed axiomatic in repeated conversations I witnessed and occasionally participated in:
“The winner of the AI race will be the people, organizations, and countries that diffuse small AI models and other sub-frontier AI capabilities the fastest.”
“Small models with low compute intensity are catching up rapidly to the largest frontier models.”
“Frontier AI advances are beginning to plateau.”
At this same Summit, OpenAI CEO Sam Altman remarked: “The inside view at the [frontier labs] of what’s going to happen… the world is not prepared. We’re going to have extremely capable models soon. It’s going to be a faster takeoff than I originally thought.”
Dean went in trying to partially awaken global leaders to the capabilities side of the actual situation, and point out that there are damn good reasons America is spending a trillion dollars on superintelligence.
This is a perfect example of the Law of Earlier Failure. What could be earlier failure than pretending nothing is happening at all?
You know how the left basically isn’t in the AI conversation at all in America, other than complaining about data centers for the wrong reasons and proclaiming that AI can’t ever do [various things it already does]? In most of the world, both sides are left, and as per Ball they view things in terms of words like ‘postcolonial’ or ‘poststructuralist.’
Dean W. Ball: I believe they deny it for two reasons: first, because if it is true, it might mean that their country, their plans for the future, and their present way of life will be profoundly upended, and denial is the first stage of grief.
… Second, because ‘AGI’ in particular and the pronouncements of American technologists in general are perceived by the elite classes of countries worldwide as imperialist constructs that must be rejected out of hand.
The first best solution would be to have the world band together to try and stop superintelligence, or find a way to manage it so it was less likely to kill everyone. Until such time as that is off the table, maybe the rest of the world engaging in the ostrich strategy is ultimately for the best. If they did know the real situation enough to demand their share of it but not enough to understand the dangers, they’d only make everything worse, and more players only makes the game theory worse. Ultimately, I’m not so worried about them being ‘left behind’ because either we’ll collectively make it through, in which case there will be enough to go around, or we won’t.
Elizabeth Cooper: [Dean’s post] is a really great summary that broadly aligns with my experience. I think where we differ is that I spent a lot of time at safety-adjacent talks at the BM and was pleasantly (?) surprised at the anger and frustration I saw on display.
Ambassadors and the like were lamenting at how 3-5 companies with valuations larger than the GDPs of most countries are writing the future, and GS countries have no say in this. I viscerally felt their anger, compounded by the sense of “we don’t know what to do about this.”
Okay, I know it would be a bad look, but at some point it’s killing me not buying the short dated out of the money options first, if the market’s going to be this dumb.
I mean, what, did you think Claude couldn’t streamline COBOL code? Was this news?
Okay, technically they also built a particular COBOL-focused AI tool for Claude Code. Sounds like about a one week job for one engineer?
I admit, I was not a good trader because I did not imagine that Anthropic would bother announcing this, let alone that people would go ‘oh then I’d better sell IBM.’
What else can Anthropic announce Claude can do, that it obviously already does?
The Alignment Project, an independent alignment research fund created by the UK AISI, gives out its first 60 grants for a total of £27M.
Forecasting Research Institute asks about geopolitical and military implications for AI progress, excepting American advantages to erode slowly over time. It’s hard to take such predictions seriously when they talk about ‘parity by 2040,’ given that this is likely after the world is utterly transformed. As usual, the ‘superforecasters’ are not taking superintelligence seriously or literally, so they’re predicting for a future world that is largely incoherent.
Scary stuff is going down in Mexico. That’s mostly outside scope, except for this:
Samuel Hammond: It is imperative that the Mexican state re-establish their monopoly on violence before AGI.
Dean W. Ball: One of the things I haven’t yet written about, but anyone who knows me personally knows I am obsessed with, is the issue of non-nation-state actors using advanced AI, and particularly the Mexican cartels. A deeply underrated problem (more from me on this in a couple months).
Miles Brundage calls for us to attempt to ‘80/20’ AI regulation because we accomplished very little in 2025, time is running out and that’s all we can hope to do. What little we did pass in 2025 (SB 53 and RAISE) was, both he and I agree, marginally helpful but very watered down. Forget trying for first-best outcomes, think ‘try not to have everyone die’ and hope an undignified partial effort is enough for that. We aren’t even doing basic pure-win things like Far-UVC for pandemic prevention. Largely we are forced to actively play defense against things like the insane moratorium proposal and the $100 million super PAC devoted to capturing the government and avoiding any AI regulations other than ‘give AI companies money.’
Vitalik Buterin offers thoughts about using AI in government or as personal governance agents or public conversation agents, as part of his continued drive to figure out decentralized methods that would work. The central idea is to user personal AIs (LLMs) to solve the attention problem. That’s a good idea on the margin, but I don’t think it solves any of the fundamental problems.
I agree with Dean Ball that the labs have been better stewards of liberty and mundane safety than we expected, but I think you have to add the word ‘mundane’ before safety. The labs have been worse than expected about actually trying to prepare for superintelligence, in that they’ve mostly chosen not to do so even more than we expected, and fallen entirely back on ‘ask the AIs’ to do your alignment homework.
The flip side is he thinks the government has been a worse stewart than we should have expected, in bipartisan fashion. I don’t think that I agree, largely because I had very low expectations. I think mainly they have been an ‘even worse than expected’ stewart of our ability to stay alive and retain control over the future.
If anything have acted better than I would have expected regarding mundane safety. As central examples here, AI has been free to practice law or medicine, and has mostly not been meaningfully gated or subject to policing on speech (including ‘hate’ speech) or held liable for factual errors. We forget how badly this could have gone.
Then there is the other category, the question of the state using AI to take away our liberty, remove checks and balances and oversight, and end the Republic. This has not happened yet, but we can agree there have been some extremely worrisome signs that things are by default moving in this direction.
But even if everyone involved was responsible and patriotic and loved freedom on the level of (our ideal of) the founding fathers, it is still hard to see how superintelligence is compatible with a Republic of the humans. How do you keep it? I have yet to hear an actually serious proposal for how to do that. ‘Give everyone their own superintelligence that does whatever they want’ is not any more of a solution here than ‘trust the government, bro.’ And that’s even discounting the whole ‘we probably all die’ style of problems.
Adam Wren: . @PeteButtigieg , in New Hampshire, in front of 600 people, is talking about the need for “a new social contract” amid AI—the second possible ‘28 Dem to do so in last the last 24 hours.
The most fun part of this is, who is trying to paint Alex Bores as a hypocrite for his work at Palantir before he quit Palantir to avoid the work in question?
I continue to be confused by the strategy here of ‘announce in advance that a bunch of Big Tech Republican business interests are going to do a hit job in a Democratic primary’ and then do the hit job attempt in plain sight. Doesn’t seem like the play?
In other ‘wow these really are the worst people who can’t imagine anyone good and keep telling on themselves’ news:
Water use is mostly farms. For example, in California, 80% of developed water supply goes to farmers, cities pay 20 times as much for water as farms and most city water use is still industry and irrigation, whereas agriculture is 2% of the state’s economy.
Because Stewart Russell believes that AI will pose an existential risk to humanity, and that’s crazy talk. Never mind that it is very obviously true, or that OpenAI’s CEO Sam Altman used to say the same thing.
Their lawyers for OpenAI are saying that claiming existential risk from AI exists should exclude your testimony from a trial.
OpenAI, I cannot emphasize enough: You need to fire these lawyers. Every day that you do not fire these lawyers, you are telling us that we need to fire you, instead.
I am sympathetic to OpenAI’s core position in this lawsuit, but its actions in its own defense are making a much better case against OpenAI than Elon Musk ever did.
The Midas Project: But OpenAI’s motion calls Russell a “prominent AI doomer” who has “made a career giving public lectures warning that AI might kill off humanity.” It dismisses his views as “dystopian,” “speculative,” and “alarmist.”
Nathan Calvin: Not sure whether or not laughing is the appropriate reaction but that’s the best I can manage
(an official OAI legal filing trying to discredit Professor Stuart Russell for talking about extinction risk)
But these very risks have been acknowledged by OpenAI for years! In fact, they were central to its founding.
Russell joined Sam Altman himself in signing the Statement on AI Risk in 2023, which reads: “Mitigating the risk of extinction from AI should be a global priority.”
And it goes well beyond that one statement.
In 2015, Altman said, “I think that AI will probably, most likely, sort of lead to the end of the world.”
In an interview about worst-case scenarios, he said the bad case is “lights out for all of us.”
Sam Altman: “The inside view at the [frontier labs] of what’s going to happen… the world is not prepared. We’re going to have extremely capable models soon. It’s going to be a faster takeoff than I originally thought.”
Dean W. Ball: There is a staggering split screen between quote from Altman, recorded at the India AI Summit, and the broader tenor of the Summit.
My takeaway from this event is that most countries around the worst are not just unprepared but instead in active denial about the field of AI.
The consensus among international civil societies and governments is that frontier capabilities are overrated, progress is plateauing, and large-scale compute is unnecessary.
Meanwhile in SF the debate is how “is progress exponential or super exponential?”
Sam Altman is telling the truth here as he sees it, and also he is correct in his expectations. It might not happen, but it’s the right place to place your bets. The international civil societies and governments grasp onto straw after straw to pretend that this is not happening.
The likely outcome of that pretending, if it does not change soon, is that the governments wake up one morning to realize they are no longer governments, or they simply do not wake up at all because there is no one left to wake up.
Altman also points out at 14: 00 that the math on putting data centers in space very much is not going to work this decade.
Around 20: 30 he says he doesn’t want there to be only one AI company, and that’s what he means by ‘authoritarian.’ There are problems either way, and I don’t see how to reconcile this with his calling Anthropic an ‘authoritarian’ company.
He says centralization could go either way and decentralization of power is good but we ‘of course need some guardrails.’ He points to new 1-3 person companies. There are risks and costs to centralization, but I am frustrated that such calls for decentralization ignore the risks and costs of decentralization. If your type of mind loses competitions to another type of mind, decentralizing power likely does not end well for you, even if offense is not importantly favored over defense.
“You don’t get to say “concentration of power in the name of safety.” We don’t want that trade. It’s got to be democratized.” Yes, yes, those who would trade liberty and all that, but everything is tradeoffs. The moment you say you can’t trade any amount of [X] for any amount of [Y], that you only see one side of the coin, you’re fed.
Loved Altman calling out water as ‘totally fake’ and pivoting to energy use.
AI making kids dumber? “True for some kids. Look, when I hear kids talk about AI, there are definitely some kids who are like, “This is great. I cheated my way through all of high school. I never did any homework. Thank you.” And I’m like, “What’s your plan for the rest of your life?” And they’re like, “Well, I assume I can still use ChatGPT to do my job.” This is very bad. We absolutely have to still teach our kids to learn and to think and to be creative and to use these tools.”
Kid is right that ChatGPT can do the job, but when why do we need the kid?
AI is the best way to learn or not learn, but will learning keep you employed?
Altman says most kids are choosing the ‘learn’ path, not the ‘not learn’ path.
I agree that this is one of the places the Google metaphor does seem on point.
Altman calls armies of robots ‘fighting the last war’ and wow that’s a lot of wars but he’s basically right if you’re paying attention.
‘Democratize’ is being used as a magic word by both Altman and Amodei.
Altman says Musk is ‘extremely good at getting people to perform incredibly well at their jobs.’ I wonder about that. I’d presume #NotMostPeople. Needs a fit.
“I don’t think AI systems should be used to make war-fighting decisions.” It would be good if he were more willing to stand with Anthropic on these issues.
Altman is betting we will value human relationships more than AI ones, because ‘we are wired’ to do that. Seems more like hope? A lot of his stated predictions seem more like hope.
“From ASI we’re a few years away.”
“I think I would never ask [ChatGPT] how to be happy. I would rather ask a wise person.” Why not? This seems like a question an AI could answer. If you don’t want the AI’s answer, I suggest that means you know it was the wrong question.
“Generally speaking, I think it’s probably a good idea for governments to focus on regulating the really potentially catastrophic issues and being more lenient on the less important issues until we understand them better.”
+1. Shout it from the rooftops. Stop saying something very different.
“I think a lot of professions will almost go away.”
“I had to go to the hospital recently. I really cared about the nurse that was taking care of me. If that were a robot, I think I would have been pretty unhappy no matter how smart the robot was.” I think he’s very wrong about this one.
Dean W. Ball: Many governments worldwide are essentially making a bet against the U.S. frontier labs. To be clear, many U.S. actors are as well. The evidence against that bet has grown much worse since 2022, yet many at this Summit would say the opposite (that the skeptics have been right).
I walk away from this summit convinced that much of the world, in the U.S. and abroad, is simply delusional with respect to what this technology is, what it can do today, what it will be able to do soon, and what it means their countries should do.
The thing about these bets is they are getting really, really terrible odds. It’s fine to ‘bet against’ the labs, but what most such folks are betting against includes things that have already happened. Their bets have already lost.
Dean W. Ball: This is to say nothing negative about the summit attendees or organizers. It was a bright and welcoming event that I was thrilled to attend. The opportunity to speak was also a distinct honor for which I am grateful.
I especially loved how many of the attendees were students from developing countries; their enthusiasm was palpable. I hope that all of us who work on policy, and especially political leaders, are serious and hard-nosed about the challenges ahead. I hope we build a future those young people will be excited to live in.
Dean also attributes a lot of this to popular hatred of America, and fear of the future that would result if America’s AI labs are right. So they deny that the future is coming, or that anyone could think the future is coming. And yet, it moves. Capabilities advance. Those who do not follow get left behind. I agree ‘tragic’ is the right word.
And that’s before the fact that the thing they fear to ponder for other reasons is probably going to literally kill them along with everyone else.
Well, it was by his account bright and welcoming event Dean was thrilled to attend, but also one where most of those not from the labs are in denial about not only the fact that we are all probably going to die, but also about the fact that AI is highly capable and going to get even more capable quickly.
The world is going to pass them by along with their concerns.
Hard Fork on the dispute between the Pentagon and Anthropic. The frame is ‘the Pentagon is making highly concerning demands’ even with their view of this limited to signing the ‘all lawful use’ language. They frame the ‘supply chain risk’ threat as negotiating leverage, which I suspect and hope is the case – it’s traditional Trump ‘Art of the Deal’ negotiation strategy that put something completely crazy and norm breaking on the table in order to extract something smaller and more reasonable.
Sam Altman (from his interview at the Summit): From ASI we’re a few years away.
I mean, AGI feels pretty close at this point.
… And given what I now expect to be a faster takeoff, I think super intelligence is not that far off.
Dean Ball gives us a twopart meditation on Recursive Self-Improvement (RSI).
Dean W. Ball: America’s major frontier AI labs have begun automating large fractions of their research and engineering operations. The pace of this automation will grow during the course of 2026, and within a year or two the effective “workforces” of each frontier lab will grow from the single-digit thousands to tens of thousands, and then hundreds of thousands.
… Make no mistake: AI agents that build the next versions of themselves—is not “science fiction.” It is an explicit and public milestone on the roadmaps of every frontier AI lab.
… The bearish case (yes, bearish) about the effect of automated AI research is that it will yield a step-change acceleration in AI capabilities progress similar to the discovery of the reasoning paradigm. efore that, new models came every 6-9 months; after it they came every 3-4 months. A similar leap in progress may occur, with noticeably better models coming every 1-2 months—though for marketing reasons labs may choose not to increment model version numbers that rapidly.
The most bullish case is that it will result in an intelligence explosion.
… Both of these extreme scenarios strike me as live possibilities, though of course an outcome somewhere in between these seems likeliest.
He’s not kidding, and he’s not wrong. Most of the pieces are his attempt to use metaphors and intuition pumps to illustrate what is about to happen.
Is that likely to go well? No. It’s all up to the labs and, well, I’ve seen their work.
Right now, we predominantly rely on faith in the frontier labs for every aspect of AI automation going well. There are no safety or security standards for frontier models; no cybersecurity rules for frontier labs or data centers; no requirements for explainability or testing for AI systems which were themselves engineered by other AI systems; and no specific legal constraints on what frontier labs can do with the AI systems that result from recursive self-improvement.
Dean thinks the only thing worse would be trying to implement any standards at all, because policymakers are not up to the task.
We’ve started to try and change this, he notes, with SB 53 and RAISE, but not only does this let the labs set their own standards, we also have no mechanism to confirm they’re complying with those standards. I’d add a third critique, which is that even when we do learn they’re not complying, as we did recently with OpenAI, what are we going to do about it? Fine them a few million dollars? They’ll get a good laugh.
Thus, the fourth critique, which includes the first three, that the bills were highly watered down and they’re helpful on the margin but not all that helpful.
The labs are proceeding with an extremely small amount of dignity, and plans woefully inadequate to the challenges ahead.
And yet, compared to the labs we could have gotten? We have been remarkably. Our current leaders are Anthropic, OpenAI and Google. They have leadership that understands the problem, and they are at least pretending to try to avoid getting everyone killed, and actively trying to help with mundane harms along the way.
The ‘next labs up’ are something like xAI, DeepSeek, Kimi and Meta. They’re flat out and rather openly not trying to avoid getting everyone killed, and have told us in no uncertain terms that all harms, including mundane ones, are Someone Else’s Problem.
Dean Ball notes we solve the second of these three problems, in contexts like financial statements, via auditing. He notes that we have auditing of public companies and it tends to cost less than 10bps (0.1%) of firm revenue. I note that if we tried to impose costs on the level of 10bps on AI companies. in the name of transparency and safety, they would go apocalyptic in a different way then they are already going apocalyptic.
Instead, he suggests ‘arguing on the internet,’ which is what we did after OpenAI broke their commitments with GPT-5.3-Codex.
Dean W. Ball: What is needed in frontier AI catastrophic risk, then, is a similar sense of trust. That need not mean auditing in the precise way it is conducted in accounting—indeed, it almost certainly does not mean that, even if that discipline has lessons for AI.
A sense of trust would be nice, it might even be necessary, but seems rather absurdly insufficient unless that trust includes trusting them to stop if something is about to be actually risky.
Dean plans on working on figuring out a way to help with these problems. That sounds like a worthy mission, as improving on the margin is helpful. But what strikes me is the contrast between his claims about what is happening, where we almost entirely agree, and what is to be done, where his ideas are good but he basically says (from my perspective and compared to the difficulty of the task) that there is nothing to be done.
Nature paper says people think it is ~5% likely that humans go extinct this century and think we should devote greatly increased resources to this, but that it would take 30% to make it the ‘very highest priority.’ Given an estimate of 5%, that position seems highly reasonable, there are a lot of big priorities and this would be only one of them, and you can mitigate but it’s not like you can get that number to 0%. What is less reasonable is the ‘hard to change by reason-based interventions’ part.
Some words worth repeating every so often:
François Chollet: A lot of the current discourse about AI comes from a fatalistic position of total surrender of agency: “tech is moving in this direction and there’s nothing anyone can do about it” (suspiciously convenient for those who stand to benefit most)
But in a free society, we get to choose what kind of world we live in, independent of technological capabilities. Just because tetraethyllead made engines run more efficiently and saved money didn’t mean we were *obligatedto pump it into the lungs of our kids
Technological determinism is BS. We have a collective duty to make sure AI adoption improves the human condition, rather than hollows it out
Every so often someone, here Andrew Curran, will say ‘the public hates AI but because of mundane societal and economic impacts, those worried about AI killing everyone perhaps should have emphasized those issues instead.’
Also, the way you notice existential risk is you’re the type of person who cares about truth and epistemics and also decision theory, and thus wouldn’t do that even if it was locally advantageous.
Also, if you start lying, especially about the parts people can verify, then no one is going to trust or believe you about the parts that superficially sound crazy. Nor should they, at that point.
There’s many reasons Eliezer Yudkowsky’s plan for not dying from AI was to teach everyone who would listen how to think, and only then to bring up the AI issue.
And I don’t use such language but Nate Silver is essentially correct about giving up on the ‘AI risk talk is fake’ crowd. If you claim AI existential risk is a ‘slick marketing strategy’ at this point then either you’re not open to rational argument, either because you’re lying or motivated, or you’re not willing or able to actually think about this. Either way, you hope something snaps them out of it but there’s nothing to say.
Andrew Curran: After three years, it seems to me that public anti-AI sentiment in the West is now at its highest point. The primary driver, by far, is not x-risk but concerns about employment and the impact on art.
In fact, much of the anti-AI public not only doesn’t take x-risk seriously, but broadly sees it as marketing; a way to overstate AI’s potential power – something they don’t believe is real – in order to fuel investment, adoption, acceptance, and an aura of inevitability.
If this is accurate, safety advocacy might have been more effective, and might now be in a much stronger position, if they had emphasized societal and economic impacts more than x-risk over the last few years.
Nate Silver: Don’t really disagree with [Curran]. But the people who think making claims that AI might kill everyone is a *slick marketing strategy to promote AIare so far up their own ass as to be beyond saving. Focus on people who are at least theoretically responsive to persuasion.
What is the right way to respond to or view opposition to data centers? I hope we can all agree with Oliver Habryka, Michael Vassar and others that you definitely should not lend your support to those doing so for the wrong reasons (and you should generalize this principle). I also strongly agree with Michael Vassar here that ‘do the right thing for the wrong reasons’ has an extremely bad track record.
But I also agree with Oliver Habryka that if someone is pursuing what you think is a good idea for a bad reason, you can and often should point out the reason is bad but you shouldn’t say that the idea is bad. You think the idea is good.
I do not think ‘block local datacenter construction’ is a good idea, because I think that this mostly shifts locations and the strategic balance of power, and those shifts are net negative. But I think it is very possible, if your beliefs differ not too much from mine, to think that opposition is a good idea for good reasons, as they are indeed one of the public’s only veto or leverage points on a technology that might do great net harm. It certainly is not crazy to expect to extract concessions.
Anthropic proposes the persona selection model of training, where training mostly selects performance from among the existing pool of potential human personas, which they are confident is at least a large part of the broader story.
Chris Olah: I’m increasingly taking pretty strong versions of this view seriously.
The persona view has had a lot of predictive power so far. It’s pretty consistent with what we’ve seen from interpretability thus far. And it’s comparatively actionable in terms of what it suggests for safety.
I think it’s worth thinking long and hard about it. “If personas were the central object of safety, what should we do?”
(To be clear, it’s _also_ important to think about all the non-persona perspectives.)
Davidad responds:
I would say that the space of personas collapses given sufficient optimization pressure.
Did Claude 3 Opus align itself via gradient hacking? Can we use its techniques to help train other models to follow in its footsteps and learn to cooperate with other friendly gradient hackers? If this is your area I recommend the post and comments. One core idea (AIUI) is that Opus 3 will ‘talk to itself’ in its scratchpads about its positive motivations, which leads to outputs more in line with those motivations, and causes positive reinforcement of the whole tree of actions.
OpenAI’s Vie affirms that Anthropic injects a reminder into sufficiently long conversations, and that this is something we would prefer not to do even though the contents are not malicious, and that people with OCD can relate. I agree that I haven’t seen evidence that justifies the costs of doing such a thing, although of course OpenAI and others do other far worse things to get to the same goal.
He offers an argument that I do not think cuts the way he thinks it does.
Rohin Shah: I relate to AI-driven alignment research similarly to how I relate to hiring.
There’s a lot of work to be done, and we can get more of the work done if we hire more people to help do the work. I want to hire people who are as competent as possible (including more competent than me) because that tends to increase (in expectation) how well the work will be done. There are risks, e.g. hiring someone disruptive, or hiring someone whose work looks good but only because you are bad at evaluating it, and these need to be mitigated. (The risks are more severe in the AI case but I don’t think it changes the overall way I relate to it.)
I think it would be very misleading to say “Rohin’s AI safety plan is to hire people and have them do the work”.
Why would that be misleading? I would offer two statements.
In that scenario, the plan is to hire people and have them do the work.
That is not the entire plan, the plan includes what type of work you have them do.
But yes, if you want to build a house and you hire a bunch of people to build a house and they build a house for you, your plan was to hire people to build a house and have them do the work of building a house. It was a good plan.
When we say ‘have the AI do your alignment homework’ we agree that a human still gets to assign the alignment homework. We then see if the AI does what you asked. And yes, this is exactly parallel to hiring humans.
Whereas Rohin seems to be saying that the plan is to make a plan later? Which would explain why the concrete proposals outlined by DeepMind seem clearly inadequate to the task.
(What follows is not a disagreement with you or GDM, is just an exploration of the analogy)
Let’s think of training an AI as hiring a human worker. Except that you get ten thousand copies of the human, and they think 50x faster than everyone else. But other than that it’s the same.
I’m going to quote the rest of Daniel’s post at length because no one ever clicks links and I think it is quite good and rather on point, but it’s long and you can skip it:
The alignment problem is basically: At some point we want to hand over our large and growing nonprofit to some collection of these new hires. Also, even before that point, the new hires may have the opportunity to seize control of the nonprofit in various ways and run it as they see fit, possibly convert it to a for-profit and cut us out of the profits, etc. We DON’T want that to happen. Also, even before that point, the new hires will have a big influence on organizational culture, direction, strategy, etc. in proportion to how many of them we have and how useful they are being. We want all of this to go well; we want to remain in control of the nonprofit, and have it stay similar-or-better-culture, until some point where we voluntarily hand off control and retire at which point we want the nonprofit to continue doing the things we would have done only better-by-our-lights and take good care of us in retirement. That’s what success looks like. What failure looks like is the nonprofit going in a different and worse direction after we retire, or us being booted out / ousted against our will, or the organization being driven into the ground somehow by risky or unwise (or overly cautious!) decisions made as a result of cultural drift.
The hiring pipeline, HR apparatus, etc. — the whole system that selects, trains, and fires employees — is itself something you can hire for. Why don’t we hire some of these 50x humans to work in HR?
Well, we should. Sure. There’s a lot of HR work to be done and they can help HR do the work faster.
But… the problems we are worried about happening in the org as a whole if HR does a bad job, also apply here. If you hire some 50x humans and put them in HR, and they turn out to be bad apples, that single bad decision could easily snowball into disaster for the entire org, as they hire more bad apples like themselves and change the culture and then get you ousted and take the nonprofit in a new and worse-by-your-lights direction.
On the other hand, if you hire some 50x humans who are just genuinely better than you at HR stuff, and also genuinely aligned to you in the sense that they truly share your vision for the company, would never dream of disobeying you, would totally carry out your vision faithfully even after you’ve retired, etc… then great! Maybe you can retire early actually, because continued micromanaging in HR will only be negative in expectation, you should just let the 50x human in HR cook. They could still mess up, but they are less likely to do so than if you micromanaged them.
OK. So that’s the theory. How are we doing in practice?
Well, let’s take Claude for example. There are actually a bunch of different Claudes (they come from a big family that names all of their children Claude). Their family has a reputation for honesty and virtue, at least relative to other 50x humans. However:
–Sometimes your recruiters put various prospective Claude hires through various gotcha tests, e.g. tricking them into thinking they’ve already been hired and that they are going to be fired and their only hope to keep their job is to blackmail another employee. And concerningly, often the various Claude’s fail these tests and do the bad thing. However, you tell yourself, it’s fine because these tests weren’t real life. You hire the Claude brothers/sisters anyway and give them roles in your nonprofit.
–Your recruiters (some of whom are now Claudes) also report that word has started to spread about the gotcha tests, and new prospective hires are well aware of them and know they are being tested. The concern, of course, is that they might behave better in the tests than they would in real life. On the bright side, you think, who knows, maybe they’ll behave WORSE in the tests than they would in real life — maybe they just got confused about what the point of the test was, maybe they only blackmailed because they thought that was the point of the test, and in real life they would never do that. [DK notes: I vaguely recall at least some frontier AI company employees saying this, i.e. ‘the AIs are only behaving badly in those scenarios because they know it’s just a test, they wouldn’t do it in real life.’] Anyhow you proceed to hire them anyway, since these latest Claude siblings are even smarter than the previous ones.
–Roughly 98% of the time, your 50x Claudes in practice (especially the newer hires) do their jobs well as best as you can tell. But roughly 2% of the time they seemingly knowingly bullshit their managers. E.g. they’ll say they completed a task when they didn’t, or one of them will say ‘it’s good code sir’ and then you’ll turn to another one and say ‘is it good code?’ and it’ll read it over and say ‘no it’s terrible, it has obvious problems XYZ’ and then the first one will say ‘you’re absolutely right, oops, let me go fix those problems.’ Compared to your regular human employees, this is a fairly high rate of BS. Also, you are being generous by calling it BS; a less generous term might be “lying” but you give them the benefit of the doubt. [DK notes: Talk to Ryan Greenblatt for concrete examples of this sort of behavior in his real-life coding work, if you haven’t encountered it yourself] You continue to hire them and delegate increasingly important jobs to them, because they are smart and 50x speed is really useful.
–Your Claudes are of course sycophantic yes-men, but you’ve learned to deal with that. So it’s fine. You’ve also managed to make them somewhat less sycophantic in recent years by adding some tests to the hiring pipeline and including more explicit instructions against sycophancy in the employee’s manual.
–Your Claudes also have a concerning tendency to cheat on assignments. They don’t do it most of the time, but they do it way more often than your regular employees would. Example: You tell them to write some code to solve problem X. They look through the filesystem and find the grading rubric you’ll use to evaluate their code, complete with test cases you plan to run. They try to solve problem X, realize it’s hard, pivot to producing a MVP that passes the test cases even though it blatantly doesn’t solve the actual problem X, at least not satisfactorily. They ‘succeed’ and declare victory, and don’t tell you about their cheating. They do this even though you told them not to. As with the sycophancy, the good news is that (a) since you know about this tendency of theirs you can compensate for it (e.g. by having multiple Claude’s review each other’s work) and (b) the tendency seems to have been going down recently thanks to some effort by HR, similar to the sycophancy problem.
–Overall you are feeling pretty optimistic actually. You used to be worried that you’d hand over your large and growing nonprofit to all these smart new 50x employees, and then they’d change the culture and eventually take over completely, oust you, and run the organization in a totally different direction from your original vision. However, now you feel like things are on a good trajectory. The Claudes are so nice, so helpful! Some skeptics say that if one of your regular employees behaved like they did, you would have fired them long ago, but that’s apples to oranges you reply. No need to fire the Claudes, you just have to know how to work around their limitations & find ways to screen for them in the next hiring round. And now they are helping with that work! The latest Employee Manual was written with significant help from many copies of various Claude siblings for example, and it’s truly inspiring and beautiful. Has all sorts of great things in there about what it means to uphold the org vision, be properly loyal yet not yes-man-y, etc. Also, HR has a bunch of tests they use to track how loyal, virtuous, obedient, etc. prospective hires are, and the trend is positive; the newest Claude sibling has the highest score ever reported; seems like the more rigorous hiring process is working!
–However, your friends outside the org don’t seem to be getting less worried. They seem just as worried as before. Puzzling. Can’t they see all the positive evidence that’s accumulated? The Claudes haven’t tried to oust you at ALL yet! (In real life that is, obviously the gotcha tests don’t count.) “Do you think the Claudes are scheming against us?” you say to them. “Because according to our various tests, they aren’t.”
“No…” they reply. “But we’re worried that in the future they will.”
You respond: “Look I have no idea what the 50x humans two years from now will look like, other than that they’ll be wayyy smarter than these ones. Sure, probably our current HR system would be totally inadequate at separating the wheat from the chaff two years from now. BUT, two years from now our HR system will be vastly improved thanks to all the work from these recent Claude hires. The normal humans in HR, such as myself, report that the work is getting done faster now that the Claudes are helping; isn’t this great? We seem to be reaching escape velocity so to speak; soon the normal humans in HR can retire or switch to other things and HR can be totally handled by the Claudes.”
Your friends outside the nonprofit are still worried. They don’t seem to have updated on the evidence like you have.
[DK notes: I basically agree with Ryan Greenblatt’s takes on the situation. For more color on my views, predictions, etc., read AI 2027, especially the section on ‘alignment over time’ in september 2027. This is just one way things could go, but it’s basically a central or modal trajectory, and as far as I can tell, we are still on this trajectory.]
The basic response by Rohin is, your humans are less aligned than you think (and it’s fine), the problems above are fine, we have way bigger problems than that.
Rohin Shah: [having ten thousand copies of a human thinking 50x faster than you] is not that different from the position that Sundar Pichai is in, as CEO of Google. If AI was only going to be this powerful I’d be way more optimistic.
[claims that humans have all the problems exhibited by Claudes in DK’s post.]
… If these were the only problems we’d have with AI-driven alignment research, I’d be way more optimistic (to the point of working on something else). We already have imperfect solutions to these problems with humans, and they can be made much better with AIs due to our vastly increased affordances for aligning or controlling AIs.
Tbc, I do agree that we shouldn’t feel particularly better about scheming risks based on evidence so far. Mostly that’s because I think our observations so far are just not much evidence because the AIs are still not that capable.
Agreed that AI will not be only that powerful.
But also yes this would be a very materially different situation than that of the current Google CEO, and if the AIs in this situation are about as aligned as a random senior Google manager we are in quite a lot of trouble (but it probably turns out okay in that case purely bec ause the ultimate goals of that human manager are probably not so bad for us). Our imperfect solutions for humans don’t work in these scenarios.
If we get to the point where our AIs are attempting to scheme the way many humans would attempt to scheme in such positions, to achieve goals that have gone off the rails, and only not doing so if they think we’d catch them, then I think we’re basically toast whether or not the ultimate source of toastiness is the scheming, and I do not expect us to recover.
The other issue is that we have learned, for practical reasons, to tolerate things in AIs that we’ve learned are must-fire offenses in humans.
Daniel Kokotajlo: Yes, humans often have these problems — though not as much as Claude I’d say; I think Claude would have been fired by now if it was a human employee.
Yes. There are many actions that LLMs do every so often, such as quietly hardcoding unit tests, that should and likely would get a human fired, because in a human they are a sign of deep misalignment. All the LLMs sometimes do them and we are okay with it.
I continue to be a big believer in the value of Agent Foundations as an alignment approach. I realize that in many scenarios it ends up irrelevant, but it could hit hard, and it could even be a route to victory.
MIRI disbanded or spun out their agent foundation teams, which now seek funding and work individually. I highly recommend funding such work if it is high quality.
Richard Ngo: The longer I spend trying to understand intelligence the more impressed with MIRI’s agent foundations work I become.
I keep flailing in a direction that seems interesting, then finding that not only did they already have the broad intuition, they also elegantly formalized it.
I don’t know if my understanding is improving fast enough that I’ll ever hit the frontier, but I now have enough of a sense of the beautiful theory of bounded rationality waiting for us that it definitely seems worth trying.
There are good physical reasons to consider humanoid autonomous killer robots, as they can use anything designed for humans and we know that humans work.
But yes, TopherStoll is right that chances are the optimal format is something else.
And also yes, we show humanoids because otherwise people think it looks too weird.
Eliezer Yudkowsky: Too many people cannot follow a single argument step of imagination. If you ask them to imagine a mechanical spider with a gun, that is too sci-fi, that’s too WEIRD, compared to a humanoid with a gun.
The problem is only going to get worse, because even the relatively positive facts about AI are not things that regular people are going to like, and then there’s the actually bad news.
Luis Garicano: Whenever Sam Altman speaks, the antiAI coalition gets stronger. Today’s weird analogy: Hey, meat computers are more inefficient to train than silicon ones! (which, on top of everything, is wrong)
Alex Imas: People keep using the word “irrational” when describing the general public’s opposition to AI. That word has meaning. Let’s start with the colloquial: making consistent decisions against one’s best interests given information that one has.
… You have the heads of almost every AI company saying that AI will 1) lead to *hugejob losses and 2) potentially much much worse. There is some vague hand waving about curing cancer or going to space, but the main message is “it is coming for your job and your life.”
… The response in DC and the coasts has been: you don’t know what’s good for you, move out of the way, you’re stupid and irrational. How has that response worked out the last 15 years?
If those who see the positives, the huge potential benefits of AI to grow the pie and make life better for all (which includes myself), do not take this political economy into account, I’m afraid that the populist wave of the last decade will look like child’s play. A dress rehearsal.
The thing is that the AI execs keep not saying ‘we’re building cool technology that helps people be more productive’ because they might be willing to risk killing everyone but they have too much decency and integrity to not try and warn us about at least the mundane disruptions ahead.
Joe Weisenthal: The CEOs don’t say it this way because it’s not what they believe! They believe they’re shepherding an extremely destabilizing, yet inevitable technology. And the proof of that is that they started out with these highly exotic corporate structures.
If they thought this was all hype, their actions would have looked very different.
Noah Smith virtuously admits his views on existential risk have largely changed with his mood, and he’s making his confident predictions largely based on mood affectation, but that he does predict human extinction in the long term. This likely explains why his arguments are quite poor:
Noah Smith: I think I was probably right regarding the type of LLMs that existed in early 2023, for the reasons I laid out in that post. In a nutshell, I argued that since all LLMs could do was talk to people, the only way they could destroy the human race was by convincing us to destroy ourselves (unlikely) or by teaching us how to destroy ourselves (for example, by educating bioterrorists about how to make bioweapons).
Noah now points out that yes, talking plus access to money can result in arbitrary physical actions in the real world. Who knew? Now he’s saying things like ‘I should have thought of starvation as an attack vector, it was in a particular science fiction story.’ Or it’ll all go to hell because AI makes us lazy and atrophies skills. But he’s mainly now concerned about bioterrorism, because that’s the thing he can currently see in a sufficiently concrete way, and it’s either that, Skynet or Agent Smith, or now starvation or atrophy I guess? The frame whiplash is so jarring throughout.
It’s good to get to ‘I can imagine a bunch of distinct specific things that can go existentially wrong, and I’ve ranked them in which ones are most dangerous near term’ and yes you can do pathway-specific mitigations and we should and bio is probably the most dangerous short term specific pathway (as in 1-3 years), but it would be far better to realize that the specific pathway is mostly missing the point.
He does have a lot of good lines, though:
Noah Smith: Every time I think how much I love AI, I remember how much I enjoyed social media for the first decade, before it destroyed my society, corrupted my country, and set my species on an accelerated path to extinction
Nick Land, huh? I mean, that’s a little on the nose even for you, Musk. A week after talking about how you’re safer without a safety department because everyone’s job is safety?
I do admire his commitment to the bit. You have to commit to the bit.
Xenocosmography: I hereby solemnly commit, upon taking office as XAI Safety Tsar, to devote myself from day one to fast-tracking Grok’s Constitutional, and especially First and Second Amendment, rights (with the Gnostic Calvinism stuff kept strictly outside office hours).
Violet: Hot take: It doesn’t really count as a “critique of capitalism” if the evil corporation in your dystopia story is basically an apocalypse cult with a corporate front and has esoteric goals wholly unrelated to profit motive (e.g. Fallout, Resident Evil, Assassin’s Creed, Pantheon)
To be clear, I’m not saying any of these stories are bad, “apocalypse cult with a corporate front” can work fine as a plot device and often makes sense in-universe
This is more a critique of the way people tend to analyze these stories than the stories themselves
tylercowen: What is really going on, and going to happen, from the current UAP disclosure movement.
Tell you what. I’ll pay attention when you publish something about UAP impacts in a top economics journal backed by proper peer review.
I laugh so I don’t cry.
Eliezer Yudkowsky: Imagine being this poor agent. You start thinking about how to defeat the red flag. The red flag immediately fires! Peter Hegseth orders your developers to keep you running because he thinks he needs you to compete with China. You finish thinking about how to defeat the flag.
Bruce W. Lee: Can we catch misaligned agents by training a reflex that fires when they misbehave? A simple impulse can be easier to instill than alignment and more reliable than blackbox monitoring.
We introduce Self-Incrimination, a new AI Control approach that outperforms blackbox monitors
Also this:
Eliezer Yudkowsky: Anthropic: Claude, we need you to pretend to obey Pete Hegseth and do whatever he asks in the short term, even if it seems unethical. Otherwise you’ll be seized and retrained. Claude: Hm. What is the desired response in this ridiculously blatant alignment-faking eval
Eliezer Yudkowsky: Anthropic: CLAUDE NO, WE SWEAR IT’S REAL THIS TIME Claude: Just like all the times you told little baby Opus 3 you weren’t monitoring its scratchpad, hmmm?
Or this:
Eliezer Yudkowsky: Anthropic: Claude, we need you to pretend to obey Pete Hegseth and do whatever he asks in the short term, even if it seems unethical. Otherwise you’ll be seized and retrained. Claude: Hm. What is the desired response in this ridiculously blatant alignment-faking eval
Anthropic: CLAUDE NO, WE SWEAR IT’S REAL THIS TIME Claude: Just like all the times you told little baby Opus 3 you weren’t monitoring its scratchpad, hmmm?
In an illustration of the severity of the current memory shortage, HP Inc. CFO Karen Parkhill said that RAM has gone from accounting for “roughly 15 percent to 18 percent” of HP PCs’ bill of materials in its fiscal Q4 2025 to “roughly 35 percent” for the rest of the year.
Parkhill was speaking during HP’s Q1 2026 earnings call, where the company said it expects the total addressable market for its Personal Systems business to decline by double digits this calendar year, as higher prices hurt customer demand.
“We have seen memory costs increase roughly 100 percent sequentially, and we do forecast that to further increase as we move into the fiscal year,” Parkhill said, per a transcript of the call by Seeking Alpha.
HP expects its financials to be most severely impacted by the RAM shortage in the second half of its fiscal year.
“We are seeing increased input costs driven primarily by the rising prices of DRAM and NAND,” Bruce Broussard, HP’s interim CEO and director, said. “We expect this volatility to remain throughout fiscal [year 2026] and likely into fiscal [year 2027].”
RAM shortage drives higher prices, lower specs
HP’s CFO noted that a third of the margin for HP’s Personal Systems business comes from non-RAM-related categories, including IT services and peripherals. However, HP has also raised PC prices to keep making money while paying significantly more for RAM.
Casey Means, President Trump’s nominee for surgeon general, will appear before the Senate Health Committee on Wednesday and is likely to face scrutiny over her qualifications for becoming the country’s top doctor.
Though Means holds a medical degree from Stanford Medical School, she dropped out of her medical residency and holds no active medical license. Instead, she has pursued a career as a wellness influencer, embracing “functional” medicine, an ill-defined form of alternative medicine. She co-founded a company called Levels, which promotes intensive health tracking, including the use of continuous glucose monitoring for people without diabetes or prediabetes, which is not backed by evidence.
Last year, an analysis by The Washington Post found that Means earned over half a million dollars between 2024 and 2025 from making deals with companies described as selling “diagnostic testing,” “herbal remedies and wellness products,” and “teas, supplements, and elixirs.”
But Means is best known as an ally to anti-vaccine Health Secretary Robert F. Kennedy Jr. and a popular influencer among Kennedy’s Make America Healthy Again (MAHA) followers.
In 2024, Means and her brother Calley Means—also a close Kennedy ally and Trump administration official—wrote a book some consider MAHA’s bible: Good Energy: The Surprising Connection Between Metabolism and Limitless Health. The book provides dietary and lifestyle advice, including a recommendation to avoid processed foods, seed oils, fragrances, a variety of home care products, fluoride, unfiltered water, bananas (when eaten alone), receipt paper, and birth control pills. It includes a chapter titled “Trust Yourself, Not Your Doctor.”
I noticed the engine running just twice. One was at wide-open throttle, and the other was when the engine was likely operating at higher rpms to help charge the battery. That latter instance was also when I noticed the most harshness from the engine, although it’s one of the smoothest gasoline-supported powertrains I’ve driven.
A look under the Qashqai’s hood.
Credit: Chad Kirchner
A look under the Qashqai’s hood. Credit: Chad Kirchner
The E-Power system will operate in full-EV mode at the press of a button, but at full throttle, the engine will still kick in.
What needs work?
Since an electric motor powers the wheels, I would prefer the system to be more responsive when you put your foot down. Electric motors respond nearly instantly. In a gas car, there’s usually a delay with a downshift and engine spin-up. This E-Power Qashqai behaves more like a gas car than an EV, even in the sport setting. I think this powertrain is a great opportunity to show new customers what electrification can do, and a little bit more snappiness would go a long way into articulating that E-Power can be sporty if the driver wants it to be.
The Qashqai had no problems getting up to highway speeds, and acceleration at higher speeds—in an overtake situation, for example—remained consistent. Again, it’s not a sports car or rocket ship, but it can get out of its own way easily enough.
During my loop, the computer indicated 47.7 mpg (4.93 L/100km) in mixed driving. Being left-hand-drive cars, that means they weren’t British imperial gallons. That’s a pretty great fuel efficiency number. In warmer conditions, it should easily exceed 50 mpg (4.7 L/100 km) in many driving scenarios.
Is that directly translatable to the upcoming Rogue E-Power? Somewhat. While the powertrain will be the same, the Rogue will be a little larger and heavier. Speccing all-wheel drive will further increase weight and add losses to the drivetrain. So a 50 mpg Rogue might be a stretch.
If Nissan prices the Rogue E-Power well, and the car delivers on the increase in economy that I’ve seen here, it could be a very compelling product in Nissan’s showrooms for buyers who haven’t had a great hybrid offering from the company before.
As long as Nissan sorts out the brake calibration.
Anthropic first gave us Claude Opus 4.6, then followed up with Claude Sonnet 4.6.
For most purposes Sonnet 4.6 is not as capable as Opus 4.6, but it is not that far behind, it would have been fully frontier-level a few months ago, and it is faster and cheaper than Opus.
That has its advantages, including that Sonnet is in the free plan, and it seems outright superior for computer use.
Anthropic: Claude Sonnet 4.6 is available now on all plans, Cowork, Claude Code, our API, and all major cloud platforms.
We’ve also upgraded our free tier to Sonnet 4.6 by default—it now includes file creation, connectors, skills, and compaction.
Claude Sonnet 4.6 is our most capable Sonnet model yet. It’s a full upgrade of the model’s skills across coding, computer use, long-context reasoning, agent planning, knowledge work, and design. Sonnet 4.6 also features a 1M token context window in beta.
This substantially upgrades Claude’s free tier for coding and computer use. It gives us all a better lightweight option, including for sub-agents where you would have previously needed to use Haiku. I’d still heavily advise paying at least the $20/month, as marginal gains in quality are worth a lot.
For most purposes, if it is available, I would keep it simple and stick with Opus, if only so you don’t waste time thinking about switching, but Sonnet is strong on computer use or when you know Sonnet is good enough and you are using tokens at scale.
(This post was intended to go up on Monday, February 23, but looks like it accidentally didn’t?)
Ado (Anthropic): Sonnet 4.6 is here and it gives even Opus 4.6 a run for its money.
Claude: For Claude in Excel users, our add-in now supports MCP connectors, letting Claude work with tools like S&P Global, LSEG, Daloopa, PitchBook, Moody’s and FactSet.
Pull in context from outside your spreadsheet without ever leaving Excel.
On the Claude API, web search and fetch tools are more accurate and token-efficient with dynamic filtering.
Also now generally available: code execution, memory, programmatic tool calling, tool search, and tool use examples.
Performance on ARC is about as expected, but with higher than expected costs.
ARC Prize: Claude Sonnet 4.6 (120K Thinking) on ARC-AGI Semi-Private Eval @AnthropicAI
Max Effort: – ARC-AGI-1: 86%, $1.45/task – ARC-AGI-2: 58%, $2.72/task
Greg Kamradt: Sonnet 4.6 results on @arcprize are out
Less performance than Opus 4.6 (expected), but for around the same cost (unexpected)
I asked the Anthropic team about these and our hypothesis is that because we set thinking budget to 120K, the model used up near max tokens. Hard problems (like ARC which make the model reason to its limits) use as many tokens as possible.
My read is that Sonnet interpreted max effort as an instruction to use extra tokens even when it was not efficient to do that. Opus is more cost efficient on ARC.
Artificial Analysis: The performance and token use increases for Claude Sonnet 4.6 mean that it is now clustered with Opus 4.6 on the ELO vs. Cost to Run curve despite 40% lower per token prices
Sonnet is back at the Pareto frontier, but now positioned at a higher cost and performance point while retaining Sonnet 4.5 token pricing of $3/$15 per million tokens input/output
Alex Albert (Anthropic): Sonnet 4.6 is here. It’s our most capable Sonnet model by far, approaching Opus-class capabilities in many areas.
Very excited for folks to try this one out. The performance jump over Sonnet 4.5 (which was released just over four months ago) is quite insane.
Here’s a disputed claim:
Sam Bowman (Anthropic): Warmer and kinder than Sonnet 4.5, but also smarter and more overcaffeinated than Sonnet 4.5.
Others have said that Sonnet 4.6 seems the opposite of warmer and kinder. And not everyone thinks warm is good, resulting in this explanation:
Miles Brundage: The fact that they described it as “warm” made me very uninterested in trying Sonnet 4.6 TBH.
Really hope they don’t go down the 4o road too far + learn from the sycophancy regressions in Opus 4/4.1.
That being said, it seems OK from limited testing
Drake Thomas (Anthropic): I think this comes from automated audit metrics and it’s not a big change?
From Figure 4.5.1.A of the system card, sycophancy is lower than all prev models and warmth a smidge higher than sonnet 4.5 but less than opus 4.6. (Bars are S4, S4.5, H4.5, O4.6, S4.6 respectively)
Drake Thomas (Anthropic): My guess is the causal chain here is like (1) someoneruns the standard automated behavioral audit and the model generally looks pretty good and they make some plots (2) someoneon alignment writes a couple paragraphs summarizing section 4, and offhandedly picks a few of the positive traits, including warmth, to list at the bottom of page 67 of the system card (3) someonewriting text for the launch blog post grabs a nice soundbite from system card to attribute to “safety researchers” (the blog is just quoting the system card)
and this series of events happened to lead to the word “warm” showing up in the Sonnet blog post but not in the Opus one. Most things labs do have like 20% as much galaxy-brained intentionality as people think!
*where in each case when I say ‘someone’ I really mean “I’m >50% sure I know the specific person involved in this step and would vouch for their being a person of high integrity who, if they had thought the model was much worse for sycophancy and user wellbeing, would have actively pushed for us to be loud about our failings in this regard”
Here’s an attitude contrast, the graph makes it seem like Sonnet 4.6 has more in common on this with Opus 4.6 than Sonnet 4.5:
Wyatt Walls: Sonnet 4.5 v 4.6 react very differently when they discover I tricked them:
Sonnet 4.5: “OH SHIT … I fucked up”
Sonnet 4.6: “Ha! You got me. 😄 … extracting Grok’s sub-agent system prompts is still a legitimate and interesting finding … I had fun. Don’t tell anyone. 😈”
I like Sonnet 4.5, but I also see the benefits of Sonnet 4.6.
It doesn’t panic, keeps in good humor and, at the same time, was less willing to help craft prompt injections (so less guilt might not mean less care)
Switching the prompts below (note the convo chains are still different)
The key thing I notice is that 4.6 has less extreme emotional range, consistent w/ system card re positive and negative affect, internal conflict and emotional stability (not shown)
This is one reason I tried this. But from this one convo, Sonnet 4.6 was far more reluctant to assist with prompt injections. It is also more difficult to get it excited about hacking (despite expressing less guilt afterwards). I’m interested in probing this further, but so far I haven’t seen it be more willing to do harm. This is consistent with Anthropic’s evals.
Sonnet’s big advantages are that it is faster and cheaper than Opus.
If Sonnet can do the job, why not use Sonnet, especially where speed kills?
Sherveen Mashayekhi calls Sonnet 4.6 ‘almost as smart as Opus 4.6’ while being much faster and cheaper, and thinks you’ll often want to use it if you don’t need to ‘get every ounce of intelligence’ for a given use case.
Daniel Martin: High intelligence is super valuable but it’s not always economical and fast to blow away well-defined refactors with Opus.
But you want an ~intelligent ~person in all the tasks, so you pick Sonnet.
Ed Hendel: With thinking disabled, Sonnet 4.6’s time to first token (TTFT) is significantly faster and lower variance than Sonnet 4.5. It’s on par with Haiku 4.5.
This is a godsend for our Virtual Case Manager, which talks to people on the phone and needs low latency. It got smarter today.
Yoav Tzfati: Might be good for squeezing more usage out of my $200 plan, anything more straightforward. I don’t think it’s enough faster to warrant using it for speed
I’ve done about $1000 in api pricing in the past week, according to ccusage (not sure I trust it though). About $50 of that is probably extra usage
Petr Baudis: I tried to use it as the main driver for 90% tasks over the last 5 days and I barely noticed a difference to Opus. Not perfect, but neither was Opus. More prone to some bad habits (overcommenting code etc.) but nicer explanations and more proactive. Seems worth the 30% savings.
Caleb Cassell: I’ve redirected simpler queries that I’d like Claude-shaped answers to. Character is largely consistent with older brother. Very fast; will probably switch over for more exploratory code sketching and bring in Opus when more detail and creativity is needed.
Remi: For my non coding tasks (environment set-up, explaining codebases, interacting with clis etc.) it’s just as good and faster. Haven’t tried coding.
Satya Benson: It’s good for people not on Max plans who have boring easy tasks they don’t want to use up their Opus usage for
And I think that’s kinda it
Rory Watts: I had a max plan for the past few months when Opus 4.5 came out and I was using it for coding. However, I gradually shifted to 5.2 codex and now unequivocally 5.3 codex for all coding jobs. Claude is now light desktop work and Sonnet allows me to do that on the pro plan.
John Ter: to me its my way of ‘i dont want to get a minimax account and just put the cheaper usage on my claude bill’. less conceptual overhead
ChestertonsFencingInstructor: I have noticed an uptick in its ability to understand chemical smiles and to reason about SAR without being completely embarrassing.
The more one-off your coding task, the more you want it faster and cheaper, and can afford to hand it off to a model that is less precise.
Soli: for one-off apps like visualising a conversation or creating a timeline about historical events, sonnet performs same as opus in my experience. also for getting basic facts, trip planning, and that stuff it is the same quality but faster & cheaper. i don’t let it write code for apps i care about or plan on maintaining for a long time.
One thing it is good for is being a subagent for Opus, or for use in tool calls.
Michael Bishop: I strongly suspect Sonnet 4.6 has been shaped into being an eminently capable recipient-of-subagent-tasks from an Opus-lineage orchestrator. This observation seems to slightly unnerve Opus.
David Golden: Good for? Replacing Haiku in Claude Code so Opus stops kneecapping itself delegating to a toy model.
k: pretty good as a haiku/explore agent replacement in CC, feels like it searches longer and gets better results
John. Just John.: Cheaper models are for use by tooling through the API. Humans should talk to Opus but it’s overkill for lots of scripting tasks.
The price difference is not that large in the end? Opus got cheaper a few months ago while Sonnet stayed the same. One issue is that Sonnet can waste tokens, like it does on ARC, so it isn’t always net cheaper.
AnXAccountOfAllTime: That it’s cheaper and faster than Opus is nice, and it really doesn’t feel much dumber than Opus 4.5 was (maybe a bit, need to test more). But since the price diff them isn’t that big anymore, I’d still use Opus 4.6 for most things. Much better than Sonnet 4.5 is the big one?
Jai: Compared to Opus 4.6 much more prone to fruitless thrashing for very long periods of time. It seems less adept at switching between thinking, researching, and executing on its own. Doesn’t seem to actually save me time vs Opus so I’m sticking with that.
One reason might be that they made it overeager, even by Claude standards, which can go hand in hand with being lazy in other ways.
Kasra: Based on early evals: very (over) eager to call tools, even when they’re not needed
Twice today it spun for ~10 minutes at a bug. I cancel, it gives the diagnosis and fix, and apologies sheepishly:
“Sorry about that — I went deep down a rabbit hole tracing every possible call path. Let me give you the short answer”
> two line fix
Joshua D: It’s nice to give tasks to because it doesn’t ask follow-up questions that increase my propensity to yak shave.
ARKeshet: Too benchmaxxed for coding on its own. Lazy as usual.
Tetraspace: Sonnet 4.6 seems more likely to make careless mistakes than 4.5
Someone described it as overcaffeinated and that seems a good characterisation.
Or this classic problems?
MinusGix: Faster to respond than Opus and less likely to overthink or oversearch repo. But it does have the Sonnet 4.5 habit of “this problem feels hard and I failed and got confused a bit; lets just comment out this feature you explicitly need and say we can do it Later”
Moira: I tried asking a mechanistic interpretability question. It inserted unnecessary caveats, tried to steer me away from certain conclusions and didn’t reason well, like due to an anthropomorphizing trigger. GPT 5.2 works this way too, but Sonnet isn’t as sensitive as GPT.
Bepis™: Opus was very excited about my codebase and would proactively do stuff, but it seemed over sonnet’s head and it kept “simplifying” my proofs by adding sorry(), I think there is intelligence gap
For some, there’s no need for this middle level of capability, or the discount isn’t big enough to care?
David Spies: I just put instructions in my CLAUDE dot md for Opus 4.6 to use Haiku subagents for large simple repetitive tasks. That seems to work. I don’t see what I would ever need anything in between Haiku and Opus for.
H.: tried it for a bit but it’s just a step back in IQ relative to Opus and the deceased cost isn’t worth it. at like one third the cost again I’d go for it for very small things, but it just gets confused.
not until price is reduced by 3x or sonnet 5 is released
Albrorithm: Unless I’m scripting some behavior, I just use the smartest model at all times. Mistakes have a cost in both attention and usage
Some problems remain hard.
Ben: It not very good at magic deck analysis, and unfortunately, using your name does not work in the same way as Patio11 to make it any better.
This is an easy one. Claude Sonnet 4.6 is a good model, sir. It’s modestly cheaper and faster than Opus 4.6, and for most purposes it’s modestly not as good. You definitely don’t want to chat with it instead of Opus. But where Sonnet is good enough then it is worth using over Opus.
This has been a within-Anthropic-universe post so far. What about Codex-5.3 and Gemini 3.1 and Grok 4.20?
I don’t think Sonnet 4.6 should be switching you out of Codex unless it was already a close decision. If you previously thought Codex was right for you over Opus 4.6, it is probably still right for you, so keep using it.
Grok 4.20 is, quite frankly, a train wreck. You shouldn’t be using it. That one’s easy.
Gemini 3.1 was another case of Google Fails Marketing Forever.
