David Karsten suggests you read the book, while noting he is biased. He reminds us that, like any other book, most conversations you have about the book will be with people who did not read the book.
David Manheim: “An invaluable primer on one side of the critical and underappreciated current debate between those dedicated to building this new technology, and those who have long warned against it. If you haven’t already bought the book, you really should.”
Steven Byrnes reviews the book, says he agrees with ~90% of it, disagreeing with some of the reasoning steps but emphasizes that the conclusions are overdetermined.
Michael Nielsen gives the book five stars on Goodreads and recommends it, as it offers a large and important set of largely correct arguments, with the caveat that there he sees significant holes in the central argument. His actual top objection is that even if we do manage to get a controlled and compliant ASI, that is still extremely destabilizing at best and fatal at worst. I agree with him this is not a minor quibble, and I worry about that scenario a lot whereas the book’s authors seem to consider it a happy problem they’d love to have. If anything it makes the book’s objections to ‘building it’ even stronger. He notes he does not have a good solution.
Nomads Vagabonds (in effect) endorses the first 2 parts of the book, but strongly opposes the policy asks in part 3 as unreasonable asks, demanding that if you are alarmed you need to make compromises to ‘help you win,’ if you think the apocalypse is coming you only propose things inside the Overton Window, that ‘even a 1% reduction in risk is worth pursuing.’
That only makes sense if and only if solutions inside the Overton Window substantially improve your chances or outlook, and thus the opportunity outweighs the opportunity cost. The reason (as Nomads quotes) Ezra Klein says Democrats should compromise on various issues if facing what they see as a crisis, is this greatly raises the value of ‘win the election with a compromise candidate’ relative to losing. There’s tons of value to protect even if you give up ground on some issues.
Whereas the perspective of Yudkowsky and Soares is that measures ‘within the Overton Window’ matter very little in terms of prospective outcomes. So you’d much rather take a chance on convincing people to do what would actually work. It’s a math problem, including the possibility that busting the Overton Window helps the world achieve other actions you aren’t yourself advocating for.
The central metaphor Nomads uses is protecting against dragons, where the watchman insists upon only very strong measures and rejects cheaper ones entirely. Well, that depends on if the cheaper solutions actually protect you from dragons.
If you believe – as I do! – that lesser changes can make a bigger difference, then you should say so, and try to achieve those lesser changes while also noting you would support the larger ask. If you believe – as the authors do – that the lesser changes can’t make much difference, then you say that instead.
I would also note that yes, there are indeed many central historical examples of asking for things outside the Overton Window being the ultimately successful approach to creating massive social change. This ranges from the very good (such as abolishing slavery) to the very bad (such as imposing Communism or Fascism).
People disagree on a lot on whether the book is well or poorly written, in various ways, at various different points in the book. Kelsey Piper thinks the writing in the first half is weak, the second half is stronger, whereas Timothy Lee (a skeptic of the book’s central arguments) thought the book was surprisingly well-written and the first few chapters were its favorites.
Peter Wildeford goes over the book’s central claims, finding them overconfident and warning about the downsides and costs of shutting AI development down. He’s a lot ‘less doomy’ than the book on many fronts. I’d consider this his central takeaway:
Peter Wildeford: Personally, I’m very optimistic about AGI/ASI and the future we can create with it, but if you’re not at least a little ‘doomer’ about this, you’re not getting it. You need profound optimism to build a future, but also a healthy dose of paranoia to make sure we survive it. I’m worried we haven’t built enough of this paranoia yet, and while Yudkowsky’s and Soares’s book is very depressing, I find it to be a much-needed missing piece.
Buck is a strong fan of the first two sections and liked the book far more than he expected, calling them the best available explanation of the basic AI misalignment risk case for a general audience. He caveats that the book does not address counterarguments Buck thinks are critical, and that he would tell people to skip section 3. Buck’s caveats come down to expecting important changes between the world today and the world where ASI is developed, that potentially change whether everyone would probably die. I don’t see him say what he expects these differences to be? And the book does hold out the potential for us to change course, and indeed find such changes, it simply says this won’t happen soon or by default, which seems probably correct to me.
Nostream addresses the book’s arguments, agreeing that existential risk is present but making several arguments that the probability is much lower, they estimate 10%-20%, with the claim that if you buy any of the ‘things are different than the book says’ arguments, that would be sufficient to lower your risk estimate. This feels like a form of the conjunction fallacy fallacy, where there is a particular dangerous chain and thus breaking the chain at any point breaks a lot of the danger and returns us to things probably being fine.
The post focuses on the threat model of ‘an AI turns on humanity per se’ treating that as load bearing, which it isn’t, and treats the ‘alignment’ of current models as meaningful in ways I think are clearly wrong, and in general tries to draw too many conclusions from the nature of current LLMs. I consider all the arguments here, in the forms expressed, not new and well-answered by the book’s underlying arguments, although not always in a form as easy to pick up on as one might hope in the book text alone (book is non-technical and length-limited, and people understandably have cached thoughts and anchor on examples).
So overall I wished the post was better and made stronger arguments, including stronger forms of its arguments, but this is the right approach to take of laying out specific arguments and objections, including laying out up front that their own view includes unacceptably high levels of existential risk and also dystopia risk or catastrophic risk short of that. If I believed what Nostream believed I’d be in favor of not building the damn thing for a while, if there was a way to not build it for a while.
Gary Marcus offered his (gated) review, which he kindly pointed out I had missed in Wednesday’s roundup. He calls the book deeply flawed, but with much that is instructive and worth heeding.
Despite the review’s flaws and incorporation of several common misunderstandings, this was quite a good review, because Gary Marcus focuses on what is important, and his reaction to the book is similar to mine to his review. He notices that his disagreements with the book, while important and frustrating, should not be allowed to interfere with the central premise, which is the important thing to consider.
He starts off with this list of key points where he agrees with the authors:
Rogue AI is a possibility that we should not ignore. We don’t know for sure what future AI will do and we cannot rule out the possibility that it will go rogue.
We currently have no solution to the “alignment problem” of making sure that machines behave in human-compatible ways.
Figuring out solutions to the alignment problem is really, really important.
Figuring out a solution to the alignment problem is really, really hard.
Superintelligence might come relatively soon, and it could be dangerous.
Superintelligence could be more consequential than any other trend.
Governments should be more concerned.
The short-term benefits of AI (eg in terms of economics and productivity) may not be worth the long-term risks.
Noteworthy, none of this means that the title of the book — If Anyone Builds It, Everyone Dies — literally means everyone dies. Things are worrying, but not nearly as worrying as the authors would have you believe.
It is, however, important to understand that Yudkowsky and Soares are not kidding about the title
Gary Marcus does a good job here of laying out the part of the argument that should be uncontroversial. I think all seven points are clearly true as stated.
Gary Marcus: Specifically, the central argument of the book is as follows: Premise 1. Superintelligence will inevitably come — and when it does, it will inevitably be smarter than us. Premise 2. Any AI that is smarter than any human will inevitably seek to eliminate all humans. Premise 3. There is nothing that humans can do to stop this threat, aside from not building superintelligence in the first place.
The good news [is that the second and third] premisses are not nearly as firm as the authors would have it.
I’d importantly quibble here, my version of the book’s argument would be, most importantly on the second premise although it’s good to be precise everywhere:
Superintelligence is a real possibility that could happen soon.
If built with anything like current techniques, any sufficiently superintelligent AI will effectively maximize some goal and seek to rearrange the atoms to that end, in ways unlikely to result in the continued existence of humans.
Once such a superintelligent AI is built, if we messed up, it will be too late to stop this from happening. So for now we have to not build such an AI.
He mostly agrees with the first premise and has a conservative but reasonable estimation of how soon it might arrive.
For premise two, the quibble matters because Gary’s argument focuses on whether the AI will have malice, pointing out existing AIs have not shown malice. Whereas the book does not see this as requiring the AI to have malice, nor does it expect malice. It expects merely indifference, or simply a more important priority, the same way humans destroy many things we are indifferent to, or that we actively like but are in the way. For any given specified optimization target, given sufficient optimization power, the chance the best solution involves humans is very low. This has not been an issue with past AIs because they lacked sufficient optimization power, so such solutions were not viable.
This is a very common misinterpretation, both of the authors and the underlying problems. The classic form of the explanation is ‘The AI does not love you, the AI does not hate you, but you are composed of atoms it can use for something else.’
His objection to the third premise is that in a conflict, the ASI’s victory over humans seems possible but not inevitable. He brings a standard form of this objection, including quoting Moltke’s famous ‘no plan survives contact with the enemy’ and modeling this as the AI getting to make some sort of attack, which might not work, after which we can fight back. I’ve covered this style of objection many times, comprising a substantial percentage of words written in weekly updates, and am very confident it is wrong. The book also addresses it at length.
I will simply note that in context, even if on this point you disagree with me and the book, and agree with Gary Marcus, it does not change the conclusion, so long as you agree (as Marcus does) that such an ASI has a large chance of winning. That seems sufficient to say ‘well then we definitely should not build it.’ Similarly, while I disagree with his assumption we would unite and suddenly act ruthlessly once we knew the ASI was against us, I don’t see the need to argue that point.
Marcus does not care for the story in Part 2, or the way the authors use colloquial language to describe the AI in that story.
I’d highlight another common misunderstanding, which is important enough that the response bears repeating:
Gary Marcus: A separate error is statistical. Over and again, the authors describe scenarios that multiply out improbabilities: AIs decide to build biological weapons; they blackmail everyone in their way; everyone accepts that blackmail; the AIs do all this somehow undetected by the authorities; and so on. But when you string together a bunch of improbabilities, you wind up with really long odds.
The authors are very much not doing this. This is not ‘steps in the chain’ where the only danger is that the AI succeeds at every step, whereas if it ever fails we are safe. They bend over backwards, in many places, to describe how the AI uses forking paths, gaming out and attempting different actions, having backup options, planning for many moves not succeeding and so on.
The scenario also does not presuppose that no one realizes or suspects, in broad terms, what is happening. They are careful to say this is one way things might play out, and any given particular story you tell must necessarily involve a lot of distinct things happening.
If you think that at the first sign of trouble all the server farms get turned off? I do not believe you are paying enough attention to the world in 2025. Sorry, no, not even if there wasn’t an active AI effort to prevent this, or an AI predicting what actions would cause what reactions, and choosing the path that is most likely to work, which is one of the many benefits of superintelligence.
Several people have pointed out that the actual absurdity is that the AI in the story has to work hard and use the virus to justify it taking over. In the real world, we will probably put it in charge ourselves, without the AI even having to push us to do so. But in order to be more convincing, the story makes the AI’s life harder here and in many other places than it actually would be.
Gary Marcus closes with discussion of alternative solutions to building AIs that he wished the book explored more, including alternatives to LLMs. I’d say that this was beyond the scope of the book, and that ‘stop everyone from racing ahead with LLMs’ would be a likely necessary first step before we can pursue such avenues in earnest. Thus I won’t go into details here, other than to note that ‘axioms such as ‘avoid causing harm to humans’’ are known to not work, which was indeed consistently the entire point of Asimov’s robot novels and other stories, where he explores some but not all of the reasons why this is the case.
Again, I very much appreciated this review, which focused on what actually matters and clearly is stating what Marcus actually believes. More like this, please.
John Pressman agrees with most of the book’s individual statements and choices on how to present arguments, but disagrees with the thesis and many editorial and rhetorical choices. Ultimately his verdict is that the book is ‘okay.’ He states up top as ‘obvious’ many ‘huge if true’ statements about the world that I do not think are correct and definitely are not obvious. There’s also a lot of personal animus going on here, has been for years. I see two central objections to the book from Pressman:
If you use parables then You Are Not Serious People, as in ‘Bluntly: A real urgent threat that demands attention does not begin with Once Upon a Time.’
Which he agrees is a style issue.
There are trade-offs here. It is hard for someone like Pressman to appreciate the challenges in engaging with average people who know nothing about these issues or facing off the various stupid objections those people have (that John mostly agrees are deeply stupid).
He thinks the alignment problems involved are much easier than Yudkowsky and Soares believe, including that we have ‘solved the human values loading problem,’ although he agrees that we are still very much on track to fail bigly.
I think (both here and elsewhere where he goes into more detail) he both greatly overstates his case and also deliberately presents the case in a hostile and esoteric manner. That makes engagement unnecessarily difficult.
I also think there’s important real things here and in related clusters of arguments, and my point estimate of difficulty is lower than the book’s, largely for reasons that are related to what Pressman is attempting to say.
As Pressman points out, even if he’s fully right, it looks pretty grim anyway.
Aella (to be fair a biased source here) offers a kind of meta-review.
Aella: man writes book to warn the public about asteroid heading towards earth. his fellow scientists publish thinkpieces with stuff like ‘well his book wasn’t very good. I’m not arguing about the trajectory but he never addresses my objection that the asteroid is actually 20% slower’
I think I’d feel a lot better if more reviews started with “first off, I am also very worried about the asteroid and am glad someone is trying to get this to the public. I recognize I, a niche enthusiast, am not the target of this book, which is the general public and policy makers. Regardless of how well I think the book accomplishes its mission, it’s important we all band together and take this very seriously. That being said, here’s my thoughts/criticisms/etc.
There is even an advertising campaign whose slogan is ‘we wish we were exaggerating,’ and I verify they do wish this.
It is highly reasonable to describe the claims as overstated or false, indeed in some places I agree with you. But as Kelsey says, focus on: What is true?
Claims the authors are overconfident, including claims this is bad for credibility.
I agree that the authors are indeed overconfident. I hope I’ve made that clear.
However I think these are reasonable mistakes in context, that the epistemic standards here are much higher than those of most critics, and also that people should say what they believe, and that the book is careful not to rely upon this overconfidence in its arguments.
Fears about credibility or respectability, I believe, motivate many attacks on the book. I would urge everyone attacking the book for this reason (beyond noting the specific worry) to stop and take a long, hard look in the mirror.
Complaints that it sucks that ‘extreme’ or eye-catching claims will be more visible. Whereas claims one thinks are more true are less visible and discussed.
Yeah, sorry, world works the way it does.
The eye-catching claims open up the discussion, where you can then make clear you disagree with the full claim but endorse a related other claim.
As Raymond says, this is a good thing to do, if you think the book is wrong about something write and say why and about how you think about it.
He provides extensive responses to the arguments and complaints he has seen, using his own framings, in the extended thread.
Then he fleshes out his full thoughts in a longer LessWrong post, The Title Is Reasonable, that lays out these questions at length. It also contains some good comment discussion, including by Nate Soares.
I agree with him that it is a reasonable strategy to have those who make big asks outside the Overton Window because they believe those asks are necessary, and also those who make more modest asks because they feel those are valuable and achievable. I also agree with him that this is a classic often successful strategy.
Yes, people will try to tar all opposition with the most extreme view and extreme ask. You see it all the time in anything political, and there are plenty of people doing this already. But it is not obvious this works, and in any case it is priced in. One can always find such a target, or simply lie about one or make one up. And indeed, this is the path a16z and other similar bad faith actors have chosen time and again.
James Miller: I’m an academic who has written a lot of journalistic articles. If Anyone Builds It has a writing style designed for the general public, not LessWrong, and that is a good thing and a source of complaints against the book.
David Manheim notes that the book is non-technical by design and does not attempt to bring readers up to speed on the last decade of literature. As David notes, the book reflects on the new information but is not in any position to cover or discuss all of that in detail.
I would turn this back at Clara here, and say she seems to have failed to update her priors on what the authors are saying. I found her response very disappointing.
I don’t agree with Rob Bensinger’s view that this was one of the lowest-quality reviews. It’s more that it was the most disappointing given the usual quality and epistemic standards of the source. The worst reviews were, as one would expect, from sources that are reliably terrible in similar spots, but that is easy to shrug off.
The book very explicitly does not depend on the foom, there only being one AI, or other elements she says it depends on. Indeed in the book’s example story the AI does not foom. It feels like arguing with a phantom.
He also points out that there being multiple similarly powerful AIs by default makes the situation more dangerous rather than less dangerous, including in AI 2027, because it reduces margin for error and ability to invest in safety. If you’ve done a tabletop exercise based on AI 2027, this becomes even clearer.
Her first charge is that a lot of AI safety people disagree with the MIRI perspective on the problem, so the subtext of the book is that the authors must think all those other people are idiots. This seems like a mix of an argument for epistemic modesty, or a claim that Yudkowsky and Soares haven’t considered these other opinions properly, and are disrespecting those who hold them? I would push back strongly on all of that.
She raises the objection that the book has an extreme ask that ‘distracts’ from other asks, a term also used by MacAskill. Yes, the book asks for an extreme thing that I don’t endorse, but that’s what they think is necessary, so they should say so.
She asks why the book doesn’t spend more time explaining why an intelligence explosion is likely to occur. The answer is the book is explicitly arguing a conditional, what happens if it does occur, and acknowledges that it may or may not occur, or occur on any given time frame. She also raises the ‘but progress has been continuous which argues against an intelligent explosion’ argument, except continuous progress does not argue against a future explosion. Extend curves.
She objects that the authors reach the same conclusions about LLMs that they previously reached about other AI systems. Yes, they do. But, she says, these things are different. Yes, they are, but not in ways that change the answer, and the book (and supplementary material, and their other writings) explain why they believe this. Reasonable people can disagree. I don’t think, from reading Clara’s objections along these lines, that she understands the central arguments being made by the authors (over the last 20 years) on these points.
She says if progress will be ‘slow and continuous’ we will have more than one shot on the goal. For sufficiently generous definitions of slow and continuous this might be true, but there has been a lot of confusion about ‘slow’ versus ‘fast.’ Current observed levels of ‘slow’ are still remarkably fast, and effectively mean we only get one shot, although we are getting tons of very salient and clear warning shots and signs before we take that one shot. Of course, we’re ignoring the signs.
She objections that ‘a future full of flourishing people is not the best, most efficient way to fulfill strange alien purposes’ is stated as a priori obvious. But it is very much a priori obvious, I will bite this bullet and die on this hill.
There are additional places one can quibble, another good response is the full post from Max Harms, which includes many additional substantive criticisms. One key additional clarification is that the book is not claiming that what we will fail to learn a lot about AI from studying current AI, only that this ‘a lot’ will be nothing like sufficient. It can be difficult to reconcile ‘we will learn a lot of useful things’ with ‘we will predictably learn nothing like enough of the necessary things.’
Clara responded in the comments, and stands by the claim that the book’s claims require a major discontinuity in capabilities, and that gradualism would imply multiple meaningfully distinct shots on goal.
Jeffrey Ladish: I do think you’re nearby to a criticism that I would make about Eliezer’s views / potential failures to update: Which is the idea that the gap between a village idiot and Einstein is small and we’ll blow through it quite fast. I think this was an understandable view at the time and has turned out to be quite wrong. And an implication of this is that we might be able to use agents that are not yet strongly superhuman to help us with interpretability / alignment research / other useful stuff to help us survive.
Anyway, I appreciate you publishing your thoughts here, but I wanted to comment because I didn’t feel like you passed the authors ITT, and that surprised me.
Peter Wildeford: I agree that the focus on FOOM in this review felt like a large distraction and missed the point of the book.
The fact that we meaningfully do get a meaningful amount of time with AIs one could think of as between village idiots and Einsteins is indeed a major source of hope, although I understand why Soares and Yudkowsky do not see as much hope here, and that would be a good place to poke. The gap argument was still more correct than incorrect, in the sense that we will likely only get a period of years in that window rather than decades, and most people are making various forms of the opposite mistake and not understanding that above-Einstein levels of intelligence are Coming Soon. But years or even months can do a lot for you if you use them well.
Will MacAskill offers a negative review, criticizing the arguments and especially parallels to evolution as quite bad, although he praises the book for plainly saying what its authors actually believe, and for laying out ways in which the tech we saw was surprising, and for the quality of the analogies.
I found Will’s quite disappointing but unsurprising. Here are my responses:
The ones about evolution parallels seem robustly answered by the book and also repeatedly elsewhere by many.
Will claims the book is relying on a discontinuity of capability. It isn’t. They are very clear that it isn’t. The example story containing a very mild form of [X] does not mean one’s argument relies on [X], although it seems impossible for there not to be some amount of jumps in capability, and we have indeed seen such jumps.
The ones about ‘types of misalignment’ seems at best deeply confused, I think he’s saying humans will stay in control because AIs will be risk averse, so they’ll be happy to settle for a salary rather than take over and thus make this overwhelmingly pitiful deal with us? Whereas the fact that imperfect alignment is indeed catastrophic misalignment in the context of a superhuman AI is the entire thesis of the book, covered extensively, in ways Will doesn’t engage with here?
Intuition pump that might help: Are humans risk averse?
Criticism of the proposal as unlikely to happen. Well, not with that attitude. If one thinks that this is what it takes, one should say so. If not, not.
Criticism of the proposal as unnecessary or even unhelpful, and ‘distracting’ from other things we could do. Standard arguments here. Not much to say, other than to note that the piecemail ban proposals he suggests don’t technically work.
Criticism of the use of fiction and parables ‘as a distraction.’ Okie dokie.
Claim that the authors should have updated more in light of developments in ML. This is a reasonable argument one can make, but people who (including Clair and also Tyler below) that the book’s arguments are ‘outdated’ are simply incorrect. The arguments and authors do take such information into account, and have updated in some ways, but do not believe the new developments change the central arguments or likely future ultimate path. And they explain why. You are encouraged to disagree with their reasoning if you find it wanting.
Zack Robinson of Anthropic’s Long Term Benefit Trust has a quite poor Twitter thread attacking the book, following the MacAskill style principle of attacking those whose tactics and messages are not cooperating with the EA-brand-approved messages designed to seek movement growth, respectability and power and donations, which in my culture we call a strategy of instrumental convergence.
The particular arguments in the thread are quite bad and mischaracterize the book. He says the book presents doom as a foregone conclusion, which is not how contingent predictions work. He uses pure modesty and respectability arguments for ‘accepting uncertainty’ in order to ‘leave room for AI’s transformative benefits,’ which is simply a non-sequitur. He warns this ‘blinds us to other series risks,’ the ‘your cause of us all not dying is a distraction from the real risks’ argument, which again simply is not true, there is no conflict here. His statement in this linked Tweet about the policy debate being only a binary false choice is itself clearly false, and he must know this.
The whole thing is in what reads as workshopped corporate speak.
The responses to the thread are very on point, and Rob Benginger in particular pulls this very important point of emphasis:
Like Buck and those who responded to expressions their disappointment, I find this thread unbecoming of someone on the LTBT and as long as he remains there I can no longer consider him a meaningful voice for existential risk worries there, which substantially lowers my estimate of Anthropic’s likely behaviors.
I join several respondents in questioning whether Zack has read the book, a question he did not respond to. One can even ask if he has logically parsed its title.
Similarly, given he is the CEO of the Center for Effective Altruism, I must adjust there, as well, especially as this is part of a pattern of similar strategies.
I think this line is telling in multiple ways:
Zack Robinson: I grew up participating in debate, so I know the importance of confidence. But there are two types: epistemic (based on evidence) and social (based on delivery). Despite being expressed in self-assured language, the evidence for imminent existential risk is far from airtight.
Zack is debating in this thread, as in trying to make his position win and get ahead via his arguments, rather than trying to improve our epistemics.
Contrary to the literal text, Zack is looking at the social implications of appearing confident, and disapproving, rather than primarily challenging the epistemics. Indeed, the only arguments he makes against confidence here are from modesty:
Argument from consequences: Thinking confidently causes you to get dismissed (maybe) or to ignore other risks (no) and whatnot.
Argument from consensus: Others believe the risk is lower. Okie dokie. Noted.
As you would expect, many others not named in this post are trotting out all the usual Obvious Nonsense arguments about why superintelligence would definitely turn out fine for the humans, such as that ASIs would ‘freely choose to love humans.’ Do not be distracted by noise.
This post compiles the reactions of others that I have seen to Biden’s Executive Order on AI, including reactions that were based only on the fact sheet, as well as my reactions to those reactions.
Reaction on the worried side was measured. It could best be described as cautious optimism.
Reaction on the unworried side was sometimes measured, but often not measured. It could perhaps be frequently described as unhinged.
It continues to be odd to see so many voices react in such horror to the idea that the government might not ultimately adapt a fully laissez faire approach to AI.
Many of them collectively seem to be, essentially, treating a request for government reports on what might be done in the future, plus some very mild reporting requirements imposed exclusively on a few giant corporations, as if it inevitably means AI, nay computers in general, nay the very core of mathematics itself, will suffer the fate of NEPA or IRBs, a slippery slope of regulatory ratcheting until all hope for the future is extinguished.
I am unusually sympathetic to this view. Such things very much do happen. They very much do often happen slowly. They are indeed strangling much of our civilization. This is all very bad. Pick almost any other hill, everyone involved, where often this is actually already happening and doing great harm, and there are not the massive externalities of potentially everyone on the planet dying, and I would be happy to stand with you.
Alas, no, all that progress energy is focused on the one place where I fear it is deeply misguided. What should be the default viewpoint and voice of reason across the board is silenced everywhere except the one place I wish it was quieter.
I’ll divide the post into three sections. First, the measured reactions, to the fact sheet and then the final executive order. Then those crying out about what can be pried from their cold dead hands.
Vivek Chilukuri: The EO is the Admin’s strongest effort yet to lead by example in the responsible development and deployment of AI, allowing it to go into the UK Summit with a far more fleshed out policy after years of seeing other nations jump out ahead in AI governance.
The Admin’s vision for AI development leans heavily into safety, privacy, civil liberties, and rights. It’s part of an urgent but incomplete effort to offer a democratic alternative for AI development to counter China’s AI model rooted in mass surveillance and social control.
At home, here’s a few ways the EO strengthens US leadership by example:
-Require companies working on advanced AI to share safety tests.
-Develop safety and security standards through NIST
-Guidance for agencies to use AI responsibly
-Support privacy-preserving technologies
Abroad, the EO intensifies US efforts to establish international frameworks, shape international standard setting, and interestingly, promote safe, responsible, and rights-affirming AI development and deployment in other countries.
A note of caution. Going big on an Executive Order is one thing. Getting the execution right is another–especially for federal agencies with an acute shortage of AI expertise. The EO nods to hiring AI experts, but it’s no small task when businesses already struggle to hire.
The establishment of a meaningful compute threshold seems like the biggest game.
David Vorick: New Executive Order is out. Already one notable item: Any AI model that required more than 1e26 floating point operations or 1e23 integer operations to build must report to the government.
Model performance and compute scale are tightly correlated. That could change in the future, but for now, FLOPs remain a highly valid proxy for capability.
Samuel Hammond: Pleasantly surprised the AI EO stuck with a compute threshold. Many will push against this, arguing that regulation should only ever be use-based, but such a threshold will be essentially for delineating AGI / ASI labs for special oversight. This *isthe light-touch approach.
The threshold draws a line for safety testing and disclosures. Emergent capabilities are inherently hard to predict in advance, and thus can’t be regulated on a discrete “use” basis, especially when we’re talking about highly generalist systems, not any one capability.
…
To be clear, no company has yet trained a model with 10^26 FLOPs. Based on Metaculus’s forecast, this particular threshold may not even bind until after 2025.
I strongly agree with this, both that it is important and also that it is the correct move. The big update from this report is that America is going to consider using compute thresholds as a measure for the potential dangers of AI systems. I know of no viable alternatives.
Will Manidis gives the health care section of the final bill a close reading, notes that giving HHS authority over healthcare AI is a broad expansion of its authority, whereas previously it was diffused over FDA, CMS and OCR. Will predicts that HHS will struggle to handle this and there will be trouble. I’d respond that diffusion across four agencies would seem like far more trouble than that.
Katie Link: I think a lot of the EO’s guidance on strengthening security for synthetic biological sequence procurement, as well as the emphasis on evaluation and monitoring of AI models in healthcare settings, are positive developments overall!
But I worry about the negative impact on biological research and open science, and I don’t think the guidelines on open models and datasets meaningfully strengthen our national biosecurity efforts and defense against bioweapon development.
IMO the hardest part about the actual idea -> bioweapon pipeline is not the how-to knowledge (can be found online), or even coming up with synthetic sequences via biological models – it’s the actual lab work, which is still really difficult, expensive, slow, and unpredictable.
…
This is a tricky and important subject that I think requires many diverse perspectives, so I’m looking forward to hearing the community’s thoughts as well as ways HF can do better in improving the transparency and positive impact of biology models and datasets 🤗
It is reasonable to worry about restrictions on bad things like bioweapons also interfering with the research necessary to defend against them. In this case, the extra security precautions seem well worth the extra time and expense to me – if your project can’t afford to pay those extra costs, it was not so valuable. The restrictions on model registration that she highlights I do not expect to much matter, so what if Google registers the next AlphaFold.
Damek Davis notes one conspicuously unmentioned item. Not only is it not required, I can recall no mention of it as a possibility in any of report requests, although it could come up in some of the privacy or discrimination concerns.
Damek Davis: Absent from new exec order: AI companies must reveal their training set. To develop safe AI, we need to know what the model is trained on. Why aren’t the AI safety orgs advocating for this?
The other things that are conspicuously missing, as is pointed out later, is any mention liability or licensing.
Matthew Yglesias: Thou shalt not make a machine in the likeness of a human mind
Matthew Yglesias: My thought is that it’s probably not a good idea to cover the administration’s AI safety efforts much because if this becomes a high-profile topic it will induce Republicans to oppose it.
Arvind + Sayash: How will the EO affect the concentration of power and resources in AI? What about the culture of open research?
We cataloged the space of AI-related policies that might impact openness and grouped them into six categories. The EO includes provisions from all but one of these categories. Notably, it does not include licensing requirements. On balance, the EO seems to be good news for those who favor openness in AI.
But the devil is in the details. We will know more as agencies start implementing the EO. And of course, the EO is far from the only policy initiative worldwide that might affect AI openness.
Licensing does seem obviously hostile to openness. It does not have to be that way, depending on implementation details, but that is the usual result.
The idea that liability is anathema to openness implies some odd things, but it is not in the EO, so not relevant here.
They point out that the reporting requirements that kick in at 10^26 flops are well above the threshold of any open source projects so far. If one did want to cross the threshold, how burdensome are the reports? I continue to say, not very, compared to the cost of that much compute. Can anyone point to a hard part? And if you can’t put together the information necessary for the report, I do not trust you to take the other safety measures appropriate to a model of that size.
They approve of the EO’s emphasis on defending attack surfaces, and say they despair of stopping bad actors from being able to access capable AI, to which I have a very ‘not with that attitude’ style of response. I see defending attack surfaces as a great idea on the margin, so we should do it, but if the bad actors (or enough different actors period) get the unrestricted highly capable models at the cutting edge, I have zero idea how that could end non-badly.
One note I appreciated was the idea that even if AI is being unduly blamed for risks like synthetic bio or cybersecurity, that shoring up those attack surfaces was a good idea anyway. I completely agree. We are not yet at the degree of security where AI is required as a justification.
On transparency they note the lack of such requirements. I would say that the EO does require a bunch of transparency over the reporting threshold, but no it does not do so for data or labor. That would really cause various people to blow their gaskets, although perhaps not in a meaningfully different way than they did already. I find such reactions hard to anticipate. I see such transparency as a nice to have relative to other things.
They note that too much AI concentration is listed as a risk, and that antitrust enforcement in the AI space is to be expected. I understand the impulse to favor this, but also it is rather horrifying. Anti-trust rules explicitly prohibit things like agreeing to proceed in a safe fashion rather than racing as fast as possible in the face of existential risks and massive potential externalities, and potentially ensure universal access to AIs that trigger competitive dynamics we cannot survive. Almost no one has faced the music on this.
It is noteworthy the positive attitude held here, by advocates of openness, versus others who think that this EO is a prelude to regulatory capture, monopoly and tyranny.
On the AI development incentives, they rightfully call them out as woefully weak. I found them so weak that I wrote such efforts off as essentially meaningless. What they want here is for the USG not only not to interfere with open source, they want USG to put its hand on the scale to allow open source to keep up with the top companies. I am hopeful that the winds are going in the opposite direction, we very much do not want open source AI models keeping up.
Overall, this is a cautiously optimistic report card on openness. Not only do they not see this is an existential threat to our freedoms or compute, they see the new rules as mild positives for openness, notice that the actions they fear the most were entirely absent, and are disappointed only that the pro-openness actions were not more substantive. Which I agree is a highly reasonable complaint if you want more openness, those provisions were highly weak sauce, but then that was true in general.
For completeness and fairness, I included everyone who came to my attention via my normal channels during the period.
The core reaction:
Kevin Fisher: A sad day for AI.
The line I expect to be the standard hyperbolic talking point:
David Johnston: The government just issued a limit on the amount of Math you are allowed to do in the United States…
I like there being a catchy phrase like this, so people can identify themselves.
The slippery slope argument, which to be clear is usually valid, and is indeed the correct reason to be worried about the Executive Order:
Suhail (Founder Playground AI, pinned tweet is literally “Run your own race”): I think we will ultimately regret that we did this to ourselves.
I am optimistic because hackers are more clever at bending these sort of silly regulations and much faster at reacting to them than the government could ever.
I do not think we need a stop-gap. Regulate applications, not technology.
One day we will have the equivalent of the gpu compute Azure has in an iPhone and this regulation will seem comical to our children.
Where regulation and reporting requirements begin according to the AI EO:
– 50K H100s at fp16 (w interconnect)
– 1.5M H100s at fp32 (w interconnect)
– 414M H100 training hours
Andrew Ng: Laws to ensure AI applications are safe, fair, and transparent are needed. But the White House’s use of the Defense Production Act—typically reserved for war or national emergencies—distorts AI through the lens of security, for example with phrases like “companies developing any foundation model that poses a serious risk to national security.”
…
It’s also a mistake to set reporting requirements based on a computation threshold for model training. This will stifle open source and innovation: (i) Today’s supercomputer is tomorrow’s pocket watch. So as AI progresses, more players — including small companies without the compliance capabilities of big tech — will run into this threshold. (ii) Over time, governments’ reporting requirements tend to become more burdensome.
The right place to regulate AI is at the application layer.
…
While the White House order isn’t currently stifling startups and open source, it seems to be a step in that direction. The devil will be in the details as its implementation gets fleshed out — no doubt with assistance from lobbyists — and I see a lot of risk of missteps. I welcome good regulation to promote responsible AI, and hope the White House can get there.
Jim Fan (Nvidia): The right place to regulate AI is at the APPLICATION layer. Requiring AI apps like healthcare, self-driving, etc. to meet stringent requirements, even pass audits, can ensure safety. But adding burdens to foundation model development unnecessarily slows down AI’s progress.
Regulate actions/outcomes, not the computing process. Here’s an example.
You only need <100M parameters to build a literal killer AI from hell. Download a Mask R-CNN object detector. Train an ethnicity classifier. Mount a gun on Spot robot dog. Pure evil with a fraction of LLMs’ FLOPs.
I’m not saying the EO is all wrong. It’s got well-intentioned guidelines like “red-teaming report”. But today’s super computer will be tomorrow’s pocket watch. It could over-regulate OSS (thus stifle innovation) while under-regulate cases like the above.
I continue to see this line of ‘regulate applications not technology’ as the failure to understand that with sufficiently capable and available AI the technology implies its application. That the existence of an existentially dangerous AI system is existentially dangerous without it needing to be pointed at any applications.
In particular, the idea that you can allow open source models, and then ‘regulate their applications’ in this context, in any meaningful way, does not make sense to me. If you allow the model to exist in open source then anything that can be done with it after modifications, will be done with it. Merely telling people what they are not allowed to do is not going to work. So how would one regulate the application rather than the model?
If you project far enough into the future, giving a sufficiently capable AI a text window that a human will read is more than sufficient to doom us, and we should be worried that even without such a window it will instead find metaphorical buffer overflows or other physical tricks.
Even if you did manage to instead regulate at the application layer, how is that not going to result in a vastly more intrusive monitoring system? You now need to check every command given to every sufficiently general model, lest it be used for a forbidden application. If you target the compute used in training, then you can limit your concern to large concentrations. If you let the models into the wild, you can’t.
Also, almost the entire EO is indeed examining what to do about applications rather than technology. Section 4, especially section 4.2, dealing with future frontier models, rightfully targets the model because there is no alternative. But the later sections are all about particular applications and particular harms.
Andrew Ng’s statement correctly identifies that the EO’s danger lies in what happens if future stages of implementation go poorly, and that as always the devil will be in the details and regulations have a way of not ending up well-aimed. Eventually, yes, if we keep the threshold at 10^26, it will start to apply to more and more processes.
If the entire process stays at ‘tell us what you are doing’ then if someone is indeed building a pocket watch or other obviously safe system, you can (not quite, but more or less) write down ‘using 4.2 x 10^26 FLOPS, at this data center, running the industry standard mundane harms evaluation package’ and be done with it, even if the threshold is not formally raised. Hell, I bet you can get your local fine-tuned version of GPT-5 to write your disclosures for you, why not, no one will read them if there is indeed no danger there.
The risk is that this report greatly expands and other requirements are added over time, in a way that do introduce an undue burden, and that this also applies to a wider range of systems, and also that none of that was actually necessary for safety. That is a possible world. I don’t expect it, but it makes sense to keep an eye.
However I would also caution that it is not obvious that it will be safe to raise this threshold any time soon. Algorithmic efficiency and data will improve rapidly over time, so 10^26 flops is going to continuously buy you more intelligence, and be more dangerous. It is even plausible that we will need to consider lowering, not raising, this threshold, if our goal is to guard against potential new superintelligent systems, either entirely or when lacking the proper safeguards.
At some point, if AI technology advances, the whole point is that we will need to impose costly safety requirements or even outright prohibitions before creating more capable AI models, because the use or later the very existence of such models poses a potential existential threat, and that threat will expand not contract until we at least solve alignment and I believe also solve other problems that creating such systems would cause even if they were aligned. Ultimately, yes, the ability to interfere with AI capabilities is the point.
Paul Graham: I don’t know what the right approach to regulating AI is, but one problem with this particular approach is that it means we’re heading toward the government regulating private individuals’ computing at an exponential rate.
Eli Tyre: What a correct way of phrasing this! I agree with both parts of this statement. I wish our options were better (more effective at preventing extinction, and imposing fewer externalities on people not doing anything dangerous).
Computing power is increasing at an exponential rate. So any move to regulate it at all is also going to have to evolve at an exponential rate if it is to be meaningful.
Paul’s concern is that it is a move to regulate computing power. Which, yes, this is (in part) absolutely a move to lay groundwork for regulating computing power. Which is the only proposed move that could plausibly prevent or substantively low the creation of AGI until such time as it would not result in everyone dying.
Paul Graham wishes that we would find such an alternative that would work, and I do as well. We would love to hear about it. But a lot of very smart highly motivated people have been searching, and this is the best we have found so far by quite a lot. I do not see ‘let anyone scale however much they want with safeguards being fully optional’ as a viable alternative. So what choice do we have other than talking price?
Zvi (on Twitter, other thread): The reward for reading 20k words of an EO that is almost entirely calls for government reports is to read the diatribes about how it is government overreach and full of premature regulations and assumptions, when you spent hours confirming that nope, it’s all report requests.
Nabeel Quereshi: Regulations usually _start out_ reasonable, the (obvious) concern is that they’re a one-way ratchet. See IRBs as an example.
Shea Levy: From most, I’d say this belies naïveté about the purpose and effects of this kind of bureaucracy and suggest that the author review the history or think for five minutes about how a petty power-luster might take advantage of and expand this. Zvi, of course, already knows this.
Maxwell Tabarrok (other thread): Hmm yeah if a law just calls for bureaucrats to prepare reports it can’t be that burdensome. There is no precedent for laws of this nature evolving into unbelievably wasteful time sinks which drain huge swathes of the economy and cripple important technologies.
This does indeed seem to be the full argumento ad absurdum. It is saying that we should never dare ask a government department to write a report on what is happening, might happen in the future or might be done about any of it, lest it be a prelude to tyranny.
Again, that is not, in general, an inherently absurd view. There are places where it would be better if the government never created any reports, so that it is not tempted to ever do anything. One need not affirm a regulation that is net positive as stated, if one expects it to then ratchet up and become negative. But as is the central theme, the only offered alternative is to do actual nothing indefinitely.
Again, reminder, I love you guys, I really do, shame we couldn’t meet anywhere else.
Adam Thierer: The new Biden AI executive order issues broad and quite amorphous calls for expanded government oversight across many other issues and agencies, raising the risk of a “death by a thousand cuts” scenario for AI policy in the US.
The EO appears to be empowering agencies to gradually convert voluntary guidelines into a sort of back-door regulatory regime for AI, a process which would be made easier by the lack of congressional action on AI issues. The danger exists that the U.S. could undermine the policy culture that made our nation a hot-bed of digital innovation and investment. We don’t want to become Europe on tech policy.
Geoffrey Manne: People really miss how important the culture of entrepreneurship is in the U.S. — and how unique it is. The relentless effort to regulate away our superpowers is truly depressing.
Indeed, this order is a call for future oversight, which could indeed lead to maybe telling people that they are not allowed to do some things, or that to do some things they might also have to do other things. Yes, the government might decide it wants to stop certain things from happening, and might do so more often than is ideal. I can confirm that is the point. On some of the points I strongly agree with the need to act, on others I am skeptical. But if you react this way to every single proposed action ever, with the same generic reasoning, your reaction isn’t information.
Reason similarly posts their standard ‘this new proposed regulation can only hurt America.’ I do appreciate them highlighting the invocation of the Defense Production Act. Someone has to point out such things in theory matter. But when the time comes to show what the great harms will be that are going to stifle innovation, they are at a loss, because the whole thing is a list of requirements for an endless set of reports.
R Street’s Adam Thierer writes the required column, and to his credit often chooses his words carefully. Note the ‘threatens to’ in the title, which is: White House Executive Order Threatens to Put AI in a Regulatory Cage.
Adam Thierer: While some will appreciate the whole-of-government approach to AI required by the order, if taken too far, unilateral and heavy-handed administrative meddling in AI markets could undermine America’s global competitiveness and even the nation’s geopolitical security.
Yes, if you take regulation too far, that would be bad. The argument seems to be once again, therefore, no regulation at all, and even no government reports? In general I do have great respect for that position, but it is deeply unworkable here, and is deeply unpopular not only here but in the places it is largely or entirely correct.
I was disappointed that he does not maintain his message discipline throughout, with derisive references to existential risk concerns, starting with ‘scenarios pulled from the plots of dystopian science fiction books and movies’ and getting less respectful from there.
Forbes also pulls out standard tropes.
James Broughel (Forbes): The government’s “regulate first, ask questions later” approach is reminiscent of when Congress created the renewable fuels program in 2005, which increased the ethanol content in gasoline.
Forbes calls the order ‘regulation run amok’ and accuses it of ‘Ready! Fire! Aim!” Except the entire report is ‘Ready! Aim!’ and no Fire. This is the Ask Questions First document, perhaps run amok in the breadth of its questions. At best this is training to run, but still at the starting gate. Where are the overly prescriptive regulations? What is the prescription of which you speak? The ‘stringent regulatory hurdles’ that involve ‘tell people what you are doing’ and only apply to a handful of large corproations? Where are all the ‘assumptions’ in the order?
The best they can come up with was a worry I had as well, that requiring reporting of safety results might discourage testing. On the other side, I would expect having to report all tests to also strongly encourage testing, because otherwise you are conspicuously not reporting any tests.
Ser Jeff Garzik: In re AI EO: This is yet again a familiar pattern of acting first from the executive branch. People naively want a chieftain-king, and Biden admin happily supplies: rule by executive fiat, achieved through arcane legal positioning.
This AI EO feels like a return to the 1990s:
– whitelisted USG Big Corp friends have no limits on GPU cycle count or parameter count.
– plebs must make do with lesser models …
Just like the early 1990s encryption, where 40-bit and below weak encryption was exportable outside the United States.
[Then in response to the Defense Production Act he invokes Godwin’s Law.]
The #AI #EO is another cog in the automated #surveillance and control machine being built. #privacy #OpenSource
This is, once again, many steps ahead of ourselves, except also assuming what he sees as the worst outcome on many different fronts at once. Is this ultimate outcome possible down the line? Is that one way we could implement compute limits? It certainly is possible. Would not be my first choice. But once again, that is not what the EO says. I presume that is how it feels to Jeff because that is his background assumption of what will always happen.
One almost suspects some of these posts were written before they knew what was in the order. Others seem to be reading an entirely different EO, that travelled back in time from a possible 2026.
[This section was added on 11/2, as I initially did not realize Ben’s post was responding to the executive order until then.]
Ben Thompson offers us a clear confirmation that the CAIS letter was indeed a big deal, in his post that is mostly another executive order reaction bemoaning the idea that anyone might ever consider regulating anything and not seeing any differences or dangers in AI as legitimate concerns. He thinks the CAIS letter was an op.
Gregory Allen: The other thing that’s happened that I do think is important just for folks to understand is, that Center for AI Safety letter that came out, that was signed by Sam Altman, that was signed by a bunch of other folks that said, “The risks of AI, including the risks of human extinction, should be viewed in the same light as nuclear weapons and pandemics.” The list of signatories to that letter was quite illustrious and quite long, and it’s really difficult to overstate the impact that that letter had on Washington, D. C. When you have the CEO of all these companies…when you get that kind of roster saying, “When you think of my technology, think of nuclear weapons,” you definitely get Washington’s attention.
Ben Thompson says that it is telling that the letter was signed by so many at OpenAI, Anthropic, Google and Microsoft, the market leaders, but no one at Apple or Amazon, as evidence that it was a cynical ploy for regulatory capture from people who claimed to know where a new technology was going, when no one can really know. This is, once again, the fully cynical perspective that saying your product might kill everyone is a clever strategic move to seek regulatory capture, what else could it be, and also conveniently ignoring all the non-commercial signatories and all the usual other arguments why this is nonsense.
But why would the letter have even asked for signatories at Apple or Amazon? Their AI strategies did not (at least at the time) visibly involve building AI, and I do not know the name of a single person on the AI team of either company. There is always someone whose signature you did not get – Tyler Cowen asked why we did not seek out priests or local car dealers or Republicans.
Ben Thompson then moves on to the executive order, which he views with utter disdain, as the wrong way to handle innovation, and he points to this further article by Steven Sinofsky. Both view the order as a conspiracy to turn the market over to incumbents and stifle innovation. Both are completely dismissive of the idea that there might be any dangers worthy of concern, or that AI might pose an existential threat, framing that as ‘alarmist.’
Even more than that, Ben even dismisses outright the possibility that there is any legitimate worry that AI might even be capable of mundane harms worth guarding against. He complains that there is only the ‘briefest of handwaves’ about AI’s upsides when talk of and explorations of ways to help with it are infused throughout the order, and while he offers a handwave outright dismissing any need to be concerned with downsides, because it is such early days.
The comparison Ben makes to mobile phones, that it would be the height of foolishness a la Bill Gates and Microsoft to think you know where phones will evolve or what they will be capable of, is in some ways highly apt. We do not know where this will be in 5 years, we do know it will have capabilities that we currently lack, potentially highly powerful and impactful ones. That is the whole idea. And it is often easier, as it is in this situation, to predict resulting endpoints than the path along the way.
From this and other writing by Ben Thompson, whose other work commercially modeling technology I respect quite a lot, I see someone who considers AI nothing but another consumer technology, whose future we cannot predict, except that it is without any transformative or potentially catastrophic impacts to worry about. I see no sign he has given the question of existential risk serious thought. And thus he falls back upon the general principle that such a technology needs to be kept as far away from any regulation or interference as possible, and any talk by incumbents to the contrary must be for regulatory capture, and anyone else calling for it is presumed either anti-technology or bought and paid for.
It is disappointing, but I have accepted that this is his perspective.
The post Ben Thompson links to, by Steven Sinofsky, is a long rant that starts by attacking the executive order as an end run around democracy and rule of law, frames it as entirely an attempt to stifle innovation, draws parallels to every other attempt in the past to regulate technology, and uses concerns with past technologies to dismiss any potential concerns that AI might have any downsides. Except, he says, when the government has access to it, at which point he warns of various dystopian outcomes, so he is against the pro-AI-in-government parts of the order as well. In general this seems like it takes every anti-government anti-big-corporate pro-free-technology-and-software rant and cause of the last 50 years, and transfers it all to AI straight up, while also treating every request for a report as for all practical purposes a declaration of a new monopoly given to insiders.
Bindu Reddy offers her usual flavor of thoughts.
Bindu Reddy (CEO AbacusAi): The AI Executive Order is a bit ridiculous and pretty hard to enforce.
Here are the issues –
1. Any foundation model that poses a serious risk to national security – How do you determine if something is a “serious risk to national security!“? If this is about mis-information, Twitter, YouTube and Meta are way more serious risks to national security than harmless AI models that merely generate content, not distribute or amplify it.
I presume that the government intends to do that evaluation itself, the same way it does with any other sort of threat. There is not some technical definition of what is a serious risk to national security, nor does a complete one exist to be found. We use phrases like ‘a clear and present danger’ and ‘probable cause’ to justify government actions.
Right now, I would agree that current models do not present such a threat. My understanding is that the current administration agrees with this.
If a model does not distribute its content in any way, and no one can or does access its outputs, then the model would need to be far more advanced and do some other physical thing to pose a threat to national security. But if there exist people, who are reading the output of the model, then that output can impact those people to do things in physical space, or the information can be utilized or amplified by those people, without any bound. Such a thing could easily pose a threat to national security. Whether or not any given system (e.g. Llama-3 or GPT-5) did so is a fact and physical-world question.
Certainly it would be unwise to say ‘we cannot technically define what is and is not a threat to national security in precise terms thus we should not respond to such threats’ any more than we would say ‘we cannot technically define what is and is not fraud in precise terms thus we should not have laws against fraud.’ Or murder, or anything else.
2. All AI-generated content has to be watermarked? – Seriously!? We may as well kill vision AI, if we actually enforced that. Are Enterprises allowed to use AI to generate images and use them in their marketing?
This is bizarre hyperbole, even if they do ultimately impose watermarking requirements. Yes, of course you are allowed to use AI images for marketing, no this will not kill vision AI. There are tradeoffs, this introduces some marginal costs, and one could argue it is not worthwhile, but I have yet to see an argument for the costs being a big deal, especially given that costs are continuously declining over time, so even a tax that looks big doesn’t ultimately do much if it is a fixed percentage.
3. Some agency is apparently going to develop tests that the AI models have to pass – How will this get enforced? Does every fine-tuned version of any open-source model have to pass these tests? What happens if a model is dropped on HuggingFace by a non-US corporation?
Yes. If there is a test you have to pass then you have to pass it.
Any reasonable regulatory regime is going to require passing some set of tests. A sufficiently capable AI model is going to be able to do things that trigger regulatory scrutiny, one way or another.
Does every fine-tuned version of any open source model have to pass these tests?
We don’t know yet. In an ideal world and in my best guess I would say no unless you are doing very expensive and extensive fine tuning and then releasing the result.
Why? If you release an open source AI, you are also releasing every fine-tuned version of that open source AI. So the original test should cover any potential threat from a (lightly) fine-tuned version, say up to 1% of original compute training costs.
If this means no open source model above some level can ever pass the test? Then you can’t have open source models above that level. That’s the whole point.
What happens if a model is dropped on HuggingFace without passing the tests? Then I would presume they are in violation of American law, and further spreading or using it would also be a violation of American. Where ‘they’ above means both the other company and also HuggingFace, who we have jurisdiction over. It would be the job of such websites to prevent the sharing of illegal models, same as any other illegal software or files one might upload. I am however not a lawyer or legal scholar, and all that.
4. It has a bunch of “privacy” protecting language – All this does is cripple AI models compared to Search where the same “privacy” isn’t being enforced. Google search will recognize a face for me, GPT-4V won’t. It’s not clear why my photo that is in the public domain, can’t be used by AI? We already have privacy protecting laws. These should be sufficient.
I agree that we should treat mandatory privacy protections with skepticism, if they are ultimately implemented. Which, so far, they are not.
The privacy argument and harm argument here, as I understand it, is that the AI could soon be shockingly good at recognizing anyone anywhere as well as locations, while also making mistakes, and also people will create fake images that the AI would identify as being the people they are pretending to be in the places they are pretending to portray, doing the things they are pretending to do, and so on. So if people start using vision AI for identification then this opens up some nasty dynamics.
Do I find this annoying and rather silly? Yeah, I basically do, but I am open to taking a cautious approach here, especially as part of a package of precautions.
It then ends with some “feel good” language around hiring and open-source. Net-net, this order won’t make any difference in the near term.
Could be harmful in the long run to companies who do develop these foundation models, especially vision models. Weirdly, it helps search engines or makes them way more powerful because they are not regulated, while AI models are🤔🤔. .
. . If anything, this isn’t good for OpenAI 🤷♀️
Did you hear that last line, everyone? There will of course be endless complaints that this whole operation is regulatory capture by OpenAI (along with DeepMind and Anthropic and perhaps others). Here is someone, who is very against regulation, saying the order is if anything not good for OpenAI.
🚩 Zooming out, what struck to me at the beginning is that this is not just an “AI” executive order, it is infact a retroactive governance regime being imposed on the entire web, while also limiting the future. The term AI is defined so broadly, that it could impact everything from Google search results, to Amazon recommendations, to Yelp restaurant suggestions, to anything else which recommends anything to you. This is the start of the “AI-industrial-complex”, which will now lead to a world of billions of dollars being spent in regulations, compliance and lobbying over the time ahead.
I mean yes the definition of AI here is not good but one I presume that gets fixed, and I will support efforts to get it fixed, and also there is no imposed regime except for reporting requirements above 10^26 flops? Potentially there could be requirements in the future.
🚩 This EO is essentially banning fully open source models over 10B parameters, as it requires reporting requirements and “..physical and cybersecurity measures taken to protect those model weights”.
No. It requires that someone involved send an email saying ‘the measures we took to protect these model weights was to post them on Torrent.’
(Also, tens means 20B, not 10B, for what that’s worth, which is not a tiny model.)
If you think that is a bad look when said out loud, or that this is an unacceptable answer… well, that is what it means to be open source.
If this leads to a ban on such models in the future, then yes I would be in favor of that. I mean obviously, if something needs to be secure then you cannot simply give it away to everyone who asks nicely.
🚩 Beyond the datacenter clouds, this would also extend to even decentralized networks providing compute, where even the location and total compute has to be reported. To Floyd in Florida running a SETI @ home node searching for extraterrestrial life signals – the government wants to know the precise location of your computer in your basement now…
Yeah, no, that is not what I expect that this means, but if it did GPS can do this for you, the government already had ways to find out, no one forced you to join that and I honestly do not particularly care?
Every time a new regulation passes, there is a class of free software style people who try to figure out exactly what a maximalist interpretation of what this might imply might do if fully enforced and they panic. That is a useful thing for someone to do, but we should not confuse it with what is likely to happen.
🚩 Compliance regime for any model which was trained with more than 10^26 floating-point operations (or 0.1 exaflops)
The compliance regime is literally ‘file a report on what you are doing.’ But I certainly hope it leads to this!
🚩 Training requests from non-US persons to be submitted. This is like opening a bank account now!
If you are trying to do a full training run that requires a report then I do not think this is much of an additional burden?
I also don’t see why one couldn’t rent a data center somewhere else if your request is lightweight – the concern I actually have here is that business can easily be taken elsewhere.
🚩 This is a remarkable regulatory capture exercise, as now the cost of compliance for anyone running a physically co-located datacenter with compute above a threshold now increases
For data centers? I have actual no idea how the requirements here could lead to datacenter regulatory capture. What is the compliance cost that is non-trivial here? That they perform KYC on major customers? That they know where their computers are? I would love if these are our regulatory capture worries, in places where I would worry about regulatory capture.
🚩 Models in this regime are basically guilty until proven innocent if they require compute > 0.1 exaflops
Models are found guilty if you do not report that they exist and forced the government to go find them, I suppose. Otherwise, this is pure fiction in terms of the EO.
But in terms of the actual situation? And what I hope is the law a few years down the line? Hell yes, of course capable foundation models are dangerous until proven safe, that is how safety works when you create such things, I have no idea what the sane alternative is.
I’ve covered his links here elsewhere on their full contents, but for full context here are the things he quoted and said about the EO.
Again, I don’t think there is anything to ‘enforce’ here? Not yet anyway.
Agustin Lebron: Even though it’s not a compute cap (just a requirement to register, for now), surely this emboldens competitors to OpenAI/etc? If I think I can build a system up to the cap, then my playing field just got more level.
This will also spur lots of research into sample efficiency again, which IMO is a very good thing. It’s been a research area that has languished for years under the “just throw more flops at it” regime.
Finally, I think the order perversely legitimizes capabilities research by showing the world “Well, the US gov’t thinks capabilities are SO important they’re making orders about it”. More attention, more dollars, more progress.
Overall this is accelerative for AI/AGI IMO.
I don’t know what to say to that, beyond being confused why any of that would change what anyone does. People reacting to USG caring about AI capabilities? All sides should be able to agree USG is playing from behind on that one. Does this actually update anyone? Under this logic, what wouldn’t be accelerative?
Also, for a potentially accelerative document, it sure has a lot of accelerationists really angry. Stay tuned for more of that.
A commentor has a better accelerationist case:
Marcel: I work for a federal agency. Right now at every single federal agency there are people trying to get the agency to approve using AI for some purpose. And there are people saying no way, that’s too risky. The existence of this EO will allow the people trying to bring AI into the government to say “look, the administration is in favor of AI” and in some cases that will win the day.
I am fairly confident this EO will result in greater use of AI by the federal government. Whether that is a good thing depends on your view of the government, I suppose.
Narrowly restricted to the government itself, I agree. This is intended to be, and will be, accelerationist. I do not expect that to result in important capabilities gains, and I’m fine with it. American state capacity is mostly good, actually?
I mean the pictures are pretty damn cool, I do love DALLE-3. Not sure if joking.
Or rather, not sure exactly how much joking. Not zero. Not completely.
Release the lawsuits?
Suhail (Founder, Playground AI): We commit at least $100K to any lawsuit that wants to litigate and get a ruling if a regulatory agency attempts to follow the EO on AI. An EO is not law, yet. We must fight back.
Pls reach out if you can match or do more so we can establish a fund.
This fight extends beyond regulatory capture. If you read between the lines, it’s about who owns the oligopoly of superintelligence of our civilization. Nobody should.
Kevin Fisher (QT of Suhail): I carefully read through the executive order on AI and it’s very clear: it’s a giant document written by OpenAI and a few others outlining a plan for AI to be entirely owned and thought policed by giant corporations with no other alternatives. But there are ways to fight back.
Kevin Fisher (other thread) I don’t think everyone fully appreciates how dire the fight for “code as speech” is.
It’s not academic – the outcome will affect the fabric of our society at fundamental level: do we have freedom of thought or will a few corporations censor or effectively moderate our collective internal dialogue?
Within a few years internal dialog will be externalized through close relationships with AIs powered by language models. We’ve already seen how “safety” limits the ability to explore ideas and concepts on our own terms.
It’s incredibly important that a totally free market is maintained without regulatory distortion to avoid a future where things like “I’m sorry I can’t talk about that” or “You shouldn’t say that” don’t start policing effectively your personal thoughts.
…
Yeah this is one step further: are you free to say something in private? We’re talking about mass surveillance and monitoring of personal and private speech
Colin O’Malley (reply to Suhail): What do find objectionable about this EO? Or do you object to any AI regulation, on principle?
Suhail: Regulating flops is the incorrect strategy.
Again, I ask, then what is the correct strategy?
Kudos, in a strange way, to Suhail and Marc for focusing on the part that matters most. They very much want to ensure people can do the things that I believe will get us all killed. Namely, they want to race to superintelligence as fast as possible, and put it in the hands of everyone at once. Which, again, I very much expect to highly reliably if implemented cause there not to exist, in the physical universe, any humans, and rather quickly. And I am taking a bold stance that I think this is bad.
And they are furious that there might be any sort of future barrier in the way of doing those things as quickly as possible.
I presume their plan is that someone attempts to file a report, and then Suhail’s group will file a lawsuit that says how dare you attempt to file a report? Or maybe OpenAI or Google files a report saying they are training a model, and Suhail somehow finds standing and files a lawsuit to claim they didn’t have to file the report? I have no idea how the actual suit would work here.
Kevin is taking the position that any restriction on code whatsoever is unconstitutional and also incompatible with our freedoms, the alternative is the end of our freedom of thought and speech and a mass surveillance state, and all of this is an OpenAI conspiracy.
It is odd that we both read the same executive order?
I think this is a good summary of all the reactions above:
Jack (MidJourney): yeah the e/acc response to this has been totally unhinged. this EO seems likely to mildly accelerate deployment of existing systems and have negligible effect on development of new ones. If you’re spending $50m on a single training run a single report is nothing.
Unhinged seems exactly right. The people crying wolf will never not cry wolf. What will they say if and when a wolf actually arrives, that is different from what they are saying now? That is a question such types often ask those claiming risk, both fairly and otherwise. It seems fair to turn it back on them here.
Roon steps in as another voice of reason.
Roon: all the executive order says as far as I can tell is that you need to report the existence of a massive compute datacenter don’t you need to report the existence of your car and your home to the government? and have them inspect your restaurant’s kitchen and shit.
this seems kind of a weird thing to freak out over. like they can definitely see it from space and it’s impossible to run one of these without a massive energy footprint.
Brian Chau: The only tangible thing in the Biden executive order seems to be creating a de-facto registry of models and datacenters. Dear lawyer friends: Is any EO authority to make anyone comply with this request?
Flopcaps are basically fine, it’s everything else the biden admin will use the registry for that’s the problem.
As I said, the EO is basically fine. Flopcaps are the most reasonable thing you could be up to and even then its just a reporting flopcap.
Nathan: According to @TheZvi, Biden’s AI order is very light on things that will slow down AI companies unduly. If I understand him correctly the main extra burden falls on bureaucrats to compile reports.
Dave Kasten: I would characterize it as the bureaucratic equivalent of the bartender theatrically rolling up his sleeves before getting ready to bounce a troublemaker.
Indeed may many other things come to pass in the future, after all those reports are filed. There are many of them you may wish to influence. In the meantime, everyone, please do your best to stay sane.
