openai

The board firing Sam Altman, then reinstating him, dominated everything else this week. Other stuff also happened, but definitely focus on that first.

Developments at OpenAI were far more important than everything else this read. So you can read this timeline of events over the weekend, and this attempt to put all the information together.

1. Introduction.

2. Table of Contents.

3. Language Models Offer Mundane Utility. Narrate your life, as you do all life.

4. Language Models Don’t Offer Mundane Utility. Prompt injection unsolved.

5. The Q Continuum. Disputed claims about new training techniques.

6. OpenAI: The Saga Continues. The story is far from over.

7. Altman Could Step Up. He understands existential risk. Now he can act.

8. You Thought This Week Was Tough. It is not getting any easier.

9. Fun With Image Generation. A few seconds of an Emu.

10. Deepfaketown and Botpocalypse Soon. Beware phone requests for money.

11. They Took Our Jobs. Freelancers in some areas are in trouble.

12. Get Involved. Dave Orr hiring for DeepMind alignment team.

13. Introducing. Claude 2.1 looks like a substantial incremental improvement.

14. In Other AI News. Meta breaks up ‘responsible AI’ team. Microsoft invests $50b.

15. Quiet Speculations. Will deep learning hit a wall?

16. The Quest for Sane Regulation. EU AI Act struggles, FTC AI definition is nuts.

17. That Is Not What Totalitarianism Means. People need to cut that claim out.

18. The Week in Audio. Sam Altman, Yoshua Bengio, Davidad, Ilya Sutskever.

19. Rhetorical Innovation. David Sacks says it best this week.

20. Aligning a Smarter Than Human Intelligence is Difficult. Technical debates.

21. People Are Worried About AI Killing Everyone. Roon fully now in this section.

22. Other People Are Not As Worried About AI Killing Everyone. Listen to them.

23. The Lighter Side. Yes, of course I am, but do you even hear yourself?

GPT-4-Turbo substantially outperforms GPT-4 on Arena leaderboard. GPT-3.5-Turbo is still ahead of every model not from either OpenAI or Anthropic. Claude-1 outscores Claude-2 and is very close to old GPT-4 for second place, which is weird.

Own too much cryptocurrency? Ian built a GPT that can ‘bank itself using blockchains.’

Paper says AI pancreatic cancer detection finally outperforming expert radiologists. This is the one we keep expecting that keeps not happening.

David Attenborough narrates your life how-to guide, using Eleven Labs and GPT-4V. Code here. Good pick. Not my top favorite, but very good pick.

Another good pick, Larry David as productivity coach.

Oh no.

Kai Greshake: PSA: The US Military is actively testing and deploying LLMs to the battlefield. I think these systems are likely to be vulnerable to indirect prompt injection by adversaries. I’ll lay out the story in this thread.

This is http://Scale.ai’s Donovan model. Basically, they let an LLM see and search through all of your military data (assets and threat intelligence) and then it tells you what you should do..

Now, it turns out to be really useful if you let the model see news and public information as well. This is called open-source intelligence or OSINT. In this screenshot, you can see them load “news and press reports” from the target area that the *adversarycan publish!

We’ve shown many times that if an attacker can inject text into your model, you get to “reprogram” it with natural language. Imagine hiding & manipulating information that is presented to the operators and then having your little adversarial minion tell them where to strike.

…

Unfortunately the goal here is to shorten the time to a decision, so cross-checking everything is impossible, and they are not afraid to talk about the intentions. There will be a “human in the loop”, but that human would get their information from the attacker’s minion!

…

@alexlevinson (head of security at scale) responded to me, saying these are “potential vulnerabilities inherent to AI systems, […] do not automatically translate into specific vulnerabilities within individual AI systems”

And that “Each AI system […] is designed with unique security measures that may or may not be susceptible to the vulnerabilities you’ve identified”.

Now, I’ve not had any access to Donovan, and am only judging based on the publicly available information and my expertise. I hope everyone can judge for themselves whether they trust Scale to have found a secret fix to this issue that gives them confidence to deploy.. or not.

Yeah, this is, what’s the term, completely insane? Not in a ‘you are going to wake up Skynet’ way, although it certainly is not helping on such matters but in a ‘you are going to get prompt injected and otherwise attacked by your enemies’ way.

This does not even get into the ways in which such a system might, for example, be used to generate leaks of classified documents and other intel.

You can hook the LLM up to the weapons and your classified data. You can hook the LLM up to outside data sources. You cannot responsibly or safely do both. Pick one.

Robin Hanson survey finds majority see not too much productivity enhancement in software yet.

Robin Hanson: Median estimate of ~7% cut in software time/cost over last 10 years (ignoring LLMs), ~4% recently due to LLMs. But high variance of estimates.

Robin Debreuil: Lots of experience, and I’m 100% sure it’s already less than 90. Also a lot of this saving are on the front end of development (finding and integrating technologies, testing, etc), so prototyping ideas much faster. Quality will improve dramatically too, but hasn’t so much yet.

I agree with Debreuil for most projects. What is less obvious is if this yet applies to the most expensive and valuable ones, where the results need to be rather bulletproof. My presumption is that it still does. I know my productivity coding is up dramatically.

There was a disputed claim in Reuters that prior to Altman’s ouster, the board was given notice by several researchers of alarming progress in tests of a new algorithmic technique called Q*, and that this contributed to Altman’s ouster. Q refers to a known type of RL algorithm, which it makes sense for OpenAI to have been working on.

The reported results are not themselves scary, but could point to scary potential later. If Altman had not shared the results with the board, that could be part of ‘not consistently candid.’ However, this story has been explicitly denied in Verge, with their editor Alex Heath saying multiple sources claimed it wasn’t true, and my prediction market has the story as 29% to be substantively true even offering ‘some give.’ This other market says 40% that Qis ‘a significant capabilities advance.’

For now I will wait for more info. Expect follow-up later.

Sam Altman was fired from OpenAI. Now he’s back. For details, see my two posts on the subject, OpenAI: Facts from a Weekend and OpenAI: Battle of the Boards.

The super short version is that Altman gave the board various reasons to fire him that we know about and was seeking to consolidate power, the board fired Altman essentially without explanation, Altman rallied investors especially Microsoft and 97% of the employees, he threatened to have everyone leave and join Microsoft, and the board agreed to resign in favor of a new negotiated board and bring Altman back.

What happens next depends on the full board chosen and who functionally controls it. The new temporary board is Brad Taylor, Larry Summers and Adam D’Angelo. The final board will have nine members, one from Microsoft at least as an observer, and presumably Altman will eventually return. That still leaves room for any number of outcomes. If they create a new board that cares about safety enough to make a stand and can serve as a check on Altman, that is a very different result than if the board ends up essentially under Altman’s control, or as a traditional board of CEOs who unlike D’Angelo prioritize investors and profits over humanity not dying. We shall see.

The loud Twitter statements continue to be that this was a total victory for Altman and for the conducting of ordinary profit-maximizing VC-SV-style business. Or that there is no other way any threat to corporate profits could ever end. That is all in large part a deliberate attempt at manifestation and self-fulfilling declaration. Power as the magician’s trick, residing where people believe it resides.

Things could turn out that way, but do not confuse such power plays with reality. We do not yet know what the ultimate outcome will be.

Nor was Altman’s reinstatement inevitable. Imagine a world in which the board, instead of remaining silent, made its case, and also brought in credible additional board members while firing Altman (let’s say Taylor, Summers, Shear and Mira Murati), and also did this after the stock sale to employees had finished. I bet that goes considerably differently.

Recommended: A timeline history of the OpenAI board. At one point Altman and Brockman were two of four board members. The board has expanded and contracted many times. No one seems to have taken the board sufficiently seriously during this whole time as a permanent ultimate authority that controls its own succession. How were things allowed to get to this point, from all parties?

Recommended: Nathan Lebenz offers a thread about his experiences red teaming GPT-4. Those at OpenAI did not realize what they had, they were too used to worrying about shortcomings to see how good their new model was. Despite the willingness to wait a long time before deployment, he finds the efforts unguided, most involved checked out, a fully inadequate process. Meanwhile for months the board was given no access to GPT-4, and when Lebenz went to the board attacks on Lebenz’s character were used to silence him.

At the ‘we’re so back’ party at OpenAI, there was a fire alarm triggered by a smoke machine, causing two fire trucks to show up. All future fire alarms are hereby discredited, as are reality’s hack writers. Do better.

Bloomberg gives additional details on Wednesday afternoon. There will be an independent investigation into the whole incident.

A thing Larry Summers once said that seems relevant, from Elizabeth Warren:

He teed it up this way: I had a choice. I could be an insider or I could be an outsider. Outsiders can say whatever they want. But people on the inside don’t listen to them. Insiders, however, get lots of access and a chance to push their ideas. People – powerful people – listen to what they have to say. But insiders also understand one unbreakable rule: They don’t criticize other insiders.

I had been warned.

Stuart Buck interprets this as siding with Altman’s criticism of Toner.

The other implication in context would be that Altman is this form of insider. Which would mean that he will not listen to anyone who criticizes an insider. Which would mean he will not listen to most meaningful criticism. I like to think that instead what we saw was that Altman is willing to use such principles as weapons.

My actual understanding of the insider rule is not that insiders will never listen to outside criticism. It is that they do not feel obligated or bound by it, and can choose to ignore it. They can also choose to listen.

A key question is whether this is Summers endorsing this rule, or whether it is, as I would hope, Summers observing that the rule exists, and providing clarity. The second insider rule is that you do not talk about the insider rules.

Also on Summers, Bloomberg notes he expects AI to come for white collar jobs. In a worrying sign, he has expressed concern about America ‘losing its lead’ to China. What a world in which our fate largely rests on the world models of Larry Summers.

Parmy Olson writes in Bloomberg that the previous setup of OpenAI was good for humanity, but bad for Microsoft, that the new board will be traditional and current members scream ‘safe hands’ to investors. And that Microsoft benefits by keeping the new tech at arms length to allow OpenAI to move faster.

Rob Bensinger asks, if Toner’s statements about OpenAI shutting down potentially being consistent with its mission are considered crazy by all employees, what does that say about potential future actions in a dangerous future?

Cate Hall reminds us that from the perspective of someone who thinks OpenAI is not otherwise a good thing, those board seats came with a very high price. If the new board proves to not be a check on Altman, and instead the week backfires, years of strategy by those with certain large purse strings made no sense.

Claim that Altman at his startup Loopt had his friends show up during a negotiation and pretend to be employees working on other major deals to give a false impression. As poster notes, this is Altman being resourceful in the way successful start-up founders are. It is also a classic con artist move, and not the sign of someone worthy of trust.

After the deal was in place, Vinod Khosla said the nonprofit control system is fine, look at companies like IKEA. Does he not understand the difference?

Fun claim on Reddit by ‘OpenAIofThrones’ (without private knowledge) of a more specific, more extreme version of what I outline in Battle of the Board. That Altman tried to convene the board without Toner to expel her, Ilya balked, that presented both the means and a short window to fire Altman before Ilya changed his mind, and that ultimately Altman blinked and agreed to real supervision.

Whatever else happens, we can all set aside our differences to point out the utter failure of The New York Times to understand what happened.

Garry Tan: NYT just going with straight ad hominem with no facts on the front page these days Putting the “capitalists” in the crosshairs with the tuxedo photos is some high class real propaganda.

Paul Graham: OpenAI’s leaders, employees, and code are all about to migrate to Microsoft. Strenuous efforts enable them to remain working for a nonprofit instead. New York Times reaction: “A.I. Belongs to the Capitalists Now.”

That is very much not how any of this works.

On the narrow issue below, performative people like to gloat. But Helen Toner is right, and Greg Brockman is wrong.

It is very much an underdog, but one sincere hope I have is Nixon Goes to China.

Altman now has the loyalty of his team, a clear ability to shut down what he helped build if crossed, and unwavering faith of investors who trust him to find a way. No one can say they do not ship. OpenAI retains a rather large lead in the AI race. The e/acc crowd has rallied its flags, and was always more into being against those against things rather than anything else.

If Altman and his team really do care deeply about the safe part of building safe AGI, he now has the opportunity to do the funniest, and also the best, possible thing.

Rather than navigating a conflict between ‘EA’ and ‘e/acc,’ or between worried and unworried, ‘doomer’ and (boomer?), he now has the credibility to say that it is time to make serious costly commitments and investments in the name of ensuring that safe AGI is actually safe.

Not because he was forced into it – surely we all know that any such secret promises the old board might have extracted are worth exactly nothing. Not to placate factions or board members. He can do it because he knows it is the right thing to do, and he now is in a position to do it without endangering his power.

That is the thing about attempting to align something more capable than you that will pursue power due to instrumental convergence. The early steps look the same whether or not you succeeded. You only find out at the end whether or not the result was compatible with humans.

Ilya Sutskever tried to put the breaks on AI development and remove what he saw at the time as a reckless CEO, from an organization explicitly dedicated to safety. Or at least, that’s the story everyone believed on the outside.

What happened? An avalanche of pressure from all sides. This despite no attempt to turn off any existing systems.

Ask yourself: What would happen if AI was integrated into the economy, or even was a useful tool everyone liked, and it suddenly became necessary to turn it off?

Never mind whether we could. Suppose we could, and also that we should. Would we?

Would anyone even dare try?

Chris Maddison: The wrath that @ilyasut is facing is just a prelude of the wrath that will be faced by anyone who tries to “unplug” an unaligned, supervaluable AI. This weekend has not been encouraging from an AI safety perspective.

David Rein: This is an extremely important and underrated point. Once AI systems are deeply integrated into the economy, there’s ~0% chance we will be able to just “turn them off”, even if they start acting against our interests.

Meta introduces Emu Video and Emu Edit. Edit means that if you get part of what you want, you can keep it and build upon it. Video is a few seconds of video. I have yet to see any useful applications of a few seconds of video that is essentially ‘things drift in a direction’ but someday, right?

Report of scam deepfake calls hitting right now, asking for bail money.

Distinct first hand report of that same scam, asking for bail money.

As a general rule, scammers are profoundly uncreative. The reason the Nigerian scam is still the Nigerian scam is that if you recognize the term ‘Nigerian prince’ as a scam then you were not about to fall for an ‘Angolan principle’ either.

So for now, while a code word is still a fine idea, you can get most of the way there with ‘any request for bail money or a suspiciously low-random-amount kidnapping is highly suspicious, probably a scam.’

An even easier, more robust rule suggests itself.

If a phone call leads to a request for money or financial information, assume until proven otherwise that this is a scam!

Good rule even without AI.

Paper suggests top freelancers are losing business due to ChatGPT. Overall demand drops for knowledge workers and also narrows gaps between them.

I would not presume this holds long term. The skill gap has narrowed, but also there is always demand for the best, although they do not find such an effect here. Mostly I would caution against generalizing too much from early impacts in quirky domains.

Vance Ginn at EconLib opposes the EO, gives the standard ‘they will never take our jobs’ speech warning that ‘red teaming’ and any other regulation will only slow down innovation, does not even bother dismissing existential risks.

Dave Orr is hiring for a new DeepMind alignment team he is joining to start. Post is light on details, including planned technical details.

Red teaming competition, goal is to find an embedded Trojan in otherwise aligned LLMs, a backdoor that lets the user do whatever they want. Submissions accepted until February 25.

Claude v2.1. 200k context window, specific prompt engineering techniques, half as many hallucinations (they say), system prompts and experimental tool use for calling arbitrary functions, private knowledge bases or browsing the web. It seems you use things like , or . You can also employ .

All seems highly incrementally useful.

GPQA, a new benchmark of 448 multiple choice science questions where experts often get them wrong and the answers are Google-proof.

ChatGPT Voice rolls out for all free users, amid all the turmoil, with Greg Brockman promoting it. This is presumably a strong cooperative sign, but it could also be a move to raise costs even further.

Microsoft to spend over $50 billion on data centers. Yikes indeed, sir.

Meta, while everyone is super distracted by OpenAI drama, breaks up its ‘Responsible AI’ team. To which we all said, Meta has a responsible AI team? I do not believe this is a team worth worrying about. Again, I expect the main effect of yelling when people disband such teams is that companies will avoid making such teams, or ensure they are illegible. Yes, they are trying to bury it, but I’m basically OK with letting them.

Jack Morris: now seems as good a time as ever to remind people that the biggest breakthroughs at OpenAI came from a previously unknown researcher [Alec Radord] with a bachelors degree from olin college of engineering.

Ethan Mollick points out that the OpenAI situation highlights the need to not enforce noncompete agreements in tech. I once sat out six months of Magic writing because I had a non-compete and my attempts to reach a win-win deal to difuse it were roundly rejected, and I keep my agreements. I do think advocates are too eager to ignore the cases where such agreements are necessary in some form, or at least net useful, so it is not as simple as all that.

New paper: Building the Epistemic Community of AI Safety. Don’t think there’s much here but included for completeness.

Will deep learning soon hit a wall? Gary Marcus says it is already hitting one based on this answer, at the end of the Sam Altman video I discuss in This Week in Audio, and writes a post suggesting Altman agrees.

Sam Altman (CEO of OpenAI): There are more breakthroughs required in order to get to AGI

Cambridge Student: “To get to AGI, can we just keep min maxing language models, or is there another breakthrough that we haven’t really found yet to get to AGI?”

Sam Altman: “We need another breakthrough. We can still push on large language models quite a lot, and we will do that. We can take the hill that we’re on and keep climbing it, and the peak of that is still pretty far away. But, within reason, I don’t think that doing that will (get us to) AGI. If (for example) super intelligence can’t discover novel physics I don’t think it’s a superintelligence. And teaching it to clone the behavior of humans and human text – I don’t think that’s going to get there.And so there’s this question which has been debated in the field for a long time: what do we have to do in addition to a language model to make a system that can go discover new physics?”

Gary Marcus: Translation: “deep learning is hitting a wall”

Rob Bensinger: What’s your rough probability on “In 2026, it will seem as though deep learning hit a non-transient wall at some point in 2023-2025, after the relatively impressive results from GPT-4?”

Michael Vassar: 85% by 2028. 65% in 2026. But economic impact will still be accelerating in 2028

AgiDoomerAnon: 25-30% maybe, it’s where most of my hope comes from 🙂

Negative Utilitarian: 80%.

Fanged Desire: 90%. Though at the same time it’s true that a *lotof things are going to be possible just by modifying the capabilities we have now in basic ways and making them applicable to different scenarios, so to a layman it may *looklike we’re still progressing at lightning speed.

CF: Lol, 0%.

Jason: 0%. There’s obvious ways forward. Adding short term memory for a start. Parallel streams of consciousness output that are compared and rated by another LLM.

Such strong disagreement. I opened up a market.

A good perspective:

Sarah Constantin: The good news is that you’d need an AI to do original science for the worst-case scenarios to occur, and it doesn’t look like LLMs are remotely close.

The bad news is that @sama apparently *wantsan AI physicist.

Not remotely close now, but I am not confident about how long until a lot closer.

Dwarkesh Patel asks why LLMs with so much knowledge don’t notice new correlations and discoveries, pretty much at all. Eliezer responds that humans are computers too, so this is unlikely to be a fundamental limitation, but we do not know how much more capacity would be required for this to happen under current architectures. Roon predicts better and more creative reasoning will solve it.

FTC is latest agency to give an absurd definition of AI.

Al includes, but is not limited to, machine-based systems that can, for a set of defined objectives, make predictions, recommendations, or decisions influencing real or virtual environments. Generative Al can be used to generate synthetic content including images, videos, audio, text, and other digital content that appear to be created by humans. Many companies now offer products and services using Al and generative Al, while others offer products and services that claim to detect content made by generative Al.

I get that a perfect legal definition of AI is hard, but this is broad enough to include essentially every worthwhile piece of software.

UK to not regulate AI at all ‘in the short term’ to avoid ‘stifling innovation.’ This could in theory be part of a sensible portfolio, where Sunak helps them lay groundwork for international cooperation and sensible regulation when we need it, while not getting in the way now. Or it could be a different way to describe the situation – Biden’s Executive Order also does not regulate AI in the short term in any meaningful way. What’s the difference? This could also reflect deep dysfunction. We will see.

Simeon attempts to explain one more time why not regulating foundation models, as in letting anyone who wants to create the most dangerous, capable and intelligent systems possible, won’t work. No, you can’t meaningfully ‘regulate applications,’ by then it is too late. Also he notes that to the extent Mistral (the most prominent voice advocating this path) has a track record on alignment, it is atrocious.

Corporate Europe on how Big Tech undermined the AI Act. Much to wince over on many fronts. It confirms the basic story of Big Tech lobbying hard against regulations with teeth, then Mistral and Aleph Alpha lobbying hard against those same regulations by claiming the regulations are a Big Tech conspiracy. Nice trick.

Politico Pro EU’s Vincent Manancourt: New: EU “crazy” to consider carve out for foundation models in AI act, ‘Godfather of AI’ Yoshua Bengio told me. He warned bloc risks “law of the jungle” for most advanced forms of the tech. [Story for pros here]

Control AI points out that Mistral’s lobbying champion is the former French tech minister for Macron, who has very much reversed his tune. I’ve heard other reports as well that this relationship has been central.

Italy seems to be with Germany and France on this. Others not so much, but that’s rather a big three, who want a regime with zero teeth whatsoever. Other officials walked out in response.

“This is a declaration of war,” a parliament official told Euractiv on condition of anonymity.

Max Tegmark and Yoshua Bengio indeed point out that this would be the worst possible thing.

Connor Dunlop of Euractiv argues regulating AI foundation models is crucial for innovation. I agree that the proposed regulations would assist, not hurt, Mistral and Aleph Alpha, the so-called would-be upstart ‘national champions.’ I do not think the linked post effectively makes that case.

One approach to dismissing any attempts to not die, or any form of regulation on AI, has always been to refer to any, and in many although not all cases I do mean any, restriction or regulation on AI as totalitarianism.

They want people to think of a surveillance state with secret police looking for rogue laptops and monitoring your every keystroke. Plus airstrikes.

What they are actually talking about is taking frontier models, the creation and deployment of new entities potentially smarter and more capable than humans, and applying a normal regulatory regime.

It is time to treat such talk the same way we treat people who get arrested because they deny the constitutional right of the United States Government to collect income taxes.

As in: I am not saying that taxation is not theft, but seriously, get a grip, stop it.

As an excellent concrete example with 500k+ views up top, I am highly disappointed in this disingenuous thread from Brian Chau.

Brian Chau: Did you guys know there’s 24-author paper by EAs, for EAs, about how Totalitarianism is absolutely necessary to prevent AI from killing everyone?

What is this ‘totalitarianism’ as it applies here, as Chau explains it?

They predictably call for exactly the kind of regulatory capture most convenient to OpenAI, Deepmind, and other large players.

[From Paper, quoted by Chau]: blocks for the regulation of frontier models are needed: (1) standard-setting processes to identify appropriate requirements for frontier AI developers, (2) registration and reporting requirements to provide regulators with visibility into frontier AI development processes, and (3) mechanisms to ensure compliance with safety standards for the development and deployment of frontier AI models.

In other words, the ‘totalitarianism’ is:

1. Standards for frontier models.

2. Registration and reporting of training runs.

3. Mechanisms allowing enforcement of the safety rules.

This is not totalitarianism. This is a completely ordinary regulatory regime.

(I checked with Claude, which was about as kind to Chau’s claims as I was.)

If you want to argue that standard regulatory interventions tend to favor insiders and large players over time? I will agree. We could then work together against that on 90% (98%?) of issues, while looking for the best solution available in the AI space.

Or you can make your true position flat out text?

Here’s how the paper describes three ways their regulatory regime might fail.

The Unexpected Capabilities Problem. Dangerous capabilities can arise unpredictably and undetected, both during development and after deployment.

The Deployment Safety Problem. Preventing deployed AI models from causing harm is a continually evolving challenge.

The Proliferation Problem. Frontier AI models can proliferate rapidly, making accountability difficult.

Here’s how Brian Chau describes this:

Brain Chau: They lay out three obstacles to their plans. If you pause for a moment and read the lines carefully, you will realize they are all synonyms for freedom.

If you want to take the full anarchist position, go ahead. But own it.

In addition to the above mischaracterization, he then ‘rebuts’ the four claims of harm. Here is his reason biological threats don’t matter.

Re 1: the limiting factor of designing new biological weapons is equipment, safety, and not killing yourself with them. No clue why this obviously false talking point is trodded out by EAs so often.

This seems to be a claim that no amount of expertise or intelligence enables the importantly easier creation of dangerous biological pandemic agents? Not only the claim, but the assertion that this is so obviously false that it is crazy to suggest?

He says repeatedly ‘show me the real examples,’ dismissing the danger of anything not yet dangerous. That is not how any of this works.

Sam Altman at Cambridge Union Society on November 1 accepting an award and answering questions. Opening speech is highly skippable. The risks and promise of AI are both front and center throughout, I provide a summary that suffices, except you will also want to watch that last question, where Sam says more breakthroughs are needed to get to AGI.

The existential risk protest interruption is at about 17: 00 and is quite brief.

At 19: 30 he describes OpenAI as tool builders. Notice every time someone assumes that sufficiently capable AIs would long remain our tools.

Right after that he says that young programmers are now outperforming older ones due to greater familiarity with AI tools.

22: 00 Sam says he learned two things in school: How to learn new things, and how to think of new things he hadn’t heard elsewhere. Learning how to learn was all the value, the content was worthless.

25: 30 Sam responds to the protest, saying that things decay without progress, the benefits can’t be ignored, there needs to be a way forward. Except of course no, there is no reason why there needs to be a way forward. Maybe there is. Maybe there isn’t.

34: 00 Sam’s primary concern remains misuse.

47: 00 Sam discusses open source, warns of potential to make an irreversible mistake. Calls immediately open sourcing any model trained ‘insanely reckless’ but says open source has a place.

From two weeks ago, I happened to listen to this right before Friday’s events: OpenAI co-founder Ilya Sutskever on No Priors. p(doom) went up as I heard him express the importance of ensuring future AIs have, his term, ‘warm feelings’ towards us, or needing it to ‘be prosocial’ or ‘humanity loving.’ That is not how I believe any of this works. He’s working alongside Leike on Superalignment, and he is still saying that, and I do not understand how or why. But assuming they can continue to work together after this and still have OpenAI’s support, they can hopefully learn and adjust as they go. It is also very possible that Ilya is speaking loosely here and his actual detailed beliefs are much better and more precise.

What is very clear here is Ilya’s sincerity and genuine concern. I wish him all the best.

Yoshua Bengio talk, towards AI safety that improves with more compute. I have not watched yet.

Davidad brief thread compares his approach to those of Bengio and Tegmark.

Some basic truth well said.

David Sacks: I’m all in favor of accelerating technological progress, but there is something unsettling about the way OpenAI explicitly declares its mission to be the creation of AGI.

AI is a wonderful tool for the betterment of humanity; AGI is a potential successor species.

By the way, I doubt OpenAI would be subject to so many attacks from the safety movement if it wasn’t constantly declaring its outright intention to create AGI.

To the extent the mission produces extra motivation for the team to ship good products, it’s a positive. To the extent it might actually succeed, it’s a reason for concern. Since it’s hard to assess the likelihood or risk of AGI, most investors just think about the former.

How true is this?

Staff Engineer: If you don’t believe in existential risk from artificial super intelligence. then you don’t believe in artificial super intelligence. You’re just looking at something that isn’t scary so you don’t have to think about the thing that is.

Jeffrey Ladish: Agree with the first sentence but not the second. Many people are just choosing to look away, but some genuinely think ASI is extremely hard / very far away / impossible. I think that’s wrong, but it doesn’t seem like a crazy thing to believe.

Eliezer Yudkowsky notes that people constantly gaslight us saying ‘creating smarter than human AIs would not be a threat to humanity’s survival’ and gets gaslit by most of the comments, including the gaslighting that ‘those who warn of AI’s existential risk deny its upsides’ and ‘those who warn of AI’s existential risk do not say over and over that not ever building it would be a tragedy.’

Your periodic reminder and reference point: AI has huge upside even today. Future more capable AI has transformational insanely great upside, if we can keep control of the future, not get killed and otherwise choose wisely. Never building it would be a great tragedy. However, it would not be as big a tragedy as everyone dying, so if those are the choices then don’t fing build it.

On Liars and Lying, a perspective on such questions very different from my own.

Your periodic reminder that ‘doomer’ is mostly a label used either as shorthand, or as a kudgel of those who want to ridicule the idea that AI could be existentially dangerous. Whereas those who do worry have widely varying opinions.

Eliezer Yudkowsky: Disturbing tendency to conflate anyone who believes in any kind of AGI risk as a “doomer”. If that’s the definition, Sam Altman is a doomer. Ilya Sutskever is a doomer. Helen Toner is a doomer. Shane Legg is a doomer. I am a doomer. Guess what? We are importantly different doomers. None of their opinions are my own, nor their plans, nor their choices. Right or wrong they are not mine and do not proceed from my own reasons.

Your periodic reminder that a for-profit business has in practice a strong economic incentive to not kill all of its customers, but only to the extent that would leave other humans alive but not leave it as many customers. If everyone is dead, the company makes no money, but no one cares or is punished for it.

Packy McCormick (a16z): The cool thing about for-profit AI, from an alignment perspective, is that it gives you a strong economic incentive to not kill all of your customers.

Rob Bensinger: If you die in all the scenarios where your customers die, then I don’t see how for-profit improves your long-term incentives. “I and all my loved ones and the entire planet die” is just as bad as “I and all my loved ones and the entire planet die, and a bunch of my customers.”

A for-profit structure may or may not be useful for other reasons, but I don’t think it’s specifically useful because of the “all my customers suddenly die (at the same time the reset of humanity does) scenario”, which is the main scenario to worry about.

Do the events of the past week doom all nuance even more than usual?

Haseeb: This weekend we all witnessed how a culture war is born.

E/accs now have their original sin they can point back to. This will become the new thing that people feel compelled to take a side on–e/acc vs decel–and nuance or middle ground will be punished.

Such claims are constant. Everything that happens nuance is presumed dead. Also every time most people with the ‘e/acc’ label speak nuance is announced dead. Them I do not worry about, they are a lost cause. The question is whether a lot of otherwise reasonable people will follow. Too soon to tell.

Not the argument you want to be making, but…

Misha Gurevich: People who think EA X-risk worries about AI are a destructive ideology: imagine the kind of ideologies artificial intelligences are gonna have.

Following up on the deceptive alignment paper from last week:

Robert Wiblin: In my mind the probability that normal AI reinforcement will produce ‘deceptive alignment’ is like… 30%. So extremely worth working on, and it’s crazy we don’t know. But it might turn out to be a red herring. What’s the best evidence/argument that actually it’s <1% or >90%?

[bunch of mostly terrible arguments in various directions in reply]

I notice a key difference here. Wiblin is saying 30% to deceptive alignment. Last week’s estimate was similar (25%) but it was conditional on the AI already having goals and being situationally aware. Conditional on all that, I am confused how such behavior could fail to arise. Unconditionally is far less clear.

I still expect to almost always see something that is effectively ‘deceptive alignment.’

The AI system will figure out to do that which, within the training environment, best convinces us it is aligned. That’s the whole idea with such techniques. I do not assume that the AI will then go ‘aha, fooled you, now that I am not being trained or tested I can stop pretending.’ I don’t rule that out, but my default scenario is that the thing we got it to do fails to generalize out of distribution the way we expected. That it is sensitive to details and context in difficult to anticipate ways that do not match what we want in both directions. That it does not generalize the ways we hope for.

We discover that we did not, after all, know how to specify what we wanted, in a way that resulted in things turning out well.

Is that ‘deceptive alignment’? You tell me.

Here’s Eliezer’s response:

Eliezer Yudkowsky: Are you imagining that it won’t be smart enough to do that? Or that deception will genuinely not be in its interests because it gets just as much of what it wants with humans believing true things as the literally optimal state of affairs? Or that someone solved soft optimization? How do you imagine the weird, special circumstances where this doesn’t happen? Remember that if MIRI is worried about a scenario, that means we think it’s a convergent endpoint and not some specific pathway; if you think we’re trying to predict a hard-to-predict special case, then you’ve misunderstood a central argument.

Robert Wiblin: Joe’s paper does a better job than me of laying out ways it might or might not happen. But ‘not being smart enough’ isn’t an important reason.

‘Not trained to be a global optimizer’ is one vision.

Another is that the reinforcement for doing things we like and not things we don’t like (with some common-sense adjustments to how the feedback works suggested by alignment folks) evolves models to basically do what we want and share our aversions, maybe because that’s the simplest / most efficient / most parsimonious way to get reward during training. The wedge between what we want and what we reward isn’t large enough to generate lots of scheming behavior, because scheming isn’t the best way to turn compute into reward in training setups.

I am so completely confused by Wiblin’s position here, especially that last sentence. Why would ‘scheming’ not be the best way to turn compute into rewards? Why would a completely honest, consistent, straightforward approach be the most rewarded one, given how humans decide how to reward things? I don’t get it.

Eliezer Yudkowsky offers what for him counts as high praise.

Eliezer Yudkowsky (QTing what follows here): This seems a very weak test of the ability of dumber judges to extract truth from smarter debaters, but the methodology could be adapted to tougher tests. Increasingly tough versions of this are a good candidate for standard evals.

Julian Michael: As AIs improve at persuasion & argumentation, how do we ensure that they help us seek truth vs. just sounding convincing? In human experiments, we validate debate as a truth-seeking process, showing that it may soon be needed for supervising AI. Paper here.

When a doctor gives a diagnosis, common advice is to get a second opinion to help evaluate whether to trust their judgment, because it’s too difficult to evaluate their diagnosis by yourself.

The idea (originally proposed by @geoffreyirving et al.) is that having equally-capable adversarial AIs critique each other’s answers will make it easier for non-expert judges to evaluate their truthfulness. But does this actually hold in practice?

We find for the first time on a realistic task that the answer is yes! We use NYU competitive debaters to stand in for future AI systems, having them debate reading comprehension questions where the judge *can’t see the passage(except for quotes revealed by the debaters).

We compare debate to a baseline we call *consultancy*, where the judge interacts with a single expert that has a 50% chance of lying. We use this to explicitly elicit dishonest behavior that may implicitly arise in methods like RLHF.

We find that judges are significantly more accurate in debate than consultancy, AND debates are much more efficient, at two-thirds the length on average.

Furthermore, many of the errors we observe in debate seem fixable with more careful judging and stronger debaters. In a third of mistakes, judges end the debate prematurely, and in nearly half, honest debaters mistakenly missed key evidence that would have helped them win.

We don’t see a difference in accuracy or efficiency between debate and consultancy when using GPT-4 as a debater — yet. In particular, GPT-4 was not very skilled at deception, which may not remain the case for future powerful AI systems.

As we move from relatively unskilled AI systems to skilled humans, non-expert judge accuracy *improveswith debate, but *decreaseswith consultancy. This suggests that training AI systems to debate may be an important alternative to methods like RLHF as models improve.

In the paper, we lay out considerations on how to train AI debaters and open problems that need to be solved.

How optimistic should we be about this in the AI case where you are trying to do this to use outputs of models you cannot otherwise trust? I continue to assume this will break exactly when you need it to not break. It could have mundane utility in the period before that, but I always worry about things I assume are destined to break.

Thread with Nora Belrose and Eliezer Yudkowsky debating deceptive alignment. Nora bites the bullet and says GPT-4 scaled up in a naive way would not have such issues. Whereas I would say, that seems absurd, GPT-4 already has such problems. Nora takes the position that if your graders and feedback suck, your AI ends up believing false things the graders believe and not being so capable, but not in a highly dangerous way. I continue to be confused why one would expect that outcome.

Roon reminds us that people acting like idiots and making deeply stupid strategic power moves, only to lose to people better at power moves, has nothing to do with the need to ensure we do not die from AI.

Roon: throughout all this please remember a few things that will be critical for the future of mankind: – this coup had nothing to do with ai safety. Sama has been a global champion of safe agi dev – the creation of new life is fraught and no one must forget that for political reasons

Sorry if the second point is vague. I literally just mean don’t turn your back on x-risk just because of this remarkably stupid event

We need better words for stuff that matters. But yes.

Roon: if a group of people are building artificial life in the lab and don’t view it with near religious significance you should be really concerned.

Axios’ Jim VandeHei and Mike Allen may or may not be worried about existential risk here, but they say outright that ‘this awesome new power’ cannot be contained, ethics never triumphs over profits, never has, never will. So we will get whatever we get.

I say this is at best midwit meme territory. Sometimes, yes, ethics or love or the common good wins. We are not living in a full-on cyberpunk dystopia of unbridled capitalism. I am not saying it will be easy. It won’t be easy. It also is not impossible.

It is important to notice this non-sequitur will be with us, until very late in the game. There is always some metaphorical hard thing that looks easy.

Katherine Dee: Had an ex who worked on self-driving cars. He once said to me, “you can’t use self-check out or self-ticketing machines at the airport reliably. No AI overlords are coming.” I think about that a lot.

Eliezer Yudkowsky: This really is just a non-sequitur. Not all machines are one in their competence. The self-check-out machines can go on being bad indefinitely, right up until the $10-billion-dollar frontier research model inside the world’s leading AI lab starts self-improving.

What term would he prefer to use for the possibility?

Stewart Brand: Maybe this is the episode that makes the term “existential risk” as passé as it needs to be.

When someone tells you who they are. Believe them.

Roon (Sunday evening): i truly respect everyone involved [in the OpenAI situation].

Eliezer Yudkowsky: I respect that.

Anton: If one has actual principles, this is not possible.

And when you see them you shall call them by their true name.

Previously: OpenAI: Altman Returns, OpenAI: The Battle of the Board, OpenAI: Facts from a Weekend, additional coverage in AI#41.

We have new stories from The New York Times, from Time, from the Washington Post and from Business Insider.

All paint a picture consistent with the central story told in OpenAI: The Battle of the Board. They confirm key facts, especially Altman’s attempted removal of Toner from the board via deception. We also confirm that Altman promised to help with the transition when he was first fired, so we have at least one very clear cut case of Altman saying that which was not.

Much uncertainty remains, especially about the future, but past events are increasingly clear.

The stories also provide additional color and key details. This post is for those who want that, and to figure out what to think in light of the new details.

The most important new details are that NYT says that the board proposed and was gung ho on Brett Taylor, and says D’Angelo suggested Summers and grilled Summers together with Altman before they both agreed to him as the third board member. And that the new board is remaining quiet while it investigates, echoing the old board, and in defiance of the Altman camp and its wish to quickly clear his name.

The New York Times finally gives its take on what happened, by Tripp Mickle, Mike Isaac, Karen Weise and the infamous Cade Metz (so treat all claims accordingly).

As with other mainstream news stories, the framing is that Sam Altman won, and this shows the tech elite and big money are ultimately in charge. I do not see that as an accurate description what happened or its implications, yet both the tech elite and its media opponents want it to be true and are trying to make it true through the magician’s trick of saying that it is true, because often power resides where people believe it resides.

I know that at least one author did read my explanations of events, and also I talked to a Times reporter not on the byline to help make everything clear, so they don’t have the excuse that no one told them. Didn’t ultimately matter.

Paul Graham is quoted as saying Altman is drawn to power more than money, as an explanation for why Altman would work on something that does not make him richer. I believe Graham on this, but also I think there are at least three damn good other reasons to do it, making the decision overdetermined.

1. If Altman wants to improve his own lived experience and those of his friends and loved ones, building safe AGI, or ensuring no one else builds unsafe AGI, is the most important thing for him to do. Altman already has all the money he will ever need for personal purposes, more would not much improve his life. His only option is to instead enrich the world, and ensure humanity flourishes and also doesn’t die. Indeed, notice the rest of his portfolio includes a lot of things like fusion power and transformational medical progress. Even if Altman only cares about himself, these are the things that make his life better – by making everyone’s life better.

2. Power and fame and prestige beget money. Altman does not have relevant amounts of equity in OpenAI, but he has used his position to raise money, to get good deal flow, and in general to be where the money resides. If Altman decided what he cared about was cash, he could easily turn this into cash. To be clear, I do not at all begrudge in general. I am merely not a fan of some particular projects, like ‘build a chip factory in the UAE.’

3. AGI is the sweetest, most interesting, most exciting challenge in the world. Also the most important. If you thought your contribution would increase the chance things went well, why would you want to be working on anything else?

Pretty much every version of Altman I can imagine would want to be doing this.

The key description of the safety issue is structured in a way that it is easy to come away thinking this was a concern of the outside board members, but both in reality and if you read the article carefully, this applies to the entire board (although we have some uncertainty about Brockman in particular):

They were united by a concern that A.I. could become more intelligent than humans.

Remember that this was and is the explicit goal of OpenAI, to safely create AI more intelligent than humans, also known as AGI. Altman signed the CAIS letter, although Brockman is not known to have done so. Altman has made the threat here very clear. Everyone involved understands the danger. Everyone is, to their credit, talking price.

The first piece of news is that we have at least one case in which we can be damn sure that Sam Altman lied to the board, in at least some important senses.

Shocked that he was being fired from a start-up he had helped found, Mr. Altman widened his eyes and then asked, “How can I help?” The board members urged him to support an interim chief executive. He assured them that he would.

Within hours, Mr. Altman changed his mind and declared war on OpenAI’s board.

I point this out because it is a common theory that Altman was a master of Exact Words and giving implications. That yes he was deceptive and misleading and played power games, but he was too smart to outright say that which was not.

So here he is, saying that which is not.

Did it matter? Maybe no. But maybe quite a lot, actually. This cooperation could have been a key factor driving the decision not to detail the issues with Altman, at least initially, when it would have worked. If Altman is going to cooperate, what he gets in return is the mission continues and also whatever he did gets left unspecified.

The article waffles on whether or not Altman actually did declare war on the board that night. The statement above says so. Then they share a narrative of others driving the revolt, including Airbnb’s CEO Brian Chesky, the executives and employees, with Altman only slowly deciding to fight back.

It can’t be both. Which is it?

I assume the topline is correct. That Altman was fighting back the whole time. And that despite being willing to explicitly say that up top, Altman’s people sufficiently sculpted the media narrative to make the rest sound like events unfolded in a very different way. It is an absolute master class in narrative sculpting and media manipulation. They should teach this in universities. Chef’s kiss.

We have confirmation that Altman was not ‘consistently candid’ about the project to build chips in the UAE:

In September, Mr. Altman met investors in the Middle East to discuss an A.I. chip project. The board was concerned that he wasn’t sharing all his plans with it, three people familiar with the matter said.

For many obvious reasons, this is an area where the board would want to be informed, and any reasonable person in Altman’s position would know this, and norms say that this means they should be informed. But not informing them would not by default strictly violate the rules, as long as Altman honestly answered questions when asked. Did he, and to what extent? We don’t know.

Now we get into some new material.

Dr. Sutskever … believed that Mr. Altman was bad-mouthing the board to OpenAI executives, two people with knowledge of the situation said. Other employees have also complained to the board about Mr. Altman’s behavior.

In October, Mr. Altman promoted another OpenAI researcher to the same level as Dr. Sutskever, who saw it as a slight. Dr. Sutskever told several board members that he might quit, two people with knowledge of the matter said. The board interpreted the move as an ultimatum to choose between him and Mr. Altman, the people said.

Dr. Sutskever’s lawyer said it was “categorically false” that he had threatened to quit.

Another conflict erupted in October when Ms. Toner published a paper…

This frames Sutskever as having been in favor of firing Altman for some time. If this is true, the board’s sense of urgency, and its unwillingness to take time to plan and get its ducks in a row, makes even less sense. If they had been discussing the issue for months, if Ilya had been not only onboard but enthusiastic for a month, I don’t get it.

The post then goes over the incident over Toner’s ignored academic paper, for which Toner agreed to apologize to keep the peace.

“I did not feel we’re on the same page on the damage of all this,” Altman wrote.

We’re definitely not. Toner and I are on the page that this was trivial and obviously so. Altman was presenting it as a major deal.

Now we get to the core issue.

Mr. Altman called other board members and said Ms. McCauley wanted Ms. Toner removed from the board, people with knowledge of the conversations said. When board members later asked Ms. McCauley if that was true, she said that was “absolutely false.”

“This significantly differs from Sam’s recollection of these conversations,” an OpenAI spokeswoman said, adding that the company was looking forward to an independent review of what transpired.

Time magazine gives this version:

Time: Altman told one board member that another believed Toner ought to be removed immediately, which was not true, according to two people familiar with the discussions. 

Whatever other reasons did or did not exist, if Altman did say that, my model of such things is that he needed to be fired and it was the board’s job to fire him. And the board really should have said so, rather than speaking in generalities.

Multiple witnesses are saying to NYT that he said it. Altman denies it.

It seems clear Altman did use private conversations with board members to give false impressions and drum up support for getting Toner off the board, thereby giving Altman board control, using the paper as an excuse. The dispute is whether Altman did it using Exact Words, or whether he lied. Altman called his attempt ‘ham fisted’ which I believe is power player code for ‘got caught lying’ but could also apply to ‘got caught technically-not-lying while implicitly lying my ass off.’

NYT does seem to be saying the board did step up their description a bit:

NYT: The board members said that Mr. Altman had lied to the board, but that they couldn’t elaborate for legal reasons.

Use of the word ‘lied’ is an escalation. And this is a clear confirmation of lawyers.

We also have confirmation of zero PR people, because we have Toner’s infamous line. I know the logic behind it but I still cannot believe that she actually said it out loud given the context, seriously WTF:

Jason Kwon, OpenAI’s chief strategy officer, accused the board of violating its fiduciary responsibilities. “It cannot be your duty to allow the company to die,” he said, according to two people with knowledge of the meeting.

Ms. Toner replied, “The destruction of the company could be consistent with the board’s mission.”

You say ‘We have no intention of doing any such thing. The company is perfectly capable of carrying on without Altman. We have every intention of continuing on OpenAI’s mission, led by the existing executive team. Altman promised to help with the transition in the board meeting. If he instead chooses to attempt to destroy OpenAI and its mission, that is his decision. It also proves he was incompatible with our mission and we needed to remove him.’

OpenAI’s executives insisted that the board resign that night or they would all leave. Mr. Brockman, 35, OpenAI’s president, had already quit.

The support gave Mr. Altman ammunition.

This sounds highly contingent.

Also the board had now already made an explicit bluff threatening to quit. The board called. The executives did not quit. Subsequent such threats become far less credible.

Skipping ahead a bit, they still tried this a second time.

By Nov. 19 [with the Microsoft offer in hand], Mr. Altman was so confident that he would be reappointed chief executive that he and his allies gave the board a deadline: Resign by 10 a.m. or everyone would leave.

Pro negotiation tip: Do not quickly pull this trick a second time once your first bluff gets called. It will not work. That is why you do not rush out the first bluff, and instead wait until your position is stronger.

Of course the board called the second bluff, appointing Emmett Shear.

The next piece of good information came before that deadline was set, which is that Bret Taylor was actually seen as a fair arbiter approved by both sides rather than being seen as in the Altman camp.

Yet even as the board considered bringing Mr. Altman back, it wanted concessions. That included bringing on new members who could control Mr. Altman. The board encouraged the addition of Bret Taylor, Twitter’s former chairman, who quickly won everyone’s approval and agreed to help the parties negotiate.

But also note that in this telling, it was the board that wanted concessions and in particular new board members rather than Altman. That directly contradicts other reports and does not make sense, unless you read it as ‘contingent on the old board agreeing to resign, they wanted concessions.’ As in, the board was going to hand over its control of OpenAI, and they wanted the concession of ‘we agree on who we give it to, and what those people agree will happen.’ At best, I find this framing bizarre.

Larry Summers was a suggestion of D’Angelo, in some key original reporting:

To break the impasse, Mr. D’Angelo and Mr. Altman talked the next day. Mr. D’Angelo suggested former Treasury Secretary Lawrence H. Summers, a professor at Harvard, for the board. Mr. Altman liked the idea.

Mr. Summers, from his Boston-area home, spoke with Mr. D’Angelo, Mr. Altman, Mr. Nadella and others. Each probed him for his views on A.I. and management, while he asked about OpenAI’s tumult. He said he wanted to be sure that he could play the role of a broker.

Mr. Summers’s addition pushed Mr. Altman to abandon his demand for a board seat and agree to an independent investigation of his leadership and dismissal.

So both sides talked to Summers, and were satisfied with his answers.

This week, Mr. Altman and some of his advisers were still fuming. They wanted his name cleared.

“Do u have a plan B to stop the postulation about u being fired its not healthy and its not true!!!” Mr. Conway texted Mr. Altman.

Mr. Altman said he was working with OpenAI’s board: “They really want silence but i think important to address soon.”

Overall this all makes me bullish on the new board. We might be in a situation with, essentially, D’Angelo and two neutral arbiters, albeit ones with gravitas and business connections. They are not kowtowing to Altman. Altman’s camp continues to fume (and somehow texts from Conway to Altman about it are leaking to NYT, theere are not many places that can come from).

Gwern offers their summary here.

Time profiled Altman, calling him ‘CEO of the year,’ a title he definitely earned. I think this is the best very short description so far, nailing the game theory:

Meanwhile, the company’s employees and its board of directors faced off in “a gigantic game of chicken,” says a person familiar with the discussions.

Sources also note the side of Altman that seeks power, and is willing to be dishonest and manipulative in order to get it.

But four people who have worked with Altman over the years also say he could be slippery—and at times, misleading and deceptive. Two people familiar with the board’s proceedings say that Altman is skilled at manipulating people, and that he had repeatedly received feedback that he was sometimes dishonest in order to make people feel he agreed with them when he did not. These people saw this pattern as part of a broader attempt to consolidate power. “In a lot of ways, Sam is a really nice guy; he’s not an evil genius. It would be easier to tell this story if he was a terrible person,” says one of them. “He cares about the mission, he cares about other people, he cares about humanity. But there’s also a clear pattern, if you look at his behavior, of really seeking power in an extreme way.”

This is the first mainstream report that correctly identifies the outcome as unclear:

It’s not clear if Altman will have more power or less in his second stint as CEO.

In addition to his other good picks, we can add… Georgist land taxes? Woo-hoo!

Altman has advocated for a land-value tax—a classic Georgist policy—in recent meetings with world leaders, he says. 

That is the kind of signal no one ever fakes. There really is a lot to love.

Including his honesty. I don’t want to punish it, but also I want to leave this here.

“We definitely accelerated the race, for lack of a more nuanced phrase,” Altman says. 

Time describes the board’s initial outreach to Altman this way:

Altman characterizes it as a request for him to come back. “I went through a range of emotions. I first was defiant,” he says. “But then, pretty quickly, there was a sense of duty and obligation, and wanting to preserve this thing I cared about so much.” The sources close to the board describe the outreach differently, casting it as an attempt to talk through ways to stabilize the company before it fell apart.

I am not saying we know for sure that this is another case of Altman lying (to Time rather than the board, a much less serious matter), but his version of events does not compute. If the board was actively asking for Altman to outright return, I do not buy that this was Altman’s reaction.

I could buy either half of Altman’s story: That Altman was asked to return, or that Altman was defiant to the board’s request and only did it out of duty and obligation (because the board was initially requesting something else.) I don’t buy both at once. It is entirely inconsistent with Paul Graham’s assessment of his character.

The WaPo piece says that in the fall the board was approached by a small number of senior leaders at OpenAI, with concerns about Altman. In this telling, the board thought OpenAI stood to lose key leaders due to what they saw as Altman’s toxicity.

Now back at the helm of OpenAI, Altman may find that the company is less united than the waves of heart emojis that greeted his return on social media might suggest.

That is always true. No large group is ever fully united, no matter what emojis says.

There are few concrete details. What details are offered sound like ordinary things that happen at a company. What is and is not abusive, in such a high-pressure and competitive environment, is in the eye of the beholder and highly context dependent. What is described here could reflect abuse, or it could reflect nothing of concern. What is concerning is that employees found it concerning enough to go to the board.

Beyond the one concrete detail of mangers going to the board with such complaints, this did not teach us much. It seems like those concerns helped confirm the board’s model of Altman’s behavior, and helped justify the decision on the margin.

Business Insider says that OpenAI employees really, really did not want to go to work at Microsoft. I wouldn’t either. The employees might have largely still seen it as the least bad alternative under some circumstances, if Altman didn’t want to start a new company. And remember, the letter said they ‘might’ do it, not that they all definitely would.

AI Safety Memes offers the following quotes:

“[The letter] was an audacious bluff and most staffers had no real interest in working for Microsoft.”

“Many OpenAI employees “felt pressured” to sign the open letter.”

“Another OpenAI employee openly laughed at the idea that Microsoft would have paid departing staffers for the equity they would have lost by following Altman.” “It was sort of a bluff that ultimately worked.”

“The letter itself was drafted by a group of longtime staffers who have the most clout and money at stake with years of industry standing and equity built up, as well as higher pay. They began calling other staffers late on Sunday night, urging them to sign, the employee explained.”

Despite nearly everyone on staff signing up to follow Altman out the door, “No one wanted to go to Microsoft.” This person called the company “the biggest and slowest” of all the major tech companies.

“The bureaucracy of something as big as Microsoft is soul crushing.”

“Even though we have a partnership with Microsoft, internally, we have no respect for their talent bar,” the current OpenAI employee told BI. “It rubbed people the wrong way to entertain being managed by them.”

Beyond the culture clash between the two companies, there was another important factor at play for OpenAI employees: money. Lots of it was set to disappear before their eyes if OpenAI were to suddenly collapse under a mass exodus of staff.

“Sam Altman is not the best CEO, but millions and millions of dollars and equity are at stake,” the current OpenAI employee said.

Microsoft agreed to hire all OpenAI employees at their same level of compensation, but this was only a verbal agreement in the heat of the moment.

A scheduled tender offer, which was about to let employees sell their existing vested equity to outside investors, would have been canceled. All that equity would have been worth “nothing,” this employee said.

The former OpenAI employee estimated that, of the hundreds of people who signed the letter saying they would leave, “probably 70% of the folks on that list were like, ‘Hey, can we, you know, have this tender go through?'”

Some Microsoft employees, meanwhile, were furious that the company promised to match salaries for hundreds of OpenAI employees.”

Roon responds that this was not accurate:

Roon: not to longpost, and I can only speak for myself, but this is a very inaccurate representation of the mood from an employee perspective.

– “employees felt pressured” -> at some point hundreds of us were in a backyard learning about the petition. people were so upset at the insanity of the board’s decisions that they were immediately fired up to sign this thing. the google doc literally broke from the level of concurrency of people all trying to sign at once. I recall many having intelligent nuanced conversations about the petition, the wording thereof, and in the end coming to the conclusion that it was the only path forward. Half the company had signed between the hours of 2 and 3am. That’s not something that can be accomplished by peer pressure.

– “it was about the money” -> at the time it sounded like signing the petition meant leaving all openai equity and starting fresh. We’re not idiots, everybody knows that the terms at newco would be up in the air at best, with a lot of bargaining chips on Microsoft’s side. People signed the petition because it was the right thing to do. You simply cannot work at the gutted husk of a company whose ultimate leadership you don’t respect.

– “no one wanted to go to Microsoft” -> you’d have to be out of your mind to prefer starting new on models and code and products being controlled by someone else rather than building in the company specifically designed to be the vehicle for safe AGI. It has nothing to do with the Microsoft talent bar or bureaucracy or brand. Not sure why some idiot leaker provocateur would frame it this way. Microsoft has been quite successful at acquiring companies under bespoke governance structures and letting them do their own thing (GitHub, LinkedIn). Even Microsoft’s own preferred outcome was continuity of OpenAI per the New Yorker article. I still bet if the board hadn’t changed their mind the company would have mostly reconstituted itself at Microsoft.

I trust that Roon is giving his honest recollection of his experience here. I also believe the two stories are more compatible than he realizes.

The employees wanted Altman back, or barring that a board and CEO that could trust, without which they would leave, but they mostly wanted OpenAI intact, and ideally to get paid, and were furious with the board. They didn’t want to go to Microsoft, we will never know how many would have actually done it versus stayed versus gone elsewhere or founded new companies, or how long the board had before that time bomb went off in earnest. My guess is that a lot of employees go to Microsoft if Altman stays there, but a lot also choose other paths.

My guess is that both the majority of employees enthusiastically signed the letter, and also those who didn’t want to sign felt pressured to do so anyway and this got a bunch of the later signatures onboard. I know I would have felt pressured even if no one applied any pressure intentionally.

Wei Dai sees the employee actions at OpenAI and the signing of the petition as a kind of OpenAI cultural revolution that he did not think was possible at a place like that, and sees it as a huge negative update. I was less surprised, and also read less into the letter. There was good reason, from their perspective, to be outraged and demand Altman’s return. There was also good reason to sign the letter even if an individual employee did not support Altman – to hold the company together, and for internal political reasons even if no direct pressure was applied. Again, the letter said ‘may’ leave, so it did not commit you to anything.

I will continue to link people to The Battle of the Board, which I believe remains the definitive synthesis of events. We now have additional detail supporting and fleshing out that narrative, but they do not alter the central story.

I am sure I will continue to often have a weekly section on developments, but hopefully things will slow down from here.

As I wrote previously, we now await the board’s investigation, and the composition of the new board. If the new board has a clear majority with a strong commitment to existential safety and the mission, and has the gravitas and experience necessary to do the job of the board, that would be a very good outcome, and if he did not do anything to render it impossible I would be happy to see Altman stay under such supervision.

If that proves not possible after an investigation, we will see who we get instead, I worry it will not be better but I also expect the company to then hold together, in a way it would not have if the board had not compromised, given how things had gone.

If the board instead ends up effectively captured by business interests and those who do not care about safety or OpenAI’s stated mission, that would be a catastrophe, whether or not Altman is retained.

If Altman ends up with effective board control and has free reign, then that is a highly worrisome outcome, and we get to find out to what extent Altman is truly aligned, wise and capable of resisting certain aspects his nature, versus the temptation to build and scale and seek power. It could end up fine, or be disastrous.

As of this morning, the new board is in place and everything else at OpenAI is otherwise officially back to the way it was before.

Events seem to have gone as expected. If you have read my previous two posts on the OpenAI situation, nothing here should surprise you.

Still seems worthwhile to gather the postscripts, official statements and reactions into their own post for future ease of reference.

What will the ultimate result be? We likely only find that out gradually over time, as we await both the investigation and the composition and behaviors of the new board.

I do not believe Qplayed a substantive role in events, so it is not included here. I also do not include discussion here of how good or bad Altman has been for safety.

Here is the official OpenAI statement from Sam Altman. He was magnanimous towards all, the classy and also smart move no matter the underlying facts. As he has throughout, he has let others spread hostility, work the press narrative and shape public reaction, while he himself almost entirely offers positivity and praise. Smart.

Before getting to what comes next, I’d like to share some thanks.

I love and respect Ilya, I think he’s a guiding light of the field and a gem of a human being. I harbor zero ill will towards him. While Ilya will no longer serve on the board, we hope to continue our working relationship and are discussing how he can continue his work at OpenAI.

I am grateful to Adam, Tasha, and Helen for working with us to come to this solution that best serves the mission. I’m excited to continue to work with Adam and am sincerely thankful to Helen and Tasha for investing a huge amount of effort in this process.

Thank you also to Emmett who had a key and constructive role in helping us reach this outcome. Emmett’s dedication to AI safety and balancing stakeholders’ interests was clear.

Mira did an amazing job throughout all of this, serving the mission, the team, and the company selflessly throughout. She is an incredible leader and OpenAI would not be OpenAI without her. Thank you.

Greg and I are partners in running this company. We have never quite figured out how to communicate that on the org chart, but we will. In the meantime, I just wanted to make it clear. Thank you for everything you have done since the very beginning, and for how you handled things from the moment this started and over the last week.

The leadership team–Mira, Brad, Jason, Che, Hannah, Diane, Anna, Bob, Srinivas, Matt, Lilian, Miles, Jan, Wojciech, John, Jonathan, Pat, and many more–is clearly ready to run the company without me. They say one way to evaluate a CEO is how you pick and train your potential successors; on that metric I am doing far better than I realized. It’s clear to me that the company is in great hands, and I hope this is abundantly clear to everyone. Thank you all.

Let that last paragraph sink in. The leadership team ex-Greg is clearly ready to run the company without Altman.

That means that whatever caused the board to fire Altman, whether or not Altman forced the board’s hand to varying degrees, if everyone involved had chosen to continue without Altman then OpenAI would have been fine. We can choose to believe or not believe Altman’s claims in his Verge interview that he only considered returning after the board called him on Saturday, and we can speculate on what Altman otherwise did behind the scenes during that time. We don’t know. We can of course guess, but we do not know.

He then talks about his priorities.

So what’s next?

We have three immediate priorities.

Advancing our research plan and further investing in our full-stack safety efforts, which have always been critical to our work. Our research roadmap is clear; this was a wonderfully focusing time. I share the excitement you all feel; we will turn this crisis into an opportunity! I’ll work with Mira on this.

Continuing to improve and deploy our products and serve our customers. It’s important that people get to experience the benefits and promise of AI, and have the opportunity to shape it. We continue to believe that great products are the best way to do this. I’ll work with Brad, Jason and Anna to ensure our unwavering commitment to users, customers, partners and governments around the world is clear.

Bret, Larry, and Adam will be working very hard on the extremely important task of building out a board of diverse perspectives, improving our governance structure and overseeing an independent review of recent events. I look forward to working closely with them on these crucial steps so everyone can be confident in the stability of OpenAI. 

I am so looking forward to finishing the job of building beneficial AGI with you all—best team in the world, best mission in the world.

Research, then product, then board. Such statements cannot be relied upon, but this was as good as such a statement can be. We must keep watch and see if such promises are kept. What will the new board look like? Will there indeed be a robust independent investigation into what happened? Will Ilya and Jan Leike be given the resources and support they need for OpenAI’s safety efforts?

Altman gave an interview to The Verge. Like the board, he (I believe wisely and honorably) sidesteps all questions about what caused the fight with the board and looks forward to the inquiry. In Altman’s telling, it was not his idea to come back, instead he got a call Saturday morning from some of the board asking him about potentially coming back.

He says he is not focused on getting back on the board, that is not his focus, but that the governance structure clearly has a problem that will take a while to fix.

Q: What does “improving our governance structure” mean? Is the nonprofit holding company structure going to change?

Altman: It’s a better question for the board members, but also not right now. The honest answer is they need time and we will support them in this to really go off and think about it. Clearly our governance structure had a problem. And the best way to fix that problem is gonna take a while. And I totally get why people want an answer right now. But I also think it’s totally unreasonable to expect it.

…

Oh, just because designing a really good governance structure, especially for such an impactful technology is not a one week question. It’s gonna take a real amount of time for people to think through this, to debate, to get outside perspectives, for pressure testing. That just takes a while.

Yes. It is good to see this highly reasonable timeline and expectations setting, as opposed to the previous tactics involving artificial deadlines and crises.

Mutari confirms in the interview that OpenAI’s safety approach is not changing, that this had nothing to do with safety.

Altman also made a good statement about Adam D’Angelo’s potential conflicts of interest, saying he actively wants customer representation on the board and is excited to work with him again. Altman also spent several hours with D’Angelo.

We also have the statement from Bret Taylor. We know little about him, so reading his first official statement carefully seems wise.

On behalf of the OpenAI Board, I want to express our gratitude to the entire OpenAI community, especially all the OpenAI employees, who came together to help find a path forward for the company over the past week. Your efforts helped enable this incredible organization to continue to serve its mission to ensure that artificial general intelligence benefits all of humanity. We are thrilled that Sam, Mira and Greg are back together leading the company and driving it forward. We look forward to working with them and all of you. 

As a Board, we are focused on strengthening OpenAI’s corporate governance. Here’s how we plan to do it:

• We will build a qualified, diverse Board of exceptional individuals whose collective experience represents the breadth of OpenAI’s mission – from technology to safety to policy. We are pleased that this Board will include a non-voting observer for Microsoft.

• We will further stabilize the OpenAI organization so that we can continue to serve our mission.  This will include convening an independent committee of the Board to oversee a review of the recent events.

• We will enhance the governance structure of OpenAI so that all stakeholders – users, customers, employees, partners, and community members – can trust that OpenAI will continue to thrive.

OpenAI is a more important institution than ever before. ChatGPT has made artificial intelligence a part of daily life for hundreds of millions of people. Its popularity has made AI – its benefits and its risks – central to virtually every conversation about the future of governments, business, and society.

We understand the gravity of these discussions and the central role of OpenAI in the development and safety of these awe-inspiring new technologies. Each of you plays a critical part in ensuring that we effectively meet these challenges.  We are committed to listening and learning from you, and I hope to speak with you all very soon.

We are grateful to be a part of OpenAI, and excited to work with all of you.

Mostly this is Brad Taylor properly playing the role of chairman of the board, which tells us little other than that he knows the role well, which we already knew.

Microsoft will get only an observer on the board, other investors presumably will not get seats either. That is good news, matching reporting from The Information.

What does ‘enhance the governance structure’ mean here? We do not know. It could be exactly what we need, it could be a rubber stamp, it could be anything else. We do not know what the central result will be.

The statement on a review of recent events here is weaker than I would like. It raises the probability that the new board does not get or share a true explanation.

He mentions safety multiple times. Based on what I know about Taylor, my guess is he is unfamiliar with such questions, and does not actually know what that means in context, or what the stakes truly are. Not that he is dismissive or skeptical, rather that he is encountering all this for the first time.

Here is the announcement via Twitter from board member Larry Summers, which raises the bar in having exactly zero content. So we still know very little here.

Larry Summers: I am excited and honored to have just been named as an independent director of @OpenAI. I look forward to working with board colleagues and the OpenAI team to advance OpenAI’s extraordinarily important mission.

First steps, as outlined by Bret and Sam in their messages, include building out an exceptional board, enhancing governance procedures and supporting the remarkable OpenAI community.

Here is Helen Toner’s full Twitter statement upon resigning from the board.

Helen Toner (11/29): Today, I officially resigned from the OpenAI board. Thank you to the many friends, colleagues, and supporters who have said publicly & privately that they know our decisions have always been driven by our commitment to OpenAI’s mission.

Much has been written about the last week or two; much more will surely be said. For now, the incoming board has announced it will supervise a full independent review to determine the best next steps.

To be clear: our decision was about the board’s ability to effectively supervise the company, which was our role and responsibility. Though there has been speculation, we were not motivated by a desire to slow down OpenAI’s work.

When I joined OpenAI’s board in 2021, it was already clear to me and many around me that this was a special organization that would do big things. It has been an enormous honor to be part of the organization as the rest of the world has realized the same thing.

I have enormous respect for the OpenAI team, and wish them and the incoming board of Adam, Bret and Larry all the best. I’ll be continuing my work focused on AI policy, safety, and security, so I know our paths will cross many times in the coming years.

Many outraged people continue to demand clarity on why the board fired Altman. I believe that most of them are thrilled that Toner and others continue not to share the details, and are allowing the situation outside the board to return to the status quo ante.

There will supposedly be an independent investigation. Until then, I believe we have a relatively clear picture of what happened. Toner’s statement hints at some additional details.

Roon gets it. The board needs to keep its big red button going forward, but still must account for its actions if it wants that button to stick.

Roon: The board has a red button but also must explain why its decisions benefit humanity. If it fails to do so then it will face an employee, customer, partner revolt. OpenAI currently creates a massive amount of value for humanity and by default should be defended tooth and nail. The for-profit would not have been able to unanimously move elsewhere if there was even a modicum of respect or good reasoning given.

The danger is that if we are not careful, we will learn the wrong lessons.

Toby Ord: The last few days exploded the myth that Sam Altman’s incredible power faces any accountability. He tells us we shouldn’t trust him, but we now know the board *can’tfire him. I think that’s important.

Rob Bensinger: We didn’t learn “they can’t fire him”. We did learn that the organization’s staff has enough faith in Sam that the staff won’t go along with the board’s wishes absent some good supporting arguments from the board. (Whether they’d have acceded to good arguments is untested.)

…

I just want us to be clear that the update about the board’s current power shouldn’t be a huge one, because it’s possible that staff would have accepted the board’s decision in this case if the board had better explained its reasoning and the reasoning had seemed stronger.

Quite so. From our perspective, the board botched its execution and its members made relatively easy rhetorical targets. That is true even if the board had good reasons for doing so. If the board had not botched its execution and had more gravitas? I think things go differently.

If after an investigation, Summers, D’Angelo and Taylor all decide to fire Altman again (note that I very much do not expect this, but if they did decide to do it), I assure you they will handle this very differently, and I would predict a very different outcome.

One of the best things about Sam Altman is his frankness that we should not trust him. Most untrustworthy people say the other thing. Same thing with Altman’s often very good statements about existential risk and the need for safety. When people bring clarity and are being helpful, we should strive to reward that, not hold it against them.

I also agree with Andrew Critch here, that it was good and right for the board to pull the plug on a false signal of supervision. If the CEO makes the board unable to supervise them, or otherwise moves against the board, then it is the duty of the board to bring things to a head, even if there are no other issues present.

Good background, potentially influential in the thinking of several board members including Helen Toner: Former OpenAI board member Holden Karnofsky’s old explanation of why and exactly how Nonprofit Boards are Weird, and how best to handle it.

Eliezer Yudkowsky proposes Paul Graham for the board of OpenAI. I see the argument, especially because Graham clearly cares a lot about his kids. My worries are that he would be too steerable by Altman, and he would be too inclined to view OpenAI as essentially a traditional business, and let that overrule other questions even if he knew it shouldn’t.

If he was counted as an Altman ally, as he presumably should, then he’s great. On top of the benefits to OpenAI, it would provide valuable insider information to Graham. Eliezer clarifies that his motivation is that he gives Graham a good chance of figuring out a true thing when it matters, which also sounds right.

Emmett Shear also seems like a clearly great consensus pick.

One concern is that the optics of the board matter. You would be highly unwise to choose a set of nine white guys. See Taylor’s statement about the need for diverse perspectives.

Matt Levine covers developments since Tuesday, especially that the valuation of OpenAI in its upcoming sale did not change, as private markets can stubbornly refuse to move their prices. In my model, private valuations like this are rather arbitrary, and based on what social story everyone involved can tell and everyone’s relative negotiating position, and what will generate the right momentum for the company, rather than a fair estimate of value. Also everyone involved is highly underinvested or overinvested, has no idea what fair value actually is, and mostly wants some form of social validation so they don’t feel too cheated on price. Thus, often investors get away with absurdly low prices, other times they get tricked into very high ones.

Gary Marcus says OpenAI was never worth $86 billion. I not only disagree, I would (oh boy is this not investment advice!) happily invest at $86 billion right now if I had that ability (which I don’t) and thought that was an ethical thing to do. Grok very much does not ‘replicate most of’ GPT-4, the model is instead holding up quite well considering how long they sat on it initially.

OpenAI is nothing without its people. That does not mean they lack all manner of secret sauce. In valuation terms I am bullish. Would the valuation have survived without Altman? No, but in the counterfactual scenario where Altman was stepping aside due to health issues with an orderly succession, I would definitely have thought $86 billion remained cheap.

A key question in all this is the extent to which the board’s mistake was that its optics were bad. So here is a great example of Paul Graham advocating for excellent principles.

Paul Graham: When people criticize an action on the grounds of the “optics,” they’re almost always full of shit. All they’re really saying is “What you did looks bad.” But if they phrased it that way, they’d have to answer the question “Was it actually bad, or not?”

If someone did something bad, you don’t need to talk about “optics.” And if they did something that seems bad but that you know isn’t, why are you criticizing it at all? You should instead be explaining why it’s not as bad as it seems.

Bad optics can cause bad things to happen. So can claims that the optics are bad, or worries that others will think the optics are bad, or claims that you are generally bad at optics.

You have two responses.

1. That means it had bad consequences, which means it was actually bad.

2. Nobly stand up for right actions over what would ‘look good.’

Consider the options in light of recent events. We all want it to be one way. Often it is the other way.