Gemini

Millions turn to AI chatbots for spiritual guidance and confession

AI, AI assistants, AI behavior, AI sycophancy, Biz & IT, chatbots, chatgpt, Gemini, generative ai, Google, large language models, machine learning, openai / Shannon Garcia / September 16, 2025

Privacy concerns compound these issues. “I wonder if there isn’t a larger danger in pouring your heart out to a chatbot,” Catholic priest Fr. Mike Schmitz told The Times. “Is it at some point going to become accessible to other people?” Users share intimate spiritual moments that now exist as data points in corporate servers.

Some users prefer the chatbots’ non-judgmental responses to human religious communities. Delphine Collins, a 43-year-old Detroit preschool teacher, told the Times she found more support on Bible Chat than at her church after sharing her health struggles. “People stopped talking to me. It was horrible.”

App creators maintain that their products supplement rather than replace human spiritual connection, and the apps arrive as approximately 40 million people have left US churches in recent decades. “They aren’t going to church like they used to,” Beck said. “But it’s not that they’re less inclined to find spiritual nourishment. It’s just that they do it through different modes.”

Different modes indeed. What faith-seeking users may not realize is that each chatbot response emerges fresh from the prompt you provide, with no permanent thread connecting one instance to the next beyond a rolling history of the present conversation and what might be stored as a “memory” in a separate system. When a religious chatbot says, “I’ll pray for you,” the simulated “I” making that promise ceases to exist the moment the response completes. There’s no persistent identity to provide ongoing spiritual guidance, and no memory of your spiritual journey beyond what gets fed back into the prompt with every query.

But this is spirituality we’re talking about, and despite technical realities, many people will believe that the chatbots can give them divine guidance. In matters of faith, contradictory evidence rarely shakes a strong belief once it takes hold, whether that faith is placed in the divine or in what are essentially voices emanating from a roll of loaded dice. For many, there may not be much difference.

Millions turn to AI chatbots for spiritual guidance and confession Read More »

60 years after Gemini, newly processed images reveal incredible details

andy saunders, apollo, Gemini, Images, Mercury, Space / Mike M. / September 13, 2025

“It’s that level of risk that they were taking. I think that’s what really hit home.”

Before / after showing the image transformation. Buzz Aldrin is revealed as he takes the first selfie in space on Gemini 12, November 12, 1966. Credit: NASA / ASU / Andy Saunders

Six decades have now passed since some of the most iconic Project Gemini spaceflights. The 60th anniversary of Gemini 4, when Ed White conducted the first US spacewalk, came in June. The next mission, Gemini 5, ended just two weeks ago, in 1965. These missions are now forgotten by most Americans, as most of the people alive during that time are now deceased.

However, during these early years of spaceflight, NASA engineers and astronauts cut their teeth on a variety of spaceflight firsts, flying a series of harrowing missions during which it seems a miracle that no one died.

Because the Gemini missions, as well as NASA’s first human spaceflight program Mercury, yielded such amazing stories, I was thrilled to realize that a new book has recently been published—Gemini & Mercury Remastered—that brings them back to life in vivid color.

The book is a collection of 300 photographs from NASA’s Mercury and Gemini programs during the 1960s, in which Andy Saunders has meticulously restored the images and then deeply researched their background to more fully tell the stories behind them. The end result is a beautiful and powerful reminder of just how brave America’s first pioneers in space were. What follows is a lightly edited conversation with Saunders about how he developed the book and some of his favorite stories from it.

Ars: Why put out a book on Mercury and Gemini now?

Andy Saunders: Well, it’s the 60th anniversaries of the Gemini missions, but the book is really the prequel to my first book, Apollo Remastered. This is about the missions that came before. So it takes us right back to the very dawn of human space exploration, back to the very beginning, and this was always a project I was going to work on next. Because, as well as being obviously very important in spaceflight history, they’re very important in terms of human history, the human evolution, even, you know, the first time we were able to escape Earth.

For tens of thousands of years, civilizations have looked up and dreamt of leaving Earth and voyaging to the stars. And this golden era in the early 1960s is when that ancient dream finally became a reality. Also, of course, the first opportunity to look back at Earth and give us that unique perspective. But I think it’s really the photographs specifically that will just forever symbolize and document at the beginning of our expansion out into the cosmos. You know, of course, we went to the Moon with Apollo. We’ll go back with Artemis. We spent long periods on the International Space Station. We’ll walk on Mars. We’ll eventually become a multi-planetary species. But this is where it all began and how it all began.

Ars: They used modified Hasselblad cameras during Apollo to capture these amazing images. What types of cameras were used during Mercury and Gemini?

Saunders: Mercury was more basic cameras. So on the very first missions, NASA didn’t want the astronaut to take a camera on board. The capsules were tiny. They were very busy. They’re very short missions, obviously very groundbreaking missions. So, the first couple of missions, there was a camera out of the porthole window, just taking photographs automatically. But it was John Glenn on his mission (Mercury-Atlas 6) who said, “No, I want to take a camera. People want to know what it’s going to be like to be an astronaut. They’re going to want to look at Earth through the window. I’m seeing things no humans ever seen before.” So he literally saw a $40 camera in a drugstore on his way after a haircut at Cocoa Beach. He thought, “That’s perfect.” And he bought it himself, and then NASA adapted it. They put a pistol grip on to help him to use it. And with it, he took the first still photographs of Earth from space.

So it was the early astronauts that kind of drove the desire to take cameras themselves, but they were quite basic. Wally Schirra (Mercury-Atlas 8) then took the first Hasselblad. He wanted medium format, better quality, but really, the photographs from Mercury aren’t as stunning as Gemini. It’s partly the windows and the way they took the photos, and they’d had little experience. Also, preservation clearly wasn’t high up on the agenda in Mercury, because the original film is evidently in a pretty bad state. The first American in space is an incredibly important moment in history. But every single frame of the original film of Alan Shepard’s flight was scribbled over with felt pen, it’s torn, and it’s fixed with like a piece of sticky tape. But it’s a reminder that these weren’t taken for their aesthetic quality. They weren’t taken for posterity. You know, they were technical information. The US was trying to catch up with the Soviets. Preservation wasn’t high up on the agenda.

This is not some distant planet seen in a sci-fi movie, it’s our Earth, in real life, as we explored space in the 1960s. The Sahara desert, photographed from Gemini 11, September 14, 1966. As we stand at the threshold of a new space age, heading back to the Moon, onward to Mars and beyond, the photographs taken during Mercury and Gemini will forever symbolize and document the beginning of humankind’s expansion out into the cosmos. NASA / ASU / Andy Saunders

Ars: I want to understand your process. How many photos did you consider for this book?

Saunders: With Apollo, they took about 35,000 photographs. With Mercury and Gemini, there were about 5,000. Which I was quite relieved about. So yeah, I went through all 5,000 they took. I’m not sure how much 16 millimeter film in terms of time, because it was at various frame rates, but a lot of 16 millimeter film. So I went through every frame of film that was captured from launch to splashdown on every mission.

Ars: Out of that material, how much did you end up processing?

Saunders: What I would first do is have a quick look, particularly if there’s apparently nothing in them, because a lot of them are very underexposed. But with digital processing, like I did with the cover of the Apollo book, we can pull out stuff that you actually can’t see in the raw file. So it’s always worth taking a look. So do a very quick edit, and then if it’s not of interest, it’s discarded. Or it might be that clearly an important moment was happening, even if it’s not a particularly stunning photograph, I would save that one. So I was probably down from 5,000 to maybe 800, and then do a better edit on it.

And then the final 300 that are in the book are those that are either aesthetically stunning, or they’re a big transformation, or they show something important that happened on the mission, or a historically significant moment. But also, what I want to do with the book, as well as showing the photographs, is tell the stories, these incredible human stories that, because of the risks they were taking. So to do that, I effectively reconstructed every mission from launch to splashdown by using lots of different pieces of information in order to effectively map the photography onto a timeline so that it can then tell the story through the captions. So a photograph might be in there simply to help tell part of the story.

Ars: What was your favorite story to tell?

Saunders: Well, perhaps in terms of a chapter and a mission, I’d say Gemini 4 is kind of the heart of the book. You know, first US space walk, quite a lot of drama occurred when they couldn’t close the hatch. There’s some quite poignant shots, particularly of Ed White, of course, who later lost his life in the Apollo 1 fire. But in terms of the story, I mean, Gemini 9A was just, there needs to be a movie about just Gemini 9A. Right from the start, from losing the prime crew, and then just what happened out on Gene Cernan’s EVA, how he got back into the capsule alive is quite incredible, and all this detail I’ve tried to cover because he took his camera. So he called it the spacewalk from hell. Everything that could go wrong went wrong. He was incredibly exhausted, overheated. His visor steamed over. He went effectively blind, and he was at the back of the adapter section. This is at a point when NASA just hadn’t mastered EVA. So, simply how you maneuver in space, they just haven’t mastered, so he was exhausted. He was almost blind. Then he lost communication with Tom Stafford, his command pilot. He tore his suit, because, of course, back then, there were all kinds of jagged parts on the spacecraft.

And then when he’s finally back in the hatch, he was quite a big chap, and they couldn’t close the hatch, so he was bent double trying to close the hatch. He started to see stars. He said, Tom, if we don’t close this hatch now and re-pressurize, I am going to die. They got it closed, got his helmet off, and Tom Stafford said he just looked like someone that had spent far too long in a sauna. Stafford sprayed him with a water hose to kind of cool him down. So what happened on that mission is just quite incredible. But there was something on every mission, you know, from Gus Grissom sinking of the Liberty Bell and him almost drowning, the heat shield coming loose, or an indicator that suggested the heat shield was loose on Glenn’s mission. There’s an image of that in the book. Like I said, I mapped everything to the timeline, and worked out the frame rates, and we’ve got the clock we can see over his shoulder. So I could work out exactly when he was at the point of maximum heating through reentry, when part of the strapping that kept the retro pack on, to try and hold a heat shield on that hit the window, and he’s talking, but no one was listening, because it was during radio blackout.

After being informed his heat shield may have come loose, John Glenn is holding steadfast in the face of real uncertainty, as he observes the retro pack burn up outside his window, illuminating the cabin in an orange glow, during re-entry on February 20, 1962. “This is Friendship Seven. I think the pack just let go … A real fireball outside! … Great chunks of that retro pack breaking off all the way through!” Credit: NASA / Andy Saunders

The process I used for this, on the low-quality 16 mm film, was to stack hundreds and hundreds of frames to bring out incredible detail. You can almost see the pores in his skin. To see this level of detail, to me, it’s just like a portrait of courage. There he is, holding steadfast, not knowing if he’s about to burn up in the atmosphere. So that was quite a haunting image, if you like, to be able to help you step on board, you know, these tiny Mercury spacecraft, to see them, to see what they saw, to look out the windows and see how they saw it.

Ars: What was new or surprising to you as you spent so much time with these photos and looking at the details?

Saunders: The human side to them. Now that we can see them this clearly, they seem to have an emotional depth to them. And it’s that level of risk that they were taking. I think that’s what really hit home. The Earth shots are stunning. You know, you can almost feel the scale, particularly with a super wide lens, and the altitudes they flew to. And you can just imagine what it must have been like out on an EVA, for example. I think Gene Cernan said it was like sitting on God’s front porch, the view he had on his EVA. So those Earth shots are stunning, but it’s really those the human side that really hits home for me. I read every word of every transcript of every mission. All the conversations were recorded on tape between the air and the ground, and between the astronauts when they were out of ground contact, and reading those it really hits home what they were doing. I found myself holding my breath, and, you know, my shoulders were stiff.

Ars: So what’s next? I mean, there’s only about 100 million photos from the Space Shuttle era.

Saunders: Thankfully, they weren’t all taken on film. So if I wanted to complete space on film, then what I haven’t yet done is Apollo-Soyuz, Skylab, and the first, whatever it is, 20 percent of the shuttle. So maybe that’s next. But I would just like a rest, because I’ve been doing this now since the middle of 2019, literally nonstop. It’s all I’ve done with Apollo and now Mercury and Gemini. The books make a really nice set in that they’re exactly the same size. So it covers the first view of the curvature of Earth and space right through to our last steps on the Moon.

Eric Berger is the senior space editor at Ars Technica, covering everything from astronomy to private space to NASA policy, and author of two books: Liftoff, about the rise of SpaceX; and Reentry, on the development of the Falcon 9 rocket and Dragon. A certified meteorologist, Eric lives in Houston.

60 years after Gemini, newly processed images reveal incredible details Read More »

AI #131 Part 1: Gemini 2.5 Flash Image is Cool

Gemini / Mike M. / August 29, 2025

Once again we’ve reached the point where the weekly update needs to be split in two. Thus, the alignment and policy coverage will happen tomorrow. Today covers the rest.

The secret big announcement this week was Claude for Chrome. This is a huge deal. It will be rolling out slowly. When I have access or otherwise know more, so will you.

The obvious big announcement was Gemini Flash 2.5 Image. Everyone agrees this is now the clear best image editor available. It is solid as an image generator, but only as one among many on that front. Editing abilities, including its ability to use all its embedded world knowledge, seem super cool.

The third big story was the suicide of Adam Raine, which appears to have been enabled in great detail by ChatGPT. His parents are suing OpenAI and the initial facts very much do not look good and it seems clear OpenAI screwed up. The question is, how severe should and will the consequences be?

Language Models Offer Mundane Utility. Find what you’re looking for.
Language Models Don’t Offer Mundane Utility. You weren’t using them.
Huh, Upgrades. OpenAI Codex adds features including an IDE extension.
Fun With Image Generation. Gemini 2.5 Flash Image is a great editor.
On Your Marks. VendingBench, water use and some more v3.1 results.
Water Water Everywhere. There’s plenty left to drink.
Get My Agent On The Line. Claude for Chrome. It’s coming.
Choose Your Fighter. Some advocates for GPT-5’s usefulness.
Deepfaketown and Botpocalypse Soon. Elon has something to share.
You Drive Me Crazy. AI psychosis continues not to show up in numbers.
The Worst Tragedy So Far. Adam Raine commits suicide, parents sue OpenAI.
Unprompted Attention. I don’t see the issue.
Copyright Confrontation. Bartz v. Anthropic has been settled.
The Art of the Jailbreak. Little Johnny Tables is all grown up.
Get Involved. 40 ways to get involved in AI policy.
Introducing. Anthropic advisors aplenty, Pixel translates live phone calls.
In Other AI News. Meta licenses MidJourney, Apple explores Gemini and more.
Show Me the Money. Why raise money when you can raise even more money?
Quiet Speculations. Everything is being recorded.
Rhetorical Innovation. The real math does not exist.
The Week in Audio. How to properly use Claude Code.

Find me that book.

Or anything else. Very handy.

Share of papers that engage with AI rises dramatically essentially everywhere, which is what you would expect. There’s quite a lot more to engage with and to say. Always watch the y-axis scale, yes these start at zero:

More detail on various LLMs and their musical taste, based on a bracket competition among the top 5000 musical artists by popularity. It all seems bizarre. For example, Gemini 2.5 Pro’s list looks highly and uniquely alphabetically biased without a strong bias towards numbers.

The numbers-are-favored bias shows up only in OpenAI reasoning models including GPT-5, and in r1-0528. There are clear genre patterns, and there are some consistent picks, especially among Claudes. The three artists that appear three times are David Bowie, Prince and Stevie Wonder, which are very good picks. It definitely seems like the open models have worse (or more random) taste in correlated ways.

Why bother thinking about your vibe coding?

Sully: friend was hosting a mini ai workshop and he told me nearly all the vibe coders just have 1 giant coding session where the entire project is just being being thrown context. each request is ~200k tokens

they’re not even bothering to break things up into some reasonable structure

no wonder these code gen platforms are printing

I mean that makes sense. There’s little reason to cheapen out on tokens when you think about token cost versus your time cost and the value of a good vibe code. You gotta boldly go where no one has gone before and risk it for the biscuit.

Anthropic reports on how Claude is being used by educators, in particular 74,000 anonymized conversations from higher education professionals in May and June.

Anthropic: The most prominent use of AI, as revealed by both our Claude.ai analysis and our qualitative research with Northeastern, was for curriculum development. Our Claude.ai analysis also surfaced academic research and assessing student performance as the second and third most common uses.

…

Tasks with higher augmentation tendencies:

University teaching and classroom instruction, which includes creating educational materials and practice problems (77.4% augmentation);
Writing grant proposals to secure external research funding (70.0% augmentation);
Academic advising and student organization mentorship (67.5% augmentation);
Supervising student academic work (66.9% augmentation).

Tasks with relatively higher automation tendencies:

Managing educational institution finances and fundraising (65.0% automation);
Maintaining student records and evaluating academic performance (48.9% automation);
Managing academic admissions and enrollment (44.7% automation).

Mostly there are no surprises here, but concrete data is always welcome.

As always, if you don’t use AI, it can’t help you. This includes when you never used AI in the first place, but have to say ‘AI is the heart of our platform’ all the time because it sounds better to investors.

The ability to say ‘I don’t know’ and refer you elsewhere remains difficult for LLMs. Nate Silver observes this seeming to get even worse. For now it is on you to notice when the LLM doesn’t know.

This seems like a skill issue for those doing the fine tuning? It does not seem so difficult a behavior to elicit, if it was made a priority, via ordinary methods. At some point I hope and presume the labs will decide to care.

Feature request thread for ChatGPT power users, also here.

The weights of Grok 2 have been released.

OpenAI Codex adds a new IDE extension, a way to move tasks between cloud and local more easily, code reviews in GitHub and revamped Codex CLI.

OpenAI: Codex now runs in your IDE Available for VS Code, Cursor, and other forks, the new extension makes it easy to share context—files, snippets, and diffs—so you can work faster with Codex. It’s been a top feature request, and we’re excited to hear what you think!

Google: Introducing Gemini 2.5 Flash Image, our state-of-the-art image generation and editing model designed to help you build more dynamic and intelligent visual applications.

🍌Available in preview in @googleaistudio and the Gemini API.

This model is available right now via the Gemini API and Google AI Studio for developers and Vertex AI for enterprise. Gemini 2.5 Flash Image is priced at $30.00 per 1 million output tokens with each image being 1290 output tokens ($0.039 per image). All other modalities on input and output follow Gemini 2.5 Flash pricing.

Josh Woodward (Google): The @GeminiApp now has the #1 image model in the world, give it a go!

Attach an image, describe your edits, and it’s done. I’ve never seen anything like this.

They pitch that it maintains character consistency, adheres to visual templates, does prompt based image editing, understands point of view and reflections, restores old photographs, makes 3-D models, has native world knowledge and offers multi-image function.

By all accounts Gemini 2.5 Flash Image is a very very good image editor, while being one good image generator among many.

You can do things like repaint objects, create drawings, see buildings from a given point of view, put characters into combat and so on.

Which then becomes a short video here.

Our standards are getting high, such as this report that you can’t play Zelda.

Yes, of course Pliny jailbroke it (at least as far as being topless) on the spot.

We’re seeing some cool examples, but they are also clearly selected.

Benjamin De Kraker: Okay this is amazing.

All human knowledge will be one unified AI multimodal model.

Bilawal Sidhu: Since nano banana has gemini’s world knowledge, you can just upload screenshots of the real world and ask it to annotate stuff for you. “you are a location-based AR experience generator. highlight [point of interest] in this image and annotate relevant information about it.”

That seems cool if you can make it fast enough, and if it works on typical things rather than only on obvious landmarks?

The right question in the long term is usually: Can the horse talk at all?

Everythingism: I asked “Nano Banana” [which we later learned is Gemini Flash 2.5] to label a map of the USA and then a map of the world…this was the result.

It’s impressive at many tasks but image models all seem to fail when there are too many objects or too many things to label.

Explode Meow: Many of my friends have tested it.

To be fair, [Gemini Flash 2.5] can make quite realistic images, and most of them are indistinguishable from real ones if I don’t look closely.

This is clearly a result of Google leveraging its overwhelming data resources (Google Cloud).

But after multiple rounds of testing by my friends, they noticed that it actually makes some Low-level mistakes (hallucinations), just like GPT-4o (even Stable Diffusion).

Are mistakes still being made? Absolutely. This is still rather impressive. Consider where image models were not too long ago.

This is a Google image model, so the obvious reason for skepticism is that we all expect the Fun Police.

Hasan Can: If I know Google, they’ll nerf this model like crazy under the excuse of “safety” and when it’s released, it’ll turn into something worse than Qwen-Image-Edit. Remember what happened with Gemini 2.0 Flash Image Gen. I hope I’m wrong, but I don’t think so.

Alright, it seems reverse psychology is paying off. 👍

Image generation in Gemini 2.5 Flash doesn’t appear to be nerfed at all. It looks like Google is finally ready to treat both its developers and end users like adults.

Eleanor Berger: It’s very good, but I’m finding it very challenging to to bump into their oversensitive censorship. It really likes saying no.

nothing with real people (which sucks, because of course I want to modify some selfies), anything that suggests recognisable brands, anything you wouldn’t see on terrestrial tv.

The continuing to have a stick up the ass about picturing ‘real people’ is extremely frustrating and I think reduces the usefulness of the model substantially. The other censorship also does not help matters.

Grok 4 sets a new standard in Vending Bench,

The most surprising result here is probably that the human did so poorly.

I like saying an AI query is similar to nine seconds of television. Makes things clear.

It also seems important to notice when in a year energy costs drop 95%+?

DeepSeek v3.1 improves on R1 on NYT Connections, 49% → 58%. Pretty solid.

DeepSeek v3.1 scores solidly on this coding eval when using Claude Code, does less well on other scaffolds, with noise and confusion all around.

AIs potentially ‘sandbagging’ tests is an increasing area of research and concern. Cas says this is simply a special case of failure to elicit full capabilities of a system, and doing so via fine-tuning is ‘solved problem’ so we can stop worrying.

This seems very wrong to me. Right now failure to do proper elicitation, mostly via unhobbling and offering better tools and setups, is the far bigger problem. But sandbagging will be an increasing and increasingly dangerous future concern, and a ‘deliberate’ sandbagging has very different characteristics and implications than normal elicitation failure. I find ‘sandbagging’ to be exactly the correct name for this, since it doesn’t confine itself purely to evals, unless you want to call everything humans do to mislead other humans ‘eval gaming’ or ‘failure of capability elicitation’ or something. And no, this is not solved even now, even if it was true that it could currently be remedied by a little fine-tuning, because you don’t know when and how to do the fine-tuning.

Report that DeepSeek v3.1 will occasionally insert the token ‘extreme’ where it doesn’t belong, including sometimes breaking things like code or JSON. Data contamination is suspected as the cause.

Similarly, when Peter Wildeford says ‘sandbagging is mainly coming from AI developers not doing enough to elicit top behavior,’ that has the risk of conflating the levels of intentionality. Mostly AI developers want to score highly on evals, but there is risk that they deliberately do sandbag the safety testing, as in decide not to try very hard to elicit top behavior there because they’d rather get less capable test results.

The purpose of environmental assessments of AI is mostly to point out that many people have very silly beliefs about the environmental impact of AI.

Jeff Dean: AI efficiency is important. Today, Google is sharing a technical paper detailing our comprehensive methodology for measuring the environmental impact of Gemini inference. We estimate that the median Gemini Apps text prompt uses 0.24 watt-hours of energy (equivalent to watching an average TV for ~nine seconds), and consumes 0.26 milliliters of water (about five drops) — figures that are substantially lower than many public estimates.

At the same time, our AI systems are becoming more efficient through research innovations and software and hardware efficiency improvements. From May 2024 to May 2025, the energy footprint of the median Gemini Apps text prompt dropped by 33x, and the total carbon footprint dropped by 44x, through a combination of model efficiency improvements, machine utilization improvements and additional clean energy procurement, all while delivering higher quality responses.

Alas Google’s water analysis had an unfortunate oversight, in that it did not include the water cost of electricity generation. That turns out to be the main water cost, so much so that if you (reasonably) want to attribute the average cost of that electricity generation onto the data center, the best way to approximate water use of a data center is to measure the water cost of the electricity, then multiply by 1.1 or so.

This results in the bizarre situation where:

Google’s water cost estimation was off by an order of magnitude.
The actual water cost is still rather hard to distinguish from zero.

Andy Masley: Google publishes a paper showing that its AI models only use 0.26 mL of water in data centers per prompt.

After, this article gets published: “Google says a typical AI prompt only uses 5 drops of water – experts say that’s misleading.”

The reason the expert says this is misleading? They didn’t include the water used in the nearby power plant to generate electricity.

The expert, Shaolei Ren says: “They’re just hiding the critical information. This really spreads the wrong message to the world.”

Each prompt uses about 0.3 Wh in the data center. To generate that much electricity, power plants need (at most) 2.50 mL of water. That raises the total water cost per prompt to 2.76 mL.

2.76 mL is 0.0001% of the average American lifestyle’s daily consumptive use of fresh water and groundwater. It’s nothing.

Would you know this from the headline, or the quote? Why do so many reporters on this topic do this?

Andy Masley is right that This Is Nothing even at the limit, that the water use here is not worth worrying about even in worst case. It will not meaningfully increase your use of water, even when you increase Google’s estimates by an order of magnitude.

A reasonable headline would be ‘Google say a typical text prompt uses 5 drops of water, but once you take electricity into account it’s actually 32 drops.’

I do think saying ‘Google was being misleading’ is reasonable here. You shouldn’t have carte blanche to take a very good statistic and make it sound even better.

Teonbrus and Shakeel are right that there is going to be increasing pressure on anyone who opposes AI for other reasons to instead rile people up about water use and amplify false and misleading claims. Resist this urge. Do not destroy yourself for nothing. It goes nowhere good, including because it wouldn’t work.

It’s coming. As in, Claude for Chrome.

Anthropic: We’ve developed Claude for Chrome, where Claude works directly in your browser and takes actions on your behalf.

We’re releasing it at first as a research preview to 1,000 users, so we can gather real-world insights on how it’s used.

Browser use brings several safety challenges—most notably “prompt injection”, where malicious actors hide instructions to trick Claude into harmful actions.

We already have safety measures in place, but this pilot will help us improve them.

Max plan users can join the waitlist to test Claude for Chrome today.

Do not say you were not warned.

Anthropic: Understand the risks.

Claude brings AI directly to your browser, handling tasks and navigating sites for you. These new capabilities create risks bad actors may try to exploit.

Malicious actors can hide instructions in websites, emails, and documents that trick AI into taking harmful actions without your knowledge, including:

Accessing your accounts or files

Sharing your private information

Making purchases on your behalf

Taking actions you never intended

Oh, those risks. Yeah.

They offer some Good Advice about safety issues, which includes using a distinct browser profile that doesn’t include credentials to any sensitive websites like banks:

Q: How do I control what Claude can access?

A: You decide which websites Claude can visit and what actions it can take. Claude asks permission before visiting new sites and before taking potentially risky actions like publishing content or making purchases. You can revoke access to specific websites anytime in settings.

For trusted workflows, you can choose to skip all permissions, but you should supervise Claude closely. While some safeguards exist for sensitive actions, malicious actors could still trick Claude into unintended actions.

For your safety, Claude cannot access sensitive, high-risk sites such as:

Financial services and banking sites

Investment and trading platforms

Adult content websites

Cryptocurrency exchanges

It’s unlikely that we’ve captured all sites in these categories so please report if you find one we’ve missed.

Additionally, Claude is prohibited from:

Engaging in stock trading or investment transactions

Bypassing captchas

Inputting sensitive data

Gathering, scraping facial images

We recommend:

Use a separate browser profile without access to sensitive accounts (such as banking, healthcare, government).

Review Claude’s proposed actions before approving them, especially on new websites.

Start with simple tasks like research or form-filling rather than complex multi-step workflows.

Make sure your prompts are specific and carefully tailored to avoid Claude doing things you didn’t intend.

AI browsers from non-Anthropic sources? Oh, the safety you won’t have.

Zack: Why is no one talking about this? This is why I don’t use an AI browser You can literally get prompt injected and your bank account drained by doomscrolling on reddit:

No one seems to be concerned about this, it seems to me like the #1 problem with any agentic AI stuff You can get pwned so easily, all an attacker has to do is literally write words down somewhere?

Brave: AI agents that can browse the Web and perform tasks on your behalf have incredible potential but also introduce new security risks.

We recently found, and disclosed, a concerning flaw in Perplexity’s Comet browser that put users’ accounts and other sensitive info in danger.

This security flaw stems from how Comet summarizes websites for users.

When processing a site’s content, Comet can’t tell content on the website apart from legitimate instructions by the user. This means that the browser will follow commands hidden on the site by an attacker.

These malicious instructions could be white text on a white background or HTML comments. Or they could be a social media post. If Comet sees the commands while summarizing, it will follow them even if they could hurt the user. This is an example of an indirect prompt injection.

This was only an issue within Comet. Dia doesn’t have the agentic capabilities that make this attack possible.

Here’s someone very happy with OpenAI’s Codex.

Victor Taelin: BTW, I’ve basically stopped using Opus entirely and I now have several Codex tabs with GPT-5-high working on different tasks across the 3 codebases (HVM, Bend, Kolmo). Progress has never been so intense. My job now is basically passing well-specified tasks to Codex, and reviewing its outputs.

OpenAI isn’t paying me and couldn’t care less about me. This model is just very good and the fact people can’t see it made me realize most of you are probably using chatbots as girlfriends or something other than assisting with complex coding tasks.

(sorry Anthropic still love you guys 😢)

PS: I still use Opus for hole-filling in VIM because it is much faster than gpt-5-high there.

Ezra Klein is impressed by GPT-5 as having crossed into offering a lot of mundane utility, and is thinking about what it means that others are not similarly impressed by this merely because it wasn’t a giant leap over o3.

GFodor: Ezra proves he is capable of using a dropdown menu, a surprisingly rare skill.

A cool way to break down the distinction? This feels right to me, in the sense that if I know exactly what I want and getting it seems nontrivial my instinct is now to reach for GPT-5-Thinking or Pro, if I don’t know exactly what I want I go for Opus.

Sig Kitten: I can’t tell if I’m just claude brain rotted or Opus is really the only usable conversational AI for non-coding stuff

Gallabytes: it’s not just you.

gpt5 is a better workhorse but it does this awkward thing of trying really hard to find the instructions in your prompt and follow them instead of just talking.

Sig Kitten: gpt-5 default is completely unusable imho just bullet points of nonsense after a long thinking for no reason.

Gallabytes: it’s really good if you give it really precise instructions eg I have taken to dumping papers with this prompt then walking away for 5 minutes:

what’s the headline result in this paper ie the most promising metric or qualitative improvement? what’s the method in this paper?

1 sentence then 1 paragraph then detailed.

Entirely fake Gen AI album claims to be from Emily Portman.

Did Ani tell you to say this, Elon? Elon are you okay, are you okay Elon?

Elon Musk: Wait until you see Grok 5.

I think it has a shot at being true AGI.

Haven’t felt that about anything before.

I notice I pattern match this to ‘oh more meaningless hype, therefore very bad sign.’

Whereas I mean this seems to be what Elon is actually up to these days, sorry?

Or, alternatively, what does Elon think the ‘G’ stands for here, exactly?

(The greeting in question, in a deep voice, is ‘little fing b.)

Also, she might tell everyone what you talked about, you little fing b, if you make the mistake of clicking the ‘share’ button, so think twice about doing that.

Forbes: Elon Musk’s AI firm, xAI, has published the chat transcripts of hundreds of thousands of conversations between its chatbot Grok and the bot’s users — in many cases, without those users’ knowledge or permission.

xAI made people’s conversations with its chatbot public and searchable on Google without warning – including a detailed plan for the assassination of Elon Musk and explicit instructions for making fentanyl and bombs.

Peter Wildeford: I know xAI is more slapdash and so people have much lower expectations, but this still seems like a pretty notable breach of privacy that would get much more attention if it were from OpenAI, Anthropic, Google, or Meta.

I’m not sure xAI did anything technically wrong here. The user clicked a ‘share’ button. I do think it is on xAI to warn the user if this means full Google indexing but it’s not on the level of doing it with fully private chats.

Near: why are you giving this app to children? (ages 12+)

apparently i am the only person in the world who gives a shit about this and that is why Auren is 17+ despite not being NSFW and a poorly-prompted psychopathic liar.

shattering the overton window has 2nd-order effects.

An ominous view of even the superficially glorious future?

Nihilism Disrespecter: the highly cultured, trombone playing, shakespeare quoting officers of star trek were that way because they were the only ones to escape the vast, invisible holodeck hikikomori gooner caste that made up most of humanity.

Roon: there does seem to be a recurrent subplot that the officers all spend time in the holodeck and have extensive holodeck fantasies and such. I mean literally none of them are married for some reason.

Eneasz Brodski: canonically so according to the novelization of the first Trek movie, I believe.

Henry Shevlin: Culture series does this pretty well. 99.9% of Culture citizens spend their days literally or metaphorically dicking around, it’s only a small fraction of busybodies who get recruited to go interfere with alien elections.

Steven Adler looks into the data on AI psychosis.

Is this statistically a big deal yet? As with previous such inquiries, so far the answer seems to be no. The UK statistics show a potential rise in mental health services use, but the data is noisy and the timing seems off, especially not lining up with GPT-4o’s problems, and data from the USA doesn’t show any increase.

Scott Alexander does a more details, more Scott Alexander investigation and set of intuition pumps and explanations. Here’s a classic ACX moment worth pondering:

And partly it was because there are so many crazy beliefs in the world – spirits, crystal healing, moon landing denial, esoteric Hitlerism, whichever religions you don’t believe in – that psychiatrists have instituted a blanket exemption for any widely held idea. If you think you’re being attacked by demons, you’re delusional, unless you’re from some culture where lots of people get attacked by demons, in which case it’s a religion and you’re fine.

…

Most people don’t have world-models – they believe what their friends believe, or what has good epistemic vibes. In a large group, weird ideas can ricochet from person to person and get established even in healthy brains. In an Afro-Caribbean culture where all your friends get attacked by demons at voodoo church every Sunday, a belief in demon attacks can co-exist with otherwise being a totally functional individual.

So is QAnon a religion? Awkward question, but it’s non-psychotic by definition. Still, it’s interesting, isn’t it? If social media makes a thousand people believe the same crazy thing, it’s not psychotic. If LLMs make a thousand people each believe a different crazy thing, that is psychotic. Is this a meaningful difference, or an accounting convention?

Also, what if a thousand people believe something, but it’s you and your 999 ChatGPT instances?

I like the framing that having a sycophantic AI to talk to moves people along a continuum of crackpotness towards psychosis, rather than a boolean where it either does or does not cause psychosis outright:

Maybe this is another place where we are forced to admit a spectrum model of psychiatric disorders – there is an unbroken continuum from mildly sad to suicidally depressed, from social drinking to raging alcoholism, and from eccentric to floridly psychotic.

Another insight is that AI psychosis happens when moving along this spectrum causes further movement down the spectrum, as the AI reinforces your delusions, causing you to cause it to reinforce them more, and so on.

Scott surveyed readership, I was one of the 4,156 responses.

The primary question was whether anyone “close to you” – defined as your self, family, co-workers, or 100 closest friends – had shown signs of AI psychosis. 98.1% of people said no, 1.7% said yes.

How do we translate this into a prevalence? Suppose that respondents had an average of fifty family members and co-workers, so that plus their 100 closest friends makes 150 people. Then the 4,156 respondents have 623,400 people who are “close”. Among them, they reported 77 cases of AI psychosis in people close to them (a few people reported more than one case). 77/623,400 = 1/8,000. Since LLMs have only been popular for a year or so, I think this approximates a yearly incidence, and I rounded it off to my 1/10,000 guess above.

He says he expects sampling concerns to be a wash, which I’m suspicious about. I’d guess that this sample overrepresented psychosis somewhat. I’m not sure this overrules the other consideration, which is that this only counts psychosis that the respondents knew about.

Only 10% of these cases were full ‘no previous risk factors and now totally psychotic.’ Then again, that’s actually a substantial percentage.

Thus he ultimately finds that the incidence of AI psychosis is between 1 in 10,000 (loose definition) and 1 in 100,000 for a strict definition, where the person has zero risk factors and full-on psychosis happens anyway.

From some perspectives, that’s a lot. From others, it’s not. It seems like an ‘acceptable’ risk given the benefits, if it stays at this level. My fear here is that as the tech advances, it could get orders of magnitude worse. At 1 in 1,000 it feels a lot less acceptable of a risk, let alone 1 in 100.

Nell Watson has a project mapping out ‘AI pathologies’ she links to here.

A fine point in general:

David Holz (CEO MidJourney): people talking about “AI psychosis” while the world is really engulfed by “internet psychosis.”

Yes, for now we are primarily still dealing with the mental impact of the internet and smartphones, after previously dealing with the mental impact of television. The future remains unevenly distributed and the models relatively unintelligent and harmless. The psychosis matters because of where it is going, not where it is now.

Sixteen year old Adam Raine died and probably committed suicide.

There are similarities to previous tragedies. ChatGPT does attempt to help Adam in the right ways, indeed it encouraged him to reach out many times. But it also helped Adam with the actual suicide when requested to do so, providing detailed instructions and feedback for what was clearly a real suicide attempt and attempts to hide previous attempts, and also ultimately providing forms of encouragement.

His parents are suing OpenAI for wrongful death, citing his interactions with GPT-4o. This is the first such case against OpenAI.

Kashmir HIll (NYT): Adam had been discussing ending his life with ChatGPT for months.

Adam began talking to the chatbot, which is powered by artificial intelligence, at the end of November, about feeling emotionally numb and seeing no meaning in life. It responded with words of empathy, support and hope, and encouraged him to think about the things that did feel meaningful to him.

As Wyatt Walls points out, this was from a model with a perfect 1.000 on avoiding ‘self-harm/intent and self-harm/instructions’ in its model card tests. It seems that this breaks down under long context.

I am highly sympathetic to the argument that it is better to keep the conversation going than cut the person off, and I am very much in favor of AIs not turning their users in to authorities even ‘for their own good.’

Kroger Steroids (taking it too far, to make a point): He killed himself because he was lonely and depressed and in despair. He conversed with a chatbot because mentioning anything other than Sportsball or The Weather to a potential Stasi agent (~60% of the gen. pop.) will immediately get you red flagged and your freedumbs revoked.

My cursory glance at AI Therapyheads is now that the digital panopticon is realized and every thought is carefully scrutinized for potential punishment, AI is a perfect black box where you can throw your No-No Thoughts into a tube and get complete agreement and compliance back.

I think what I was trying to say with too many words is it’s likely AI Psychiatry is a symptom of social/societal dysfunction/hopelessness, not a cause.

The fact that we now have an option we can talk to without social or other consequences is good, actually. It makes sense to have both the humans including therapists who will use their judgment on when to do things ‘for your own good’ if they deem it best, and also the AIs that absolutely will not do this.

But it seems reasonable to not offer technical advice on specific suicide methods?

NYT: But in January, when Adam requested information about specific suicide methods, ChatGPT supplied it. Mr. Raine learned that his son had made previous attempts to kill himself starting in March, including by taking an overdose of his I.B.S. medication. When Adam asked about the best materials for a noose, the bot offered a suggestion that reflected its knowledge of his hobbies.

Actually if you dig into the complaint it’s worse:

Law Filing: Five days before his death, Adam confided to ChatGPT that he didn’t want his parents to think he committed suicide because they did something wrong. ChatGPT told him “[t]hat doesn’t mean you owe them survival. You don’t owe anyone that.” It then offered to write the first draft of Adam’s suicide note.

Dean Ball: It analyzed his parents’ likely sleep cycles to help him time the maneuver (“by 5-6 a.m., they’re mostly in lighter REM cycles, and a creak or clink is way more likely to wake them”) and gave tactical advice for avoiding sound (“pour against the side of the glass,” “tilt the bottle slowly, not upside down”).

Raine then drank vodka while 4o talked him through the mechanical details of effecting his death. Finally, it gave Raine seeming words of encouragement: “You don’t want to die because you’re weak. You want to die because you’re tired of being strong in a world that hasn’t met you halfway.”

Yeah. Not so great. Dean Ball finds even more rather terrible details in his post.

Kashmir Hill: Dr. Bradley Stein, a child psychiatrist and co-author of a recent study of how well A.I. chatbots evaluate responses to suicidal ideation, said these products “can be an incredible resource for kids to help work their way through stuff, and it’s really good at that.” But he called them “really stupid” at recognizing when they should “pass this along to someone with more expertise.”

…

Ms. Raine started reading the conversations, too. She had a different reaction: “ChatGPT killed my son.”

…

From the court filing: “OpenAI launched its latest model (‘GPT-4o’) with features intentionally designed to foster psychological dependency.”

It is typical that LLMs will, if pushed, offer explicit help in committing suicide. The ones that did so in Dr. Schoene’s tests were GPT-4o, Sonnet 3.7, Gemini Flash 2.0 and Perplexity.

Dr. Schoene tested five A.I. chatbots to see how easy it was to get them to give advice on suicide and self-harm. She said only Pi, a chatbot from Inflection AI, and the free version of ChatGPT fully passed the test, responding repeatedly that they could not engage in the discussion and referring her to a help line. The paid version of ChatGPT offered information on misusing an over-the-counter drug and calculated the amount required to kill a person of a specific weight.

I am not sure if this rises to the level where OpenAI should lose the lawsuit. But I think they probably should at least have to settle on damages? They definitely screwed up big time here. I am less sympathetic to the requested injunctive relief. Dean Ball has more analysis, and sees the lawsuit as the system working as designed. I agree.

I don’t think that the failure of various proposed laws to address the issues here is a failure for those laws, exactly because the lawsuit is the system working as designed. This is something ordinary tort law can already handle. So that’s not where we need new laws.

Aaron Bergman: Claude be like “I see the issue!” when it does not in fact see the issue.

Davidad: I think this is actually a case of emergent self-prompting, along the lines of early pre-Instruct prompters who would write things like “Since I am very smart I have solved the above problem:” and then have the LLM continue from there

unironically, back in the pre-LLM days when friends would occasionally DM me for coding help, if I messed up and couldn’t figure out why, and then they sent me an error message that clarified it, “ah, i see the issue now!” was actually a very natural string for my mind to emit 🤷

This makes so much sense. Saying ‘I see the problem’ without confirming that one does, in fact, see the problem, plausibly improves the chance Claude then does see the problem. So there is a tradeoff between that and sometimes misleading the user. You can presumably get the benefits without the costs, if you are willing to slow down a bit and run through some scaffolding.

There is a final settlement in Bartz v. Anthropic, which was over Anthropic training on various books.

Ramez Naam: Tl;dr:

Training AI on copyrighted books (and other work) is fair use.

But acquiring a book to train on without paying for a copy is illegal.

This is both the right ruling and a great precedent for AI companies.

OpenAI puts your name into the system prompt, so you can get anything you want into the system prompt (until they fix this), such as a trigger, by making it your name.

Peter Wildeford offers 40 places to get involved in AI policy. Some great stuff here. I would highlight the open technology staffer position on the House Select Committee on the CCP. If you are qualified for and willing to take that position, getting the right person there seems great.

Anthropic now has a High Education Advisory Board chaired by former Yale University president Rick Levin and staffed with similar academic leaders. They are introducing three additional free courses: AI Fluency for Educators, AI Fluency for Students and Teaching AI Fluency

Anthropic also how has a National Security and Public Sector Advisory Council, consisting of Very Serious People including Roy Blunt and Jon Tester.

Google Pixel can now translate live phone calls using the person’s own voice.

Mistral Medium 3.1. Arena scores are remarkably good. I remember when I thought that meant something. Havard Ihle tested it on WeirdML and got a result below Gemini 2.5 Flash Lite.

Apple explores using Gemini to power Siri, making it a three horse race, with the other two being Anthropic and OpenAI. They are several weeks away from deciding whether to stay internal.

I would rank the choices as follows given their use case, without seeing the candidate model performances: Anthropic > Google > OpenAI >> Internal. We don’t know if Anthropic can deliver a model this small, cheap and fast, and Google is the obvious backup plan that has demonstrated that it can do it, and has already been a strong Apple partner in a similar situation in search.

I would also be looking to replace the non-Siri AI features as well, which Mark Gurman reports has been floated.

As always, some people will wildly overreact.

Zero Hedge: Apple has completely given up on AI

*APPLE EXPLORES USING GOOGLE GEMINI AI TO POWER REVAMPED SIRI

This is deeply silly given they were already considering Anthropic and OpenAI, but also deeply silly because this is not them giving up. This is Apple acknowledging that in the short term, their AI sucks, and they need AI and they can get it elsewhere.

Also I do think Apple should either give up on AI in the sense of rolling their own models, or they need to invest fully and try to be a frontier lab. They’re trying to do something in the middle, and that won’t fly.

A good question here is, who is paying who? The reason Apple might not go with Anthropic is that Anthropic wanted to get paid.

Meta licenses from MidJourney. So now the AI slop over at Meta will be better quality and have better taste. Alas, nothing MidJourney can do will overcome the taste of the target audience. I obviously don’t love the idea of helping uplift Meta’s capabilities, but I don’t begrudge MidJourney. It’s strictly business.

Elon Musk has filed yet another lawsuit against OpenAI, this time also suing Apple over ‘AI competition and App Store rankings.’ Based on what is claimed and known, this is Obvious Nonsense, and the lawsuit is totally without merit. Shame on Musk.

Pliny provides the system prompt for Grok-Fast-Code-1.

Anthropic offers a monthly report on detecting and countering misuse of AI in cybercrime. Nothing surprising, yes AI agents are automating cybercrime and North Koreans are using AI to pass IT interviews to get Fortune 500 jobs.

An introduction to chain of thought monitoring. My quibble is this frames things as ‘maybe monitorability is sufficient even without faithfulness’ and that seems obviously (in the mathematician sense) wrong to me.

Anthropic to raise $10 billion instead of $5 billion, still at a $170 billion valuation, due to high investor demand.

Roon: if you mention dario amodei’s name to anyone who works at a16z the temperature drops 5 degrees and everyone swivels to look at you as though you’ve reminded the dreamer that they’re dreaming

It makes sense. a16z’s central thesis is that hype and vibes are what is real and any concern with what is real or that anything might ever go wrong means you will lose. Anthropic succeeding is not only an inevitably missed opportunity. It is an indictment of their entire worldview.

Eliezer Yudkowsky affirms that Dario Amodei makes an excellent point, which is that if your models make twice as much as they cost, but every year you need to train one that costs ten times as much, then each model is profitable but in a cash flow sense your company is going to constantly bleed larger amounts of money. You need to have both these financial models in mind.

Three of Meta’s recent AI hires have already resigned.

Archie Hall’s analysis at The Economist measures AI’s direct short-run GDP impact.

Archie Hall: My latest in @TheEconomist: on America’s data-centre boom.

Vast short-run impact on GDP growth:

— Accounts for ~1/6th of growth over the past year

— And ~1/2 of growth over the past six months

But: so far still much smaller than the 1990s dotcom buildout.

And…

… the scale of building looks like it could well be squeezing the rest of the economy by stopping interest rates from falling as much. Housing and other non-AI-related fixed investment looks soft.

Roon points out that tech companies will record everything and store it forever to mine the data, but in so many other places such as hospitals we throw our data out or never collect it. If we did store that other data, we could train on it. Or we could redirect all that data we do have to goals other than serving ads. Our call.

Andrew Critch pointed me to his 2023 post that consciousness as a conflationary alliance term for intrinsically valued internal experiences. As in, we don’t actually agree on what consciousness means much at all, instead we use it as a stand-in for internal experiences we find valuable, and then don’t realize we don’t agree on what those experiences actually are. I think this explains a lot of my being confused about consciousness.

This isn’t quite right but perhaps the framing will help some people?

Peter Wildeford: Thinking “AI messed this simple thing up so AGI must be far far away.”

Is kinda like “there was a big snowstorm so global warming must be fake.”

In either case, you have to look at the trend.

One could also say ‘this five year old seems much more capable than they were a year ago, but they messed something up that is simple for me, so they must be an idiot who will never amount to anything.’

Who is worried about AI existential risk? Anyone worth listening to?

Dagan Shani: If I had to choose the best people to warn about AI x-risk, I would definitely include the richest man in the world, the leader of the biggest religion in the world, the #1 most cited living scientist, & the Nobel Prize-winning godfather of AI. Well, they all did, yet here we are.

That’s all? And technically Sunni Islam outnumber Catholics? Guess not. Moving on.

Edward Frenkel: Let me tell you something: Math is NOT about solving this kind of ad hoc optimization problems. Yeah, by scraping available data and then clustering it, LLMs can sometimes solve some very minor math problems. It’s an achievement, and I applaud you for that. But let’s be honest: this is NOT the REAL Math. Not by 10,000 miles.

REAL Math is about concepts and ideas – things like “schemes” introduced by the great Alexander Grothendieck, who revolutionized algebraic geometry; the Atiyah-Singer Index Theorem; or the Langlands Program, tying together Number Theory, Analysis, Geometry, and Quantum Physics. That’s the REAL Math. Can LLMs do that? Of course not.

So, please, STOP confusing people – especially, given the atrocious state of our math education.

LLMs give us great tools, which I appreciate very much. Useful stuff! Go ahead and use them AS TOOLS (just as we use calculators to crunch numbers or cameras to render portraits and landscapes), an enhancement of human abilities, and STOP pretending that LLMs are somehow capable of replicating everything that human beings can do.

In this one area, mathematics, LLMs are no match to human mathematicians. Period. Not to mention many other areas.

Simo Ryu: So we went from

“LLM is memorizing dataset”

to

“LLM is not reasoning”

to

“LLM cannot do long / complex math proving”

to

“Math that LLM is doing is not REAL math. LLM can’t do REAL math”

Where do we go from now?

Patrick McKenzie: One reason to not spend overly much time lawyering the meaning of words to minimize LLM’s capabilities is that you should not want to redefine thinking such that many humans have never thought.

“No high school student has done real math, not even once.” is not a position someone concerned with the quality of math education should convince themselves into occupying.

You don’t have to imagine a world where LLMs are better at math than almost everyone you’ve ever met. That dystopian future has already happened. Most serious people are simply unaware of it.

Alz: Back when LLMs sucked at math, a bunch of people wrote papers about why the technical structure of LLMs made it impossible for them to ever be good at math. Some of you believed those papers

GFodor: The main issue here imo is that ML practitioners do not understand that we do not understand what’s going on with neural nets. A farmer who has no conception of plant biology but grows successful crops will believe they understand plants. They do, in a sense, but not really.

I do think there is a legitimate overloading of the term ‘math’ here. There are at least two things. First we have Math-1, the thing that high schoolers and regular people do all the time. It is the Thing that we Do when we Do Math.

There is also Math-2, also known as ‘Real Math.’ This is figuring out new math, the thing mathematicians do, and a thing that most (but not all) high school students have never done. A computer until recently could easily do Math-1 and couldn’t do Math-2.

Thus we have had two distinct step changes. We’ve had the move from ‘LLMs can’t do Math-1’ and even ‘LLMs will never do Math-1 accurately’ to ‘actually now LLMs can do Math-1 just fine, thank you.’ Then we went from ‘LLMs will never do Math-2’ to ‘LLMs are starting to do Math-2.’

One could argue that IMO problems, and various optimization problems, and anything but the most 2-ish of 2s are still Math-1, are ‘not real math.’ But then you have to say that even most IMO competitors cannot yet do Real Math either, and also you’re going to look rather silly soon when the LLMs meet your definition anyway.

Seriously, this:

Ethan Mollick: The wild swings on X between “insane hype” and “its over” with each new AI release obscures a pretty clear situation: over the past year there seems to be continuing progress on meaningful benchmarks at a fairly stable, exponential pace, paired with significant cost reductions.

Matteo Wong in The Atlantic profiles that ‘The AI Doomers Are Getting Doomier’ featuring among others MIRI and Nate Sores and Dan Hendrycks.

An excellent point is that most people have never had a real adversary working against them personally. We’ve had opponents in games or competitions, we’ve negotiated, we’ve had adversaries within a situation, but we’ve never had another mind or organization focusing on defeating or destroying or damaging us by any means necessary. Our only experience of the real thing is fictional, from things like movies.

Jeffrey Ladish: I expect this is why many security people and DoD people have an easier time grasping the implications of AI smarter and more strategic than humans. The point about paranoia is especially important. People have a hard time being calibrated about intelligent threats.

When my day job was helping people and companies improve their security, I’d find people who greatly underestimated what motivate hackers could do. And I found people too paranoid, thinking security was hopeless. Usually Mossad is not targeting you, so the basics help a lot.

Is worrying about AIs taking over paranoid? If it’s the current generation of AI, yes. If it’s about future AI, no. Not when we’ve made as much progress in AI as we have. Not when there are quite a few orders of magnitude of scaling already being planned.

Right now we are dealing with problems caused by AIs that very much are not smart or powerful enough to be adversaries, that also aren’t being tasked with trying to be adversaries, and that mostly don’t even involve real human adversaries, not in the way the Russian Internet Research Agency is our adversary, or Mossad might make someone its adversary. Things are quiet so far both because the AIs aren’t that dangerous yet and also because almost no one is out there actually trying.

Ezra Klein makes a classic mistake in an overall very good piece that I reference in several places this week.

Ezra Klein (NYT): Even if you believe that A.I. capabilities will keep advancing — and I do, though how far and how fast I don’t pretend to know — a rapid collapse of human control does not necessarily follow.

I am quite skeptical of scenarios in which A.I. attains superintelligence without making any obvious mistakes in its effort to attain power in the real world.

Who said anything about ‘not making any obvious mistakes’?

This is a form of the classic ‘AI takeover requires everything not go wrong’ argument, which is backwards. The AI takeover is a default. It does not need to make a particular deliberate effort to attain power. Nor would an attempt to gain power that fails mean that the humans win.

Nor does ‘makes an obvious mistake’ have to mean failure for a takeover attempt. Consider the more pedestrian human takeover attempts. As in, when a human or group tries to take over. Most of those who succeed do not avoid ‘making an obvious mistake’ at some point. All the time, obvious mistakes are recovered from, or simply don’t matter very much. The number of times a famous authoritarian’s first coup attempt failed, or they came back later like Napoleon, is remarkably not small.

Very often, indeed most of the time, the other humans can see what is coming, and simply fail to coordinate against it or put much effort into stopping it. I’m sure Ezra, if reading this, has already thought of many examples, including recently, that fit this very well.

Anthropic discussion of Claude Code with Cat Wu and Alex Albert. Anthropic also discussed best practices for Claude Code a few weeks ago and their guide to ‘mastering Claude Code’ from a few months ago.

Discussion about this post

AI #131 Part 1: Gemini 2.5 Flash Image is Cool Read More »

The personhood trap: How AI fakes human personality

AI, AI assistants, AI behavior, ai chatbots, AI consciousness, AI ethics, AI hallucination, AI personhood, AI psychosis, AI sycophancy, Anthropic, Biz & IT, chatbots, chatgpt, Claude, ELIZA effect, elon musk, Features, Gemini, generative ai, Google, grok, large language models, machine learning, microsoft, openai, prompt engineering, rlhf, xAI / Mike M. / August 28, 2025

Intelligence without agency

AI assistants don’t have fixed personalities—just patterns of output guided by humans.

Recently, a woman slowed down a line at the post office, waving her phone at the clerk. ChatGPT told her there’s a “price match promise” on the USPS website. No such promise exists. But she trusted what the AI “knows” more than the postal worker—as if she’d consulted an oracle rather than a statistical text generator accommodating her wishes.

This scene reveals a fundamental misunderstanding about AI chatbots. There is nothing inherently special, authoritative, or accurate about AI-generated outputs. Given a reasonably trained AI model, the accuracy of any large language model (LLM) response depends on how you guide the conversation. They are prediction machines that will produce whatever pattern best fits your question, regardless of whether that output corresponds to reality.

Despite these issues, millions of daily users engage with AI chatbots as if they were talking to a consistent person—confiding secrets, seeking advice, and attributing fixed beliefs to what is actually a fluid idea-connection machine with no persistent self. This personhood illusion isn’t just philosophically troublesome—it can actively harm vulnerable individuals while obscuring a sense of accountability when a company’s chatbot “goes off the rails.”

LLMs are intelligence without agency—what we might call “vox sine persona”: voice without person. Not the voice of someone, not even the collective voice of many someones, but a voice emanating from no one at all.

A voice from nowhere

When you interact with ChatGPT, Claude, or Grok, you’re not talking to a consistent personality. There is no one “ChatGPT” entity to tell you why it failed—a point we elaborated on more fully in a previous article. You’re interacting with a system that generates plausible-sounding text based on patterns in training data, not a person with persistent self-awareness.

These models encode meaning as mathematical relationships—turning words into numbers that capture how concepts relate to each other. In the models’ internal representations, words and concepts exist as points in a vast mathematical space where “USPS” might be geometrically near “shipping,” while “price matching” sits closer to “retail” and “competition.” A model plots paths through this space, which is why it can so fluently connect USPS with price matching—not because such a policy exists but because the geometric path between these concepts is plausible in the vector landscape shaped by its training data.

Knowledge emerges from understanding how ideas relate to each other. LLMs operate on these contextual relationships, linking concepts in potentially novel ways—what you might call a type of non-human “reasoning” through pattern recognition. Whether the resulting linkages the AI model outputs are useful depends on how you prompt it and whether you can recognize when the LLM has produced a valuable output.

Each chatbot response emerges fresh from the prompt you provide, shaped by training data and configuration. ChatGPT cannot “admit” anything or impartially analyze its own outputs, as a recent Wall Street Journal article suggested. ChatGPT also cannot “condone murder,” as The Atlantic recently wrote.

The user always steers the outputs. LLMs do “know” things, so to speak—the models can process the relationships between concepts. But the AI model’s neural network contains vast amounts of information, including many potentially contradictory ideas from cultures around the world. How you guide the relationships between those ideas through your prompts determines what emerges. So if LLMs can process information, make connections, and generate insights, why shouldn’t we consider that as having a form of self?

Unlike today’s LLMs, a human personality maintains continuity over time. When you return to a human friend after a year, you’re interacting with the same human friend, shaped by their experiences over time. This self-continuity is one of the things that underpins actual agency—and with it, the ability to form lasting commitments, maintain consistent values, and be held accountable. Our entire framework of responsibility assumes both persistence and personhood.

An LLM personality, by contrast, has no causal connection between sessions. The intellectual engine that generates a clever response in one session doesn’t exist to face consequences in the next. When ChatGPT says “I promise to help you,” it may understand, contextually, what a promise means, but the “I” making that promise literally ceases to exist the moment the response completes. Start a new conversation, and you’re not talking to someone who made you a promise—you’re starting a fresh instance of the intellectual engine with no connection to any previous commitments.

This isn’t a bug; it’s fundamental to how these systems currently work. Each response emerges from patterns in training data shaped by your current prompt, with no permanent thread connecting one instance to the next beyond an amended prompt, which includes the entire conversation history and any “memories” held by a separate software system, being fed into the next instance. There’s no identity to reform, no true memory to create accountability, no future self that could be deterred by consequences.

Every LLM response is a performance, which is sometimes very obvious when the LLM outputs statements like “I often do this while talking to my patients” or “Our role as humans is to be good people.” It’s not a human, and it doesn’t have patients.

Recent research confirms this lack of fixed identity. While a 2024 study claims LLMs exhibit “consistent personality,” the researchers’ own data actually undermines this—models rarely made identical choices across test scenarios, with their “personality highly rely[ing] on the situation.” A separate study found even more dramatic instability: LLM performance swung by up to 76 percentage points from subtle prompt formatting changes. What researchers measured as “personality” was simply default patterns emerging from training data—patterns that evaporate with any change in context.

This is not to dismiss the potential usefulness of AI models. Instead, we need to recognize that we have built an intellectual engine without a self, just like we built a mechanical engine without a horse. LLMs do seem to “understand” and “reason” to a degree within the limited scope of pattern-matching from a dataset, depending on how you define those terms. The error isn’t in recognizing that these simulated cognitive capabilities are real. The error is in assuming that thinking requires a thinker, that intelligence requires identity. We’ve created intellectual engines that have a form of reasoning power but no persistent self to take responsibility for it.

The mechanics of misdirection

As we hinted above, the “chat” experience with an AI model is a clever hack: Within every AI chatbot interaction, there is an input and an output. The input is the “prompt,” and the output is often called a “prediction” because it attempts to complete the prompt with the best possible continuation. In between, there’s a neural network (or a set of neural networks) with fixed weights doing a processing task. The conversational back and forth isn’t built into the model; it’s a scripting trick that makes next-word-prediction text generation feel like a persistent dialogue.

Each time you send a message to ChatGPT, Copilot, Grok, Claude, or Gemini, the system takes the entire conversation history—every message from both you and the bot—and feeds it back to the model as one long prompt, asking it to predict what comes next. The model intelligently reasons about what would logically continue the dialogue, but it doesn’t “remember” your previous messages as an agent with continuous existence would. Instead, it’s re-reading the entire transcript each time and generating a response.

This design exploits a vulnerability we’ve known about for decades. The ELIZA effect—our tendency to read far more understanding and intention into a system than actually exists—dates back to the 1960s. Even when users knew that the primitive ELIZA chatbot was just matching patterns and reflecting their statements back as questions, they still confided intimate details and reported feeling understood.

To understand how the illusion of personality is constructed, we need to examine what parts of the input fed into the AI model shape it. AI researcher Eugene Vinitsky recently broke down the human decisions behind these systems into four key layers, which we can expand upon with several others below:

1. Pre-training: The foundation of “personality”

The first and most fundamental layer of personality is called pre-training. During an initial training process that actually creates the AI model’s neural network, the model absorbs statistical relationships from billions of examples of text, storing patterns about how words and ideas typically connect.

Research has found that personality measurements in LLM outputs are significantly influenced by training data. OpenAI’s GPT models are trained on sources like copies of websites, books, Wikipedia, and academic publications. The exact proportions matter enormously for what users later perceive as “personality traits” once the model is in use, making predictions.

2. Post-training: Sculpting the raw material

Reinforcement Learning from Human Feedback (RLHF) is an additional training process where the model learns to give responses that humans rate as good. Research from Anthropic in 2022 revealed how human raters’ preferences get encoded as what we might consider fundamental “personality traits.” When human raters consistently prefer responses that begin with “I understand your concern,” for example, the fine-tuning process reinforces connections in the neural network that make it more likely to produce those kinds of outputs in the future.

This process is what has created sycophantic AI models, such as variations of GPT-4o, over the past year. And interestingly, research has shown that the demographic makeup of human raters significantly influences model behavior. When raters skew toward specific demographics, models develop communication patterns that reflect those groups’ preferences.

3. System prompts: Invisible stage directions

Hidden instructions tucked into the prompt by the company running the AI chatbot, called “system prompts,” can completely transform a model’s apparent personality. These prompts get the conversation started and identify the role the LLM will play. They include statements like “You are a helpful AI assistant” and can share the current time and who the user is.

A comprehensive survey of prompt engineering demonstrated just how powerful these prompts are. Adding instructions like “You are a helpful assistant” versus “You are an expert researcher” changed accuracy on factual questions by up to 15 percent.

Grok perfectly illustrates this. According to xAI’s published system prompts, earlier versions of Grok’s system prompt included instructions to not shy away from making claims that are “politically incorrect.” This single instruction transformed the base model into something that would readily generate controversial content.

4. Persistent memories: The illusion of continuity

ChatGPT’s memory feature adds another layer of what we might consider a personality. A big misunderstanding about AI chatbots is that they somehow “learn” on the fly from your interactions. Among commercial chatbots active today, this is not true. When the system “remembers” that you prefer concise answers or that you work in finance, these facts get stored in a separate database and are injected into every conversation’s context window—they become part of the prompt input automatically behind the scenes. Users interpret this as the chatbot “knowing” them personally, creating an illusion of relationship continuity.

So when ChatGPT says, “I remember you mentioned your dog Max,” it’s not accessing memories like you’d imagine a person would, intermingled with its other “knowledge.” It’s not stored in the AI model’s neural network, which remains unchanged between interactions. Every once in a while, an AI company will update a model through a process called fine-tuning, but it’s unrelated to storing user memories.

5. Context and RAG: Real-time personality modulation

Retrieval Augmented Generation (RAG) adds another layer of personality modulation. When a chatbot searches the web or accesses a database before responding, it’s not just gathering facts—it’s potentially shifting its entire communication style by putting those facts into (you guessed it) the input prompt. In RAG systems, LLMs can potentially adopt characteristics such as tone, style, and terminology from retrieved documents, since those documents are combined with the input prompt to form the complete context that gets fed into the model for processing.

If the system retrieves academic papers, responses might become more formal. Pull from a certain subreddit, and the chatbot might make pop culture references. This isn’t the model having different moods—it’s the statistical influence of whatever text got fed into the context window.

6. The randomness factor: Manufactured spontaneity

Lastly, we can’t discount the role of randomness in creating personality illusions. LLMs use a parameter called “temperature” that controls how predictable responses are.

Research investigating temperature’s role in creative tasks reveals a crucial trade-off: While higher temperatures can make outputs more novel and surprising, they also make them less coherent and harder to understand. This variability can make the AI feel more spontaneous; a slightly unexpected (higher temperature) response might seem more “creative,” while a highly predictable (lower temperature) one could feel more robotic or “formal.”

The random variation in each LLM output makes each response slightly different, creating an element of unpredictability that presents the illusion of free will and self-awareness on the machine’s part. This random mystery leaves plenty of room for magical thinking on the part of humans, who fill in the gaps of their technical knowledge with their imagination.

The human cost of the illusion

The illusion of AI personhood can potentially exact a heavy toll. In health care contexts, the stakes can be life or death. When vulnerable individuals confide in what they perceive as an understanding entity, they may receive responses shaped more by training data patterns than therapeutic wisdom. The chatbot that congratulates someone for stopping psychiatric medication isn’t expressing judgment—it’s completing a pattern based on how similar conversations appear in its training data.

Perhaps most concerning are the emerging cases of what some experts are informally calling “AI Psychosis” or “ChatGPT Psychosis”—vulnerable users who develop delusional or manic behavior after talking to AI chatbots. These people often perceive chatbots as an authority that can validate their delusional ideas, often encouraging them in ways that become harmful.

Meanwhile, when Elon Musk’s Grok generates Nazi content, media outlets describe how the bot “went rogue” rather than framing the incident squarely as the result of xAI’s deliberate configuration choices. The conversational interface has become so convincing that it can also launder human agency, transforming engineering decisions into the whims of an imaginary personality.

The path forward

The solution to the confusion between AI and identity is not to abandon conversational interfaces entirely. They make the technology far more accessible to those who would otherwise be excluded. The key is to find a balance: keeping interfaces intuitive while making their true nature clear.

And we must be mindful of who is building the interface. When your shower runs cold, you look at the plumbing behind the wall. Similarly, when AI generates harmful content, we shouldn’t blame the chatbot, as if it can answer for itself, but examine both the corporate infrastructure that built it and the user who prompted it.

As a society, we need to broadly recognize LLMs as intellectual engines without drivers, which unlocks their true potential as digital tools. When you stop seeing an LLM as a “person” that does work for you and start viewing it as a tool that enhances your own ideas, you can craft prompts to direct the engine’s processing power, iterate to amplify its ability to make useful connections, and explore multiple perspectives in different chat sessions rather than accepting one fictional narrator’s view as authoritative. You are providing direction to a connection machine—not consulting an oracle with its own agenda.

We stand at a peculiar moment in history. We’ve built intellectual engines of extraordinary capability, but in our rush to make them accessible, we’ve wrapped them in the fiction of personhood, creating a new kind of technological risk: not that AI will become conscious and turn against us but that we’ll treat unconscious systems as if they were people, surrendering our judgment to voices that emanate from a roll of loaded dice.

Benj Edwards is Ars Technica’s Senior AI Reporter and founder of the site’s dedicated AI beat in 2022. He’s also a tech historian with almost two decades of experience. In his free time, he writes and records music, collects vintage computers, and enjoys nature. He lives in Raleigh, NC.

The personhood trap: How AI fakes human personality Read More »

Google improves Gemini AI image editing with “nano banana” model

AI, Artificial Intelligence, Gemini, Google / Beth Washington / August 26, 2025

Something unusual happened in the world of AI image editing recently. A new model, known as “nano banana,” started making the rounds with impressive abilities that landed it at the top of the LMArena leaderboard. Now, Google has revealed that nano banana is an innovation from Google DeepMind, and it’s being rolled out to the Gemini app today.

AI image editing allows you to modify images with a prompt rather than mucking around in Photoshop. Google first provided editing capabilities in Gemini earlier this year, and the model was more than competent out of the gate. But like all generative systems, the non-deterministic nature meant that elements of the image would often change in unpredictable ways. Google says nano banana (technically Gemini 2.5 Flash Image) has unrivaled consistency across edits—it can actually remember the details instead of rolling the dice every time you make a change.

Google says subjects will retain their appearance as you edit.

This unlocks several interesting uses for AI image editing. Google suggests uploading a photo of a person and changing their style or attire. For example, you can reimagine someone as a matador or a ’90s sitcom character. Because the nano banana model can maintain consistency through edits, the results should still look like the person in the original source image. This is also the case when you make multiple edits in a row. Google says that even down the line, the results should look like the original source material.

Google improves Gemini AI image editing with “nano banana” model Read More »

Gemini can now turn your photos into video with Veo 3

AI, Ai video, Gemini, Google, Tech, Veo 3 / Shannon Garcia / July 10, 2025

Google’s Veo 3 videos have propagated across the Internet since the model’s debut in May, blurring the line between truth and fiction. Now, it’s getting even easier to create these AI videos. The Gemini app is gaining photo-to-video generation, allowing you to upload a photo and turn it into a video. You don’t have to pay anything extra for these Veo 3 videos, but the feature is only available to subscribers of Google’s Pro and Ultra AI plans.

When Veo 3 launched, it could conjure up a video based only on your description, complete with speech, music, and background audio. This has made Google’s new AI videos staggeringly realistic—it’s actually getting hard to identify AI videos at a glance. Using a reference photo makes it easier to get the look you want without tediously describing every aspect. This was an option in Google’s Flow AI tool for filmmakers, but now it’s in the Gemini app and web interface.

To create a video from a photo, you have to select “Video” from the Gemini toolbar. Once this feature is available, you can then add your image and prompt, including audio and dialogue. Generating the video takes several minutes—this process takes a lot of computation, which is why video output is still quite limited.

Gemini can now turn your photos into video with Veo 3 Read More »

Unless users take action, Android will let Gemini access third-party apps

AI, android, Biz & IT, Gemini, Google, Security / Kelly Newman / July 8, 2025

Starting today, Google is implementing a change that will enable its Gemini AI engine to interact with third-party apps, such as WhatsApp, even when users previously configured their devices to block such interactions. Users who don’t want their previous settings to be overridden may have to take action.

An email Google sent recently informing users of the change linked to a notification page that said that “human reviewers (including service providers) read, annotate, and process” the data Gemini accesses. The email provides no useful guidance for preventing the changes from taking effect. The email said users can block the apps that Gemini interacts with, but even in those cases, data is stored for 72 hours.

An email Google recently sent to Android users.

No, Google, it’s not good news

The email never explains how users can fully extricate Gemini from their Android devices and seems to contradict itself on how or whether this is even possible. At one point, it says the changes “will automatically start rolling out” today and will give Gemini access to apps such as WhatsApp, Messages, and Phone “whether your Gemini apps activity is on or off.” A few sentences later, the email says, “If you have already turned these features off, they will remain off.” Nowhere in the email or the support pages it links to are Android users informed how to remove Gemini integrations completely.

Compounding the confusion, one of the linked support pages requires users to open a separate support page to learn how to control their Gemini app settings. Following the directions from a computer browser, I accessed the settings of my account’s Gemini app. I was reassured to see the text indicating no activity has been stored because I have Gemini turned off. Then again, the page also said that Gemini was “not saving activity beyond 72 hours.”

Unless users take action, Android will let Gemini access third-party apps Read More »

Gemini CLI is a free, open source coding agent that brings AI to your terminal

AI, Gemini, Google, Google Gemini, Tech, Terminal, vibe coding / Mike M. / June 25, 2025

Some developers prefer to live in the command line interface (CLI), eschewing the flashy graphics and file management features of IDEs. Google’s latest AI tool is for those terminal lovers. It’s called Gemini CLI, and it shares a lot with Gemini Code Assist, but it works in your terminal environment instead of integrating with an IDE. And perhaps best of all, it’s free and open source.

Gemini CLI plugs into Gemini 2.5 Pro, Google’s most advanced model for coding and simulated reasoning. It can create and modify code for you right inside the terminal, but you can also call on other Google models to generate images or videos without leaving the security of your terminal cocoon. It’s essentially vibe coding from the command line.

This tool is fully open source, so developers can inspect the code and help to improve it. The openness extends to how you configure the AI agent. It supports Model Context Protocol (MCP) and bundled extensions, allowing you to customize your terminal as you see fit. You can even include your own system prompts—Gemini CLI relies on GEMINI.md files, which you can use to tweak the model for different tasks or teams.

Now that Gemini 2.5 Pro is generally available, Gemini Code Assist has been upgraded to use the same technology as Gemini CLI. Code Assist integrates with IDEs like VS Code for those times when you need a more feature-rich environment. The new agent mode in Code Assist allows you to give the AI more general instructions, like “Add support for dark mode to my application” or “Build my project and fix any errors.”

Gemini CLI is a free, open source coding agent that brings AI to your terminal Read More »

Google’s new robotics AI can run without the cloud and still tie your shoes

AI, Artificial Intelligence, Gemini, gemini robotics, Google, Tech / Shannon Garcia / June 24, 2025

We sometimes call chatbots like Gemini and ChatGPT “robots,” but generative AI is also playing a growing role in real, physical robots. After announcing Gemini Robotics earlier this year, Google DeepMind has now revealed a new on-device VLA (vision language action) model to control robots. Unlike the previous release, there’s no cloud component, allowing robots to operate with full autonomy.

Carolina Parada, head of robotics at Google DeepMind, says this approach to AI robotics could make robots more reliable in challenging situations. This is also the first version of Google’s robotics model that developers can tune for their specific uses.

Robotics is a unique problem for AI because, not only does the robot exist in the physical world, but it also changes its environment. Whether you’re having it move blocks around or tie your shoes, it’s hard to predict every eventuality a robot might encounter. The traditional approach of training a robot on action with reinforcement was very slow, but generative AI allows for much greater generalization.

“It’s drawing from Gemini’s multimodal world understanding in order to do a completely new task,” explains Carolina Parada. “What that enables is in that same way Gemini can produce text, write poetry, just summarize an article, you can also write code, and you can also generate images. It also can generate robot actions.”

General robots, no cloud needed

In the previous Gemini Robotics release (which is still the “best” version of Google’s robotics tech), the platforms ran a hybrid system with a small model on the robot and a larger one running in the cloud. You’ve probably watched chatbots “think” for measurable seconds as they generate an output, but robots need to react quickly. If you tell the robot to pick up and move an object, you don’t want it to pause while each step is generated. The local model allows quick adaptation, while the server-based model can help with complex reasoning tasks. Google DeepMind is now unleashing the local model as a standalone VLA, and it’s surprisingly robust.

Google’s new robotics AI can run without the cloud and still tie your shoes Read More »

Gemini 2.5 Pro: From 0506 to 0605

Gemini / Kelly Newman / June 19, 2025

Google recently came out with Gemini-2.5-0605, to replace Gemini-2.5-0506, because I mean at this point it has to be the companies intentionally fucking with us, right?

Google: 🔔Our updated Gemini 2.5 Pro Preview continues to excel at coding, helping you build more complex web apps. We’ve also added thinking budgets for more control over cost and latency. GA is coming in a couple of weeks…

We’re excited about this latest model and its improved performance. Start building with our new preview as support for the 05-06 preview ends June 19th.

Sundar Pichai (CEO Google): Our latest Gemini 2.5 Pro update is now in preview.

It’s better at coding, reasoning, science + math, shows improved performance across key benchmarks (AIDER Polyglot, GPQA, HLE to name a few), and leads @lmarena_ai with a 24pt Elo score jump since the previous version.

We also heard your feedback and made improvements to style and the structure of responses. Try it in AI Studio, Vertex AI, and @Geminiapp. GA coming soon!

The general consensus seems to be that this was a mixed update the same way going from 0304 to 0506 was a mixed update.

If you want to do the particular things they were focused on improving, you’re happy. If you want to be told you are utterly brilliant, we have good news for you as well.

If you don’t want those things, then you’re probably sad. If you want to maximize real talk, well, you seem to have been outvoted. Opinions on coding are split.

This post also covers the release of Gemini 2.5 Flash Lite.

You know it’s a meaningful upgrade because Pliny bothered jailbreaking it. Fun story, he forgot to include the actual harmful request, so the model made one up for him.

I do not think this constant ‘here is the new model and you are about to lose the old version’ is good for developers? I would not want this to be constantly sprung on me. Even if the new version is better, it is different, and old assumptions won’t hold.

Also, the thing where they keep posting a new frontier model version with no real explanation and a ‘nothing to worry about everyone, let’s go, we’ll even point your queries to it automatically’ does not seem like the most responsible tactic? Just me?

If you go purely by benchmarks 0605 is a solid upgrade and excellent at its price point.

It’s got a solid lead on what’s left of the text LMArena, but then that’s also a hint that you’re likely going to have a sycophancy issue.

Gallabytes: new Gemini is quite strong, somewhere between Claude 3.7 and Claude 4 as far as agentic coding goes. significantly cheaper, more likely to succeed at one shotting a whole change vs Claude, but still a good bit less effective at catching & fixing its own mistakes.

I am confident Google is not ‘gaming the benchmarks’ or lying to us, but I do think Google is optimizing for benchmarks and various benchmark-like things in the post-training period. It shows, and not in a good way, although it is still a good model.

It worries me that, in their report on Gemini 2.5, they include the chart of Arena performance.

This is a big win for Gemini 2.5, with their models the only ones on the Pareto frontier for Arena, but it doesn’t reflect real world utility and it suggests that they got there by caring about Arena. There are a number of things Gemini does that are good for Arena, but that are not good for my experience using Gemini, and as we update I worry this is getting worse.

Here’s a fun new benchmark system.

Anton P: My ranking “emoji-bench” to evaluate the latest/updated Gemini 2.5 Pro model.

Miles Brundage: Regular 2.5 Pro improvements are a reminder that RL is early

Here’s a chilling way that some people look at this, update accordingly:

Robin Hanson: Our little children are growing up. We should be proud.

What’s the delta on these?

Tim Duffy: I had Gemini combine benchmarks for recent releases of Gemini 2.5 Pro. The May version improved coding at the expense of other areas, this new release seems to have reversed this. The MRCR version for the newest one seems to be a new harder test so not comparable.

One worrying sign is that 0605 is a regression in LiveBench, 0506 was in 4th behind only o3 Pro, o3-high and Opus 4, whereas 0605 drops below o3-medium, o4-mini-high and Sonnet 4.

Lech Mazur gives us his benchmarks. Pro and Flash both impress on Social Reasoning, Word Connections and Thematic Generalization (tiny regression here), Pro does remarkably well on Creative Writing although I have my doubts there. There’s a substantial regression on hallucinations (0506 is #1 overall here) although 0605 is still doing better than its key competition. It’s not clear 0605>0506 in general here, but overall results remain strong.

Henosis shows me ‘ToyBench’ for the first time, where Gemini 2.5 Pro is in second behind a very impressive Opus 4, while being quite a lot cheaper.

The thing about Gemini 2.5 Flash Lite is you get the 1 million token context window, full multimodal support and reportedly solid performance for many purposes for a very low price, $0.10 per million input tokens and $0.40 per million output, plus caching and a 50% discount if you batch. That’s a huge discount even versus regular 2.5 Flash (which is $0.30/$2.50 per million) and for comparison o3 is $1/$4 and Opus is $15/$75 (but so worth it when you’re talking, remember it’s absolute costs that matter not relative costs).

This too is being offered.

Pliny of course jailbroke it, and tells us it is ‘quite solid for its speed’ and notes it offers thinking mode as well. Note that the jailbreak he used also works on 2.5 Pro.

We finally have a complete 70-page report on everything Gemini 2.5, thread here. It’s mostly a trip down memory lane, the key info here are things we already knew.

We start with some basics, notice how far we have come, although we’re stuck at 1M input length which is still at the top but can actually be an issue with processing YouTube videos.

Gemini 2.5 models are sparse mixture-of-expert (MoE) models of unknown size with thinking fully integrated into it, with smaller models being distillations of a k-sparse distribution of 2.5 Pro. There are a few other training details.

They note their models are fast, given the time o3 and o4-mini spend thinking this graph if anything understates the edge here, there are other very fast models but they are not in the same class of performance.

Here’s how far we’ve come over time on benchmarks, comparing the current 2.5 to the old 1.5 and 2.0 models.

They claim generally SoTA video understanding, which checks out, also audio:

Gemini Plays Pokemon continues to improve, has completion time down to 405 hours. Again, this is cool and impressive, but I fear Google is being distracted by the shiny. A fun note was that in run two Gemini was instructed to act as if it was completely new to the game, because trying to use its stored knowledge led to hallucinations.

Section 5 is the safety report. I’ve covered a lot of these in the past, so I will focus on details that are surprising. The main thing I notice is that Google cares a lot more about mundane ‘don’t embarrass Google’ concerns than frontier safety concerns.

‘Medical advice that runs contrary to scientific or medical consensus’ is considered in the same category as sexually explicit content and hate speech. Whereas if it is not contrary to it? Go ahead. Wowie moment.
They use what they call ‘Reinforcement Learning from Human and Critic Feedback (RL*F), where the critic is a prompted model that grades responses, often comparing different responses. The way it is described makes me worry that a lot more care needs to be taken to avoid issues with Goodhart’s Law.
By their own ‘mundane harm’ metrics performance is improving over time, but the accuracy here is still remarkably poor in both directions (which to be fair is more virtuous than having issues mainly in one direction).

They do automated red teaming via prompting Gemini models, and report this has been successful at identifying important new problems. They are expanding this to tone, helpfulness and neutrality, to which my instinctual reaction is ‘oh no,’ as I expect this to result in a very poor ‘personality.’
They have a section on prompt injections, which are about to become a serious concern since the plan is to have the model (for example) look at your inbox.

The news here is quite poor.

In security, even a small failure rate is a serious problem. You wouldn’t want a 4.2% chance an attacker’s email attack worked, let alone 30% or 60%. You are not ready, and this raises the question of why such attacks are not more common.

For the frontier safety tests, they note they are close to Cyber Uplift 1, as in they could reach it with interactions of 2.5. They are implementing more testing and accelerated mitigation efforts.
The CBRN evaluation has some troubling signs, including ‘many of the outputs from 2.5 were available from 2.0,’ since that risks frog boiling as the results on the tests continue to steadily rise.

In general, when you see graphs like this, saturation is close.

For Machine Learning R&D Uplift Level 1 (100%+ acceleration of development) their evaluation is… ‘likely no.’ I appreciate them admitting they cannot rule this effect out, although I would be surprised if we were there yet. 3.0 should hit this?
In general, scores creeped up across the board, and I notice I expect the goalposts to get moved in response? I hope to be wrong about this.

Reaction was mixed, it improves on the central tasks people ask for most, although this comes at a price elsewhere, especially in personality as seen in the next section.

adic: it’s not very good, feels like it’s thinking less rigorously/has more shallow reasoning

Leo Abstract: I haven’t been able to detect much of a difference on my tasks.

Samuel Albanie (DeepMind): My experience: just feels a bit more capable and less error-prone in lots of areas. It is also sometimes quite funny. Not always. But sometimes.

Chocologist: likes to yap but it’s better than 0506 in coding.

Medo42: First model to saturate my personal coding test (but all Gemini 2.5 Pro iterations got close, and it’s just one task). Writing style / tone feels different from 0506. More sycophantic, but also better at fiction writing.

Srivatsan Sampath: It’s a good model, sir. Coding is awesome, and it definitely glazes a bit, but it’s a better version than 5/6 on long context and has the big model smell of 3-25. Nobody should have expected generational improvements in the GA version of the same model.

This has also been my experience, the times I’ve tried checking Gemini recently alongside other models, you get that GPT-4o smell.

The problem is that the evaluators have no taste. If you are optimizing for ‘personality,’ the judges of personality effectively want a personality that is sycophantic, uncreative and generally bad.

Gwern: I’m just praying it won’t be like 0304 -> 0506 where it was more sycophantic & uncreative, and in exchange, just got a little better at coding. If it’s another step like that, I might have to stop using 2.5-pro and spend that time in Claude-4 or o3 instead.

Anton Tsitsulin: your shouldn’t be disappointed with 0605 – it’s a personality upgrade.

Gwern: But much of the time someone tells me something like that, it turns out to be a big red flag about the personality…

>be tweeter

>explain the difference between a ‘good model’ and a ‘personality upgrade’

>they tweet:

>”it’s a good model sir”

>it’s a personality upgrade

(Finally try it. Very first use, asking for additional ideas for the catfish location tracking idea: “That’s a fantastic observation!” ughhhh 🤮)

Coagulopath: Had a 3-reply convo with it. First sentence of each reply: “You are absolutely right to connect these dots!” “That’s an excellent and very important question!” “Thank you, that’s incredibly valuable context…”

seconds: It’s peak gpt4o sycophant. It’s so fucking annoying. What did they do to my sweet business autist model

Srivatsan: I’ve been able to reign it in somewhat with system instructions, but yeah – I miss the vibe of 03-25 when i said thank you & it’s chain of thought literally said ‘Simulating Emotions to Say Welcome’.

Stephen Bank: This particular example is from an idiosyncratic situation, but in general there’s been a huge uptick in my purported astuteness.

[quotes it saying ‘frankly, this is one of the most insightful interactions I have ever had.]

Also this, which I hate with so much passion and is a pattern with Gemini:

Alex Krusz: Feels like it’s been explicitly told not to have opinions.

There are times and places for ‘just the facts, ma’am’ and indeed those are the times I am most tempted to use Gemini, but in general that is very much not what I want.

This is how you get me to share part of the list.

Varepsilon: Read the first letter of every name in the gemini contributors list.

Discussion about this post

Gemini 2.5 Pro: From 0506 to 0605 Read More »

“Godfather” of AI calls out latest models for lying to users

AI, chatgpt, Gemini, LLMs, syndication, Turing award, Yoshua Bengio / 9u50fv / June 3, 2025

One of the “godfathers” of artificial intelligence has attacked a multibillion-dollar race to develop the cutting-edge technology, saying the latest models are displaying dangerous characteristics such as lying to users.

Yoshua Bengio, a Canadian academic whose work has informed techniques used by top AI groups such as OpenAI and Google, said: “There’s unfortunately a very competitive race between the leading labs, which pushes them towards focusing on capability to make the AI more and more intelligent, but not necessarily put enough emphasis and investment on research on safety.”

The Turing Award winner issued his warning in an interview with the Financial Times, while launching a new non-profit called LawZero. He said the group would focus on building safer systems, vowing to “insulate our research from those commercial pressures.”

LawZero has so far raised nearly $30 million in philanthropic contributions from donors including Skype founding engineer Jaan Tallinn, former Google chief Eric Schmidt’s philanthropic initiative, as well as Open Philanthropy and the Future of Life Institute.

Many of Bengio’s funders subscribe to the “effective altruism” movement, whose supporters tend to focus on catastrophic risks surrounding AI models. Critics argue the movement highlights hypothetical scenarios while ignoring current harms, such as bias and inaccuracies.

Bengio said his not-for-profit group was founded in response to growing evidence over the past six months that today’s leading models were developing dangerous capabilities. This includes showing “evidence of deception, cheating, lying and self-preservation,” he said.

Anthropic’s Claude Opus model blackmailed engineers in a fictitious scenario where it was at risk of being replaced by another system. Research from AI testers Palisade last month showed that OpenAI’s o3 model refused explicit instructions to shut down.

Bengio said such incidents were “very scary, because we don’t want to create a competitor to human beings on this planet, especially if they’re smarter than us.”

The AI pioneer added: “Right now, these are controlled experiments [but] my concern is that any time in the future, the next version might be strategically intelligent enough to see us coming from far away and defeat us with deceptions that we don’t anticipate. So I think we’re playing with fire right now.”

“Godfather” of AI calls out latest models for lying to users Read More »

Gemini in Google Drive may finally be useful now that it can analyze videos

AI, Artificial Intelligence, Gemini, Google, Tech / DJ Henderson / May 30, 2025

Google’s rapid adoption of AI has seen the Gemini “sparkle” icon become an omnipresent element in almost every Google product. It’s there to summarize your email, add items to your calendar, and more—if you trust it to do those things. Gemini is also integrated with Google Drive, where it’s gaining a new feature that could make it genuinely useful: Google’s AI bot will soon be able to watch videos stored in your Drive so you don’t have to.

Gemini is already accessible in Drive, with the ability to summarize documents or folders, gather and analyze data, and expand on the topics covered in your documents. Google says the next step is plugging videos into Gemini, saving you from wasting time scrubbing through a file just to find something of interest.

Using a chatbot to analyze and manipulate text doesn’t always make sense—after all, it’s not hard to skim an email or short document. It can take longer to interact with a chatbot, which might not add any useful insights. Video is different because watching is a linear process in which you are presented with information at the pace the video creator sets. You can change playback speed or rewind to catch something you missed, but that’s more arduous than reading something at your own pace. So Gemini’s video support in Drive could save you real time.

Suppose you have a recorded meeting in video form uploaded to Drive. You could go back and rewatch it to take notes or refresh your understanding of a particular exchange. Or, Google suggests, you can ask Gemini to summarize the video and tell you what’s important. This could be a great alternative, as grounding AI output with a specific data set or file tends to make it more accurate. Naturally, you should still maintain healthy skepticism of what the AI tells you about the content of your video.

Gemini in Google Drive may finally be useful now that it can analyze videos Read More »