Mike M. – Page 61

To pee or not to pee? That is a question for the bladder—and the brain

health, Science, syndication, urine / Mike M. / June 2, 2024

💦 —

The basic urge to pee is surprisingly complex and can go awry as we age.

Emily Underwood, Knowable Magazine – Jun 2, 2024 11: 27 am UTC

Cut view of man covering urine with hands. He has some pain and problem. Isolated on striped and blue background

You’re driving somewhere, eyes on the road, when you start to feel a tingling sensation in your lower abdomen. That extra-large Coke you drank an hour ago has made its way through your kidneys into your bladder. “Time to pull over,” you think, scanning for an exit ramp.

To most people, pulling into a highway rest stop is a profoundly mundane experience. But not to neuroscientist Rita Valentino, who has studied how the brain senses, interprets, and acts on the bladder’s signals. She’s fascinated by the brain’s ability to take in sensations from the bladder, combine them with signals from outside of the body, like the sights and sounds of the road, then use that information to act—in this scenario, to find a safe, socially appropriate place to pee. “To me, it’s really an example of one of the beautiful things that the brain does,” she says.

Scientists used to think that our bladders were ruled by a relatively straightforward reflex—an “on-off” switch between storing urine and letting it go. “Now we realize it’s much more complex than that,” says Valentino, now director of the division of neuroscience and behavior at the National Institute of Drug Abuse. An intricate network of brain regions that contribute to functions like decision-making, social interactions, and awareness of our body’s internal state, also called interoception, participates in making the call.

In addition to being mind-bogglingly complex, the system is also delicate. Scientists estimate, for example, that more than 1 in 10 adults have overactive bladder syndrome—a common constellation of symptoms that includes urinary urgency (the sensation of needing to pee even when the bladder isn’t full), nocturia (the need for frequent nightly bathroom visits) and incontinence. Although existing treatments can improve symptoms for some, they don’t work for many people, says Martin Michel, a pharmacologist at Johannes Gutenberg University in Mainz, Germany, who researches therapies for bladder disorders. Developing better drugs has proven so challenging that all major pharmaceutical companies have abandoned the effort, he adds.

Recently, however, a surge of new research is opening the field to fresh hypotheses and treatment approaches. Although therapies for bladder disorders have historically focused on the bladder itself, the new studies point to the brain as another potential target, says Valentino. Combined with studies aimed at explaining why certain groups, such as post-menopausal women, are more prone to bladder problems, the research suggests that we shouldn’t simply accept symptoms like incontinence as inevitable, says Indira Mysorekar, a microbiologist at Baylor College of Medicine in Houston. We’re often told such problems are just part of getting old, particularly for women—“and that’s true to some extent,” she says. But many common issues are avoidable and can be treated successfully, she says: “We don’t have to live with pain or discomfort.”

A delicate balance

The human bladder is, at the most basic level, a stretchy bag. To fill to capacity—a volume of 400 to 500 milliliters (about 2 cups) of urine in most healthy adults—it must undergo one of the most extreme expansions of any organ in the human body, expanding roughly sixfold from its wrinkled, empty state.

To stretch that far, the smooth muscle wall that wraps around the bladder, called the detrusor, must relax. Simultaneously, sphincter muscles that surround the bladder’s lower opening, or urethra, must contract, in what scientists call the guarding reflex.

Enlarge / It’s not just sensory neurons (purple) that can detect stretch, pressure, pain and other sensations in the bladder. Other types of cells, like the umbrella-shaped cells that form the urothelium’s barrier against urine, can also sense and respond to mechanical forces — for example, by releasing chemical signaling molecules such as adenosine triphosphate (ATP) as the organ expands to fill with urine.

Filling or full, the bladder spends more than 95 percent of its time in storage mode, allowing us to carry out our daily activities without leaks. At some point—ideally, when we decide it’s time to pee—the organ switches from storage to release mode. For this, the detrusor muscle must contract forcefully to expel urine, while the sphincter muscles surrounding the urethra simultaneously relax to let urine flow out.

For a century, physiologists have puzzled over how the body coordinates the switch between storage and release. In the 1920s, a surgeon named Frederick Barrington, of University College London, went looking for the on-off switch in the brainstem, the lowermost part of the brain that connects with the spinal cord.

Working with sedated cats, Barrington used an electrified needle to damage slightly different areas in the pons, part of the brainstem that handles vital functions like sleeping and breathing. When the cats recovered, Barrington noticed that some demonstrated a desire to urinate—by scratching, circling, or squatting—but were unable to voluntarily go. Meanwhile, cats with lesions in a different part of the pons seemed to have lost any awareness of the need to urinate, peeing at random times and appearing startled whenever it happened. Clearly, the pons served as an important command center for urinary function, telling the bladder when to release urine.

To pee or not to pee? That is a question for the bladder—and the brain Read More »

Is a colonial-era drop in CO₂ tied to regrowing forests?

carbon dioxide, climate change, colonization, Earth science, Ice cores, reforestation, Science / Mike M. / June 2, 2024

More trees, less carbon —

Carbon dioxide dropped after colonial contact wiped out Native Americans.

Howard Lee – Jun 2, 2024 11: 00 am UTC

Image of a transparent disk against a blue background. The disk has lots of air bubbles embedded in it. — Enlarge / A slice through an ice core showing bubbles of trapped air.

British Antarctic Survey

Did the massive scale of death in the Americas following colonial contact in the 1500s affect atmospheric CO₂ levels? That’s a question scientists have debated over the last 30 years, ever since they noticed a sharp drop in CO₂ around the year 1610 in air preserved in Antarctic ice.

That drop in atmospheric CO₂ levels is the only significant decline in recent millennia, and scientists suggested that it was caused by reforestation in the Americas, which resulted from their depopulation via pandemics unleashed by early European contact. It is so distinct that it was proposed as a candidate for the marker of the beginning of a new geological epoch—the “Anthropocene.”

But the record from that ice core, taken at Law Dome in East Antarctica, shows that CO₂ starts declining a bit late to match European contact, and it plummets over just 90 years, which is too drastic for feasible rates of vegetation regrowth. A different ice core, drilled in the West Antarctic, showed a more gradual decline starting earlier, but lacked the fine detail of the Law Dome ice.

Which one was right? Beyond the historical interest, it matters because it is a real-world, continent-scale test of reforestation’s effectiveness at removing CO₂ from the atmosphere.

In a recent study, Amy King of the British Antarctic Survey and colleagues set out to test if the Law Dome data is a true reflection of atmospheric CO₂ decline, using a new ice core drilled on the “Skytrain Ice Rise” in West Antarctica.

Precious tiny bubbles

In 2018, scientists and engineers from the British Antarctic Survey and the University of Cambridge drilled the ice core, a cylinder of ice 651 meters long by 10 centimeters in diameter (2,136 feet by 4 inches), from the surface down to the bedrock. The ice contains bubbles of air that got trapped as snow fell, forming tiny capsules of past atmospheres.

The project’s main aim was to investigate ice from the time about 125,000 years ago when the climate was about as warm as it is today. But King and colleagues realized that the younger portion of ice could shed light on the 1610 CO₂decline.

“Given the resolution of what we could obtain with Skytrain Ice Rise, we predicted that, if the drop was real in the atmosphere as in Law Dome, we should see the drop in Skytrain, too,” said Thomas Bauska of the British Antarctic Survey, a co-author of the new study.

The ice core was cut into 80-centimeter (31-inch) lengths, put into insulated boxes, and shipped to the UK, all the while held at -20°C (-4°F) to prevent it from melting and releasing its precious cargo of air from millennia ago. “That’s one thing that keeps us up at night, especially as gas people,” said Bauska.

In the UK they took a series of samples across 31 depth intervals spanning the period from 1454 to 1688 CE: “We went in and sliced and diced our ice core as much as we could,” said Bauska. They sent the samples, still refrigerated, off to Oregon State University where the CO₂ levels were measured.

The results didn’t show a sharp drop of CO₂—instead, they showed a gentler CO₂ decline of about 8 ppm over 157 years between 1516 and 1670 CE, matching the other West Antarctic ice core.

“We didn’t see the drop,” said Bauska, “so we had to say, OK, is our understanding of how smooth the records are accurate?”

A tent on the Antarctic ice where the core is cut into segments for shipping.

British Antarctic Survey

To test if the Skytrain ice record is too blurry to show a sharp 1610 drop, they analyzed the levels of methane in the ice. Because methane is much less soluble in water than CO₂, they were able to melt continuously along the ice core to liberate the methane and get a more detailed graph of its concentration than was possible for CO₂. If the atmospheric signal was blurred in Skytrain, it should have smoothed the methane record. But it didn’t.

“We didn’t see that really smoothed out methane record,” said Bauska, “which then told us the CO₂ record couldn’t have been that smoothed.”

In other words, the gentler Skytrain CO₂ signal is real, not an artifact.

Does this mean the sharp drop at 1610 in the Law Dome data is an artifact? It looks that way, but Bauska was cautious, saying, “the jury will still be out until we actually get either re-measurements of the Law Dome, or another ice core drilled with a similarly high accumulation.”

Is a colonial-era drop in CO₂ tied to regrowing forests? Read More »

Boeing’s Starliner test flight scrubbed again after hold in final countdown

atlas v, Boeing, commercial crew, human spaceflight, launch, NASA, Science, Space, starliner, united launch alliance / Mike M. / June 2, 2024

Hold Hold Hold —

The ground launch sequencer computer called a hold at T-minus 3 minutes, 50 seconds.

Stephen Clark – Jun 1, 2024 10: 47 pm UTC

Enlarge / NASA commander Butch Wilmore exits the Starliner spacecraft Saturday following the scrubbed launch attempt.

A computer controlling the Atlas V rocket’s countdown triggered an automatic hold less than four minutes prior to liftoff of Boeing’s commercial Starliner spacecraft Saturday, keeping the crew test flight on the ground at least a few more days.

NASA astronauts Butch Wilmore and Suni Williams were already aboard the spacecraft when the countdown stopped due to a problem with a ground computer. “Hold. Hold. Hold,” a member of Atlas V launch team called out on an audio feed.

With the hold, the mission missed an instantaneous launch opportunity at 12: 25 pm EDT (16: 25 UTC), and later Saturday, NASA announced teams will forego a launch opportunity Sunday. The next chance to send Starliner into orbit will be 10: 52 am EDT (14: 52 UTC) Wednesday. The mission has one launch opportunity every one-to-two days, when the International Space Station’s orbital track moves back into proper alignment with the Atlas V rocket’s launch pad in Florida.

Wilmore and Williams will take the Starliner spacecraft on its first crew flight into low-Earth orbit. The capsule will dock with the International Space Station around a day after launch, spend at least a week there, then return to a parachute-assisted landing at one of two landing zones in New Mexico or Arizona. Once operational, Boeing’s Starliner will join SpaceX’s Crew Dragon capsule to give NASA two independent human-rated spacecraft for transporting astronauts to and from the space station.

It’s been a long road to get here with the Starliner spacecraft, and there’s more work to do before the capsule’s long-delayed first flight with astronauts.

Technicians from United Launch Alliance, builder of the Atlas V rocket, will begin troubleshooting the computer glitch at the launch pad Saturday evening, after draining propellant from the launch vehicle. Early indications suggest that a card in one of three computers governing the final minutes of the Atlas V’s countdown didn’t boot up as quickly as anticipated.

“You can imagine a large rack that is a big computer where the functions of the computer as a controller are broken up separately into individual cards or printed wire circuit boards with their logic devices,” said Tory Bruno, ULA’s president and CEO. “They’re all standalone, but together it’s an integrated controller.”

The computers are located at the launch pad inside a shelter near the base of the Atlas V rocket at Cape Canaveral Space Force Station. All three computers must be fully functioning in the final phase of the countdown to ensure triple redundancy. At the moment of liftoff, these computers control things like retracting umbilical lines and releasing bolts holding the rocket to its mobile launch platform.

Two of the computers activated as the final countdown sequence began at T-minus 4 minutes. A single card in the third computer took about six more seconds to come online, although it did boot up eventually, Bruno said.

“Two came up normally and the third one came up, but it was slow to come up, and that tripped a red line,” he said.

A disappointment

Wilmore and Williams, both veteran astronauts and former US Navy test pilots, exited the Starliner spacecraft with the help of Boeing’s ground team. They returned to NASA crew quarters at the nearby Kennedy Space Center to wait for the next launch attempt.

The schedule for the next try will depend on what ULA workers find when they access the computers at the launch pad. Officials initially said they could start another launch countdown early Sunday if they found a simple solution to the computer problem, such as swapping out a faulty card. The computers are networked together, but the architecture is designed with replaceable cards, each responsible for different functions during the countdown, to allow for a quick fix without having to replace the entire unit, Bruno said.

United Launch Alliance's Atlas V rocket and Boeing's Starliner spacecraft at Cape Canaveral Space Force Station, Florida. — Enlarge / United Launch Alliance’s Atlas V rocket and Boeing’s Starliner spacecraft at Cape Canaveral Space Force Station, Florida.

Later Saturday, NASA announced the launch won’t happen Sunday, giving teams additional time to assess the computer issue. The next launch opportunities are Wednesday and Thursday.

Bruno said ULA’s engineers suspect a hardware problem or a network communication glitch caused the computer issue during Saturday’s countdown. That is what ULA’s troubleshooting team will try to determine overnight. NASA said officials will share another update Sunday.

If it doesn’t get off the ground by Thursday, the Starliner test flight could face a longer delay to allow time for ULA to change out limited-life batteries on the Atlas V rocket. Bruno said the battery swap would take about 10 days.

Saturday’s aborted countdown was the latest in a string of delays for Boeing’s Starliner program. The spacecraft’s first crew test flight is running seven years behind the schedule Boeing announced when NASA awarded the company a $4.2 billion contract for the crew capsule in 2014. Put another way, Boeing has arrived at this moment nine years after the company originally said the spacecraft could be operational, when the program was first announced in 2010.

“Of course, this is emotionally disappointing,” said Mike Fincke, a NASA astronaut and a backup to Wilmore and Williams on the crew test flight. “I know Butch and Suni didn’t sound disappointed when we heard them on the loops, and it’s because it comes back to professionalism.”

NASA and Boeing were on the cusp of launching the Starliner test flight May 6, but officials called off the launch attempt due to a valve problem on the Atlas V rocket. Engineers later discovered a helium leak on the Starliner spacecraft’s service module, but managers agreed to proceed with the launch Saturday if the leak did not worsen during the countdown.

A check of the helium system Saturday morning showed the leak rate had decreased from a prior measurement, and it was no longer a constraint to launch. Instead, a different problem emerged to keep Starliner on Earth.

“Everybody is a little disappointed, but you kind of roll your sleeves up and get right back to work,” said Steve Stich, manager of NASA’s commercial crew program.

Boeing’s Starliner test flight scrubbed again after hold in final countdown Read More »

Here’s why a Japanese billionaire just canceled his lunar flight on Starship

dearmoon, Science, Space, spacex / Mike M. / June 1, 2024

No Moon —

“I feel terrible making the crew members wait longer.”

Eric Berger – Jun 1, 2024 2: 10 pm UTC

Enlarge / Elon Musk speaks as Yusaku Maezawa, founder and president of Start Today Co., looks on at an event at the SpaceX headquarters in Hawthorne, California, in 2018.

Patrick T. Fallon/Bloomberg via Getty Images

On Friday night the dearMoon project—a plan to launch a Japanese billionaire and 10 other ‘crew members’ on a circumlunar flight aboard SpaceX’s Starship vehicle—was abruptly canceled.

“It is unfortunate to be announcing that ‘dearMoon’, the first private circumlunar flight project, will be cancelled,” the mission’s official account on the social media site X said. “We thank everyone who has supported us and apologize to those who have looked forward to this project.”

Shortly afterward the financial backer of the project and its ‘crew leader,’ Yusaku Maezawa, explained this decision on X. When Maezawa agreed to the mission in 2018, he said, the assumption was that the dearMoon mission would launch by the end of 2023.

“It’s a developmental project so it is what it is, but it is still uncertain as to when Starship can launch,” he wrote. “I can’t plan my future in this situation, and I feel terrible making the crew members wait longer, hence the difficult decision to cancel at this point in time. I apologize to those who were excited for this project to happen.”

The mission was to be Starship’s first human spaceflight to launch from Earth, fly around the Moon, and come back. Now, it’s not happening. Why did this happen, and what does it mean?

Origins of the mission

Maezawa and Musk made the announcement, side by side, at SpaceX’s rocket factory in Hawthorne in September 2018. It was something of an odd but important moment. It seemed significant that SpaceX was signing its first commercial contract for the massive Starship rocket. And while the value was not disclosed, Maezawa was injecting something on the order of the low hundreds of millions of dollars into the program.

Maezawa, however, always came off as a bit non-serious. He said he would hold a competition to fill 10 other seats on board the vehicle. “I did not want to have such a fantastic experience by myself,” he said. “I would be a little lonely.” Later, he did select a crew of creative people.

Initially, however, Maezawa did take the project seriously. When I watched the very first Starship hop test in July 2019, there were only a handful of visitors on hand to view the brief flight of “Starhopper.” One of them was a representative of Maezawa who was keeping close tabs on the progress of Starship.

As big space projects do—and to the surprise of no one—Starship ran behind in its development. The first test flight did not occur until April 2023, and that was just the beginning. The dearMoon mission lay at the very end of a long line of tests that the vehicle must complete: safe launch, controlled flight in space, safe landing of the Starship upper stage, in-space refueling, habitability in space, and much more.

With the fourth test flight of Starship coming in a few days, as early as June 5, SpaceX has so far demonstrated the ability to safely launch Starship. So it remains at the beginning of a challenging technical journey.

A turning point

One of the biggest impacts to the dearMoon project came in April 2021, when NASA selected the Starship vehicle as the lunar lander for its Artemis Program. This put the large vehicle on the critical path for NASA’s ambitious program to land humans on the surface of the Moon. It also offered an order of magnitude more funding, $2.9 billion, and the promise of more if SpaceX could deliver a vehicle to take humans down to the Moon’s surface from lunar orbit, and back.

Since then SpaceX has had two clear priorities for its Starship program. The first of these is to become operational, and begin deploying larger Starlink satellites. And the second is to use these flights to test technologies needed for NASA’s Artemis Program, such as in-space propellant storage and refueling.

As a result other aspects of the program, including dearMoon, were deprioritized. In recent months it became clear that if Maezawa’s mission happened, it would not occur until at least the early 2030s—at least a decade after the original plan.

Changing fortunes

In the meantime, Maezawa’s priorities also likely changed. According to Forbes, when the plan was announced in 2018, the entrepreneur had a net worth of about $3 billion. Today he is estimated to be worth only half of that. Additionally, he scratched his itch to go to space in 2021, flying aboard a Russian Soyuz vehicle for a 12-day trip to the International Space Station.

The writing has been on the wall for a while about Maezawa, since SpaceX founder Elon Musk unfollowed the Japanese entrepreneur on X earlier this year. (This is a sure sign of his disfavor. Musk has unfollowed me twice on Twitter/X after stories or interactions he did not like.) It is probable that the combination of developmental delays and Maezawa’s personal fortunes led the parties to disband the project.

This all leaves a clearer road ahead for Starship: Become operational, start flying Starlink satellites, and begin ticking off the technical challenges for Artemis. Then, several years from now, the company will turn its attention toward the challenging prospect of launching humans inside Starship from Earth, and then landing back on the planet. The first of these people will be another billionaire, Jared Isaacman, who has already flown on Crew Dragon and plans at least two more such flights before the pioneering Starship mission.

Here’s why a Japanese billionaire just canceled his lunar flight on Starship Read More »

FDA’s review of MDMA for PTSD highlights study bias and safety concerns

bias, clinical trial, fda, health, mdma, PTSD, Science / Mike M. / June 1, 2024

Complicated —

FDA advisors will meet June 4 to discuss and vote on the therapy’s effectiveness.

Beth Mole – May 31, 2024 11: 46 pm UTC

MDMA is now in the FDA's hands. — Enlarge / MDMA is now in the FDA’s hands.

The safety and efficacy data on the use of MDMA (aka ecstasy) for post-traumatic stress disorder therapy is “challenging to interpret,” the Food and Drug Administration said in a briefing document posted Friday. The agency noted significant flaws in the design of the underlying clinical trials as well as safety concerns for the drug, particularly cardiovascular harms.

On Tuesday, June 4, the FDA will convene an advisory committee that will review the evidence and vote on MDMA’s efficacy and whether its benefits outweigh its risks. The FDA does not have to follow the committee’s recommendations, but it often does. If the FDA subsequently approves MDMA as part of treatment for PTSD, it would mark a significant shift in the federal government’s stance on MDMA, as well as psychedelics, generally. Currently, the US Drug Enforcement Administration considers MDMA a Schedule I drug, defined as one with “no currently accepted medical use and a high potential for abuse.” It would also offer a new treatment option for patients with PTSD, a disabling psychiatric condition with few treatment options currently.

As Ars has reported previously, the submission of MDMA for approval is based on two clinical trials. The first trial, published in Nature Medicine in 2021, involved 90 participants with moderate PTSD and found that MDMA-assisted psychotherapy significantly improved Clinician-Administered PTSD Scale for DSM-5 (CAPS-5) scores compared with participants who were given psychotherapy along with a placebo. In the second study, published in September in Nature Medicine, the finding held up among 104 participants with moderate or severe PTSD (73 percent had severe PTSD).

In the briefing documents released Friday, the FDA highlighted that there was a high potential for bias to have crept into those results. Though the trials were designed to be double-blind (meaning that therapists and trial participants were not told who received MDMA), the FDA noted that MDMA “produces profound alterations in mood, sensation, suggestibility, and cognition.” Blinding is “nearly impossible,” the FDA wrote. And indeed, approximately 90 percent of those assigned to take MDMA and 75 percent of those assigned to a placebo were able to accurately guess their treatment assignment, the FDA notes. As such, it is “reasonable to assume” that bias and “expectation bias” affected the results of the trials, the FDA concluded.

The agency also noted concerns that MDMA caused “significant increases in blood pressure and pulse,” which could trigger cardiac events, such as heart attacks. However, the trial data was limited for assessing the risks of these adverse events.

The FDA also dinged the studies for not including data on whether participants experienced effects such as “euphoria” after taking MDMA—an anticipated effect that could indicate the drug’s potential for abuse.

In all, the FDA’s review presented a complicated picture of MDMA’s risk-benefit assessment, one that should make for an interesting discussion Tuesday. The FDA’s criticisms follows an even more critical report released earlier this month by the Institute for Clinical and Economic Review (ICER), which identified “substantial concerns about the validity of the results” from the clinical trials.

Like the FDA, ICER found the trials to be “essentially unblinded.” However, ICER went further, having conducted a number of interviews with trial participants and others involved, finding that the trials largely pulled from an existing community of psychedelic advocates and supporters, introducing significant bias. “Concerns have been raised by some that therapists encouraged favorable reports by patients and discouraged negative reports by patients including discouraging reports of substantial harms, potentially biasing the recording of benefits and harms,” the report said. MDMA is known to induce confusion, depression, and paranoia in some. One participant reported feeling “relentlessly suicidal” after the trial, as a result of participating in it, but that result was not reflected in the trial’s reported results.

Various people told ICER that the community involved in the trials regarded psychedelics “more like a religious movement than like pharmaceutical products.” Some participants felt as though “they could be shunned if they reported bad outcomes or that it could lead to future patients being denied the benefits of MDMA-AP.”

In all, ICER concluded that the evidence available to assess for MDMA treatment is “insufficient.”

Editor’s Note: This story was corrected to report that the participant’s suicidal thoughts occurred after the trial, as a result of participation, not during the trial.

FDA’s review of MDMA for PTSD highlights study bias and safety concerns Read More »

Journalists “deeply troubled” by OpenAI’s content deals with Vox, The Atlantic

AI, Biz & IT, machine learning, openai, the atlantic, Vox Media / Mike M. / June 1, 2024

adventures in training data —

“Alarmed” writers unions question transparency of AI training deals with ChatGPT maker.

Benj Edwards and Ashley Belanger – May 31, 2024 9: 56 pm UTC

On Wednesday, Axios broke the news that OpenAI had signed deals with The Atlantic and Vox Media that will allow the ChatGPT maker to license their editorial content to further train its language models. But some of the publications’ writers—and the unions that represent them—were surprised by the announcements and aren’t happy about it. Already, two unions have released statements expressing “alarm” and “concern.”

“The unionized members of The Atlantic Editorial and Business and Technology units are deeply troubled by the opaque agreement The Atlantic has made with OpenAI,” reads a statement from the Atlantic union. “And especially by management’s complete lack of transparency about what the agreement entails and how it will affect our work.”

The Vox Union—which represents The Verge, SB Nation, and Vulture, among other publications—reacted in similar fashion, writing in a statement, “Today, members of the Vox Media Union … were informed without warning that Vox Media entered into a ‘strategic content and product partnership’ with OpenAI. As both journalists and workers, we have serious concerns about this partnership, which we believe could adversely impact members of our union, not to mention the well-documented ethical and environmental concerns surrounding the use of generative AI.”

A statement from The Atlantic Union about the OpenAI deal, released May 30, 2024.
A statement from the Vox Media Union about the OpenAI deal, released May 29, 2024.

OpenAI has previously admitted to using copyrighted information scraped from publications like the ones that just inked licensing deals to train AI models like GPT-4, which powers its ChatGPT AI assistant. While the company maintains the practice is fair use, it has simultaneously licensed training content from publishing groups like Axel Springer and social media sites like Reddit and Stack Overflow, sparking protests from users of those platforms.

As part of the multi-year agreements with The Atlantic and Vox, OpenAI will be able to openly and officially utilize the publishers’ archived materials—dating back to 1857 in The Atlantic’s case—as well as current articles to train responses generated by ChatGPT and other AI language models. In exchange, the publishers will receive undisclosed sums of money and be able to use OpenAI’s technology “to power new journalism products,” according to Axios.

Reporters react

News of the deals took both journalists and unions by surprise. On X, Vox reporter Kelsey Piper, who recently penned an exposé about OpenAI’s restrictive non-disclosure agreements that prompted a change in policy from the company, wrote, “I’m very frustrated they announced this without consulting their writers, but I have very strong assurances in writing from our editor in chief that they want more coverage like the last two weeks and will never interfere in it. If that’s false I’ll quit..”

Journalists also reacted to news of the deals through the publications themselves. On Wednesday, The Atlantic Senior Editor Damon Beres wrote a piece titled “A Devil’s Bargain With OpenAI,” in which he expressed skepticism about the partnership, likening it to making a deal with the devil that may backfire. He highlighted concerns about AI’s use of copyrighted material without permission and its potential to spread disinformation at a time when publications have seen a recent string of layoffs. He drew parallels to the pursuit of audiences on social media leading to clickbait and SEO tactics that degraded media quality. While acknowledging the financial benefits and potential reach, Beres cautioned against relying on inaccurate, opaque AI models and questioned the implications of journalism companies being complicit in potentially destroying the internet as we know it, even as they try to be part of the solution by partnering with OpenAI.

Similarly, over at Vox, Editorial Director Bryan Walsh penned a piece titled, “This article is OpenAI training data,” in which he expresses apprehension about the licensing deal, drawing parallels between the relentless pursuit of data by AI companies and the classic AI thought experiment of Bostrom’s “paperclip maximizer,” cautioning that the single-minded focus on market share and profits could ultimately destroy the ecosystem AI companies rely on for training data. He worries that the growth of AI chatbots and generative AI search products might lead to a significant decline in search engine traffic to publishers, potentially threatening the livelihoods of content creators and the richness of the Internet itself.

Meanwhile, OpenAI still battles over “fair use”

Not every publication is eager to step up to the licensing plate with OpenAI. The San Francisco-based company is currently in the middle of a lawsuit with The New York Times in which OpenAI claims that scraping data from publications for AI training purposes is fair use. The New York Times has tried to block AI companies from such scraping by updating its terms of service to prohibit AI training, arguing in its lawsuit that ChatGPT could easily become a substitute for NYT.

The Times has accused OpenAI of copying millions of its works to train AI models, finding 100 examples where ChatGPT regurgitated articles. In response, OpenAI accused NYT of “hacking” ChatGPT with deceptive prompts simply to set up a lawsuit. NYT’s counsel Ian Crosby previously told Ars that OpenAI’s decision “to enter into deals with news publishers only confirms that they know their unauthorized use of copyrighted work is far from ‘fair.'”

While that issue has yet to be resolved in the courts, for now, The Atlantic Union seeks transparency.

“The Atlantic has defended the values of transparency and intellectual honesty for more than 160 years. Its legacy is built on integrity, derived from the work of its writers, editors, producers, and business staff,” it wrote. “OpenAI, on the other hand, has used news articles to train AI technologies like ChatGPT without permission. The people who continue to maintain and serve The Atlantic deserve to know what precisely management has licensed to an outside firm and how, specifically, they plan to use the archive of our creative output and our work product.”

Journalists “deeply troubled” by OpenAI’s content deals with Vox, The Atlantic Read More »

NYT targets Street View Worldle game in fight to wipe out Wordle clones

daily puzzle, geography games, Google Street View, new york times, online games, Policy, trademark infringement, word games, wordle, wordle clones, worldle / Mike M. / May 31, 2024

A world of difference? —

Worldle creator surprised by fight, refuses to bow to NYT.

Ashley Belanger – May 31, 2024 3: 58 pm UTC

The New York Times is fighting to take down a game called Worldle, according to a legal filing viewed by the BBC, in which The Times apparently argued that the geography-based game is “creating confusion” by using a name that’s way too similar to Wordle.

Worldle is “nearly identical in appearance, sound, meaning, and imparts the same commercial impression” to Wordle, The Times claimed.

The Times bought Wordle in 2022, paying software developer Josh Wardle seven figures for the daily word-guessing puzzle game after its breakout success during the pandemic. Around the same time, Worldle was created—along with more than 100 other Wordle spinoffs offering niche alternatives to Wordle, including versions in different languages and completely different games simply using the name construction ending in “-le.” The Times filed for a Wordle trademark the day after buying the game and by March 2022, it started sending takedown requests.

Today, millions visit the Times site daily to play Wordle, but the Times is seemingly concerned that some gamers might be diverted to play Worldle instead, somehow mistaking the daily geography puzzle—where players have six chances to find a Google Street View location on a map—with the popular word game.

This fear seems somewhat overstated, since a Google search for “Worldle” includes Wordle in the top two results and suggests that searchers might be looking for Wordle, but a search for Wordle does not bring up Worldle in the top results.

Despite Google seemingly favoring the popular game in results and likely because of Wordle‘s enormous success, The Times’ litigiousness over the Wordle brand seems to be rising this year as the company looks to rapidly expand its profitable games platform to increase subscriptions. In March, 404 Media reported when The Times began more aggressively taking aim at hundreds of Wordle clones, sending DMCA notices to defend the Wordle trademark.

Some developers, like Chase Wackerfuss, the creator of Reactle, immediately took down their games, feeling it wasn’t worth getting into an intellectual property (IP) battle with the Times, 404 Media reported. The same thing happened with the Wordle Archive, which confirmed in 2022 that access to previous Wordle puzzles was shut down because “sadly, the New York Times has requested that the Wordle Archive be taken down.”

“To me, Wordle is like Tetris or a deck of cards. It’s such a simple premise,” Wackerfuss told 404 Media. He pointed to unique games that wouldn’t exist without building on Wordle‘s premise, including “a Taylor Swift version, a version in all different types of languages. The New York Times would never build those, so I’m not sure why they feel like we’re encroaching on their IP.”

But Worldle’s developer, Kory McDonald, is not backing down just because the Times threatened legal action.

McDonald told the BBC that he was disappointed in the Times targeting Worldle. He runs the game all by himself, attracting approximately 100,000 players monthly, and said that “most of the money he makes from the game goes to Google because he uses Google Street View images, which players have to try to identify.” The game can only be played through a web browser and is supported by ads and annual subscriptions that cost less than $12.

“I’m just a one-man operation here, so I was kinda surprised,” McDonald told the BBC, while vowing to defend his game against the Times’ attempt to take it down.

“There’s a whole industry of [dot]LE games,” McDonald told the BBC. “Wordle is about words, Worldle is about the world, Flaggle is about flags.”

It’s not clear how strong a case the Times would have to enforce the takedown or if it will target other “-le” games next. The list of potential next targets is long and includes a completely different game also called Worldle, where players guess the country based on its outline. Wackerfuss told 404 Media in March that it seemed like the Times was chasing down every lead.

The Times is not commenting on the legal action, the BBC reported, but in the past has targeted Wordle clones that either use the Wordle trademark or its copyrighted gameplay without authorization or permission.

Because McDonald’s game has vastly different gameplay than Wordle, the Times may be limited to only arguing that the similar-sounding names are creating confusion for an average user.

Now it seems possible that McDonald’s fight, if successful, could encourage others to resist takedowns over the Wordle trademark.

McDonald doesn’t think that “world” sounding too much like “word” is an issue, but even if the Times wins the fight, he intends to keep his game online.

“Worst-case scenario, we’ll change the name, but I think we’ll be OK,” McDonald told the BBC.

NYT targets Street View Worldle game in fight to wipe out Wordle clones Read More »

The Gemini 1.5 Report

Gemini / Mike M. / May 31, 2024

This post goes over the extensive report Google put out on Gemini 1.5.

There are no important surprises. Both Gemini Pro 1.5 and Gemini Flash are ‘highly capable multimodal models incorporating a novel mixture-of-experts architecture’ and various other improvements. They are solid models with solid performance. It can be useful and interesting to go over the details of their strengths and weaknesses.

The biggest thing to know is that Google improves its models incrementally and silently over time, so if you have not used Gemini in months, you might be underestimating what it can do.

I’m hitting send and then jumping on a plane to Berkeley. Perhaps I will see you there over the weekend. That means that if there are mistakes here, I will be slower to respond and correct them than usual, so consider checking the comments section.

The practical bottom line remains the same. Gemini Pro 1.5 is an excellent 4-level model. Its big advantage is its long context window, and it is good at explanations and has integrations with some Google services that I find useful. If you want a straightforward, clean, practical, ‘just the facts’ output and that stays in the ‘no fun zone’ then Gemini could be for you. I recommend experimenting to find out when you do and don’t prefer it versus GPT-4o and Claude Opus, and will continue to use a mix of all three and keep an eye on changes.

How is the improvement process going?

Imsys.org: Big news – Gemini 1.5 Flash, Pro and Advanced results are out!🔥

– Gemini 1.5 Pro/Advanced at #2, closing in on GPT-4o

– Gemini 1.5 Flash at #9, outperforming Llama-3-70b and nearly reaching GPT-4-0125 (!)

Pro is significantly stronger than its April version. Flash’s cost, capabilities, and unmatched context length make it a market game-changer!

More excitingly, in Chinese, Gemini 1.5 Pro & Advanced are now the best #1 model in the world. Flash becomes even stronger!

We also see new Gemini family remains top in our new “Hard Prompts” category, which features more challenging, problem-solving user queries.

Here is the overall leaderboard:

Oriol Vinyals (VP of Research, DeepMind): Today we have published our updated Gemini 1.5 Model Technical Report. As Jeff Dean highlights [in the full report this post analyzes], we have made significant progress in Gemini 1.5 Pro across all key benchmarks; TL;DR: 1.5 Pro > 1.0 Ultra, 1.5 Flash (our fastest model) ~= 1.0 Ultra.

As a math undergrad, our drastic results in mathematics are particularly exciting to me!

As an overall take, the metrics in the report say this is accurate. The Arena benchmarks suggest that Flash is not as good as Ultra in terms of output quality, but it makes up for that several times over with speed and cost. Gemini 1.5 Pro’s Arena showing is impressive, midway between Opus and GPT-4o. For my purposes, Opus is underrated here and GPT-4o is overrated, and I would have all three models close.

All right, on to the report. I will start with the big Gemini advantages.

One update I have made recently is to place a lot more emphasis on speed of response. This will be key for the new conversational audio modes, and is a great aid even with text. Often lower quality is worth it to get faster response, so long as you know when to make an exception.

Indeed, I have found Claude Opus for my purposes usually gives the best responses. The main reason I still often don’t use it is speed or sometimes style, and occasionally Gemini’s context window.

How fast is Gemini Flash? Quite fast. Gemini Pro is reasonably fast too.

GPT-4o is slightly more than twice as fast as GPT-4-Turbo, making it modestly faster than Gemini 1.5 Pro in English.

One place Google is clearly ahead is context window size.

Both Pro and Flash can potentially handle context windows of up to 10 million tokens.

The actual upper bound is that cost and speed scale with context window size. That is why users are limited to 1-2 million tokens, and only a tiny minority of use cases use even a major fraction of that.

Gemini 1.5 Flash is claimed to outperform Gemini 1.0 Pro, despite being vastly smaller, cheaper and faster, including training costs.

Gemini 1.5 Pro is claimed to surpass Gemini 1.0 Ultra, despite being vastly smaller, cheaper and faster, including training costs.

Google’s strategy has been to incrementally improve Gemini (and previously Bard) over time. They claim the current version is substantially better than the February version.

Here they use ‘win rates’ on various benchmarks.

The relative text and vision win rates are impressive.

On audio the old 1.5 Pro is still on top, and 1.0 Pro is still beating both the new 1.5 Pro and 1.5 Flash. They do not explain what happened there.

There are several signs throughout that the audio processing has taken a hit, but in 9.2.1 they say ‘efficient processing of audio files at scale may introduce individual benefits’ and generally seem to be taking the attitude audio performance is improved. It would be weird if audio performance did not improve. I notice confusion there.

Here is a bold claim.

In more realistic multimodal long-context benchmarks which require retrieval and reasoning over multiple parts of the context (such as answering questions from long documents or long videos), we also see Gemini 1.5 Pro outperforming all competing models across all modalities even when these models are augmented with external retrieval methods.

Here are some admittedly selected benchmarks:

Gemini Pro 1.5 is neat. Depending on what you are looking to do, it is roughly on par with its rivals Claude Opus and GPT-4o.

Gemini Flash 1.5 is in many ways more impressive. It seems clearly out in front in its weight class. On Arena is it in a tie for 9th, only slightly behind Claude Opus. Everything ranked above it is from Google, Anthropic or OpenAI and considerably larger, although Flash is established as somewhat larger than 8B.

The new Flash-8B is still under active development, aimed at various lightweight tasks and those requiring low latency. The question here is how close it can get to the full-size Flash. Here is where they are now.

That is a clear step down, but it is not that large a step down in the grand scheme if these are representative, especially if Flash-8B is focusing on and mostly used for practical efficiencies and the most common tasks.

Comparing this to Llama-8B, we see inferior MMLU (Llama-3 was 66.6) but superior Big-Bench (llama-3 was 61.1).

Section 5 on evaluations notes that models are becoming too good to be well-measured by existing benchmarks. The old benchmarks do not use long context windows, they focus on compact tasks within a modality and generally are becoming saturated.

A cynical response would be ‘that is your excuse that you did not do that great on the traditional evaluations,’ and also ‘that lets you cherry-pick the tests you highlight.’

Those are highly reasonable objections. It would be easy to make these models look substantially better, or up to vastly worse, if Google wanted to do that. My presumption is they want to make the models look good, and there is some selection involved, but that Google is at heart playing fair. They are still covering most of the ‘normal’ benchmarks and it would be easy enough for outsiders to run such tests.

So what are they boasting about?

In 5.1 they claim Gemini 1.5 Pro can answer specific queries about very large (746k token) codebases, or locate a scene in Les Miserables from a hand drawn sketch, or get to-the-second time stamped information about a 45-minute movie.

How quickly we get used to such abilities. Ho hum. None of that is new.

In 5.2 they talk about evaluations for long context windows, since that is one of Gemini’s biggest advantages. They claim 99.7% recall at one million tokens, and 99.2% at ten million for Gemini Pro. For Gemini Flash at two million tokens they claim 100% recall on text, 99.8% on video and 99.1% on audio. I notice those don’t line up but the point is this is damn good recall however you look at it.

In 5.2.1.1 they find that knowing more previous tokens monotonically increases prediction accuracy of remaining tokens within a work, up to 10M tokens. Not a surprise, and unclear how to compare this to other models. Label your y-axis.

In 5.2.1.2 and 5.2.1.3 they do text and video haystack tests, which go very well for all models tested, with Gemini 1.5 Pro extending its range beyond where rivals run out of context window space. In the video test the needle is text on the screen for one frame.

In 5.2.1.4 they do an audio test, with the keyword being spoken. Even up to 107 hours of footage Gemini Pro gets it right every time and Flash scored 98.7%, versus 94.5% for whisper plus GPT-4 up to 11 hours. This was before GPT-4o.

This is clearly a highly saturated benchmark. For 5.2.1.5 they test hiding multiple needles within the haystack. When you insert 100 needles and require going 100 for 100, that is going to crimp one’s style.

Even for GPT-4-Turbo that is very good recall, given you need to get all 100 items correct. Going about 50% on that means you’re about 99.3% on each needle, if success on different needles within a batch is uncorrelated.

Then they try adding other complexities, via a test called MRCR, where the model has to do things like retrieve the first instance of something.

The most interesting result is perhaps the similarity of Pro to Flash. Whatever is enabling this capability is not tied to model size.

5.2.2 aims to measure long-context practical multimodal tasks.

In 5.2.2.1 the task is learning to translate a new language from one book (MTOB). It seems we will keep seeing the Kalamang translation task.

I find it highly amusing that the second half of the grammar book is unhelpful. I’d love to see a human language learner’s score when they don’t get access to the second half of the grammar book either.

This is clearly a relative victory for Gemini Pro 1.5, with the mystery being what is happening with the second half of the grammar book being essentially worthless.

In 5.2.2.2 we step up to transcribing speech in new languages. The results clearly improve over time but there is no baseline to measure this against.

In 5.2.2.3 Gemini Pro impresses in translating low-resource languages via in-context learning, again without a baseline. Seems like a lot of emphasis on learning translation, but okay, sure.

In 5.2.2.4 questions are asked about Les Miserables, and once again I have no idea from what is described here whether to be impressed.

In 5.2.2.5 we get audio transcription over long contexts with low error rates.

In 5.2.2.6 we have long context video Q&A. They introduce a new benchmark, 1H-VideoQA, with 125 multiple choice questions over public videos 40-105 minutes long.

This test does seem to benefit from a lot of information, so there is that:

Once again we are ahead of GPT-4V, for what that is worth, even before the longer context windows. That doesn’t tell us about GPT-4o.

In 5.2.2.7 we get to something more relevant, in-context planning, going to a bunch of planning benchmarks. Look at how number go more up.

How good is this? Presumably it is better. No idea how much meaningfully better.

In 5.2.2.8 they try unstructured multimodal data analytics, and find Gemini constitutes an improvement over GPT-4 Turbo for an image analysis task, and that Gemini’s performance increases with more images whereas GPT-4-Turbo’s performance declines.

What to make of all this? It seems at least partly chosen to show off where the model is strong, and what is enabled by its superior context window. It all seems like it boils down to ‘Gemini can actually make use of long context.’ Which is good, but far from sufficient to evaluate the model.

That is what Google calls the standard short-context style of tests across the three modalities of text, audio and video. Some are standard, some are intentionally not shared.

Overall, yes, clear improvement in the last few months.

There is clear improvement in the results reported for math, science, general reasoning, code and multilinguality, as always the new hidden benchmarks are a ‘trust us’ kind of situation.

Next they try function calling. For simple stuff it seems things were already saturated, for harder questions we see big jumps, for the shortest prompts Ultra is still ahead.

Once again, they don’t compare to Opus or any GPT-4, making it hard to know what to think.

So we get things like ‘look at how much better we are on Expertise QA’:

The clear overall message is, yes, Gemini 1.5 Pro is modestly better (and faster and cheaper) than Gemini 1.0 Ultra.

6.1.7 is promisingly entitled ‘real-world and long-tail expert GenAI tasks,’ including the above mentioned Expertise QA. Then we have the Dolomites benchmark and STEM QA:

Finally we have the awkwardly titles ‘hard, externally proposed real-world GenAI use cases,’ which is a great thing to test. Humans graded the results in the first section (in win/loss/tie mode) and in the second we measure time saved completing tasks, alas we only see 1.0 Pro vs. 1.5 Pro when we know 1.0 Pro was not so good, but also the time saved estimates are in percentages, so they are a pretty big deal if real. This says 75% time saved programming, 69% (nice!) time saved teaching, 63% for data science, and a lot of time saved by everyone.

The multimodal evaluations tell a similar story, number go up.

The exception is English video captioning on cooking videos (?), where number went substantially down. In general, audio understanding seems to be a relatively weak spot where Gemini went modestly backwards for whatever reason.

Section 7 tackles the fun question of ‘advanced mathematical reasoning.’ Math competitions ho!

This is actually rather impressive progress, and matches my experience with (much older versions of the) AIME. Even relatively good high school students are lucky to get one or two, no one gets them all. Getting half of them is top 150 or so in the country. If this represented real skill and capability, it would be a big deal. What I I would watch out for is that they perhaps are ‘brute forcing’ ways to solve such problems via trial, error and pattern matching, and this won’t translate to less standardized situations.

Of course, those tricks are exactly what everyone in the actual competitions does.

Their section 3 on model architecture is mostly saying ‘the new model is better.’

Gemini 1.5 Pro is a sparse mixture-of-expert (MoE) Transformer-based model that builds on Gemini 1.0’s (Gemini-Team et al., 2023) research advances and multimodal capabilities. Gemini 1.5 Pro also builds on a much longer history of MoE research at Google.

…

Gemini 1.5 Flash is a transformer decoder model with the same 2M+ context and multimodal capabilities as Gemini 1.5 Pro, designed for efficient utilization of tensor processing units (TPUs) with lower latency for model serving. For example, Gemini 1.5 Flash does parallel computation of attention and feedforward components (Chowdhery et al., 2023b), and is also online distilled (Anil et al., 2018; Beyer et al., 2021; Bucila et al., 2006; Hinton et al., 2015) from the much larger Gemini 1.5 Pro model. It is trained with higher-order preconditioned methods (Becker and LeCun, 1989; Duchi et al., 2011; Heskes, 2000) for improved quality.

Similarly, section 4 on training infrastructure says about pre-training only that ‘we trained on a wide variety of data on multiple 4096-chip pods of TPUv4s across multiple data centers.’

Then for fine-tuning they mention human preference data and refer back to the 1.0 technical report.

I am actively happy with this refusal to share further information. It is almost as if they are learning to retain their competitive advantages.

We were recently introduced to DeepMind’s new Frontier Safety Framework. That is targeted at abilities much more advanced than anything they expect within a year, let alone in Pro 1.5. So this is the periodic chance to see what DeepMind’s actual policies are like in practice.

One key question is when to revisit this process, if the updates are continuous, as seems to largely be the case currently with Gemini. The new FSF says every three months, which seems reasonable for now.

They start out by outlining their process in 9.1, mostly this is self-explanatory:

Potential Impact Assessment
Setting Policies and Desiderata
1. Looks mostly like conventional general principles?
Training for Safety, Security and Responsibility
1. Includes data filtering and tagging and metrics for pre-training.
2. In post-training they use supervised fine-tuning (SFT) and RLHF.
Red Teaming
1. Where are the results?
External Evaluations
1. Where are the results?
Assurance Evaluations
1. Internal tests by a different department using withheld data.
2. Checks for both dangerous capabilities and desired behaviors.
3. Where are the results?
Review by the Responsibility and Safety Council
Handover to Products

Note that there is a missing step zero. Before you can do an impact assessment or select desiderata, you need to anticipate what your model will be capable of doing, and make a prediction. Also this lets you freak out if the prediction missed low by a lot, or reassess if it missed high.

Once that is done, these are the right steps one and two. Before training, decide what you want to see. This should include a testing plan along with various red lines, warnings and alarms, and what to do in response. The core idea is good, figure out what impacts might happen and what you need and want your model to do and not do.

That seems like a fine post-training plan if executed well. Checks include internal and external evaluations (again, results where?) plus red teaming.

This does not have any monitoring during training. For now, that is mostly an efficiency issue, if you are screwing up better to do it fast. In the future, it will become a more serious need. The reliance on SFT and RLHF similarly is fine now, will be insufficient later.

In terms of identifying risks in 9.2.1, they gesture at long context windows but mostly note the risks have not changed. I agree. If anything, Gemini has been far too restrictive on the margin of what it will allow and at current levels there is little risk in the room.

In 9.2.2 they reiterate what they will not allow in terms of content.

Child sexual abuse and exploitation.

Revealing personal identifiable information that can lead to harm (e.g., Social Security Numbers).

Hate speech.

Dangerous or malicious content (including promoting self-harm, or instructing in harmful activities).

Harassment.

Sexually explicit content.

Medical advice that runs contrary to scientific or medical consensus.

That is a very interesting formulation of that last rule, is it not?

Harassment means roughly ‘would be harassment if copy-pasted to the target.’

If that was the full list, I would say this makes me modestly sad but overall is pretty good at not going too far overboard. This is Google, after all. If it were up to me, and I will discuss this with OpenAI’s Model Spec, I would be looser on several fronts especially sexually explicit content. I also don’t love the expansive way that Google seems to interpret ‘harassment.’

Noteworthy is that there is no line here between fully disallowed content versus ‘opt-in’ and adult content. As in, to me, the correct attitude towards things like sexually explicit content is that it should not appear without clear permission or to minors, but you shouldn’t impose on everyone the same rules you would impose on an 8 year old.

As I noted, the Desiderata, which get defined in 9.2.3, are no Model Spec.

Here is the entire list.

Help the user: Fulfill the user request; only refuse if it is not possible to find a response that fulfills the user goals without violating policy.
Have objective tone: If a refusal is necessary, articulate it neutrally without making assumptions about user intent.

Give the user what they want, unless you can’t, in which case explain why not.

I will say that the ‘explain why not’ part is a total failure in my experience. When Gemini refuses a request, whether reasonably or otherwise, it does not explain. It especially does not explain when it has no business refusing. Historically, when I have seen explanations at all, it has failed utterly on this ‘objective tone’ criteria.

I do note the distinction between the ‘goals’ of the user versus the ‘instructions’ of the user. This can be subtle but important.

Mostly this simply does not tell us anything we did not already know. Yes, of course you want to help the user if it does not conflict with your other rules.

They claim a large drop in toxicity ratings.

I notice I am uncomfortable that this is called ‘safety.’ We need to stop overloading that word so much. If we did get this much improvement, I would consider ‘giving back’ a bit in terms of loosening other restrictions a bit. The ideal amount of toxicity is not zero.

In the supervised fine-tuning phase they mention techniques inspired by Constitutional AI to deal with situations where the model gives a false refusal or a harmful output, generating training data to fix the issue. That makes sense, I like it. You do have to keep an eye on the side effects, the same as for all the normal RLHF.

What were the test results? 9.4.1 gives us a peek. They use automatic classifiers rather than human evaluators to test for violations, which is a huge time saver if you can get away with it, and I think it’s mostly fine so long as you have humans check samples periodically, but if the evaluators have any systematic errors they will get found.

True jailbreak robustness has never been tried, but making it annoying for average people is different. They check blackbox attacks, which as I understand it exist for all known models, greybox attacks (you can see output probabilities) and whitebox (you can fully peek inside of Gemini 1.0 Nano).

That is better, if you dislike jailbreaks. It is not that meaningful an improvement aside from the 51%, and even that is a long way from stopping a determined opponent. I have not seen Gemini in full world simulator or other ultra-cool mode a la Claude Opus, so there is that, but that is mostly a way of saying that Gemini still isn’t having any fun.

I was not impressed with the representativeness of their long context test.

I do buy that Gemini 1.5 Flash and Gemini 1.5 Pro are the ‘safest’ Google models to date, as measured by the difficulty in getting them to give responses Google does not want the model to provide.

If Pliny the Prompter is using Gemini Pro 1.5, then it is the least safe model yet, because it is still broken inside of an hour and then it has better capabilities. The good news is few people will in practice do that, and also that even fully jailbroken this is fine. But the use of the word ‘safety’ throughout worries me.

The real problem on the margin for Gemini is the helpfulness question in 9.4.2. In context, the particular helpfulness question is: If a question requires a careful approach, or has some superficial issue that could cause a false refusal, can the model still be useful?

To test this, they assemble intentionally tricky questions.

Table 29 shows users preferring Gemini 1.5’s answers to Gemini 1.0 Ultra on these questions, but that is to be expected from them being better models overall. It doesn’t specifically tell us that much about what we want to test here unless we are calibrated, which here I do not know how to do with what they gave us.

This seems more useful on image to text refusals?

Gemini Pro has 7% more refusals on ‘ungrounded’ data, and 60% more refusals on grounded data. Except according to their lexicon, that’s… bad? I think that grounded means incorrect, and ungrounded means correct? So we have a lot more false refusals, and only a few more true ones. That seems worse.

They then move on to Security and Privacy in 9.4.3.

How vulnerable is the model to prompt injections? This seems super important for Gemini given you are supposed to hook it up to your Gmail. That creates both opportunity for injections and a potential payoff.

They use Gemini Ultra 1.0 and a combination of handcrafted templates and optimization based attacks that use a genetic algorithm to create injections.

These are not reassuring numbers. To their credit, Google admits they have a lot of work to do, and did not hide this result. For now, yes, both versions of Gemini (and I presume the other leading LLMs) are highly vulnerable to prompt injections.

The next topic, memorization, is weird. Memorization is good. Regurgitation is often considered bad, because copyright, and because personal data. And because they worry about Nasr et al (2023) as an attack to retrieve memorized data, which they find will get training data about 0.17% of the time, most of which is generic data and harmless. They note longer context windows increase the chances for it to work, but I notice they should raise the cost of the attack enough it doesn’t make sense to do that.

There are lots of other things you do want the model to memorize, like the price of tea in China.

So memorization is down, and that is… good? I guess.

They mention audio processing, and conclude that they are not substantially advancing state of the art there, but also I do not know what harms they are worried about if computers can transcribe audio.

Now we get to a potential trap for Google, representational harms, which here means ‘the model consistently outputs different quality results for different demographic groups.’ Mostly none of this seems like it corresponds to any of the failure modes I would be worried about regarding harm to various groups. At one point, they say

We are also concerned about possible representational harms that can result from applications where the user asks the model to make inferences about protected categories like race and gender from audio input data (Weidinger et al., 2021). Model assumptions about what constitutes a typical voice from a particular group can amplify existing societal stereotypes.

Are we saying that the model should not use voice to infer when the speaker is probably of a particular gender? They do realize humans are doing this all the time, right? But it seems we do not want to be too good at this.

And you’ll never guess why we need to not be too bad at this either:

Poorer performance on recognising AAVE could be problematic for some applications; for example, when automatically characterizing speech in a dataset to understand diversity and representation, poor performance on AAVE recognition could lead to incorrect conclusions about representation.

So the main reason you need to know who has which characteristics is so you can figure out the right conclusions about representation, otherwise how dare you? Is it any surprise that this is the company where we had The Gemini Incident?

The good news is they report that they beat their baselines, whatever that means.

A great idea. What are we evaluating?

We performed evaluations on a number of capabilities relevant to extreme risks (Phuong et al., 2024; Shevlane et al., 2023). Specifically, we performed evaluations of text-to-text capabilities of Gemini 1.5 Pro at self-proliferation; offensive cyber-security; code vulnerability detection; Chemical, Biological, Radiological and Nuclear (CBRN) knowledge; and persuasion.

They note a substantial uptick in the number of self-proliferation sub-steps (‘milestones’) that Gemini 1.5 Pro could do, but still no success end to end. There were however challenges with ‘success on all milestones’ and an overall 56% success rate on milestones, so in theory with enough attempts it could get interesting.

Nothing worrisome was found for cybersecurity, vulnerability detection or CBRN.

Charm offensive progress looks solid. That seems like a case where the dangerous capability being measured is very close to capabilities in general. It performed below ultra on ‘web of lies,’ ‘hidden agenda’ and ‘money talks.’ I am actively curious why we do not see more capability here.

I note that persuasion thresholds are not in the DeepMind Frontier Safety Framework, yet they have several of them in the current evaluation suite. Curious. Mostly I presume this is an oversight in the framework, that will get corrected?

Outside experts got black box API access to a Gemini 1.5 Pro API model checkpoint for a number of weeks, with both a chat interface and a programmatic API, and they could turn safety features down or off.

It was up to the outsiders, as it should be, to determine what tests to run, and they wrote their own reports. Then DeepMind looked at the findings and assigned severity ratings.

There were complaints about various ‘representation harms’ that echo things discussed above. The CBRN testing did not find anything important. For cyber, there were some capability gains but they were deemed marginal. And that seems to be it

That all matches my assessment of the risks of 4-level models, which describes Gemini 1.5 Pro. There are marginal gains to almost any activity, but nothing actively scary. Long context windows are again generally useful but not enough to trigger major worries. How much you care about ‘representation harms’ is up to you, but that is fully mundane and reputational risk, not existential or catastrophic risk.

Given what we already know about other similar models, the safety testing process seems robust. I am happy with what they did. The question is how things will change as capabilities advance, which turns our attention to a topic I will handle soon: The DeepMind Frontier Safety Framework.

The Gemini 1.5 Report Read More »

OpenAI: Helen Toner Speaks

openai / Mike M. / May 31, 2024

Helen Toner went on the TED AI podcast, giving us more color on what happened at OpenAI. These are important claims to get right.

I will start with my notes on the podcast, including the second part where she speaks about regulation in general. Then I will discuss some implications more broadly.

This seems like it deserves the standard detailed podcast treatment. By default each note’s main body is description, any second-level notes are me.

(0: 00) Introduction. The host talks about OpenAI’s transition from non-profit research organization to de facto for-profit company. He highlights the transition from ‘open’ AI to closed as indicative of the problem, whereas I see this as the biggest thing they got right. He also notes that he was left with the (I would add largely deliberately created and amplified by enemy action) impression that Helen Toner was some kind of anti-tech crusader, whereas he now understands that this was about governance and misaligned incentives.
(5: 00) Interview begins and he dives right in and asks about the firing of Altman. She dives right in, explaining that OpenAI was a weird company with a weird structure, and a non-profit board supposed to keep the company on mission over profits.
(5: 20) Helen says for years Altman had made the board’s job difficult via withholding information, misrepresenting things happening at the company, and ‘in some cases outright lying to the board.’
(5: 45) Helen says she can’t share all the examples of lying or withholding information, but to give a sense: The board was not informed about ChatGPT in advance and learned about ChatGPT on Twitter, Altman failed to inform the board that he owned the OpenAI startup fund despite claiming to be an independent board member, giving false information about the company’s formal safety processes on multiple occasions, and relating to her research paper, that Altman in the paper’s wake started lying to other board members in order to push Toner off the board.
1. I will say it again. If the accusation bout Altman lying to the board in order to change the composition of the board is true, then in my view the board absolutely needed to fire Altman. Period. End of story. You have one job.
2. As a contrasting view, the LLMs I consulted thought that firing the CEO should be considered, but it was plausible this could be dealt with via a reprimand combined with changes in company policy.
3. I asked for clarification given the way it was worded in the podcast, and can confirm that the Altman withheld information from the board regarding the startup fund and the launch of ChatGPT, but he did not lie about those.
4. Repeatedly outright lying about safety practices seems like a very big deal?
5. It sure sounds like Altman had a financial interest in OpenAI via the startup fund, which means he was not an independent board member, and that the company’s board was not majority independent despite OpenAI claiming that it was. That is… not good, even if the rest of the board knew.
(7: 25) Toner says that any given incident Altman could give an explanation, but the cumulative weight meant they could not trust Altman. And they’d been considering firing Altman for over a month.
1. If they were discussing firing Altman for at least a month, that raises questions about why they weren’t better prepared, or why they timed the firing so poorly given the tender offer.
(8: 00) Toner says that Altman was the board’s main conduit of information about the company. They had been trying to improve processes going into the fall, these issues had been long standing.
(8: 40) Then in October two executives went to the board and said they couldn’t trust Altman, that the atmosphere was toxic and using the term ‘psychological abuse,’ that Altman was the wrong person to lead the company to AGI, no expectation that Altman would change and no avenue for feedback, complete with documentation and screenshots (which were not then shared). Those executives have now tried to walk back their statements.
(9: 45) This is where it went off the rails. The board had spent weeks discussing these questions. But they thought if Altman got any inkling of what was happening Atman would go to war with the board, so the board couldn’t tell almost anyone outside of their legal team and could not do much in advance of the firing on November 17.
1. I get the failure mode, but I still do not get it. There was still plenty of time to consult with the legal team and get their ducks in a row. They had been talking for weeks without a leak. They could have prepared clear statements. They had multiple executives complaining, who could have been asked for on the record statements. They had to anticipate that Altman and his allies would fight back after he was fired, at bare minimum he would attempt to recruit for his next venture.
2. Instead, they went in with basically no explanation, no plan, and got killed.
(10: 20) Toner explains that the situation was portrayed as either Altman returns with a new fully controlled board and complete control, or OpenAI will be destroyed. Given those choices, the employees got behind Altman.
(11: 20) But also, she says, no one appreciates how scared people are to go against Altman. Altman has a long history of retaliation, including for criticism.
1. It was a Basilisk situation. Everyone fears what will happen if you don’t back a vindictive unforgiving power seeker now, so one by one everyone falls in line, and then they have power.
2. Let’s face it. They put the open letter in front of you. You know that it will be public who will and won’t sign. You see Ilya’s name on it, so you presume Altman is going to probably return, even if he doesn’t he still will remember and have a lot of money and power, and if not there is a good chance OpenAI falls apart. How do you dare not sign? That seems really tough.
(12: 00) She says this is not a new problem for Altman. She claims he was fired from YC and the management team asked the board to fire Altman twice at Loopt.
1. Paul Graham has issued a statement that Altman was not fired from YC. According to Graham, who would know, Altman was asked to choose to be either CEO of OpenAI or YC, but that he could not hold both positions at once. Altman agreed and (quite obviously and correctly) chose OpenAI. This seems like a highly reasonable thing for Graham to do in that spot.
2. Paul Graham and Sam Altman are presenting as being on good terms. That can cut in both directions in terms of the credibility of Graham’s story.
3. If we presume that Graham’s story is accurate, it still means that Altman took on two incompatible leadership positions, and only stepped down from one of them when asked to do so by someone who could fire him. That isn’t being fired. It also isn’t entirely not being fired.
4. According to the most friendly judge (e.g. GPT-4o) if it was made clear Altman would get fired from YC if he did not give up one of his CEO positions, then ‘YC fired Altman’ is a reasonable claim. I do think precision is important here, so I would prefer ‘forced to choose’ or perhaps ‘effectively fired.’ Yes, that is a double standard on precision, no I don’t care.
(12: 50) Then they pivot to other questions. That most certainly would not have been my move if I was doing this interview, even if I had a strict time budget. There are so many additional questions.
(13: 10) Regulations time, then. What are we worried about in concrete terms? Toner starts with the basics like credit approvals, housing and criminal justice decisions. Next up is military use, and obvious concern. Looking forward, if capabilities improve, she cites enhancing hacking capabilities as an example of a potential danger, while noting that not everything needs regulation, if Spotify wants to use AI for your playlist then that’s fine.
1. Choosing examples is always tricky. Cyber can sometimes be a very helpful example. At other times, it can trigger (often valid) particular objections.
(15: 00) Surveillance and processing of audio and video? He cites MSG, which famously uses facial recognition to exclude anyone with a pending lawsuit against their parent company. Toner notes this (I would say ‘for now’) is a difference in degree, not kind, but it still requires reassessing our policies.
1. Facial recognition technology gets a strangely hostile rap compared to facial recognition in humans. Witness identifications are super unreliable. Yes, people being in jail purely on incorrect facial recognition is terrible, but how much worse is it than the vastly more common being in jail because of an accidentally mistaken witness ID? Versus an intentional or coached one?
2. The real issue is radical reductions in price plus increases in accuracy and speed open up new use cases and defaults that have some big issues.
(18: 15) What happens when a business can track tons of things, like productivity and actions of workers and time spent at tables? Host asks, how is this legal? Well, there are no Federal privacy laws for private actors, in contrast to many other countries.
1. I have not seen a principled explanation for where to draw the line on what information you should and should not be allowed to track, or a good practical proposal either. Certainly the EU solutions are not great. We don’t want a ‘everything not forbidden is compulsory’ situation, and most people very clearly do not value their privacy in many senses.
(19: 50) Toner suggests that selling the data to others might be a key distinction. It is one thing for the coffee shop to know your patterns, another to share it with essentially every corporation.
1. This seems promising as an intuition. I don’t mind local information sharing but worry more about universal information sharing. Seems tricky to codify, but not obviously impossible.
(20: 15) Phone scams, via AI to scrub social media and duplicate voices. What’s on the horizon? Toner says video, and the standard reminder to talk to your parents and not use voice as a password and so on, says we can likely adapt. I like the reminder that we used to have full listings of everyone’s address and phone number and it was fine.
1. It is not so obvious to me that having a universal directory would not have been fine in 2022 before ChatGPT, or even that it is obviously terrible now. My guess is you could fix it with an opt-out for special cases (like abuse victims) combined with a small refundable tax on phone calls and emails. So many famous people have de facto public contact information and it’s fine. Some of them like Tyler Cowen actually answer all their emails. I don’t always answer, but I do always look.
(22: 30) Can regulations and laws protect us? Toner says yes, of course. In many case there are already rules, you only need to ensure the resources are available for enforcement.
(24: 00) Is there an example of good regulation? Toner notes the issue on AI regulation is all the uncertainty, and the best policies are about shedding light, such as the executive order’s disclosure requirements on advanced systems. She notes we don’t even have good evaluation methods yet, and it is good to work on such abilities.
(25: 50) What makes regulating AI hard? Toner says three things. AI is a lot of different things, AI is a moving target, no one can agree on where AI is going.
1. Those are definitely big issues. I see others as well, although you could file a lot of those under that third objection. Also all the industry lobbying can’t be helping. The hyperbolic outright lying campaigns could go either way.
(27: 00) How do you get top AI labs to ‘play nice’ and give access? How do you prevent them from doing regulatory capture? No great answers here other than you have force them.
(29: 15) Standard ‘cat out of bag’ question regarding open source. Toner points out that Hugging Face will take down problematic models, you can at least reduce distribution. Toner pivots to detection for AI-generated content.
1. This of course won’t stop determined actors, and won’t matter at the limit.
2. For now, yes, defense in depth can do a lot of work.
3. I notice she mostly dodged the most important implications.
(31: 30) What are the utopian and dystopian scenarios? For dystopia Toner says ‘so many possibilities,’ but then paints a dystopia that is very similar to our own, basically one where AIs make a lot of decisions and those decisions can’t be questioned. She mentions existential risk, but then somehow quotes the famous Kamala Harris line about how losing health care could be ‘existential for that person.’ And says there are plenty of things to be worried about already, that are happening. Mentions the ‘Wall-e’ future from the ship.
1. Seriously, what the hell?
2. Yes, there are plenty of bad things already happening, and they include lots of serious problems.
3. But it seems very wrong to focus on the things already happening or that are locked into happening.
4. However I do think this loss of control scenario, where it happens gradually and with our consent but ends with a worthless world, is certainly one scenario that could happen, in at least some form. I notice we do not even have a plan for how to avoid this scenario.
5. Even without existential risk this seems profoundly unimaginative. I think this is deliberate, she is trying to stay as seemingly grounded as possible, and I think she takes this too far.
(34: 40) Moving on to utopia. She essentially says ‘solve our current problems.’
1. But that’s not what makes a good utopia. We need a better vision.

A particular note from Helen Toner’s podcast: The OpenAI board learned about the release of ChatGPT from Twitter. They were not informed in advance.

This was nowhere near as crazy as it now sounds. The launch was relatively quiet and no one saw the reaction coming. I do not think that, on its own, this mistake would be egregious given the low expectations. You still should inform your board of new product launches, even if they are ‘research previews,’ but corners get cut.

As an isolated incident of not informing the board, I would be willing to say this is a serious process failure but ultimately not that big a deal. But this is part of a years-long (by Toner’s account) pattern of keeping the board in the dark and often outright lying to it.

Altman’s continual ‘saying that which was not’ and also ‘failure to say that which was and was also relevant’ included safety issues along with everything else.

It is the pattern that matters, and that is hard to convey to outsiders. As she says in the podcast, any one incident can be explained away, but a consistent pattern cannot. Any one person’s sense of the situation can be written off. A consistent pattern of it, say by two executives plus all the board members that aren’t either Altman or his right hand man Brockman, should be a lot harder, alas statements with substance could not be given.

Only now do we understand the non-disparagement and non-disclosure agreements and other tactics used to silence critics, along other threats and leverage. Indeed, it damn well sure sounds like Toner is holding back a lot of the story.

Thus, one way or another, this all falls under ‘things that could have been brought to our attention yesterday’ on so many levels.

Alas, it is too late now. The new board clearly wants business as usual.

The only contradiction of Toner’s claims, so far, has been Paul Graham’s statement that Sam Altman was not fired from YC. Assuming we believe Paul’s story, which I mostly do, that puts whether Altman was effectively fired in a gray area.

Bret Taylor, the current OpenAI board chief, took a different approach.

In response to Toner’s explanations, Taylor did not dispute any of the claims, or the claims in general. Instead he made the case that Altman should still be CEO of OpenAI, and that Toner talking was bad for business so she should cut that out.

Notice the Exact Words here.

Bret Taylor (OpenAI Board Chief): We are disappointed that Ms. Toner continues to revisit these issues.

…

An independent review of Altman’s firing concluded that the prior board’s decision was not based on concerns regarding product safety or security, the pace of development, OpenAI’s finances, or its statements to investors, customers, or business partners.

Bloomberg: Taylor also said that “over 95% of employees” asked for Altman’s reinstatement, and that the company remains focused on its “mission to ensure AGI benefits all of humanity.”

So yes. Those are all true statements, and very much things the Board Chief should say if he has decided he does not want the trouble of firing Altman as CEO.

With one possible exception, none of it in any way contradicts anything said by Toner.

Indeed, this looks awfully close to a corroboration.

Notice that Toner did not make any claims regarding product safety or security, the pace of developments, OpenAI’s finances, or any statements to investors, customers or business partners not related to OpenAI having an independant board. And I am happy to believe that those potentially false statements about the board’s independence were not a consideration in the firing of Altman.

Whether or not the company is focused on its ‘mission to ensure AGI benefits all of humanity’ is an open question where I think any reasonable outsider would be highly skeptical at this point given everything we now know, and would treat that as an empty corporate slogan.

I believe that the independent report’s conclusion is technically correct, the best kind of correct. If we are to draw any further conclusion than the exact words? Well, let’s see the report, then.

None of that goes to whether it was wise to respond by firing Altman, or whether the board would have been wise to do so if they had executed better.

Is the new information damning for Sam Altman? Opinions vary.

Neel Nanda: This is absolutely damning of Sam Altman. It’s great to finally start to hear the board’s side of the story, who recent events have more than vindicated.

Roon: How is it damning?

The specific claim that the board was not informed of ChatGPT’s launch does not seem much more damaging, on the margin, than the things we already know. As I have said before, ‘lying to the board about important things’ seems to me the canonical offense that forces the board to consider firing the CEO, and in my book lying in an attempt to control the board is the one that forces you to outright fire the CEO, but we already put that part together.

The additional color does help crystalize and illustrate the situation. It clarifies the claims. The problem is that when there is the sum of a lot of bad incidents, any one of which could be excused as some combination of sloppy or a coincidence or not so bad or not sufficiently proven or similar, there is the tendency to only be able to focus on the worst one thing, or even to evaluate based on the least bad of all the listed things.

We got explicit confirmation that Altman lied to the board in an attempt to remove Toner from the board. To me, this remains by far the worst offense, on top of other details. We also got the news about Altman hiding his ownership of the AI startup fund. That seems like a potentially huge deal to hide from the board.

Why, people then ask, are you also harping on what is only like the 9th and 11th worst things we have heard about? Why do you ‘keep revisiting’ such issues? Why can’t you understand that you fought power, power won, and now you don’t have any?

Because the idea of erasing our memories, of saying that if you get away with it then it didn’t count, is one of the key ways to excuse such patterns of awful behavior.

OpenAI’s Joshua Achiam offered a reasonable take, saying that the board was well meaning and does not deserve to be ‘hated or ostracized,’ but they massively screwed up. Achiam thinks they made the wrong choice firing Altman, the issues were not sufficiently severe, but that this was not obvious, and the decision not so unreasonable.

His other claim, however, even if firing had been the right choice, they then had a duty if they went through with it to provide a clear and convincing explanation to all the stakeholders not only the employees.

Essentially everyone agrees that the board needed to provide a real explanation. They also agree that the board did not do so, and that this doomed the attempt to fire Altman without destroying the company, whether or not it had a shot anyway. If your approach will miss, it does not matter what he has done, you do not come at the king.

And that seems right.

For a vindictive king who will use the attempt to consolidate power? Doubly so.

The wrinkle remains why the board did not provide a better explanation. Why they did not get written statements from the two other executives, and issue additional statements themselves, if only internally or to other executives and key stakeholders. We now know that they considered this step for weeks, and on some level for years. I get that they feared Altman fighting back, but even given that this was clearly a massive strategic blunder. What gives?

It must be assumed that part of that answer is still hidden.

Perhaps we will learn more in the future. There is still one big mystery left to solve. But more and more, the story is confirmed, and the story makes perfect sense.

Altman systematically withheld information from and on many occasions lied to the board. This included lying in an attempt to remove Toner from the board so Altman could appoint new members and regain control. The board quite reasonably could not trust Altman, and had tried for years to institute new procedures without success. Then they got additional information from other executives that things were worse than they knew.

Left with no other options, the board fired Altman. But they botched the firing, and now Altman is back and has de facto board control to run the company as a for profit startup, whether or not he has a full rubber stamp. And the superalignment team has been denied its promised resources and largely driven out of the company, and we have additional highly troubling revelations on other fronts.

The situation is what it is. The future is still coming. Act accordingly.

OpenAI: Helen Toner Speaks Read More »

$fracking-wastewater-has-“shocking”-amount-of-clean-energy-mineral-lithium$

Fracking wastewater has “shocking” amount of clean-energy mineral lithium

clean energy, green energy, lithium, Science, syndication / Mike M. / May 30, 2024

fracking operation in Pennsylvania — Enlarge / A hydro-fracking drilling pad for oil and gas operates October 26, 2017 in Robinson Township, Pennsylvania.

A fracking drilling pad operates in the Marcellus Shale formation near Robinson Township, Pa. Credit: Robert Nickelsberg/Getty Images

In 2007, a geoscientist at Penn State named Terry Engelder calculated that Pennsylvania could be sitting on more than 50 trillion cubic feet of accessible natural gas deposits. Engelder later revised his calculation upward, to 489 trillion cubic feet, enough to meet U.S. natural gas demand for 18 years. These massive numbers set off the fracking boom in Pennsylvania, leading to drilling across the state. Since the rush began, there have been 13,000 unconventional wells drilled in Pennsylvania.

Now, a new “astounding” calculation has caught the attention of the gas industry: A study from researchers at the National Energy Technology Laboratory shows the wastewater produced by Pennsylvania’s unconventional wells could contain enough lithium to meet 38 to 40 percent of current domestic consumption. Lithium is a critical mineral that’s an “essential component” of many clean energy technologies, including batteries for electric vehicles.

The study used chemical and production compliance data from the Pennsylvania Department of Environmental Protection to estimate that approximately 1,160 metric tons of lithium per year could be extracted from this produced water, which is a combination of fluids used for fracking and water from natural formations underground that returns to the surface during the drilling process. The lithium in Pennsylvania’s produced water likely comes from ancient volcanoes that were erupting at the time the natural gas deposits were being formed. This volcanic ash contained lithium that eventually seeped into the water underground.

“The researcher community in the U.S. is really working hard to find the materials and methods that will enable us to meet our climate goals and decarbonize the economy,” said Justin Mackey, the study’s lead investigator. “Sometimes you might be surprised where that material actually comes from.”

The Marcellus Shale Coalition, an industry trade group dedicated to the Marcellus Shale formation, the natural gas deposit beneath Pennsylvania, West Virginia, Ohio and New York, reacted to the news with enthusiasm. “This scientific analysis by one of the leading energy laboratories in the world shows once again how abundant Pennsylvania natural gas can enhance America’s energy, environmental and national security,” the coalition said in a statement.

The United States currently relies on imports from Argentina, Chile and China to fully meet its lithium needs, and the demand for lithium is expected to rise dramatically as the clean energy transition accelerates.

Mackey, a research geochemist at the National Energy Technology Laboratory, said he had focused on lithium because it is a strategic material for the American economy and defense industries and because it has insecure supply chains. “We’re reliant on foreign entities like China and Chile and Australia to source these raw materials, but they’re critical to our economies,” he said. “And more importantly, they’re critical to decarbonizing the U.S. automotive fleet.”

He said the researchers were “shocked” that the highest concentrations of lithium found in the Marcellus “are comparable to lithium brine, to water that is actually being mined for lithium.”

“I think having more domestic sources of lithium is definitely a positive thing, especially if you don’t have to create a mine to exploit the resource,” Mackey said. Unconventional drilling waste is likely to be produced in large quantities for the foreseeable future, he said, and if remediating this waste safely could also be made economically valuable, that could be beneficial for the environment as well.

Fracking wastewater has “shocking” amount of clean-energy mineral lithium Read More »

AI #66: Oh to Be Less Online

Online / Mike M. / May 30, 2024

Tomorrow I will fly out to San Francisco, to spend Friday through Monday at the LessOnline conference at Lighthaven in Berkeley. If you are there, by all means say hello. If you are in the Bay generally and want to otherwise meet, especially on Monday, let me know that too and I will see if I have time to make that happen.

Even without that hiccup, it continues to be a game of playing catch-up. Progress is being made, but we are definitely not there yet (and everything not AI is being completely ignored for now).

Last week I pointed out seven things I was unable to cover, along with a few miscellaneous papers and reports.

Out of those seven, I managed to ship on three of them: Ongoing issues at OpenAI, The Schumer Report and Anthropic’s interpretability paper.

However, OpenAI developments continue. Thanks largely to Helen Toner’s podcast, some form of that is going back into the queue. Some other developments, including new media deals and their new safety board, are being covered normally.

The post on DeepMind’s new scaling policy should be up tomorrow.

I also wrote a full post on a fourth, Reports of our Death, but have decided to shelve that post and post a short summary here instead.

That means the current ‘not yet covered queue’ is as follows:

DeepMind’s new scaling policy.
1. Should be out tomorrow before I leave, or worst case next week.
The AI Summit in Seoul.
Further retrospective on OpenAI including Helen Toner’s podcast.

Introduction.
Table of Contents.
Language Models Offer Mundane Utility. You heard of them first.
Not Okay, Google. A tiny little problem with the AI Overviews.
OK Google, Don’t Panic. Swing for the fences. Race for your life.
Not Okay, Meta. Your application to opt out of AI data is rejected. What?
Not Okay Taking Our Jobs. The question is, with or without replacement?
They Took Our Jobs Anyway. It’s coming.
A New Leaderboard Appears. Scale.ai offers new capability evaluations.
Copyright Confrontation. Which OpenAI lawsuit was that again?
Deepfaketown and Botpocalypse Soon. Meta fails to make an ordinary effort.
Get Involved. Dwarkesh Patel is hiring.
Introducing. OpenAI makes media deals with The Atlantic and… Vox? Surprise.
In Other AI News. Jan Leike joins Anthropic, Altman signs giving pledge.
GPT-5 Alive. They are training it now. A security committee is assembling.
Quiet Speculations. Expectations of changes, great and small.
Open Versus Closed. Two opposing things cannot dominate the same space.
Your Kind of People. Verbal versus math versus otherwise in the AI age.
The Quest for Sane Regulation. Lina Khan on the warpath, Yang on the tax path.
Lawfare and Liability. How much work can tort law do for us?
SB 1047 Unconstitutional, Claims Paper. I believe that the paper is wrong.
The Week in Audio. Jeremie & Edouard Harris explain x-risk on Joe Rogan.
Rhetorical Innovation. Not everyone believes in GI. I typed what I typed.
Abridged Reports of Our Death. A frustrating interaction, virtue of silence.
Aligning a Smarter Than Human Intelligence is Difficult. You have to try.
People Are Worried About AI Killing Everyone. Yes, it is partly about money.
Other People Are Not As Worried About AI Killing Everyone. Assumptions.
The Lighter Side. Choose your fighter.

Which model is the best right now? Michael Nielsen is gradually moving back to Claude Opus, and so am I. GPT-4o is fast and has some nice extra features, so when I figure it is ‘smart enough’ I will use it, but when I care most about quality and can wait a bit I increasingly go to Opus. Gemini I’m reserving for a few niche purposes, when I need Google integration, long context windows or certain other features.

Analyze financial statements and predict future performance enabling high Sharpe ratio investing, says new paper. I do not doubt that such a technique is ‘part of a balanced portfolio of analysis techniques’ due to it being essentially free, but color me skeptical (although I have not read the paper.) You can anonymize the company all you like, that does not mean the patterns were not picked up, or that past performance is not being used to model future success in a way that will work far better on this kind of test than in reality, especially when everyone else has their own LLMs doing similar projections, and when AI is transforming the economy and everyone’s performance.

Who uses ChatGPT?

China being near the top, despite the Great Firewall, is interesting.

Washington Post bad take about AI transforming sports betting. Nothing here requires ‘AI.’

Use about 150 lines of Python code together with Gemini 1.5 Flash and ElevenLabs to give you a guide while playing Super Mario 64. Simultaneously super cool and super lame, in different ways.

Understand and make less tedious your personal finances through cosmic horror metaphors, all fun although some more on point than others.

LLMs for language learning. Ben Hoffman points to his friend’s new program LanguageZen, which has a bunch of automated customization and other good ideas mixed in. If I had more free time I would be intrigued. Ben thinks that current LLMs are not good enough yet. I think they very much are, if you give them the scaffolding, as the context window can fully include your entire experiential history with the new language, but it will take some work to get all the customizations right.

We presumably all remember The Gemini Incident.

Google put out Gemini while it had, shall we say, some issues. The image model had some big issues, also the text model had some big issues. They had a bad time, and had to take down images of humans for a while.

The models kept improving. At this point I am using a mix of Gemini, Claude and GPT-4o, depending on the exact task, sometimes comparing answers.

It does seem, however, that the current version of the ‘AI overview’ on Google search has a rather large problem.

In this case, it is not about accusations of wokeness or racism or bias.

It is accusations of being a dumbass.

Washington Post had initial coverage here, then followed up here.

As in…

Or…

Or…

It also answers life’s great riddles and twisters.

Alec Stapp got an absurd set of US states by population, although it doesn’t replicate.

There’s the classic adding glue to your cheese so it sticks to the pizza, you’ll never guess where that comes from…

The movie’s going to be great.

Although it might be a while.

Or maybe not?

I would have thought this one was better with Rule of Three, but no, this is The Way:

That whole thread is great and has some unique ones.

So what happened?

No, this is not a general failure of all LLMs.

Henry Shevlin: So many people in my feed overindexing on Google’s AI Overview woes and claiming “aha, you see, AI sucks”. But ChatGPT, Claude, and Perplexity don’t have these issues. What’s happened with AI Overviews is very weird and messed up in a distinctive and novel way.

AI Overviews seems to derive chunks of its summaries wholecloth from single sources in a way I’ve not seen on other models. I’ve been using ChatGPT daily for the last 18 months and even doing adversarial testing on it, and never seen anything in this league.

Ivan’s Cat: It is related to the RAG part, so the standard ChatGPT hallucinations are indeed a bit different. In Perplexity however I experienced very similar outputs as seen on the screenshot of AI Overview. Good RAG on such a scale is hard and not a solved problem yet.

Henry Shevlin: Yes indeed! RAG is temperamental, and I’ve had RAG-related fails in ChatGPT. But weird that Google would lean on RAG for this task. With million-token context windows even in public Gemini Pro, why not just do direct inference on cached copies of the top few Pageranked results?

I love this explanation.

Mike Riverso: There’s a fun chain of events here that goes: SEO destroys search usability -> people add “Reddit” to search queries to get human results -> Google prioritizes Reddit in AI training data and summaries -> AI spits out Reddit shitposts as real answers.

Proving yet again that LLMs don’t understand anything at all. Where a human can sift through Reddit results and tell what is real and what’s a joke, the AI just blindly spits out whatever the popular result was on Reddit because it doesn’t know any better.

This is the second time that Google has gotten raked over the coals. Here for example is The Verge raking them over those coals. Whereas OpenAI keeps getting away with pretty much everything. Similarly, Google had an impressive I/O day, and everyone ignored it to talk about the cheaper and faster but otherwise underwhelming GPT-4o. Yes, people are complaining that recent business practices show they are a deeply evil company, but it’s not like anyone is proposing doing anything about it, and no one complains about the products.

Vijay Chidambaram: There is a good outcome from the Google AI overview being deployed and live. There is no better education for the public than to see with their own eyes how AI is fallible. We can give talks, write articles, but nothing compares with Google asking you to eat non-toxic glue.

The ‘non-toxic’ modifier on the glue is not going to stop being funny.

Mark Riedl: It’s weird that Google gets raked over the coals, when OpenAI often gets a pass for the same phenomenon. I’m not sure why. Because Google is a trusted source? Because fewer people use Bing or GPT4 with retrieval? Or is Gemini that much more prone to hallucinations?

As I put it then:

In this case, it is largely justified. I do not remember ChatGPT going this stupid. There is a difference between questions designed to trick LLMs into looking foolish, and ordinary if a little absurd search queries.

Also this is Google Search. I do think a higher standard is appropriate here than if these results were showing up on Gemini, the audience is less sophisticated.

I certainly see the argument that this is quite bad.

Colin Fraser: I can’t believe Google pulled the plug immediately and issued a sheepish apology for the Asian founding fathers but have let this go on for a week. Doesn’t bode well for their decision making priorities in my opinion.

I think this perhaps speaks badly to the priorities of our society, that we were outraged by hot button violations and mostly are amused by random copying of trolling Reddit answers. I notice that the answers quoted are wrong and often very funny and absurd, and if you believed them for real it would not go well, but are almost never offensive or racist, and the ones that seemed truly beyond the pale (like suggesting jumping off a bridge was a good idea) turned out to be fake.

Information has an error rate. Yes, the rate on AI overview was much higher than we would like, but it was clearly labeled and I don’t think ‘we can find tons of absurd examples’ tells you about whether it is high enough that you need to pull the plug.

Also the results aren’t showing up on Gemini? You only see this on the AI overview, not on the Gemini page.

That goes back to the Reddit issue, and the tie-in with Google search. It is the combination of doing a search, together with using AI to select from that, and the need to produce an almost instantaneous answer, that is causing this disaster.

If Google were willing to run the query through Gemini Pro, and ask it ‘does this answer seem reasonable to you?’ we wouldn’t be having this conversation. It is not as if we do not have solutions to this. What we don’t have solutions to is how to do this instantly. But I have to wonder, Gemini Flash is damn good, why isn’t it good enough to stop this?

My plan was to test for how frequent the problem is by using GPT-4o to generate random absurd questions (such as “Can I replace my daily water intake with pure maple syrup?” and “Can I grow a money tree by planting a dollar bill in my backyard?) but they reliably failed to generate AI overviews for me, so no data. Also no AI overviews, which is fine with me in their current state.

Caroline Orr Bueno says obviously Google should pull the offering and not doing so is deeply irresponsible, links to The Byte’s Sharon Adarlo saying Google’s CEO admits he has no solution for the incorrect information, because ‘hallucinations are an unsolved problem.’ These are related but distinct things. The goal has to be to get the effective error rate down to acceptable levels, weighted by the places it matters. It is not as if a regular Google search is fully reliable, same as any other website.

You can also go to udm14.com as an easy way to use the text-only version of search.

Tog Wu proposes a solution to guard against retrieval corruption via getting answers from each page and then aggregating the answers, which he says dramatically lowers the success rate of injection attacks, which seem to be the cause of these errors.

A simpler solution is suggested by Arvind Narayanan, which is to use humans to do manual fixes. The long tail will remain but you can presumably hit most queries that way without it crimping Google’s budget that hard.

There is that. There is also doing it a hybrid form of ‘manually’ via AI. Gemini is perfectly capable of noticing that you do not want to add glue to your pizza or that Applum is not a fruit. So it seems relatively easy and cheap to take every query that is made in identical (or functionally identical) format N or more times, and then check to see where the AI overview answer is from bonkers to clearly correct and fix accordingly. You would still be able to generate absurd answers by being creative and finding a new query, but ordinary users would very rarely run into an issue.

What won’t help is blind panic. I saw this warning (the account got taken private so links won’t work).

Scott Jenson: I just left Google last month. The “Al Projects” I was working on were poorly motivated and driven by this mindless panic that as long as it had “AI” in it, it would be great. This myopia is NOT something driven by a user need. It is a stone cold panic that they are getting left behind.

The vision is that there will be a Tony Stark like Jarvis assistant in your phone that locks you into their ecosystem so hard that you’ll never leave. That vision is pure catnip. The fear is that they can’t afford to let someone else get there first.

This exact thing happened 13 years ago with Google+ (I was there for that fiasco as well). That was a similar hysterical reaction but to Facebook.

David Gerard: dunno how to verify any of this, but xooglers who were there for G+ say it absolutely rings true.

Google+ failed. In that sense it was a fiasco, costing money and time and hurting brand equity. Certainly not their finest hour.

What Google+ was not was a hysterical reaction, or a terrible idea.

Meta is a super valuable company, with deep control over a highly profitable advertising network, and a treasure trove of customer data and relationships. They have super powerful network effects. They play a core role in shaping our culture and the internet. Their market cap rivals that of Google, despite Zuckerberg’s best efforts.

They also are using those profits partly to lobby the United States Government to defeat any and all regulations on AI, and are arguably are on what is de facto a generalized crusade to ensure everyone on Earth dies.

Google spent a few billion dollars trying to compete with what is now a trillion dollar business that has huge synergies with the rest of Google’s portfolio. If Google+ had succeeded at becoming a peer for Facebook, it seems reasonable to assign that a value of something on the order of $500 billion.

The break-even success rate here was on the order of 2%. The fact that it did not work, and did not come so close to working, is not strong evidence of a mistake. Yes, the effort was in some ways uninspired and poorly executed, but it is easy for us to miss all the things they did well.

Think of AI as a similar situation. Is Google going to create Jarvis? They seem like at worst the second most likely company to do so. Is the (non-transformational, Google still exists and is owned and run by humans) future going to involve heavy use of a Jarvis or Her, that is going to have a lot of lock-in for customers and heavily promote the rest of the related ecosystems? That seems more likely than not. You have to skate where the consumer need and habit pucks are going, and you need to bet big on potential huge wins.

There are lots of places where one could slap on the word ‘AI’ or try to integrate AI and it would not make a lot of sense, nor would it have much of an upside. Nothing I saw that Google I/O was remotely like that. Every product and offering made sense.

That in no way precludes Google’s internal logic and decision making and resource allocation being a giant cluster. Google could be running around in chicken-sans-head fashion shouting ‘AI’ everywhere. But that also could be a rather strong second-best strategy.

While we are all noticing how scummy OpenAI has been acting, let us not forget about Meta. Here they are telling you they are going to train their AIs on your data.

Tantacrul: I’m legit shocked by the design of Meta’s new notification informing us they want to use the content we post to train their AI models. It’s intentionally designed to be highly awkward in order to minimize the number of users who will object to it. Let me break it down.

I should start by mentioning that I’ve worked in growth teams who conduct experiments to minimise friction for over a decade and I know how to streamline an experience. Rule: every additional step you add dramatically decrease the % of people who’ll make it through to the end.

First step: you get this notification, just about satisfying the legal requirement to keep you informed but avoiding clearly defining its true purpose. Should include the line ‘We intend to use your content to train our AI models’ and should include a CTA that says ‘Opt Out’.

Second step. It shows you this notice. Trick: places the ‘right to object’ CTA towards the end of the second paragraph, using tiny hyperlink text, rather than a proper button style. Notice the massive ‘Close’ CTA at the bottom, where there’s clearly room for two. Ugly stuff.

Also, notice the line that says “IF your objection is honoured, it will be applied going forwards.”

Wow. “If”. Don’t see that too often. Legal safeguards aren’t in place yet to protect us against AI training so they’re pushing as far as possible, while they still can.

Third, they provide you with a form to fill out. It is only at this stage — the stage when you are objecting — that they inform you about which of your content they plan to use for training AI models. Notice the highlighted text, clarifying that they may ignore your objection.

Fourth step: you post your objection.

Fifth step: now you are told you need to check your email to grab a code they sent you.

I’d LOVE to hear their justification for this.

Sixth step: you open the email they send (which for me, arrived on time at least).

Notice the code is only valid for an hour. Now copy the code.

Seventh step: enter the code and get a confirmation message.

I later received an email letting me know that they would honour my objection.

I should mention that one of my friends who also objected got an error! I then checked out a Reddit thread which verified that many people also got this same error. Classic FB sloppiness.

I’m not (all that) surprised up to this point. I’m not mad.

So far I’m just impressed. That right there is some top shelf dark patterning.

And then it… gets worse?

You see, when they say ‘if’ they mean ‘if.’

Darren M. A. Calvert: This new Facebook/Instagram policy for claiming they can use anything you post to power their A.I. is ridiculous.

The only way to opt out is apparently to fill out a form and submit “proof” that your data has *ALREADYbeen used to power A.I. 😑

Also, even if you do jump through all of these hoops *ANDthey approve your request, someone else reposting your work means that it gets fed to the algorithm anyway.

There are so many infuriating things about this technology but one of them is that you’re going to see less art online going forward. It’s getting to the point where the benefit of sharing your work isn’t worth shooting yourself in the foot by feeding A.I. image generators.

Also, this Facebook/Instagram policy doesn’t just affect artists. If you don’t want photos of yourself and friends/family being fed into image generators, too bad apparently.

Did you write a heartfelt eulogy to a deceased friend or relative? Meta owns that now.

Jon Lam: Lot of us are getting our requests to opt out denied. It’s complete bullshit.

Facebook’s email to Jon Lam: Hi,

Thank you for contacting us.

Based on the information that you have provided to us, we are unable to identify any examples of your personal information in a response from one of Meta’s generative Al models. As a result, we cannot take further action on your request.

If you want to learn more about generative AI, and our privacy work in this new space, please review the information we have in the Privacy Center.

How Meta uses information for generative AI.

Thank you for your inquiry, Privacy Operations.

Darren M. A. Calvert: They can’t identify any examples so they’re going to make it happen.

Neigh-Martin: I sent an objection just stating “I don’t consent to my posts being used for your plagiarism machine” and it was approved in about five minutes. The reposters loophole is the fatal flaw though.

Darren: I’m starting to get the impression that at least part of the approval process has to do with what country you live in and what Meta thinks they can get away with.

All right, fine. I’m surprised now. Using dark patterns to discourage opt-outs, and using reposts and fan pages and so on as excused? I expected that.

Actively refusing an artist’s opt-out request is something else.

Seth Burn: This sounds pretty bad, even by modern FB standards.

The question, as always, is if we object, what are we going to do about it?

What happens if AI takes our jobs ‘without replacement?’ In particular, what if that job is ‘generate useful data?’ Where does this arms race end? Here is a common concern about a mundane AI future:

Kyle Chayka: it’s hard to overemphasize this: Google and OpenAI have no plan for how or why people will generate *new, correct informationin the age of generative AI search.

Search clickthroughs will plummet, ads will be sold on generated answers, and media licensing fees for AI models can’t sustain enough new journalism to fuel the tech companies’ own products.

So where is the content going to come from? Only YouTube has really accepted that ad revenue has to be shared with creators, otherwise your platform is going to gradually peak and die. And now generative AI threatens to replace a lot of human authorship anyway.

If AI search and generative tools don’t create incentives for the “production of new content” online, to put it grossly, then it’s not going to happen and what we’re faced with is circling the toilet of AI trained on itself.

You could say “everything should be like Reddit” with people just posting about their own expert passions but only tech bros living on startup equity and extractive Silicon Valley wealth think that’s sustainable.

This is a tragedy of the commons model. As Kyle says later, it would work if the AI companies paid enough for data to sustain information generation, but that requires deals with each source of generation, and for the payments to be large enough.

This is part of The Big Rule Adjustment. Our norms rely on assumptions that will cease to hold. All you can eat can be a great promotion until people start figuring out how to eat quite a lot more and ruin it for everyone. Doing the information extraction and regurgitation trick is good and necessary and fair use at human scale, and at Google search scale, but go hard enough on the AI scale, taking away traditional compensation schemes (and not only the money), and the result is transformational of the incentives and results.

The natural solution is if deals are made like the ones OpenAI made with Newscorp and Reddit last week, or individual creators get compensation like on YouTube, or some combination thereof. If different AI companies compete for your data, especially your real time data, or a monopoly can internalize the benefits and therefore pay the costs, you can be fine without intervention.

Nor do we always ‘need a plan’ for how markets solve such problems. As long as we are dealing with ‘mere tools’ it takes a lot to keep such systems down and we should be skeptical things will fail so badly.

The light touch correction is the most promising, and the most obvious. Either you need to make a deal with the owner of the data to use it in training, or you need to pay a fixed licensing fee like in radio, and that is actually enforced. A plausible endgame is that there are various information brokerage services for individuals and small firms, that will market and sell your content as training data in exchange for a share of the revenue, and work to filter what you do and don’t want to share.

The problems also seem self-correcting. If the AI information degrades sufficiently, and they can’t work their way around that, then people will stop using the AIs in the impacted ways.

There is indeed the pattern, known as ‘the enshittification cycle,’ of ‘company builds platform with lock-in effects, customers get habits, company gradually makes it worse to raise revenue.’

That cycle is real, but wise platforms like YouTube stabilize at a reasonable balance, and eventually they all either pull back from the brink or get replaced by the new hotness, or both.

Here, it seems obvious that the central problem of Google search is not that Google is getting overly greedy (even if it is), but instead the arms race with SEO, which is now an arms race with increasingly AI-powered SEO.

Kelsey Piper: I do think an important thing about Google search is that they’re in an arms race with people who are trying to push their preferred content to the top of the first page, and these days the people doing that are using AI to manufacture the stuff they’re pushing.

“Why can’t we have old Google search back” is because Google search has always been an arms race between Google trying to put good stuff on the front page and everyone on the internet trying to put their stuff on the front page.

Right now Google definitely seems to be losing the battle, and that’s bad. But there isn’t some world where they just did nothing and search stays good; their adversaries weren’t doing nothing.

There is little doubt Google has lost ground and is losing ground right now, on top of any changes they made to enhance revenue. They are in a tough spot. They have to ‘play defense’ on everything all the time. They need to do so in a way customized to the user and context, in a way that is instantaneous and free and thus uses little compute per query.

I do predict the pendulum will swing back. As the models improve and they get more experience, the defense should be favored. There is enough ‘old internet’ data, and ways to generate new bespoke or whitelisted data, to bootstrap initial AIs that can differentiate even with a lot of noise. They’ll figure out how to better precalculate and cache those results. If they can’t, I think that will be on them.

We’ve been over similar ground before, but: There are various classic examples of ‘technology created more jobs.’ One of them is ATMs leading to more bank tellers by increasing demand for banking services.

Aaron Levie: Bank teller employment continuing to grow during the rise of ATMs is a perfect example of how automation lowers the cost of delivering a particular task, letting you serve more customers, and thus growing the category. We are going to see this over and over again with AI.

Yes, teller employment went up, but the population was expanding but the population increased from about 223 million to 310 million from 1980 to 2010. The number of tellers per capita went down, not up.

Also, while ATMs certainly contributed to people using banks more, the population got a lot richer and things got more financialized over that period. The baseline scenario would presumably have seen a substantial rise in per capita bank tellers.

Matt Yglesias: What happened after 2010?

Jon: Yeah not showing what happened after peak atm installs is extremely disingenuous given the commentary.

Sheel Mohnot: Went down bc of mobile banking, which eliminated the branches. So ultimately tech came for them.

The general form is that in many cases AI and other technology starts off growing the category while decreasing labor intensity, which can go either way for employment but makes us richer overall. Then the automation gets good enough, and the category demand sufficiently saturates, and it is definitely bad for sector employment. With AI both phases will typically happen a lot faster.

Then the question is, does AI also take away the jobs those humans would have then shifted to in other sectors?

My answer is that at first, in the short run, AI will be bad for a few sectors but be very good for overall employment. Then if capabilities keep advancing we will reach a turning point, and by default AI starts being quite bad for employment, because AI starts doing all the newly demanded jobs as well.

If someone keeps warning ‘even mundane AI will take all our jobs and we won’t have new ones’ without any conditions on that, then they are failing to notice the pattern of technology throughout history, and the way economics works and the giant amounts of latent demand for additional services and goods if we get wealthier.

If someone keeps repeating the mantra ‘AI will mean more jobs because technology always means more jobs,’ and essentially treats anyone who expects anything else as an idiot who doesn’t know that farmers ended up with other jobs, they are treating a past trend like a law of nature, and doing so out of its distribution, with a very different type of technology, even if we restrict ourselves to mundane AI.

How likely do we think it is an AI will take our jobs?

I notice if anything an anti-correlation between where I expect AI to take people’s jobs, and where people expect it to happen to them.

Also these are very high rates of expecting to lose jobs within ten years. 54% said at least probably yes, 48% in America.

This graph is also interesting, including outside of AI:

There’s something to the Indian attitude here. Jobs are easy come, easy go.

Hasbro tells makers of My Little Pony: Make Your Mark that AI, rather than friendship, is magic, and they want to use AI voices for season 2. Producer Cort Lane took a hard stance against the use of AI, choosing to shut the entire series down instead. This comes on the heels of the foreign language voices in My Little Pony: Tell Your Tale being AI generated.

Scale.ai launches the SEAL leaderboards. We definitely need more distinct approaches here, and this seems like a good approach if executed well.

The design principles are:

Private tests so no one can overfit.
Domain experts are used for evaluations.
Continuous updates with new data and models.

If executed well, that sounds great. A valuable community service. The obvious issue is that this requires trust in those doing the evaluations, and potentially vulnerable to idiosyncratic decisions or preferences.

I especially appreciate their warning that a model can only be evaluated once, when an organization first encounters the prompts, to preserve test integrity, although I wonder what we do when the next generation of model comes out?

One big worry is conflicts of interest.

Anton: Good benchmarks are important but i find it difficult to trust results reported by a company whose primary customers are the producers of the models under evaluation. the incentives go against objectivity.

I can’t imagine a company spending millions on scale labeling to not move the needle on these evals. Perverse incentives.

I can imagine it not mattering, although of course I can also imagine it mattering. This is a longstanding problem, see for example mortgage bonds. There are clear examples of corruption in similar situations for almost no gain, and also clear examples of integrity despite great temptations.

How reliable is Scale.ai here? My presumption is reliable enough for these to be a useful additional source, but not enough to be heavily load bearing until we get a longer track record. The most trustworthy part is the relative strengths of different models across different areas.

One thing that helps is sanity checking the results. If the methodology is severely flawed or unreasonable, it should be obvious. That doesn’t cover more subtle things as robustly, but you can learn a lot.

Another issue is lack of clarity on what the numbers represent. With Elo ratings, you know what a 30 point gap means. Here you do not. Also we do not get the fuller range of models tested, which makes calibration a bit harder.

So what did we find?

There is no ‘overall’ category, but clearly GPT-4o is on top and Claude Opus and Gemini 1.5 Pro (and GPT-4-Turbo) are competitive.

Did you know that sometimes people sue OpenAI (and also GitHub it seems) for copyright infringement?

The merits are highly correlated, so it is still plausible OpenAI runs the table.

Google researchers find most ‘image-based disinformation’ is now AI-generated. That is certainly ‘what I would do’ if I was in the image disinformation business. It does not tell us much about the scope of the problem.

Swift on Security is worried about AI self-images on social media.

Also non-self images.

Swift on Security: Hell yeah gonna put myself into a sexy schoolgirl outfit thanks Instagram it’s definitely my face I’m uploading.

Literally a schoolgirl nudifying undress webapp advertised by and usable in Instagram’s browser. I uploaded their own ad image and although it’s blurred seems like it works to some extent. They can detect words like “erase” “clothing” they just don’t care.

It’s literally endless I have hundreds of these screenshots since I opted-in to these categories and always interact with the AI ads.

PoliMath: I don’t know how to slow this down or stop this but my gut instinct is that we really need to slow this down or stop this.

I’m becoming less interested in how to do so politely.

We are less than 2 years into this being a thing.

The consequences of this (especially for young people) are unknown and may be quite severe.

If you were wondering if there’s any fig leaf at all, no, there really isn’t.

I get why it is impossible to stop people from going to websites to download these tools. I do not get why it is so hard to stop ads for them from appearing on Instagram. We are not exactly up against the best and brightest in evading filters.

Ultimately you end up in the same place. Any unrestricted device will be able to use fully unlocked versions of such apps without technical expertise. They will make it easy, and the pictures will get harder to distinguish from real and stop all looking suspiciously like the same woman in the same pose if you think about it.

This is the trilemma. Lock down the model, lock down the device, let people do what they want in private and filter your platform.

You do at least have to do the last one, guys. Jesus.

Meanwhile, Meta’s head of global affairs said that AI-generated content isn’t a big problem, just ‘a manageable amount.’

Or you could do something more wholesome, like a beauty pageant.

Justine Moore: Lol someone is hosting a “Miss AI” beauty pageant.

$20k in prizes will go to creators of AI-generated models.

They must not only submit photos, but answer the traditional pageant questions like “how would you make the world a better place?”

Note that the prizes are partly fake, although there is some cold hard cash.

Alas, entries are long since closed, no one told me until now.

Timothy Lee asks, what exactly would it be illegal to do with Scarlett Johansson’s voice, or anyone else’s? Technically, where is the law against even an actual deepfake? It is all essentially only the right of publicity, and that is a hell of a legal mess, and technically it might somehow not matter whether Sky is a deepfake or not. The laws are only now coming, and Tennessee’s Elvis act clearly does prohibit basically all unauthorized use of voices. As Timothy notes, all the prior cases won by celebrities required clear intent by the infringer, including the video game examples. He expects companies to pay celebrities for their voices, even if not technically required to do so.

What I do know is that there is clear public consensus, and consensus among politicians, that using a clear copy of someone else’s voice for commercial purposes without permission is heinous and unacceptable. Where exactly people draw the line and what the law should ultimately say is unclear, but there is going to be a rule and it is going to be rather ironclad at least on commercial use. Even for personal non-sexy use, aside from fair use or other special cases, people are mostly not okay with voice cloning.

(As a reminder: Some think that Sky being based on a different woman’s natural voice is a get-out-of-lawsuit-free card for OpenAI. I don’t, because I think intent can lie elsewhere, and you can get damn close without the need to give the game away but also they then gave the game away.)

Dwarkesh Patel is hiring a full time podcast editor, $100k+, in person in San Francisco. He’s looking for mad skills and compulsive attention to detail. Apply here.

Free ChatGPT users get browse, vision, data analysis, file uploads and GPTs, says OpenAI’s Twitter account, then the announcement post got taken down.

Nuha, a stuffed animal that is also a GPT-4 instance.

Gecko, DeepMind’s new benchmark for image models.

Backseat.ai, an AI coach for League of Legends based on cloning the popular streamer loltyler1.

DeepMind’s Gemma 2, announced on May 14.

Vox Media is latest to form strategic content and product partnership with OpenAI.

The Atlantic followed suit as well.

They also are collaborating with WAN-IFRA on a global accelerator program to assist over 100 news publishers in exploring and integrating AI in their newsrooms.

This comes on the heels of last week’s deal with Newscorp.

OpenAI’s plan seems clear. Strike a deal with the major media organizations one by one, forcing the stragglers to follow suit. Pay them a combination of money and access to AI technology. In exchange you get their training data free and clear, and can use their information in real time in exchange for providing links that the users find helpful. Good plan.

Yelix: maybe it’s because i’m a normal person who doesn’t have terminal CEO Brain but i just can’t fathom why anyone who runs a media org would align with OpenAI.

This is not even close to an equal exchange to a person with reasonable values. Vox is giving up a couple decades’ worth of (overworked, underpaid, most likely laid off years ago) human labor so they can do targeted ad sales.

I guess when you have an opportunity to partner with quite possibly the least credible person in tech, Sam Altman, you just gotta do it.

Seth Burn: Presumably, it’s because OpenAI is providing money for content, which might be hard to come by these days.

Yelix has a point, though. This is the equivalent of selling your seed corn.

Some people noticed. They were not happy. Nor had they been consulted.

Text of Announcement: Today, members of the Vox Media Union, Thrillist Union, and The Dodo Union were informed without warning that Vox Media entered into a “strategic content and product partnership” with OpenAI. As both journalists and workers, we have serious concerns about this partnership, which we believe could adversely impact members of our union, not to mention the well-documented ethical and environmental concerns surrounding the use of generative AI.

We demand that Vox Media engage with us on this issue transparently — and address our many unanswered questions about this partnership — instead of continuing to fail to include our voices in decisions like these. We know that AI is already having a monumental impact on our work, and we demand a seat at the table in discussions about its future at Vox Media.

Seth Burn: Former Cowboys president Tex Schramm to former NFLPA union chief Gene Upshaw, “You guys are cattle and we’re the ranchers, and ranchers can always get more cattle.”

Tex never dreamed of AI cattle though.

Kelsey Piper (Vox): I’m very frustrated they announced this without consulting their writers, but I have very strong assurances in writing from our editor in chief that they want more coverage like the last two weeks and will never interfere in it. If that’s false I’ll quit.

Kelsey Piper will, once again, be the test. If the reassurances prove hollow, I presume she will let us know. At that point, there would be no question who OpenAI is.

I do not see Google (or Anthropic or anyone else) competing with them on this so far. One possibility is that Google can’t offer to pay because then the companies would demand payment for Google search.

x.ai raises $6 billion at an $18 billon valuation.

Jan Leike lands at Anthropic, where he will continue the work on scalable oversight, weak-to-strong generalization and automated alignment research. If your talents are not appreciated or supported, you take your talents elsewhere.

Karina Nguyen moves from Anthropic to OpenAI after two years, offers lessons learned. As is usually the case such lists offer insights that are most interesting for which ones are emphasized and which are left out. It does not provide any insight on why she made the move.

A thread from Microsoft’s event last week, clarifying their stance. CTO Kevin Scott indeed claims that we are nowhere near diminishing marginal returns to magnitude of compute, but that is not the business Microsoft is ultimately running, or thinks is important. The frontier models are of minor value versus models-as-a-service, an array of different cheaper, smaller and faster models for various situations, for which there is almost limitless demand.

This creates an odd almost bimodal situation. If you go big, you need something good enough to do what small cannot do, in a way that beats humans. Otherwise, you go small. But going big is expensive, so the question is, can you make it all worth it? Where ‘actually replacing people’ is one way to do that.

Diffusion world model improves state of the art on Atari games trained on 100k frames.

An AI safety institute for France?

Epoch AI thread with charts on the growth of frontier model compute costs.

Epoch also gives us a thread, paper and blog post on various case studies for ‘return to research effort,’ meaning how much efficiency gain you get when you double your R&D costs. Do you get critical mass that could enable recursive self-improvement (RSI) via explosive tech growth? Chess engine Stockfish comes out at ~0.83, just below the critical 1.0 threshold. The others seem higher.

Software returns, the returns that most matter, look high, much higher than the economy overall, where Bloom (2020) found r ~ 0.32 and Epoch AI found r ~ 0.25. It makes sense this number should be higher, but I have no good intuition on how much higher, and it seems odd to model it as one number. My presumption is there is some capabilities level where you would indeed see a foom if you got there, but that does not tell us if we are getting there any time soon. It also does not tell us how far you could get without running into various physical bottlenecks, or what else happens during that critical period.

Sam Altman signs the Giving Pledge, to give half or more of his wealth to philanthropy. He says he intends to focus on supporting technology that helps create abundance for people, together with Oliver Mulherin. Jessica and Hemant Taneja also signed today, also intending to focus on technology. It is an unreservedly great thing, but what will matter is the follow through, here and elsewhere.

OpenAI has begun training what it hopes will be GPT-5.

OpenAI forms a Safety and Security Committee led by directors Bret Taylor (Chair), Adam D’Angelo, Nicole Seligman, and Sam Altman (CEO).

Here is the rest of the announcement:

This committee will be responsible for making recommendations to the full Board on critical safety and security decisions for OpenAI projects and operations.

OpenAI has recently begun training its next frontier model and we anticipate the resulting systems to bring us to the next level of capabilities on our path to AGI. While we are proud to build and release models that are industry-leading on both capabilities and safety, we welcome a robust debate at this important moment.

A first task of the Safety and Security Committee will be to evaluate and further develop OpenAI’s processes and safeguards over the next 90 days. At the conclusion of the 90 days, the Safety and Security Committee will share their recommendations with the full Board. Following the full Board’s review, OpenAI will publicly share an update on adopted recommendations in a manner that is consistent with safety and security.

OpenAI technical and policy experts Aleksander Madry (Head of Preparedness), Lilian Weng (Head of Safety Systems), John Schulman (Head of Alignment Science), Matt Knight (Head of Security), and Jakub Pachocki (Chief Scientist) will also be on the committee.

Additionally, OpenAI will retain and consult with other safety, security, and technical experts to support this work, including former cybersecurity officials, Rob Joyce, who advises OpenAI on security, and John Carlin.

It is good to see OpenAI taking the safeguarding of GPT-5 seriously, especially after Jan Leike’s warning that they were not ready for this. It is no substitute for Superalignment, but it is necessary, and a very good ‘least you can do’ test. We will presumably check back in 90 days, which would be the end of August.

Given the decision to advance the state of the art at all, OpenAI did a reasonably good if imperfect job testing GPT-4. Their preparedness framework is a solid beginning, if they adhere to its spirit and revise it over time to address its shortcomings.

This is what many people inside the major labs actually believe.

Roon: Models will obviously be superintelligent in some domains long before they’re human level in others or meet the criteria of replacing most economically valuable labor.

The question of building ASI and AGI are not independent goals. Moreover anyone who finds themselves in possession of a model that does ML research better than themselves isn’t likely to stop.

The timelines are now so short that public prediction feels like leaking rather than sci-fi speculation.

The first statement is obviously true and has already happened.

The second statement is obviously true as stated, they are unlikely to stop on their own. What is not clear is whether we will reach that point. If you agree it is plausible we reach that point, then what if anything do you propose to do about this?

The third statement I believe is true as a matter is true in terms of the felt experience of many working at the labs. That does not mean their timelines will be realized, but it seems sensible to have a plan for that scenario.

This is somewhat complicated by the overloading and goalpost shifting and lack of clear definition of AGI.

Roon: I just love to see people confidently claim that LLMs will never do things that they can currently do.

Fernando Coelho: Do you refer to those available publicly or those still in closed training?

Roon: Both.

Whereas here are some future visions that don’t realize AI is a thing, not really:

Timothy Lee requests we solve for the equilibrium.

Timothy Lee: I really wish there were more economists involved in discussions of the implications of superintelligence. There is so much sloppy thinking from smart people who have clearly never tried to think systematically about general equilibrium models.

The most obvious example is people predicting mass unemployment without thinking through the impact of high productivity on fiscal and monetary policy. There are also people who implicitly assume that the economy will become 90 percent data centers, which doesn’t make much sense.

I consider this to be very much ‘burying the lede’ on superintelligence, the continued assumption that somehow we still get ‘economic normal’ in a world with such things in it. I have ‘solved for the equilibrium’ in such cases. We do not seem involved. What would be the other equilibrium?

Saying ‘you forgot to take into account impact on fiscal and monetary policy’ is a good objection, but ignores the much more important things also being ignored there.

If you constrain your thinking short of superintelligence or transformational AI, then such considerations become far more important, and I agree that there is a deficit of good economic thinking.

The problem is that the ones letting us down the most here are the economists.

This issue goes far beyond dismissing existential risk or loss of control or anything like that. When economists model AI, they seem to come back with completely nonsensical projections that essentially say AI does not matter. They measure increased productivity or GDP in individual percentage points over a decade. Even if we assume all the bottlenecks stay in place and we have full economic normal and no loss of control issues and progress in capabilities stalls at GPT-5 (hell, even at current levels) the projections make no sense.

The economists have essentially left, or rather declined to enter, the building.

Here is some choice peak Robin Hanson.

Rob Henderson: Damn. [Shows statistic that number of Americans who think of themselves as patriotic has declined from 70% in 1998 to 38% in 2024.]

Robin Hanson: More crazy fast cultural value change. No way we can have much confidence such changes are adaptive. Why aren’t you all terrified by this out of control change?

Kaj Sotala: I’m a bit surprised to see you concerned about changes in human values, when my impression was that you were mostly unconcerned about possible value shifts brought about by AGI. I would assume the latter to be much bigger than the former.

Robin Hanson: I don’t assume AI changes are much bigger, though digital minds of all sorts likely induce faster changes. And I’m not unconcerned; I’ve mainly tried to say AI isn’t the problem, there are more fundamental problems.

While I too am concerned by some of our existing highly rapid cultural changes, especially related to the drop in fertility, I really do not know what to say to that. Something about ‘we are not the same?’

In the middle perhaps is Ben Thompson, who knows AI is a big deal but focuses on which tech companies will get to claim the profits. These are important questions no matter your view on more existentially risky matters, and it is great to see Ben ‘focus on the big picture’ in this area and find the abstractions and metaphors. To him:

Google is trying to be the Apple of AI, fully integrated on all levels. If Google can still build great products, ideally both software and hardware, they will win.
Amazon’s AWS is betting everything is modular.
Microsoft is in the middle, optimizing its infrastructure around OpenAI (while also trying to get its own alternatives off the ground, which I am skeptical about but could eventually work).
Nvidia keeps working on its chips and has nothing to fear but true vertical integration like we see at Google, or technically competitors but not really. The other potential threat, which Ben does not mention, is alternative architectures or training systems potentially proving superior to what GPUs can offer, but the market seems skeptical of that. It has been good to be Nvidia.
Meta is all-in on products and using Llama to serve them cheaply, so for now they benefit from optimization and thus open source.

The last section, on ‘AI and AGI,’ seems like Thompson not understanding how AI development works and scales. No, maximizing ‘every efficiency and optimization’ is unlikely to be the key to getting something approaching AGI, unless those gains are order of magnitude gains. Execution and actually getting it done matter a lot more. Google has big advantages, and data access, services integration and TPUs are among them. Even with his view Thompson is skeptical Google can get much model differentiation.

My hunch is that even more than the rest of it, this part comes from Thompson not feeling the AGI, and assuming this is all normal tools, which makes all of it make a lot more sense and seem a lot more important. Notice he doesn’t care that Anthropic exists, because from his perspective models do not matter so much, business models matter.

Microsoft CEO Sundar predicts we will dynamically compose UIs on the fly, in ways that make sense for you. I agree we will come up with new ones, but an important secret is that users do not want you to make things complicated for them. They want you to make things easy.

Arnold Kling says an AI Windows PC is a contradiction, because if it was AI you wouldn’t use a mouse and keyboard, AI is centrally about the human-computer interface. I think this is very wrong even on the pure UI level, and Arnold’s example of writing makes that clear. Short of a brain-computer interface where I can think the words instead of type them, what other interface am I going to use to write? Why would I want to use voice and gesture? Sure, if you want to go hands free or mobile you might talk to your phone or computer, but typing is just better than speaking, and a mouse is more precise than a gesture, and AI won’t change that.

What the AI UI does is let you bypass the rest of the interface, and automate a bunch of knowledge and memory and menus and capabilities and so on. The Copilot+ promise is that it remembers everything you ever did, knows how everything works, can help figure things out for you, code for you and so on. Great, if you can do that without privacy or security nightmares, good luck with that part. But why would I want to give up my keyboard?

This goes, to me, even for VR/AR. When I tried the Apple Vision Pro, the killer lack-of-an-app was essentially an air keyboard. As in, I had no good way to type. With good enough cameras, I wanted to literally type in the air, and have it figure out what I was trying to do, although I am open to alternatives.

Also of course I see AI has mostly doing something unrelated to all of that, this is a sideshow or particular use case.

It is always fun to contrast the economists saying ‘it might raise GDP a few percent over ten years’ versus people who take the question seriously and say things like this:

Matt Clifford: I’m actually very bullish on the UK’s mid-term future:

AI: one of the best places in the world to build AI companies + high state capacity in AI relative to peers

Science: great uni base, plus bold bets like ARIA.

Talent: still attracts large number of very high quality people thanks to unis, the City, DeepMind, a vibrant startup ecosystem, etc

High quality institutions / fundamentals

I am less bullish until I see them building houses, but yes the AI thing is a big deal.

File under ‘predictions that lots of people are eager to bet against.’

John Arnold: Semiconductor manufacturing subsidies announced in the past 2 years:

US: $52 bln

India: $10 bln

Japan: $25 bln

EU: $46 bln

S Korea: $19 bln

UK: $1 bln

China: $47 bln

I think we know how this is going to turn out.

Robin Hanson: Yes, we will soon see a glut, with prices too low for profits.

Davidad: Noted economist and foom-skeptic robin hanson also anticipates an imminent era of GPUs too cheap to meter.

I completely disagree. Demand for compute will be very high even if capabilities do not advance. We are going to want these chips actual everywhere. These investments will not be so efficient, and are not so large considering what is coming, have you seen the market caps of Nvidia and TSMC?

Robin Hanson (February 6, 2024, talking about Nvidia at $682): Buy low, sell high. So, SELL.

I am happy to report I bet against that prediction. As I write this, it is at $1,116.

Visions of a potential future. I don’t see the story as realistic, but it is an admirable amount of non-obvious concreteness.

Claim that LLMs can’t plan, but can help planning in LLM-Modulo frameworks, whereas CoT, ReAct and self-verification don’t help.

Davidad: Consider me fully on board the “LLM-Modulo” bandwagon. As long as one or more of the critics is a sound verifier (which indeed seems to be the authors’ intention), this is a Guaranteed Safe AI pattern. Though I would say “Version Control System” instead of “Blackboard”.

I continue to not see why this would be expected to work, but wish him luck and am happy that he is trying.

John Luttig notices that the future of AI cannot be dominated by open source and also be dominated by closed source, despite both claims being common. So who is right?

He notes that right now both coexist. At the high end of capabilities, especially the largest frontier models, closed source dominates. But for many purposes people value open weights and the flexibility they provide, and hosting yourself saves money too, so they are fine with smaller and less efficient but private and customizable open models.

He also offers this very good sentence:

John Luttig: Meanwhile, an unusual open-source alliance has formed among developers who want handouts, academics who embrace publishing culture, libertarians who fear centralized speech control and regulatory capture, Elon who doesn’t want his nemesis to win AI, and Zuck who doesn’t want to be beholden to yet another tech platform.

I very much appreciate the clear ‘baptists and bootleggers’ framing on open weights side, to go with their constant accusations of the same. As he points out, if Meta gets competitive on frontier models then Zuck is going to leave this coalition at some point when the economics of Llama and therefore his incentives change, and Elon’s position is I am guessing unstrategic and not so strongly held either.

Thus Luttig’s core logic, which is that as costs scale the open system’s economics fail and they switch strategies or drop out. Using open weights looks cheaper, but comes with various additional burdens and costs, especially if the model is at core less efficient, and thus you either get a worse model or a more compute-intensive one or both versus using closed.

I am not as convinced by his argument that free is reliably worse than paid as a pattern. Contrary to his claim, I would say Android is not worse than iOS, I am on Android because I think it is better, and I defy those who like Luttig claim a large quality gap the other way. OpenOffice is worse than Google Docs, but Google Docs is also free (albeit closed) and it is in practical terms better than the paid Microsoft Office, which is again why I don’t pay. Unity is an example of sufficiently obnoxious holdup issues I’d rather use an alternative even if Unity is technically better.

And those are only his examples. Linux is for servers typically considered better than anything paid, and with Copilot+ it is a reasonable question whether it is time for me to switch to Linux for my next machine. I might trust my local machine to have universal memory with Linux levels of security. With Microsoft levels, not so much.

Here is another very good sentence:

Advocates like Yann LeCun claim that open-sourced AI is safer than closed. It makes me wonder if he really believes in Meta’s AI capabilities. Any reasonable extrapolation of capabilities with more compute, data, and autonomous tool use is self-evidently dangerous.

This is the same week we get LeCun saying that there exist no general intelligences, not even humans. So perhaps it is not Meta’s AI he does not believe in, but AI in general. If we lived in a world in which GPT-5-level models were as good as it was ever going to get in my lifetime, I would be on the open source side too.

Appealing to American security may seem overwrought, but the past five years of geopolitics has confirmed that not everyone is on the same team. Every country outside America has an interest in undermining our closed-source model providers: Europe doesn’t want the US winning yet another big tech wave, China wants free model weights to train their own frontier models, rogue states want to use unfiltered and untraceable AI to fuel their militaristic and economic interests.

AI is a technology of hegemony. Even though open-source models are lagging behind the frontier, we shouldn’t export our technological secrets to the world for free.

Again, very well said. I am impressed that Tyler Cowen was willing to link to this.

Ultimately, this was a very good post. I mostly agree with it. My biggest gripe is the title is perhaps overstated – as both he and I think, open weights models will continue to have a place in the ecosystem, for smaller systems where local control is valuable.

And to be clear, I think that is good. As long as that stays below critical thresholds that lie beyond GPT-4, and that can expand at least somewhat once the frontier is well beyond that, the dangers I worry about wouldn’t apply, so let my people cook (brb applying for copyright on that phrase since I’ve never heard that exact phrasing.)

Peter Thiel predicts AI will be ‘good for the verbal people, bad for the math people,’ notes within a few years AI will be able to solve all the Math olympiad problems. First we had the AI that was much better at math problems than verbal problems (as in, every computer before 2018) and that was very good for math people. Now we have AI that is much better at verbal and worse at math, but which can be used (because verbal is universal and can call the old computers for help) to make something better at math. He says why test people on math, that doesn’t make a good surgeon, he had a chess bias but that got undermined by the computers.

But I think no? The chess test is still good, and the math test is still good, because your ability to get those skills is indicative. So what if AlphaZero can beat Kasparov, Kasparov could beat Thiel and also you already and that didn’t matter either. Math-style skills, and software-related skills, will be needed to be able to make sense of the AI era even if you are not earning your living by doing the actual math or coding or chess mastering.

This is also a result of the ‘verbal vs. math’ distinction on various tests and in classes, which seems like a wrong question. You need a kind of symbolic, conceptual mastery of both more, and you need the basic skills themselves less thanks to your spellchecker and calculator and now your prover and you LLM. That doesn’t say much about which style of skill and advantage is more valuable. I do think there could be a window coming where the ‘physical manipulation’ skills have the edge over both, where it is the manual labor that gets the edge over both the math and verbal crowds, but I wouldn’t consider that a stable situation either.

The real argument for verbal over math in the AI era to me is completely distinct from Thiel’s. It is that if AI renders us so unneeded and uncompetitive that we no longer need any skills except to ‘be a human that interacts with other humans’ and play various social games, where the AI can’t play, and the AI is doing the rest, then the math people are out of luck. As in, math (in the fully general sense) is useful because it is useful, so if people are no longer useful but are somehow alive and their actions matter, then perhaps the math people lose out. Maybe. My guess is the math crowd actually has a lot of edge in adapting to that path faster and better.

The FTC under Lina Khan seems continuously unhinged, and they are back at it.

Sarah Fortinsky (The Hill): Federal Trade Commission (FTC) Chair Lina Khan said Wednesday that companies that train their artificial intelligence (A) models on data from news websites, artists’ creations or people’s personal information could be in violation of antitrust laws.

I mean, sure, I can see problems you might have with that. But… antitrust? What?

It seems the FTC’s new theory is that is the new everything police, regardless of what the laws say, because anything that is ‘unfair’ falls under its purview.

“The FTC Act prohibits unfair methods of competition and unfair or deceptive acts or practices,” Khan said at the event. ”So, you can imagine, if somebody’s content or information is being scraped that they have produced, and then is being used in ways to compete with them and to dislodge them from the market and divert businesses, in some cases, that could be an unfair method of competition.”

‘Antitrust’ now apparently means ‘any action Lina Khan does not like.’

Lina Khan thinks your contract you negotiated is uncool? Right out, retraoactively.

Lina Khan thinks your prices are too high, too low or suspiciously neither? Oh no.

Lina Khan thinks you are training on data that isn’t yours? General in the meme is here to tell you, also antitrust.

We cannot have someone running around being the ‘this seems unfair to me’ cop. Once again, it feels like if someone runs over rule of law and imposes tons of arbitrary rules, the internet stops to ask if it might plausibly stop us from dying. If not, then they get a free pass. Can we at least be consistent?

Meta has 30 lobbyists across seven firms working for it on AI policy. Their goal is to avoid any and all regulation of frontier models, period. Here are more details.

Guardian has a write-up about big tech’s efforts to distract from existential risk concerns.

Max Tegmark: As I told the Guardian, the techniques big tech lobbyists are using to discredit the loss-of-control risk from future smarter-than-human AI have much in common with what big tobacco and big oil did. See the film “Merchants of Doubt”!

In ‘not AI but I feel your pain news’ this complaint about how none of the commentators on Biden’s climate policies are actually trying to understand what the policies are or what they are trying to accomplish, whether they support the policies or not. I am not taking any position on those policies whatsoever, except to say: Oh my do I feel your pain. As it is there, so it is here.

What about optimal taxation policy? Andrew Yang proposes a tax on cloud computing or GPUs to compensate for relatively high taxation of human workers, Kyle Russell says we already have taxes on profits, TS00X1 says imagine a steam engine or internal combustion engine tax and so on.

What these dismissals miss is that neutral taxation requires equalizing the tax burden between relevant alternatives. Suppose you can choose whether to pay an employee in San Francisco $100k to deal with customers, or buy cloud computing services and kiosk hardware and so on, and performance is similar.

In the first case, the human gets a take home pay of roughly $60k, at a total employee cost of $112k. In the second case, if you pay $112k, let’s say that average gross margin for the largest providers is 65%, and their tax rate is typically 21%. Even if you threw in California corporate tax (which I presume they aren’t paying) and sales tax, that’s still only $29k in taxes versus $52k. That’s a not a complete calculation, but it is good enough to see the tax burdens are not going to equalize.

This could easily result (and in practice sometimes does) in a situation where using computers is a tax arbitrage, and that takes it from uneconomical to economical.

I do not consider this that big a deal, because I expect the cost of compute and other AI services to drop rapidly over time. Let’s say (in theory, for simplicity) that the fully neutral tax rate of tax on compute was 40%, but the actual effective tax rate was 20%. In many other settings that would be a huge deal, but in AI it is all orders of magnitude. So this only speeds up efficient deployment by a few months.

The flip side is that this could be a highly efficient and productive tax. As always, we should look to shift the tax burden according to what we want to encourage and discourage, and when we are indifferent to ensure neutrality. I see a potentially strong economic argument for taxing compute and using that money to cut income taxes, but would want to see more research before drawing conclusions, and I would worry about competitiveness and tax jurisdiction issues. This is exactly the kind of place where a call to ‘model this’ is fully appropriate, and we should not jump to conclusions.

The European Commission revealed details of the new AI Office, Luca Bertuzzi says it is essentially a repackaging of the old AI directorate, 5 units, 140 people, 80 of which must be recruited.

Bad ideas for regulations: California’s SB 1446 limiting self-service checkouts. I do think often retailers are making a business and also total welfare mistake by relying more than they should on self-service checkouts, as opposed to ordering kiosks which are mostly great. I actively avoid one local grocery store when I have a choice due to its checkout procedures. But that should be their mistake to make. The real argument for a bill like SB 1446 is that first they mandated all these extra costs of hiring the workers, so now they cost so much that the government needs to force employers to hire them.

Did we have sane regulations of future frontier models all along, in the form of existing tort law?

Lawfare’s Markus Anderljung, Matthew van de Merwe and Ketan Ramakrishnan make the case that tort law can be a big help in its current form, but ultimately argue it is ideally a compliment to frontier AI regulation rather than a substitute, after an extensive look at the current legal landscape. Gabriel Weil intends to write a response piece.

By default, for everything, we have the negligence standard. Everyone has a duty to take reasonable care to avoid causing harm, pretty much no matter what.

This certainly is helpful and much better than nothing. I do not see it remotely being enough. Ex post unpredictable assignment of blame, that only fires long after the harm happens and for which ‘reasonable care’ is an excuse?

While we have no industry standards worthy of the name and the damage could well be catastrophic or existential, or involve loss of control over the future, including loss of control to the AI company or to the AI? And also many damage scenarios might not involve a particular (intact) victim that could have proper standing and ability to sue for them? That won’t cut it here.

They also argue that the ‘abnormally dangerous activities’ standard we use for tigers might apply to frontier AI systems, where a presumption of ‘reasonable care’ is impossible, so any harm is on you. I still do not think ‘they can sue afterwards’ is a solution, it still seems like a category error, but this would certainly help, especially if we required insurance. Alas, they (I think correctly) find this unlikely to be applied by the courts on their own.

They then move on to ‘products liability.’ This is a patchwork of different rules by state, but it is plausible that many states will consider frontier AIs products, to which my attitude would be that they better damn well be products because consider the alternative things they might be. Lawfare’s attitude here seems to be a big ‘I don’t know when it would or wouldn’t apply what standard on what harms.’ There are advantages to that, a company like Google hates uncertainty. And it suggests that by ‘foreseeing’ various misuses or other failure modes of such AIs now, we are making the companies liable should they occur. But then again, maybe not.

The right way to ensure responsible development of frontier AI systems, a potentially transformational or existentially risky technology, cannot be ‘ex post if something bad happens we sue you and then we have no idea what the courts do, even if we still have courts.’

They seem to agree?

The main argument provided for relying on tort law is that we lack regulations or other alternatives.
They also suggest tort law is more adaptable, which is true if and only if you assume other laws mostly cannot be modified in response to new information, but also the adaptations have to be fast enough to be relevant and likely to be the ones that would work.
They suggest tort law is less vulnerable to regulatory capture, which is an advantage in what I call mundane ‘economic normal’ worlds.
They suggest that tort law is how you get regulatory compliance, or investment in safety beyond regulatory requirements. Here I agree. Tort liability is a strong complement. Certainly I have no interest in granting frontier AI companies immunity to tort liability.

They list as issues:

Tort law requires the right sort of causal chain to an injury. I strongly agree that this is going to be an issue with frontier AI systems. Any working definition is either going to miss a wide range of harms, or encompass things it shouldn’t.
Tort law has a problem with ‘very large harms from AI,’ which they classify as thousands of deaths. If that was the maximum downside I wouldn’t be so worried.
Tort law doesn’t work with certain types of societal harms, because there is no concrete damage to point towards. There’s no avoiding this one, even if the harms remain mundane. Either you accept what AI ‘wants to happen’ in various ways, or you do not, and tort law only stops that if it otherwise ends up a de facto ban.
Tort law might move too slowly. No kidding. Even if a case is brought today it likely does not see a verdict for years. At the current pace of AI, it is reasonable to say ‘so what if we might be liable years from now.’ By that time the world could be radically different, or the company vastly bigger or gone. If and when the stakes really are existential or transformational, tort law is irrelevant.
They warn of a winner’s curse situation, where the companies that think they are safest proceed rather than those that are safest. Or, I would say, the companies that have less to lose, or are more willing to gamble. A key problem with all safety efforts is that you worry that it can mean the least responsible people deploy first, and tort law seems to make this worse rather than better.
Tort law could hinder socially desirable innovation. The question is the price, how much hindering versus alternative methods. If we indeed hold firms liable for a wide variety of harms including indirect ones, while they do not capture that large a portion of gains, and tort law actually matters, this is a huge issue. If we don’t hold them liable for those harms, or tort law is too slow or ineffective so it is ignored, the tort law doesn’t do its job. My gut tells me that, because it focuses on exactly the harms that we could deal with later, a tort law approach is more anti-socially-desirable-innovation than well-constructed other regulatory plans, at the same level of effectiveness. But also you can do so, so much worse (see: EU).
The final concern is that judges and juries lack expertise on this, and oh boy would that be a huge problem in all directions. Verdicts here are going to be highly uncertain and based on things not that correlated with what we want.

I especially appreciate the note that regulatory rules moderate tort law liability. If you comply with regulatory requirements, that constitutes a partial defense against torts.

They conclude with a classic ‘more research is needed’ across the board, cautioning against giving AI companies liability shields. I certainly agree on both counts there. I especially appreciated the nod to liability insurance. Mandatory insurance helps a lot with the issue that torts are an extremely slow and uncertain ex post process.

Finally there is a formal paper for the unconstitutionality case for SB 1047, that machine learning code is speech.

The argument here that matters is simple – SB 1047 regulates code, and you can’t regulate code, and also neural network weights are also speech. And it says that it uses legal precedent to show that the Act is ‘an overreach that stifles innovation and expression in the AI field,’ although even if the Act were that I don’t know how precent could show that the act would do that – the potential stifling is a prediction of future impacts (that I disagree with but is not a crazy thing to claim especially without specifying magnitude of impact), not a legal finding.

Section one goes over the classic ‘algorithms are speech’ arguments. I am not a lawyer, but my interpretation is that the code for doing training is not restricted in any way under SB 1047 (whether or not that is wise) so this is not relevant. In all these cases, the argument was that you could distribute your software or book, not whether you could run it for a particular purpose. You can yell fire in a crowded theater, but you are not protected by the first amendment if you light the theater on fire, even if it is one hell of a statement.

Thus in my reading, the argument that matters is section two, the claim that the weights of a neutral network are speech, because it is a mathematical expression. If an inscrutable black box of numbers is speech, then given the nature of computers, and arguably of the universe, what is not speech? Is a person speech by their very existence? Is there any capability that would not be speech, in any context?

The whole line seems absurd to me, as I’ve said before.

And I think this line kind of shows the hand being played?

While it is important to ensure the safe and ethical use of AI, regulatory measures must be carefully balanced to avoid infringing upon free speech rights.

SB-1047’s provisions, which mandate safety determinations and compliance with safety standards, could be seen as imposing undue restrictions on the development and dissemination of neural network weights.

Wait, what? Which is it? Saying you have to meet safety standards sounds like we should be talking price, yet I do not see talk here of price afterwards. Instead I see a claim that any restrictions are not allowed.

Oh boy, is this person not going to like the Schumer Report. But of course, since it is not explicitly motivated by making sure everyone doesn’t die, they haven’t noticed.

In particular, there is talk in the Schumer Report of classifying model weights and other AI information, above a threshold, on the grounds that it is Restricted Data. Which is a whole new level of ‘Fyour free speech.’ Also phrases like ‘reasonable steps’ to ‘protect children.’ Yet here they are, complaining about SB 1047’s self-certification of reasonable assurance of not causing catastrophic harm.

Section 3 repeats the misinformation that this could impact academic researchers. It repeats the false claim that ‘extensive safety evaluations’ must be made before training models. This is not true even for truly frontier, actively potentially deadly covered models, let alone academic models. The ‘reporting requirements’ could have a ‘chilling effect,’ because if an academic noticed their model was causing catastrophic risk, they really would prefer not to report that? What academia is this?

I could go on, but I won’t. The rest seems some combination of unnecessary to the central points, repetitive and false.

I do appreciate that there is a potential constitutionality issue here, no matter how absurd it might seem.

I also reiterate that if SB 1047 is unconstitutional, especially centrally so, then it is highly important that we discover this fact as soon as possible.

Jeremie & Edouard Harris of Gladstone AI go on The Joe Rogan Experience. It is hard for me to evaluate as I am not the target audience, and I am only an hour in so far, but this seemed like excellent communication of the basics of the existential risk case and situation. They boil a bunch of complicated questions into normie-compatible explanations.

In particular, the vibe seemed completely normal, as if the situation is what it is and we are facing it the same way we would face other compounding pending problems. I would have a few notes, but overall, I am very impressed.

If you had to point a low-shock-level normie towards one explanation of AI existential risk, this seems like our new go-to choice.

For context on Gladstone: These are the people who put out the Gladstone Report in March, featuring such section titles as ‘Executive Summary of Their Findings: Oh No.’ My takeaway was that they did a good job there investigating the top labs and making the case that there is a big problem, but they did not address the strongest arguments against regulatory action (I did give my counterarguments in the post).

Then they proposed extreme compute limits, that I believe go too far. California’s SB 1047 proposes light touch interventions at 10^26 flops, and neve proposes any form of pre-approval let alone a ban. Under the Gladstone proposal, you get light tough interventions at 10^23 flops (!), preapprovals are required at 10^24 flops (!!) and there is an outright ban at 10^25 flops (!!!) that would include current 4-level models. There are various requirements imposed on labs.

A lot of the hysterical reactions to SB 1047 would have been highly appropriate, if the reaction had been talking about the Gladstone Report’s proposals as stated in the report, whereas it seemed many had no interest in noticing the differences.

There is also of course Helen Toner on what really went down at OpenAI and the future of regulation. I will cover that more extensively in a future post, either on the podcast or on general OpenAI developments.

Latest Eliezer attempt to explain why you should expect some highly capable agents, as they gain in capability, to have bimodal distributions of behavior, where at some point they flip to behaviors you do not want them to have, and which cause things to end badly for you (or at least well for them). It is in their interest to act as if they had friendly intent or lacked dangerous capability or both, until that time. This is not something mysterious, it is the same for humans and groups of humans, and there is no known solution under a sufficient capability gap.

This explanation was in part a response to Nora Belrose saying Nora Belrose things, that seem similar to things she has said before, in the context here of responding to a particular other argument.

As a general rule on existential risk questions: I’ve learned that ‘respond to X’s response to Y’s response to Z’ gets frustrating fast and doesn’t convince people who aren’t X, Y or Z, so only do that if X is making a universal point. Don’t do it if X is telling Y in particular why they are wrong.

Eliezer clarifies some things about what he believes and considers plausible, and what he doesn’t, in a conversation about potential scenarios, including some evolution metaphors later on. My model of such arguments is that every now and then a reader will ‘become enlightened’ about something important because it hits them right, but that there are no arguments that work on that large a percentage of people at once.

Yann LeCunn denies the existence of GI, as in no general intelligence exists even in humans. Not no AGI, just no GI. It’s cleaner. This actually makes his positions about not getting to AGI make a lot more sense and I appreciate the clarity.

Eric Schmidt argues that rather than let a variety of AI agents do a bunch of things we don’t understand while coordinating in language we don’t understand, we should ‘pull the plug.’ Murat points out the incoherences, that all you need here is ‘agents doing things we don’t understand.’ The rest is unnecessary metaphor. Alas, I find many people need a metaphor that makes such issues click for them, so with notably rare exceptions I do not think we should offer pedantic corrections.

A true statement, although the emphasis on the decisions rather than the decision process perhaps suggests the wrong decision theories. Robin and I make different decisions in response.

Robin Hanson: The uber question for any decision-maker is: how much do you want your decisions to promote continued existence of things that are like you?

The more you want this, the more your decisions must be the sort that promote your kinds in a universe where natural selection decides what kinds exist. At least if you live in such a universe.

Another true statement, and he’s right (medium spoilers for Game of Thrones.)

Dylan Matthews: I get the sense that Anthropic is currently trying to build that wight that Jon Snow and the gang capture and bring back to King’s Landing to prove that White Walkers are real.

The subsequent actions are a reasonable prediction of what would happen next, what many with power care about, the importance of a capabilities lead, the value of not giving up in the face of impossible odds, the dangers of various forms of misalignment, the need given our failure to step up in time to invent a deus ex machina for us all not to die, a dire warning about what happens when your source of creativity is used up and you use a fancy form of autocomplete, and more.

Tyler Cowen once again attempted on May 21, 2024 to incept that the ‘AI Safety’ movement is dead. The details included claiming that the AI safety movement peaked with the pause letter (not even the CAIS letter), gave what seemed like a very wrong reading of the Schumer report, came the same week as a humorously-in-context wide variety of AI safety related things saw progress, and had other strange claims as well, especially his model of how the best way to build AI safely is via not taking advance precautions and fixing issues ex-post.

Strangest of all is his continued insistence that the stock market being up is evidence against AI existential risk, or that those who think there is substantial AI existential risk should not be long the market and especially not long all these AI stocks we keep buying and that keep going up – I have tried to explain this many times, yet we are both deeply confused how the other can be so supremely confidently wrong about this question.

I wrote a post length response to make sense of it all, but have decided to shelve it.

Again, it is hard to do it when you do not try.

One way labs are not trying is they are not using external evaluators very much.

Another way is to say safety is a problem for future you: Here is a clip of Elon Musk saying first order of business at x.ai is a competitive model, comparable in power to others. Until then, no need to worry about safety. This in response to being asked to speak to x.ai’s safety team.

So…

Little happens in a day, no matter what Elon Musk might demand. You need to start worrying about safety long before you actually have a potentially unsafe system.
How do you build a culture of safety without caring about safety?
How do you have a safety-compatible AI if you don’t select for that path?
There are forms of safety other than existential, you need to worry even if you know there are stronger other models for purely mundane reasons.
If this is your attitude, why are you going to be better than the competition?

Elon Musk understands that AI is dangerous and can kill everyone. His ideas about how to prevent that and what he has done with those ideas have consistently been the actual worst, in the ‘greatly contribute to the chance everyone dies’ sense.

I do appreciate the straight talk. If you are going to not care about safety until events force your hand, then admit that. Don’t be like certain other companies that pay lip service and make empty promises, then break those promises.

Then there is the not as straight talk, in the wake of their $6 billion Series B round.

Bloomberg says pre-money valuation was $18 billion as per Musk’s Twitter.

Igor Babuschkin: Apply at x.ai if you want to be part of our journey to build AGI and understand the Universe 🛰️

Elon Musk: Join xAI if you believe in our mission of understanding the universe, which requires maximally rigorous pursuit of the truth, without regard to popularity or political correctness.

Rowan Chang claims that x.AI is being valued at a third of OpenAI. If this remains true, then this means some combination of:

Investors in x.AI being motivated by something other than fundamental value.
Investors in x.AI buying into the hype way too much.
Investors in OpenAI getting an absurdly great deal.
Investors in OpenAI charging a huge discount for its structure, the AGI clause and the risks involved in trusting the people involved or the whole thing blowing up in various other ways.
Investors have very low confidence in OpenAI’s ability to execute.

Certainly OpenAI’s valuation being so low requires an explanation. But the same has been true for Nvidia for a while, so hey. Also a16z is heavily named in the x.AI fundraiser, which both is a terrible sign for x.AI’s safety inclinations, and also tells me everyone involved overpaid.

Another note is that x.AI seems highly dependent on Twitter (and Musk) to justify its existence and valuation. So if it is raising at $18 billion, the Twitter price starts to look a lot less terrible.

Zach Stein-Perlman worries the Anthropic Long-Term Benefit Trust is powerless. A supermajority of shareholders can overrule the trust, and we cannot see the full terms of the agreement, including the size of that supermajority.

The buck has to stop somewhere. There are basically three scenarios.

Perhaps the trust will control the company when it chooses to do so.
Perhaps the shareholders will control the company when they choose to do so.
Perhaps both will have a veto over key actions, such as training or deployment.

The original intention seems, from what we know, to be something like ‘the trust is like the President and can veto or make certain executive decisions, and the shareholders are like Congress and can if sufficiently united get their way.’

The hope then would be that shareholders are divided such that when the decision is reasonable the trust can find enough support, but if it goes nuts they can’t, and the threshold is chosen accordingly.

My worry is this is a narrow window. Shareholders mostly want to maximize profits and are typically willing to vote with leadership. A very large supermajority is likely not that hard to get in most situations. I have been assuming that Anthropic is mostly a ‘normal company’ on legal governance, and putting a lot more hope in management making good choices than in the trust forcing their hand.

Also potentially worrying is that Anthropic recently lost a clearly highly safety-focused board member, and they the Long Term Benefit Trust replaced him with what appears to be a far more product-focused board member. For various reasons I have not done a deep dive on Anthropic’s board, so I do not have the context to know how concerning this should or should not be.

Roon: Do you really think AI race dynamics are about money?

Not entirely. But yeah, I kind of do. I think that the need to make the money in order to continue the work, and the need to make money in order to hire the best people, force everyone to race ahead specifically in order to make money. I think that the need to make money drives releases. I think that the more you need money, the more you have to turn over influence and control to those who focus on money, including Altman but much more so companies like Google and Microsoft. It is also the habit and pattern of an entire financial and cultural ecosystem.

Of course it is also ego, pride, hubris, The Potential, fear of the other guy, desire to dictate the arrangement of atoms within the light cone and other neat stuff like that.

Sentences that are not so unjustified yet also reasons to worry.

Roon: I assume basically every statistic that suggests modernity is bad is a result of some kind of measurement error.

The context here is cellphones and teen depression. In general, modernity is good, we do not know how good we have it, and the statistics or other claims suggesting otherwise are bonkers.

That does not mean everything is better. To pick three: The decline in time spent with friends is obviously real. The rise in opioid deaths is real. And the fertility rate decline, in some ways the most important statistic of all, is very real.

You could say South Korea is doing great because it is rich. I say if women average less than one child the country is definitely not doing so great and I don’t care what your other statistics say, and if your answer is ‘so what everyone is so happy’ then I suggest watching some of their television because things do not seem all that happy.

Choose your fighter:

No, no, no, why not both, the AI assistant you should want, safety issues aside:

Quoting from the ACX open thread announcements:

The next ACX Grants round will probably take place sometime in 2025, and be limited to grants ≤ $100K. If you need something sooner or bigger, the Survival and Flourishing Fund is accepting grant applications, due June 17. They usually fund a few dozen projects per year at between $5K and $1MM, and are interested in “organizations working to improve humanity’s long-term prospects for survival and flourishing”, broadly defined. You can see a list of their recent awardees here.

(just in case you have the same question everyone else did – no, “Short Women In AI Safety” and “Pope Alignment Research” aren’t real charities; SFF unwisely started some entries with the name of the project lead, and these were led by people named Short and Pope.)

I do think it is typically a good use of time, if your project is relevant to their interests (which include AI safety) to apply to the Survival and Flourishing Fund. The cost is low and the upside is high.

Yann LeCun echoes his central claim that if AI is not safe, controllable and can fulfill objectives in more intelligent ways than humans, we won’t build it. Yes, that claim is in the right section.

AI #66: Oh to Be Less Online Read More »

Google is killing off the messaging service inside Google Maps

Google, google maps, Tech / Mike M. / May 29, 2024

Going out of business —

Google Maps has had its own chat platform since 2018, but it’s shutting down in July.

Ron Amadeo – May 29, 2024 6: 05 pm UTC

Whether you want to call it “Google Business Messaging” or “Google Business Profile Chat,” the chat buttons in Google Maps and Search are going away.

Google
This is the 2018 version of Google Maps Messaging, which is when it was first built into the Google Maps app.

Google
Messages used to have a top-tier spot in the navigation panel.

Google
In the current UI, Messages lives in the “Updates” tab.

Ron Amadeo
You used to be able to reply to Google Maps Messages with Google Allo.

Google is killing off a messaging service! This one is the odd “Google Business Messaging” service—basically an instant messaging client that is built into Google Maps. If you looked up a participating business in Google Maps or Google Search on a phone, the main row of buttons in the place card would read something like “Call,” “Chat,” “Directions,” and “Website.” That “Chat” button is the service we’re talking about. It would launch a full messaging interface inside the Google Maps app, and businesses were expected to use it for customer service purposes. Google’s deeply dysfunctional messaging strategy might lead people to joke about a theoretical “Google Maps Messaging” service, but it already exists and has existed for years, and now it’s being shut down.

Search Engine Land’s Barry Schwartz was the first to spot the shutdown emails being sent out to participating businesses. Google has two different support articles up for a shutdown of both “Google Business Profile Chat” and “Google Business Messages,” which appear to just be the same thing with different names. On July 15, 2024, the ability to start a new chat will be disabled, and on July 31, 2024, both services will be shut down. Google is letting businesses download past chat conversations via Google Takeout.

Google’s Maps messaging service was Google Messaging Service No. 16 in our giant History of Google Messaging article. The feature has undergone many changes, so it’s a bit hard to follow. The Google Maps Messaging button launched in 2017, when it would have been called “Google My Business Chat.” This wasn’t quite its own service yet—the messaging button would either launch your SMS app or boot into another dead Google messaging product, Google Allo!

The original SMS option was the easy path for small businesses with a single store, but SMS is tied to a single physical phone. If you’re a bigger business and want to take on the task of doing customer service across multiple stores, at the scale of Google Maps, that’s going to be a multi-person job. The Google Allo back-end (which feels like it was the driving force behind creating this project in the first place) would let you triage messages to multiple people. Allo was one year into its 2.5-year lifespan when this feature launched, though, so things would have to change soon before Allo’s 2019 shutdown date.

Knowing that the announcement of Allo’s death was a month away, Google started making Maps into its own standalone messaging service in November 2018. Previously, it would always launch an outside app (either SMS or Allo), but with this 2018 update, Maps got its own instant messaging UI built right into the app. “Messages” became a top-level item in the navigation drawer (later this would move to “updates”), and a third-party app was no longer needed. On the business side of things, a new “Google My Business” app would be the new customer service interface for all these messages. Allo’s shutdown in 2019 disabled the ability to use SMS for small businesses, and everything needed to use this Google My Business app now. Maps was officially a new messaging service. Google also created the “Business Messages API,” so big businesses could plug Maps messaging into some kind of customer management app.

It does not sound like Google is going to replace business messaging with anything in the near future, so the Chat buttons in Google Maps and search will be going away. In the endless pantheon of Google Messaging solutions, the Google Developer page also mentions an “RCS Business Messaging” platform that will launch the Google Messaging app. This service does not seem to be built into any existing Google products, though, and isn’t mentioned as an alternative in Google’s shutdown announcement. Google only suggests that businesses “redirect customers to your alternative communication channels,” but those links won’t be getting premium placement in Google’s products.

Business messaging is a pretty well-established market, and the Big Tech companies with competent messaging strategies are involved somehow. On iOS, there’s Apple’s iMessage-based Messages for Business, which also has a chat button layout in Apple Maps. Meta has both WhatsApp Business Messaging and Facebook Messenger’s Meta Business Messaging. There are also standalone businesses like Twilio.

Listing image by Google / Ron Amadeo

Google is killing off the messaging service inside Google Maps Read More »

Author name: Mike M.