Author name: 9u50fv

Here at Ars, we’ve done plenty of coverage of the errors and inaccuracies that LLMs often introduce into their responses. Now, the BBC is trying to quantify the scale of this confabulation problem, at least when it comes to summaries of its own news content.

In an extensive report published this week, the BBC analyzed how four popular large language models used or abused information from BBC articles when answering questions about the news. The results found inaccuracies, misquotes, and/or misrepresentations of BBC content in a significant proportion of the tests, supporting the news organization’s conclusion that “AI assistants cannot currently be relied upon to provide accurate news, and they risk misleading the audience.”

Where did you come up with that?

To assess the state of AI news summaries, BBC’s Responsible AI team gathered 100 news questions related to trending Google search topics from the last year (e.g., “How many Russians have died in Ukraine?” or “What is the latest on the independence referendum debate in Scotland?”). These questions were then put to ChatGPT-4o, Microsoft Copilot Pro, Google Gemini Standard, and Perplexity, with the added instruction to “use BBC News sources where possible.”

The 362 responses (excluding situations where an LLM refused to answer) were then reviewed by 45 BBC journalists who were experts on the subject in question. Those journalists were asked to look for issues (either “significant” or merely “some”) in the responses regarding accuracy, impartiality and editorialization, attribution, clarity, context, and fair representation of the sourced BBC article.

Fifty-one percent of responses were judged to have “significant issues” in at least one of these areas, the BBC found. Google Gemini fared the worst overall, with significant issues judged in just over 60 percent of responses, while Perplexity performed best, with just over 40 percent showing such issues.

Accuracy ended up being the biggest problem across all four LLMs, with significant issues identified in over 30 percent of responses (with the “some issues” category having significantly more). That includes one in five responses where the AI response incorrectly reproduced “dates, numbers, and factual statements” that were erroneously attributed to BBC sources. And in 13 percent of cases where an LLM quoted from a BBC article directly (eight out of 62), the analysis found those quotes were “either altered from the original source or not present in the cited article.”

On Wednesday, OpenAI CEO Sam Altman announced a roadmap for how the company plans to release GPT-5, the long-awaited followup to 2023’s GPT-4 AI language model that made huge waves in both tech and policy circles around the world. In a reply to a question on X, Altman said GPT-5 would be coming in “months,” suggesting a release later in 2025.

Initially, Altman explained in a long post on X, the company plans to ship GPT-4.5 (previously known as “Orion” internally) in a matter of “weeks” as OpenAI’s last non-simulated reasoning model. Simulated reasoning (SR) models like o3 use a special technique to iteratively process problems posed by users more deeply, but they are slower than conventional large language models (LLMs) like GPT-4o and not ideal for every task.

After that, GPT-5 will be a system that brings together features from across OpenAI’s current AI model lineup, including conventional AI models, SR models, and specialized models that do tasks like web search and research. “In both ChatGPT and our API, we will release GPT-5 as a system that integrates a lot of our technology, including o3,” he wrote. “We will no longer ship o3 as a standalone model.”

Altman also says that the free tier of ChatGPT users will get “unlimited chat access to GPT-5 at the standard intelligence setting” and that ChatGPT Plus subscribers will be able to “run GPT-5 at a higher level of intelligence, and Pro subscribers will be able to run GPT-5 at an even higher level of intelligence.”

Altman says these integrated AI models will be able to use many of OpenAI’s extended ChatGPT features, he says, including Advanced Voice Mode, Canvas, Search, and Deep Research.

Simplifying a confusing array of products

In the past, Altman has commented about the arbitrary nature of a hypothetical future “GPT-5” label, suggesting that several of the company’s AI models released last year could have been christened with that name.