AI Assistants Show Major Issues In 45% Of News Answers

A new study finds that AI Assistants shows major issues in 45% of the news answers, which raises questions about the accuracy and reliability of information.

A thorough cross-market study of 2709 AI assistant responses revealed that 45% of the answers had significant imperfections, and 81% showed at least one form of errors. The study, which was conducted by the European Broadcasting Union (EBU) in collaboration and with BBC, raises serious issues with the way AI handles news and information across different countries and languages.

Systemic Issues in AI News Reporting

The study analyzed the most popular consumer models such as ChatGPT, Copilot, Google’s Gemini and Perplexity. The assistants answered inquiries about news in 14 distinct languages, and covered 22 media outlets of public service in 18 different countries.

According to EBU:

“AI’s systemic distortion of news is consistent across languages and territories.”

The wide-spread nature of these errors suggests the existence of fundamental problems in AI process of news, and not isolated glitches or platform-specific issues.

From the total of 2,709 assessed responses, 45% had at the very least one major error however 81% of them had any kind of problem. The sourcing aspect was identified as the biggest issue with 31% of the responses displaying significant difficulties in correctly the citation or attribution of information.

Google Gemini Shows Highest Error Rates

The assistants that were tested on Google’s Gemini had the highest frequency and significant issues. It was found that 76% of the answers were faulty, mostly because of sourcing issues, which impacted 72% of its results. However, other assistants displayed serious issues with 37% or fewer answers, with sourcing issues less than 25%.

The study found the most common errors in accuracy like outdated information or inaccurate information. Many AI responses incorrectly were able to identify Pope Francis as the current pope in May, months following that he died in April. Gemini also gave incorrect explanations of the recent changes to legislation pertaining to disposable vapes.

Study Framework and Data Collection

The data was gathered between May 24 to June 10, using an established set of 30 fundamental questions and additional local inquiries. The study was focused solely on the free, publically available versions of AI tools used to mimic the everyday, real-world use.

Usually, technical barriers block AI agents from gaining access to specific media content. The limitations were lifted temporarily during this research to give AI models to access all media content to the media, and then restored after data collection.

Toolkit and Recommendations

The EBU and BBC have released a complimentary News Integrity in the AI assistants toolkit to aid technology providers as well as media organisations and researchers to address these issues.

According to Reuters The EBU warns that the increased dependence on AI to consume news can undermine trust in journalism if issues with accuracy do not get resolved.

EBU Media Director Jean Philip De Tender suggests:

“When people don’t know what to trust, they end up trusting nothing at all, and that can deter democratic participation.”

Final Thought

These results are a vital reminder to check information before you rely on AI aid in research and reporting or content planning. For media companies and publishers the high percentage of errors is a concern about how their reports may be distorted or incorrectly presented by AI-driven systems.