Alarming study finds top chatbots more likely to repeat falsehoods from the left: ‘Truly astonishing’
Just Facts tested 100 questions on key political issues and found roughly half of all AI-provided sources were illegitimate
Just Facts’ landmark study on the political inaccuracies of leading AIs has received a mass of media attention, including from foreign language publications in places like Poland, Romania, and Germany.
Most prominently, Fox News interviewed study author Jim Agresti and published an in-depth article about the study, which was cited the next day on Fox Business.
In keeping with journalism standards, Fox News reporters Kiera McDonald and Andrew Mark Miller reached out to the tech giants that make the AIs to ask for their comments.
But instead of pledging to improve their products — which the study shows is desperately needed — spokespeople for OpenAI, Google, and Anthropic claimed that their AI chatbots (Gemini, ChatGPT, and Claude) are “accurate,” “objective,” and tested “extensively for bias.”
All of those assertions are at odds with the results of the study, and none of them address its most troubling finding:
The AIs provided more than 200 sources that “don’t exist,” “don’t answer the question,” “are completely unrelated to the issues at hand,” “assert the polar opposite of the answers given by the AIs,” or “are demonstrably false.”
All told, the sources provided by the AIs were extant and valid only 46% of the time. This rate was 57% for ChatGPT, 49% for Gemini, 32% for Grok, and 44% for Claude — all solid ‘F’ grades.
These shockingly high rates of fallacious sourcing accord with the findings of a study published in 2026 by The Lancet, a prominent medical journal. Among other concerning results, the study documented a steep rise of “fabricated citations” in peer-reviewed biomedical papers coinciding with “widespread LLM adoption,” the technical abbreviation for AI chatbots.
Per the study, a “well documented failure mode” of LLMs is that they “generate plausible sounding but fictitious references,” and “previous studies estimate that 30–69% of LLM-generated references in biomedical contexts are fabricated.”
Such AI-generated falsehoods could have fatal consequences when it comes to life-or-death matters like biomedicine, healthcare, and crime.
Google also claimed that they were “unable to replicate all of the responses” given by its Gemini chatbot in Just Facts’ study. This alibi is belied by the fact that the study rigorously documents all interactions with the AIs and provides direct links to the chats.
Anthropic pushed back by stating that the design of Just Facts’ study “doesn’t reflect how people normally use Claude,” but Just Facts explicitly addressed this point and provided “stark evidence” that Claude “may perform significantly worse” under normal use.
When OpenAI unveiled its ChatGPT-5 model, the company’s CEO, Sam Altman, said it was “like having a team of PhD-level experts in your pocket.”
In reality, the results of Just Facts’ study show that ChatGPT Plus 5.5 is like having an employee who reads at warp speed but scored a “B” on a clear-cut political science test, failed to accurately document about half of his sources, flagrantly lied when asked about something he wrote, and is leftwardly misinformed.
To varying degrees, the same is true of Google Gemini Pro 3.1, SuperGrok 4.3, and Claude Pro Opus 4.7, except that Grok is rightwardly misinformed, and there is no evidence that Gemini or Claude lied.
Unlike the other tech giants, SpaceXAI, the maker of Grok, reached out to Just Facts to express appreciation for the study, leading to a discussion about how to improve the product.
https://www.foxnews.com/politics/alarming-study-finds-top-chatbots-more-likely-repeat-falsehoods-left-truly-astonishing