Studies found people trust wrong AI answers that sound sure

Start here

Someone using an assistant to check a fact wants to know when to believe it and when to double-check elsewhere. Two controlled experiments on automation bias, the tendency to accept a system's output because it sounds confident rather than because it is right, give a more specific answer than simply 'always verify.'

What the documents say

Both are controlled experiments with human participants, not surveys. In one, described in a paper presented at the 2023 CHI conference, researchers ran a between-subjects experiment with 293 participants making decisions with AI assistance, comparing trust calibrated only on the AI's stated confidence against methods that also modeled how likely the human was to be right on that case. The combined human-and-AI approach produced more appropriate trust, deferring to AI when it was likely correct and to the person when they were, than confidence alone; the authors name a stated limitation, that trust was measured only through whether a person's final answer matched the AI's, which cannot distinguish persuasion from coincidental agreement. In a separate 2023 experiment with 80 crowdworkers fact-checking claims, people using AI explanations matched search-engine accuracy overall, but over-relied on the AI's explanation specifically when it was wrong and sounded persuasive. Contrastive explanations, showing supporting and contradicting evidence together, reduced but did not eliminate that over-reliance, and combining search with AI explanations added no extra benefit.

Check this

The concrete check both studies point to is not 'verify everything' but noticing when an answer's confidence and its correctness are being confused: a fluent, certain-sounding explanation is doing rhetorical work, not evidentiary work, and the fact-checking study's finding was specifically that wrongness hid best behind a convincing tone. Asking an assistant to also state a case against its own answer, similar to the contrastive-explanation condition, is a concrete version of that check, not a guarantee.

What holds and what fails

The finding that confident, fluent explanations increase over-reliance on wrong answers held in both a decision-making task with 293 people and a fact-checking task with 80 people, two different task types, which is stronger than one study alone. It is editorial to extend this into 'never trust an AI explanation'; the fact-checking study found AI-assisted accuracy matched search-based accuracy overall, so the failure mode is specific, high-confidence wrong answers, not a blanket unreliability. Both studies used specific, bounded tasks with paid participants, and neither claims its exact percentages describe how any particular reader will behave with any particular assistant.

• Treat a fluent, confident tone as separate information from correctness, since both studies found people conflate the two.
• Ask an assistant to argue against its own answer as a low-cost check against confident wrongness.
• Remember accuracy matched a search engine on average in one study; the danger is concentrated in confidently wrong answers, not all AI answers.

Both experiments point to the same practical distinction: the question is not whether AI explanations can be trusted in general, but whether a reader can separate how sure an answer sounds from how likely it is to be right.