AI Hallucinations: Why Chatbots Invent Facts and How to Catch Them

AI hallucinations are confident, fluent statements that turn out to be false, such as an invented citation, a wrong number or a quote nobody said. They happen because a language model predicts likely words instead of looking facts up, and because, as OpenAI argues, standard training and evaluation reward guessing over admitting uncertainty. Search, source documents and careful prompts lower the risk, but nothing removes it, so anything that matters needs a check.

What are AI hallucinations?

OpenAI defines hallucinations as plausible but false statements generated by a language model. Its ChatGPT help page lists the forms people meet most: incorrect definitions, dates or facts, fabricated quotes, studies and citations, and overconfident answers to ambiguous or complex questions. The same page adds that confidence is not reliability, because a model can sound sure while being wrong.

No vendor is exempt. Google's Gemini help page says Gemini can make mistakes, tells users to double-check responses and warns against relying on them for professional advice.

Why does ChatGPT make things up?

Language models learn by predicting the next word across huge amounts of text. In OpenAI's explanation, pretraining shows the model only examples of fluent language and no true or false labels, so valid and invalid statements are hard to tell apart. Consistent patterns such as spelling fade as errors with scale. Arbitrary, rarely mentioned facts, like one person's birthday, cannot be predicted from patterns, so the model guesses.

The second cause is how models are graded. OpenAI compares most benchmarks to a multiple-choice test where a blank answer scores zero and a lucky guess can score points, so a model that always guesses looks better on accuracy-only scoreboards. Its example from the GPT-5 system card, on the SimpleQA test, shows the trade-off.

Model	Abstained	Right	Wrong
gpt-5-thinking-mini	52%	22%	26%
OpenAI o4-mini	1%	24%	75%

The older model scores slightly higher on accuracy and makes far more errors. OpenAI concludes that accuracy will never reach 100% because some questions cannot be answered from the available information, and that hallucinations are not inevitable, since a model can say it does not know.

What do hallucinations look like in practice?

Form	Example	How to catch it
Invented citations	A paper, case or URL that does not exist	Search the exact title and open the link
Wrong numbers	A statistic, price or date stated precisely	Find the original table or recompute it
Outdated facts	A rule or price that changed after the model's training data ends	Use search and check the page date
Fake quotes	Words attributed to a real person or document	Search the exact phrase and read the source
Misgrounded citations	A real source that does not say what the answer claims	Open the source and read the passage

Real cases show the stakes. In Mata v. Avianca, a US federal court sanctioned lawyers who filed non-existent judicial opinions with fake quotes created by ChatGPT, and imposed a $5,000 penalty in June 2023. The AI Hallucination Cases database, last updated on 2 October 2026, tracks 2,125 court and tribunal decisions in which hallucinated content was found or implied, 840 of them involving lawyers.

Outside courtrooms, Deloitte Australia agreed to partly refund A$440,000 for a government report that contained a fabricated quote from a court judgment and references to nonexistent research papers. In Moffatt v. Air Canada, a tribunal ordered the airline to pay $812 after its chatbot gave a customer wrong information about bereavement fares, as CBC reported.

How do search and tools reduce hallucinations?

Search and tools give the model something to read instead of something to remember. OpenAI's help page lists search (cited answers from the web), data analysis (accurate calculations) and deep research (multi-source cited answers) as tools for more verifiable answers, and says that without them responses rest on what the model learned in training. Google's Gemini prompting guide says Gemini can use tools to avoid hallucinations. It advises enabling grounding with Google Search for obscure or recent facts and code execution for any arithmetic, counting or calculation. Which assistant offers which tool varies, and our comparison of the best AI model for each task covers the current options.

Fetching source text before answering is called retrieval-augmented generation, or RAG. It helps and it is not a cure. Stanford researchers tested commercial legal research tools built on RAG with more than 200 legal questions. The tools made fewer errors than a general model such as GPT-4, yet Lexis+ AI and Ask Practical Law AI gave incorrect information more than 17% of the time, and Westlaw's AI-Assisted Research more than 34%. The study also found misgrounded answers, where a tool cites a real source that does not support its claim, and sycophancy, where it accepts a false premise from the user.

Errors remain even when a model summarises a document you supply. Vectara's public leaderboard, updated on 11 May 2026, measures how often models add unsupported claims while summarising a source text. Rates ranged from 1.8% to 24.2%, and several flagship models landed between 7% and 12%. The metric checks consistency with the supplied text, not general truthfulness, and the leaderboard predates the newest models.

Which prompting habits help?

Anthropic's guide to reducing hallucinations lists techniques that work with any chatbot.

• Give permission to say it does not know. Anthropic says this simple step can drastically reduce false information.
• For long documents, over 20,000 tokens, ask for word-for-word quotes first and base the answer only on those quotes.
• Ask for a supporting quote for each claim, and have the model remove any claim it cannot support.
• Run the same prompt several times and compare. Inconsistent answers can signal a hallucination.
• Tell the model to use only the documents you provide and not its general knowledge.

A prompt that combines them:

Answer using only the text between [TEXT] and [/TEXT]. For each claim, add a short quote from the text that supports it. If the text does not contain the answer, reply: Not in the text. Do not use outside knowledge.
[TEXT] ... [/TEXT]
Question: ...

Anthropic is clear that these techniques do not eliminate hallucinations and that critical information still needs validation. Our guide on how to use AI effectively covers the wider prompting habits.

What is a good workflow for verifying AI answers?

1. Rate the stakes. A casual question needs a skim. Money, health, law, safety and reputation need a full check.
2. List the checkable claims. Names, numbers, dates, quotes and citations.
3. Open every source. Search each title, follow each link and read the sentence the claim rests on.
4. Cross-check key numbers. Use a second independent source or recompute them in a spreadsheet.
5. Check dates. Look at the publication date and whether a newer rule replaced the one you found.
6. Ask for a second look. In a fresh chat, paste the answer and ask what could be wrong. Treat the reply as a pointer, not as proof.
7. Keep the sources. For work documents, note the source next to each claim so a colleague can recheck it.

The Mata v. Avianca opinion records the lawyer's testimony that he was operating under the false perception that ChatGPT could not possibly be fabricating cases. Checking each case in a legal database would have exposed the problem.

When do you need a human expert?

Use a professional when an error is costly and hard to reverse: medical decisions, legal filings and advice, tax and financial choices, safety-critical work and regulated documents. Google tells Gemini users not to rely on it for professional advice. In Mata v. Avianca, the judge wrote that existing rules give attorneys a gatekeeping role to ensure the accuracy of their filings. The Air Canada tribunal found the airline did not take reasonable care to ensure its chatbot was accurate, so a company answers for what its chatbot tells customers.

AI still helps here. Use it to prepare questions for the expert, explain terms and summarise your own documents, then have the expert confirm anything you plan to act on.

Where Danube Labs fits

If you are building an AI feature that must not invent things, we start with an evaluation set so you can measure error rates before launch. See our AI systems service or book a free intro call.