Why Agents Hallucinate: Confidence Is Not Evidence

Learning outcome: Recognize unsupported answers and missing evidence before an agent's output influences a decision.

This is Day 11 of AI Agents in 30 Days. We are beginning the trusted-knowledge phase of the series. Before connecting an agent to notes or documents, we need to understand why a fluent answer is not automatically a factual one.

What is an AI hallucination?

An AI hallucination is an output that appears plausible but is false, invented, or unsupported by the available evidence. OpenAI describes hallucinations as plausible but false statements, while the U.S. National Institute of Standards and Technology uses the related term confabulation for confidently presented erroneous content.

The important word is not only false. An answer may also be risky when the system cannot show where the claim came from. A correct guess is still a guess if no supporting evidence was available.

Sources:

• OpenAI: Why language models hallucinate

• NIST Generative AI Risk Management Profile

Why can a language model produce an unsupported answer?

A language model generates text by predicting likely continuations from patterns learned during training and from the context supplied at runtime. This process can produce accurate and useful answers, but fluent prediction is not the same operation as checking every claim against a source.

When a required fact is absent, the model may still generate a completion that fits the surrounding pattern. The result can be specific, grammatical, and confident—even though the evidence does not support it.

OpenAI's research also highlights an incentive problem: evaluations that reward correct guesses but do not reward appropriate abstention can encourage guessing over admitting uncertainty. That does not mean every hallucination has one simple cause. It means reliable systems should make room for answers such as “I do not have enough information.”

The mental model: plausible is not proven

Use this three-step model:

1. A fact is missing. The prompt, memory, retrieved documents, or tool result does not contain the needed information.

2. The model produces likely text. It generates a response that fits the question and surrounding context.

3. Fluency hides the gap. The wording sounds trustworthy, so the user may not notice that the claim lacks evidence.

The failure is especially dangerous when an agent can take action. A fabricated date in a casual chat is inconvenient. The same fabricated date passed to a calendar tool could create a real reminder, reschedule work, or trigger another workflow.

Practical example: the invented Friday deadline

Imagine a Study Assistant receives these notes:

Assignment: Research summary
Topic: Renewable energy policy
Deadline: not provided

The learner asks:

When is my research summary due?

The assistant replies:

Your research summary is due Friday.

“Friday” sounds reasonable because assignments often have weekday deadlines. But it appears nowhere in the notes. The answer is unsupported.

The safer response does not pretend the agent knows more than it does. It identifies the missing information and asks for the next useful input.

Three labels for auditing an AI answer

For each factual claim, assign one label:

1. Supported

The claim can be traced to a relevant source supplied to the system, a verified tool result, or reliable external evidence.

Example: “The assignment is a research summary” is supported by the notes.

2. Uncertain

Some relevant information exists, but it is incomplete, ambiguous, outdated, or insufficient for a confident conclusion.

Example: the course calendar says assignments are usually due on Fridays, but it does not identify this assignment's deadline.

3. Invented

The answer presents a detail that does not appear in the available evidence and is not clearly marked as a guess.

Example: “It is due Friday” when no deadline exists in the notes.

These labels are a thinking tool, not a perfect automated detector. A source can itself be wrong, stale, or irrelevant. Evidence must be assessed for both presence and quality.

Try this today

Take one factual answer from an AI tool and follow this exercise:

1. Underline every factual claim.

2. Ask what evidence supports each claim.

3. Label each one supported, uncertain, or invented.

4. Rewrite any uncertain or invented claim so that it clearly communicates the evidence gap.

Completed example

Original AI answer:

Your research summary is due Friday, so you should finish the outline by Wednesday.

Audit:

• “Your research summary is due Friday.” → Invented. No deadline appears in the notes.

• “You should finish the outline by Wednesday.” → Uncertain recommendation. It depends on a Friday deadline that has not been verified.

Evidence-aware rewrite:

The notes list a research summary but do not include its deadline. If you share the assignment page or calendar, I can help build a schedule around the verified date.

A reusable instruction for your agent

You can add an evidence-aware rule to an agent's instructions:

Answer using the available sources and tool results.
For important factual claims, identify the supporting evidence.
If the evidence is missing or ambiguous, say that clearly and ask for the information needed to continue.
Do not invent a specific fact to make the answer feel complete.

This instruction can encourage better behaviour, but prompting alone cannot guarantee factual accuracy. Reliable applications also need appropriate sources, retrieval quality, tool checks, evaluations, and human review when the stakes are high.

Common mistakes

Mistake 1: Treating confidence as a probability

A confident writing style does not provide a calibrated probability that a claim is true. Tone and correctness are separate properties.

Mistake 2: Assuming citations automatically solve the problem

A citation may be fabricated, irrelevant, or fail to support the exact claim. Open the source and check the connection between claim and evidence.

Mistake 3: Asking the model to “never hallucinate”

That instruction expresses a goal but does not supply missing facts or create a verification mechanism. Give the system trusted data and define what it should do when evidence is insufficient.

Mistake 4: Forcing an answer

If an evaluation or workflow treats “I don't know” as failure, the system may be pushed toward guessing. A reliable design should allow abstention or clarification when appropriate.

Mistake 5: Believing retrieval guarantees truth

Retrieval can ground an answer in documents, but the retrieved material may be irrelevant, incomplete, or incorrect. Day 12 will begin building the search layer needed for trusted-knowledge agents.

Knowledge check

1. An answer is fluent and specific. Does that prove it is correct?
No. Fluency describes presentation, not evidential support.

2. The notes do not contain a deadline. What should the Study Assistant do?
State that the deadline is missing and ask for a source such as the assignment page or calendar.

3. What are the three audit labels?
Supported, uncertain, and invented.

4. Can retrieval eliminate every hallucination?
No. Retrieval can improve grounding, but source relevance, source quality, generation, and verification can still fail.

Download the handwritten notes

Use the one-page notes as a quick reminder of the evidence-first mental model, the Friday-deadline example, and today's claim-auditing exercise.

Continue learning with Korshub

Korshub helps learners discover course listings and continue to the official course platform. If you want to study AI agents, retrieval-augmented generation, or language-model reliability in more depth, explore relevant course options and check the recorded price history where available before choosing.

Explore courses on Korshub

Series navigation

• Previous: Day 10 — Build a Study Planner Agent

• Roadmap: AI Agents in 30 Days

• Next: Day 12 — Embeddings and Semantic Search