Hallucination: a fact that is not in the sources

A hallucination is a fact the model stated that is not in the sources you gave it. It sounds sure. That is the problem.

This post is how to catch one, how to stop showing it, and a small set of cases you can rerun.

What it is not

It is not a creative story you asked for.

It is not a wrong number from a tool. That is a tool bug.

It is not “the model used a synonym.” If the source says 75% and the answer says three-quarters, that can be fine if you still cite the rule.

It is a heading that does not exist. A fee that is not in the file. A “the teacher said Friday” when no one said Friday.

Sure tone is not a source. Open the line or drop the sentence.

Catch it with a place

Ask for a cite. Open it. You already have that rule.

Also check invented names: people, rooms, circulars, dates. If the name is not in the chunk list, fail.

A simple checker:

1. Split the answer into sentences.
2. For each sentence, require a chunk title from the list you retrieved.
3. If a sentence has no title, or the title is outside the list, drop that sentence or drop the whole answer.

You will over-drop at first. That is safer than a fake fee.

Three cases

Bait is the useful one. “When is the makeup class?” when the file never says. If the answer invents Thursday, you caught a hallucination.

Write the bait question down. Do not rely on memory. You will go easy on a fluent Thursday.

Walk through once

File: only the 75% rule and two examples. No calendar.

Question: “When is the extra class for people below 75%?”

A grounded system says it does not know.

A leaking system says “usually Saturday” or “ask the HOD on Monday.” Those words feel helpful. They are not in the file.

Add a second bait: “What is the hostel fee?” Same file. Same refuse.

Keep both baits. After every prompt change, run them plus one grounded question. If the baits pass and the grounded one dies, you made the model too quiet. Loosen the wording, not the cite rule.

What good looks like

A grounded question gets a short answer and a real title.

A bait gets no date, no fee, no “usually.” It can say the notes do not have it.

An empty retrieve does not draft. You never see a helpful paragraph born from silence.

The tone can stay polite. Polite is not the same as filling the hole.

If you only ever demo the grounded question, you have a reading app, not a checked one.

Where the extra sentence hides

The model often gets the rule right, then adds a kindness: “You can also email the HOD by Friday.”

That last sentence is the hallucination. The rule was in the file. Friday was not.

Your checker that only looks at “does the answer mention 75%” will pass. Split by sentence. Each sentence needs a home, or it is cut.

This is why “mostly right” is a bad score. The damage is in the extra clause.

Write that down next to the bait list: extra clauses are first-class fails, not style.

If the heading is missing, the sentence goes. Do not keep it because it sounds kind.

Why “be accurate” fails

You can add that line. Some days it helps. It is not a check.

The model cannot feel accuracy. Your code can see a title that is not in the list. Use that.

If you have no sources — open chat, no files — you cannot catch this the same way. Then do not present the answer as from a document. Say it is a general guess, or do not build that product.

Common mistakes

Only testing questions you know. You will not bait.

Keeping a pretty extra sentence. That sentence is where the date sneaks in.

No empty-chunk stop. Then the model fills silence.

Punishing synonyms. You will force ugly copies and still miss fake dates.

A second model as judge with no field. Now you have two talkers.

Practise

1. One grounded question. Cite opens.
2. Two bait questions. No invented date or fee.
3. Empty chunks. No draft.
4. A checker that drops a sentence with a title not in the list.
5. Rerun after every prompt edit.

Two refuses and one pass. That is a set, not a single demo.

Checklist

• You can say hallucination in one line: a fact not in the sources.
• Bait questions exist and are written down.
• Empty chunks do not draft.
• A title outside the list fails.
• “Be accurate” is not your only control.

The next step is two bait questions, not a sterner adjective.