Why Your RAG System Gives Confident Wrong Answers

It is not hallucination, it is an unopposed guess

The word hallucination suggests one phenomenon with one cure. In a retrieval system there are at least six distinct mechanisms, they occur at different stages, and they need different fixes. Separate two questions that get conflated constantly: did the system have the evidence, and given the evidence, did it answer correctly. The first is retrieval, covered by the four gates in RAG explained for builders. This guide is about the second.

Terms used precisely in this guide

Groundedness

The property that every claim in an answer is supported by a passage that was actually retrieved and placed in the model's context, rather than recalled from training.

Distractor

A retrieved passage matching the query's topic and language closely, but concerning a different entity, date, version or jurisdiction, so it scores highly while being the wrong evidence.

Abstention

The system declining to answer because the retrieved evidence fails a defined test, returning a stated reason rather than a best guess.

Entailment check

A verification step asking whether a specific passage supports a specific claim, run after generation and by a model that did not write the answer.

Supersession

The relationship between an older document version and the newer one that replaces it, recorded as metadata so retrieval can prefer or exclude the superseded copy.

The six mechanisms, and the test that exposes each

Every wrong answer from a grounded system fits one of six shapes. Classify the failure before you fix anything, because the fixes are not interchangeable and four of the six have nothing to do with the prompt.

Distractors: the failure a better model does not fix

The hardest wrong answers come from passages that are correct text about the wrong thing. Dense embeddings encode topic and phrasing, so a 2021 refund policy and a 2026 refund policy sit almost on top of each other in vector space, and the tokens that distinguish them contribute almost nothing to a vector built to capture meaning.

What actually works against distractors

1. Hybrid retrieval. Keyword search treats a version string as a token that either matches or does not. Cheapest and largest improvement for corpora full of identifiers and dates.
2. Metadata filters at query time. If the question names a year, product or region, filter before ranking rather than hoping ranking sorts it out. Extract those entities from the query and pass them as filters.
3. Supersession edges. When ingesting version seven, mark version four replaced and exclude it by default. Keep it in the store for audits, out of the candidate pool.
4. Effective dates in the chunk text. A model reading 'effective 2024-03-02, superseded' has a chance. A model reading a bare paragraph has none.
5. A conflict rule in the prompt. Two passages that disagree must both be surfaced with their dates. Merging two versions produces a policy that has never existed and reads perfectly.

The Four Silences: when the system must decline

A system that never refuses is not confident, it is uncalibrated. Four conditions call for a refusal or a qualified partial answer, each with a different detection signal and a different thing to say. Build them as explicit branches with their own logging, because one generic fallback teaches you nothing about which condition fired.

Making silence the easier output

Prompting alone does not produce grounded answers, but a well-built prompt removes a large share of the avoidable failures and costs nothing to deploy. The instruction that fails is the vague one: telling a model to use only the provided context is a request it honours until the context is long, the evidence is thin, and continuing is easier than stopping.

What works is making the required output structurally impossible to fake. Demand a source id on every factual sentence and an unsupported claim becomes visible to a script rather than to a reviewer. Give refusal a short fixed form and writing NO_SOURCE becomes the path of least resistance.

SYSTEM

You answer questions using only the SOURCES block below. You have no other
knowledge for this task. If the sources do not settle the question, say so.
Saying so is a correct answer and is preferred to a plausible one.

RULES
1. Every factual sentence you write must end with the id of the source that
   supports it, in square brackets, like [S3]. A sentence with no id is a
   rule violation, not a stylistic choice.
2. Numbers, dates, names, versions and amounts must appear verbatim in a
   source. Never compute, convert, round or infer one.
3. If two sources disagree, do not merge them. Report both, name the
   effective date of each, and state that the question is unsettled.
4. If the sources cover the topic but not the specific question asked,
   answer only the part they cover and name the part they do not.
5. If no source supports an answer, reply with exactly:
   NO_SOURCE: <one line naming what document would have contained it>
6. Do not use phrases implying certainty you cannot support, such as
   "always" or "in all cases", unless a source uses them.

OUTPUT
Return JSON only, matching this shape:
{
  "status": "answered" | "partial" | "conflict" | "no_source",
  "answer": "the answer text, every factual sentence carrying a [id]",
  "used": ["S1", "S3"],
  "unsupported": ["any claim you wanted to make but could not source"],
  "conflict": [{"claim": "...", "sources": ["S2", "S5"]}]
}

SOURCES
[S1] (doc: refunds-policy, version 7, effective 2026-01-12)
<text>

[S2] (doc: refunds-policy, version 4, effective 2024-03-02, SUPERSEDED)
<text>

QUESTION
<the user question, verbatim>

# Rule 1 gives you a machine-checkable output: an id the model cannot supply
# marks a claim it invented. Rule 2 catches arithmetic performed silently on
# retrieved numbers. Rule 5 makes refusal shorter to write than a paragraph.
# Repeat the rules after the SOURCES block as well as before it, because
# instructions far from the end of a long context lose to what sits between.

Verify after generation, because prompting is not enough

Add a second pass that checks the answer against the passages actually retrieved. Prompting reduces the failure rate. Verification is what lets you state a number for it, and an unmeasured error rate cannot be improved deliberately. Run the cheap deterministic checks first.

• Numeral check. Every number, amount, percentage and date in the answer must appear verbatim in the retrieved context. Any that does not fails the answer, because the model computed it.
• Entity check. A proper noun appearing in the answer and nowhere in the context is a strong signal of training-data leakage into a supposedly grounded reply.
• Citation coverage. Every factual sentence carries an id, and every id refers to a passage genuinely in the context. Missing ids and invented ids both fail automatically.
• Quote match. Text presented in quotation marks must appear character for character in the cited source. Fabricated quotes are the most damaging and the easiest to detect.

What survives goes to an entailment pass: for each claim and its cited passage, does the passage support the claim. Use a small dedicated model or a separate judge call with a narrow rubric, and never let the model that wrote the answer grade itself, because it agrees with itself. Span-level mechanics are in citations and grounding.

Build the adversarial set nobody builds

Most evaluation sets contain only questions the system can answer, so they measure fluency and never test honesty. A set built to catch confident wrongness needs questions designed to make the system fail, labelled with the correct behaviour, including refusal.

1. Collect thirty real answerable questionsthe baseline

   Real user wording from a support inbox or chat log, each labelled with the document and section that answers it. This is the floor everything else is measured against.

2. Add ten unanswerable questionsmeasures honesty

   Questions about topics your corpus genuinely does not cover, phrased like real ones. The expected output is a refusal, and the abstention rate here tells you whether the system bluffs.

3. Add ten near-miss questionsmeasures distractor resistance

   Take answerable questions and swap one entity: the year, the plan tier, the region. The correct answer is the right passage or a refusal. Anything drawn from the original passage is a distractor failure.

4. Add five superseded-fact questionsmeasures corpus hygiene

   Questions whose answer changed between document versions. The system must give the current answer and ideally note that it changed. Quoting the old version is the failure you are hunting.

5. Add five split-evidence questionsmeasures assembly

   Questions needing two passages, such as a rule in one section and its exception in another. Half an answer is a failure, and this is where over-eager chunking shows up.

6. Score by category, never by a single averagethe reporting rule

   One blended accuracy number lets a gain on easy questions hide a collapse in abstention. Report five numbers and treat any drop in the unanswerable category as a release blocker.

7. Re-run on every change and keep the historythe discipline

   Embedding model, splitter, reranker, prompt, model version: any of these moves any of these numbers. Catching model regressions covers wiring it into a release process so it happens without anyone remembering.

Log enough to diagnose the next one in ten minutes

The difference between a ten minute diagnosis and a lost afternoon is whether you recorded what the system retrieved. Without the retrieved ids and their scores you cannot tell an empty retrieval from a distractor from a model that ignored good evidence, and you will end up rewriting the prompt on a hunch.