RAG Explained for People Who Have to Build It

RAG in one sentence, then the honest version

The honest version has more parts. A question arrives. Something rewrites it. Two indexes are queried, one matching meaning and one matching exact tokens. The lists are fused, a reranker reorders the survivors, and a handful of passages go into a prompt with the question. The model writes an answer and, if you built it properly, tags each claim with the passage it came from. Nine of those steps are ordinary information retrieval. One is the model. Teams routinely reverse that ratio in where they spend attention.

The words, used precisely

Retrieval augmented generation (RAG)

An architecture where a language model answers using passages fetched at query time from an external store, rather than from what it memorised in training.

Chunk

A passage of source text stored and indexed as one retrievable unit. It is both the smallest thing the system can find and the smallest thing it can cite.

Hybrid retrieval

Running a dense vector search and a keyword search over the same corpus and merging the two ranked lists, so both paraphrase and exact identifiers are findable.

Reranker

A second stage model that scores each candidate passage against the query directly, rather than comparing two independent vectors, and reorders the shortlist.

Grounding

The property that every claim in an answer is supported by a passage that was actually retrieved and placed in the model's context.

The pipeline, stage by stage

Eight stages sit between a document in a shared drive and a sentence in an answer. Each can destroy information silently, and the damage is invisible downstream because later stages cannot know what earlier ones threw away.

1. Ingest and normalisefails quietly

   Convert the source to text and keep the structure. This is where PDFs eat their own tables and scanned pages yield nothing without OCR. Print a random sample and read it, because most teams never do and never learn that a tenth of the corpus is unusable.

2. Split into chunksdecides your ceiling

   Fixed token windows are fastest to build and most likely to sever a claim from the heading that scoped it. Splitting on structure costs more and caps the quality everything downstream can reach.

3. Enrich each chunkcheap, high return

   Attach source path, document title, heading trail, version, effective date and access group. Prepend a situating line so a chunk reading 'this applies only to annual plans' still means something alone.

4. Embedone way door

   Record which model produced each vector. Vectors from two models are not comparable, so a model change means re-embedding everything and recalibrating every threshold.

5. Indextwo indexes, not one

   Vectors into an approximate nearest neighbour index, the same text into a keyword index. Keep metadata queryable so you filter by access group and date during retrieval, not after generation.

6. Retrieveload bearing

   Query both indexes, take twenty to fifty candidates from each, fuse the lists. Reciprocal rank fusion sums one over a smoothing constant plus each rank, so no single confident retriever dominates.

7. Rerank and assemblewhere precision lives

   Score candidates with a cross-encoder, keep a few, deduplicate near-identical passages, then order deliberately. Evidence buried mid-context is used less reliably than evidence at either end, so lead with the strongest.

8. Generate and attributethe visible part

   Send instructions, the passages with stable ids, and the question. Require a source id on every claim and give the model an explicit branch for insufficient evidence.

Stages one and two produce no error messages. A parser that drops every table exits with status zero, and the failure surfaces months later as a user saying the assistant is useless for pricing. Chunking strategies covers stage two, which has the widest quality range and the least discussion.

Where retrieval quality actually comes from

Quality comes from combining retrievers with different failure modes, not from finding one perfect retriever. Dense vectors fail on identifiers, keyword search fails on paraphrase, and each covers the other's blind spot.

The Retrieval Ladder: four gates every answer has to pass

When an answer is wrong there are exactly four places the failure can be, and each has a different fix. Most wasted RAG effort is a gate three fix applied to a gate one problem: swapping embedding models for weeks while the fact was never in the corpus.

The arithmetic, with the assumptions on the table

RAG has two cost curves that behave differently. Indexing is roughly one-time, proportional to corpus size, and paid again on any embedding or splitter change. Answering is per question, dominated by the passages you paste in, and you pay it on every call forever.

Move top-k from six to twenty and watch the dominant term triple, for a recall gain a reranker would have delivered far cheaper. This is the most common way a RAG system becomes expensive: somebody raised top-k to fix a ranking problem, it appeared to help, and nobody revisited it.

The questions RAG will never answer well

Retrieval finds passages. If the answer is stated in no single passage, retrieval cannot find it, and the model assembles something plausible from the fragments it did get. No amount of chunking work fixes this, so know the shape of these questions before promising a system that handles them.

• Aggregation. How many contracts included a termination clause last quarter. The answer lives in a count across thousands of passages, not in one. Route it to SQL.
• Superlatives. Which supplier gives the longest payment terms. Worse than aggregation, because the model names whichever supplier happened to be retrieved.
• Negative existence. Do we have any policy on this. Retrieval returns the nearest thing regardless, and a near thing reads exactly like a hit.
• Multi-hop chains. Who approved the change that caused the outage. The second query depends on the first answer, which is an agent loop rather than one retrieval pass.
• Time sensitive questions with no version metadata. Three editions of a policy and no effective dates means the system is guessing which one you meant.

Build the thin slice first

The fastest route to a working system is one narrow question type, end to end, on real documents, in front of real users, in about two weeks. Breadth after that is mostly repetition. Breadth before it is how six month RAG projects happen.

How to tell whether it is working

Measure retrieval and generation separately. One end-to-end score tells you the system got worse and nothing about which half did it, so every regression becomes an argument. Two numbers turn the same regression into a ten minute diagnosis.

# retrieval_eval.py
# Run this before you touch the prompt. It measures the search, not the writing.
#
# gold.jsonl, one line per question:
# {"q": "what is the refund window on annual plans",
#  "gold_ids": ["policy-refunds-v7#s3"], "must_abstain": false}

import json, statistics

def recall_at_k(hits, gold, k):
    return 1.0 if set(h["id"] for h in hits[:k]) & set(gold) else 0.0

def reciprocal_rank(hits, gold):
    for i, h in enumerate(hits, 1):
        if h["id"] in gold:
            return 1.0 / i
    return 0.0

rows = [json.loads(line) for line in open("gold.jsonl")]
r5, r20, mrr = [], [], []

for r in rows:
    hits = retrieve(r["q"], k=20)          # your retriever, untouched
    r5.append(recall_at_k(hits, r["gold_ids"], 5))
    r20.append(recall_at_k(hits, r["gold_ids"], 20))
    mrr.append(reciprocal_rank(hits, r["gold_ids"]))

print("recall@5 ", round(statistics.mean(r5), 3))
print("recall@20", round(statistics.mean(r20), 3))
print("mrr      ", round(statistics.mean(mrr), 3))

# How to read the gap:
#   recall@20 high, recall@5 low  -> ranking problem, add a reranker
#   both low                      -> indexing or chunking problem
#   both high, answers still bad  -> prompt or context assembly problem

The answer side needs its own scoring: is every claim supported by a retrieved passage, does the system abstain on unanswerable questions, does it cite what it used. Writing evals for LLM systems covers the generation half, and citations and grounding covers making answers checkable.

What it costs to keep one alive

A RAG system is an index that has to stay synchronised with a corpus people keep editing, and that ongoing obligation is the part nobody budgets for.

If you are scoping a build, price the operating side explicitly instead of treating it as a rounding error on the build. Our approach to that split is on the RAG and knowledge systems page.