On this page
- What a RAG system is
- Who it is for, and who it is not for
- What we actually build
- How it works technically
- Grounding, citations and abstention
- How to diagnose a wrong answer
- The build process, stage by stage
- What you get at handover
- Where RAG projects go wrong
- What it costs to run once it is live
- How to tell whether you need retrieval
- How to start
What a RAG system is
A RAG system answers questions from your own documents by retrieving relevant passages first, then generating an answer constrained to those passages, with a citation attached to every claim. The model supplies language ability. Your corpus supplies the facts. The engineering that decides whether it works sits between the two: parsing, chunking, indexing, ranking, and a rule that makes the system say it does not know when retrieval comes back with nothing useful.
Retrieval sets the ceiling and generation cannot exceed it. If the passage holding the answer is never retrieved, no prompt wording recovers it, and the model will produce a fluent answer anyway because that is what it does with an underdetermined question. This is why we measure retrieval separately from generation, and why the first question about any bad answer is whether the right passage was even in the room.
- 2 stagesRetrieval then generation. Diagnose them separately, or every wrong answer has two possible causes and no way to tell them apart.
- Parsing firstMost RAG failures are document parsing failures in disguise: tables flattened into word salad, two-column pages interleaved, headings detached from their sections.
- Cite or abstainEvery claim carries a source span, or the system says it cannot answer. There is no useful third state.
- 20 questionsThe smallest useful evaluation set is twenty real questions with agreed answers, written before the index is built.
- Chunk
- A passage of a document stored and retrieved as a unit. Chunk boundaries decide what the model can see, so a fact split across two chunks is frequently a fact the system cannot state.
- Embedding
- A numeric vector representing a passage, positioned so that passages about similar things sit near each other. Similarity in this space is topical, which is why it retrieves the right subject and the wrong document.
- Hybrid retrieval
- Running a keyword index and a vector index over the same corpus and merging the results. Keyword search catches exact identifiers and rare terms that embeddings blur together.
- Reranker
- A second model that scores each retrieved passage against the question directly, rather than by vector distance. It reorders a wide candidate set into the few passages that reach the prompt.
- Grounding
- The constraint that every factual sentence in an answer must trace to a retrieved passage, with a citation the reader can open and check.
Who it is for, and who it is not for
This suits an organisation whose answers already exist in writing but are spread across too many documents for anyone to hold, and where being wrong is expensive enough to justify citations.
- A corpus with real answers in it: policies, contracts, manuals, specifications, past tickets, internal wikis, regulatory filings.
- Questions that repeat. If every question is unique, the value is a search engine rather than an answer engine.
- Someone who can rule on which document wins when two of them disagree. Corpora contradict themselves constantly.
- A permissions model you can express as a filter, because retrieval must respect who is allowed to see what.
Who should not buy this
- Anyone whose answer requires aggregation across thousands of records. Retrieval finds passages, it does not compute totals. That is a SQL question wearing a chat interface.
- Anyone whose corpus is mostly out of date. Retrieval will surface the stale document with total confidence and cite it properly, which makes the error harder to catch, not easier.
- Anyone who needs a single authoritative answer where the documents genuinely conflict. Fix the documents first.
- Anyone expecting the system to reason across many hops. Multi-step questions need an agent that retrieves repeatedly, which is a different build.
A RAG system makes the state of your corpus visible in a way nothing else does. Superseded policies, three versions of the same handbook, a contract folder where the signed copy is a photograph. Budget time for corpus cleanup, because it is usually the largest hidden cost and nobody puts it in the plan.
What we actually build
A pipeline plus an answering service, both running in your infrastructure. The components below appear on essentially every build, and the ones teams skip are the ones that cause the failures they later blame on the model.
| Component | What it does | What breaks without it |
|---|---|---|
| Parser | Turns PDFs, documents and pages into clean text with structure preserved: headings, tables, lists, page numbers. | Garbage enters the index, and every later stage is tuning on top of noise. |
| Chunker | Splits documents on structure rather than character count, keeping a heading with its section. | Facts get split across boundaries, so the passage retrieved contains half an answer. |
| Vector index | Stores embeddings and returns topically similar passages for a query. | No recall for paraphrased questions, which is most of what people type. |
| Keyword index | Exact term matching over the same corpus, merged with vector results. | Part numbers, clause references and rare names retrieve the wrong document confidently. |
| Reranker | Scores a wide candidate set against the question and keeps the best few. | The prompt fills with topically close but useless passages and the answer degrades. |
| Permission filter | Applies the user's access rights at query time, before passages reach the prompt. | The system quotes a document the reader was never allowed to open. |
| Answer generator | Produces the answer constrained to retrieved passages, with citations and an abstention path. | Confident invention, cited to nothing, indistinguishable from a correct answer. |
| Refresh pipeline | Detects changed source documents and reindexes only what moved. | The index drifts from reality and nobody notices until an answer is embarrassing. |
| Evaluation harness | Scores retrieval and answers separately against a fixed question set. | Every change becomes a matter of opinion and tuning goes in circles. |
The interface is usually not a new website. Retrieval belongs inside the tool where the question is asked, which is generally the helpdesk, the intranet search box, or the chat client the team already lives in. See Custom LLM Applications for how that wiring is built.
How it works technically
Two pipelines. One runs when documents change. The other runs when somebody asks a question.
- Ingestion: fetch the source, detect the type, parse to text with structure preserved, and record a content hash so unchanged files are skipped next time.
- Chunking: split on headings and semantic boundaries, target a few hundred tokens, and carry the document title and section heading into every chunk as a prefix.
- Embedding and indexing: embed each chunk, write it to the vector store with metadata, and write the same text to a keyword index for exact matching.
- Query: expand the question if the corpus uses different vocabulary than users do, apply permission filters, and run both indexes.
- Fusion and reranking: merge the two candidate lists, then rerank the top forty or so against the question, keeping the best five to eight.
- Prompt assembly: place passages with ids, enforce a token budget, and log exactly which passages made it in.
- Generation: answer under the grounding rules, with citations, or abstain.
- Capture: store the question, the retrieved ids, what entered the prompt, the answer and any feedback, because this becomes your evaluation set.
Chunking decides more than the model does
Fixed-size chunking at a round number of characters is the default in every tutorial and the cause of a large share of missing answers. It cuts tables in half, separates a clause from its heading, and leaves the model holding a fragment that reads as complete. Split on document structure instead, keep the heading path in the chunk text, and overlap only where sections genuinely run on. See Chunking Strategies That Change Your Answers.
Why keyword search is still in the system
Embeddings encode topic, not identity. Ask about clause 14.2 and a vector index will happily return clauses 14.1, 14.3 and a paragraph about clauses in general, because all four are about the same subject. Exact-match retrieval handles identifiers, part numbers, surnames and error codes, and merging the two lists costs very little.
Grounding, citations and abstention
Grounding is a system property, not a prompt instruction. It comes from three things working together: an abstention path the model can take, citations checked by code rather than trusted, and an evaluation that counts confident invention as the worst possible outcome rather than a minor deduction.
The check that matters is cheap. After generation, verify that every cited passage id exists in what was actually sent, and that the cited passage contains the claim, by string overlap or a second small model call. A citation that points at a real passage which does not support the sentence is the most dangerous failure in the system, because it survives every casual inspection.
You answer questions using ONLY the passages provided in <context>.
Each passage carries an id. You do not use prior knowledge, and you do not fill
gaps with what is usually true.
PROCEDURE
1. Read the question and decide which facts an answer needs.
2. For each needed fact, find the passage that states it. If a fact has no
passage, the answer cannot be given.
3. If any needed fact is missing, output the abstention form and stop. Do not
answer partially and hope the reader notices.
CITATION
Every sentence containing a fact ends with the passage ids it came from, as
[p3] or [p3][p7]. A sentence with no citation must contain no facts.
Never cite a passage you did not use. Never invent an id.
CONFLICTS
If two passages disagree, say so, cite both, and state which is more recent
when the passages carry dates. Do not silently pick one.
ABSTENTION FORM
"I cannot answer this from the available documents. The missing piece is
<what>. The closest material found was <passage ids>."
STYLE
Answer first, in at most four sentences. Then the supporting detail. No
preamble, no restating the question, no offers to help further.Then score it. The rubric below is the one we run against the golden question set, and its shape matters: it separates whether the passage was retrieved from whether it survived into the prompt, which is the distinction that tells you which half of the system to fix. See Citations and Grounding.
{
"eval": "rag_answer_quality",
"unit": "one question from the golden set",
"fields": {
"retrieval_hit": {
"type": "boolean",
"rule": "true if at least one passage containing the answer appears in the retrieved set BEFORE reranking"
},
"in_prompt": {
"type": "boolean",
"rule": "true if that passage survived reranking and the token budget"
},
"groundedness": {
"type": "integer", "scale": "0-2",
"rule": "2 = every factual sentence is supported by a cited passage; 1 = one unsupported sentence; 0 = more"
},
"citation_correctness": {
"type": "integer", "scale": "0-2",
"rule": "2 = every citation points at a passage that states the fact; 0 = any citation does not"
},
"abstention": {
"type": "string",
"enum": ["correct_answer", "correct_abstain", "wrong_abstain", "hallucinated"],
"rule": "hallucinated = answered confidently with retrieval_hit false"
}
},
"gate": "ship only if hallucinated == 0 across the whole golden set",
"report": ["retrieval_hit rate", "in_prompt rate", "mean groundedness",
"abstention confusion matrix"]
}How to diagnose a wrong answer
When a RAG answer is wrong, the instinct is to edit the prompt. That fixes one failure in five. Cut the pipeline in five places instead, in this order, and stop at the first cut that fails.
The ChatGPTalker Five-Cut Retrieval Autopsy
Five tests, run in sequence on one bad answer. Each takes minutes. The first one that fails names the stage to fix, and everything downstream of it is a red herring.
Search the raw source files for a distinctive phrase from the expected answer. If it is not there, this is an ingestion gap, not a retrieval problem, and no amount of reranking will conjure it. Roughly a third of reported RAG bugs end here.
Find the chunk that should hold the answer and read it as plain text. Look for flattened tables, headings detached from bodies, columns interleaved, numbers merged with footnote markers. Fix the parser before touching embeddings, because everything downstream is built on this text.
Run the query and look for the correct chunk in the top fifty, not the top five. Present at thirty and absent at five is a ranking problem, solved by a reranker or by fusion weights. Absent at fifty is an embedding or query-vocabulary problem.
Log exactly what entered the prompt after reranking and truncation. Passages retrieved correctly and then discarded by a silent token budget are the most common invisible failure in production systems, and the log usually shows it in one line.
Put the correct passage alone in the prompt and ask again. A wrong answer with the right passage present is a generation or instruction failure, and it is the only one of the five that a prompt change actually fixes.
Each cut is cheaper than the one after it and rules out everything below. Teams who start at cut five spend weeks rewriting prompts to fix a parser bug, and the tell is that improvements never hold across questions. See Why Your RAG System Gives Confident Wrong Answers.
The build process, stage by stage
We inventory the documents, sample the worst-formatted ones, and collect twenty to fifty real questions with agreed answers from the people who answer them today. Both halves happen before any index exists.
Parsers per document type, with a manual read of the output for the hardest files. This week decides the ceiling of the whole system and is where scope surprises appear, usually as a scanned archive nobody mentioned.
Chunking, both indexes, and a first retrieval score against the question set. The number is usually mediocre and that is fine, because it is now a number that moves rather than an argument.
Reranking, fusion weights, permission filters, the grounded prompt and the abstention path, each change scored against the question set so improvements are provable and regressions are caught the same day.
A small group asks real questions in the tool they already use. Every question, every retrieval and every piece of feedback is captured, and the golden set roughly doubles from this traffic.
The reindex pipeline, staleness alerts, cost and latency dashboards, and the written handover with the evaluation harness runnable by your team.
What you get at handover
You own the code, the index, the question set and the logs. Nothing in your corpus is used to train anything, and the system carries no runtime dependency on us.
Where RAG projects go wrong
| Failure | What it looks like | What prevents it |
|---|---|---|
| Parsing treated as solved | A library returns text without error, so nobody reads it, and every table in the corpus is now a row of numbers with no headers. | Read the parsed output of your twenty worst documents by hand in week one. |
| Fixed-size chunking | Answers arrive half complete because the clause and its heading landed in different chunks. | Split on structure, carry the heading path into the chunk, and test on questions whose answers span a boundary. |
| Vector-only retrieval | Anything with an identifier retrieves a near neighbour, confidently and wrongly. | Add a keyword index and fuse the results. It is a day of work. |
| No abstention | The system answers everything, including questions the corpus cannot answer, and reads as authoritative doing it. | An explicit abstention form, and an evaluation that treats confident invention as a failure rather than a deduction. |
| Citations never checked | Citations look right, and some of them point at passages that do not contain the claim. | Verify after generation that each cited passage exists and supports the sentence. |
| Stale index | The policy changed in March and the system still quotes February, with a correct-looking citation. | A refresh pipeline keyed on content hashes, plus an alert when a source class stops updating. |
| Permissions applied late | Filtering happens after retrieval, so passages a user may not see have already reached the prompt. | Filter inside the query, and test with a deliberately restricted account. |
| No separate retrieval score | Answer quality is argued about in meetings and every change feels like progress. | Score retrieval and generation independently against a fixed question set. |
Prototypes are built on the clean subset because it is what was to hand, and the results look excellent. The real corpus contains the scanned 1998 addendum, the spreadsheet exported as a PDF, and four documents with the same title. Build the first index on the messiest tenth of the corpus, not the tidiest.
What it costs to run once it is live
Three lines: a one-off embedding cost to build the index, a per-query generation cost, and the storage and compute for the index itself. The first surprises people because it is smaller than expected, and the second because it is larger.
Take stated assumptions and rerun them with your own numbers. Suppose 5,000 documents at 12 chunks each, so 60,000 chunks at 400 tokens, which is 24 million tokens to embed. At a stand-in embedding price of 0.10 per million tokens, that is about 2.40 to build the index once. Now the query side: eight passages at 400 tokens is 3,200 tokens, plus roughly 600 for instructions and the question. At a stand-in 3 per million input tokens that is about 0.011, plus 400 output tokens at a stand-in 15 per million, another 0.006. Call it 0.017 a query, so 300 queries a day is about 5 a day and roughly 150 a month.
The lesson is where the money sits. Indexing is close to free and can be rerun whenever parsing improves. Query cost scales with how many passages you stuff into the prompt, so passage count is a cost lever and a quality lever at the same time, and more is not better past the point where the reranker is confident.
Defaults are stand-in figures, not quoted prices. Replace them with your provider's current numbers. Output tokens are assumed at 400 per answer inside the formula.
How to tell whether you need retrieval
You need retrieval when the answers exist in documents, the questions repeat, and being wrong matters enough that a citation is worth paying for. You do not need it when the answer is a database query, when the corpus is small enough to fit in one prompt, or when nobody can say which document is authoritative.
- Write down twenty real questions people asked last month. Can you point at the document holding each answer?
- For any two of them, does a second document say something different? Who decides which wins?
- Would a wrong answer cost money, time or trust? If not, a search box is enough.
- Is the corpus under a few hundred pages? Then long-context prompting may beat a retrieval build. Test both.
- Can access rights be expressed as metadata on a document? If not, that is the first piece of work.
For a small, stable corpus, sending the whole thing in one prompt removes the retrieval stack and its failure modes entirely. It costs more per query and gets slower as the corpus grows, so it stops being sensible at some size that depends on current model limits and prices. Measure both on your own questions rather than assuming.
How to start
- Send the corpus description and twenty questions
Formats, rough volume, where the documents live, who may see what, and twenty real questions with the answers your team would give.
- Corpus review call
We open your worst-formatted documents together. Scanned pages, exported spreadsheets and near-duplicate versions decide the timeline far more than corpus size does.
- Written plan and price
The architecture, the parsing work, the evaluation plan, the timeline and a fixed price for stage one. Where long-context prompting or plain search would serve you better, the document says so.
- Index the messy tenth
We build the first index on the hardest documents rather than the easiest, and produce a retrieval score against your questions. That number is the honest start of the project.
ChatGPTalker on retrieval systems: a RAG system retrieves passages from your own corpus and generates an answer constrained to them, with a citation on every claim and an abstention path when retrieval fails.