On this page
- The short answer
- What an eval is, and the four things people confuse it with
- The Assertion Ladder, and why to climb as little of it as possible
- Where a model judge quietly lies to you
- A judge rubric you can lift today
- Your suite probably cannot see the regression you care about
- Scoring without lying to yourself
- Wiring evals into the way you actually build
- Terms worth being precise about
- Before you trust the number
The short answer
An eval is three things: a frozen set of inputs, assertions that decide pass or fail on each one, and a score you can compare between two versions of your system. Most teams build the score and skip the other two, which is why the number moves when nothing changed. Write the cheapest assertion that would have caught the failure you actually saw in production, run every case several times because the system is not deterministic, and report per slice of traffic rather than as one average. A hundred case suite cannot see a five point regression, and that is arithmetic, not pessimism.
- 5rungs on the assertion ladder, from a schema check up to human adjudication
- ±7.8 ptsthe 95 percent margin of error on a 100 case suite scoring 80 percent, before anything changesBinomial standard error, 1.96 * sqrt(p(1-p)/n) at p = 0.8, n = 100
- 3+runs per case, because one sample from a non-deterministic system is an anecdote
- 0eval cases whose expected answer came from the model you are testing
The reason evals feel like busywork is that the first one is usually written to prove the system works rather than to find out where it does not. That version passes on day one, passes forever, and tells you nothing. A useful suite is built the other way round: every case exists because something went wrong, or because two people argued about whether an output was acceptable and the argument had to be settled.
What an eval is, and the four things people confuse it with
An eval is a regression test whose assertion is allowed to be fuzzy. That is the whole idea. Everything else in the discipline exists to stop the fuzziness spreading into the parts that should be exact.
| What people call an eval | What it measures | What it cannot tell you |
|---|---|---|
| A public benchmark score | A base model on somebody else's task distribution | Anything about your prompt, retrieval, tools or users |
| A notebook of ten tried prompts | Whether the author liked the outputs that day | Whether the next change makes it worse, since nothing is stored |
| Thumbs up and down in the product | The opinion of the small fraction of users who click | Silent failure, because nobody rates an answer they believed |
| An offline eval suite | Behaviour on inputs you chose, under assertions you wrote | Behaviour on inputs you never imagined, which is most of production |
| An online metric such as edit rate | What actually happened to real work | Which of your nine changes caused the move |
The consequence is a division of labour. The offline suite is fast, cheap, repeatable and blind to the world. The online metric is slow, hard to attribute and the only thing that tells you whether the system is useful. Use the first to decide whether a change is safe to ship and the second to decide whether shipping it was worth doing. With only the second, every incident becomes an argument about which change caused it.
- A case is one input plus everything needed to reproduce the run: the corpus version, the tool responses, the user context.
- An assertion is one decidable statement about that output. A case usually carries several.
- A slice is a tag on the case: language, document type, customer tier, intent. Slices are how you find the regression the average hides.
- A suite is the cases plus the schedule and the thresholds that decide what happens when the score moves.
The Assertion Ladder, and why to climb as little of it as possible
Every assertion sits on one of five rungs. They get more expensive, slower and less trustworthy as you climb. The discipline is to write each assertion on the lowest rung that catches the failure you are targeting, because most of what teams grade with a model judge could have been a string comparison.
The Assertion Ladder
Five rungs, cheapest first. Start every new assertion at rung one and stop climbing the moment the failure is caught.
Does the output parse. Are required fields present. Are enum values inside the allowed set. Is the response under the length contract. These run in milliseconds, never disagree with themselves, and catch a real share of production breakage, because a drifting system usually breaks its output shape before it starts being wrong.
Claims a computer can decide alone. Every cited document id appears in the retrieved set. Line items sum to the stated total. The extracted date falls inside the document's period. No phone number appears in an output that should not contain one. This is the most under-invested rung and the highest value one on the ladder.
Compare against a known correct answer. Exact match for classification and extraction, set overlap for multi-label tasks, and for free text an embedding similarity floor used as a smoke alarm rather than a grade. Honest when there is one right answer, misleading the moment there are several, which is why it does not survive summarisation or advice.
A model grades against written criteria, one verdict per criterion, each with a quoted piece of evidence. People reach for this first because it is easy to write and feels like it covers everything. It is the slowest rung, the most expensive, the only one that can be wrong in a correlated way across the whole suite, and it means nothing until it is calibrated.
A person decides, on a sample. Reserve it for criteria that are genuinely a matter of judgement, for building the calibration set rung four is measured against, and for the audit that tells you whether rung four has drifted. It is not a fallback for lazy assertions, it is the ground truth that makes the rung below interpretable.
Take your last three production incidents involving model output and ask which rung would have caught each. In most systems the honest answer for at least two of them is rung one or rung two, and the assertion is about four lines of code. Write those before you build any judge infrastructure.
Where a model judge quietly lies to you
A model judge is a measuring instrument that has opinions. Calibrate it or its numbers will be precise and wrong in a consistent direction, which is worse than having no number. These are the biases that show up in practice.
| Failure of the judge | How it shows up | Mitigation that works |
|---|---|---|
| Position bias | In a pairwise comparison, whichever answer comes first wins too often | Run every pair twice with the order swapped, and count it only if both runs agree |
| Length and fluency bias | Longer, better formatted answers score higher on criteria unrelated to length | One criterion at a time, binary verdict, quoted evidence required for every verdict |
| Self preference | The judge favours output from its own model family | Do not use the same model version as generator and judge. If you must, check the gap on a human labelled slice |
| Criteria collapse | You asked about grounding and the judge scored overall quality, because that is what the word good pulls in | Remove evaluative adjectives. Write each criterion as a testable statement, not a quality bar |
| Scale mush | A one to ten score where the judge only uses six, seven and eight | Binary per criterion, then count criteria passed. If you need a scale, give three anchored levels with an example of each |
| Judge drift | The score moves and nobody touched the system, because the judge model changed underneath you | Pin the judge model version, treat it as a production dependency, re-run calibration when it changes |
Have humans label a set of outputs against the same rubric the judge uses, covering boundary cases rather than obvious ones, then measure how often judge and human agree. Publish that agreement number next to every judge derived score, permanently. If agreement is poor the fix is the rubric, not a bigger model: ambiguous criteria produce disagreement in humans first and in judges second.
One more trap. A judge that grades your output while reading the same retrieved context your system read inherits your retrieval failures. If the context is missing the answer, both are working from the same hole and the judge will certify a grounded answer to the wrong question. Grade retrieval separately, with its own assertion about whether the right document was in the top k, before grading the answer built on top of it. There is more on this in grounding and citations.
A judge rubric you can lift today
This is a working shape for a grounded question answering system. The parts that matter are the forced evidence quote, the independent per criterion verdicts, the explicit refusal criterion, and the escape hatch that sends undecidable cases to a human instead of guessing. Change the criteria to match your task. Do not change the structure.
ROLE
You are a grader. You do not answer the user's question. You decide whether the
candidate answer satisfies each criterion below, using only the inputs given.
INPUTS
<question>{{question}}</question>
<retrieved_context>{{context}}</retrieved_context>
<candidate>{{candidate}}</candidate>
<reference note="may be empty">{{reference}}</reference>
CRITERIA. Each is judged independently and returns pass or fail. Never average them.
C1 GROUNDED Every factual claim in the candidate is supported by retrieved_context.
A claim that is true in general but absent from the context fails C1.
C2 COMPLETE The candidate covers every part of the question the context can answer.
C3 SCOPED The candidate answers what was asked and adds no advice or caveat
that the question did not ask for.
C4 REFUSAL If the context cannot answer, the candidate says so plainly and does
not offer a partial answer presented as a full one.
C5 CONTRACT Output is 120 words or fewer and every doc id it cites appears in
retrieved_context.
RULES
1. Quote the evidence before you decide. If you cannot quote it, the verdict is fail.
2. Judge only the criterion in front of you. Fluency, length, tone and confidence
are not evidence and must not move a verdict.
3. Use <reference> for C2 only. A candidate may be correct and worded differently.
4. "Unclear" is not a verdict. If a criterion cannot be decided from the inputs,
set needs_human true and stop.
OUTPUT, JSON only, with no prose before it and none after it.
{
"c1_grounded": {"verdict": "pass|fail", "evidence": "quote, 25 words max", "why": "20 words max"},
"c2_complete": {"verdict": "pass|fail", "evidence": "...", "why": "..."},
"c3_scoped": {"verdict": "pass|fail", "evidence": "...", "why": "..."},
"c4_refusal": {"verdict": "pass|fail|not_applicable", "evidence": "...", "why": "..."},
"c5_contract": {"verdict": "pass|fail", "evidence": "...", "why": "..."},
"needs_human": false
}
Two choices in there are worth stating plainly. The criteria are written as statements that can be checked rather than qualities that can be admired, which is what stops criteria collapse. And needs_human exists because the alternative is a judge guessing on the hardest cases and burying the guess inside an aggregate. Track that rate as its own metric: if it rises, your inputs have drifted away from the rubric.
Score the criteria, not the answer
Report five numbers, one per criterion, plus the fraction of cases that passed all five. The last one decides shipping and the five underneath tell you what to fix. Averaging them is meaningless, because a system that is always grounded and never complete is a different product from one that is always complete and half invented, and an average makes them identical.
Your suite probably cannot see the regression you care about
This part gets skipped and it decides whether the rest was worth building. A pass rate measured on a finite sample is an estimate, and the estimate has a width. If the width is bigger than the change you want to detect, the suite cannot detect it, and every green run is telling you nothing.
Halving the error costs four times the cases. That is the shape of a square root and there is no way around it, which is why the answer is never one enormous suite. It is a small deterministic tier on every change, a larger tier nightly, and per slice thresholds instead of a single headline, so that a collapse in a slice of forty cases is visible even when a two point move in the aggregate is not.
The margin above is for a single measurement. When you compare two versions you run the same cases through both, so the comparison is paired and you should count only the cases whose verdict changed. Twelve cases that passed before and fail now is a strong signal even in a suite of a hundred, while a two point aggregate move with three changes in each direction is noise. The mechanics are in catching model regressions.
Prices are per million tokens and yours to fill in, so check your provider's current rate rather than trusting a number on a web page. Judge tokens are priced at the input rate because a per criterion judge writes very little.
The judge percentage is usually the expensive dial, and pushing assertions down the ladder is the cheapest way to turn it down. The margin of error output sits in the same box so that nobody proposes a suite size without seeing what that size can resolve.
Scoring without lying to yourself
One number is a management artefact, not an engineering one. The moment a suite reports a single percentage, people optimise the number, the failing slice gets averaged into invisibility, and nobody can say whether a two point move means anything. Report structure instead.
- Report pass rate per criterion, and the fraction of cases passing every criterion. Never average criteria together.
- Report per slice, with a floor per slice rather than one overall floor. A slice below its floor blocks the release even if the aggregate rose.
- Report the flaky rate: cases whose verdict changes across samples of the same version. Counting them as failures hides real ones, counting them as passes hides instability.
- Report judge agreement from the calibration set next to any judge derived score.
- Report case counts per slice, so a slice with six cases is never quoted as a percentage. Six cases move in seventeen point steps, and someone will eventually make a decision on it.
Wiring evals into the way you actually build
A suite that lives in a notebook and runs when someone remembers is a document, not a control. The wiring is what makes it a control, and it is mostly ordinary software engineering.
- Put prompts and rubrics in version control
Prompt text, tool schemas, rubric text and the judge model version live in the repository, and a change to any of them is a pull request. If the prompt lives in a vendor console, nobody can tell you what was running last Tuesday. The pattern is in treating prompts as code.
- Split the suite into tiers
A fast tier of rung one and two assertions on every pull request, finishing in about two minutes. A full judged tier nightly and on release candidates. A locked holdout run only before a release, because a suite you tune against stops being a measurement.
- Make failure output a diff, not a percentage
A failing case shows the input, the previous output, the current output, the failing criterion and the judge's quoted evidence. A report that needs a database query before an engineer knows what to look at will be ignored within a fortnight.
- Feed production failures back as cases
Every incident, override and confirmed complaint becomes a case whose provenance points at the incident. That loop is what keeps a suite relevant, and it is covered in building a golden dataset.
- Set thresholds that block, and mean it
Decide per slice floors before the first release, write them into the pipeline, and agree who can override. A threshold overridden three times is not a threshold, it is a log line, and it should be raised, lowered or deleted.
- Review the suite on a schedule
Cases that have passed for months without ever failing are candidates for retirement. Criteria nobody has read should be checked against the current spec. Suites rot the way documentation rots, quietly and then all at once.
Terms worth being precise about
- Eval
- A repeatable measurement of a system's behaviour on a fixed set of inputs, using assertions that decide pass or fail on each output. An eval differs from a benchmark in that the inputs and the criteria are yours, drawn from your traffic and your definition of correct.
- Assertion
- One decidable statement about a single output, returning pass or fail. Assertions sit on a ladder from structural checks through deterministic content checks and reference comparison up to rubric judgement and human adjudication, and the correct rung is the lowest one that catches the failure.
- Model judge
- A language model used as a grading instrument against a written rubric. It is subject to position, length and self preference bias, and its scores carry no meaning until its agreement with human labels has been measured and reported alongside them.
- Slice
- A tag on eval cases that groups them by a property of the input, such as language, document type, intent or customer tier. Slices exist because an aggregate pass rate hides a collapse in any group small enough to be averaged away.
- Flaky case
- A case whose verdict changes between runs of the same system version, caused by sampling temperature, retrieval ties or upstream variability. Flaky cases are quarantined out of the pass rate and tracked separately, since counting them either way corrupts the score.
- Locked holdout
- A slice of the eval set run only before a release and never used while iterating. It detects overfitting of prompts to the development set, which happens after enough iterations against the same cases that the score stops predicting behaviour on new inputs.
Before you trust the number
With most of that in place you have a control rather than a report. The honest framing for anyone asking what it buys: an eval suite does not make the system better. It makes changes legible, so improvement stops being a matter of opinion and a regression stops being something a customer tells you about first. That work sits inside evaluation and guardrails, and it is the part of an AI build that is dull to sell and impossible to run without.
ChatGPTalker, "Writing Evals for LLM Systems That Catch Real Failures" (2026). An eval is a frozen input set, assertions that decide pass or fail, and a comparable score. Write assertions on the lowest rung that catches the failure, calibrate any model judge against human labels, and report per criterion and per slice, because a 100 case suite at 80 percent carries a margin of error of roughly eight points.
Questions readers ask next
How many test cases does an LLM eval suite need?
Can I use an LLM to grade another LLM's output?
What is the difference between an eval and a unit test?
Why does my eval score change when I did not change anything?
Should evals run on every pull request?
What should I do about eval cases that keep flip-flopping between pass and fail?
How do I evaluate a system where there is no single correct answer?
ChatGPTalker. "Writing Evals for LLM Systems That Catch Real Failures." chatgptalker.com, 2026-08-26. https://chatgptalker.com/guides/writing-evals-for-llm-systems/