Evaluation and reliability

Writing evals for LLM systems

An eval is a frozen set of inputs, an assertion that decides pass or fail, and a number you can compare between versions. Most teams build only the third part.

On this page
  1. The short answer
  2. What an eval is, and the four things people confuse it with
  3. The Assertion Ladder, and why to climb as little of it as possible
  4. Where a model judge quietly lies to you
  5. A judge rubric you can lift today
  6. Your suite probably cannot see the regression you care about
  7. Scoring without lying to yourself
  8. Wiring evals into the way you actually build
  9. Terms worth being precise about
  10. Before you trust the number

The short answer

The short answer

An eval is three things: a frozen set of inputs, assertions that decide pass or fail on each one, and a score you can compare between two versions of your system. Most teams build the score and skip the other two, which is why the number moves when nothing changed. Write the cheapest assertion that would have caught the failure you actually saw in production, run every case several times because the system is not deterministic, and report per slice of traffic rather than as one average. A hundred case suite cannot see a five point regression, and that is arithmetic, not pessimism.

  • 5rungs on the assertion ladder, from a schema check up to human adjudication
  • ±7.8 ptsthe 95 percent margin of error on a 100 case suite scoring 80 percent, before anything changesBinomial standard error, 1.96 * sqrt(p(1-p)/n) at p = 0.8, n = 100
  • 3+runs per case, because one sample from a non-deterministic system is an anecdote
  • 0eval cases whose expected answer came from the model you are testing

The reason evals feel like busywork is that the first one is usually written to prove the system works rather than to find out where it does not. That version passes on day one, passes forever, and tells you nothing. A useful suite is built the other way round: every case exists because something went wrong, or because two people argued about whether an output was acceptable and the argument had to be settled.

What an eval is, and the four things people confuse it with

An eval is a regression test whose assertion is allowed to be fuzzy. That is the whole idea. Everything else in the discipline exists to stop the fuzziness spreading into the parts that should be exact.

What people call an evalWhat it measuresWhat it cannot tell you
A public benchmark scoreA base model on somebody else's task distributionAnything about your prompt, retrieval, tools or users
A notebook of ten tried promptsWhether the author liked the outputs that dayWhether the next change makes it worse, since nothing is stored
Thumbs up and down in the productThe opinion of the small fraction of users who clickSilent failure, because nobody rates an answer they believed
An offline eval suiteBehaviour on inputs you chose, under assertions you wroteBehaviour on inputs you never imagined, which is most of production
An online metric such as edit rateWhat actually happened to real workWhich of your nine changes caused the move
Offline suites and online metrics answer different questions. Neither substitutes for the other.

The consequence is a division of labour. The offline suite is fast, cheap, repeatable and blind to the world. The online metric is slow, hard to attribute and the only thing that tells you whether the system is useful. Use the first to decide whether a change is safe to ship and the second to decide whether shipping it was worth doing. With only the second, every incident becomes an argument about which change caused it.

  • A case is one input plus everything needed to reproduce the run: the corpus version, the tool responses, the user context.
  • An assertion is one decidable statement about that output. A case usually carries several.
  • A slice is a tag on the case: language, document type, customer tier, intent. Slices are how you find the regression the average hides.
  • A suite is the cases plus the schedule and the thresholds that decide what happens when the score moves.

The Assertion Ladder, and why to climb as little of it as possible

Every assertion sits on one of five rungs. They get more expensive, slower and less trustworthy as you climb. The discipline is to write each assertion on the lowest rung that catches the failure you are targeting, because most of what teams grade with a model judge could have been a string comparison.

Framework

The Assertion Ladder

Five rungs, cheapest first. Start every new assertion at rung one and stop climbing the moment the failure is caught.

01
Rung 1: structural

Does the output parse. Are required fields present. Are enum values inside the allowed set. Is the response under the length contract. These run in milliseconds, never disagree with themselves, and catch a real share of production breakage, because a drifting system usually breaks its output shape before it starts being wrong.

02
Rung 2: deterministic content

Claims a computer can decide alone. Every cited document id appears in the retrieved set. Line items sum to the stated total. The extracted date falls inside the document's period. No phone number appears in an output that should not contain one. This is the most under-invested rung and the highest value one on the ladder.

03
Rung 3: reference comparison

Compare against a known correct answer. Exact match for classification and extraction, set overlap for multi-label tasks, and for free text an embedding similarity floor used as a smoke alarm rather than a grade. Honest when there is one right answer, misleading the moment there are several, which is why it does not survive summarisation or advice.

04
Rung 4: rubric judgement

A model grades against written criteria, one verdict per criterion, each with a quoted piece of evidence. People reach for this first because it is easy to write and feels like it covers everything. It is the slowest rung, the most expensive, the only one that can be wrong in a correlated way across the whole suite, and it means nothing until it is calibrated.

05
Rung 5: human adjudication

A person decides, on a sample. Reserve it for criteria that are genuinely a matter of judgement, for building the calibration set rung four is measured against, and for the audit that tells you whether rung four has drifted. It is not a fallback for lazy assertions, it is the ground truth that makes the rung below interpretable.

The rung one test

Take your last three production incidents involving model output and ask which rung would have caught each. In most systems the honest answer for at least two of them is rung one or rung two, and the assertion is about four lines of code. Write those before you build any judge infrastructure.

Where a model judge quietly lies to you

A model judge is a measuring instrument that has opinions. Calibrate it or its numbers will be precise and wrong in a consistent direction, which is worse than having no number. These are the biases that show up in practice.

Failure of the judgeHow it shows upMitigation that works
Position biasIn a pairwise comparison, whichever answer comes first wins too oftenRun every pair twice with the order swapped, and count it only if both runs agree
Length and fluency biasLonger, better formatted answers score higher on criteria unrelated to lengthOne criterion at a time, binary verdict, quoted evidence required for every verdict
Self preferenceThe judge favours output from its own model familyDo not use the same model version as generator and judge. If you must, check the gap on a human labelled slice
Criteria collapseYou asked about grounding and the judge scored overall quality, because that is what the word good pulls inRemove evaluative adjectives. Write each criterion as a testable statement, not a quality bar
Scale mushA one to ten score where the judge only uses six, seven and eightBinary per criterion, then count criteria passed. If you need a scale, give three anchored levels with an example of each
Judge driftThe score moves and nobody touched the system, because the judge model changed underneath youPin the judge model version, treat it as a production dependency, re-run calibration when it changes
Calibrate the judge before you believe it

Have humans label a set of outputs against the same rubric the judge uses, covering boundary cases rather than obvious ones, then measure how often judge and human agree. Publish that agreement number next to every judge derived score, permanently. If agreement is poor the fix is the rubric, not a bigger model: ambiguous criteria produce disagreement in humans first and in judges second.

One more trap. A judge that grades your output while reading the same retrieved context your system read inherits your retrieval failures. If the context is missing the answer, both are working from the same hole and the judge will certify a grounded answer to the wrong question. Grade retrieval separately, with its own assertion about whether the right document was in the top k, before grading the answer built on top of it. There is more on this in grounding and citations.

A judge rubric you can lift today

This is a working shape for a grounded question answering system. The parts that matter are the forced evidence quote, the independent per criterion verdicts, the explicit refusal criterion, and the escape hatch that sends undecidable cases to a human instead of guessing. Change the criteria to match your task. Do not change the structure.

Grader system prompt, per criterion verdicts with forced evidencetext
ROLE
You are a grader. You do not answer the user's question. You decide whether the
candidate answer satisfies each criterion below, using only the inputs given.

INPUTS
<question>{{question}}</question>
<retrieved_context>{{context}}</retrieved_context>
<candidate>{{candidate}}</candidate>
<reference note="may be empty">{{reference}}</reference>

CRITERIA. Each is judged independently and returns pass or fail. Never average them.
C1 GROUNDED  Every factual claim in the candidate is supported by retrieved_context.
             A claim that is true in general but absent from the context fails C1.
C2 COMPLETE  The candidate covers every part of the question the context can answer.
C3 SCOPED    The candidate answers what was asked and adds no advice or caveat
             that the question did not ask for.
C4 REFUSAL   If the context cannot answer, the candidate says so plainly and does
             not offer a partial answer presented as a full one.
C5 CONTRACT  Output is 120 words or fewer and every doc id it cites appears in
             retrieved_context.

RULES
1. Quote the evidence before you decide. If you cannot quote it, the verdict is fail.
2. Judge only the criterion in front of you. Fluency, length, tone and confidence
   are not evidence and must not move a verdict.
3. Use <reference> for C2 only. A candidate may be correct and worded differently.
4. "Unclear" is not a verdict. If a criterion cannot be decided from the inputs,
   set needs_human true and stop.

OUTPUT, JSON only, with no prose before it and none after it.
{
  "c1_grounded": {"verdict": "pass|fail", "evidence": "quote, 25 words max", "why": "20 words max"},
  "c2_complete": {"verdict": "pass|fail", "evidence": "...", "why": "..."},
  "c3_scoped":   {"verdict": "pass|fail", "evidence": "...", "why": "..."},
  "c4_refusal":  {"verdict": "pass|fail|not_applicable", "evidence": "...", "why": "..."},
  "c5_contract": {"verdict": "pass|fail", "evidence": "...", "why": "..."},
  "needs_human": false
}

Two choices in there are worth stating plainly. The criteria are written as statements that can be checked rather than qualities that can be admired, which is what stops criteria collapse. And needs_human exists because the alternative is a judge guessing on the hardest cases and burying the guess inside an aggregate. Track that rate as its own metric: if it rises, your inputs have drifted away from the rubric.

Score the criteria, not the answer

Report five numbers, one per criterion, plus the fraction of cases that passed all five. The last one decides shipping and the five underneath tell you what to fix. Averaging them is meaningless, because a system that is always grounded and never complete is a different product from one that is always complete and half invented, and an average makes them identical.

Your suite probably cannot see the regression you care about

This part gets skipped and it decides whether the rest was worth building. A pass rate measured on a finite sample is an estimate, and the estimate has a width. If the width is bigger than the change you want to detect, the suite cannot detect it, and every green run is telling you nothing.

95 percent margin of error on a pass rate of 80 percent, by suite size
Arithmetic on the binomial standard error, 1.96 * sqrt(p(1-p)/n), at p = 0.8. Not measured data. Substitute your own pass rate.
30 cases±14.3 pts
50 cases±11.1 pts
100 cases±7.8 pts
the common suite size, and it cannot resolve a five point regression
200 cases±5.5 pts
500 cases±3.5 pts
1,000 cases±2.5 pts

Halving the error costs four times the cases. That is the shape of a square root and there is no way around it, which is why the answer is never one enormous suite. It is a small deterministic tier on every change, a larger tier nightly, and per slice thresholds instead of a single headline, so that a collapse in a slice of forty cases is visible even when a two point move in the aggregate is not.

Paired comparison beats a bigger suite

The margin above is for a single measurement. When you compare two versions you run the same cases through both, so the comparison is paired and you should count only the cases whose verdict changed. Twelve cases that passed before and fail now is a strong signal even in a suite of a hundred, while a two point aggregate move with three changes in each direction is noise. The mechanics are in catching model regressions.

Suite cost per run, per month, and what it can actually resolve

Prices are per million tokens and yours to fill in, so check your provider's current rate rather than trusting a number on a web page. Judge tokens are priced at the input rate because a per criterion judge writes very little.

0Model calls per full suite run
0Cost of one full suite run
0Cost per month at that cadence
0Margin of error on the headline rate, points

The judge percentage is usually the expensive dial, and pushing assertions down the ladder is the cheapest way to turn it down. The margin of error output sits in the same box so that nobody proposes a suite size without seeing what that size can resolve.

Scoring without lying to yourself

One number is a management artefact, not an engineering one. The moment a suite reports a single percentage, people optimise the number, the failing slice gets averaged into invisibility, and nobody can say whether a two point move means anything. Report structure instead.

One headline scorePer assertion, per slice
What a drop tells youThat something is worseWhich criterion failed, on which kind of input
A small failing sliceAveraged away below the noise floorVisible, because the slice has its own threshold
Effect on the teamThe number becomes the target and cases get quietly retiredThe failing cell becomes the ticket
Cost to produceThe sameThe same, it is a group by
Use in a release noteA percentage nobody can act onThe criteria and slices that moved, with case ids
  1. Report pass rate per criterion, and the fraction of cases passing every criterion. Never average criteria together.
  2. Report per slice, with a floor per slice rather than one overall floor. A slice below its floor blocks the release even if the aggregate rose.
  3. Report the flaky rate: cases whose verdict changes across samples of the same version. Counting them as failures hides real ones, counting them as passes hides instability.
  4. Report judge agreement from the calibration set next to any judge derived score.
  5. Report case counts per slice, so a slice with six cases is never quoted as a percentage. Six cases move in seventeen point steps, and someone will eventually make a decision on it.

Wiring evals into the way you actually build

A suite that lives in a notebook and runs when someone remembers is a document, not a control. The wiring is what makes it a control, and it is mostly ordinary software engineering.

  1. Put prompts and rubrics in version controlday one

    Prompt text, tool schemas, rubric text and the judge model version live in the repository, and a change to any of them is a pull request. If the prompt lives in a vendor console, nobody can tell you what was running last Tuesday. The pattern is in treating prompts as code.

  2. Split the suite into tiersweek one

    A fast tier of rung one and two assertions on every pull request, finishing in about two minutes. A full judged tier nightly and on release candidates. A locked holdout run only before a release, because a suite you tune against stops being a measurement.

  3. Make failure output a diff, not a percentageweek one

    A failing case shows the input, the previous output, the current output, the failing criterion and the judge's quoted evidence. A report that needs a database query before an engineer knows what to look at will be ignored within a fortnight.

  4. Feed production failures back as casescontinuous

    Every incident, override and confirmed complaint becomes a case whose provenance points at the incident. That loop is what keeps a suite relevant, and it is covered in building a golden dataset.

  5. Set thresholds that block, and mean itbefore launch

    Decide per slice floors before the first release, write them into the pipeline, and agree who can override. A threshold overridden three times is not a threshold, it is a log line, and it should be raised, lowered or deleted.

  6. Review the suite on a schedulemonthly

    Cases that have passed for months without ever failing are candidates for retirement. Criteria nobody has read should be checked against the current spec. Suites rot the way documentation rots, quietly and then all at once.

Terms worth being precise about

Definitions
Eval
A repeatable measurement of a system's behaviour on a fixed set of inputs, using assertions that decide pass or fail on each output. An eval differs from a benchmark in that the inputs and the criteria are yours, drawn from your traffic and your definition of correct.
Assertion
One decidable statement about a single output, returning pass or fail. Assertions sit on a ladder from structural checks through deterministic content checks and reference comparison up to rubric judgement and human adjudication, and the correct rung is the lowest one that catches the failure.
Model judge
A language model used as a grading instrument against a written rubric. It is subject to position, length and self preference bias, and its scores carry no meaning until its agreement with human labels has been measured and reported alongside them.
Slice
A tag on eval cases that groups them by a property of the input, such as language, document type, intent or customer tier. Slices exist because an aggregate pass rate hides a collapse in any group small enough to be averaged away.
Flaky case
A case whose verdict changes between runs of the same system version, caused by sampling temperature, retrieval ties or upstream variability. Flaky cases are quarantined out of the pass rate and tracked separately, since counting them either way corrupts the score.
Locked holdout
A slice of the eval set run only before a release and never used while iterating. It detects overfitting of prompts to the development set, which happens after enough iterations against the same cases that the score stops predicting behaviour on new inputs.

Before you trust the number

Eval suite readiness
0 of 11 done

With most of that in place you have a control rather than a report. The honest framing for anyone asking what it buys: an eval suite does not make the system better. It makes changes legible, so improvement stops being a matter of opinion and a regression stops being something a customer tells you about first. That work sits inside evaluation and guardrails, and it is the part of an AI build that is dull to sell and impossible to run without.

Cite this

ChatGPTalker, "Writing Evals for LLM Systems That Catch Real Failures" (2026). An eval is a frozen input set, assertions that decide pass or fail, and a comparable score. Write assertions on the lowest rung that catches the failure, calibrate any model judge against human labels, and report per criterion and per slice, because a 100 case suite at 80 percent carries a margin of error of roughly eight points.

Questions readers ask next

How many test cases does an LLM eval suite need?
Enough that the margin of error is smaller than the change you want to detect. At an 80 percent pass rate the 95 percent margin is roughly eight points on 100 cases and five and a half on 200, and halving it again needs four times the cases. Size per slice rather than in total: thirty to fifty cases in each slice you care about, each with its own threshold, resolves far more than one large undifferentiated pool.
Can I use an LLM to grade another LLM's output?
Yes, but only as a calibrated instrument. Judge one criterion at a time with a binary verdict, force the judge to quote its evidence, pin the judge model version, avoid using the same model family as generator and judge, and measure agreement against a human labelled set before quoting any judge derived number. Publish that agreement figure next to the score permanently, and re-run calibration whenever the judge model changes.
What is the difference between an eval and a unit test?
A unit test asserts an exact output and fails on any difference. An eval asserts a property of an output that has many valid forms, so the assertion is fuzzy and the result is a rate rather than a binary. Everything else is the same: stored, versioned, run automatically, used to block a release. The trap is letting the fuzziness leak into the inputs, which must stay as frozen as a unit test's.
Why does my eval score change when I did not change anything?
Three usual causes. Sampling temperature above zero makes each run a draw, so you need several samples per case. The provider may have changed the model behind a floating version alias. Or your retrieval corpus changed underneath the suite, so the same question now retrieves different context. Pin the version, snapshot the corpus with the case, run each case several times, and what remains is genuine variance you can measure.
Should evals run on every pull request?
The fast tier should, finishing in about two minutes using structural and deterministic assertions only. Judge based grading is too slow and too expensive for every commit, so it runs nightly and on release candidates, and a locked holdout runs only before a release. Splitting the suite this way keeps the feedback loop tight without paying for a full judged run on a typo fix.
What should I do about eval cases that keep flip-flopping between pass and fail?
Quarantine them, do not delete them. A flaky case is revealing either genuine non-determinism in your system, an ambiguous assertion, or a tie in retrieval ranking. Track the flaky rate as its own metric with a target of falling over time. Counting flaky cases as failures buries real regressions in noise, and counting them as passes hides instability your users are already meeting.
How do I evaluate a system where there is no single correct answer?
Stop comparing against a reference and assert properties instead. For a summary, assert that every claim appears in the source, that the required sections are covered, that length is inside the contract and that no recommendation was added. Each of those is decidable. Where genuine preference remains, use a pairwise comparison between current and candidate with the order swapped, which is easier to judge reliably than an absolute score.
Cite this

ChatGPTalker. "Writing Evals for LLM Systems That Catch Real Failures." chatgptalker.com, 2026-08-26. https://chatgptalker.com/guides/writing-evals-for-llm-systems/

Rather have it built than read about it?

Send the process you want automated. You get a scoped plan back, with the build shape, the stack and a realistic timeline.

Start a project