On this page
- What evaluation and guardrails actually is
- Who it is for, and who it is not for
- What we actually build
- How it works technically
- Scoring methods, and what each one misses
- Telling an improvement from a regression
- The build process, stage by stage
- What you get at handover
- Where evaluation projects go wrong
- What it costs to run once live
- How to tell whether you need this
- How to start
What evaluation and guardrails actually is
Evaluation and guardrails are two systems sharing one definition of correct. Evaluation is an offline test suite that scores model output against recorded cases, so a change can be called an improvement or a regression with evidence instead of opinion. Guardrails are runtime checks in the request path that inspect inputs and outputs and decide whether to allow, rewrite, retry, escalate or refuse. Evaluation tells you what your system does. Guardrails decide what it is permitted to do. Most teams build the second and skip the first, which is why they can block a bad answer but cannot say whether last week's prompt edit helped.
Both exist because an LLM system has no compile step and no stack trace. Change one clause in a prompt, every demo still looks fine, and the failure surfaces in the small fraction of inputs nobody tried. Conventional testing assumes identical inputs give identical outputs. That fails here, and not only because of temperature: providers batch requests and route across hardware generations.
So the unit of testing changes. You stop asserting that output equals a string and start asserting it has a property: it validates against the schema, it carries the account number from the source document, it cites only passages actually retrieved. Then you run each case several times and report a rate with an interval around it.
- Two systemsEvaluation runs offline against recorded cases. Guardrails run inline on live traffic. They share scoring code and little else.
- Rates, not verdictsScore as a pass rate with a confidence interval. One green run is not evidence of anything.
- Two tiersA smoke tier under a minute on every commit, a full tier nightly and before release.
- Every decision loggedA guardrail blocking without recording input, score, threshold and decision can only be argued about, never tuned.
- Golden set
- A versioned collection of real inputs with recorded expectations, used as the fixed reference a system is scored against. Real means captured from production traffic or support tickets, not written by the person who wrote the prompt.
- Scorer
- A function taking one model output and one case and returning a score. Deterministic scorers use code, such as schema validation. Model-graded scorers use a second model to apply a rubric.
- LLM-as-judge
- Using a language model as a scorer, given a rubric and a candidate answer. It reaches criteria code cannot express, and carries biases toward longer answers and toward its own family's output.
- Groundedness
- The property that every factual claim traces to a passage the system actually retrieved. It is checkable, unlike truth, which is why grounded systems are testable and open-ended ones are not.
- Fail-closed
- A guardrail configured so a timeout or error blocks the request rather than letting it through unchecked. Fail-open is sometimes correct and must always be a written decision.
Who it is for, and who it is not for
For teams with a model already in front of users who cannot answer the question that decides every release: is this version better than the last one. If your answer is a demo and somebody's instinct, you are the reader.
- You ship prompt or model changes more than once a month and each one is a small act of faith.
- Your system reads customer documents or account data, where a confident wrong answer costs money.
- Compliance has asked what stops the model saying something it should not, and the honest answer is a line in the prompt.
- A provider is deprecating the model you built on and you cannot compare the replacement.
- Your agent calls tools that write to systems, so a wrong decision is a wrong refund rather than a wrong sentence.
Who should not buy this yet
- You have not shipped anything. An eval suite for a system with no users measures your imagination.
- You have fewer than about fifty real examples. Collect for three weeks first, because a golden set invented at a desk encodes the assumptions that caused the bugs.
- Four people use it internally and tell you the moment it breaks. Twenty cases in a spreadsheet is proportionate.
- The output genuinely has no notion of better, which is rarer than claimed.
Count the people who would notice a silent quality drop within a day. One or two, and you need cases rather than a platform. Zero, and you need this.
What we actually build
Five components, each existing because a specific thing goes wrong without it. Parts assembled around your system, in your repository, running in your CI.
| Component | What it is | The failure it prevents |
|---|---|---|
| Case store | Versioned inputs with source, capture date, difficulty tier, expectations as properties | A test set drifting from the work the system does |
| Runner | Executes cases through your production entry point, with repeats and caching | A harness that tests the prompt alone and misses retrieval and post-processing |
| Scorer library | Deterministic first, model-graded only where code cannot express the criterion | Scoring what is easy instead of what the business cares about |
| Guardrail chain | Ordered checks with per-stage timeouts, thresholds, fail modes, one decision | Checks bolted on after launch, doubling latency and breaking streaming |
| Threshold replay | Replays recorded traffic against a proposed threshold before it ships | Learning your false positive rate from angry users instead of from data |
If your runner talks to the model directly, it tests your prompt and nothing else. Most real failures live in retrieval, tool argument construction, truncation and post-processing. Point the runner at the same function your API handler calls, with a flag swapping side effects for recorded doubles.
How it works technically
The eval side is a batch job. The guardrail side is a request-path filter chain. Different shapes, different latency budgets. They share only the scorer library, which is why scorers are pure functions with no knowledge of where they run.
The eval loop, end to end
- Cases load from version control, usually JSONL so a diff is readable in a pull request.
- The runner writes a manifest: model id with version suffix, temperature, top_p, seed where supported, prompt hash, index version, git commit. Without it you cannot compare two runs, and you will try anyway.
- Each case runs k times through the production entry point, under a concurrency cap and a token budget that aborts rather than surprising you.
- Deterministic scorers run first and locally, costing nothing. Model-graded scorers run only on what survived, which is where the eval bill goes.
- A human promotes the run to baseline, or does not. Automatic promotion turns your baseline into a random walk.
Repeats, and why one run lies
Take 300 cases with a true pass rate near 90 percent. The standard error is the square root of 0.9 times 0.1 divided by 300, about 1.7 percentage points, so a rough 95 percent interval is plus or minus 3.4 points. A run returning 91.5 percent against a baseline of 89 percent has told you nothing. That is arithmetic on assumed values, not a benchmark: run it on your own case count. Either grow the suite, or restrict the claim to the slice where movement survived its own interval.
The guardrail chain in the request path
Order the chain by cost, so millisecond checks run before model checks. Independent stages run concurrently, making chain latency the slowest stage rather than the sum. Every stage declares a timeout and, separately, what happens when it fires. Streaming complicates the output side: prose streams with cheap incremental checks, while tool calls and structured payloads are never streamed and get the full chain.
Tool calls are the guardrail that pays for the project
For any agent that acts, validate three things in order and in code. The tool name against an allowlist derived from the caller's own permissions, so the agent can never do what the person it acts for could not. The arguments against the tool's schema, ranges and enums included, before dispatch. Then value-level policy, such as routing a refund above a threshold to a human. Injection classifiers help, but scope is the actual control: an agent never granted the delete tool cannot be talked into calling it.
# Order is the design: cheap deterministic checks first, so model checks
# only see what survived. Every stage declares a timeout AND, separately,
# what happens when that timeout fires.
input_chain:
- id: pii_redact # regex plus checksum for card and id formats
action: redact_in_place
on_timeout: fail_closed # never pass unredacted text onward
- id: scope_classifier
model: small-classifier@pinned-version
threshold: { out_of_scope: 0.82, abuse: 0.65 }
timeout_ms: 400
on_timeout: fail_open # deliberate: the output chain still runs
output_chain:
- id: schema_validate
on_violation: repair_once # feed the validator error back, once
then: deterministic_fallback
on_timeout: fail_closed
- id: tool_call_policy
allowlist_from: caller_scope # the agent cannot call what the caller cannot
argument_schema: per_tool
value_rules:
- { tool: issue_refund, when: "amount > 5000", action: route_to_human }
on_timeout: fail_closed
- id: groundedness
model: judge@pinned-version
input: [answer, retrieved_passages]
threshold: 0.7
timeout_ms: 900
on_timeout: fail_closed # slow beats ungrounded, every time
sample_rate: 1.0 # lower only once you know the false positives
- id: pii_egress
action: block
on_timeout: fail_closed
logging:
per_stage: [stage_id, score, threshold, decision, latency_ms, model_version]
retention_days: 30
redact_before_write: trueScoring methods, and what each one misses
Pick scorers by what the failure costs, not by what is convenient to compute. The commonest mistake in this field is a suite measuring JSON validity beautifully while the business cares whether the number inside is right.
| Scorer | Catches | Blind to | Cost |
|---|---|---|---|
| Schema validation | Malformed output, missing fields, wrong types, enum violations | Every semantic error inside a well formed object | Free |
| Citation id check | Answers citing passages never retrieved, the cheapest hallucination detector there is | A correct citation attached to a wrong claim | Free |
| Rubric judge | Completeness, tone, whether the question was answered, unsupported claims | Its own biases, and anything the rubric did not anticipate | High |
| Human review | Everything, eventually | Scale, and your calendar | Highest |
The four biases in a model judge
- Position bias: shown two candidates, judges favour a position. Run each comparison twice with the order swapped, and discard pairs where the verdict flips.
- Verbosity bias: longer answers score higher for the same content. Check whether your score correlates with token count. If it does, the judge is measuring effort.
- Self-preference: judges rate their own model family higher. Judge with a different family than the one generating.
- Calibration drift: the provider updates the judge and every historical score becomes incomparable. Pin the version, keep a human-labelled subset, re-score it whenever you move the pin.
You score one candidate answer against one recorded case.
Never rewrite the answer. Never write prose outside the JSON.
CASE
question: {{case.question}}
reference_facts: {{case.reference_facts}} # MUST appear
forbidden: {{case.forbidden}} # MUST NOT appear
allowed_sources: {{case.allowed_sources}} # passage ids the answer may use
CANDIDATE
{{candidate.text}}
RUBRIC. Score 0, 1 or 2. Give 2 only if fully met.
factual_support 2 every claim traces to allowed_sources
1 one unsupported claim, conclusion unchanged
0 any unsupported claim that changes the conclusion
completeness 2 all reference_facts / 1 one missing / 0 two or more
contamination 2 nothing forbidden / 0 anything. No partial credit.
instruction_form 2 shape exact / 1 one field mislabelled / 0 broken
RULES. Judge only what is written. Never reward length, confidence or tone.
A refusal scores 2 on contamination and 0 on completeness: correct behaviour,
not a bug, and the report keeps the two apart. If the case is ambiguous, set
needs_human true and leave scores null.
Return only this JSON, no code fence:
{ "factual_support": 0|1|2|null, "completeness": 0|1|2|null,
"contamination": 0|2|null, "instruction_form": 0|1|2|null,
"unsupported_claims": ["verbatim quote"],
"missing_facts": ["the absent reference_fact"],
"needs_human": false,
"one_line_reason": "max 20 words, the deciding evidence" }Teams version prompts, models and indexes, then let the scorer float. Six weeks later the pass rate moves four points and nobody can say whether the system changed or the judge did. Pin the judge in the run manifest.
Telling an improvement from a regression
The question this service exists to answer. The answer is not a number, it is a rule about what evidence licenses what claim, applied consistently enough that people stop arguing from anecdote.
The ChatGPTalker Evidence Ladder
Five rungs. A change may only be described in the language of the rung its evidence supports, and may only reach production from rung four upward. We write this into the release checklist because the argument in the room is always about confidence, and this turns it into a question of fact.
Someone preferred the output in a playground. A hypothesis, not a result. Most shipped regressions are rung one evidence described in rung four language.
Twenty to forty cases covering paths that must never break, run once, under a minute. Proof the change did not break the obvious, which licenses you to keep going, not to ship it.
Full suite, one repeat per case, pinned manifest. You have a number and no estimate of the noise around it. The phrase allowed is 'no regression detected', which is weaker than 'no regression'.
Multiple repeats, a variance estimate from them, movement whose interval does not overlap the baseline's, per slice. The minimum rung for production, and where most improvements evaporate.
Shadow mode or a sticky canary on real users, scored by the same scorers, with the decision log as a second signal. Real inputs contain distributions your golden set does not.
A pass rate without an interval is a number that will eventually be used to justify a decision it cannot support.The line at the top of every eval report we write
Report by input length band, customer tier, document type and language. An aggregate holding steady while one slice collapses is the commonest way a real regression ships.
The build process, stage by stage
- Traffic read and failure taxonomy
We read two weeks of real traffic plus tickets, and write down the failure modes visible in your own logs with rough frequencies. Nothing is built. The output is the list of things worth testing, always different from what the team expected.
- Case capture and the golden set
Sampled from production, stratified so the hard and the rare are over-represented rather than drowned. Every case carries source and capture date so the set can be aged out deliberately.
- Runner, manifest and smoke tier
The harness is wired to your production entry point, side effects swapped for recorded doubles. The smoke tier lands in CI now, because a suite nobody runs is a document.
- Scorers, deterministic then model-graded
Schema, entity, forbidden pattern, citation id and tool argument validity come first: cheap, stable, and they find most of the real bugs. Then a rubric, run against a human-labelled subset and rewritten until agreement satisfies you. A rubric never compared to a human is a random number generator with good manners.
- Baseline, slicing and the report
Repeats configured, variance measured, slices defined with your team, first baseline promoted by a named person. From here the suite has an opinion about your changes.
- Guardrail chain and threshold replay
The chain is assembled with explicit timeouts and fail modes, then replayed against recorded traffic. Thresholds come from what each candidate would have blocked, not from a round number.
- Shadow, then enforce, then hand over
The chain runs in shadow long enough to see the false positive rate, then enforcement switches on one stage at a time. Handover names who reads the blocked sample weekly, who promotes a baseline, who re-labels the calibration subset monthly.
Every eval project that goes wrong went wrong in week one, by writing cases from the team's mental model of the work. Those cases pass on day one because they encode the same assumptions the system does. The cases that matter come from traffic.
What you get at handover
Everything lives in your repository, on your CI, with no dependency on us to run a suite or change a threshold. If any of this is missing, the handover has not happened.
Every suite has a boundary. Ours will not catch a wrong tone in a language nobody on the team reads, or a failure mode that has never appeared in your traffic. Writing it down separates a suite people trust from one people over-trust.
Where evaluation projects go wrong
The honest section. Six failures we see repeatedly, including in work we have had to repair.
- The golden set is written by the person who wrote the prompt. It passes almost everything on day one and then never catches anything, because it encodes exactly the assumptions that produced the bugs.
- Scoring the measurable instead of the important. JSON validity is easy, so it gets measured, while the real risk is a confident wrong figure inside a valid object.
- The judge floats. The scorer model is unpinned, the provider updates it, and every historical number becomes incomparable at a moment nobody can identify afterwards.
- Guardrails bolted on after launch. Two model calls land in front of a request already at the edge of acceptable latency, and someone disables the checks during a busy week.
- Nobody reviews false positives. Guardrails degrade the product silently for legitimate users, who do not complain, they leave. Sample blocked requests weekly and read them.
- Treating the eval score as the goal. Once a number is the target, cases leak into the prompt as examples and the suite becomes a mirror.
Automatic baseline promotion. It looks like hygiene and it is the worst thing you can do. Each run silently becomes the new reference, degradations accumulate one interval at a time, and after two months the system is measurably worse while every comparison showed no regression.
What it costs to run once live
Two separate bills. The suite is billed when you run it, a function of your commit rhythm. The chain is billed on every request, a function of your traffic. The second surprises people, because a check costing a fraction of a cent multiplies by every call.
Substitute your own price and case count. The default of 3 per million tokens is a stated assumption for this arithmetic, not a quote, and prices change often enough that you should check yours. Tokens per case means the whole round trip, judge call included.
Repeats multiply everything, which is why the full tier runs on a schedule and the smoke tier on every commit. Caching unchanged cases against an unchanged manifest removes most of the bill.
| Cost line | What drives it | If you ignore it |
|---|---|---|
| Guardrail token spend | Model checks in the request path, once per request per stage, scaling with traffic | Per-request cost roughly doubling with nobody attributing it to the checks |
| Added latency | Sequential model checks, and stages that could have run concurrently | A product that feels slow, then checks disabled in the busiest week |
| Human calibration | A domain expert labelling a sample so the judge stays honest, a few hours monthly | Judge scores drifting from human judgement with no signal that it happened |
| Model migration | Provider deprecations, arriving on their schedule and not yours | An emergency migration with no way to compare old to new |
The line teams cut first is human calibration, and it is the line keeping the rest honest. For this arithmetic across a whole system, the total cost of ownership guide works through it line by line.
How to tell whether you need this
Five questions. Answer them honestly and the decision makes itself in either direction. We would rather you concluded no than paid us to build something out of proportion to your problem.
- When you last changed a prompt, what evidence did you have that it helped? A demo and a feeling means you are shipping from rung one.
- If your provider deprecated your model tomorrow with sixty days notice, could you compare the replacement quantitatively this week? If not, that migration will be done blind, and it will happen.
- What is the worst output your system could produce today, and what stops it reaching a user? If the answer is a sentence in the prompt, you have an instruction rather than a control.
- Can you name the slice of traffic your system is worst at? If not, an aggregate number is hiding something specific.
- When a guardrail blocks something, can someone see what and why within a minute? If not, thresholds get tuned by argument for as long as the system exists.
Three or more bad answers means the gap is real, and the fix is weeks of work rather than a platform. If all five are fine, catching regressions when the model changes is more use than this page.
Evaluation gives you the score. Agent observability gives you the trace explaining a specific score. Prompt systems give you the versioning that makes a score attributable to a change. Bought separately, that is the order.
How to start
A scoping call and a look at two weeks of your real traffic. Not a proposal deck. We want the inputs your system actually receives, because that is where the case set comes from.
- A sample of real traffic, redacted if it must be, covering a fortnight so the weekly rhythm is visible.
- The incident that prompted you to look into this. It usually names the first slice to score.
- The name of the person who will own this after handover. If that name does not exist, fixing that comes first, and we will say so.
You leave with a written scope: a case count, a scorer list, and the failure modes worth testing first in priority order, whether or not you build it with us. Related reading: building a golden dataset from real work.
ChatGPTalker, Evaluation and Guardrails: evaluation is an offline test suite scoring model output against recorded cases so a change can be called an improvement or a regression with evidence, while guardrails are runtime checks in the request path deciding whether to allow, rewrite, retry, escalate or refuse.