Catching model regressions

The short answer

The uncomfortable property of these systems is that the code is the smallest part of what determines the output. A deployment that changed nothing in your repository can behave differently on Tuesday than it did on Monday, and unless you have pinned the other inputs you have no way to tell whether that is true or whether somebody is imagining it.

The Four Clocks: why the system changes when you did not

Four things tick independently underneath an LLM system. Each fails in a different way and each needs its own detector, which is why a single eval run on a schedule catches only some of them.

Pin it, or there is nothing to compare

A comparison is only meaningful if exactly one thing differs between the two runs. That takes more pinning than most teams expect, because several of the inputs do not look like inputs.

One warning about seeds. A fixed seed is worth setting where a provider honours it, and it is not a substitute for pinning anything else, because seed behaviour is not guaranteed across model versions or infrastructure changes. Record the seed you used, run several samples per case anyway, and treat determinism as a convenience rather than a property you can rely on.

What to diff when the output is prose

A naive text diff between two runs is almost entirely noise, because the model rewords freely and none of that rewording matters. Split the output into parts that must match exactly and parts where only material change counts.

• Exact diff for structured fields, extracted values, classification labels, tool calls made, document ids cited and refusal or non-refusal. Any difference here is a difference, full stop.
• Set diff for lists: which tools were called, which documents were retrieved, which entities were extracted. Order usually does not matter and membership always does.
• Material diff for free text, judged by a model asked one question only: does the new answer differ from the old one in a way that changes what a reader would do. Not whether it is better, and not whether it is different.
• Numeric tolerance for anything computed: assert a band rather than equality, and set the band from what the downstream consumer can absorb.

Why material diff has to be a separate judge

Asking your quality judge whether the answer got worse pulls its whole rubric into the question, and you get a preference rather than a change detection. The material diff judge should see both answers with no labels indicating which is old and which is new, in a randomised order, and return one of three verdicts: same meaning, materially different, or cannot tell. The cannot tell rate is a useful health signal on its own. Then send only the materially different cases to the full grading run, which is also where most of the cost saving in a large suite comes from.

A detail that catches people: run the material diff on the same case sampled several times from the same version first, to establish the baseline rate at which your own system disagrees with itself. If a third of same-version pairs come back materially different, your temperature is too high for the task and no regression test you build on top will be readable.

The aggregate pass rate will hide the regression

This is not a subtle statistical point, it is ordinary weighted averaging, and it happens on real systems constantly. A change improves the common case and damages the rare one. The common case dominates the average, so the headline number goes up while the system gets worse where it matters.

The fix is per slice thresholds rather than one overall floor, and a release gate that blocks when any slice breaches its own floor regardless of what the aggregate did. That only works if every slice has enough cases to be measured, which is the argument for quota sampling made in building a golden dataset. A slice with four cases moves in twenty-five point steps and cannot hold a threshold.

Is the drop real? Count the cases that flipped

Because you run the same cases through both versions, the comparison is paired, and the only cases carrying information are the ones whose verdict changed. Everything that passed in both runs, and everything that failed in both, tells you nothing about the difference between the versions. This changes conclusions more often than people expect.

Call b the count of cases that passed before and fail now, and c the count that failed before and pass now. Two scenarios, both a 200 case suite, both showing the same three point drop in headline pass rate.

Scenario B is also a warning in its own right. Thirty-six cases changing verdict on a change you believed was small means the system is less stable than you thought, even though the net is nothing. Treat a high discordant total as a finding, not as a clean bill of health.

1. Freeze the comparison setbefore anything runs

   Same cases, same corpus version, same tool fixtures, same judge version. If the candidate run uses a newer corpus, you are measuring two changes at once and cannot separate them.

2. Run several samples per case on both sides3 to 5

   Classify each case per version as stable pass, stable fail or flaky. A case that is flaky on either side leaves the paired comparison and goes to the flaky bucket, because a coin flip contributes to b or c at random.

3. Compute b and c on the stable cases onlyone query

   New failures and new passes. Print the counts before you print any percentage, and put both in the report header where nobody can skip them.

4. Apply the test only when there is enough to testb + c >= 10

   Below roughly ten discordant cases, no significance test is meaningful. Read the individual cases instead. Six new failures is small enough to open one at a time, and that is faster than any statistic.

5. Read every new failure by handalways

   Statistical significance decides whether the aggregate moved. It never decides whether a specific failure is acceptable. One new failure in the legal escalation slice can block a release that the arithmetic called noise.

6. Record the counts in the release noteone line

   Version compared against, b, c, the per slice table and the flaky rate. Six weeks later this is the only thing that answers whether a behaviour was already there before the release everybody suspects.

Flaky is not the same as regressed

The two look identical in a single run and need opposite responses. Separating them is the reason to sample each case several times, and the reason a suite that runs each case once cannot support a release decision.

Flaky rate deserves a threshold of its own in the release gate. A system whose flaky rate is climbing is becoming harder to evaluate, which means every future release decision gets weaker, and that degradation is invisible in any accuracy metric. The usual causes are worth fixing directly: lower the temperature for tasks that have one right answer, break ties in retrieval deterministically, and rewrite assertions that two humans would grade differently.

What runs when, expressed as configuration

The tiering and the gates should live in a file that CI reads, not in a runbook. A gate that depends on somebody remembering to check a dashboard is not a gate. This is a working shape, and the parts that matter are the pins at the top, the discordant pair gate rather than an aggregate gate, and the per slice floors.

# evals/pipeline.yaml
# read by CI and by the release gate. Nothing here is typed into a vendor console.

pins:
  generator_model: "vendor:model-name@2026-05-01"   # a dated version, never a moving alias
  judge_model:     "vendor:other-model@2026-03-19"  # a release dependency in its own right
  embedding_model: "vendor:embed@2025-11-02"        # changing this rebuilds the whole index
  temperature: 0.2
  seed: 20260826                # only some providers honour it, record it regardless
  corpus: "policies@2026-07-14"
  prompt_sha: auto              # CI fills this from the file hash, never a human

tiers:
  fast:
    runs_on: [pull_request]
    partitions: [dev]
    assertions: [structural, deterministic]
    samples_per_case: 1
    budget_minutes: 2
  nightly:
    runs_on: [schedule]
    partitions: [dev]
    assertions: [structural, deterministic, reference, rubric]
    samples_per_case: 3
  release:
    runs_on: [release_candidate]
    partitions: [holdout]
    assertions: [structural, deterministic, reference, rubric]
    samples_per_case: 5

gates:                          # any true condition blocks the release
  - new_failures:            { max: 3 }          # cases that passed on the pinned version
  - mcnemar_chi_square:      { max: 3.84, only_when: "b + c >= 10" }
  - structural_failures:     { max: 0 }
  - flaky_rate:              { max: 0.05 }
  - per_slice_pass_rate:
      legal_escalation: { min: 0.95 }
      non_english:      { min: 0.85 }
      default:          { min: 0.80 }

report:
  emit: [per_criterion, per_slice, discordant_case_ids, judge_agreement, flaky_case_ids]
  attach_diff_for: [new_failures, new_passes]
  fail_loudly_on_missing_pin: true

The last line is there because of a specific incident pattern. When a pin is missing, the run usually still succeeds, quietly falling back to a provider default, and the results look normal. A missing pin has to be a hard failure of the pipeline, in the same way a missing test fixture is, or the whole comparison silently stops meaning anything. The wider habit of treating prompts and configuration as reviewed code is covered in prompts as code.

The provider migration drill

Pinning a model version buys you stability and hands you a deadline, because pinned versions get retired. Run the migration as a planned exercise on your own schedule instead of an emergency two days before a shutdown. Check your provider's deprecation policy for the actual notice period rather than assuming one, because it differs by vendor and it changes.

Definitions, and the check before you ship

Definitions

Regression

A case that produced acceptable output under the previous configuration and produces unacceptable output under the new one. It is defined against a pinned baseline, so a system with no pinned baseline cannot have a measurable regression, only complaints.

Discordant pair

A case whose verdict differs between two versions of a system run on the same input. New failures and new passes are the only cases carrying information about a paired comparison, and their counts, not the aggregate pass rate, decide whether a change is real.

Material diff

A comparison of two free text outputs that asks only whether they differ in a way that changes what a reader would do, rather than whether one is better. It is judged blind, with the order randomised, and it is the filter that keeps prose comparison affordable.

Shadow run

Running a candidate configuration on live production traffic in parallel with the version being served, without showing its output to anyone. It exposes the candidate to real inputs no fixed dataset contains, at the cost of paying for every request twice.

Pin

An explicit dated version recorded in configuration for every input that can change behaviour: model, judge model, embedding model, prompt hash, corpus version, sampling settings. An unpinned input turns any comparison between two runs into an unattributable difference.

Flaky rate

The share of eval cases whose verdict varies across repeated samples of the same configuration. It is tracked separately from the pass rate, because counting flaky cases as failures hides real regressions and counting them as passes hides instability users already meet.

Regression testing does not stop an LLM system from changing. Nothing does, because most of what determines its behaviour is outside your repository. What it buys is the ability to notice within a day instead of within a quarter, and to say which clock moved. Pair it with production monitoring, described in monitoring AI in production, and with the assertion design in writing evals for LLM systems. Together they are the ordinary machinery of evaluation and guardrails, and they are what separates a system somebody operates from a demonstration that used to work.