Evaluation and reliability

Building a golden dataset

The set of cases you measure against decides what you can see. Here is how to source it, stratify it, label it, and keep it honest once the product moves underneath it.

On this page
  1. The short answer
  2. One name, three jobs, and they must not share a pool
  3. The Golden Five: what a case must carry to get in
  4. Where cases actually come from
  5. Stratify, or the suite is blind to the thing that breaks
  6. Labelling: two people, one spec, and what disagreement means
  7. Contamination, leakage, and the score that flatters you
  8. The case record, in a shape that survives
  9. Keeping it from rotting
  10. Definitions, and what to check before you call it golden

The short answer

The short answer

A golden dataset is a versioned set of inputs with agreed expected behaviour, built so that a change to your system produces a comparable number. Source it from production failures and stratified real traffic rather than from imagination, give every case a provenance record and a retire condition, and never let an expected answer come from the model you are testing. Size it per slice, not in total, because a slice with three cases cannot register a regression. Then accept that it decays: the product changes, the source documents change, and a case whose expected answer is now wrong is worse than no case at all.

  • 5properties every case must carry before it is allowed into the set
  • 2independent labellers per case, because a case one person cannot decide is a spec bug
  • 2 of 200cases a one percent slice receives under proportional sampling, which is why quotas existArithmetic on a stated traffic mix, not measured data
  • Neverhow often a case in the eval set may also appear in fine-tuning data or few-shot examples

The golden dataset is the part of an eval programme that cannot be bought, borrowed or generated. Assertions are code and code is portable. The cases are the encoded judgement of people who know what correct looks like in your business, and that is why the set is valuable and why nobody wants to build it.

One name, three jobs, and they must not share a pool

Teams build one pile of cases and use it for everything, which quietly destroys its value. There are three jobs and each needs its own partition, drawn from the same traffic but never mixed.

PartitionWhat it is forHow often it is runWhat breaks if you merge it
Development setIterating on prompts, retrieval and toolsConstantly, on every ideaNothing, this is what it is for
Locked holdoutDeciding whether a release is safeOnly at release, never during iterationAfter enough iterations you have tuned to the dev set, and the score stops predicting real behaviour
Calibration setMeasuring whether your model judge agrees with humansWhen the rubric changes or the judge model changesJudge agreement is measured on cases the judge was tuned against, so it looks better than it is
Live sampleChecking that the fixed set still resembles productionMonthly, drawn fresh and never frozenYou cannot tell whether the set has drifted away from your users
A rough starting split for the fixed partitions is around 60 percent development, 25 percent holdout and 15 percent calibration, adjusted so that every slice clears its own floor in each one.

The locked holdout is the partition people push back on, because it feels like withholding data from the work. It is. That is the point. Prompt engineering against a fixed set of sixty cases for forty iterations produces a prompt fitted to those sixty cases, and the gap between the dev score and the holdout score is the only measurement you have of how much of your improvement is real. If that gap is wide, you have been optimising a memory rather than a system.

The Golden Five: what a case must carry to get in

Most golden datasets fail as a records problem rather than a modelling one. A case arrives with an input and an expected answer, and eight months later nobody can say who decided that answer, whether it is still correct, or why the case exists. These five properties are the intake gate.

Framework

The Golden Five

Five properties, all five required. A candidate case missing any of them goes to the staging area, not the set.

01
Provenance

Where the case came from and why it exists: an incident id, the disagreement it settled, the slice quota it fills, or the adversarial pattern it probes. Without provenance you cannot tell a case that encodes a hard-won lesson from one somebody typed to make the file look fuller, and both will be defended equally in review.

02
Decidable expectation

The expectation must be checkable by a stated method: an exact value, a set of required document ids, a list of criteria, or a reference answer plus the rule for comparing against it. If the expectation is a paragraph describing a good answer with no procedure attached, it is a wish and it will be graded differently by every reviewer.

03
Distinctness

The case must not be a near duplicate of one already in the set. Duplicates inflate the apparent size, skew the pass rate towards whatever they repeat, and make the suite slower without making it more sensitive. Check new candidates against the existing set with an embedding similarity threshold and put anything above it in front of a person.

04
Named owner

A team that owns the definition of correct for that case. Not the engineer who added it, the function that decides the policy: people ops for expense questions, legal for contract wording, support leadership for tone. When the expected answer becomes contested, the owner settles it, and without one the case is settled by whoever is loudest that week.

05
Retire condition

The event that makes this case wrong, written when the case is added. Usually the version of a source document, a policy, a price list or a product behaviour. A case with no retire condition survives the thing it was testing and then quietly asserts an answer that is no longer true, which is the most expensive failure in this whole discipline.

Where to keep the staging area

Candidates that fail an intake gate go into a separate file rather than being thrown away, with the gate they failed recorded. A large staging file is a useful signal: many candidates failing distinctness means you are over-sampling one intent, and many failing decidable expectation means your product spec is vague in a way that will hurt you somewhere else too.

Where cases actually come from

Four sources, and a set fed by only one goes stale in a predictable way. Write the mix down as a target and check it quarterly, because whichever source is easiest will silently take over.

SourceWhat it gives youHow it fails alone
Production failuresCases that encode a real loss, with an incident to point atEvery case is a past fire, so the set only ever tests yesterday's problem
Stratified traffic samplingA set shaped like reality, including the boring majorityAlmost every sampled case is easy, so the pass rate sits high and moves slowly
Hand-written adversarial casesCoverage of the failure you fear but have not yet suffered: injection attempts, contradictory documents, out of scope questions, empty retrievalWritten by people who know the system, so they probe what the authors already imagined
Perturbations of existing casesCheap sensitivity tests: same question with a typo, a different date format, a second language, a longer documentCorrelated with their parent case, so ten perturbations are not ten independent cases and must not be counted as such

The perturbation trap is worth expanding. A suite of 300 cases where 200 are perturbations of 40 originals has roughly the statistical power of a suite of 140, not 300, because the perturbations move together. Tag perturbations with their parent id and report both counts, so nobody quotes a case count that overstates what the set can resolve.

Do not generate the expected answers with the model

Using a model to draft candidate inputs is fine, and it is a fast way to fill an adversarial quota. Using a model to produce the expected answers is not. If the expectation comes from the system under test, you have frozen current behaviour as the definition of correct, and the suite will report a perfect score while agreeing with every mistake the system already makes. The same applies to using a stronger model as an oracle: you have then built a suite that measures similarity to that model, which is a different product from the one you are shipping.

Stratify, or the suite is blind to the thing that breaks

Proportional sampling gives you a set shaped like your traffic, and traffic is shaped so that the hard cases are rare. That is the whole problem, and it is arithmetic rather than opinion.

Cases landing in each slice when 200 are sampled proportionally from a stated traffic mix
Arithmetic on an illustrative traffic mix of 62, 18, 11, 6, 2 and 1 percent. Not measured data. Substitute your own mix.
Simple lookup, 62 percent124 cases
Status question, 18 percent36 cases
Refund or credit, 11 percent22 cases
Non-English, 6 percent12 cases
Multi-document, 2 percent4 cases
Legal escalation, 1 percent2 cases
two cases cannot register a regression, and this is the slice where being wrong costs the most

The slices at the bottom are usually the expensive ones. Legal escalation is rare and a wrong answer there is the one that reaches a lawyer. So sample by quota rather than by proportion: set a floor of thirty to fifty cases per slice you care about, fill it, and then weight the slices back to traffic proportions only when you want a figure that represents the user experience. Keep both numbers and label them. The unweighted per slice table is for engineering and the weighted figure is for the business review.

How to choose the slices

  • By input property: language, document type, length, format, channel. Cheap to tag and usually where the first surprise lives.
  • By intent: what the user is trying to do. This is the slicing that predicts business impact, and it needs a taxonomy somebody owns.
  • By cost of being wrong: a tier you assign, not one you observe. Legal, financial and irreversible actions get their own slice even when they are rare.
  • By retrieval difficulty: answer in one document, answer spread across several, answer absent. The third group is the one that produces confident invention, covered in why RAG gives confident wrong answers.

Labelling: two people, one spec, and what disagreement means

The labelling process is the quality of the dataset. A set labelled by one engineer at speed is a set of that engineer's assumptions, and its errors are invisible because there is nothing to compare against.

  1. Write the labelling spec before the first labelhalf a day

    One page: what counts as correct, what counts as an acceptable variation, what to do with a case that is ambiguous, and three worked examples including one boundary case. Every argument you have during labelling belongs in this document afterwards.

  2. Label independently, two people per casethe bulk of the work

    Two labellers who cannot see each other's answers. This costs twice as much and it is the only way to find out whether the task is decidable. Choose the second labeller from the owning function rather than from engineering, because engineers agree with each other about things users care about differently.

  3. Measure agreement before you look at any modelone hour

    Count how often the two labellers matched. Report raw agreement and, if your labels are categorical, a chance corrected figure. This number is the ceiling on what any evaluation of your system can mean: the model cannot be measured more precisely than your humans can agree.

  4. Adjudicate disagreements, and file the reasonongoing

    A third person settles it, and the reason goes back into the spec. Most disagreements are not labeller error. They are the spec failing to cover a real situation, and each one you resolve improves the product definition as well as the dataset.

  5. Quarantine what cannot be settledongoing

    Cases where the adjudicator also cannot decide move to an ambiguity bucket and out of the scoring set. Then fix the product spec. A case nobody can grade is not a hard case, it is an unanswered product question wearing a costume, and grading a model on it produces a number that means nothing.

  6. Re-label a sample laterquarterly

    Pull thirty already labelled cases and label them again, blind. If today's labels disagree with the originals, your definition of correct has moved and the whole set needs a review, not just those thirty.

What labelling this set will actually cost you in calendar time

The last output is the one that changes plans. Labelling is usually squeezed into whatever hours people can spare, and a set that needs seventy hours of attention at six hours a week is a quarter of the year, not a sprint task.

0Labelling hours
0Adjudication hours
0Cost of the first pass
0Calendar weeks at that capacity

Two levers move the calendar number honestly. Cut the case count by tightening slice quotas rather than by dropping the second labeller, and cut minutes per case by improving the labelling interface so the reviewer sees the retrieved context beside the question instead of hunting for it. Cutting to one labeller does not save half the cost, it removes the only measurement of whether the labels are any good.

Contamination, leakage, and the score that flatters you

A golden dataset can be corrupted in four ways, and every one of them makes the score go up. That is what makes them hard to spot: nobody investigates good news.

  1. Oracle contamination. The expected answers were generated by a model, so the set measures agreement with that model rather than correctness.
  2. Few-shot leakage. Cases from the eval set are used as examples inside the prompt. The system has been given the answer sheet, and the leak usually happens by accident when someone picks good examples from the same file.
  3. Fine-tune leakage. Eval cases end up in training data, often through a shared export of production conversations. Any case that has been trained on has to leave the set, not be flagged.
  4. Iteration leakage. Nobody copied anything, but forty rounds of prompt tuning against the same dev cases has fitted the prompt to them. This is the common one, it leaves no trace, and the locked holdout exists specifically to detect it.

The controls are boring and they work. Keep eval case ids in a deny list that the training data pipeline checks, tag every few-shot example with its origin and forbid origins inside the eval set, hash the input text of every case and check for those hashes in any training export, and always report the dev score and the holdout score side by side rather than only the better one. If you also plan on fine-tuning, the interaction between these two data sets is worth thinking through early, and fine-tuning versus prompting covers the decision itself.

The case record, in a shape that survives

One file, one line per case, in the repository next to the code. Not a spreadsheet, because a spreadsheet has no diff and no review, and the moment two people edit it the history of who decided what is gone. This shape carries the Golden Five in fields rather than in a convention people remember.

Golden set case record, one object per line in a JSONL filejson
// one JSON object per line, one line per case, in version control
{
  "case_id": "gs_00417",
  "input": {
    "question": "Can I expense a taxi to the airport on a domestic trip?",
    "user_role": "employee",
    "locale": "en-GB"
  },
  "fixtures": {
    "corpus_version": "policies@2026-07-14",
    "retrieval": "live",
    "tool_stubs": { "hr_lookup": "fixtures/hr_lookup_00417.json" }
  },
  "expectation": {
    "kind": "criteria",
    "must_cite_doc_ids": ["POL-TRV-11"],
    "must_not_cite_doc_ids": ["POL-TRV-09"],
    "criteria": ["c1_grounded", "c2_complete", "c4_refusal"],
    "forbidden_content": ["a reimbursement percentage", "an approval promise"],
    "reference": "Standard class ground transport to and from the airport is reimbursable on domestic travel (POL-TRV-11)."
  },
  "slices": {
    "intent": "policy_lookup",
    "locale": "en-GB",
    "doc_type": "policy",
    "tier": "employee",
    "difficulty": "tail"
  },
  "provenance": {
    "source": "incident",
    "ref": "INC-2026-0713",
    "added_by": "r.mehta",
    "added_on": "2026-07-16",
    "note": "system answered from POL-TRV-09, which was superseded in May"
  },
  "labelling": {
    "owner_team": "people-ops",
    "labelled_by": ["r.mehta", "s.okafor"],
    "agreement": "full",
    "adjudicated_by": null
  },
  "lifecycle": {
    "status": "active",
    "retire_when": "POL-TRV-11 is superseded or withdrawn",
    "review_by": "2027-01-31",
    "supersedes": null
  }
}

Three details matter more than they look. corpus_version pins the documents so a case can be re-run in a year and mean the same thing. must_not_cite_doc_ids encodes the specific mistake the incident produced, which is a rung two assertion and costs nothing to check. And supersedes exists because cases are never edited in place: when an expected answer changes, you add a new case that supersedes the old one and mark the old one retired, so the history of what you believed remains readable.

Keeping it from rotting

Every golden dataset decays, and the decay is silent because a stale case keeps passing until the day the system gets it right and the suite calls that a failure. Put a maintenance rhythm around it from the start rather than discovering the rot during an incident.

Every release
Freeze the version you measured against

Tag the dataset version in the release record. Six weeks later, when somebody asks whether a behaviour regressed, the question is answerable only if you know which cases and which expectations produced the number you shipped on.

Weekly
Harvest new cases from failures

Pull the week's incidents, human overrides and confirmed complaints into the staging area with provenance attached. Do this weekly and it is twenty minutes. Do it quarterly and it becomes an archaeology project nobody volunteers for.

Monthly
Compare the set against live traffic

Draw a fresh unfrozen sample from production and check its slice distribution against the set's. When a new intent appears in traffic and has no cases, the suite is blind to your newest users, who are exactly the ones most likely to break something.

Quarterly
Run the retire conditions

For every case, check whether its retire condition has fired: source document superseded, policy changed, product behaviour deliberately changed. Retire what fired. This is the single highest value hour in the maintenance calendar.

Quarterly
Re-label thirty cases blind

If the new labels disagree with the old ones, your definition of correct has moved. That is not a problem with the labellers, it is information about the product, and it means the spec and a slice of the set both need revisiting.

Yearly
Rebuild the calibration set

Judge rubrics change, judge models change and the boundary cases that were hard last year are often routine now. A calibration set that no longer contains genuinely difficult examples reports high agreement and tells you nothing.

The stale expectation is the expensive failure

A case whose expected answer is out of date does active damage. It fails a correct system, an engineer spends a day investigating, and the eventual fix is to change the case. Do that twice and the team learns that suite failures are usually the suite's fault, which is the moment the whole apparatus stops working as a control. Retire conditions are the cheap insurance against it, and they only work if they were written when the case was added, not afterwards.

Definitions, and what to check before you call it golden

Definitions
Golden dataset
A versioned set of inputs paired with agreed expected behaviour, used to measure a system's output across changes. It differs from a test fixture in that expectations are decided by the function that owns the definition of correct, and each case carries provenance and a condition under which it is retired.
Locked holdout
A partition of the golden dataset that is run only before a release and never used while iterating. Its purpose is to measure overfitting: the gap between the development score and the holdout score is how much of an apparent improvement was fitted to the cases rather than to the task.
Stratified sampling
Drawing eval cases by quota per slice rather than in proportion to traffic. Proportional sampling gives rare slices too few cases to detect a change in them, and rare slices are usually the ones where being wrong is most expensive.
Inter-annotator agreement
How often two independent labellers assign the same label to the same case. It is the ceiling on evaluation precision, because a system cannot be measured more finely than the people defining correctness can agree, and a low figure indicates an ambiguous spec rather than careless labellers.
Contamination
Any route by which the system under test has already seen the eval cases or their answers, through few-shot examples, fine-tuning data, or expected answers generated by a model. Contamination raises the score without changing behaviour, which is why it is rarely investigated.
Retire condition
The event recorded at intake that makes a case's expected answer wrong, usually the version of a source document or policy. Without one, a case outlives the thing it tested and begins failing a system that has become correct.
Golden dataset readiness
0 of 12 done

That set is the asset. The assertions that read it, described in writing evals for LLM systems, and the comparison machinery that runs it on every change, described in catching model regressions, are both replaceable. The encoded judgement about what your business considers a correct answer is not, and it is the reason evaluation and guardrails work starts with people rather than with tooling.

Cite this

ChatGPTalker, "Building a Golden Dataset That Does Not Rot in Six Months" (2026). A golden dataset is a versioned set of inputs with agreed expected behaviour. Every case needs provenance, a decidable expectation, distinctness, a named owner and a retire condition, sampled by quota per slice rather than in proportion to traffic.

Questions readers ask next

How many cases should a golden dataset contain?
Size it per slice rather than in total. A floor of thirty to fifty cases in each slice you care about is a workable starting point, which for six meaningful slices puts the set somewhere between two and three hundred. Total size matters less than distribution: a thousand cases drawn proportionally from traffic will still leave your rarest and most expensive slice with a handful of cases and no ability to detect a regression in it.
Can I use synthetic data to build a golden dataset?
Use it for inputs, not for expectations. Generating candidate questions, adversarial phrasings and format variations is a reasonable way to fill a quota quickly, provided a human then decides what the correct behaviour is for each one. Generating the expected answers with a model means your suite measures agreement with that model rather than correctness, and it will report a high score while endorsing every mistake your system already makes.
Who should label the eval cases?
The function that owns the definition of correct, paired with someone from the build team. People ops decides expense policy answers, legal decides contract wording, support leadership decides tone and escalation. Two independent labellers per case, with the agreement rate recorded, because a case two informed people cannot agree on is a gap in your product specification rather than a hard test case.
How often should a golden dataset be updated?
Harvest new cases from failures weekly, compare slice distribution against live traffic monthly, and run the retire conditions quarterly. The quarterly retire pass is the one people skip and the one that prevents the worst failure mode, which is a case whose expected answer has gone out of date and now fails a system that has become correct.
What is the difference between a golden dataset and training data?
Training data teaches the system, a golden dataset measures it, and any overlap between the two destroys the measurement. Keep eval case ids in a deny list that your training pipeline checks, hash the input text and scan any training export for those hashes, and treat a case that has been trained on as retired rather than flagged. This applies to few-shot examples inside the prompt as much as to fine-tuning.
How do I handle cases where reviewers disagree about the right answer?
Send it to an adjudicator, record the reason, and put the reason back into the labelling spec. If the adjudicator also cannot decide, move the case to an ambiguity bucket outside the scoring set and treat it as an open product question. Grading a model on a case no human can grade produces a number that looks like measurement and is not, and it will be quoted in a meeting eventually.
Do I need a golden dataset if I already have production monitoring?
Yes, because they answer different questions. Monitoring tells you what happened to real work and cannot isolate which change caused it. A fixed dataset tells you whether a specific change is safe to ship, before any user meets it. Teams that have only monitoring end up debugging every incident as an argument about which of nine recent changes was responsible.
Cite this

ChatGPTalker. "Building a Golden Dataset That Does Not Rot in Six Months." chatgptalker.com, 2026-08-26. https://chatgptalker.com/guides/building-a-golden-dataset/

Rather have it built than read about it?

Send the process you want automated. You get a scoped plan back, with the build shape, the stack and a realistic timeline.

Start a project