On this page
- The short answer
- One name, three jobs, and they must not share a pool
- The Golden Five: what a case must carry to get in
- Where cases actually come from
- Stratify, or the suite is blind to the thing that breaks
- Labelling: two people, one spec, and what disagreement means
- Contamination, leakage, and the score that flatters you
- The case record, in a shape that survives
- Keeping it from rotting
- Definitions, and what to check before you call it golden
The short answer
A golden dataset is a versioned set of inputs with agreed expected behaviour, built so that a change to your system produces a comparable number. Source it from production failures and stratified real traffic rather than from imagination, give every case a provenance record and a retire condition, and never let an expected answer come from the model you are testing. Size it per slice, not in total, because a slice with three cases cannot register a regression. Then accept that it decays: the product changes, the source documents change, and a case whose expected answer is now wrong is worse than no case at all.
- 5properties every case must carry before it is allowed into the set
- 2independent labellers per case, because a case one person cannot decide is a spec bug
- 2 of 200cases a one percent slice receives under proportional sampling, which is why quotas existArithmetic on a stated traffic mix, not measured data
- Neverhow often a case in the eval set may also appear in fine-tuning data or few-shot examples
The golden dataset is the part of an eval programme that cannot be bought, borrowed or generated. Assertions are code and code is portable. The cases are the encoded judgement of people who know what correct looks like in your business, and that is why the set is valuable and why nobody wants to build it.
One name, three jobs, and they must not share a pool
Teams build one pile of cases and use it for everything, which quietly destroys its value. There are three jobs and each needs its own partition, drawn from the same traffic but never mixed.
| Partition | What it is for | How often it is run | What breaks if you merge it |
|---|---|---|---|
| Development set | Iterating on prompts, retrieval and tools | Constantly, on every idea | Nothing, this is what it is for |
| Locked holdout | Deciding whether a release is safe | Only at release, never during iteration | After enough iterations you have tuned to the dev set, and the score stops predicting real behaviour |
| Calibration set | Measuring whether your model judge agrees with humans | When the rubric changes or the judge model changes | Judge agreement is measured on cases the judge was tuned against, so it looks better than it is |
| Live sample | Checking that the fixed set still resembles production | Monthly, drawn fresh and never frozen | You cannot tell whether the set has drifted away from your users |
The locked holdout is the partition people push back on, because it feels like withholding data from the work. It is. That is the point. Prompt engineering against a fixed set of sixty cases for forty iterations produces a prompt fitted to those sixty cases, and the gap between the dev score and the holdout score is the only measurement you have of how much of your improvement is real. If that gap is wide, you have been optimising a memory rather than a system.
The Golden Five: what a case must carry to get in
Most golden datasets fail as a records problem rather than a modelling one. A case arrives with an input and an expected answer, and eight months later nobody can say who decided that answer, whether it is still correct, or why the case exists. These five properties are the intake gate.
The Golden Five
Five properties, all five required. A candidate case missing any of them goes to the staging area, not the set.
Where the case came from and why it exists: an incident id, the disagreement it settled, the slice quota it fills, or the adversarial pattern it probes. Without provenance you cannot tell a case that encodes a hard-won lesson from one somebody typed to make the file look fuller, and both will be defended equally in review.
The expectation must be checkable by a stated method: an exact value, a set of required document ids, a list of criteria, or a reference answer plus the rule for comparing against it. If the expectation is a paragraph describing a good answer with no procedure attached, it is a wish and it will be graded differently by every reviewer.
The case must not be a near duplicate of one already in the set. Duplicates inflate the apparent size, skew the pass rate towards whatever they repeat, and make the suite slower without making it more sensitive. Check new candidates against the existing set with an embedding similarity threshold and put anything above it in front of a person.
A team that owns the definition of correct for that case. Not the engineer who added it, the function that decides the policy: people ops for expense questions, legal for contract wording, support leadership for tone. When the expected answer becomes contested, the owner settles it, and without one the case is settled by whoever is loudest that week.
The event that makes this case wrong, written when the case is added. Usually the version of a source document, a policy, a price list or a product behaviour. A case with no retire condition survives the thing it was testing and then quietly asserts an answer that is no longer true, which is the most expensive failure in this whole discipline.
Candidates that fail an intake gate go into a separate file rather than being thrown away, with the gate they failed recorded. A large staging file is a useful signal: many candidates failing distinctness means you are over-sampling one intent, and many failing decidable expectation means your product spec is vague in a way that will hurt you somewhere else too.
Where cases actually come from
Four sources, and a set fed by only one goes stale in a predictable way. Write the mix down as a target and check it quarterly, because whichever source is easiest will silently take over.
| Source | What it gives you | How it fails alone |
|---|---|---|
| Production failures | Cases that encode a real loss, with an incident to point at | Every case is a past fire, so the set only ever tests yesterday's problem |
| Stratified traffic sampling | A set shaped like reality, including the boring majority | Almost every sampled case is easy, so the pass rate sits high and moves slowly |
| Hand-written adversarial cases | Coverage of the failure you fear but have not yet suffered: injection attempts, contradictory documents, out of scope questions, empty retrieval | Written by people who know the system, so they probe what the authors already imagined |
| Perturbations of existing cases | Cheap sensitivity tests: same question with a typo, a different date format, a second language, a longer document | Correlated with their parent case, so ten perturbations are not ten independent cases and must not be counted as such |
The perturbation trap is worth expanding. A suite of 300 cases where 200 are perturbations of 40 originals has roughly the statistical power of a suite of 140, not 300, because the perturbations move together. Tag perturbations with their parent id and report both counts, so nobody quotes a case count that overstates what the set can resolve.
Using a model to draft candidate inputs is fine, and it is a fast way to fill an adversarial quota. Using a model to produce the expected answers is not. If the expectation comes from the system under test, you have frozen current behaviour as the definition of correct, and the suite will report a perfect score while agreeing with every mistake the system already makes. The same applies to using a stronger model as an oracle: you have then built a suite that measures similarity to that model, which is a different product from the one you are shipping.
Stratify, or the suite is blind to the thing that breaks
Proportional sampling gives you a set shaped like your traffic, and traffic is shaped so that the hard cases are rare. That is the whole problem, and it is arithmetic rather than opinion.
The slices at the bottom are usually the expensive ones. Legal escalation is rare and a wrong answer there is the one that reaches a lawyer. So sample by quota rather than by proportion: set a floor of thirty to fifty cases per slice you care about, fill it, and then weight the slices back to traffic proportions only when you want a figure that represents the user experience. Keep both numbers and label them. The unweighted per slice table is for engineering and the weighted figure is for the business review.
How to choose the slices
- By input property: language, document type, length, format, channel. Cheap to tag and usually where the first surprise lives.
- By intent: what the user is trying to do. This is the slicing that predicts business impact, and it needs a taxonomy somebody owns.
- By cost of being wrong: a tier you assign, not one you observe. Legal, financial and irreversible actions get their own slice even when they are rare.
- By retrieval difficulty: answer in one document, answer spread across several, answer absent. The third group is the one that produces confident invention, covered in why RAG gives confident wrong answers.
Labelling: two people, one spec, and what disagreement means
The labelling process is the quality of the dataset. A set labelled by one engineer at speed is a set of that engineer's assumptions, and its errors are invisible because there is nothing to compare against.
- Write the labelling spec before the first label
One page: what counts as correct, what counts as an acceptable variation, what to do with a case that is ambiguous, and three worked examples including one boundary case. Every argument you have during labelling belongs in this document afterwards.
- Label independently, two people per case
Two labellers who cannot see each other's answers. This costs twice as much and it is the only way to find out whether the task is decidable. Choose the second labeller from the owning function rather than from engineering, because engineers agree with each other about things users care about differently.
- Measure agreement before you look at any model
Count how often the two labellers matched. Report raw agreement and, if your labels are categorical, a chance corrected figure. This number is the ceiling on what any evaluation of your system can mean: the model cannot be measured more precisely than your humans can agree.
- Adjudicate disagreements, and file the reason
A third person settles it, and the reason goes back into the spec. Most disagreements are not labeller error. They are the spec failing to cover a real situation, and each one you resolve improves the product definition as well as the dataset.
- Quarantine what cannot be settled
Cases where the adjudicator also cannot decide move to an ambiguity bucket and out of the scoring set. Then fix the product spec. A case nobody can grade is not a hard case, it is an unanswered product question wearing a costume, and grading a model on it produces a number that means nothing.
- Re-label a sample later
Pull thirty already labelled cases and label them again, blind. If today's labels disagree with the originals, your definition of correct has moved and the whole set needs a review, not just those thirty.
The last output is the one that changes plans. Labelling is usually squeezed into whatever hours people can spare, and a set that needs seventy hours of attention at six hours a week is a quarter of the year, not a sprint task.
Two levers move the calendar number honestly. Cut the case count by tightening slice quotas rather than by dropping the second labeller, and cut minutes per case by improving the labelling interface so the reviewer sees the retrieved context beside the question instead of hunting for it. Cutting to one labeller does not save half the cost, it removes the only measurement of whether the labels are any good.
Contamination, leakage, and the score that flatters you
A golden dataset can be corrupted in four ways, and every one of them makes the score go up. That is what makes them hard to spot: nobody investigates good news.
- Oracle contamination. The expected answers were generated by a model, so the set measures agreement with that model rather than correctness.
- Few-shot leakage. Cases from the eval set are used as examples inside the prompt. The system has been given the answer sheet, and the leak usually happens by accident when someone picks good examples from the same file.
- Fine-tune leakage. Eval cases end up in training data, often through a shared export of production conversations. Any case that has been trained on has to leave the set, not be flagged.
- Iteration leakage. Nobody copied anything, but forty rounds of prompt tuning against the same dev cases has fitted the prompt to them. This is the common one, it leaves no trace, and the locked holdout exists specifically to detect it.
The controls are boring and they work. Keep eval case ids in a deny list that the training data pipeline checks, tag every few-shot example with its origin and forbid origins inside the eval set, hash the input text of every case and check for those hashes in any training export, and always report the dev score and the holdout score side by side rather than only the better one. If you also plan on fine-tuning, the interaction between these two data sets is worth thinking through early, and fine-tuning versus prompting covers the decision itself.
The case record, in a shape that survives
One file, one line per case, in the repository next to the code. Not a spreadsheet, because a spreadsheet has no diff and no review, and the moment two people edit it the history of who decided what is gone. This shape carries the Golden Five in fields rather than in a convention people remember.
// one JSON object per line, one line per case, in version control
{
"case_id": "gs_00417",
"input": {
"question": "Can I expense a taxi to the airport on a domestic trip?",
"user_role": "employee",
"locale": "en-GB"
},
"fixtures": {
"corpus_version": "policies@2026-07-14",
"retrieval": "live",
"tool_stubs": { "hr_lookup": "fixtures/hr_lookup_00417.json" }
},
"expectation": {
"kind": "criteria",
"must_cite_doc_ids": ["POL-TRV-11"],
"must_not_cite_doc_ids": ["POL-TRV-09"],
"criteria": ["c1_grounded", "c2_complete", "c4_refusal"],
"forbidden_content": ["a reimbursement percentage", "an approval promise"],
"reference": "Standard class ground transport to and from the airport is reimbursable on domestic travel (POL-TRV-11)."
},
"slices": {
"intent": "policy_lookup",
"locale": "en-GB",
"doc_type": "policy",
"tier": "employee",
"difficulty": "tail"
},
"provenance": {
"source": "incident",
"ref": "INC-2026-0713",
"added_by": "r.mehta",
"added_on": "2026-07-16",
"note": "system answered from POL-TRV-09, which was superseded in May"
},
"labelling": {
"owner_team": "people-ops",
"labelled_by": ["r.mehta", "s.okafor"],
"agreement": "full",
"adjudicated_by": null
},
"lifecycle": {
"status": "active",
"retire_when": "POL-TRV-11 is superseded or withdrawn",
"review_by": "2027-01-31",
"supersedes": null
}
}
Three details matter more than they look. corpus_version pins the documents so a case can be re-run in a year and mean the same thing. must_not_cite_doc_ids encodes the specific mistake the incident produced, which is a rung two assertion and costs nothing to check. And supersedes exists because cases are never edited in place: when an expected answer changes, you add a new case that supersedes the old one and mark the old one retired, so the history of what you believed remains readable.
Keeping it from rotting
Every golden dataset decays, and the decay is silent because a stale case keeps passing until the day the system gets it right and the suite calls that a failure. Put a maintenance rhythm around it from the start rather than discovering the rot during an incident.
Tag the dataset version in the release record. Six weeks later, when somebody asks whether a behaviour regressed, the question is answerable only if you know which cases and which expectations produced the number you shipped on.
Pull the week's incidents, human overrides and confirmed complaints into the staging area with provenance attached. Do this weekly and it is twenty minutes. Do it quarterly and it becomes an archaeology project nobody volunteers for.
Draw a fresh unfrozen sample from production and check its slice distribution against the set's. When a new intent appears in traffic and has no cases, the suite is blind to your newest users, who are exactly the ones most likely to break something.
For every case, check whether its retire condition has fired: source document superseded, policy changed, product behaviour deliberately changed. Retire what fired. This is the single highest value hour in the maintenance calendar.
If the new labels disagree with the old ones, your definition of correct has moved. That is not a problem with the labellers, it is information about the product, and it means the spec and a slice of the set both need revisiting.
Judge rubrics change, judge models change and the boundary cases that were hard last year are often routine now. A calibration set that no longer contains genuinely difficult examples reports high agreement and tells you nothing.
A case whose expected answer is out of date does active damage. It fails a correct system, an engineer spends a day investigating, and the eventual fix is to change the case. Do that twice and the team learns that suite failures are usually the suite's fault, which is the moment the whole apparatus stops working as a control. Retire conditions are the cheap insurance against it, and they only work if they were written when the case was added, not afterwards.
Definitions, and what to check before you call it golden
- Golden dataset
- A versioned set of inputs paired with agreed expected behaviour, used to measure a system's output across changes. It differs from a test fixture in that expectations are decided by the function that owns the definition of correct, and each case carries provenance and a condition under which it is retired.
- Locked holdout
- A partition of the golden dataset that is run only before a release and never used while iterating. Its purpose is to measure overfitting: the gap between the development score and the holdout score is how much of an apparent improvement was fitted to the cases rather than to the task.
- Stratified sampling
- Drawing eval cases by quota per slice rather than in proportion to traffic. Proportional sampling gives rare slices too few cases to detect a change in them, and rare slices are usually the ones where being wrong is most expensive.
- Inter-annotator agreement
- How often two independent labellers assign the same label to the same case. It is the ceiling on evaluation precision, because a system cannot be measured more finely than the people defining correctness can agree, and a low figure indicates an ambiguous spec rather than careless labellers.
- Contamination
- Any route by which the system under test has already seen the eval cases or their answers, through few-shot examples, fine-tuning data, or expected answers generated by a model. Contamination raises the score without changing behaviour, which is why it is rarely investigated.
- Retire condition
- The event recorded at intake that makes a case's expected answer wrong, usually the version of a source document or policy. Without one, a case outlives the thing it tested and begins failing a system that has become correct.
That set is the asset. The assertions that read it, described in writing evals for LLM systems, and the comparison machinery that runs it on every change, described in catching model regressions, are both replaceable. The encoded judgement about what your business considers a correct answer is not, and it is the reason evaluation and guardrails work starts with people rather than with tooling.
ChatGPTalker, "Building a Golden Dataset That Does Not Rot in Six Months" (2026). A golden dataset is a versioned set of inputs with agreed expected behaviour. Every case needs provenance, a decidable expectation, distinctness, a named owner and a retire condition, sampled by quota per slice rather than in proportion to traffic.
Questions readers ask next
How many cases should a golden dataset contain?
Can I use synthetic data to build a golden dataset?
Who should label the eval cases?
How often should a golden dataset be updated?
What is the difference between a golden dataset and training data?
How do I handle cases where reviewers disagree about the right answer?
Do I need a golden dataset if I already have production monitoring?
ChatGPTalker. "Building a Golden Dataset That Does Not Rot in Six Months." chatgptalker.com, 2026-08-26. https://chatgptalker.com/guides/building-a-golden-dataset/