Structured Output: Getting JSON You Can Trust

Valid JSON and correct JSON are different problems

The progression is familiar. A prompt asking for JSON returns the occasional trailing comma, markdown fence or helpful preamble. You switch on a schema and those failures disappear overnight. Six weeks later somebody in finance notices that a supplier name has been quietly filled in on documents where it never appeared, and it has been happening since the day the shape errors stopped.

That is not a coincidence. Removing the shape errors removed your only visible symptom. A malformed response is loud and lands in an error queue. A well-formed wrong response looks exactly like a well-formed right one, flows into your database, and is found weeks later or never.

Terms used precisely

Structured output

A generation mode where the model's output is required to conform to a declared schema, so the response parses into a known shape without any post-processing or repair step.

Constrained decoding

The mechanism behind it: at each generation step the sampler is restricted to tokens that can still lead to a valid document under the schema, which makes an invalid response impossible rather than unlikely.

Abstention channel

An explicit, schema-legal way for the model to report that it has no answer, usually a nullable value paired with a required reason code, so that saying nothing is a valid response rather than a rule violation.

Grounding

The check that an extracted value actually appears in the source material, most cheaply done by requiring the model to return the exact span it copied and verifying that span exists in the input.

Fabrication rate

The share of cases where a field is genuinely absent from the source and the system returns a value anyway. It is the single most useful number for a structured extractor and almost nobody measures it.

The four ways to ask for JSON, and what each one guarantees

These are not four flavours of the same thing. They guarantee different properties and fail in different places, and mixing them up produces the wrong debugging instinct.

Schema design is where the reliability actually lives

The schema is not a type declaration. It is part of the prompt, it is read on every call, and its structure changes what the model produces. Six design choices carry most of the outcome.

Order fields so that evidence comes before conclusions

Generation is autoregressive. Every token is conditioned on the tokens already emitted, which means a field's position in your schema is a causal choice and not a cosmetic one. Put the quoted source span before the extracted value and the model has to find text before it can commit to an answer. Put it after and you get a value first, then a span invented to justify it. The same logic applies to a reasoning field: before the answer it is reasoning, after the answer it is a press release.

Prefer closed sets, and split large ones

An enum is the only part of a schema that constrains meaning rather than type, so use it wherever the legal answers are known. The cost is tokens, since the list is sent on every call, and a taxonomy with a hundred members is both expensive and harder to discriminate within. Above roughly twenty options, a coarse call followed by a narrow one usually beats a single long list on both accuracy and cost.

Make fields required and nullable, not optional

An optional field lets the model skip it, and you cannot distinguish deliberate omission from a field it forgot. A required non-nullable field forces a value even when there is none, which is where fabrication comes from. Required and nullable is the combination that works: the model must address every field, and null is an available answer. Pair it with a required reason code and null becomes a data point rather than a hole.

Keep it shallow and avoid top-level unions

Every level of nesting is structure the model holds open while it generates, and deep objects are where truncation does the most damage, since a cut at depth four destroys the whole document. A top-level union is worse, because the model commits to a branch early and spends the rest of the generation defending that commitment. For genuinely different document types, make the type a first field with an enum and let a second call handle the branch.

Cap every array and let the model report the cap

An unbounded array is an invitation to generate until the token limit, and truncation inside an array is the most common cause of an unparseable response even with constrained decoding, because the ceiling is enforced by the sampler and not by the grammar. Set a maximum, and add a flag the model can set to say the list was longer than the cap. Silence about truncation is worse than the truncation.

Take numbers and dates out of the model's hands

Models format numbers the way the surrounding text does, so you will receive thousands separators, currency symbols and decimal commas. Declare amounts as strings with a strict pattern and parse them in your own code where a failure is visible. Dates are the same problem with a sharper edge: a document showing 03/04/2026 is genuinely ambiguous, and a model that resolves it is guessing. Require ISO format and give it a way to report the ambiguity instead.

Field descriptions are the last lever. They act as instructions and are billed on every call, so a hundred-field schema with a paragraph on each becomes a permanent tax. Write them short and imperative, and keep the long explanation in the system prompt where it appears once.

Give the model a legal way to say nothing

This is the highest-value change on most extraction systems, and it takes an afternoon. A model constrained to produce a string will produce a string. If the document has no purchase order number, and your schema says purchase_order is a required string, the only legal moves available to the sampler are strings that look like purchase order numbers. The system is behaving exactly as designed, which is the problem.

The fix has three parts. Make the value nullable, add a required reason code with a small enum so that null carries information rather than mere absence, and say plainly in the system prompt that a blank field is a correct answer. The model's default posture is helpfulness, and helpfulness looks like filling things in.

• absent is the ordinary case: the field is not in this document and that is normal.
• illegible separates a scanning problem from a content problem, and routes to a different queue.
• ambiguous_multiple_candidates is the one that saves you. Two plausible supplier names is a human decision, not a coin flip.
• wrong_document_kind catches the delivery note that arrived in the invoice pipeline before it corrupts twelve other fields.

The reason codes then become operational signal. A rising illegible rate is an upstream scanner problem, not a prompt problem. A rising ambiguity rate on one supplier means their template changed. Neither is visible in a system whose only vocabulary is a value or a crash.

The four-layer output contract

Teams talk about structured output as one property, which is why they stop work after the first layer is solved. It is four properties, each enforced in a different place, each with its own failure and its own test. Only the first is free.

The layers are also a build order. Do them in sequence and each one narrows what the next has to handle. Skip to the fourth and you will find yourself writing business rules that exist only to catch invented values, which is an expensive way to solve a problem that a nullable field would have prevented.

Validate, then repair in the right order

The failure path matters more than the happy path, because the happy path is now guaranteed. Run the stages in this order and log the stage that stopped each call, since the distribution across stages is the most useful debugging artifact you will have.

1. Check the finish reason before you check the outputfree

   If generation stopped because it hit the token ceiling, the output is truncated and the content is irrelevant. This is not a retry case. Retrying the same request truncates again, at the same place, for the same money. Reduce the field set, raise the output reservation, or split the document.

2. Parse and validate against the schemamicroseconds

   With constrained decoding this should never fail. If it does, you have a provider incident, a schema silently rejected and fallen back from, or a length cut you missed at step one. Alert rather than repair, because the interesting information is that it happened at all.

3. Run the domain validatormilliseconds

   Dates parse, amounts parse, identifiers resolve against your own tables, enums map to live internal codes. This is your code and it should be strict. Every rule here started as a real bad row, so keep a comment naming it.

4. Verify the evidence spansmilliseconds

   For every non-null tracked field, confirm the returned span exists in the input after whitespace normalisation. Mismatches are fabrication, and they go to a human queue rather than into a retry loop, because a retry usually produces a different fabrication rather than an admission.

5. Retry once, with the error attachedone extra call

   Send the failed output back with the specific validation error and an instruction to fix only that field. One retry catches most transient failures. A second rarely helps and a third never does, so cap it at two and make the cap a constant somebody can find.

6. Dead-letter with everything attachedhuman time

   Store the input, the prompt version, the schema version, the raw output and the stage that failed. A dead letter without the prompt version is close to useless three weeks later. Automation error handling covers the queue design and who reads it.

One big schema or several small ones

Once a schema passes about twenty fields, splitting the extraction into field groups usually wins, and the reason is not accuracy in the abstract. It is that failure stops being all-or-nothing.

The deciding question is whether the fields are genuinely interdependent. Header fields and totals belong together because they constrain each other. Line items belong in their own call because they are repetitive and long. Anything requiring judgement, a category or a routing decision, belongs in a third call that can be evaluated and versioned on its own. If prefix caching applies to your stable document prefix, the cost argument for a single call gets weaker, which is worth checking before you assume splitting is expensive.

What it costs, and where the retries hide

Two line items get left out of structured extraction budgets. The schema is re-sent on every call, so a large schema is a fixed tax multiplied by volume. And the retry rate multiplies everything, including the document tokens you already paid for once. Both are easy to compute and neither shows up until the invoice does.

Run it with your real schema size and the answer is often uncomfortable. Full-sentence field descriptions, a long enum and a system prompt nobody has trimmed since launch can push the fixed overhead close to the document itself on short inputs. Token cost arithmetic has the general version of this calculation.

Testing an extractor without fooling yourself

The metric that matters is per field, and the case that matters most is the one where the answer is nothing. An extractor that scores well on documents containing every field, and fabricates on documents that do not, will look excellent on any test set drawn from your happy path.

Build the set from real documents and include the awkward population: fields that are genuinely absent, partly illegible scans, a document of the wrong type that slipped into the pipeline, two invoices in one file, and one where the field label appears but the value does not. That last case is the sharpest test there is, because the label alone is enough to pull a plausible value out of a model with no way to abstain.

Freeze the set, version it, and run it on every prompt change and every model change. Providers update models behind stable names, and an extractor steady for months can shift in a week with nothing in your repository changing. Writing evals for LLM systems covers the mechanics, and prompts as code covers pinning prompt and schema together.

The schema and prompt, ready to lift

The pair below is a working starting point. The schema carries the abstention channel, evidence-before-value ordering, string amounts with a pattern, a capped notes array, and a document-kind field decided first. The system prompt states the trade the schema alone cannot express.

{
  "name": "invoice_extraction",
  "strict": true,
  "schema": {
    "type": "object",
    "additionalProperties": false,
    "required": ["document_kind", "supplier_name", "invoice_number", "invoice_date",
                 "total_amount", "currency", "line_item_count", "notes"],
    "properties": {

      "document_kind": {
        "type": "string",
        "enum": ["invoice", "credit_note", "statement", "delivery_note", "unreadable", "other"],
        "description": "Decide this FIRST. Everything below is conditioned on it."
      },

      "supplier_name": {
        "type": "object",
        "additionalProperties": false,
        "required": ["evidence", "value", "not_found_reason"],
        "properties": {
          "evidence": {
            "type": ["string", "null"],
            "maxLength": 200,
            "description": "The exact span copied from the document. Emitted BEFORE the value."
          },
          "value": {
            "type": ["string", "null"],
            "description": "Null unless the evidence field above contains it verbatim."
          },
          "not_found_reason": {
            "type": ["string", "null"],
            "enum": ["absent", "illegible", "ambiguous_multiple_candidates", "wrong_document_kind", null],
            "description": "Required when value is null. This is the legal way to say nothing."
          }
        }
      },

      "invoice_number": { "$ref": "#/$defs/tracked_string" },
      "purchase_order":  { "$ref": "#/$defs/tracked_string" },

      "invoice_date": {
        "type": ["string", "null"],
        "description": "ISO 8601 only. For an ambiguous form such as 03/04/2026, return null and set date_ambiguity."
      },
      "date_ambiguity": {
        "type": ["string", "null"],
        "enum": ["day_month_order_unclear", "year_missing", "multiple_dates_present", null]
      },

      "total_amount": {
        "type": ["string", "null"],
        "pattern": "^-?[0-9]+(\\.[0-9]{1,2})?$",
        "description": "Digits and a decimal point only. No separators, no symbol. String, not number: your code parses it."
      },
      "currency": {
        "type": ["string", "null"],
        "enum": ["GBP", "EUR", "USD", "INR", "AED", null]
      },

      "line_item_count": { "type": ["integer", "null"], "minimum": 0, "maximum": 500 },

      "notes": {
        "type": "array",
        "maxItems": 5,
        "items": {
          "type": "string",
          "enum": ["handwritten_annotation_present", "page_missing", "poor_scan_quality",
                   "multiple_invoices_in_one_file", "totals_do_not_add_up"]
        }
      }
    },
    "$defs": {
      "tracked_string": { "$comment": "same three-property shape as supplier_name above" }
    }
  }
}

You extract fields from supplier documents. You do not interpret them, summarise them,
or correct them.

RULES, in priority order:

(1) Evidence before value. For every tracked field, write the `evidence` span first, copied
   character for character from the document. Then write `value`. If you cannot copy a span
   that contains the value, `value` is null.

(2) Null is a correct answer. A missing field is normal. When you set a value to null, set
   `not_found_reason` to the code that describes why. Never infer a value from another field
   or from what an invoice usually contains.

(3) Never compute. Do not add line items, convert currencies, reconcile totals, or normalise a
   date you had to guess at. If the printed total disagrees with the line items, report the
   printed total and add "totals_do_not_add_up" to `notes`.

(4) Ambiguity is reported, not resolved. Two candidate supplier names means
   `ambiguous_multiple_candidates`, not the one that looks more likely. A date of 03/04/2026
   with no other signal means null plus `day_month_order_unclear`.

(5) One document per call. If the file contains more than one invoice, extract the first and add
   "multiple_invoices_in_one_file" to `notes`.

You are evaluated on how often you correctly return null, not only on how often you return a
value. A blank field is visible to a human. A fabricated one is not.

Adapt the field names and enums, keep the four structural choices, and add each new reason code only when a real document forces it. Codes invented in advance go unused and cost tokens forever. This pattern sits underneath most of our custom LLM applications work, and the mechanics of reading messy files are in extracting data from documents.