On this page
- Most extraction failures happen before the model sees anything
- Four document classes, four different pipelines
- The Four Gates of a Trusted Field
- Grounding: make every value point at the text it came from
- Confidence is an agreement score, not a number the model gives you
- Validation rules catch what the model cannot
- The fields that break every pipeline
- Routing, review and the arithmetic of a human queue
- Measuring extraction so you can tell improvement from regression
- A configuration worth starting from
Most extraction failures happen before the model sees anything
Reliable document data extraction is three problems stacked on each other: a parse that produces honest text, a field contract that forces every value to point back at the place it came from, and a routing rule that decides who checks it. The model sits in the middle and is rarely the weakest part. The dominant production failure is not a model misreading a digit. It is a model handed degraded, reordered or empty text that returns a perfectly well-formed value for a field that was never on the page.
Two examples make the point concrete. A naive text extractor run over a two-column contract reads across the page rather than down each column, so clause text from the left column interleaves with unrelated text from the right. Every sentence is grammatical and no sentence is real. Send that to a model with a schema and it will fill the schema. Second example: a scanned purchase order with no text layer returns an empty string. A permissive prompt over an empty string produces a complete invoice record, correct in shape, invented in every field.
Both failures share a signature. There is no error, no exception, no red log line. The record lands in your finance system looking exactly like the good ones. This is why extraction pipelines need verification that is structural rather than statistical, and why the first thing to build is not a better prompt.
- Parse firstThe text layer sets the ceiling on accuracy. No prompt recovers a field that the parser never produced or scrambled into the wrong reading order.
- Nulls are the hard partA language model completes. Absence has to be an explicit, named, exemplified output, or the model will supply something shaped like the answer.
- Confidence is computedA self-reported 0.95 tells you the completion was fluent. Agreement between independent signals tells you the value came off the page.
- Score per fieldDocument-level accuracy averages the field nobody cares about with the field that moves money. Report them apart or you are measuring nothing.
- Document data extraction
- Turning an unstructured or semi-structured document into named fields with known types, so a downstream system can act on the values without a person reading the page.
- Text layer
- The machine-readable characters embedded in or derived from a document. A digital PDF carries one already, a scan has none until an OCR engine creates it, and its quality bounds everything downstream.
- Span grounding
- Requiring the model to return the exact source substring and page for each extracted value, so a deterministic string match can prove the value existed in the document rather than in the model.
- Field-level confidence
- A score computed per field from independent signals such as span match, character-level OCR confidence, agreement across passes and rule outcome, used to decide whether a human sees that field.
- Straight-through processing rate
- The share of documents that complete end to end with no human touch. It is the single number that drives the cost of an extraction pipeline, because reviewer time usually costs more than model calls.
Four document classes, four different pipelines
Classify before you extract. Documents fall into four classes that fail in different ways and need different machinery, and a pipeline built for the average of all four is bad at each of them. Classification is cheap: file type, presence and density of a text layer, and a small classifier over the first page will separate them well enough to route.
| Class | Typical example | What the pipeline needs | Where it breaks |
|---|---|---|---|
| Digital, one fixed template | Invoices from a single ERP, bank statements from one bank | Positional rules and regular expressions. A model is optional and often slower and worse | Silently, on the morning the vendor upgrades their system and every coordinate shifts |
| Digital, many templates | Supplier invoices from four hundred vendors | Layout-aware parsing, then a schema-constrained model call per page or per section | Multi-page tables, wrapped line items, and totals that appear twice with different meanings |
| Scanned or photographed | Delivery notes, phone photos of receipts, faxed forms | Deskew and dewarp, OCR with per-word confidence, page images kept for review | Rotation, glare, fold shadows, stamps printed over the numbers you need |
| Handwritten or hybrid | Forms with handwritten annotations, ticked boxes, signatures | A vision model plus human review as the default path, not the exception | Everywhere. Treat any handwritten field as review-by-default regardless of score |
If your accuracy number pools all four populations, it will drift every time the mix of incoming documents changes and you will chase a model regression that never happened. Log the class on every record and report accuracy per class. The same discipline applies to vendors inside the multi-template class, because that is where template drift hides.
The Four Gates of a Trusted Field
A field earns the right to touch a downstream system by passing four gates in order. The ordering is deliberate and it is about cost: each gate is cheaper than the one after it, and each rejects a failure class the later gates cannot see. Running them out of order means paying for model calls on documents that were never legible and paying for human minutes on values that a string match could have rejected for nothing.
The Four Gates of a Trusted Field
Apply the gates per field, not per document. A single invoice routinely has ten fields through gate four and one field stuck at gate two, and the whole point is that the eleventh field is the only one a person needs to look at.
The page produced a text layer with plausible density for its area, and the page image is not blank, rotated or upside down. Test: characters per square inch against a floor you set from your own corpus, plus an orientation check. Cost: milliseconds, no model call. This gate exists because an empty parse is the input that produces the most confident nonsense, and rejecting the document here is far cheaper than detecting the fabrication later.
The value appears verbatim in the parsed text of the page the model claims it came from. Test: normalise whitespace and case, then check that the returned span is a substring of that page, and that the value is a substring of the span. Cost: a string match. A field that fails this gate is not low confidence, it is invented, and the only safe action is to discard the value and mark the field ungrounded.
The field satisfies its own format rule and any arithmetic that links it to its neighbours. Test: pattern and checksum for structured identifiers, sum of line items plus tax against the stated total within a rounding tolerance, dates inside a sane window relative to the received date. Cost: pure code. This gate catches the errors grounding cannot, because an OCR engine that reads 5 as S produces a value that really is on the page.
The routing decision, made by what the field does rather than by how the model felt. A field that triggers a payment, creates a legal obligation or overwrites a master record goes to a human whenever any earlier gate was marginal. A field that populates a search facet or a dashboard filter does not. Write the consequence class into the schema next to the type, because it is the only part of the design that a reviewer, a finance lead and an auditor will all agree on.
The gates also give you a clean vocabulary for incidents. When a bad record reaches production you ask which gate should have caught it, and the answer names the fix. Gate one means a parser or intake problem, gate two means the grounding check was skipped or too lenient, gate three means a missing rule, and gate four means the routing policy was wrong. None of those fixes is a prompt edit, which is where most teams start and stay.
Grounding: make every value point at the text it came from
The highest-value change you can make to an extraction prompt is to stop asking for values and start asking for evidence. Instead of a flat object of strings, require every field to return four things: the value, the exact source span it was copied from, the page that span sits on, and a reason when the field is absent. Then verify the span in code. That verification is deterministic, costs nothing, and catches the entire class of invented fields that no confidence score will.
# OUTPUT CONTRACT. Every field is an object, never a bare value.
# Copy this, swap the field names, keep the four keys.
{
"invoice_number": {"value": null, "span": null, "page": null, "absent_reason": null},
"invoice_date": {"value": null, "span": null, "page": null, "absent_reason": null},
"currency": {"value": null, "span": null, "page": null, "absent_reason": null},
"total_amount": {"value": null, "span": null, "page": null, "absent_reason": null},
"line_items": [
{"description": {...}, "quantity": {...}, "unit_price": {...}, "amount": {...}}
]
}
SYSTEM
You extract printed fields from one document. You do not summarise it and you
do not reason about the business behind it.
Rules, in priority order:
1. Copy, do not compose. Every value must appear word for word in the text you
were given. Keep the original characters, including the currency symbol and
the separators. Do not reformat, translate, correct spelling or convert
units. Normalisation happens in code, after you.
2. Show where it came from. "span" is the exact substring you copied from, extended by
at least three words on each side so it can be located. "page" is the page
that substring sits on.
3. Absence is an answer. If a field is not in the document, return value null,
span null, and set absent_reason to NOT_IN_DOCUMENT, ILLEGIBLE or AMBIGUOUS.
An empty field is correct and expected. A plausible guess is a defect.
4. Never do arithmetic. Do not add line items, derive tax from a total, or infer a
due date from payment terms. Return only what is printed on the page.
5. If two values compete for one field, return the one closest to its printed
label and set absent_reason to AMBIGUOUS alongside the value you chose.
6. Output the JSON object and nothing else.
USER
<page n="1">{page_1_text}</page>
<page n="2">{page_2_text}</page>
# AFTER THE CALL, IN CODE. This is the part teams skip.
#
# def norm(s): return re.sub(r"\s+", " ", s).strip().casefold()
#
# for name, f in fields.items():
# if not f.span or norm(f.span) not in norm(pages[f.page]):
# f.value, f.status = None, "UNGROUNDED"
# elif norm(f.value) not in norm(f.span):
# f.value, f.status = None, "SPAN_MISMATCH"
#
# An ungrounded field is not a low confidence field. It is an invented one,
# and the only safe thing to do with it is throw the value away.Spans make the output longer and the token bill higher. That is the trade, and it is usually worth it, because the alternative cost is a human reading the whole document. The span also gives your review interface somewhere to jump to, which is the difference between a reviewer spending twenty seconds and two minutes. Keep the page images and the parse output addressable by document id for as long as the record lives, because a span with no page behind it proves nothing in an audit.
Some fields legitimately do not appear on the page: a net amount that has to be computed, a normalised vendor id, a category. Do not let those pass through the same path. Mark them derived in the schema, compute them in code from grounded inputs, and record which fields they were derived from. A derived field that a model produced is the easiest place for a wrong number to enter a system unchallenged.
Confidence is an agreement score, not a number the model gives you
Asking a model how confident it is gives you a number that reflects fluency rather than correctness. Invented values are exactly the sort of output a model rates highly, because a well-formed invoice number is an easy completion. Build the score yourself out of signals that can disagree with each other.
The four signals worth combining
- Span match, binary. Normalised substring hit against the page text. A miss is not a low score, it is a rejection, and it should never be weighted into an average where a strong OCR score can outvote it.
- OCR character confidence over the span. Most engines expose per-word or per-character confidence. Take the minimum across the span rather than the mean, because a single broken digit is what ruins an amount and a mean hides it behind twenty clean characters.
- Two-pass agreement. Run the extraction a second time with the pages in a different order, or with a second model, and compare per field after normalisation. Disagreement on a field is the most predictive review signal available for the price of one more call. Apply it only to fields whose consequence justifies the spend.
- Rule outcome. A field that fails its own format, checksum or arithmetic rule scores zero regardless of the other three signals. Rules are deterministic and they outrank estimates.
Setting the threshold is an empirical exercise, not a judgement call. Score every field on a labelled set, sort by score, and walk down the list until the observed error rate crosses the tolerance you can defend for that field. That cut is your auto-accept threshold, and it will be different for a total amount than for a description string. Re-derive it after every change to the model, the parser or the normaliser, because all three move the distribution and none of them will tell you.
Validation rules catch what the model cannot
Rules are the cheapest quality in this pipeline. They are deterministic, they explain their own failures, and they catch errors that are genuinely on the page and therefore invisible to grounding. Write them as data rather than as scattered code, so a finance lead can read them and a change is a diff.
| Rule class | Example | What it catches that nothing else does |
|---|---|---|
| Format | Country-specific tax number pattern, ISO date after normalisation | Glyph confusions such as 0 against O and 5 against S in reference numbers |
| Checksum | IBAN modulo check, card and account check digits | Single-character OCR errors in identifiers that no pattern would reject |
| Arithmetic | Line items plus tax against the stated total, within a rounding tolerance | A missed line item, a wrapped row read twice, a decimal separator misread |
| Cross-document | Invoice matched to a purchase order and a goods receipt | A correctly extracted document that should not be paid at all |
| Temporal | Document date inside a sane window relative to the date received | Two-digit year misreads and a date field that picked up the print timestamp |
| Referential | Vendor name resolved against the vendor master before the record is written | A near-match new vendor created by a typo, which is also a fraud surface |
# Deterministic. Runs after extraction, before any human is involved.
# Every rule names what it touches and what happens when it fails.
normalise:
amounts:
strip: ["currency symbols", "thin and non-breaking spaces"]
locale_from: vendor.country # 1.234,56 and 1,234.56 are not guessable
negatives: ["(1,250.00)", "1250.00-"] # both mean minus, both parse to plus
dates:
order_from: vendor.country # 03/04/2026 is two different real days
on_ambiguous: flag # day <= 12 and no known locale: review
rules:
- id: total_matches_lines
when: line_items.length > 0
check: abs(sum(line_items.amount) + tax_amount - total_amount) <= 0.02
on_fail: route_to_review
note: computed in code, never asked of the model
- id: date_window
check: doc_date <= received_date and doc_date > received_date - 400d
on_fail: route_to_review
- id: tax_id_format
check: match(vendor_tax_id, tax_pattern[vendor.country])
on_fail: clear_field_and_review
note: a format failure is usually a glyph error, not a different vendor
- id: known_vendor
check: vendor_name in vendor_master or fuzzy(vendor_name, vendor_master) > 0.92
on_fail: route_to_review
- id: not_a_duplicate
check: sha256(file_bytes) not in seen
and (vendor_id, doc_number, total_amount) not in ledger
on_fail: reject_as_duplicate
note: the same document arrives twice by two channels and hashes differently
routing:
auto_accept: all_gates_pass and agreement == 1.0 and total_amount < 2000
review: any_rule_failed or agreement < 1.0 or handwritten == true
reject: text_density < floor or page_count == 0Extract the line items and the stated total as separate grounded fields, then compute the sum in code and compare. When they disagree you have learned something real: a missed row, a duplicated row, or a separator misread. If the model does the sum, the disagreement is resolved silently in its head and the signal is gone. This is the same discipline described in structured output from LLMs, applied to documents.
The fields that break every pipeline
The list below is not exotic. These are the recurring defects in production document data extraction, and every one of them is fixed in code and metadata rather than in the prompt.
- Ambiguous dates. 03/04/2026 has two valid readings and no amount of prompting resolves it, because the information is not in the document. Carry the vendor country as metadata, resolve in code, and flag for review when the day is twelve or lower and the locale is unknown.
- Negative numbers. Credit notes print negatives as (1,250.00) or as 1250.00 with a trailing minus. A naive float parse turns a credit into a charge, the arithmetic rule still passes if the total was misread the same way, and the error surfaces at month end.
- Decimal separators. 1.234,56 and 1,234.56 differ by three orders of magnitude. This is locale, never inference.
- Multi-page tables. The header sits on page one and rows continue on page four with no header at all. Extract per page, then stitch with a continuation rule based on column count and horizontal position, not on the model's opinion about whether a table continues.
- Wrapped line items. One item wraps across three visual rows and becomes three items, two of them with empty quantity and price. Merge any row whose quantity and amount are both empty into the row above it before validation runs.
- Checkboxes. Ticks, crosses, filled circles and scratched-out boxes all mean different things to different people. Extract three states rather than a boolean: checked, unchecked, indeterminate. Vision models read these poorly and rate themselves highly while doing it.
- Stamps and overprints. A paid stamp across the total merges glyphs, and the amount comes out plausible and wrong. Keep the page crop so a reviewer can see what the parser saw.
- Similar glyphs in identifiers. 0 and O, 1 and l, 5 and S, 8 and B. A checksum rule catches these. A confidence score usually does not, because the character was read cleanly, just as the wrong character.
- Missing currency. An amount with no symbol anywhere on the page. The currency belongs in the vendor record, not in the model's assumption about where you are.
- Duplicates. The same invoice arriving by email and through a portal, minutes apart, hashing differently because one was re-rendered on the way. Hash the bytes and also enforce a business key, using the pattern in idempotency in automation.
- Rotation and skew. A page ninety degrees off produces an empty or garbled text layer, which is the input that produces the most confident nonsense.
Of every failure listed above, the blank page is the one that produces a complete, well-formed, entirely fictional record with no warning attached. Put a hard floor on extracted characters relative to page area, measured on your own corpus, and fail the document before the model call rather than after it. This single check removes more bad records than any prompt improvement you will make this quarter.
Routing, review and the arithmetic of a human queue
Extraction economics are dominated by the review queue rather than the model bill, and that ratio decides where engineering time should go. Run the numbers on your own volumes first. In most configurations, moving the straight-through rate by fifteen points is worth more than any token saving available to you, which is why an extra verification call that lifts auto-accept pays for itself.
The prices here are stand-ins so the arithmetic runs. They are not quotes and they are not current figures. Replace the token price with your provider's published number on the day you read this, and replace the reviewer rate with your own loaded cost. Then change the straight-through percentage and watch which lever actually moves the total.
Two consequences fall out of that arithmetic. A second extraction pass for agreement scoring doubles a small number, so it is affordable as long as it lifts the straight-through rate at all. And the review interface deserves real engineering attention, because minutes per document multiplies against every routed document forever. Show the page crop beside the field, prefill the value, let the reviewer confirm with one key, and store every correction as a labelled example. That queue is designed in human in the loop design.
Measuring extraction so you can tell improvement from regression
Document pipelines are unusually easy to measure badly. The two common mistakes are scoring whole documents rather than fields, and scoring only the documents where the field is present.
- Build a golden set from real production traffic, stratified
Sample per class and per high-volume vendor, then deliberately add a slice of documents where a field is legitimately absent and a slice of the ugly ones people complain about. Label by hand, twice, by two people. Keep the disagreements attached to the label, because the fields your labellers argue about are the fields your model will get wrong.
- Freeze a normaliser and version it with the set
Comparison happens after normalisation: case, whitespace, currency symbols, thousands separators, dates to ISO. Write it once and version it alongside the golden set. A silent normaliser change moves every score at once and looks exactly like a model improvement, which is how teams celebrate a rounding change.
- Report per field, and report nulls separately
For each field, track precision and recall over non-null values, and track null behaviour on its own: how often the system correctly says absent, and how often it invents a value where none existed. A pooled accuracy number hides the invention rate, and invention is the failure that costs money.
- Run the full set on every change, and diff at field level
Prompt edits, model versions, parser upgrades and normaliser changes all go through the same gate. Read the fields that moved rather than the headline. A change that lifts the average while dropping total_amount by two points is a regression wearing a nice hat. The general approach is in writing evals for LLM systems.
- Track two production metrics, not one
The straight-through rate tells you what the pipeline costs. The review correction rate, meaning the share of reviewed fields a human actually changed, tells you whether routing is sending the right work to people. A correction rate near zero means you are paying humans to rubber-stamp and your thresholds are too cautious.
- Re-label a rolling sample weekly, per source
Template drift is the quiet killer. A vendor changes their layout without telling anyone and one source degrades while the global number barely moves. Alert on per-vendor review rate rather than the pooled rate, and when it spikes, look at the parser output before you touch the prompt.
A twelve-field invoice with one wrong field scores zero, the same as one with twelve wrong fields. The number moves for reasons you cannot attribute and it cannot tell you which change helped. Keep it as a reporting line for stakeholders if you must, and never make a decision with it.
A configuration worth starting from
This is a sane default for a mixed corpus of digital and scanned business documents. It is deliberately conservative on routing, because it is far easier to widen auto-accept once you have labelled data than to explain a month of wrong payments.
Build the gates and the golden set in the first week, before the prompt is any good. The prompt improves on its own once you can see which fields fail and why, and the machinery around it is what keeps the pipeline honest when the documents change. If you want this built and measured as a system rather than a script, that is the shape of our document processing work.
Reliable document data extraction is three problems, not one: a parse that produces honest text, a field contract that makes every value point back at the span it came from, and a routing rule that sends work to a human by consequence rather than by a confidence number the model invented.
Questions readers ask next
Do I still need OCR if all my PDFs are digital?
Can a vision model replace the OCR and parsing step entirely?
Should I fine-tune a model for document data extraction?
How accurate can an AI document extraction pipeline actually get?
What do I do about handwriting?
How do I stop the same document being processed twice?
How long before an extraction pipeline is worth trusting without review?
ChatGPTalker. "Extracting Data from Messy Documents Reliably." chatgptalker.com, 2026-08-26. https://chatgptalker.com/guides/extracting-data-from-documents/