On this page
- What document processing is
- Who it is for, and who it is not for
- What we actually build
- How it works technically
- Confidence, and what a score actually means
- The build process stage by stage
- What you get at handover
- Where document processing projects go wrong
- What it costs to run once live
- How to tell whether you need this
- How to start
What document processing is
Document processing is a pipeline that turns invoices, contracts, forms and scans into structured records your systems can act on. It classifies the document, extracts named fields with a confidence score and the page location each value came from, checks the result against arithmetic and your own master data, then routes it: post automatically, send to a review queue, or refuse to answer. The extraction is the easy half. The half that decides whether it works is knowing which results to trust.
The old approach was template matching. You drew boxes on a supplier's invoice layout and the system read those coordinates, which worked until the supplier moved their logo. Modern extraction reads the document rather than its coordinates, so it handles layouts it has never seen. That is a real change, and it introduces a new problem: a system that always produces an answer, including for the documents it has misread.
So the engineering effort concentrates on the gates rather than the reading. A per-field confidence score, calibrated against documents you have labelled by hand. Arithmetic checks the document must pass. A comparison against your supplier master. And an abstain path, because in any workflow that posts to a ledger, no answer is cheaper than a wrong one.
- Per fieldConfidence, provenance and the verbatim characters are stored for every field, not once per document. Document-level accuracy hides the field that matters.
- Abstain allowedThe system is permitted to say it does not know. A refusal costs one review. A confident wrong total costs a payment and an investigation.
- Calibrated, not guessedThresholds come from a labelled set of your own documents. A score copied from a vendor demo describes their documents, not yours.
- Review is a productThe queue is designed for speed: only flagged fields, the source region highlighted, keyboard first. If reviewing is slower than typing, the project is negative.
Who it is for, and who it is not for
This is for teams where people key structured facts out of documents every day and the volume is steady. The economics come from volume and repetition, not from the cleverness of the extraction.
| Your situation | Verdict | Why |
|---|---|---|
| Thousands of documents a month in a handful of recurring shapes | Good fit | Volume amortises the calibration work, and recurring shapes let you monitor accuracy per template. |
| A shared inbox where invoices or forms arrive as attachments | Good fit | Intake, classification and routing are the same build, and the inbox is already the bottleneck. |
| A few hundred documents a month, all different | Not a fit | Calibration and review design cost the same, and there is not enough volume to repay them. |
| Documents that decide something legal or clinical on their own | Not a fit | Extraction can assist the reader. It should not be the only thing standing between the document and the decision. |
| No labelled examples and no appetite to create any | Fix first | Without labelled documents nobody can measure accuracy, so nobody can tell whether the build worked. |
| Handwriting, photos taken at angles, scans of scans | Careful | Achievable, but the pre-processing tier becomes most of the project. Price it as its own piece of work. |
Three things you need before the build
- A sample of real documents, including the ugly ones. Not the clean examples from the finance system, the ones from the shared inbox with three invoices in one PDF and a photo taken on a phone.
- A definition of correct, per field, in writing. Is a date correct if the format differs? Is a supplier name correct if it omits the legal suffix? These arguments are cheaper before the build than during acceptance.
- A person who currently does the work and will spend a day labelling with us. Their edge cases are the specification, and they know which fields nobody actually reads.
Who should not buy this
- Anyone who wants a single accuracy number promised in advance. That number depends on your documents and cannot be known before we see them.
- Teams whose real problem is that documents arrive late, not that they take time to key. Chasing is a different automation.
- Organisations that cannot store documents outside a specific environment and have not checked whether the intended provider can operate inside it.
- Anyone planning to remove the reviewer entirely on day one. The queue shrinks with evidence, and the evidence takes months to accumulate.
What we actually build
Six components. Two of them involve a model. The other four are where the accuracy actually comes from.
Intake and pre-processing
Documents arrive from an inbox, a portal, a folder or an API. Then the unglamorous tier: de-skew and de-noise scans, split multi-document PDFs, drop blank and duplicate pages, detect password protection, hash the file so the same invoice arriving twice is recognised. A large share of what people call model error is a splitting failure two steps upstream.
Classification and routing
What is this document, and which extraction contract applies. Invoice, credit note, statement, delivery note, contract, or unknown. Unknown is a legitimate and useful answer, and routing it to a human beats forcing it through the invoice extractor and receiving a plausible set of fields.
Extraction against a strict contract
A JSON schema defines exactly what may come back: field names, types, nullability, and for every field a value, the verbatim characters on the page, a confidence score and the page and bounding box it came from. Structured output enforcement stops shape errors. It does not stop content errors, which is what the next two components are for. See structured output from LLMs for the mechanics.
The validation tier
Rules that are cheap, deterministic and unforgiving. Line items sum to the subtotal. Subtotal plus tax equals the total. The date is a real date and within a sensible window. The currency is one you trade in. The supplier resolves to a record in your master data and the bank details match the ones on file. This tier catches errors no confidence score will.
The review queue
Built as a product, not an admin page. Only flagged fields are shown, each with the source region highlighted on the page image. Keyboard-first, one document at a time, corrections written back as labelled training and evaluation data. Every correction is a free label, and a queue that does not capture them is throwing away the most valuable output of the system.
The evaluation harness
A frozen labelled set of real documents, scored per field, run on every prompt change, model change and schema change. It is the only way to distinguish an improvement from a regression when the underlying model is not deterministic. More on building one in building a golden dataset.
Template systems fail loudly and model extraction fails quietly, which is the entire reason the gating tier exists. If your documents come from twenty suppliers who never change their layouts, a template system may still be the right answer, and it will be cheaper to run.
How it works technically
Ingest, normalise, classify, extract, validate, score, route, learn. The design decisions that matter are how pages reach the model, what the contract permits, and where the thresholds sit.
Text layer or image, and why it matters
A born-digital PDF has an embedded text layer that can be read directly, cheaply and exactly. A scan has none, so pages go to the model as images or through OCR first. Image input is billed differently from text and usually costs considerably more per page, so check your provider's current image token accounting before estimating anything. Many pipelines are cheaper simply because they detect a usable text layer and take it, falling back to images only when they have to.
Chunking long documents without losing the table
Contracts and statements exceed a comfortable single call. Splitting at a page boundary cuts tables in half and separates a clause from its definitions. Split on structure where the document has any, overlap the boundaries, and give each chunk enough context to be interpreted alone. Then reconcile the results, because the same field appearing twice with different values is information rather than a nuisance.
Routing on the score
Three lanes. Above the auto-post threshold on every required field, and passing every validation rule, the record posts. Below it, the document goes to the queue with only the failing fields flagged. Where the document cannot be read or classified at all, the system abstains and a human sees the original. Thresholds are set per field from labelled data, and they are not the same number: an invoice total deserves a stricter bar than a purchase order reference.
{
"name": "invoice_extraction_v4",
"strict": true,
"schema": {
"type": "object",
"additionalProperties": false,
"required": ["document_type", "abstained", "fields", "line_items"],
"properties": {
"document_type": { "enum": ["invoice", "credit_note", "statement", "unknown"] },
"abstained": {
"type": "boolean",
"description": "true when the page is unreadable or not a known type. Abstaining always beats guessing."
},
"fields": {
"type": "object",
"additionalProperties": false,
"required": ["invoice_number", "invoice_date", "currency", "total_amount", "supplier_name"],
"properties": {
"invoice_number": { "$ref": "#/$defs/text_field" },
"invoice_date": { "$ref": "#/$defs/text_field" },
"currency": { "$ref": "#/$defs/text_field" },
"total_amount": { "$ref": "#/$defs/number_field" },
"supplier_name": { "$ref": "#/$defs/text_field" }
}
},
"line_items": {
"type": "array",
"items": {
"type": "object",
"additionalProperties": false,
"required": ["description", "quantity", "unit_price", "line_total"],
"properties": {
"description": { "$ref": "#/$defs/text_field" },
"quantity": { "$ref": "#/$defs/number_field" },
"unit_price": { "$ref": "#/$defs/number_field" },
"line_total": { "$ref": "#/$defs/number_field" }
}
}
}
},
"$defs": {
"provenance": {
"type": "object",
"required": ["page", "bbox"],
"properties": {
"page": { "type": "integer", "minimum": 1 },
"bbox": { "type": "array", "minItems": 4, "maxItems": 4, "items": { "type": "number" } }
}
},
"text_field": {
"type": "object",
"additionalProperties": false,
"required": ["value", "verbatim", "confidence", "provenance"],
"properties": {
"value": { "type": ["string", "null"] },
"verbatim": { "type": ["string", "null"], "description": "exact characters on the page, before normalisation" },
"confidence": { "type": "number", "minimum": 0, "maximum": 1 },
"provenance": { "$ref": "#/$defs/provenance" }
}
},
"number_field": {
"type": "object",
"additionalProperties": false,
"required": ["value", "verbatim", "confidence", "provenance"],
"properties": {
"value": { "type": ["number", "null"] },
"verbatim": { "type": ["string", "null"] },
"confidence": { "type": "number", "minimum": 0, "maximum": 1 },
"provenance": { "$ref": "#/$defs/provenance" }
}
}
}
}
}Three details in that schema do the heavy lifting. verbatim keeps the raw characters next to the normalised value, so a reviewer can see that 1/3 was read as a date and not a quantity. provenance forces every value to name a page and a region, which means a field with no source region on the page was invented. And abstained gives the model a legitimate way out, which it will not take unless the schema and the instructions permit it.
- Straight-through rate
- The share of documents that clear every gate and post without a human touching them. It is the number that decides whether the build pays back, and it is only meaningful once measured on your own documents.
- Provenance
- The page and bounding box a value was read from, stored alongside the value. Without it a reviewer must hunt through the document to check a field, which is most of the cost of reviewing.
- Abstain
- A deliberate output meaning the system will not answer. In any workflow that posts to a ledger, an abstention costs one review while a confident wrong answer costs a payment and an investigation.
- Golden set
- A frozen, hand-labelled sample of real documents used to score accuracy before and after every change. Without one, you cannot tell an improvement from a regression when the model is not deterministic.
- Calibration
- The process of mapping raw confidence scores onto observed correctness using labelled data, so that a threshold means something. Uncalibrated scores rank documents. They do not measure the chance of being right.
Confidence, and what a score actually means
A raw confidence number out of a model is a ranking signal, not a probability of correctness. It is high on fluent mistakes, which are exactly the mistakes that get through. So we build the score from four independent signals, weight them per field and calibrate the result against documents somebody labelled by hand.
The ChatGPTalker Four-Signal Confidence Score
Four signals that fail in different ways, combined per field and calibrated on your own labelled set. The point of using four is that a wrong value rarely fools all of them at once.
Token probabilities or a self-rated score. A weak prior and dangerous alone, because a clean wrong date reads exactly like a clean right one. Useful mainly for ranking within a field, never as the gate by itself.
Line items sum to the subtotal, subtotal plus tax equals the total, quantity times unit price matches the line. A document that fails its own arithmetic is wrong somewhere even when every individual field looks plausible.
Does the supplier resolve to your master data, does the purchase order exist, does the bank account match the one on file. This signal catches fraud as well as error, which is why it is worth building even at low volume.
Every value names the page and region it came from. A value with no source region was invented. A total pulled from the header block of a page where totals never appear is suspect even at high model confidence.
Weight the signals per field and set thresholds from labelled data rather than intuition, because fields differ in how much a mistake costs. Recalibrate when a template or a model version changes, and treat calibration drift as a release blocker.
A field marked 0.94 does not mean a 94 percent chance of being right unless somebody measured that against ground truth. Reporting it as a probability is how a number invented for routing ends up in a control document, where it will eventually be quoted back to you in an audit.
The build process stage by stage
Six stages. Two of them are labelling and measurement, which is the part clients want to compress and the part that determines whether anyone believes the result.
- Document survey and sampling
A representative sample from where documents really arrive, not the tidy archive. Count the shapes, languages, scanned proportion, multi-document files and duplicates. This sample decides the shape of the whole build.
- Labelling and the golden set
A few hundred documents labelled field by field with the person who does the work today, disagreements resolved in writing. This produces both the evaluation set and the definition of correct that acceptance will be measured against.
- Pre-processing and classification
Splitting, de-skewing, deduplication, text-layer detection, document typing. Measured on its own, because a splitting error looks like an extraction error and gets debugged in the wrong place for a week.
- Extraction contract and first scores
Schema, extraction, validation rules and the first confidence model, scored against the golden set per field. The first honest accuracy table appears here, and it is usually lower than the demo suggested and higher than the sceptic expected.
- Calibration and the review queue
Thresholds set per field from labelled data, then the queue built around the fields that actually get flagged and timed with a real reviewer. If a review takes longer than keying from scratch, the design changes.
- Shadow running and handover
The pipeline runs alongside the existing process, both outputs compared daily, disagreements reviewed. Only then does anything post automatically, and it starts with the field and document type where the evidence is strongest.
Every extraction tool looks excellent on a born-digital invoice from a large supplier. Insist that any evaluation, ours included, runs on a sample drawn from your real intake, including the phone photographs and the multi-invoice PDFs. The gap between those two samples is the whole project.
What you get at handover
The pipeline, the queue, the labelled data and the harness all sit in your environment. The labelled set matters most in the long run, because it is the asset that lets you switch model, provider or vendor later without losing your ability to measure.
We also hand over the list of document types the system is deliberately not allowed to handle, and why. That page prevents somebody quietly pointing a new intake at the pipeline eighteen months later.
Where document processing projects go wrong
Six failure modes, and only one of them is about the model being bad at reading.
Accuracy measured per document rather than per field
A headline accuracy figure is usually an average across fields, carried by the easy ones. The total, the date and the supplier account for nearly all the business risk, and they are frequently the hardest. Always demand a per-field table, and read the row for the field that would cost you money.
No golden set, so nothing can be compared
Without frozen labelled data, a prompt change, a model version change and a new supplier template are indistinguishable from each other. Teams end up arguing from anecdotes: somebody remembers a bad week. Build the labelled set before the pipeline, even though it feels like the boring part.
The review queue that is slower than typing
If a reviewer opens the whole document, hunts for the value and retypes it, you have added a step rather than removed one. Show only the flagged fields, highlight the source region, make it keyboard driven, and time it against the old process with a stopwatch before you call it finished.
Documents that are not what the pipeline assumes
Three invoices in one PDF. A statement attached to a covering email where the real invoice sits in the signature block. A 400-page bundle where four pages matter. A photograph at an angle with a thumb over the total. Most accuracy complaints trace back to intake, and intake is where the cheapest wins are.
Silent template drift
A supplier redesigns their invoice. Accuracy for that supplier falls while the overall average barely moves, because that supplier is a small share of volume. Monitor accuracy per template, alert on a drop within a segment, and treat the global average as a vanity metric.
Personal data spreading through the pipeline
Documents contain personal data, and a pipeline copies them into storage, logs, prompts, caches and possibly a provider's systems. Decide retention, redaction and region at design time rather than at the security review, and record what is sent where.
A template system that fails returns nothing and somebody notices. Model extraction returns a well-formed date, a real-looking invoice number and a total that is off by one digit. It passes every shape check and posts. Every gate on this page exists because of that specific failure, and a vendor who does not talk about it has not run one of these in production.
What it costs to run once live
Four lines. Review labour usually dominates for the first year, which is why the queue design matters more than the model choice.
| Cost line | What drives it | How it scales |
|---|---|---|
| Model and OCR | Pages per document and whether pages go as text or images. Image input is billed differently and usually costs more. | With pages, not documents. A ten-page contract is not a one-page invoice. |
| Review labour | Straight-through rate and seconds per review. Usually the largest line early on. | Falls as thresholds are tuned and as corrections improve the system. |
| Storage and retention | Original files, page images, extracted records, audit trail. | With volume and the retention period your policy requires. |
| Maintenance | New document types, template drift, model version changes, recalibration. | With the number of document families and how often your suppliers change. |
The arithmetic worth running is not the token cost. It is whether the review queue is genuinely faster than what you do now. Put your own numbers in below. Take the straight-through rate from a measured pilot on your own documents, never from a vendor's figure, including any figure quoted at you by us before we have seen your intake.
The straight-through rate here is your assumption, not a prediction. Measure it on a labelled sample of your own documents before you rely on this arithmetic.
Run it twice: once with the rate you hope for, once with that rate halved and the review time doubled, which is the honest pessimistic case for a first year. If the pessimistic case is still positive, the project is sound. If only the optimistic case works, you are buying a bet rather than a system.
Page count and image handling drive model cost far more than model choice does, and providers change image token accounting without much notice. Detect the text layer, use it when it exists, and keep the arithmetic in a spreadsheet you can rerun when prices move rather than in a slide.
How to tell whether you need this
The test is not whether documents annoy you. It is whether the same fields are copied out of the same shapes of document, at volume, by people who could be doing something else, and whether an error in those fields is currently caught by somebody rather than by an auditor.
That last one is worth sitting with. Manual keying has an error rate too, and it is rarely measured. A pilot that produces the first honest accuracy figure your organisation has ever had is valuable even if you decide to keep doing the work by hand. For finance-specific workflows, the approvals and reconciliation layer sits in finance automation.
Take one hundred real documents, have two people key them independently, and compare. You will learn your true manual error rate, which fields people disagree about, and how long it actually takes. That day of work makes every subsequent vendor conversation shorter and more honest.
How to start
It starts with documents, not with a call about capabilities. We would rather look at fifty of your real files than describe what is possible in general, because the answer is entirely determined by what your intake actually contains.
- Send a sample from where documents really arrive, including the difficult ones, and the list of fields you need out of them.
- We survey the sample: shapes, languages, scan quality, multi-document files, duplicates, and the fields that will be hard.
- You get a labelled pilot on a subset with a per-field accuracy table, or a recommendation that this is not worth building yet.
- If it proceeds, the first document type goes end to end with the queue in place before a second type is added.
The measurement discipline underneath all of this is the same one used for any non-deterministic system, and it is covered in evaluation and guardrails. The extraction techniques themselves are set out in extracting data from messy documents.
ChatGPTalker. Document Processing With Confidence Scores and a Review Queue. chatgptalker.com/services/document-processing/