Business process automation, rebuilt end to end

What business process automation is

A process is not a list of tasks. It is a list of tasks plus the gaps between them, and the gaps are where the calendar goes. A supplier invoice can take four minutes of human effort and eleven days to clear, because it waits in a shared inbox, then in an approval queue, then in a batch job that runs on Thursdays. Task automation attacks the four minutes. Process automation attacks the eleven days.

If the process has not been checked against real timestamps yet, start with mapping a process before you automate it. A rebuild priced from an unchecked diagram is priced from fiction.

Terms used precisely on this page

Business process automation

The rebuild of an entire process so software moves work between steps, applies the rules, and routes only genuine exceptions to a person. The unit of work is the process, not the task.

Straight-through processing rate

The share of items travelling from trigger to final record with no human touch. Everything it excludes carries a reason code.

Exception

An item the system decided it could not finish alone. A designed exception queues with a machine-readable reason. An undesigned exception is a failure that looks like silence.

Idempotency

Running a step twice produces the same result as running it once. Without it, every retry risks a duplicate payment or a duplicate ledger entry.

Who it is for, and who it is not for

This is for processes that run often, have a definable finish line, and pass through more than two pairs of hands. It fits badly where volume is low, where every item is genuinely unique, or where the rules are still being argued about.

The handoff census, and why elapsed time is the real number

Before designing anything we count custody changes, not steps. It takes two days, it is the cheapest work in the project, and it usually changes the design.

The census produces the baseline as a side effect, which matters because a rebuild with no baseline can never be proven to have worked. The baseline you must take before any automation covers what to record so it stays comparable.

What we actually build

Six components, in every build. Names change between clients, responsibilities do not, because each absorbs a specific class of failure.

The intake layer

Normalises the trigger, whatever form it arrives in: email, webhook, form, a file on shared storage, a scheduled poll. It deduplicates on a natural key, issues an item id, and stores the raw payload immutably. Everything downstream refers to the item id, so a resend does not create a second run.

The orchestrator

A durable state machine with named states, allowed transitions and a maximum age per state. It persists to a database rather than memory, so a deployment mid-run resumes instead of losing the item. We use a workflow engine rather than a chain of queue consumers, because "where is item 4471 and how long has it been there" must be answerable with one query.

The decision units

Small, single-purpose, independently testable. Deterministic rules live in code, because a rule that can be written as code should never be handed to a model. Model calls are reserved for unstructured input or genuinely fuzzy judgement, and each returns JSON validated against a schema, with a confidence value and a reason string.

The system-of-record adapters

One module per external system, and the only components permitted to write. Each owns its idempotency keys, retry policy, rate limits and dead letter path. Centralising writes is the difference between a bug that produces one duplicate invoice and a bug that produces four hundred.

The exception console

A real interface, not a channel in a chat tool. It shows the item, the reason code, the evidence used, and two actions: approve as it stands, or correct and return. Corrections are captured as labelled examples, so the queue that costs money in month one shrinks it by month four.

The evidence store

Every input, model prompt and response, rule outcome, write and confirmation, keyed by item id, with an agreed retention policy. Teams cut this in week three to save time and rebuild it under pressure in month five, after the first argument with a customer about what the system did.

How it works technically

The architecture is a durable state machine with pluggable decision units and a single write path. Nine things happen to an item, in this order, and every one is observable.

1. The trigger arrives. Intake normalises it, hashes a natural key, checks it against recent runs, and either drops a duplicate or issues an item id.
2. The orchestrator creates a run record in the first state and stores the raw payload. Nothing has touched a system of record yet.
3. Classification runs. Deterministic checks go first because they are free and certain; a model call handles what is left, validated against a JSON schema.
4. Enrichment fetches what the decision needs on read-only credentials. A failed enrichment leaves the item waiting rather than proceeding with a hole in its inputs.
5. The decision executes: rules first, model second, then a threshold check on confidence. Below threshold, the item transitions to review rather than to a write.
6. The adapter writes, using an idempotency key derived from the item id and the state name, so a retried write lands once.
7. The adapter re-reads what it wrote and compares it against what it intended. This catches the silent partial write, the failure producing the worst data and the fewest alerts.
8. The item reaches a terminal state, completed or exception, both carrying reason codes. There is no third outcome and no state meaning nothing in particular.
9. Timers run underneath all of it. Every non-terminal state has a maximum age, and exceeding it alerts with the item, the state and the age. An item stuck six hours is a bug even when nothing threw an error.

None of that is bespoke glue. The process is declared once in a specification file, and the orchestrator, the runtime dashboard and the documentation are generated from it. Copy the shape below as a starting point.

{
  "process": "supplier_invoice_to_posting",
  "version": 7,
  "idempotency_key": "{item_id}:{state}",
  "states": [
    {
      "name": "RECEIVED",
      "on_enter": "extract_fields",
      "max_age_minutes": 15,
      "next": { "ok": "CLASSIFIED", "low_confidence": "REVIEW", "error": "EXCEPTION" }
    },
    {
      "name": "CLASSIFIED",
      "on_enter": "match_purchase_order",
      "max_age_minutes": 60,
      "next": { "matched": "APPROVED", "unmatched": "REVIEW", "error": "EXCEPTION" }
    },
    {
      "name": "REVIEW",
      "queue": "ap_exceptions",
      "requires_human": true,
      "max_age_minutes": 1440,
      "next": { "resolved": "APPROVED", "rejected": "CLOSED_NO_ACTION" }
    },
    {
      "name": "APPROVED",
      "on_enter": "post_to_ledger",
      "write": true,
      "verify_after_write": true,
      "max_age_minutes": 30,
      "next": { "ok": "COMPLETED", "error": "EXCEPTION" }
    }
  ],
  "decisions": {
    "extract_fields": {
      "kind": "model",
      "schema": "schemas/invoice_fields.json",
      "confidence_field": "confidence",
      "threshold": 0.90,
      "on_below_threshold": "low_confidence"
    },
    "match_purchase_order": { "kind": "rules", "module": "rules/po_match" },
    "post_to_ledger": {
      "kind": "adapter",
      "system": "erp",
      "retries": 3,
      "backoff": "exponential",
      "dead_letter": "ap_dead_letter"
    }
  },
  "exception_reasons": [
    "missing_supplier",
    "po_not_found",
    "amount_mismatch",
    "duplicate_invoice",
    "erp_write_failed",
    "state_timeout"
  ],
  "evidence": { "store_raw_payload": true, "store_model_io": true, "retention_days": 400 }
}

Two details carry more weight than they look. The verify_after_write flag makes the confirmation read a declared property rather than something a developer remembers. And exception_reasons is a closed list, so an item cannot fail for a reason nobody named in advance.

The build process, stage by stage

1. Process captureWeek 1 to 2

   We watch people run the process rather than asking them to describe it. Description gives the intended path. Watching gives the workarounds, and the workarounds are where the exception rules come from. Output is a written definition the team signs.

2. Baseline and handoff censusWeek 2

   Volume, elapsed time, touch time, rework rate, error rate, and handoffs ranked by waiting caused, recorded so they can be measured identically later. Skip it and you can improve the process without ever proving it.

3. Specification and sign-offWeek 2 to 3

   States, decision units, confidence thresholds, exception reason codes, and a deliberately modest straight-through target for release one. A first release aiming at the whole distribution spends its budget on the long tail and ships late.

4. Build the spineWeek 3 to 6

   Intake, orchestrator, adapters, evidence store and console wired end to end with decision units stubbed. Real items run through the spine before any model is involved, because most hard bugs are integration bugs and they are easier to find with nothing non-deterministic in the trace.

5. Decision units and evaluationWeek 5 to 8

   Rules written and unit tested. Prompts built against labelled real historical items, not invented examples. Thresholds chosen by plotting precision against threshold on that set, not by picking a number that sounds careful.

6. Shadow runWeek 7 to 10

   The system processes live items alongside the humans and writes nowhere. Every disagreement is logged and reviewed. Two weeks minimum, longer where the process has a monthly cycle, because a system that has never seen a month end has not been tested.

7. Cutover by cohortWeek 9 to 12

   One supplier, one region, one item type at a time, with the old path closed for that cohort rather than left open beside it. Cohort cutover gives a clean rollback and an honest comparison. Big bang gives neither.

8. Exception burn-downWeek 12 to 14

   Weekly, take the top three reason codes by volume and fix the cause. The distribution is steep, so three fixes move the straight-through rate more than twenty small improvements. Skipping this is why automations plateau.

Two stages surprise people. Watching instead of interviewing always surfaces something that was in no document. And running the spine with stubbed decisions feels wasteful until the week a broken adapter is diagnosed in ten minutes.

What you get at handover

Handover is a defined event with a defined pack, not a gradual fading out. Everything below is yours, in your repository, under your accounts, and none of it depends on us staying available.

Where these projects go wrong

The failures are consistent and almost none of them are technical. Six we see repeatedly, in rough order of damage done.

The process was never actually agreed

Three people describe the process three ways and everyone assumes the differences are details. The build encodes one version, the other two turn up in the exception queue, and the people holding them conclude the system is broken. Settle it in writing before the specification.

The exception rate was estimated rather than measured

A team guesses that a small fraction of items will need review. In month one the real figure is several times that, because the model has never seen the strange third of the distribution. The console is staffed for the guess, the queue grows, people revert to the old path, and the system dies of neglect.

One write path was left open

The old manual route stays available because somebody was nervous about closing it. Under pressure people use it, the two paths write different state, and reconciliation becomes a permanent job nobody budgeted for. A cohort with two open paths is not a cohort, it is a preference.

Confidence thresholds were set by feel

Somebody picks a number that sounds careful. Either almost everything routes to review and the automation delivers nothing, or wrong items pass silently carrying the system's authority. The fix takes an afternoon: label two hundred real items, plot precision against the threshold, and choose knowing what you are trading.

Nobody owned it after go-live

The build team leaves, operations believes it inherited a finished thing, and nobody has authority to change a rule. Six weeks later an upstream system adds a required field and the response is a chat message rather than a fix. Who owns the automation after launch is worth reading before the handover meeting.

Success was measured only on the automated path

The report shows completed items and their impressive cycle times, and excludes everything that went to review. That number can improve while the process gets slower overall, because the exception tail is where the elapsed time hides.

What it costs to run once live

Four running cost lines: model tokens, infrastructure, human review of exceptions, and maintenance. Human review is usually the largest for the first quarter and model tokens are smaller than people expect. We will not print a model price here, because provider pricing changes often enough that any number would be wrong within months.

Work the default row by hand once. Four thousand runs at six thousand input and one thousand output tokens each, assuming three per million in and fifteen per million out, gives 4000 times (6 times 3 plus 1 times 15) divided by 1000, which is 132 of model spend. The review line: twelve percent of four thousand is 480 exceptions, at six minutes each that is 48 hours, at 40 an hour that is 1,920. Review costs roughly fifteen times the tokens. Replace every assumption; the ranking between lines is the point.

That ratio is why exception burn-down is the highest-value work after go-live. Halving the exception rate halves the largest line. Moving to a cheaper model to halve the smallest line saves less and often raises the exception rate, which costs more than it saved. Token cost arithmetic works through the token side properly.

How to tell whether you need this

Answer these seven questions about one specific process. Four or more answers on the left means a rebuild is likely to pay. Fewer means task automation, better documentation, or nothing, and all three are legitimate.

1. Does the item wait longer than it is worked on? A queue-to-touch ratio above three says the shape is the problem, not the speed of any step.
2. Can you say where a given item is right now? If answering needs two messages to two people, the process has no state, and giving it state is most of the value.
3. Does the same correction get made repeatedly? Recurring rework is encodeable. One-off corrections are judgement and should stay with a person.
4. Is there a defined finish line? A process ending in a clear terminal record can be automated. One that ends when somebody feels satisfied cannot.
5. Could you show an auditor how a decision was made? If not, you have an evidence problem now, and automating without an evidence store makes it permanent.
6. Do items get lost? Loss is the clearest symptom of uncontrolled handoffs, and the easiest thing for an orchestrator with timers to eliminate.
7. Is anyone accountable end to end? If nobody is, the rebuild will not survive its first disagreement. Fix ownership before spending on engineering.

How to start

The first conversation is a scoping call about one process. Bring the volume, the rough number of people who touch an item, and one recent example that went badly. That is enough for us to say whether it is worth measuring, and to say so honestly when it is not.

If it is, the next step is the handoff census: two days, and yours afterwards whether or not you build anything with us. Clients have run it, found the fix was a policy change rather than software, and made that change themselves. That is a good outcome and we will say so when it is the one in front of you.

For a first process, expect six to fourteen weeks from kickoff to cohort cutover. A second process in the same organisation costs materially less, because intake, orchestration, evidence and the console already exist.