Measuring whether an automation actually worked

What measuring automation impact actually requires

The reason to be strict about this is not accounting. It is that the first time the system does something visibly stupid, and it will, somebody senior will ask whether the whole thing was worth building. If the only evidence is a chart with a line going down and a launch date marked on it, the honest answer is that nobody knows, because at least four other things changed inside that window. Designing the comparison before launch costs a few days. Reconstructing one afterwards is usually impossible, because the pre-period has already gone.

The Realisation Ladder

The ladder is a rank order of how hard each rung is to argue with. Most disagreements about automation value are actually disagreements about which rung two people are standing on. An engineer reports an instrumented delta accurately. A finance lead hears a claim about realised money. Naming the rung in the first line of the report ends that argument before it starts: this is a mix-adjusted delta measured against a ten percent holdback, no capacity has been released yet, and no money has moved.

Why the before and after number is almost always wrong

Assume the naive comparison is wrong until you have eliminated the alternatives, because a process metric in a live business moves for a dozen reasons every quarter and your automation is one of them. The list below is worth walking every single time, in rough order of how often each one turns out to be the real explanation.

Two of these deserve extra attention. Mix shift is the most common and the least visible, because difficulty is rarely recorded anywhere, so the only defence is to split by a proxy you do have: document type, channel, customer tier, line count, region. The selection effect is the more dangerous one because it has a signature people misread. The automated path looks superb, the human queue looks worse than it did before, and both numbers are true. The automation took the easy two thirds, the humans now face a harder average case with the same staffing, and total time to clear the queue may not have moved at all. If you have not yet taken a proper before measurement, start with the pre-launch baseline and come back to this guide afterwards.

Three ways to get a comparison you can defend

Pick the strongest design your volume and your ethics allow, and decide it before launch rather than after the first result you dislike. In descending order of how convincing the result will be to somebody who was not involved.

1. Holdbackstrongest, needs volume

   Route a random share of eligible work down the old path for a fixed window, usually somewhere between five and twenty percent depending on volume. Randomise per unit of work using a hash of the unit id, never per person and never by anyone choosing, because the moment a human routes cases the comparison is dead. Write the window and the share into the config so nobody quietly turns it off in week two. A holdback removes seasonality, mix shift and concurrent change in one move, because both arms experience all of them.

2. Staged rollout by segmentthe usual compromise

   Enable by team, region, document type or customer tier in a planned order, so every segment that has not been switched on is a temporary control. The catch is that segments differ from each other, so you compare each segment to its own pre-period rather than to the other segments, and you look for the step change to land in the week each one was switched. Randomise the order where you can, because rolling out to the enthusiastic team first produces a number driven by the team.

3. Switchbackfor shared queues

   Alternate whole days or whole weeks between the old and the new path when both feed one queue and cannot run side by side without contaminating each other. You need enough periods to average out day-of-week effects, so a fortnight of alternating days is usually the floor. Switchback is the design most people forget, and it is often the only one available for a shared inbox or a single processing team.

4. Interrupted time series with a control serieswhen nothing can be held back

   Use it when the old path has to be switched off completely. You need enough pre-period points to establish a trend rather than a level, ideally a year to catch seasonality, plus a related series that the project did not touch and that responds to the same business conditions. The claim then becomes the change in your series relative to the change in the control series, which is far weaker than a holdback but far stronger than a launch date on a chart.

The metrics that get gamed, and the guard metric for each

Every headline metric in automation has a cheap way to move it that does not involve doing the work better, and teams find those ways without anyone deciding to cheat. The defence is structural: publish each headline metric permanently paired with the metric that would deteriorate if the headline were being moved dishonestly.

The pairing has a second benefit that matters more over time. Guard metrics are early warnings. A rising rework rate or a rising reopen rate shows up weeks before anyone complains loudly enough to reach the project channel, and it is usually the first visible symptom of a model change, a prompt edit or a shift in the incoming work. Wiring those pairs into a live view rather than a monthly slide is covered in monitoring an AI system in production.

Hours saved are not money until somebody spends them differently

The single largest gap between reported and believed automation value sits in one variable: the share of freed hours that turned into anything. Six minutes per unit across a large queue is a genuinely large number of hours. Those hours become money only if headcount changes, if overtime or contractor spend falls, if a hiring plan is cancelled, or if the time is pointed at named work that somebody would otherwise have paid for. Otherwise the hours are absorbed, the process gets a little more comfortable, and the finance system shows nothing.

Run it twice. Once with the reallocation share your business case assumed, and once with the share you can actually evidence from a rota change, a cancelled requisition or a named piece of work. The gap between those two payback figures is the real subject of the conversation you are about to have with finance, and it is far better to raise it yourself in month one than to have it raised for you in month nine.

Count what the system costs, including the parts that hide

The run cost of an AI automation is routinely understated by a factor that surprises people, not because anyone lies but because the components sit on different invoices owned by different teams. Build the cost side of the report from a fixed list and refuse to publish a value figure without it.

Inference is usually the line people expect to dominate and usually is not, once volume is real and the prompt has been trimmed. The arithmetic for that side sits in token cost arithmetic. If you want the reporting itself built and maintained rather than assembled by hand every month, that is reporting automation.

Quality is a constraint, not a metric you get to trade

Speed and cost are things you optimise. Quality is a line you agreed not to cross, and the difference matters because optimisers will happily trade a constraint they have been handed as a metric. Write the constraint as a rule with a number in it, before you see any results, and put it in the same file as the target.

Write the acceptance rule before you read the results

An acceptance rule names the improvement you are claiming, the guard metrics, the direction each must move, and what happens if a guard fails. The wording that works looks like this: cycle time at p90 falls by at least a quarter against the holdback, while rework rate, escalation rate and seven-day reopen rate are each no worse than the baseline plus one point, measured over four full weeks, with a minimum of four hundred completed units. If a guard breaches, the rollout pauses at its current share rather than continuing, and the decision to resume is taken by a named person.

• State the improvement threshold as a number, not as an aspiration, so the result cannot be renegotiated after the fact.
• Give every guard metric a tolerance, because measurement noise will move any rate a little and a zero-tolerance rule pauses the rollout every fortnight for nothing.
• Set a minimum sample size alongside the tolerance, so a rate computed from thirty cases cannot trigger anything.
• Name the person who decides on a breach. A rule with no owner is a rule that gets overruled by whoever is in the room.
• Record what you will do if the improvement lands and a guard fails anyway, because that is the case people avoid planning for and it is the one that actually happens.

Where the quality bar has to be evaluated on model output rather than on process events, the mechanics of writing that evaluation belong in writing evals for systems that are not deterministic. Process guard metrics and output evals are complementary. Guards catch a system that is doing the wrong work correctly. Evals catch a system that is doing the right work badly.

The monthly report that survives a hostile reading

Write the report as a file, not a slide, and keep it in version control next to the process code. A slide gets rebuilt by a different person each month with a slightly different definition, which is how a metric silently improves. A file has a diff, and a diff makes a definition change visible. The spec below is the structure worth copying: it forces the mix, the change log, the guard metrics and the rung to be present, because those are the four things a hostile reader will ask for and the four things people leave out.

# monthly-impact-report.yml
# One file per automated process, versioned next to the code that runs it.
# Every value in angle brackets is yours to fill. Nothing here is a benchmark.

process: invoice_intake
owner: <the individual accountable, not a team name>
period: 2026-08-01/2026-08-31

comparison:
  method: holdback              # holdback | staged_rollout | switchback | interrupted_time_series
  holdback_share: 0.10
  randomised_by: unit_id        # never by person, never by anyone choosing
  window_days: 28
  change_log:                   # everything else that moved in the same window
    - date: 2026-08-12
      change: new supplier onboarding form went live
    - date: 2026-08-19
      change: two contractors joined the processing team

mix:
  dimension: document_type
  weighted_to: baseline         # recompute on the baseline mix, always
  volumes:
    standard: <n>
    multi_page: <n>
    handwritten: <n>

headline:
  metric: cycle_time_p90_hours
  baseline: <from the pre-period file, not from memory>
  treated: <value>
  control: <value>
  attributed_delta: <treated minus control, with the sign spelled out>

guard_metrics:                  # reported whether they pass or fail
  - metric: rework_rate
    rule: not worse than baseline
    value: <value>
  - metric: escalation_rate
    rule: not worse than baseline
    value: <value>
  - metric: reopen_rate_7d
    rule: not worse than baseline
    value: <value>

capacity:
  hours_removed_gross: <value>
  review_hours_added_back: <value>
  hours_reallocated: <named work, or the number zero>
  headcount_change: <state zero if it is zero>

cost:
  inference: <from a tagged invoice line, including failed and retried runs>
  orchestration_hosting: <value>
  human_review: <review minutes times the loaded rate>
  maintenance_allocation: <agreed at handover, not guessed later>
  loaded_hourly_rate_used: <state it here so it can be argued with>

ladder_rung: 3                  # 1 instrumented 2 mix_adjusted 3 attributed 4 released 5 realised
known_limitations:
  - <the objection you would raise yourself if you were reading this>
query_sha: <git hash of the SQL that produced every number above>

Two fields carry most of the weight. The change log is what stops the report claiming credit for a staffing change, and it costs one line a fortnight to maintain. The query hash is what stops the definition drifting, because if the numbers moved and the hash did not change, the process changed, and if the hash changed, somebody has to say why in the commit message. Neither field is glamorous and both of them will save an argument in a room where you are outnumbered.

Definitions, and when to stop measuring

Definitions

Counterfactual

What the process would have done in the same period without the automation. Every impact claim is a comparison against a counterfactual, and the only question is whether yours was estimated deliberately or assumed by accident.

Holdback

A randomly selected share of eligible work that keeps running on the old path after launch, for a fixed window, so both paths meet the same conditions. Randomisation happens per unit of work, never per person and never by anyone choosing which case goes where.

Mix adjustment

Recomputing a metric weighted to the composition of work seen in the baseline period, so that a change in what arrived cannot be misread as a change in how well the process ran.

Guard metric

A second metric published alongside every headline claim, chosen because it is precisely what would deteriorate if the headline metric were being moved dishonestly. Deflection rate travels with reopen rate. Cycle time travels with rework rate.

Released capacity

Hours or headcount fractions that came free and are now doing named different work, or have left the cost base entirely. Capacity that is free but unassigned is not released, and valuing it as money is the most common way an impact report loses credibility.

Fully loaded cost per unit

The cost of one completed unit including inference on successful and failed runs, orchestration and hosting, the human review minutes that remain, an agreed maintenance allocation, and amortised vendor fees.

Stop the formal measurement when the comparison stops being possible or stops being useful, which is usually somewhere between three and six months after launch. By then the old path has decayed, the people who ran it have moved on, and the counterfactual has become genuinely unknowable, so a holdback is no longer measuring what it was measuring on day one. At that point the impact report retires and the guard metrics stay, moving from a monthly document into the live dashboard. What you keep forever is the pre-period file, the acceptance rule, and the three or four reports that covered the window when a real comparison existed. That is the evidence that gets pulled out two years later when somebody proposes ripping the system out.