The baseline you must take before any automation

What a baseline is and why after-only numbers are worthless

The reason to care is not reporting. It is that a baseline changes what you build. Teams that measure first regularly discover that the step they were about to automate holds a small share of the elapsed time, that most of the delay is a queue nobody owns, or that the process runs a third as often as everyone believed. Each of those findings redirects the project before money is spent, which is worth more than any dashboard produced afterwards.

The Five-Number Baseline

The set is deliberately small. Baselines fail more often from being too ambitious than from being too thin, because a measurement plan with twenty metrics never gets finished and the project starts anyway. Five numbers and one artifact can be assembled in a few days by one person with database access.

Take it from logs, not from people

Asking a team how long something takes produces a number shaped by memory, and memory over-weights the worst case and forgets the waiting entirely. Ask people to time themselves and you get a second problem on top: the act of measuring changes the work, usually downwards, and it costs you goodwill at the exact moment you need the team on side.

Where the timestamps already live

Nearly every process leaves a trail in systems you already pay for. The work is knowing which field means what, and which ones lie. Each row below has a trap attached, and every one of those traps has quietly corrupted somebody's business case.

Because these records already exist, most baselines are retrospective rather than prospective. You are not waiting four weeks to start the project; you are querying the last six months this afternoon. Pull six months rather than one, so you can see whether the month you would have chosen was typical.

Measure the p90, or you will be judged on it

Report a mean and you will describe a process nobody experiences. Most operational work is bimodal: a large population of straightforward cases and a smaller population of complicated ones, with the average sitting in the empty space between them. The complaints, the escalations and the service level breaches all come from the second population, so that is the number your stakeholders carry in their heads.

• Report p50 and p90 together, always. The pair describes the shape. A p90 close to the p50 means a predictable process; a p90 several times the p50 means two processes wearing one name, and they should probably be measured, and automated, separately.
• Keep the count next to the percentile. With thirty completed units, the ninetieth percentile is essentially the third worst case, and a single unusual week moves it. Publishing a p90 without its sample size invites a comparison that the data cannot support.
• Never mix variants in one percentile unless you also publish the mix. Otherwise a change in mix reads as a change in performance, which is the most common way an honest team accidentally reports a false improvement.
• Store the raw distribution, not just the summary. Six months from now somebody will ask a question your summary cannot answer, and the query will be cheap to rerun only if you kept the extract.

-- baseline.sql, Postgres flavoured, adapt the names and keep the shape.
-- Produces the five numbers per month from timestamps you already store.
-- Run it before the build, save the output, and never edit that file again.

WITH work AS (
  SELECT
    id,
    variant,
    date_trunc('month', created_at) AS period,
    EXTRACT(EPOCH FROM (closed_at - created_at))      / 3600.0 AS cycle_hours,
    EXTRACT(EPOCH FROM (first_touch_at - created_at)) / 3600.0 AS wait_hours,
    reopened_count,
    assignee_changes
  FROM cases
  WHERE closed_at IS NOT NULL
    AND created_at >= now() - interval '6 months'
)
SELECT
  period,
  variant,
  count(*)                                                  AS volume,
  round(percentile_cont(0.5) WITHIN GROUP (ORDER BY cycle_hours)::numeric, 2) AS cycle_p50,
  round(percentile_cont(0.9) WITHIN GROUP (ORDER BY cycle_hours)::numeric, 2) AS cycle_p90,
  round(percentile_cont(0.9) WITHIN GROUP (ORDER BY wait_hours)::numeric,  2) AS wait_p90,
  round(avg(CASE WHEN reopened_count > 0 THEN 1.0 ELSE 0.0 END), 3)          AS rework_rate,
  round(avg(assignee_changes), 2)                                            AS handoffs_per_unit
FROM work
GROUP BY period, variant
ORDER BY period, volume DESC;

-- Record alongside the output, in the same file:
--   window: the exact dates covered
--   n: the number of completed units in the window
--   excluded: any filter you applied and the reason for it
--   owner: the person who ran it and can answer questions about it later
--   note: anything unusual in the window, such as a system outage or a hiring gap

The block of metadata at the bottom of that file matters more than the query. A baseline with no recorded window, no sample size and no owner becomes uncheckable within a quarter, and an uncheckable baseline is treated as a marketing number by everyone who reads it afterwards.

The denominator trap

Pick the denominator before the build, and make it the end to end cycle. Automating a step that holds a small share of the elapsed time produces a large improvement in that step and a small one overall, which is Amdahl's law applied to operations. Teams that choose the denominator afterwards end up reporting a big percentage against the step they automated, next to a process that feels exactly the same to everybody outside the team. That gap is how automation programmes lose credibility even when the engineering was good.

There is a fair counter-argument worth stating: cycle time is not the only thing worth improving. Removing tedious work has a real effect on error rates, on staff retention and on capacity for the work only people can do. State which of those you are optimising for in the baseline document, and measure that one. Choosing the outcome after seeing the results is the thing to avoid, not the ambition itself.

Baseline the quality, not just the speed

Speed baselines are easy and quality baselines are the ones that decide arguments. Take fifty completed units, record what the correct output was for each, and record who decided that. This gives the accuracy conversation a reference point, it turns into the acceptance test in your automation brief, and it exposes a problem most teams do not know they have.

Measure the disagreement between your own experts first

Give two senior people the same twenty units and have them label independently, without discussing them. Then compare. If they differ on three of the twenty, no automated system will be judged consistently better than that, because there is no stable definition of correct to build against. Human agreement is the ceiling on measurable accuracy, and finding out where that ceiling sits costs one hour of two people's time.

1. Where they agree, you have a rule, and a rule can be automated and tested.
2. Where they disagree and can resolve it in conversation, you have an unwritten rule. Write it down. It belongs in the config the automation reads, not in the model's head.
3. Where they disagree and cannot resolve it, you have found a judgement call, and that class of unit should route to a person by design rather than be handed to a model that will answer confidently either way.

Keep the labelled set in version control with the query output. It is a small file with a long life: acceptance testing at handover, regression testing after launch, and the reference for the first time somebody claims the system has got worse.

How long to measure, and when a baseline expires

Measure one full cycle of the process at minimum, and long enough to hold at least a hundred completed units. For a daily process that is a month. For a monthly close it is a year, which is why those baselines get reconstructed from history rather than collected forward. Include a peak period or state plainly that you excluded one, because a baseline taken in a quiet month will be used against you later.

1. Confirm the unit and the windowone hour

   Agree what one unit is and what trigger and terminal events define it. This should already be settled if you have done the process map. Then pick a window of at least six months of history and write down the dates.

2. Pull the raw extracthalf a day

   One row per completed unit with every timestamp you can reach and the variant attached. Store the extract itself, not only the summary, and keep it where a colleague can find it in a year.

3. Compute the five numbers by month and by varianthalf a day

   Never as one blended figure across the whole window. The monthly series is what tells you whether the process is stable, and stability is what decides whether a single number means anything at all.

4. Label fifty units and measure expert agreementone day, two people

   Twenty units labelled independently by two seniors, thirty labelled once for the reference set. The disagreements are the most valuable output of the whole exercise.

5. Write the baseline document and freeze itone hour

   Numbers, window, sample size, exclusions, assumptions, owner, date. Then make it read only. Any later correction goes in a new version with a note explaining the change, because an edited baseline is not evidence.

A baseline expires when the world underneath it moves: volume changes materially, the team changes size, an upstream system is replaced, or the mix shifts. Re-baseline then, and archive the old one rather than overwriting it. Comparing across a regime change without saying so is the most common way honest teams produce misleading before-and-after numbers, and it is also how you lose an argument with a finance director who noticed.

Definitions, and the check before the build starts

Baseline terms

Process baseline

A dated record of how a process performed before it was changed, expressed as volume and mix, cycle time percentiles, the split of touch and wait time, a rework rate and a fully loaded unit cost, taken from system events rather than from estimates.

Cycle time

Wall clock time from the trigger event to a terminal state, including every queue the item sits in. It is not the same as touch time, and confusing the two is the most common error in an automation business case.

Touch time

The portion of cycle time during which a person is actively working on the unit. Automation usually reduces touch time first and wait time only if the queue itself is redesigned.

Rework rate

The share of units that re-enter an earlier state or get reassigned before completing. It works as a quality proxy because it is derived from system events rather than from a definition of correctness that has to be negotiated.

Expert agreement ceiling

The rate at which two qualified people independently produce the same answer on the same units. It sets the practical upper bound on measurable automated accuracy, because a system cannot be reliably scored above the consistency of its own reference standard.

That list takes a few days and it survives longer than the project does. It is also the input to every later argument about whether the system worked, which is the subject of measuring automation impact. If the numbers need to keep arriving without a person rebuilding a spreadsheet every month, that recurring version is what reporting automation is for.