How to map a process before you automate it

What a process map for automation has to show

Process mapping has a bad reputation inside engineering teams, and the reputation is earned. Most maps are drawn to be shown rather than used, they encode what a manager believes happens, and they stop at the point where the interesting behaviour starts. A map for automation is a different artifact with a different test of quality: could a builder who has never met your team implement the happy path from this file, and would they know in advance which cases will defeat them?

Follow the artifact, not the people

Map the thing that moves. One requisition, one ticket, one invoice, one candidate. Write down the states it occupies and what causes each transition. Mapping by department produces a picture of your org chart, which changes every reorganisation and tells a builder nothing. Mapping by artifact produces a state machine, which is what the automation will literally be.

• Name every state as a condition the item is in, not as an activity someone performs. 'Awaiting budget check' is a state. 'Finance reviews it' is an activity, and activities hide the queue time where most of the elapsed time actually goes.
• Mark every transition observable or not. Observable means a timestamp exists somewhere a query can reach. Unobservable transitions are the ones you will not be able to measure after launch either, so they are a reporting problem as much as a build problem.
• Record the terminal states, all of them. Teams remember the successful ending and forget rejected, withdrawn, duplicated and abandoned. Those cases still consume human time and they still need defined behaviour.
• Count the loops. Any state the item can re-enter is a rework loop, and rework loops are usually the biggest single cost in the process. A map with no loops on it is a map drawn from memory.

Interviews give you the happy path

People describe processes the way they wish they ran. This is not dishonesty, it is compression: after two hundred repetitions the odd cases stop registering as part of the job and become interruptions to it, so they are left out of the description. The exceptions you are told about are the memorable ones, which is not the same as the frequent ones.

Use both, in this order. Pull the records first, count what happened, then take the counts into the interview and ask about the cases you cannot explain. That conversation is short, specific and productive, and it avoids the hour of narrative that a blank-page interview produces. The three questions that repay themselves are: show me the last one that went wrong, what do you do when the field is empty, and who do you ask when you are unsure.

Ask for four exports: the case records for the last full month with all their timestamps, the exception or escalation queue, the rework log if one exists, and the channel where the team asks each other questions. The fourth is the least official and often the most informative, because every recurring question in it is a rule that exists but was never written down.

The Four-Layer Process Map

The layers are ordered by how expensive it is to discover them late. A missing state costs you a sprint. A missing input costs you the trust of the team that has to use the system. A misjudged exception tail costs you the business case. An undesigned reversal path costs you an incident, and incidents are how automation programmes get cancelled.

Dark inputs are what kill the pilot

A dark input is information a person uses to decide something that is not present in any system the automation can read. It lives in memory, in a side spreadsheet, in a message thread, or in the fact that someone recognised the supplier name. Dark inputs are the single most common reason a demo works and the production system does not, because the builder picked demo cases where the visible data was sufficient, and about one case in six is not like that.

Where dark inputs hide, and what to do about each

The method for finding them is unglamorous and it works: sit beside someone processing real cases and after every click ask what they just looked at and why. Ten cases is usually enough for the pattern to appear. A screen recording of ten real cases beats an hour of interview, because the recording contains the glances at the second monitor that the person genuinely does not remember making.

The exception spectrum sets the budget

Rank the variants by share of cases and the shape is nearly always the same: a small number of variants cover most of the volume, then a long tail of situations that each occur a handful of times. The head is cheap to automate and the tail is the entire project risk, so the useful question is never how accurate the system will be. It is which slice of the distribution you are buying, and what happens to the rest.

Two traps live in this arithmetic. The first is coverage of volume against coverage of value: the rare variants are often the expensive ones, so a system that handles most of the items can still leave most of the money on the human side. Weight the ranking by value as well as by count before deciding what to build. The second is that a residual queue below roughly one unit a day tends to be handled badly, because nobody builds a habit around it. Sometimes the right design is to route more to humans, not less, and keep the queue busy enough to stay reliable.

1. Route it. The tail goes to a person with the record and the reasoning attached. Cheapest to build, and it is the correct default. See human in the loop design for how to make that queue survive contact with a busy team.
2. Degrade it. The system does the part it can and leaves the rest in a defined half-finished state that a person completes. This works when the unit decomposes cleanly and fails badly when it does not.
3. Refuse it. The system declines the unit, logs the reason, and does nothing else. Underrated. A refusal a human can read is worth more than a guess a human has to audit.

Write the map as a file, not a picture

A diagram cannot be diffed, tested or read by a program. Write the map as structured text in the repository, review it the way you review code, and update it when the process changes. The file becomes the input to the automation brief, the source of the evaluation cases, and the thing a new engineer reads in their first hour instead of asking six people.

# process-map.yaml
# One file per unit of work, versioned beside the code that will automate it.
# Every value here is an example, replace them with counts from your own systems.

unit: "one purchase requisition"
observed_window: "one calendar month"
observed_cases: 214            # counted from an export, never estimated in a meeting

states:
  - id: submitted
    entered_by: "requester, web form"
    system: "helpdesk"
    observable: true
  - id: budget_checked
    entered_by: "finance analyst"
    system: "spreadsheet, copied fresh each month"
    observable: false          # no timestamp exists for this transition
  - id: approved
    entered_by: "cost centre owner"
    system: "email reply"
    observable: true
  - id: raised
    entered_by: "buyer"
    system: "ERP"
    observable: true
  - id: rejected
    terminal: true
  - id: closed
    terminal: true

decisions:
  - at: budget_checked
    question: "is there budget left in this cost centre this quarter?"
    inputs:
      - name: "committed spend to date"
        location: "ERP report BC-14"
        machine_readable: true
        freshness: "rebuilt nightly"
      - name: "orders raised but not yet in that report"
        location: "the buyer's own spreadsheet"
        machine_readable: false        # dark input, decides roughly one case in six
    rule: "approve when remaining budget covers the amount, otherwise escalate"
    escalates_to: "finance manager"
    disagreement_rate: "two analysts differ on about one case in twenty"

exceptions:
  - variant: "amount over the delegated limit"
    share_of_cases: 0.11
    handling: "second approver, stays human by policy"
    separable: true
  - variant: "cost centre missing on the form"
    share_of_cases: 0.07
    handling: "buyer looks it up, derivable from the requester's department"
    separable: true
  - variant: "supplier not on the approved list"
    share_of_cases: 0.04
    handling: "procurement review, out of scope"
    separable: true
  - variant: "everything else, one-off situations"
    share_of_cases: 0.03
    handling: "route to a human with the full record attached"
    separable: false

reversal:
  detected_by: "monthly reconciliation, or a supplier calling about a wrong order"
  detection_lag: "up to one reporting cycle"
  undo_path: "credit note plus a manual correction in the ERP"
  blast_radius: "one order per incident, unless the rule itself is wrong"
  who_owns_the_fix: "buyer, with finance sign-off"

Notice what the file forces. You cannot fill in machine_readable without checking, you cannot fill in share_of_cases without counting, and you cannot fill in detection_lag without admitting how long a mistake survives. Every one of those fields is a question a vendor would otherwise have to price as unknown, and the value of the file is mostly in the fields you find yourself unable to complete.

Running the mapping in five working days

1. Pull the recordsday one

   One month of case records with every timestamp, the escalation queue, and the question channel. Do this before you book any interviews, because the counts change which questions are worth asking.

2. Build the artifact pathday one and two

   States, transitions, actors, systems, and the observable flag on every transition. Reconstruct it from the records first, then have one practitioner correct it. Correcting a draft takes twenty minutes; producing one from scratch in a workshop takes half a day and produces the idealised version.

3. Sit with two operators on real casesday two and three

   Ten cases each, recorded with permission. Ask after every click what they just read. Everything they consulted goes into the input inventory with a machine readable flag. Two operators rather than one, because the differences between them are unwritten rules.

4. Count the variantsday three and four

   Classify every case in the window into a variant and rank them by share. Add a value column. Stop when the classification stops changing, then write the tail down honestly as one bucket rather than pretending it is enumerable.

5. Trace one failure end to endday four

   Take a real case that went wrong, and follow how it was noticed, by whom, after how long, and what it took to undo. This gives you detection lag and blast radius, and it is the layer teams skip because failures are uncomfortable to discuss.

6. Write the file and circulate itday five

   Fill in the YAML, send it to the people you observed, and ask them to correct it. Their corrections are the most valuable review you will get, and they are also the moment the team starts trusting the project.

Definitions, and when the map is finished

Terms used in the file

Dark input

Information a person uses to make a decision that exists in no system the automation can read. Dark inputs are the most common reason a working demonstration becomes an unreliable production system.

Observable transition

A change of state that leaves a timestamp a query can reach. Transitions that are not observable cannot be measured before or after automation, so they are a reporting gap as well as a build risk.

Exception spectrum

The ranked distribution of process variants by share of cases, counted from records rather than recalled. It determines how much of the volume an automated path can cover and what the residual human work will cost.

Detection lag

The time between a wrong outcome being produced and a human noticing it. It is the governing input to how much autonomy an automated step should be given, and it is usually longer than anyone estimates.

Blast radius

The number of units affected by one instance of a fault before it is caught. A per-unit fault is an incident; a fault in a shared rule is a project-scale problem, and the map should say which class each decision belongs to.

A map that meets that list is enough to get a fixed price from a competent vendor, and enough to reject an incompetent one, because the questions it answers are exactly the ones a serious builder asks in the first meeting. If the mapping exercise itself is the part you would rather hand over, it is the first phase of any business process automation engagement, and it is worth doing even when the conclusion is not to automate the process at all.