How to Pick the First Process to Automate with AI

Pick the process you can grade, not the one people complain about

The usual method is to ask which job the team hates most. That surfaces the process with the worst ergonomics, rarely the one with the best economics. The hated job is usually hated because it is ambiguous, owned by two departments at once, or full of exceptions that need somebody to phone a customer. Those three properties are what make a system hard to grade.

Grading is the constraint because everything downstream depends on it. An eval suite is a set of graded examples. A rollout decision is graded output against a graded baseline. Incident response is a graded sample showing when quality moved. Without a verdict on output four hundred, none of that exists and the project becomes a matter of opinion.

The four properties that make a process automatable

Four properties matter: a machine-readable trail of past cases, a discrete verdict at the end of each case, enough repetition to build an eval set this month, and a blast radius you can contain. Most of a standard scoring sheet is decoration around these four.

A machine-readable trail

The process must already leave evidence a query can reach: resolution codes in a ticket queue, coded invoices in a ledger, disposition fields in a CRM. That trail is three assets at once. It is your golden dataset, your baseline, and the only honest way to estimate the exception rate before committing.

A discrete verdict

Each case must end in something checkable: a category, an amount, a routing destination, a field that either matches the source or does not. Processes ending in a paragraph somebody reads and feels good about can be automated, and cannot be defended when a stakeholder claims quality dropped in March.

Repetition above the eval threshold

The fixed cost of an automation barely moves with volume. Build, integration, eval suite and monitoring cost roughly the same at fifty cases a month as at five thousand. That is why a low-volume process with expensive handling time can beat a high-volume one that takes ninety seconds a case.

A blast radius you can contain

Ask what a wrong output touches before anyone notices, and how long you have to undo it. A misrouted internal ticket is recoverable in minutes. A wrong ledger code is recoverable until the period closes. An email to a customer is not recoverable at all. Reversal window is the number to write down.

The Adjudication Ladder ranks candidates by the cost of finding out

Rank every candidate by what it costs to learn whether one output was right. The rung it lands on decides the shape of the build, the size of the review queue, and whether an eval suite is possible at all.

The rule we work to: a first build sits on rung one or rung two. Later builds climb. If every candidate sits on rung four, the correct first project is not an automation, it is the instrumentation that will let you grade one.

Score the shortlist without falling for the loudest candidate

Score each candidate from evidence you pulled, not from what people said in the workshop. One file per candidate, filled with counts and names. A field nobody can fill is a finding about how ready that process is.

# One file per candidate. Fill it from evidence, not from memory.
# A field you cannot fill is itself a finding: record who could not fill it.

id: ap-invoice-coding
name: Coding supplier invoices to the right ledger account
owner: whose week gets worse when it breaks, by name
sponsor: who signs        # not the same person
system_of_record: which system holds the truth today

volume:
  cases_last_90_days: 0        # counted from the system, never estimated
  peak_day: 0

handling_time:
  median_minutes: 0            # one case, competent person, timed
  p90_minutes: 0               # the tail is what the review queue fills with
  source: timed sample | system timestamps | estimate

adjudication:
  rung: 1 | 2 | 3 | 4 | 5      # the Adjudication Ladder
  who_decides: role, by name if there is only one
  time_to_verdict: minutes | hours | days | weeks
  ground_truth_available: yes | no

trail:
  past_cases_exportable: yes | no
  export_method: API | scheduled report | database read | screen only
  fields_present: [input, decision, decider, timestamp]
  export_tested_by: your name, on the day you tried it
  export_took_minutes: 0

exceptions:
  followed_documented_path_pct: 0   # measured on last month
  top_three_exceptions: []

blast_radius:
  writes_to: [systems a wrong output would touch]
  customer_visible: yes | no
  financially_material: yes | no
  reversal_window: how long you have to undo it
  worst_case: one sentence, from the owner, not a vendor

constraints:
  personal_data: yes | no
  regulated: name the regime, or none

baseline:
  metric: e.g. median hours from receipt to posted
  value: 0
  measured_on: YYYY-MM-DD
  agreed_by: owner name

veto:
  one_sentence: what would make you refuse this candidate outright

Five rules for turning those files into a decision, each learned by watching a scoring sheet produce the wrong answer.

• Score volume as cases per month, not importance. Importance is an argument you can lose. Cases are a count.
• Treat rungs four and five as a veto, not a low score. A weighted average lets a big volume number drown a process nobody can grade, and that is the project that stalls at month five.
• Score containment by reversal window. Under an hour is green, same day is amber, irreversible is a veto until a human approval step is designed in.
• If the trail only comes out via a screen scrape or a vendor ticket, add integration weeks rather than deducting a point. Access difficulty is a schedule problem, not a quality problem.
• Refuse to collapse the sheet into one number. A composite score hides the veto column, and the veto column is the only part that protects you.

Run the arithmetic on your own numbers before you commit

Do the sums in public, with your numbers, before the project starts. Most first automations are justified by a saving that assumes every case goes straight through and nobody reviews anything. Both assumptions are wrong, and the gap is where the disappointment lives in month four.

Review minutes bite hard: one minute of review on a six minute task gives back a sixth of the saving before anything goes wrong, and most plans budget zero. Straight-through share moves the result more than any other input, which is why it deserves a two week shadow run rather than an assumption.

Automating the handling and automating the decision are different projects

Decide which of the two you are buying before picking the process. They have different costs, failure modes and verification. Roughly half the work brought to us as an AI problem is a handling problem with one small decision buried in the middle.

Handling is cheaper, verifiable by counting rather than by judgement, and it builds the pipes a decision model needs anyway. Do it first, leave the human decision inside the pipe as an explicit stop, then replace that stop with a model once you have three months of recorded decisions to grade against. Workflow automation and AI agent development are the two shapes.

Four traps that pick the process for you

Selection is usually decided by one of four forces, none of which has anything to do with fit. Name them out loud and they lose their power.

• The demo trap. A process is chosen because it looks good on a screen in a steering meeting. Screen-friendly and gradeable are unrelated properties.
• The vendor-shaped trap. It is chosen because a tool you already pay for ships a template for it. Good reason to try the template, poor reason to make it the flagship build.
• The volume mirage. A huge case count at two minutes a case saves less than a mid-volume process at forty. Multiply the two columns before anyone gets excited about the first.
• The export trap. The pilot runs on a CSV, never needs a write path, and looks finished at week six. Then the real write path needs permissions, idempotency and an audit trail nobody scoped. That mistake explains most of why AI pilots fail.

The right first process is the one whose answers you can grade cheaply, whose mistakes you can undo before lunch, and whose owner will still be in the job in six months.

A two-week selection you can run without a consultant

Two weeks is enough. The deliverable is three candidates, one recommendation, and a written veto for the other two.

1. Days one and two: list candidates from systems, not meetings2 days

   Pull case counts from the systems of record: tickets by queue, invoices by type, records by status transition. Anything you cannot count is not yet a candidate. Hunt repetition in data, not enthusiasm in a room.

2. Days three and four: time the work honestly2 days

   Time ten cases each with two people doing the process, or pull timestamps where the system records them. Report median and ninetieth percentile separately. The tail is what the review queue fills with once the easy cases are automated away.

3. Day five: place each candidate on the ladder1 day

   One question per candidate: who says whether this output was right, and how long do they take. Write the name and the duration. If the honest answer is a committee, that candidate drops regardless of how good its volume looked on Monday.

4. Week two, days one to three: run the export test3 days

   For the top three, export two hundred past cases with inputs, decisions and outcomes. Do it yourself, and time it. An export needing a vendor ticket or a warehouse project is the most reliable predictor we know of an estimate doubling.

5. Week two, days four and five: baseline, then write the veto2 days

   Measure the current state before anyone builds: cases per month, median and ninetieth percentile handling time, error rate as defined today. Then write the one sentence that would make you refuse each candidate. The one with an empty veto and a rung one or two verdict is your first build. Method in the baseline you must take before any automation.

The vocabulary this decision needs

Four terms that keep the conversation honest, because most selection arguments are two people using one word for different things.

Definitions

Adjudication cost

The time and expertise needed to decide whether one automated output was correct. It sets the price of every eval run, every incident investigation and every model change, which is why it belongs at the top of a selection sheet.

Straight-through rate

The share of cases that complete with no human touch. It decides whether an automation saves anything at all, and it is lower in month one than in the demo, because the demo used cases somebody chose.

Blast radius

The set of systems, records and people a wrong output reaches before somebody catches it. A one hour reversal window with no customer visibility is small, and small is what makes a first build safe to get wrong.

Golden dataset

A fixed set of real past cases with agreed correct outputs, used to score every version of the system against the same bar. It is assembled from the trail a process already leaves, which is why a process with no trail costs more than its case count suggests.