On this page
- Readiness is what you can prove in an afternoon, not what a questionnaire says
- The Five Probes
- What each probe tells you, and what a failure adds to the project
- The agreement probe is the one everybody skips
- The afternoon, hour by hour
- The double-labelling pack you can send today
- Scoring the afternoon: go, narrow, or fix this first
- Definitions worth agreeing on before the meeting
Readiness is what you can prove in an afternoon, not what a questionnaire says
Run five live probes against your own systems in a single afternoon: try the export yourself, try to get a scoped credential, have two experts label the same twenty cases blind, count last month's exceptions, and try to undo a test write. What you learn in those four hours predicts the project better than any maturity questionnaire, because a questionnaire records what people believe and a probe records what the systems do.
Readiness questionnaires fail for a structural reason. They are answered by the people who own the systems, and those people answer from the documented state rather than the current one. Everyone reports the API as available because it exists. Nobody reports that the token was issued to a contractor in 2024 and that the rate limit is enforced per organisation rather than per key. The gap between the documented state and the current state is exactly where an automation project loses its first month.
A probe closes that gap by producing evidence. You are not asking whether an export is possible. You are exporting two hundred rows and writing down how long it took and who you had to ask. The output of the afternoon is a page of facts with timestamps, and those facts feed straight into an estimate.
- 5 probesExport, credential, agreement, exception and undo. Each is a thing you do, not a question you ask.
- 1 afternoonFour to five hours of your own time, plus about an hour each from two experienced practitioners.
- 20 casesEnough for a smoke test on expert agreement. Not enough for a benchmark, and the difference matters.
- CeilingWhere two experts disagree, no system can be graded as correct. Human agreement is the ceiling on measurable accuracy.
The Five Probes
Each probe replaces a question that people answer optimistically with an action that produces a fact. Run them in order, because a failure in probe one changes what the later probes are worth doing on.
The Five Probes
Five small experiments, each finished in under an hour, each producing a number or a name you can put in an estimate. Run them yourself rather than delegating, because the friction you meet is the finding.
Pull two hundred real past cases out of the system of record, with input, decision, decider and timestamp on each row. Do it personally and time it. Whether the answer is an API call, a scheduled report, a database read or a screen you cannot get behind, that answer sets the integration cost for the whole project.
Try to obtain a scoped service account with write access to one non-production record. You are measuring elapsed time and the number of approvals, not technical difficulty. Two weeks and four approvers is a schedule risk that belongs in the plan on day one, and it is invisible to every readiness questionnaire ever written.
Give two experienced practitioners the same twenty past cases, separately, with only the inputs the automation would see. Compare their decisions. The share where they agree is the ceiling on accuracy you can meaningfully measure. Where they disagree, no system can be graded, so those cases need a rule, an escalation path, or removal from scope.
Take last month's real cases and count how many followed the documented path from start to finish. Do not sample the ones people remember. The share that deviated tells you whether you are building an unattended system or a review queue, and it is usually further from the documented process than the process owner expects.
Write one test record into a non-production environment and try to reverse it. If there is no non-production environment, that is the finding and it is a large one. Record how the reversal is done, who is allowed to do it, and whether anything downstream already consumed the record. This is your blast radius, measured rather than assumed.
The common sequence is to scope a project, sign it, and then discover the export needs a warehouse ticket and the service account needs a security review. Reversing that order costs one afternoon and routinely moves an estimate by weeks in either direction. It also occasionally saves the whole budget, because a failed export probe on a closed vendor system is a genuine reason not to start.
What each probe tells you, and what a failure adds to the project
| Probe | What you actually do | A pass looks like | A fail adds |
|---|---|---|---|
| Export | Pull 200 past cases with inputs, decisions and timestamps | You have the file within an hour, unaided | Integration weeks, plus a labelling project if the fields are absent |
| Credential | Request a scoped service account with one write permission | Issued within days, by a named process | Schedule risk, and a launch date that depends on another team's queue |
| Agreement | Two experts label the same 20 cases blind | They agree on most cases and cite the same evidence | Scope reduction, written rules, or a decision to keep a human in the loop |
| Exception | Count last month's cases that followed the documented path | The documented path matches what actually happened | A review queue, staffing for it, and a longer path to straight-through running |
| Undo | Write a test record and reverse it | A non-production environment exists and reversal is documented | Approval gates on every write, and a much slower rollout |
The agreement probe is the one everybody skips
Two experts doing the same job disagree more than either of them expects, and the disagreement is not noise. It is the part of the process that was never actually decided, carried for years by whoever happened to pick up the case. An automation forces that ambiguity into the open, which is uncomfortable and is also the most valuable thing the exercise produces.
The arithmetic matters here. If your two reviewers agree on sixteen of twenty cases, an eval later reporting ninety-five percent accuracy is not measuring your system. It is measuring how closely the system imitates whoever wrote the labels. Set the accuracy target below the agreement rate, or fix the ambiguity with a written rule first and re-run the probe.
Twenty cases is a smoke test. The margin output uses the standard rough approximation for a proportion, one over the square root of the sample size, which is why a twenty case reading carries a margin wide enough to hide almost anything. Use it to decide whether to keep going, never to report an accuracy figure.
When the promised target sits above the expert agreement rate, the target cannot be verified, only asserted. There are three honest responses: lower the target, narrow the scope to the cases experts agree on, or write the rule that settles the disputed cases and re-run the probe. Building anyway produces a system that passes its eval and loses its first argument with a stakeholder.
The afternoon, hour by hour
Book the room, get the two reviewers to hold an hour, and work in this order. The sequence matters because the export gives the agreement probe its cases.
Try the export yourself. Record the method, the elapsed time, who you had to ask, and which of the four fields are missing. If it fails, note precisely where it failed and continue: the rest of the afternoon still has value.
Submit the real request for a scoped service account. You are starting a clock, not finishing a task. Note the approvers by role and diary the date it lands, because that date is a dependency in the plan.
Draw twenty cases at random from last month, add four known-difficult ones deliberately, strip everything the automation would not see, and send the pack to the two reviewers separately. Their hour runs in parallel with yours.
Count how many of last month's cases followed the documented path end to end, and list the three most common deviations by volume. This number sizes the review queue, which is the part of the design people forget to staff.
Write a test record into staging, reverse it, and time both. Record who is permitted to reverse a live record and whether anything downstream has already consumed it by then.
Collect the two label sets, compute agreement, and write a single page: five findings, five numbers, and one recommendation of go, go with a narrower scope, or fix this first. Circulate it the same day, while the evidence is still fresh and unarguable.
The double-labelling pack you can send today
Copy this, fill in the process name and the closed decision list, and send it to two people separately. The closed list is the important part: free-text answers cannot be compared, so they cannot produce an agreement number.
# Double-labelling pack: <process name>
# Two reviewers, working separately. Do not let them compare notes.
## Rules for the person running it
1. Draw 20 cases at random from last month. Do not hand-pick them.
2. Add 4 cases that were escalated or corrected. Random draws under-sample hard cases.
3. Give reviewers only what the automation would see. No hallway context, no phone calls.
4. Ask for a decision, a confidence, and the one piece of evidence that settled it.
5. Record seconds taken. If the real decision needs longer than the automation
would allow, that is a finding about scope, not about the reviewer.
## One JSON object per case, per reviewer
{
"case_id": "INV-2026-04417",
"reviewer": "initials",
"decision": "<one value from the closed list below>",
"confidence": "high | medium | low",
"deciding_evidence": "the field, sentence or attachment that settled it",
"seconds_taken": 0,
"note_if_ambiguous": ""
}
## Closed decision list, edit for your process, keep it closed
["approve", "route_to_finance", "route_to_legal", "reject",
"cannot_decide_from_input"]
## What you compute afterwards
agreement = cases where both decisions match / total cases
ambiguity = cases where either reviewer chose cannot_decide_from_input
evidence_match = cases where both cited the same deciding evidence
slow_tail = cases above 90 seconds for either reviewer
## How to read it
agreement -> the ceiling on accuracy you can measure. Targets above it are fiction.
ambiguity -> your minimum escalation rate. Design the queue for at least this.
evidence_match -> low match means one retrieved snippet is not enough context.
slow_tail -> these cases need the full document, or a human, or both.
Scoring the afternoon: go, narrow, or fix this first
Three verdicts, not a score out of ten. A composite readiness score averages away the one probe that should stop you, which is the same mistake as averaging away a veto when choosing what to automate.
- Go. The export worked unaided, the credential path is known and measured in days, experts agreed on most cases and cited the same evidence, exceptions are a manageable share, and a reversal path exists. Build, and size the review queue from the exception number rather than from hope.
- Go with a narrower scope. One probe failed in a bounded way. Experts disagreed on a specific case type, or the export is missing the decider field. Cut the disputed case type out of scope, write the rule for it, and keep it as a human path until the rule is proven.
- Fix this first. The export is impossible without a vendor project, or there is no non-production environment, or two experts agreed on barely half the cases. The first project is now the fix, not the automation. This verdict is unpopular and it is the one that saves the most money.
Take the baseline in the same week using the method in the baseline you must take before any automation, and map the process properly before design starts with mapping a process before you automate it. Across a whole department rather than one team, this is what an AI readiness audit produces.
Definitions worth agreeing on before the meeting
- Inter-rater agreement
- The share of cases on which two independent reviewers reach the same decision from the same inputs. It is the practical ceiling on accuracy for any automated system doing that job, because a case two experts dispute has no agreed correct answer to grade against.
- System of record
- The one system whose version of a fact is authoritative when systems disagree. Naming it is the first step of any integration, because automations that write to a system which is not the record of truth create quiet divergence that surfaces weeks later during a reconciliation.
- Scoped service account
- A machine identity with only the permissions one automation needs, separate from any human login. It matters because it makes actions attributable in an audit log and revocable in one step, and because obtaining one is often the longest lead time in a project plan.
- Shadow run
- Running the automation alongside the existing process without letting it act, then comparing its output against what the humans did. It converts assumptions about straight-through rate into a measurement, and it is the cheapest honest test available before go-live.
Questions readers ask next
How is this different from a vendor readiness questionnaire?
What if two experts disagree on most of the cases?
Can we run the probes on a sample rather than the real system?
Who should run the check, us or a vendor?
How long do the findings stay valid?
ChatGPTalker. "The AI Automation Readiness Check, Run in an Afternoon." chatgptalker.com, 2026-08-26. https://chatgptalker.com/guides/ai-automation-readiness-check/