Keeping a human in the loop where it matters

The short answer

Oversight designs fail in one of two directions. Either every action needs a signature, the queue backs up, and people approve in batches without reading. Or nothing does, and the first anyone hears of a problem is a customer. Both come from treating oversight as a moral stance rather than a budget to allocate.

The three shapes of oversight

There are three real shapes, plus a fourth that is a filter over the others. Most systems need two at once, on different slices of traffic.

The important cell is the last one. Your routing rules define what you can see: if a human only ever reviews flagged items, every quality number you have describes the flagged population and you know nothing about the rest. The fix is boring. Route a random one or two percent of unflagged traffic to review as well, forever, and keep the two streams separate in the data.

Why approval on everything decays into a rubber stamp

A gate that sees a high volume of mostly correct items stops functioning as a check, and the mechanism is well understood outside AI. Three forces push the same way.

• Low base rates. When almost everything in the queue is fine, careful reading rarely changes an outcome, so attention drifts to the default action. This is the vigilance decrement seen in any monitoring task where signals are rare.
• Automation bias. A confident, well formatted proposal is read as evidence of correctness. Fluent output gets approved faster than clumsy output of the same quality, which is exactly backwards.
• Queue pressure. Approve is one click and reject requires a reason and a conversation. When the queue is 200 deep at 5pm, the asymmetry decides the outcome.

The countermeasures that work are structural, not motivational. Cut queue volume so each item gets attention, show the difference rather than the whole record, require a closed set of reject reasons, and audit a sample of approvals rather than only rejections. An unaudited approve is the cheapest way for a mistake to become permanent.

The Interrupt Budget

Treat human attention as a fixed daily quantity and gates as a portfolio competing for it. This reframes the argument from whether a human should approve something to whether this gate is worth more than the gate it displaces.

Two things fall out of this every time. Capacity used passes 100 percent sooner than anyone expects, and value per review is independent of volume, so a gate is either worth the interruption for each item or not worth it at all. Above roughly 70 percent utilisation, queue latency stops being linear and items start expiring, which looks like a backlog and is a design error.

Put the gate at the irreversible step

The gate belongs immediately before the effect leaves your control, not before the model call. Approving a plan is close to worthless, because the plan is not what executes. Approve the tool call, with its exact arguments, at the moment it would run.

Better still, move the irreversibility. Most actions people gate can be made reversible with an hour of engineering, and a reversible action needs a smaller gate or none.

• Draft state. The agent writes the reply, the record, the invoice into a draft that exists but does nothing. A human promotes it. Nothing is at stake until promotion.
• Delayed effect. Sends go out after a hold, commonly thirty minutes, during which anyone can pull them back. It costs almost nothing and removes most of the queue.
• Compensating action. Every side effecting tool ships with its inverse: refund reverses charge, credit note reverses invoice. Store the compensating call alongside the action so undo is one operation rather than an investigation.
• Limits in code. A ceiling that refuses the action beats a gate that asks about it. If nobody should refund above a number without finance, the tool should reject it, not route it.
• Canary slices. New behaviour runs on a small slice with tighter gating and widens only once the slice is clean. This is where a gate earns its cost, because error rates are highest when behaviour is new.

The decision packet

A reviewer should decide in well under a minute from one screen, which needs a packet built for the decision rather than a dump of the run. The highest value field is the one naming why this item was routed here, because it teaches the policy to the person applying it.

{
  "packet_version": "3",
  "run_id": "run_2b91f7",
  "routed_because": "amount 420.00 exceeds auto-approve ceiling 250.00",
  "proposed_action": {
    "tool": "issue_refund",
    "args": {"invoice_id": "INV-88213", "amount": 420.00, "currency": "GBP", "reason_code": "late_delivery"},
    "reversible": false,
    "undo_path": "manual credit note, finance, about two days"
  },
  "diff": {
    "before": {"invoice_status": "paid", "balance": 0.00},
    "after":  {"invoice_status": "partially_refunded", "balance": -420.00}
  },
  "evidence": [
    {"kind": "email", "id": "msg_7712", "quote": "parcel arrived 9 days late", "received_at": "2026-08-22T11:04:00Z"},
    {"kind": "policy", "id": "policy/refunds#4.2", "version": "2026-03", "quote": "full refund where delivery exceeds 7 days"},
    {"kind": "record", "id": "orders/88213", "fetched_at": "2026-08-26T08:02:11Z"}
  ],
  "model_view": {
    "summary": "Policy 4.2 applies. Delay was 9 days. Customer has no prior refunds.",
    "self_reported_confidence": 0.86,
    "caution": "self reported, not calibrated, do not threshold on this"
  },
  "cost_if_wrong": {"money": "420.00", "who": "cust_1099", "detectable_within": "one billing cycle"},
  "options": [
    {"id": "approve"},
    {"id": "approve_with_edit", "editable_fields": ["amount", "reason_code"]},
    {"id": "reject", "reason_required": true,
     "reasons": ["policy_does_not_apply", "wrong_customer", "amount_wrong",
                 "evidence_insufficient", "needs_human_owner", "should_never_be_automated"]}
  ],
  "deadline": "2026-08-26T17:00:00Z",
  "on_timeout": "hold_and_alert"
}

1. Show the diff, not the record. Before and after, on the fields that change. Reviewers cannot audit a whole object at speed and will stop trying.
2. Quote the evidence with ids. A pasted sentence with a document id and a version is checkable. A summary of the evidence is another model output to trust.
3. Print the model's confidence and label it. Self reported scores correlate with fluency more than correctness, so display it as context and never as a threshold.
4. Closed enum reject reasons. They become the labels for your eval set. Free text becomes archaeology nobody performs.
5. on_timeout is hold_and_alert. Auto-approve on timeout converts your gate into a delay. Whoever chose that default did it to keep the queue moving, and it is the quiet way irreversible actions ship unread.
6. Every decision writes a record. Who, when, which packet version, what they saw. That is the same record your auditors want, described in making an agent's decisions auditable.

Choosing what reaches a human

Route on properties of the action, not on the model's opinion of itself. Rules are legible, testable and stable across model versions, which self reported confidence is not.

• Money above a ceiling, or any action touching a payment method.
• First interaction with a counterparty, or a customer flagged as sensitive.
• Bulk actions above a record count, because blast radius scales while attention does not.
• Anything irreversible with no compensating action registered.
• Retrieval returned nothing above your similarity floor, the classic setup for a confident wrong answer.
• A tool errored and the agent retried its way to a different plan.
• The verifier disagreed with the author, which is the cheapest high value signal you will find.
• A random sample of everything else, so you can measure the population you are not worried about.

Retiring a gate without losing control

Gates should be removable, with a bar agreed in advance. Otherwise every gate is permanent and the budget only shrinks.

1. Run silent shadowTwo to four weeks

   The agent proposes, the proposal is stored without being shown, and the human decides as usual. Comparing afterwards gives an uncontaminated agreement rate, which showing the proposal would destroy.

2. Write the bar before you lookThirty minutes

   Agreement rate on that slice, a minimum item count, and zero severe misses, with severe defined in advance. Setting the bar after seeing the data is how every gate gets approved for removal.

3. Open the narrowest safe bandWeek one

   Auto-approve only the safest slice, for example amounts under a ceiling with a known counterparty and a policy match, rather than the whole class. Bands can widen later.

4. Keep sampling and probes at the new levelOngoing

   The newly ungated band still gets a sampled audit and seeded probes. Removing a gate removes the interruption, not the measurement.

5. Rehearse the undo once, for realOne afternoon

   Reverse a real action in production using the documented path, with a stopwatch. An undo path nobody has run is a story, and you learn on the worst day whether it was true.

6. Make the band close itselfIn code

   If override rate or probe catch rate crosses the bar, the band narrows automatically and pages someone. A rollback that needs a meeting is not a rollback.

The vocabulary, used precisely

Terms worth pinning down

Human in the loop

A design in which a person must act before the system's action takes effect. The person is inside the execution path, so the action does not happen without them, and system throughput is bounded by their availability.

Human on the loop

A design in which the system acts and a person monitors, samples and intervenes afterwards. Throughput is not bounded by the person, so it requires an undo path and a sampling regime to be a real control rather than a label.

Seeded probe

A known-bad item deliberately inserted into a review queue at a fixed rate to measure the share of defects reviewers actually catch. It is the only direct measurement of whether an approval step is still functioning as a check.

Compensating action

The registered inverse of a side effecting operation, such as a credit note against an invoice. Its existence turns an irreversible action into a reversible one, which usually removes the need for a blocking gate.

Override rate

The share of proposed actions a human rejects or edits before they execute. A rising rate means the system got worse or the population shifted. A rate near zero means either the system is very good or nobody is reading, and only seeded probes distinguish the two.

Undo window

The interval between an action being taken and it becoming irreversible, during which a person can pull it back. Deliberately introducing one is often cheaper and more effective than approving each action in advance.

The gates, the probe harness and the eval set that grows out of reject reasons are evaluation and guardrails. What an agent may attempt at all is a separate control, in controlling what an agent is allowed to do, and the failures these gates catch are in how agents fail.