On this page
- What monitoring an AI system means when nothing is throwing errors
- The Four-Plane Monitor
- Which signals are worth waking someone for
- The silent failures, the ones that return 200 and a confident answer
- Online evaluation: judging a sample of live traffic
- How to threshold a rate that is naturally noisy
- Token spend is a reliability signal, not just a finance one
- The runbook, and alert rules you can copy
- Definitions, and a two week instrumentation plan
What monitoring an AI system means when nothing is throwing errors
Monitoring an AI system means watching four things at once. Transport is ordinary service monitoring: latency, status codes, timeouts, saturation. Behaviour is what the run did on the way to an answer: step count, which tools were called, retries, tokens spent, terminal reason. Output is what it produced, checked without a human: schema validity, refusal rate, groundedness against the retrieved evidence, and a pinned judge score on a sampled slice. Outcome is what the business did next: accepted, edited, overridden, escalated, reversed. Nearly every team instruments the first plane, some reach the second, and the failures that damage a business live in the third and the fourth. The status code of a serious AI incident is almost always 200.
- 4planes to watch: transport, behaviour, output and outcome. Most systems are instrumented only on the first
- 200 OKis the response code of nearly every serious AI failure, which is why classic uptime monitoring misses them
- 1 idshould connect the trigger event, every model call, every tool call and the final outcome record
- 2 gateson every alert rule: a minimum denominator, and a comparison against a rolling baseline rather than a fixed number
The gap between transport monitoring and the other three planes is the whole subject. A model that has started returning fluent, well formatted, entirely outdated answers because the retrieval index stopped updating will show perfect latency, zero errors and a completely healthy dashboard for as long as you let it. Nothing in your existing observability stack is wrong. It is measuring the wrong layer, and no amount of tuning the alert thresholds on that layer will surface the problem.
The Four-Plane Monitor
The Four-Plane Monitor
Four layers, each answering a different question, each with its own detection window and its own owner. Build them in this order, because each one is only interpretable once the one before it is in place.
Requests per minute, latency at p50 and p99, provider status codes split by class, timeouts, retry counts at the HTTP layer, and saturation of whatever runs the orchestration. This is identical to monitoring any other network service and every team already has it. It tells you the system is up. It tells you nothing about whether the answers are right, which is why an all green dashboard during a serious incident is the normal case rather than a surprise.
Steps per run, tool call sequence, retries at the agent layer, tokens in and out, cache hit share, wall clock per step, and the terminal reason for every run: finished, hit the step ceiling, hit a token ceiling, was cancelled, errored. Watch distribution shape rather than the mean. A small share of runs walking to the step ceiling is the early signal of a looping agent, and it will not move an average until it is far too late.
Properties of the output computed by machine: does it parse against the schema and which path failed if not, does every cited claim resolve to a chunk that was actually retrieved, is the refusal or empty answer rate stable, is the length distribution stable, does a version pinned judge score a sampled slice above the agreed bar. Cheap checks run on every output. Expensive checks run on a stratified sample.
What the rest of the business did with the artifact: accepted, edited with a diff, overridden, escalated, reopened, reversed. This is the only plane that measures usefulness rather than correctness, and it is almost always the last one built because it needs a write back from a downstream tool. Instrument the accept or edit action in the interface where a human touches the result, and the override rate becomes the highest signal metric you own.
The four planes are only useful joined. One correlation id has to travel from the trigger event through every model call, every tool call, the output record, and the outcome write back that may arrive three days later when a human finally reviews the item. Without the join you have four dashboards that each say something is wrong and no way to see the same run in all four. The field level detail of that record is in what to log so future you can debug it.
Which signals are worth waking someone for
Page on output and on hard availability. Ticket on behaviour and outcome. Graph transport unless a user is waiting synchronously on it. The reason is detection speed against remediation speed: a schema failure breaks every downstream consumer within minutes and has an immediate action, while an override rate drifting upward needs a day of reading runs and has no action at three in the morning.
| Signal | Plane | Window | Response and why |
|---|---|---|---|
| Schema validation failure rate above its rolling baseline | Output | 15 min | Page. A parse rate that jumps is nearly always a prompt edit or a model change, and it breaks everything downstream at once |
| Retrieval returning zero chunks, or fewer than k | Behaviour | 15 min | Page. An index that stopped updating looks perfectly healthy on every other signal |
| Age of the newest document in the retrieved set | Behaviour | 30 min | Page. This is the only cheap detector for a stalled ingest pipeline |
| Groundedness failures on answers that carry citations | Output | 1 hour | Page if the surface is customer facing, ticket if internal |
| Share of runs terminating at the step ceiling | Behaviour | 30 min | Ticket. A rise means a tool is failing quietly and the agent is retrying around it |
| Tokens per run at p95 | Behaviour | 1 hour | Ticket. Cost is where a loop appears first, usually well before anything errors |
| Human override rate against the trailing week | Outcome | 24 hours | Ticket. This is how a real quality regression reaches you, and it needs reading not paging |
| Provider 429 and 5xx share | Transport | 5 min | Alert, with the current backoff policy visible on the same panel |
| Latency p99 | Transport | 5 min | Graph. Page only where a person is waiting on the response in real time |
Note what is missing from that list. There is no alert on average quality score, because an average across a mixed traffic mix moves for reasons that have nothing to do with your system and it will train people to ignore the page. There is also no alert on total request volume alone, which is a business metric that will fire every Monday morning. Both belong on a graph a human reads once a day.
The silent failures, the ones that return 200 and a confident answer
These are the failure modes that motivate the whole apparatus. Every one of them produces a successful response, a normal latency and a clean error log, and every one of them has a specific detector that costs very little to add once you know to look for it.
- The index stopped updating. Every answer is fluent, correctly cited and describes last month's policy. Nothing errors. The detector is the age of the newest document in the returned set, published as a gauge and alerted at your ingest cadence plus one cycle.
- A prompt edit doubled the refusal rate. Refusals are 200 responses containing a polite paragraph. Detect with a refusal classifier on a sample, or in the cheap version a small regex family covering the model's standard refusal openings, and alert on the rate rather than on any single case.
- The provider moved the alias. You requested a stable name and got a different underlying version. Log the model identity returned in the response rather than the one you sent, and alert when the count of distinct identities seen in a day rises above one.
- Context truncation dropped the system instruction. As conversation history or retrieved context grows, the oldest part of the assembled prompt falls off the front. The symptom is behaviour drift on long sessions only, invisible in the aggregate. Log assembled prompt token count and alert when it approaches your assembly ceiling.
- A tool schema changed underneath the agent. The model now passes a string where an integer is expected, the tool errors, the agent apologises and improvises an answer from memory, and the run completes successfully by every transport measure. Alert on tool error rate per tool, not on run error rate.
- One tenant dominates the aggregate. A single large customer with unusual documents moves the global average and hides a regression affecting everyone else. Every quality metric needs a per tenant breakdown, or at minimum a per tenant alert with a volume floor.
- The judge model was upgraded. Scores shift, the team celebrates or panics, and nothing about the system under test changed. Pin the judge by exact version, treat a judge change as a release with its own before and after run, and record the judge identity on every score.
- Valid JSON with invented values. Schema validation passes because the shape is right and the content is fabricated. Only a groundedness check against the retrieved source, or a human sample, catches this one. Structural validity is necessary and never sufficient.
The practical consequence is that your first customer report of a quality problem will arrive while every panel is healthy, and the on-call engineer will lose thirty minutes deciding whether to believe the customer or the dashboard. Say this out loud in the runbook. Write the sentence into the top of the on-call page: transport being green is expected during output and outcome incidents, and it is not evidence against the report.
The taxonomy behind most of these is covered in how AI agents fail. The point of listing them here is that each one maps to a cheap detector, and the detector is worth building before the incident rather than after, because after the incident you will build the one detector that would have caught the last one.
Online evaluation: judging a sample of live traffic
Offline evals gate changes you make. Online evaluation catches changes you did not make. You need both, they answer different questions, and running only the first is the most common gap in an otherwise well built system.
Sample by segment rather than uniformly. A uniform sample of live traffic is dominated by your most common case, which is also your best understood case, so it tells you the least per unit of judging cost. Stratify by the dimensions where behaviour actually differs: document type, language, customer tier, whether retrieval returned anything, whether a guard rail fired. Oversample the rare and expensive segments deliberately, weight the results back when you report a headline number, and always keep a small unweighted random slice for the failure modes you have not thought of yet. The mechanics of building the fixed side of this sit in writing evals for systems that are not deterministic.
How to threshold a rate that is naturally noisy
Do not put an absolute threshold on a rate computed from a small denominator, because binomial noise will breach it regularly and the team will mute the alert within a fortnight. Use two gates on every rule: a comparison against a rolling baseline from the same hour of the same weekday, and a minimum number of events before the rule is allowed to fire at all.
The arithmetic is worth internalising because it decides how sensitive you are allowed to be. For a rate p measured over n runs, the standard error is the square root of p times one minus p divided by n. Take a refusal rate sitting at three percent measured over one hundred runs: the standard error is about 1.7 percentage points, so an observed six percent is under two standard errors from baseline and will occur by chance often enough to wake you up for nothing. Push n to one thousand and the same standard error falls to about half a point, and that same six percent becomes a real signal. Sensitivity is bought with volume, and if you do not have the volume you have to accept a longer detection window instead.
Binomial arithmetic on your own numbers. Put your baseline rate, the rate you just observed, and the number of runs the observation came from. The third output tells you how many runs you would need before a shift of that size clears three standard errors, which is a sensible bar for something that pages a human.
Two more rules keep the pager quiet. Require the breach to persist across a short window and a long window before it fires, which is the burn rate pattern from ordinary service alerting and works identically here. And write the rule as a ratio to baseline rather than a fixed number, because a fixed number encodes today's traffic mix and will be wrong the first time a new customer segment arrives.
The failure mode of AI monitoring is not missing alerts, it is too many. A rule that fires falsely twice gets muted, and a muted rule is indistinguishable from no rule at all except that it appears on the dashboard and reassures people. Audit your muted and snoozed rules monthly, and delete rather than mute anything you no longer act on, so the list reflects what is actually being watched.
Token spend is a reliability signal, not just a finance one
Cost moves before errors do. An agent that has entered a retry loop, a retrieval config that quietly started returning forty chunks instead of eight, a prompt that grew a large example block, a conversation history that is no longer being truncated: all of these show up as tokens per run long before they show up as a failure. Put spend on the reliability dashboard, not only on the finance one.
- Alert on tokens per run at p95, not on the daily total. The total is dominated by volume and will hide a per run doubling behind a quiet Tuesday.
- Set a hard per run token ceiling and a hard step ceiling in code, and count terminations against each separately. A ceiling with no counter is a silent truncation.
- Set a per tenant daily budget with a soft alert and a hard stop. One customer uploading a thousand page document is the usual cause of a spend spike, and it is also an availability problem for everyone else.
- Track cached against uncached input tokens as separate series. A cache hit rate that collapses is a config regression that looks exactly like a price rise.
- Attribute every call to a feature and a tenant at call time. Reconciling an untagged provider invoice to a feature after the fact is not possible, and the reconciliation is what makes the whole series trustworthy.
The unit arithmetic for turning those series into a per run figure, using whatever your provider charges on the day you read it, is in token cost arithmetic.
The runbook, and alert rules you can copy
An AI on-call runbook differs from a service runbook in one way that matters: the first action is almost never a restart. Restarting a stateless model call fixes nothing, and the reflex wastes the first ten minutes of every incident. The first action is to establish which plane the problem lives on, because that determines who can fix it and how long it will take.
- Establish the plane
Check provider error rate and latency first. If transport is genuinely degraded, treat it as an ordinary dependency incident and follow the backoff and shedding policy. If transport is clean, which it usually is, move immediately to output and behaviour and stop looking at the infrastructure.
- Pull ten affected traces end to end
Not metrics, traces. Read the assembled prompt reference, the retrieved chunk ids and their scores, the tool calls with the arguments the model produced, and the raw output. Nine times out of ten the cause is visible in the first three traces, and no dashboard would have shown it.
- Check the four things that change without a deploy
The model identity returned by the provider, the index version and its freshness, any prompt or template edit shipped outside the normal release, and the tool schemas of every dependency. These four are the usual causes of an incident with no corresponding deploy in your own repository.
- Mitigate before you diagnose
Pin the previous model version, roll the prompt back to the last released hash, raise the human review share, or route the affected segment to the fallback path. All four should be a config change, not a deploy. If any of them requires a code release, that is the first thing to fix after the incident.
- Add the detector that would have caught it
Every incident review ends with one new signal and, ideally, one deleted signal. Monitoring that only ever grows becomes noise, and noise is how the next incident goes unnoticed. Record the detector in the same file as the rule set so the reason it exists is not lost.
# alerts.yml
# Every number below is a starting shape, not a benchmark. Set your own from two
# weeks of your own traffic, then keep the rule and change only the number.
# Two gates on every rule: a minimum denominator, and a comparison to a rolling baseline.
defaults:
min_events: 200 # never fire on a denominator smaller than this
baseline: rolling_7d_same_hour
compare: ratio_to_baseline # relative, not an absolute value pinned in 2026
rules:
- name: schema_validation_failures
plane: output
query: rate(llm_output_invalid_total) / rate(llm_output_total)
fire_when: ratio > 2.0
for: 15m
severity: page
runbook: pin the previous prompt version and the previous model id, then diff
- name: refusal_rate_shift
plane: output
query: rate(llm_refusal_total) / rate(llm_output_total)
fire_when: ratio > 1.5
for: 1h
severity: ticket
runbook: read the last prompt commit and the last safety setting change
- name: retrieval_returned_nothing
plane: behaviour
query: rate(retrieval_empty_total) / rate(retrieval_total)
fire_when: absolute > 0.02
for: 15m
severity: page
runbook: check index freshness first, embedding model id second
- name: index_staleness
plane: behaviour
query: max_chunk_age_hours
fire_when: absolute > 26 # your ingest cadence plus one cycle of slack
for: 30m
severity: page
runbook: the ingest job is the suspect, not the model
- name: model_identity_changed
plane: behaviour
query: count_distinct(response_model_id) over 24h
fire_when: absolute > 1
for: 0m
severity: ticket
runbook: an alias moved under you; pin the exact version and rerun the eval suite
- name: step_ceiling_terminations
plane: behaviour
query: rate(run_terminated_step_limit) / rate(run_total)
fire_when: ratio > 2.0
for: 30m
severity: ticket
runbook: look for a tool returning errors the agent is retrying in a loop
- name: tokens_per_run_p95
plane: behaviour
query: histogram_quantile(0.95, tokens_per_run)
fire_when: ratio > 1.4
for: 1h
severity: ticket
runbook: spend moves before errors do, so find the loop before the invoice does
- name: human_override_rate
plane: outcome
query: rate(result_overridden_total) / rate(result_reviewed_total)
fire_when: ratio > 1.3
for: 24h
severity: ticket
runbook: read twenty overridden runs; treat as a quality regression until proven otherwise
- name: provider_errors
plane: transport
query: rate(provider_5xx_total + provider_429_total) / rate(provider_calls_total)
fire_when: absolute > 0.05
for: 5m
severity: alert
runbook: confirm backoff is active before escalating to the provider
Adapt the query syntax to whatever you actually run. The part worth copying is the structure: each rule names its plane, carries a minimum denominator, compares against a rolling baseline rather than a fixed number, and has a one line runbook attached to the rule itself rather than in a wiki page nobody opens at four in the morning.
Definitions, and a two week instrumentation plan
- Silent failure
- A run that returns a successful response containing a wrong, stale or fabricated answer. It produces no error, normal latency and a healthy dashboard, which is why detection has to come from properties of the output and from what humans did with it rather than from the transport layer.
- Groundedness check
- An automated test that every factual claim in a generated answer traces back to a chunk that was actually retrieved for that run. It catches valid output containing invented content, which schema validation cannot detect because the shape is correct.
- Online evaluation
- Scoring a stratified sample of live production traffic on a schedule, using automated property checks and a version pinned judge model, to detect quality drift that no deployment caused.
- Override rate
- The share of AI outputs that a human corrected, replaced or escalated rather than accepting. It is the highest signal quality metric available in most systems because it needs no ground truth, only an instrumented accept or edit action in the interface.
- Two gate alert rule
- An alert that fires only when both conditions hold: the metric has moved by a stated ratio against a rolling baseline from the same hour of the same weekday, and the denominator exceeds a minimum event count. The second gate is what stops binomial noise paging a human.
Generate a correlation id at the trigger, carry it through every model call, tool call and downstream write, and make it visible on the customer facing artifact. Nothing else in this plan works without it, and retrofitting it later is far more expensive than it looks.
Emit steps per run, tool calls, tokens in and out, terminal reason, model identity returned, retrieval count and newest chunk age. Add schema validation with the failing path recorded. All of this is free to compute and covers most silent failures.
Run a full week with no alert rules at all and look at the distributions. Setting thresholds before you have seen a week of your own traffic produces rules calibrated to your imagination, and those are the rules that get muted.
Write the rule set as a file in the repository with min_events and ratio_to_baseline on every rule and a runbook line attached to each. Start with page severity on output only, and leave everything else as a ticket for the first month.
Instrument accepted, edited, overridden and escalated wherever a human touches the result, joined on the correlation id. It is the smallest piece of work on this list and the one that pays for the rest, because it turns your logs into an evaluation dataset for free.
Every incident review adds a detector and, where possible, removes one that has never fired usefully. Review muted rules monthly and delete rather than snooze, so the rule file always describes what is genuinely being watched.
If this is being built for an agent rather than a single model call, the trace shape and the permission boundaries matter more than the metric list, and that is the subject of agent observability. If the system is simple enough that one person can read every output for the first fortnight, do that instead of building any of this, and build the plan above only when the volume makes reading impossible. Monitoring you cannot yet interpret is a cost with no benefit.
ChatGPTalker, "Monitoring an AI System in Production" (2026). Monitor four planes rather than one: transport, behaviour, output and outcome. The status code of a serious AI failure is almost always 200, so detection has to come from output properties and from what humans did with the result, with every alert rule carrying a minimum denominator and a rolling baseline.
Questions readers ask next
Can we use our existing observability stack for AI monitoring?
How much live traffic should we judge with a model as a judge?
What is the single most valuable metric if we can only build one?
How do we monitor quality when there is no correct answer to compare against?
Should alerts page a human at night for an AI quality problem?
How do we tell a model change from a data change?
ChatGPTalker. "Monitoring an AI System in Production Without Alert Noise." chatgptalker.com, 2026-08-26. https://chatgptalker.com/guides/monitoring-ai-in-production/