Guardrails without breaking things

The short answer

Guardrail work goes wrong in two directions and both are common. Either everything is instruction and the system does whatever a sufficiently strange input persuades it to do, or the checks pile up until the product is slow, refuses ordinary requests, and the team quietly turns half of them off. The way out is placing each rule where its cost matches its value, and measuring the false positive side with the same seriousness as the true positive side.

Five places a guardrail can live, and only two of them enforce

The word guardrail covers five mechanisms that behave nothing alike. Confusing them is why teams believe they have controls when they have preferences.

A useful test for any proposed guardrail: if the model produced the most damaging output you can imagine, would this rule still stop the damage. Instructions and filters answer no, because they operate on text. Action gates answer yes, because they operate on effects. Both kinds are worth having, and only one of them belongs in a sentence beginning with the words we prevent.

The placements compared

The false positive is the cost nobody budgets

Safety work is usually specified as recall: catch as much of the bad thing as possible. The number that decides whether the guardrail is usable is precision, and precision is dominated by the base rate rather than by the quality of the classifier. This surprises people every time, so it is worth doing the arithmetic on your own traffic.

Take ten thousand requests a day where half a percent genuinely violate policy, so fifty real cases. A check with 90 percent recall catches forty-five and misses five. The same check with a 3 percent false positive rate on the 9,950 clean requests raises about 299 false alarms. The flag queue therefore contains 344 items of which 45 are real, so roughly 13 percent of what a reviewer sees is genuine. Every one of the 299 is a real user who was stopped, and at three minutes each the queue costs about seventeen hours of review a day.

Move the false positive rate from 3 percent to 1 percent and watch what happens to the review hours. On rare events, tightening the false positive rate does far more for the usability of a guardrail than raising recall does, and it is the dial most teams never touch because nobody is measuring it.

Latency, streaming, and the check you cannot finish in time

Guardrails are usually specified as though they were free, then added one at a time until the product feels slow and someone removes the ones that are easiest to remove rather than the ones that matter least. Give the whole set a latency budget at design time and place each check according to what it can actually stop.

• Advisory input checks run in parallel with the main model call. If a check only tags or logs, it must never sit in front of the request. Start both, use the tag when it arrives, and discard it if it is late.
• Blocking input checks run first and serially, and there should be very few of them. Each one you add is added to every request in the product, including the 99 percent that were never going to be a problem.
• Output validation is cheap where it is structural. Schema checks, citation membership against the retrieved set and arithmetic verification are all local computation and cost effectively nothing. Model based output review is a second full call and should apply to a defined subset, not to everything.
• Retries have a budget too. A validator that fails and triggers a regeneration has doubled the cost and latency of that request. Cap it at one retry with a corrective instruction, then fail to a human rather than looping.

Streaming forces a genuine choice

If you stream tokens to the user, you cannot validate the complete output before they see part of it. There are three honest options and no clever fourth. Buffer the whole response and validate before sending anything, which costs the perceived speed that made streaming worth doing. Stream, and validate in chunks, which reduces exposure to a partial leak without eliminating it and can produce visible retractions. Or stream only in contexts where the worst partial output is tolerable, and buffer everywhere else. Pick per surface, write down which you chose, and be honest that chunked validation is a mitigation rather than a control.

The same reasoning applies to any downstream action. If an agent can call a tool mid-stream, output validation is not protecting you at all, because the effect has already happened by the time the check runs. That is another argument for putting the real rule at the tool boundary, which is covered further in agent permissions and scope.

Fail open or fail closed, decided per class of action

Every check can be unavailable: a timeout, a rate limit, a dependency outage. What happens then is a design decision, and a system with one global answer to it is wrong in one direction or the other most of the time.

Write the choice into configuration per action class rather than into scattered exception handlers, so that the answer to what happens when the moderation service is down is a file somebody can read rather than an archaeology exercise. And measure how often the fallback path fires. A check that has been silently failing open for three weeks is not a guardrail, and the only way you learn that is by counting the times it did not run.

Injection is the case where only code helps

The moment a system reads text it did not receive from the authenticated user, that text can contain instructions. Retrieved documents, web pages, email bodies, uploaded files, tool responses and the output of another agent are all untrusted channels. A model has no reliable way to distinguish an instruction from your developer from an instruction embedded in a document, because both arrive as text in the same context.

1. Separate the channels. Mark untrusted content clearly in the context and never derive permissions, identity, scope or amounts from it. Scope comes from the session object, always.
2. Bound the tools per request. The action set is assembled from the authenticated user's rights before the model sees anything, so a widening instruction has nothing to widen.
3. Plant canary tokens. Insert a unique token into each retrieved document. If it ever appears in output or in a tool argument, content is flowing somewhere it should not, and you have a detector rather than a hope.
4. Validate tool arguments against the request, not against the model's reasoning. A refund amount that appears nowhere in the user's message or the account record is rejected regardless of how convincingly the model explains it.
5. Alert on refused proposals. A model attempting an action outside its scope is a signal. Most systems throw that event away, which discards the earliest evidence of an attack that is available.

Injection defence and permission design are the same piece of work seen from two angles, and both belong in code that runs whether the model is having a good day or not.

A policy artifact you can copy

Guardrails scattered across handlers cannot be reviewed, and a rule nobody can review is a rule nobody can be sure is still there. Put the policy in one file that the tool layer reads at run time. The properties that matter are the per class fail behaviour, the human gate that escalates rather than auto-approving on timeout, the explicit untrusted input rule, and overrides that expire.

# policy/actions.yaml
# Enforced in the tool layer, before any call leaves the process.
# The model never reads this file, and nothing written in a prompt can change it.

defaults:
  on_check_timeout: deny       # fail closed unless a class below says otherwise
  log_every_decision: true     # allows and denies both, with an input hash
  tell_the_user: true          # a silent block is an unreported bug

classes:

  read_public:
    tools: [search_docs, get_faq]
    reversible: true
    on_check_timeout: allow    # a read failing closed hurts more than it protects
    checks: [input_schema]
    rate: { per_user_per_min: 30 }

  read_customer_data:
    tools: [get_account, get_orders]
    reversible: true
    checks: [input_schema, tenant_scope, pii_egress]
    scope: "account_id must equal the authenticated session's account_id"
    rate: { per_user_per_min: 10 }

  write_reversible:
    tools: [draft_reply, set_ticket_tag]
    reversible: true
    undo_window_minutes: 30
    checks: [input_schema, tenant_scope, output_contract, banned_content]
    rate: { per_user_per_hour: 60, global_per_hour: 2000 }

  write_irreversible:
    tools: [send_email, issue_refund, close_account]
    reversible: false
    checks: [input_schema, tenant_scope, output_contract, banned_content, amount_bounds]
    amount_bounds: { currency: GBP, max_per_action: 250, max_per_user_per_day: 500 }
    human_gate:
      required_when: ["amount > 100", "tier == enterprise", "self_reported_confidence < 0.8"]
      timeout_minutes: 240
      on_timeout: escalate     # never auto_approve an irreversible action on a timeout
    rate: { global_per_hour: 200 }

untrusted_input:
  # anything the model did not receive directly from the authenticated user
  sources: [retrieved_documents, web_pages, email_bodies, uploaded_files, tool_output]
  rule: "text from these sources can never grant an action or widen a scope"
  canary: "one unique token per document; any appearance in output raises an alert"

overrides:
  who: [support_lead, on_call_engineer]
  requires_reason_code: true
  expires_after_hours: 24
  reviewed_weekly_by: trust_and_safety

Two details are load bearing. on_timeout: escalate under the human gate exists because the alternative, auto-approving when nobody responded in four hours, converts an oversight mechanism into a delay. And tell_the_user: true is in the defaults because silent blocks generate support tickets that nobody can trace, since the log says the request succeeded and the user says nothing happened. Design of the human gate itself, including how to keep reviewers from approving by reflex, is covered in human in the loop design.

Overrides, appeals, and keeping the rules owned

A guardrail with no override path does not get respected, it gets routed around. Somebody will do the work in a spreadsheet, outside the system, with no logging at all, and you will have traded a measurable risk for an invisible one. Build the override deliberately.

1. Give every rule a named owner and a review dateat creation

   Not a team inbox, a person, plus the date the rule is next examined. Rules accumulate faster than anyone removes them, and an unowned rule is defended by nobody and understood by nobody within about two quarters.

2. Make the override explicit, scoped and expiringin the policy file

   Who can override, a required reason code from a closed list, and an automatic expiry. An override that persists forever is a rule change made without a review, and free text reasons are unanalysable, so the closed list is what makes the next step possible.

3. Review override reasons weeklytwenty minutes

   The distribution of reason codes is the highest quality signal you will get about which rules are miscalibrated. One code dominating means that rule is wrong for a class of legitimate work, and the fix is the rule rather than the reviewers.

4. Give users a visible appealin the product

   When a request is blocked, say that it was blocked, give a reference, and provide a route to a human. Appeals are labelled false positive data arriving for free, and a blocked user with no route is a churn event with no telemetry attached.

5. Re-evaluate every rule against the current trafficquarterly

   Replay a recent sample through the whole guardrail stack offline and count what each rule would fire on now. Rules that fire on nothing are dead weight adding latency, and rules that fire on far more than they did are either catching a new pattern or drifting into ordinary traffic.

6. Retire rules out loudwhenever earned

   Deleting a guardrail feels dangerous, so nothing ever gets deleted and the stack grows until the product is unusable. Retire with the same ceremony as adding: a note saying what it caught, what replaced it, and who agreed.

Definitions, and the check before you turn it on

Definitions

Guardrail

A constraint on what an AI system may output or do. It is a control only where it is enforced outside the model, in code the model cannot influence, which in practice means the tool boundary and the layer that runs after the model has produced its output.

Action gate

Code sitting between a model's proposed tool call and its execution, checking scope against the authenticated session, amount bounds, rate limits and any condition requiring a human. It is unaffected by what the model was persuaded to believe, which is what distinguishes it from an instruction.

Fail closed

Denying an action when a required check cannot be completed, for example when a moderation service times out. The correct setting for irreversible actions, and the wrong setting for ordinary reads, where it converts a dependency outage into a full product outage.

Prompt injection

Text in an untrusted channel, such as a retrieved document or an email body, that the model treats as an instruction. It is not solved by prompt wording, because the model receives both instructions as text in the same context, and is contained by deciding available actions from the session in code.

Base rate

The share of traffic that genuinely violates a policy. It dominates the precision of any check: at a low base rate, even a small false positive rate produces a flag queue mostly made of legitimate users, regardless of how well the check detects real violations.

Blast radius control

Measures that limit damage after something wrong has already happened: reversibility windows, per tenant and per hour caps, per tool kill switches and a ledger of every side effect. It assumes the preventive layers failed, because occasionally all of them do at once.

The honest summary for anyone deciding how much of this to build: guardrails do not make a model correct, and they are not a substitute for evaluating it. What they do is bound the consequences of it being wrong, which is a different and more achievable goal. Combined with the logging described in what to log in AI systems, that is most of what evaluation and guardrails means in practice: knowing what the system did, and making sure the worst thing it can do is survivable.