Guardrails for a production AI system

A guardrailA check that runs around a model, not inside it, to keep a request or an action inside safe bounds is any check that sits around the model rather than inside it. The job of guardrails is simple to state: validate what goes in, validate what comes out, block actions you cannot undo, and when something goes wrong, fail in a safe and visible way. Done well, the reader never notices them. Done badly, they either let bad things through or block so much that the system is useless.

This guide covers the four layers worth building and how to keep them light enough that real users still get useful answers.

What is a guardrail in AI?

A guardrail is a deterministic check you control. It runs before, around, or after a model call, and it has the final say. A model instruction like "do not reveal secrets" is not a guardrail. It is a request the model usually honours and sometimes does not. Guardrails assume the model can be wrong, confused, or manipulated, and they hold regardless.

The model is the smart, fallible part. The guardrails are the dumb, reliable part. You want both.

The four layers map to that diagram: input validation, output validation, action gating, and safe failure. Treat them as separate concerns. A weak link in any one undoes the rest.

Layer 1: how do I validate the input?

Everything starts with what the user, or another system, sends in. You check the input before it ever reaches the model, for two reasons: to protect the model from junk and attacks, and to protect your costs and latency from obviously bad requests.

Useful input checks, roughly in order of value:

• Shape and size. Reject empty input, input over a length limit, and the wrong content type. This alone removes a surprising amount of noise.
• Authorisation. Confirm the caller is allowed to ask this, for this account, in this context. The model should never be the thing deciding who is allowed in.
• Known-bad patterns. Strip or reject content that tries to override your instructions, the classic prompt injectionInput crafted to make the model ignore its instructions and follow the attacker's instead. You will not catch every variant with patterns, which is why later layers exist, but you catch the lazy ones cheaply.Common gotchaDo not only screen what the user typed.Documents, web pages, and tool results that land in the model's context carry injection too, so run the same checks on retrieved and fetched text before it reaches the model.
• Sensitive data. Detect and redact things like card numbers or personal identifiers before they are logged or sent onward.

A practical habit: write input rules as data, not as scattered code. A small config block is easy to review, version, and reason about.

{
  "input": {
    "maxChars": 8000,
    "allowedContentTypes": ["text/plain", "application/json"],
    "rejectIfMatches": [
      "ignore (all|previous) instructions",
      "reveal your system prompt"
    ],
    "redact": ["credit_card", "email", "ssn"]
  }
}

Keep these checks fast and boring. Input validation that takes a second per request is a guardrail your users will learn to hate.

Layer 2: how do I validate the output?

The model produced something. Before you show it to a user or hand it to another system, check it. This is the layer people skip, and it is the one that catches the failures that reach customers.

Two kinds of output check matter.

Structural checks. If you asked the model for JSON in a specific shape, validate it against a schemaA formal description of the fields, types, and required keys an output must have before you trust it. If validation fails, you can retry once with the error fed back, or fall back to a safe default. Do not pass unvalidated model output straight into code that assumes it is well formed.

{
  "type": "object",
  "required": ["category", "confidence"],
  "properties": {
    "category": { "enum": ["billing", "technical", "other"] },
    "confidence": { "type": "number", "minimum": 0, "maximum": 1 }
  },
  "additionalProperties": false
}

Content checks. Scan the text for things that must never leave: secrets, internal hostnames, another customer's data, disallowed claims. For higher-stakes systems, a second cheap model pass can judge whether the output is safe or on-topic. That is the same idea as an evaluator step in the multi-agent architectures guide, used here purely as a gate.

A guardrail is only as good as what it does on failure. An output check that detects a leaked secret and then sends the message anyway is decoration. Wire the failing path to a safe refusal, not to the original output.

Layer 3: how do I gate dangerous actions?

Reading is cheap to get wrong. Acting is not. The actions that need a guardrail are the ones that are hard or impossible to reverse: sending an email, charging a card, deleting a record, writing to a production database, posting in public.

The rule of thumb: the harder an action is to undo, the more it should cost to perform.

When the model drives tools, the cleanest place to enforce this is the tool layer, not the prompt. Give each tool a clear definition and an explicit permission posture, and keep the irreversible ones behind a confirmation. If you connect tools through MCP, describe each server's capability narrowly so a tool can only do the one thing it is meant to do.

With the Claude Code CLI you can see this concretely. Permission rules decide what runs without asking, what asks first, and what is forbidden outright. You scope them so safe, frequent commands run freely and dangerous ones are denied.

# Inspect and change what the agent is allowed to do
claude config get permissions
claude config set permissions.deny '["Bash(rm -rf *)", "Bash(git push --force *)"]'

For programmatic checks, a hookA script Claude Code runs automatically at a lifecycle point, such as before a tool call on the PreToolUse event can inspect a tool call and block it before it runs. The hook is a script that receives the tool input on stdin and decides. Returning a deny decision stops the call; a normal exit lets the usual permission flow continue.

{
  "hooks": {
    "PreToolUse": [
      {
        "matcher": "Bash",
        "hooks": [
          { "type": "command", "command": ".claude/hooks/block-destructive.sh" }
        ]
      }
    ]
  }
}

The hook itself reports its decision as JSON. A permissionDecision of deny blocks the tool call and feeds the reason back; ask escalates to a confirmation; allow lets it proceed.

{
  "hookSpecificOutput": {
    "hookEventName": "PreToolUse",
    "permissionDecision": "deny",
    "permissionDecisionReason": "Destructive command blocked by policy"
  }
}

This is the heart of action gating: a deterministic check, outside the model, with the final word on whether an irreversible thing happens.

Layer 4: how do I make the system fail safely?

Things will go wrong. A check will reject, a model will time out, a tool will error, a budget will run out. What separates a system you can run from a demo is what happens next.

Safe failure has a few properties:

• Closed, not open. When a guardrail is unsure, it blocks. A spam filter that passes everything when it crashes is worse than no filter. Default to the safe outcome.Common gotchaThe quiet killer is the try/catch around a check that returns "allow" on error, so the day your content scanner throws, every output sails through unchecked and you never notice.Make the catch branch return a block, not a pass, and log the exception loudly so a broken guardrail surfaces as refusals instead of silent leaks.
• Honest to the user. Return a clear refusal or a partial result with the reason, not a fabricated answer and not a raw stack trace. "I cannot complete that action automatically; it has been sent for review" beats both.
• Bounded. Cap retries, set a token and cost ceiling per request, and define a stop condition. An agent that retries forever is its own outage.
• Logged. Every rejection, block, and fallback gets recorded with enough context to understand it later. You cannot improve a guardrail you cannot see firing.

Backoff matters here too: wait a little longer before each retry so you do not hammer a service that is already struggling. After the last retry, hand off to the safe fallback rather than looping.

How do I keep guardrails from blocking real users?

This is the part teams get wrong in the other direction. It is easy to add so many checks that legitimate requests get blocked, every answer is hedged into uselessness, and users route around your system entirely. A guardrail that everyone disables is worth nothing.

Some ways to keep guardrails proportionate:

• Match the check to the stakes. A read-only summary needs almost nothing. A refund needs a lot. Do not put the refund-level process on the summary.
• Prefer narrow rejects over broad ones. Block the specific dangerous thing, not the whole category around it. Overbroad rules generate false positives, and false positives are how users learn to ignore you.
• Measure the false positive rate. Track how often each guardrail blocks something that was fine. A rule with a high false-positive rate is a bug, not safety.
• Make refusals recoverable. When you block, tell the user what would make the request acceptable. A dead end frustrates; a path forward keeps trust.
• Tune from real logs. Your logs show which checks fire, on what, and whether the block was right. Tighten and loosen based on that, not on guesses.

The goal is not maximum blocking. It is the lowest amount of friction that still holds the line on the things that matter.

A starting checklist

When you are about to put a model in front of real users or real systems, walk this list:

• Input is size-limited, type-checked, and authorised before it reaches the model.
• Obvious injection and sensitive data are stripped or redacted on the way in.
• Structured output is validated against a schema before any code trusts it.
• Output is scanned for content that must never leave, with a safe path on failure.
• Irreversible actions are gated behind confirmation or human approval at the tool layer.
• Every guardrail fails closed, logs the event, and surfaces an honest reason.
• Retries, tokens, and cost are bounded per request.
• You track the false-positive rate and tune from logs, not guesses.

You can ask Claude to audit an existing system against exactly this shape before you ship.

Guardrails are not the exciting part of an AI system, and that is the point. They are the boring, deterministic layer that lets the smart, unpredictable model be useful in front of real users without putting anything important at risk. Build the four layers, keep them proportionate, and tune them from what you see in the logs.