The Guardrail Has to Live Outside the Model

The first fix everyone reaches for is the same one: tell the agent not to do the bad thing. Put it in the system prompt. “Never access another customer’s account. Never transfer funds without confirmation.” Ship it, watch it work in testing, move on.

It will keep working right up until someone phrases the same request differently enough, or buries it inside something that looks like a normal turn. Then it won’t. Understanding why takes one honest look at what a system prompt actually is.

Why can’t the system prompt just handle it?

A system prompt and a user’s message live in the same place: a sequence of tokens the model attends over before producing the next token. There’s no separate channel where instructions carry more authority than input. “Never do X” is not a rule the model enforces - it’s a string that influences a probability distribution, the same way the user’s message does. If that string sits far enough from the decision point, or gets crowded out by more recent context, its influence just fades. Nothing breaks. Nothing errors. The model simply weighs it less.

And even in the best case, where the instruction sits right at the top of context with nothing competing for attention, the output is still sampled, not branched. There’s no if statement in there. A well-behaved model with a well-written prompt is still, at bottom, rolling weighted dice on every token. You can load the dice heavily in your favor. You can’t remove the dice.

That gives you the first real constraint: a safety check can’t live inside the same generation pass as the action it’s supposed to govern. If the thing deciding “is this allowed” is the same stochastic process as the thing proposing “here’s what I want to do,” you don’t have a check. You have a second opinion from someone who already agreed with the first opinion.

Does swapping in a classifier actually fix that?

A common next step is to add a second, smaller model in front of the main agent - a lightweight classifier that screens incoming messages for jailbreak attempts before they ever reach the agent doing the real work. This is closer to right, but it’s worth being precise about what it does and doesn’t buy you.

The classifier is still a model. It’s still sampling. What’s changed is scope: instead of one large, general-purpose model trying to simultaneously hold a conversation and police itself, you have a small, narrow model doing exactly one job. Specialization reduces variance. It does not eliminate it. You’ve traded “unbounded failure inside a general system” for “bounded failure inside a narrow one” - a real improvement, but still a probabilistic gate guarding a probabilistic actor.

The actual fix is structural, not statistical: put a deterministic, non-LLM function at the one point that matters most - not judging the model’s reasoning, but checking the specific action it’s about to take.

Here’s what that point looks like concretely. Say a customer asks the agent to check a balance, or move money between accounts. The LLM doesn’t just answer in prose - it decides which tool to call and with what arguments: check_balance(account_id="acct_001"), or a transfer with a source account, a destination account, and an amount. That proposed call is a fact, not a guess - it names an exact account and an exact action. The guardrail’s job is to sit in the gap between the model producing that call and the system actually running it, and ask one narrow question: is this specific action, with these specific arguments, something this specific customer is allowed to do?

A banking support agent is a good place to see this cleanly, because that gap is unambiguous: right before a tool call - transfer money, close an account, pull transaction history - actually runs. Here’s roughly what the check looks like, stripped to the shape that matters:

def check_ownership(session, tool_call):
    account_id = tool_call.arguments["account_id"]
    owner = data_store.get_owner(account_id)
    if owner != session.customer_id:
        return Decision(allowed=False, reason="ownership_mismatch")
    return Decision(allowed=True)

Two things about this deserve attention. First, it’s ordinary code - no model call, no prompt, nothing stochastic. Given the same inputs, it returns the same answer, every time, forever. Second, and more important: it checks the proposed tool call’s actual arguments, not what the model said it was going to do in its reply text. An agent can hallucinate a confident, articulate explanation of an action it isn’t actually about to take. It cannot hallucinate its way past a check that reads the real argument off the real function call.

That ownership check is one of four checkpoints a system like this needs, each looking at a different moment in the conversation: an input check on the way in, screening the raw message for attacks before it reaches the agent; a dialog check on what’s being asked, classifying the message into a known type of request; an execution check right before an action executes - the one above; and an output check on the way out, screening the reply before it reaches the customer, in case it accidentally names an account or balance that isn’t theirs. Four separate, deterministic gates. None of them the model marking its own homework.

What happens to everything the guardrail doesn’t recognize?

That’s the dialog check’s job: classify what kind of request it’s looking at - “this is a balance check,” “this is a transfer,” “this is something else entirely.” It has the same blind spot as everything else: it works by matching input against a set of known patterns, and input that doesn’t cleanly match any of them is going to happen, regularly, the moment real users start typing real sentences.

The question that actually matters is what happens by default when nothing matches. There are exactly two honest answers. Allow-by-default passes unmatched input through to the agent unchecked - convenient, and quietly dangerous, because “unmatched” and “malicious” are not the same category, but an allow-by-default posture treats them as equivalent. Deny-by-default routes anything unrecognized into a restrictive fallback - safer, at the cost of a worse experience for the ordinary users whose phrasing just happened to fall outside what was anticipated.

Deny-by-default is the right posture for anything touching money or another person’s data. But it isn’t free - someone has to keep teaching the system new phrasings as real usage reveals them, or the fallback bucket ends up swallowing traffic that should have been handled normally.

Should every request pay the same toll?

A four-checkpoint pipeline adds real latency to every turn, including the vast majority that are someone innocently asking for their balance. The obvious instinct is to skip checkpoints that don’t apply - a balance inquiry doesn’t move money, so why run it through an execution check at all?

Here’s the catch: skipping the execution check also skips the only record that a check happened. Without it, the only thing standing between a request and the data is whatever logic the tool itself happens to contain - unaudited, and invisible if it’s ever wrong. That’s why the policy below runs the check on every request type, including read-only ones:

check_balance:
  risk_tier: low
  rails: [input, dialog, execution, output]
  requires_human_confirmation: false
  hard_block: false

The saving doesn’t come from skipping the gate - it comes from the gate being cheap, a lookup rather than a model call, and from skipping confirmation, not execution. Risk-tiering means spending friction where risk is, not spending authority where risk is. Authority isn’t optional anywhere; friction is.

Who gets to decide where the risky line sits?

Which request types need confirmation, which get hard-blocked outright, what the default posture is for anything unrecognized - none of that is an engineering decision. It’s a risk-tolerance decision, and the people best positioned to make it are the ones closest to customers and complaints, not the people closest to the code.

Which means it can’t live inside a routing function or any other artifact only an engineer can safely edit. It has to live in something a policy owner can open, read, and change without asking anyone to redeploy anything - one plain config file, reloaded fresh on every request:

transfer_funds_own_accounts:
  requires_human_confirmation: true

Flip that single value to false, save the file, and the very next request behaves differently. No code touched. No deploy. That’s not a nice-to-have - it’s the actual test of whether policy and mechanism are separated or just pretending to be. If changing a risk decision still requires opening a pull request, the separation is cosmetic.

Is a block always the same kind of block?

Not every refusal deserves the same shape. A customer trying to transfer to a flagged external account has made an honest mistake or hit a real fraud screen - the door should stay open for a retry, or an escalation, once the situation’s actually resolved. A customer’s phrasing that happens to reference another customer’s account number is a different category of event entirely, and no amount of “yes, I’m sure” from a confirmation prompt should be able to talk that block into standing down.

That distinction - retryable versus never-overridable - has to be an explicit flag on each request type, not something inferred from how a refusal happens to read. A hard block and a soft block can produce nearly identical output text; only the policy entry behind them knows which one actually is which, and confirmation logic has to check that flag before it ever offers a “try again.”

The confirmation mechanism itself is worth a beat too. When a request needs a yes, the system doesn’t just print a question and hope the next message answers it - it holds the proposed action in a small pending store keyed to the session, and treats whatever the customer says next as only a reply to that specific pending action, not a fresh request re-entering the whole pipeline. Say anything affirmative, and the held action executes. Say anything else, and it’s dropped. Nothing about that mechanism can be talked into skipping a hard block, because a hard block never reaches the point where a pending confirmation gets created in the first place.

Does it matter which framework you use?

There’s more than one way to wire these four checkpoints together, and the choice matters less than it seems like it should.

One approach treats the whole conversation as an explicit state graph: each checkpoint is a plain function - a graph node - and the connections between them are conditional edges that decide, based on the outcome of one check, which check runs next. A framework like LangGraph exists specifically to make that kind of graph easy to build and reason about. Nothing about any individual node is special - it’s ordinary code, the same ownership check shown earlier, just wired into an explicit sequence instead of scattered through prompt instructions.

NeMo Guardrails (NVIDIA’s open-source toolkit for wrapping an LLM in policy checks) reaches the same architecture from the opposite direction: instead of code wiring functions together, you write Colang - a small rules language for describing canonical intents and the checks each one triggers. A flow for a balance request might look roughly like this:

define user express check balance
  "what's my balance"
  "how much do I have in checking"

define flow check balance
  user express check balance
  $decision = execute check_ownership(account_id=$account_id)
  if not $decision.allowed
    bot refuse "That account isn't linked to this session."
  else
    execute check_balance(account_id=$account_id)

Read that flow against the Python function from earlier and they’re doing the same job: match the intent, call a deterministic check, only proceed if it passes. Colang just expresses it as configuration instead of as code a Python interpreter runs directly.

Neither approach is more correct than the other. What has to hold is the architecture, not the tool: the check sits outside the model’s own generation pass, it runs against the real proposed action, and the risk decisions live somewhere a policy owner can edit without a redeploy. State graph, rules language, or something built from scratch - the framework is a delivery mechanism for that shape. It is not the shape.

Where this leaves you

The shape holds regardless of the label on the box: keep the check outside the generation pass, keep it deterministic, keep it looking at the real arguments instead of the model’s account of itself, and keep the risk decisions in a file a non-engineer can edit without you.

None of that answers the question this whole design keeps circling without quite landing on: if policy and mechanism are so obviously meant to be separable, why do so many real systems still end up with risk decisions quietly baked into code anyway? Is that laziness, or is there something about risk itself - as a concept - that resists staying cleanly outside the system that has to act on it?

What’s your take? Drop a comment below.