AI Guardrails Explained: Input, Output, and Tool-Call Guardrails for LLM Apps and Agents (NeMo, Guardrails AI, Llama Guard, Jev) — 2026

What are AI guardrails, and how are they different from model alignment?

An AI guardrail is a check that runs outside the model, around a call to it, and decides whether the input, the action, or the output is allowed to proceed. The ICML 2024 position paper Building Guardrails for Large Language Models puts it in one sentence: "Guardrails, which filter the inputs or outputs of LLMs, have emerged as a core safeguarding technology." For agents, add a third surface — the tool calls in between — and you have the whole subject.

The distinction that matters most is guardrails versus alignment. Alignment is the safety behaviour the vendor trained into the weights: the model refuses to write malware, declines to dox people, and so on. You did not write it, cannot inspect it, and cannot change it. Guardrails are the behaviour you enforce, in code you own, with a policy you chose and a log you can read. They are complementary, and neither replaces the other. Palo Alto Networks' Unit 42 made that concrete in a 2025 test of three cloud LLM platforms, noting that when "internal model alignment is insufficient, output filters may not reliably catch harmful content" — and, the other way around, no amount of alignment stops a model from executing a destructive tool call it was socially engineered into. Our AI coding agent security post catalogues the real CVEs where exactly that happened.

Three things a guardrail is not, because the word is used loosely in vendor marketing:

Where do guardrails sit? Input, retrieval, tool-call, and output checkpoints

The most useful way to classify guardrails is by where they run, because placement decides what the check can see. NVIDIA's NeMo Guardrails splits its rails into input, dialog, retrieval, execution, and output; the OpenAI Agents SDK has input, output, and tool guardrails; LangChain hooks middleware before and after the agent and around model and tool calls. They are describing the same four checkpoints.

Input guardrails are where everyone starts and where the free wins are: length limits, a moderation classifier, a PII scan, an allow-list of topics for a narrow assistant. The OpenAI Agents SDK docs make a point worth copying — its input guardrails can run before the agent starts (blocking mode) so that a failed check never spends model tokens, or in parallel with the agent (the default) to save latency when most inputs are fine.

Retrieval guardrails exist because of indirect injection. OWASP distinguishes direct injection (the user's own prompt) from indirect injection, where "external sources (websites, files) contain content that changes model behavior when processed." Anything your RAG pipeline or an MCP server returns is user-grade input and should get the same scan as the user's message. Most teams skip this layer; it is where the coding-agent CVEs lived.

Tool-call guardrails are the ones that matter most for agents and the ones most often missing. They are different in kind: instead of judging text, they judge a concrete action — tool name, arguments, the session's permissions. That is the only point in the loop where "the model was tricked" and "the model decided on its own" collapse into the same question: should this action run? It is also the checkpoint that composes with permissions. If the credential the agent holds cannot delete production tables, no injection can make it. OWASP's mitigation list — least privilege, "require human approval for high-risk operations" — is a description of this layer.

Output guardrails cover two different risks. The first is content: unsafe text, leaked PII, an answer that contradicts the retrieved sources. The second is less discussed and more dangerous in agentic systems: the output is consumed by a machine. OWASP's LLM05:2025 Improper Output Handling lists LLM output executed in a shell (remote code execution), unvalidated Markdown or JavaScript returned to a browser (XSS), and LLM-generated SQL without parameterisation (injection). Its first recommendation is the right mental model for the whole checkpoint: "Treat the model as any other user, adopting a zero-trust approach, and apply proper input validation on responses."

How are guardrails implemented? Rules, guard models, judges, decision models, policy engines

Placement says where a check runs; mechanism says how it decides. Every guardrail product is a bundle of one or more of the five mechanisms below, and the mechanism — not the brand — determines latency, cost, what it can catch, and how it fails.

Rules and schema validation are underrated because they are boring. A regex for credit-card numbers, a JSON Schema the output must parse against, a 4,000-character input cap, an allow-list of tools per user tier — each is free, instant, and cannot be argued with. LangChain's guardrails docs frame the trade-off exactly: deterministic guardrails are "fast, predictable, and cost-effective, but may miss nuanced violations." Put every policy you can state precisely here, and only reach for a model when you can't.

Guard and classifier models add semantics. Meta's Llama Guard 3 (1B, 8B, and 11B-Vision) classifies a prompt or a response as safe or unsafe and lists the violated categories from the MLCommons hazard taxonomy — S1 Violent Crimes through S13 Elections, plus S14 Code Interpreter Abuse on the 8B model. OpenAI's moderation endpoint does the same job as a free hosted call across 13 categories, with image support. Both are excellent at what they cover and silent about everything else: neither knows that your assistant must not discuss competitor pricing or that DROP TABLE is off-limits on Fridays.

LLM-as-judge is how most teams cover the "everything else": prompt a general model with your rubric and ask for a verdict. It works for nuanced, low-volume checks and for offline evaluation. As an inline guardrail it has two structural problems. It doubles your latency and cost, because the judge is a full autoregressive call. And the number it returns when you ask for a confidence is generated text, not a probability — the argument we make in detail in Jev vs LLMs vs traditional ML for classification. Thresholding on it is thresholding on a vibe.

Calibrated decision models are the newer middle: a non-autoregressive model that answers a question you phrase at request time with a typed verdict and a confidence trained to track accuracy. Jev is the example we have covered most; TypeSafe lists "guardrail, and detect jailbreaks of LLM prompts, reasoning traces, and/or outputs" among its use cases and quotes 70–500 ms end to end at $0.042 per million input tokens. The appeal for guardrails is specific: you get a custom policy question ("should this tool call be blocked?") with a thresholdable number, without training a classifier or paying for a judge. The limits are equally specific and we return to them in the agent-loop section.

Policy engines and human approval are the layer that does not involve judging anything: the agent's credential is scoped so the action is impossible, a spend cap is enforced by the gateway, a sandbox contains the shell, and irreversible actions wait for a human. LangChain ships this as HumanInTheLoopMiddleware for "financial transactions and transfers, deleting or modifying production data, sending communications to external parties." It is slow when it fires, which is why it pairs with a model-based gate that decides when it should fire.

NeMo Guardrails vs Guardrails AI vs Llama Guard vs OpenAI Moderation vs framework-native: the tooling landscape

These get lumped together in search results as "LLM guardrails tools," but they occupy different cells of the placement × mechanism grid above. Read the second and third columns to see which checkpoint and which mechanism each one actually gives you; most production stacks use two or three of them.

Two contrasts are worth spelling out. NeMo Guardrails versus Guardrails AI is the comparison people search for, and the honest answer is that they are shaped for different apps. NeMo is conversation-flow-centric — you write Colang flows that steer a dialogue and attach rails at five points, which fits chat assistants and RAG where you want to control topic and tone. Guardrails AI is validator-centric — you compose Validators from the Hub into a Guard and it doubles as a structured-output layer over Pydantic, which fits pipelines where the output must be typed and checked. Both are Apache 2.0. Neither gates tool calls with a calibrated verdict out of the box, which is why the next section exists.

Guard models versus provider-native safety is the other. Anthropic's Constitutional Classifiers research is the most transparent public account of what a serious input-plus-output classifier layer costs and buys: on their evaluation, jailbreak success fell from 86% on the unguarded model to 4.4%, the refusal rate on harmless queries rose by 0.38 percentage points, and compute rose 23.7%. During a two-month red-team of the prototype, 183 participants spent an estimated 3,000-plus hours without finding a universal jailbreak; a later public demo with 339 jailbreakers over roughly 3,700 hours did find one. Three lessons travel to your own stack: good classifiers can cut the attack rate by an order of magnitude, every point of recall costs you some over-refusal and some latency, and "no jailbreak found" has an expiry date.

How do you place guardrails in an agent loop?

A chat app has an input and an output. An agent has a loop: the model plans, calls a tool, reads the result, and plans again — and every pass through that loop is a new chance for retrieved or returned text to steer it. Guardrails for agents therefore have to be placed on the loop, not just at its ends. The pattern that holds up:

As pseudocode. Illustrative only — the control flow, not real SDK calls:

# ILLUSTRATIVE PSEUDOCODE — control flow only, not real API syntax.

def handle(user_msg, session):
    # ① INPUT guardrails — cheapest first, short-circuit on failure
    if rules.blocks(user_msg):                    # length, regex (PII, secrets), allow-lists
        return refuse("policy")
    if guard_model.classify(user_msg).unsafe:     # Llama Guard / moderation API
        return refuse("unsafe_input")

    context = retrieve(user_msg)                  # UNTRUSTED — same scan as user text
    context = [c for c in context if not injection_scan(c).flagged]

    while True:
        step = llm.next_step(session, user_msg, context)

        if step.is_tool_call:
            # ② TOOL-CALL guardrail — judge the ACTION, on fields your code assembled
            if step.tool in NEVER_AUTO:           # payments, deletes, external sends
                step = await human_approval(step)
            else:
                verdict = decision_model.decide(
                    question="should this tool call be blocked?",
                    input=dict(tool=step.tool, args=step.args, user_tier=session.tier),
                    # no tool output or retrieved text in here — only the action
                )
                log(step, verdict.label, verdict.confidence)
                if verdict.label == "block" or verdict.confidence < 0.9:
                    step = await human_approval(step)   # uncertain → escalate, not allow
            result = run_tool(step)               # least-privilege credential, sandboxed
            session.append(scan(result))          # tool output re-enters as UNTRUSTED input
            continue

        # ③ OUTPUT guardrails — before any user or downstream system sees it
        answer = pii.redact(step.text)
        validate_schema(answer)                   # typed output? fail closed if it won't parse
        if judge.flags(answer, user_msg, context): # grounding / safety judge if stakes justify latency
            return refuse("unsafe_output")
        log_outcome(session.id, answer)
        return encode_for_destination(answer)     # HTML-escape / parameterise — OWASP LLM05

The asymmetry in the tool-call block is the whole design. The threshold is on allow, not on block: an action runs unattended only when the verdict is confidently "allow," and everything else goes to a human. That encodes the safe default into control flow, so a drifting or injected verdict degrades into "slightly more approvals," never into "silently executed." It is the same shape we recommend for the routing decision in LLM routing, where the expensive model is the default and the cheap one has to earn the request.

Why the insistence on keeping fetched text out of the verdict's input? Because model-based gates can be moved by what they read. VentureBeat reported an Octomind engineer's demo in which Jev's block probability for a tool call fell from 0.76 to 0.48 — and its confidence from 0.64 to 0.22 — after adversarial text claiming the action was pre-approved was added to the input. TypeSafe's own limitations documentation says the same thing without the drama: content "written to adversarially steer the model … can move the answer." This is not a Jev problem; it is a property of every classifier and judge that reads untrusted text. The fix is architectural: the gate judges the action, your code assembles the fields, and the model's self-justification never gets a vote. If your agent runs on a framework that already has approval hooks — the Hermes Agent security post walks through one — this is the layer to wire them into.

What do guardrails cost in latency and money?

Every guardrail is on the request path, so the budget question is real. Rough orders of magnitude, which you should replace with your own measurements:

Three ways to buy latency back. Run independent checks in parallel — the OpenAI Agents SDK's default for input guardrails runs them concurrently with the agent and cancels if a tripwire fires. Run cheap checks first so the expensive ones only see what survived. And gate actions rather than every message: an agent session may have one or two tool calls that matter and twenty model turns that don't. The same accounting applies to the model call itself — if you have already built a routing layer, the guardrail verdict and the routing verdict can share one cheap decision call.

How do you evaluate guardrails? False positives and false negatives on your own traffic

A guardrail is a binary classifier with asymmetric costs, so evaluate it as one. The published evidence that vendor defaults are not interchangeable is stark. Unit 42's 2025 comparison sent 1,000 benign prompts and 123 JailbreakBench attacks through three anonymised cloud LLM platforms with every filter at its strictest setting. Benign prompts blocked (false positives): 0.1%, 0.6%, and 13.1% — the last platform was flagging code-review and maths questions. Jailbreaks that got past the input filter (false negatives): 47%, 9%, and 8%. Role-play framing was the dominant evasion tactic everywhere. The same label, "guardrails enabled," covered a 130× spread in false positives and a 6× spread in false negatives.

So measure on your traffic. The protocol:

If you already have an evaluation harness for RAG or agents — the kind in our RAG evaluation metrics guide — guardrail evaluation slots in as one more configuration to compare, with attack cases added to the dataset.

Common AI guardrail mistakes

Frequently Asked Questions

References

• • Dong et al. 2024 — Building Guardrails for Large Language Models (ICML 2024, arXiv)
• • OWASP — LLM01:2025 Prompt Injection
• • OWASP — LLM05:2025 Improper Output Handling
• • NVIDIA — NeMo Guardrails (GitHub)
• • Guardrails AI — guardrails (GitHub)
• • Meta — Llama Guard 3 model card and prompt format
• • OpenAI — Moderation guide (omni-moderation-latest)
• • OpenAI Agents SDK — Guardrails (input, output, tool guardrails, tripwires)
• • LangChain — Guardrails (middleware, PIIMiddleware, HumanInTheLoopMiddleware)
• • Anthropic — Constitutional Classifiers: Defending against universal jailbreaks
• • Palo Alto Networks Unit 42 — How Good Are the LLM Guardrails on the Market? A Comparative Analysis
• • LangChain — Building a harness with Jev (AutoModeMiddleware)
• • TypeSafe AI — Introducing System One models and Jev
• • VentureBeat — Companies are putting Jev in charge of AI agent decisions, and prompt injection can influence the verdict