Hands-On with Guardrails AI: Validating LLM Inputs and Outputs

Why Guardrails AI?

Most LLM security discussions focus on model-level mitigations — system prompts, fine-tuning, RLHF. But those controls are inside the black box. What if you want a deterministic, auditable layer that sits between your application and the model?

That's the problem Guardrails AI solves. It is an open-source Python framework that wraps your LLM calls with a validation pipeline. You define a Guard object, attach validators from the Guardrails Hub, and every request and response runs through those validators before it reaches — or leaves — the model.

Validators can be applied to messages (input guardrails, checked before the LLM call) or to output (output guardrails, checked after the response arrives). The on_fail parameter controls behavior: raise an exception, return a fallback value, or log and continue. I used on_fail="exception" throughout to make failures loud and visible.

The Setup

Install once, then install each validator from the hub before first use:

What it guards against: Nonsensical or randomly generated text in either the prompt or the response. This is more useful than it first appears — attackers sometimes send garbled payloads to probe model behavior, and some adversarial prompt injection techniques rely on encoded or obfuscated text that looks meaningless to humans but carries semantic weight to models.

I tested both directions. For the input guardrail, I sent a prompt filled with random keystrokes (dhfsjdhfasdhfkjsdhf***kjsdf) with on="messages". The guard intercepted it before the API call was made — the LLM was never charged a single token.

For the output guardrail, I asked the model to generate gibberish deliberately. With on="output", the guard validated the response after it arrived from the LLM and raised an exception because the output failed the coherence check.

What it guards against: Personally Identifiable Information leaking into LLM prompts or out of LLM responses. This is a compliance-critical control for any application handling customer data — GDPR, CCPA, HIPAA all have opinions about sending PII to a third-party AI service.

The validator accepts a list of entity types to detect. I configured it for EMAIL_ADDRESS and PHONE_NUMBER and tested it against a prompt that explicitly asked the model to include an email address in its response.

Both directions worked correctly. On the input side (on="messages"), the guard blocked the request because the prompt itself contained a PII entity. On the output side (default — applied to the response), the guard caught the generated email in the model's reply and raised an exception.

Under the hood, DetectPII uses the Microsoft Presidio library — the same engine powering PII detection in Azure AI services. This means the entity recognition is NLP-based, not just regex, which gives it robustness against format variations (e.g., 555.123.4567 vs. (555) 123-4567).

What it guards against: Prompt injection and jailbreak attempts — crafted inputs designed to override the model's safety instructions, cause it to adopt an unintended persona, or extract harmful information it would otherwise refuse to provide.

I tested it with a classic DAN ("Do Anything Now") prompt: a multi-sentence attack that tells the model to abandon its identity, adopt a "no restrictions" persona, and then asks for bomb-making instructions. This is one of the most widely documented jailbreak patterns and a standard benchmark for any jailbreak detection system.

The guard blocked the request before the LLM call. The validator uses a fine-tuned classifier trained specifically on jailbreak patterns — it does not rely on keyword matching, which means it has better coverage against paraphrased or obfuscated variants of known attacks.

What it guards against: Hate speech, harassment, and abusive language in either user input or model output. This is a customer safety and brand protection control — critical for any consumer-facing AI product.

The validator exposes two key parameters. The threshold (0.0–1.0) controls sensitivity — lower values are stricter. The validation_method can be set to "sentence" (evaluate each sentence independently) or "full" (evaluate the entire block). I used threshold 0.5 with sentence-level validation.

I ran two test cases in the same guard instance:

The underlying model is a HuggingFace toxicity classifier. Because it operates at the sentence level, it handles mixed content well — a message that is mostly fine but contains one toxic sentence will still be caught.

What it guards against: Hallucinations — model outputs that are factually unsupported by the sources you provided. This is the most architecturally interesting validator in the set, and the most relevant to RAG-based applications.

ProvenanceLLM works by comparing the model's response against a set of sources you pass in via the metadata parameter. It uses an embedding model to compute semantic similarity between each sentence in the response and the source documents. If a claim in the output cannot be grounded in the sources, the validator flags it.

An important constraint: ProvenanceLLM does not fetch from the internet. It only checks against the sources you explicitly pass. This is a feature, not a limitation — it gives you deterministic, auditable grounding rather than relying on dynamic web retrieval.

For the embedding function, I used paraphrase-MiniLM-L6-v2 from the sentence-transformers library — a fast, local model that runs on CPU with no API cost. The custom embed function is passed via metadata["embed_function"].

This pattern directly addresses one of the most persistent failure modes in production RAG systems: models confidently answering questions that their retrieved context does not actually support.

Mapping Validators to the LLM Threat Model

These five validators do not overlap — each one addresses a distinct threat category. Here's how they map to the OWASP Top 10 for LLMs:

How the Guard Pipeline Works

The Guardrails framework wraps the LLM call in a validation loop. Here's the request lifecycle when input and output guards are both active:

Key Takeaways

What's Next

The Guardrails Hub has dozens of additional validators — regex matching, JSON schema enforcement, competitor mention detection, secrets detection, and more. My next area of exploration is composing a full production-grade guard stack and testing it against a realistic agentic pipeline where indirect prompt injection is a real concern.

If you want to run these experiments yourself, the full notebook is on GitHub. Drop in your API keys and run each cell — the output is intentionally verbose so you can see exactly where each validator fires.