Large Language Models (LLMs) respond flexibly to natural language requests, but this very flexibility can become an operational risk. Users might ask about prohibited topics, prompt injections could be mixed into retrieved documents, and models might call unauthorized tools or include sensitive information in their responses.
NVIDIA NeMo Guardrails is an open-source toolkit that places a programmable control flow between the application and the LLM, rather than leaving these issues to the nature of a single model. It’s not just a simple profanity filter; it applies different policies to inputs, conversations, search results, tool executions, and final outputs, and orchestrates their sequence.
This article will delve into what NeMo Guardrails is, how its five types of Rails operate, why typical system prompts alone are insufficient, and what level of responsibility should be entrusted to it. The latest development branch of the official repository can change rapidly, so when actually implementing, it’s important to check the release version along with the documentation.
NeMo Guardrails in One Sentence
NeMo Guardrails is a library that adds multiple security checks before and after an LLM, and it is an LLM processing pipeline runtime that determines which checks to execute and when.
Instead of the application directly calling the model, it calls LLMRails.generate() or the Guardrails Server. The NeMo Runtime inspects the input, performs necessary retrievals and actions, calls the LLM, and then re-inspects the output, all according to its configuration and Colang flows.
Application
→ NeMo Guardrails Runtime
→ Input Rail
→ Dialog Rail
→ Retrieval Rail
→ Execution Rail
→ LLM
→ Output Rail
→ ApplicationThe important point here is that a Rail does not refer to a specific model. A Rail is “the policy and flow to be executed at this stage,” and internally, it can combine rules, separate safety models, external APIs, Python Custom Actions, or other LLMs.
Why System Prompts Alone Are Insufficient
While it’s necessary to write “Do not output personal information” or “Do not execute administrator functions” in system prompts, this does not provide enforceable access control. Models generate sentences probabilistically and can be influenced by long conversations, conflicting instructions, or malicious phrases within retrieved documents.
The reason for using NeMo Guardrails is to separate policies into executable stages rather than hiding them in a single prompt sentence.
- Block explicit attacks and prohibited topics before sending requests to the LLM.
- Inspect raw chunks returned by RAG before inserting them into the prompt.
- Select only allowed next actions based on the conversation state.
- Validate arguments and results before and after tool calls.
- Inspect model responses for sensitive information and policy violations before they are delivered to the user.
This structure does not guarantee perfect defense. Instead, it makes it easier to isolate failure points and record policy execution history.
Five Types of Rails
The NVIDIA official repository describes five Rails: Input, Dialog, Retrieval, Execution, and Output.
| Rail | Application Timing | Primary Role | Result on Blockage |
| — | — | — | — |
| Input Rail | Immediately after user input | Jailbreak, prompt injection, sensitive info check | Terminate before LLM call or transform input |
| Dialog Rail | When determining conversation flow | Determine next action based on intent, state, standard procedures | Pre-defined response or allowed flow selection |
| Retrieval Rail | After RAG retrieval | Check retrieved chunks for contamination, sensitive info, relevance | Remove or mask risky chunks |
| Execution Rail | Before/after Tool/Custom Action | Validate arguments and execution results | Deny tool call or refine results |
| Output Rail | After LLM generation | Check for harmful content, info exposure, factual accuracy | Block, modify, or regenerate response |
Input Rail
The Input Rail is the outermost gate. It executes before the user’s raw text enters the model context, allowing it to block or mask explicit attack strings, prohibited topics, and personal information. However, one should not assume that a single input filter can catch all bypass expressions.
Dialog Rail
The Dialog Rail is key to distinguishing NeMo from a simple filter. It interprets user utterances into normalized intents and expresses what responses or actions are allowed in the current conversation state as a flow. For a customer support chatbot, this might mean restricting the conversation sequence to prevent moving to an account change step before identity verification.
However, the actual success of authentication must be determined by the application or authentication service. The Dialog Rail can adjust the flow of verification procedures, but it is not a security boundary that grants final user permissions itself.
Retrieval Rail
The Retrieval Rail sits between the raw chunks retrieved from RAG and the LLM prompt. It can remove or transform hidden instructions, unnecessary personal information, or irrelevant chunks within external documents.
It generally inspects readable raw chunks returned after retrieval, rather than checking embeddings themselves before vector search. Tenant filters and document access permissions should be enforced first at the search API or data layer, and the Retrieval Rail’s role is to perform additional content safety checks on those results.
Execution Rail
The Execution Rail wraps the inputs and outputs of Custom Actions or external tools called by the LLM. For example, it can block negative amounts or values exceeding allowed ranges from entering a transfer tool, and remove secret values contained in internal API results before returning them to the model.
Whether a tool can actually be executed must be re-authenticated and authorized by the downstream API. The fact that “the model chose this tool” is not proof of authorization.
Output Rail
The Output Rail executes just before the model’s response is delivered to the user. It can check for harmful content, prohibited topics, personal information, unsubstantiated answers, etc., and choose to block, modify, or regenerate the response. For streaming responses, the timing of inspection, latency, and the irrevocability of already transmitted tokens must be considered separately.
What Do Colang and Configuration Files Do?
NeMo Guardrails configures policies using config.yml, Colang files, and Python Actions. config.yml declares the models to be used and active Rails, .co files express conversation and policy flows, and actions.py implements custom behaviors like external service calls.
models:
- type: main
engine: openai
model: your-model
rails:
input:
flows:
- check jailbreak
output:
flows:
- self check outputColang is a language for modeling conversation flows. Instead of scattering all policies across Python conditional statements, it allows managing structures like “if the user shows this intent, execute this flow” in separate files. However, as complex policies increase, managing the duplication of responsibilities between Colang and application code becomes necessary.
Python Library vs. Server Approach
The simplest usage is to call the Python library within the application process.
from nemoguardrails import LLMRails, RailsConfig
config = RailsConfig.from_path("./config")
rails = LLMRails(config)
response = rails.generate(
messages=[{"role": "user", "content": "질문 내용"}]
)
print(response)If multiple applications use common policies, a Guardrails Server can be deployed as a separate service. The official repository provides HTTP API in the form of /v1/chat/completions and container execution methods. While the library approach is simpler to deploy and has less call latency, the server approach offers better policy centralization and language independence but requires consideration of network failures and bottlenecks.
What NeMo Does Well and What Not to Entrust It With
| Responsibilities Suitable for NeMo | Responsibilities of Application/Trust Layer |
| — | — |
| Input, retrieval, output content inspection | User authentication and session validation |
| Orchestrating conversation and LLM processing flow | Tenant determination and data access permissions |
| Pre/post Custom Action call inspection | Final authorization of tools/APIs |
| Integrating safety models, rules, external checks | Final approval for financial, deletion, permission changes |
| Tracking Rail execution | Legal audit trails and business ledgers |
NeMo can be at the center of the LLM pipeline, but it should not be the ultimate arbiter of business authority. The most practical division of responsibility is: “Application is the center of security/business control, NeMo is the center of the LLM processing pipeline.”
Limitations to Know Before Adoption
Guardrails have false positives and false negatives. Repeated LLM-based checks across multiple Rails increase token costs and latency. If the same policy is applied redundantly to input and output, or if application policies and Colang flows make conflicting decisions, it becomes difficult for operators to trace the cause.
Furthermore, the official repository does not guarantee that built-in guardrails are suitable for all operating environments as-is, and advises organizations to independently review their requirements and potential unforeseen misuses. Test sets should include normal requests, bypass expressions, multilingual content, long contexts, contaminated search documents, tool errors, and streaming.
The latest repository documentation also describes anonymous usage statistics telemetry and how to disable it. In closed networks or strict personal information environments, the telemetry items of the version in use and the terms of external inference endpoints should be checked before deployment.
When to Use NeMo Guardrails
Its adoption is highly valuable under the following conditions:
- A single service needs to consistently execute input, RAG, tool, and output policies.
- There’s a need to model conversation states and standard procedures beyond simple filters.
- Multiple safety models and external checkers need to be connected into a single Rail flow.
- Policy execution locations and results need to be observed and evaluated.
Conversely, for a small internal tool that only checks a few forbidden words in a single model, a custom middleware might be simpler. NeMo’s value is determined not by the number of filters, but by whether there is a need to orchestrate the entire LLM processing flow with policies.
Conclusion
NeMo Guardrails is not a magic box that creates safe models. It is a runtime that structures scattered checks before and after model calls into Input, Dialog, Retrieval, Execution, and Output Rails, and allows programming the flow of conversations and tool executions.
The biggest reason to use it is to transform “policies expected to be followed by the model itself” into “a pipeline that is actually executed and observed.” However, authentication/authorization, tenant boundaries, and final business approvals should not be handed over to NeMo. When these boundaries are maintained, NeMo becomes a useful layer that enhances both the safety and operational viability of LLM applications.
Leave a Reply