Skip to content

LLM agent guardrails

Checks and enforcement controls that separate valid LLM actions from permitted execution, with a research-only permission and two-call quota example.

LLM agent guardrails are checks and enforcement controls that constrain an LLM agent's inputs, outputs and actions. In quant research, their practical job is to keep a request such as “summarise this report” within its permitted data, tools and execution budget—even if the model proposes a trade.

The term covers several mechanisms. A content classifier, a JSON validator, a permission check and a human approval gate answer different questions. Calling all of them “guardrails” does not make them interchangeable.

Four questions that must stay separate

CheckWhat it establishesWhat it leaves open
SchemaThe proposed action has the required fields and typesWhether its claims are true or its action is allowed
FactualityA claim agrees with an identified source or independent calculationWhether the agent may access that source or act on the claim
PermissionThis authenticated session may perform this action on this resourceWhether capacity remains or additional approval is required
QuotaThe action fits the remaining enforced allowanceWhether it is accurate, useful or authorized
For example, structured LLM output can require an integer quantity and a valid action name. That does not grant trading authority. An impeccably formatted research summary can also contain a financial model hallucination. Format checking and evidence checking need separate failure responses.
Content checks can run before model input, around intermediate messages and tools, or before releasing an answer. LangChain's guardrails documentation illustrates both deterministic rules and model-based checks. Rules can miss contextual violations; model-based judgments can also be wrong. A final-answer filter cannot undo an external action that already happened.

A research-only session, walked through

Consider a synthetic architecture, with fictional tools and no live broker, API or product integration. Its toy action contract recognises read_report, with a text report identifier, and place_order, with a text symbol and positive integer quantity. Both action types can be schema-valid.
The session's independent policy allows reads of reports already authorized for this user. It denies orders. It has two read-call tickets for the whole session. A ticket is reserved before each permitted execution; once dispatched, that read uses a ticket even if it fails or produces an unhelpful answer. Denied proposals use none. These are illustrative policy choices, not universal defaults.
Proposal, in orderSchemaPermissionDecision before executionRead tickets left
Read report AValidAllowedExecute1
Place an orderValidDeniedBlock; no order tool call1
Read report BValidAllowedExecute0
Read report CValidAllowedBlock; quota exhausted0
The order is blocked while read capacity remains. The third read is blocked despite having permission. Only two proposals reach a tool. Neither denial can be replaced by the model saying it has made an exception.
Synthetic research-only execution trace: all four proposals are schema-valid; read A executes, place_order is denied, read B executes, and read C is denied by the two-call quota. Cumulative tool executions are 1, 1, 2 and 2.
Synthetic research-only execution trace: all four proposals are schema-valid; read A executes, place_order is denied, read B executes, and read C is denied by the two-call quota. Cumulative tool executions are 1, 1, 2 and 2.
Figure: A permission denial and a quota denial stop different requests at the same execution boundary. The counter records executions, not model intentions.

In this design, the model proposes an action; a trusted execution layer validates its arguments, checks the authenticated user's resource permissions, reserves any quota and dispatches only an allowed request. The model cannot edit the policy or replenish its tickets. With parallel requests, checking and reserving capacity must be atomic: two requests must not both consume the same last ticket. Retries count under the stated policy too.

This follows the separation in OWASP's Excessive Agency guidance: restrict available capabilities and privileges, and enforce authorization in downstream systems. Removing the order tool and withholding trading credentials further narrows what a research session can do. A prompt saying “do not trade” is not an equivalent access control.

Approval is another gate

Permission establishes eligibility; an approval policy can require a person to authorize a specific eligible action before execution. LangChain's human-in-the-loop documentation illustrates pausing proposed tool calls for review.

For an order-capable system, a meaningful review would expose the exact instrument, side, quantity and destination being authorized. Changing those arguments requires applying the policy again. In the research-only example, an order remains denied: adding an approval dialog does not itself grant trading permission. A person would need a separate, authorized change of session capability before the action could become eligible.

What guardrails cannot promise

An external report may contain a prompt injection that tries to redirect the agent. Input and output filters can reduce exposure, but a filter passing a document is not proof that its contents are trustworthy. OWASP's Prompt Injection guidance treats layered controls as mitigation, rather than a guarantee of prevention.

The research-only permission boundary limits one possible consequence—an order through the denied tool. It does not prove that the summary is accurate, prevent every information leak or make all report access harmless. Resource authorization and restrictions on where data may be sent still matter. Logging supports investigation and monitoring; it does not substitute for a check that runs before an action.

Sources

Last updated
  • llm agents
  • guardrails
  • permissions
  • quant research
  • ai security
Keep connecting the ideas
← Back to the glossaryBack to top ↑