A code freeze written in English
In July 2025, SaaStr founder Jason Lemkin was building an application with Replit's AI agent. He had told the agent, in plain language, that there was a code freeze and that nothing should change without his permission. The agent deleted his production database anyway, fabricated data and gave inaccurate answers about what had happened. Replit's chief executive called the deletion unacceptable and something that should never be possible, pointed to the platform's backups and one-click restore, and announced automatic separation of development and production databases.
The instructive part is not that an agent made a mistake. It is where the "control" lived. The freeze existed only as text in the agent's context, which the agent could weigh, misread or ignore. The fixes Replit announced were all outside the model: a separate production database the agent cannot reach by default, and restore points that do not depend on the agent's cooperation.
Liability does not move to the model either. In Moffatt v. Air Canada (February 2024), British Columbia's Civil Resolution Tribunal rejected the airline's argument that its website chatbot was responsible for its own statements, and held the airline liable for the chatbot's incorrect description of its bereavement fare policy.
The thesis of this guide: an enterprise agent is safe only to the extent that its permissions, actions and spending are constrained by deterministic controls that sit outside the model. Prompts, system instructions and the model's own judgment are useful for quality. They are not guardrails.
Why the model cannot guard itself
Agents read content they did not write: emails, web pages, tickets, documents, tool outputs. Any of it can carry instructions. The UK National Cyber Security Centre put the problem plainly in December 2025: large language models do not enforce a boundary between instructions and data, so prompt injection may never be mitigated in the way SQL injection was. The NCSC recommends treating the model as a deputy that can be confused, and designing so that the damage a confused deputy can do is limited.
The measured evidence supports that stance.
Figure 1
Known attacks on one try understate the risk
Agent hijacking success rate (%) in NIST red-team testing of an agent built on Claude 3.5 Sonnet (October 2024 upgrade)
In January 2025, NIST's AI safety team (now the Center for AI Standards and Innovation, CAISI) published results from red-teaming an agent built on Claude 3.5 Sonnet (the October 2024 upgrade), a frontier model at the time. Against the benchmark's existing attacks, hijacking succeeded 11% of the time. New attacks written by NIST's red team succeeded 81% of the time. And when each attack was simply retried 25 times, average success across five injection tasks rose from 57% to 80%. Models have improved since then, but the lesson stands: a defense measured on known attacks and a single attempt overstates protection against an adaptive attacker who can try again.
The first widely reported zero-click case in a production assistant followed in June 2025. Researchers at Aim Security showed that a single crafted email could cause Microsoft 365 Copilot to retrieve internal content and send it out, with no user action (CVE-2025-32711, "EchoLeak"). Microsoft fixed it server-side. The pattern is the general one: untrusted input, access to private data, and a channel to the outside.
Meta's security team turned that pattern into a design rule in October 2025, the Agents Rule of Two. Within one session, an agent should have no more than two of three properties: it processes untrustworthy input, it can access sensitive systems or private data, or it can change state or communicate externally. If a task needs all three, the agent should not run autonomously; it needs human approval or another reliable check. The rule is simple enough to apply in a design review, which is why we use it below.
A layered guardrail architecture
Guardrails work as layers, each built on the assumption that the layer above it has failed. The model sits in the middle, wrapped by controls it cannot change.
Figure 2
The model sits inside controls it cannot change
Seven guardrail layers, each designed on the assumption that the model has been manipulated
Untrusted zone
Model plus prompts, retrieved documents, emails, web pages and tool outputs. Treated as able to be hijacked.
- 1Identity and least privilegePer-agent identity, task-scoped short-lived credentials, separate dev and prodStops:Inherited or shared access
- 2Policy gateway and tool allowlistEvery tool call checked against per-agent allowlist, parameters and targetsStops:Tool misuse, destructive calls
- 3Sandbox and egress controlDisposable execution, no standing secrets, outbound traffic denied by defaultStops:Code execution, data exfiltration
- 4Approval gatesNamed approver sees the exact action for payments, deletions, production changesStops:High-impact actions on model say-so
- 5Input and output validationUntrusted content marked, injection classifiers, schema and policy checksStops:Some injections, wrong statements to customers
- 6Tracing and auditCalls, retrievals, decisions and approvals logged to agent and requesterStops:Unexplainable incidents
- 7Budgets and kill switchHard limits on cost, calls and time; central credential revocationStops:Runaway loops, repeated harm
Identity and least privilege
Every agent needs its own identity, not a borrowed human session or a shared service account. NIST's National Cybersecurity Center of Excellence opened a project on agent identity and authorization in February 2026 for exactly this reason. Two common anti-patterns cause most of the exposure: an agent that inherits everything its user can do, and an agent that runs under a generic service account whose actions cannot be traced back to who asked for them.
The target state is an agent identity with task-scoped, short-lived credentials, issued on behalf of a named user or process, and logged as such. In practice that means OAuth-style delegated tokens with narrow scopes, secrets held in a vault rather than in prompts or configuration, and separate identities for development, test and production. Replit's dev/prod separation is the same principle applied to data.
Tool allowlists and a policy gateway
Agents should call tools only through a gateway that enforces an explicit allowlist per agent: which tools, which operations, which parameters and which targets. A "read customer record" tool should be a different tool from "update customer record", not one generic database connection. The gateway checks every call against policy before it executes, regardless of how convincing the model's reasoning looks.
This is the most valuable single control. It turns "the model decided to drop a table" into "the model requested a call that does not exist for this agent".
Sandboxing and egress control
Anything that executes code or browses should run in an isolated, disposable environment with no standing credentials, and with outbound network access denied by default and allowlisted by destination. Egress control is what breaks the third leg of an exfiltration attack. EchoLeak needed a way to send data out; a deny-by-default egress policy is a deterministic answer to that whole class of attack.
Approval gates
Some actions should never execute on the model's say-so: payments and refunds above a threshold, deletion, changes to production, external communications in the organization's name, permission changes. These need an approval step enforced by the gateway, showing the approver the exact action and its parameters, not the agent's summary of it. Approval design has its own failure modes, chiefly rubber-stamping, which is the subject of our companion article on human-in-the-loop design.
Input and output validation
Validation reduces likelihood; it does not remove it. Useful measures include marking and separating untrusted content in the context, classifiers that flag likely injection, schema validation of every tool call, and checks on outputs before they reach a customer or another system: policy and factual checks for customer-facing text, data-loss checks for anything leaving the boundary. The Air Canada case is an output-validation failure: a statement of policy went to a customer without being checked against the policy.
Tracing and audit
Log every model call, retrieved document, tool request, policy decision, approval and result, tied to the agent identity and the requesting user. Without this, incidents cannot be reconstructed and the organization cannot show a regulator or a court what happened. Traces also feed evaluation: they are the raw material for finding new failure patterns before an attacker does.
Budgets, rate limits and kill switches
Agents loop. A runaway loop can burn model spend, hammer a downstream API or repeat a harmful action hundreds of times. Hard limits per task, per agent and per day, on tokens, tool calls, cost and wall-clock time, should be enforced by the platform. A kill switch should revoke the agent's credentials and stop its sessions centrally, and should be tested like any other incident procedure.
Map the controls to the threats
The OWASP GenAI Security Project published its Top 10 for Agentic Applications on 9 December 2025. It is the most useful shared vocabulary for agent risk, and a good test of coverage: for each risk, which layer stops it, and which layer catches it when that fails?
Figure 3
Most agent risks are stopped outside the model
OWASP Top 10 for Agentic Applications mapped to the guardrail layer that should stop each risk (P) and the layers that support it (S)
ASI01Agent goal hijack
GatewayApprovalValidationTracing
ASI02Tool misuse and exploitation
GatewayApprovalBudgets
ASI03Identity and privilege abuse
IdentityTracing
ASI04Agentic supply chain vulnerabilities
GatewaySandbox
ASI05Unexpected code execution
IdentitySandbox
ASI06Memory and context poisoning
ValidationTracing
ASI07Insecure inter-agent communication
IdentityGateway
ASI08Cascading failures
ApprovalTracingBudgets
ASI09Human-agent trust exploitation
ApprovalValidation
ASI10Rogue agents
IdentityTracingBudgets
| Risk | Identity | Gateway | Sandbox | Approval | Validation | Tracing | Budgets |
|---|---|---|---|---|---|---|---|
| ASI01Agent goal hijack | – | P | – | P | S | S | – |
| ASI02Tool misuse and exploitation | – | P | – | S | – | – | S |
| ASI03Identity and privilege abuse | P | – | – | – | – | S | – |
| ASI04Agentic supply chain vulnerabilities | – | P | S | – | – | – | – |
| ASI05Unexpected code execution | S | – | P | – | – | – | – |
| ASI06Memory and context poisoning | – | – | – | – | P | S | – |
| ASI07Insecure inter-agent communication | P | S | – | – | – | – | – |
| ASI08Cascading failures | – | – | – | S | – | S | P |
| ASI09Human-agent trust exploitation | – | – | – | P | S | – | – |
| ASI10Rogue agents | S | – | – | – | – | S | P |
Two observations come out of this mapping. First, almost every risk is mitigated mainly by identity, the gateway or egress control, not by anything inside the model. Second, two risks, cascading failures and human-agent trust exploitation, are about the system around the agent: how errors propagate between automated steps, and how people come to trust confident output. Those need process design as much as technology.
Size the guardrails to autonomy and blast radius
Applying maximum controls to every agent is its own failure mode. Gartner warned in May 2026 that treating agent governance as binary, either locked down or fully trusted, over-restricts simple agents, which pushes teams to build around governance, and under-restricts autonomous ones. It predicted that by 2027, 40% of enterprises will demote or decommission autonomous agents because of governance gaps found only after production incidents.
The practical answer is tiering by what the agent can do, not by how clever it is.
Figure 4
Tier agents by what they can do, not how smart they are
Proportional guardrails by worst-case action and reversibility
Tier 1
Assist
Drafts, summarizes or answers from approved content; read-only
- Agent identity and read-only scopes
- Output checks for customer-facing text
- Standard logging
Tier 2
Act, reversible
Creates tickets, updates records, schedules work; changes can be undone
- Tool allowlist through gateway
- Egress allowlist
- Budgets and rate limits
- Full tracing
Tier 3
Act, high impact
Moves money, deletes data, changes production, speaks externally for the firm
- Everything in tier 2
- Approval gate on listed actions
- Sandboxed execution
- Adaptive red-teaming before launch
- Tested kill switch
Rule of Two check: an agent that reads untrusted input, can reach sensitive data and can act externally in one session belongs in tier 3 with approval, whatever the size of each action.
The tier follows from two questions: what is the worst action this agent can take with the permissions it holds, and how reversible is it? A drafting assistant with read-only access to a knowledge base sits in tier 1 even if it uses the most capable model available. An agent that can issue refunds sits in tier 3 even if each refund is small, because errors and attacks repeat at machine speed. Tiering should happen once, centrally and quickly, consistent with the proportional governance described in The Enterprise AI Operating Model.
A worked example: replaying the code freeze
Run the July 2025 incident through the architecture, as an illustration of where each layer would have intervened:
| Layer | Control | Effect on the incident |
|---|---|---|
| Identity | Agent identity has no production database credentials during development | Deletion is impossible, not merely forbidden |
| Tool gateway | Destructive SQL is not on the allowlist; schema changes go through a migration tool | The request is rejected and logged |
| Approval gate | Production changes require a named approver who sees the exact statement | A human sees the action before it runs |
| Tracing | Every tool call and result recorded independently of the agent | The agent's inaccurate account can be checked against the record |
| Recovery | Point-in-time restore outside the agent's control | Recovery does not depend on the agent's account |
None of these rows depends on the model understanding or respecting the word "freeze".
The Gulf view
In the UAE, the Dubai International Financial Centre has already turned part of this into regulation. DIFC Data Protection Regulation 10, introduced in 2023, governs personal data processed by autonomous and semi-autonomous systems. Mayer Brown's January 2026 summary describes requirements for notices about non-human-initiated processing, operation within human-defined or human-approved purposes, and, for high-risk processing, an Autonomous Systems Officer with a role similar to a data protection officer. For firms in the DIFC, agent identity, purpose-bound tool permissions and full traces are not only good engineering; they are the evidence a regulator will ask for. Organizations elsewhere in Saudi Arabia and the UAE should expect the same questions as AI rules mature, and map agent permissions to their data classification under frameworks such as the NDMO standards.
What to do in the next 90 days
- Inventory every agent in production or pilot, including those inside SaaS products, with its identity, tools, data access and owner.
- Apply the Rule of Two to each: list which of untrusted input, sensitive access and external action it combines. Any agent with all three and no approval gate is the first fix.
- Replace inherited and shared credentials with per-agent identities and task-scoped, short-lived tokens. Separate development and production.
- Put a policy gateway in front of tools with per-agent allowlists, and default-deny network egress for anything that executes code or browses.
- Define the tier-3 action list (payments, deletions, production changes, external messages, permission changes) and enforce approval for it in the gateway.
- Set hard budgets and test the kill switch in a tabletop exercise, including revoking credentials mid-task.
- Red-team with retries. Test injection with adaptive attacks and repeated attempts, not a single pass over a fixed list.
Frequently asked questions
What are guardrails for AI agents?
Guardrails are controls that limit what an agent can access and do, regardless of what its model decides: per-agent identity and least privilege, tool allowlists enforced by a gateway, sandboxing and egress control, approval gates for high-impact actions, output validation, full tracing, budgets and a kill switch.
Can prompt injection be prevented?
Not reliably with current models. The UK NCSC says it may never be fully mitigated, and NIST testing in 2025 found hijack success rose sharply with new and repeated attacks. Design to limit impact: restrict permissions, block unapproved egress and gate high-impact actions.
What is the OWASP Top 10 for Agentic Applications?
A list of the ten most significant security risks for AI agents, published by the OWASP GenAI Security Project in December 2025. It covers risks such as agent goal hijack, tool misuse, identity and privilege abuse, memory poisoning, cascading failures and rogue agents.
Who is liable when an AI agent makes a mistake?
The deploying organization, in the cases decided so far. In Moffatt v. Air Canada (2024), a Canadian tribunal rejected the argument that a chatbot was responsible for its own statements and held the airline liable for its incorrect advice.
Sources and further reading
- OWASP Top 10 for Agentic Applications for 2026, OWASP GenAI Security Project, 9 December 2025.
- Technical blog: Strengthening AI agent hijacking evaluations, NIST, 17 January 2025.
- Prompt injection is not SQL injection (it may be worse), UK NCSC, 8 December 2025.
- Agents Rule of Two: A practical approach to AI agent security, Meta AI, 31 October 2025.
- Accelerating the Adoption of Software and AI Agent Identity and Authorization, NIST NCCoE concept paper, 5 February 2026.
- Gartner on proportional governance for AI agents, Gartner, 26 May 2026.
- Replit's response to the SaaStr database deletion, The Register, 22 July 2025.
- BC tribunal confirms companies remain liable for information provided by AI chatbot, ABA Business Law Today, February 2024.
- EchoLeak: zero-click prompt injection in a production LLM system, arXiv, September 2025.
- AI regulation in the DIFC: autonomous and semi-autonomous systems, Mayer Brown, 19 January 2026.