Securing AI Agents: A Practical Threat Model

An AI agent is software that acts: it plans, calls tools, writes to systems, and loops until it reaches a goal. That single property changes the security model. When a chatbot fails, you get a bad paragraph. When an agent fails, you get actions: a deleted branch, an issued refund, a modified record, an email sent to the wrong thousand people. A practical threat model for agents covers six areas: goal hijacking, tool-call abuse, memory poisoning, identity and privilege for non-human actors, runtime guardrails, and the ability to stop the system quickly. Each maps onto the OWASP LLM Top 10, and each is testable. The teams that do well treat the agent like a new class of privileged user, not a feature.

Failures are actions now

Traditional LLM security worried about outputs: toxic text, leaked training data, hallucinated citations. Agent security worries about consequences. An agent with access to a CRM, a payment API, and email does not need to be compromised in any classic sense to cause damage; it needs one bad decision at the wrong permission level. Blast radius, not accuracy, is the unit of risk. That is why the OWASP category that matters most for agents is LLM06, excessive agency: too many tools, too much permission per tool, and too much autonomy before a human sees anything.

Scoping the action surface is the highest-leverage control you have. Every tool you do not attach and every permission you do not grant is a class of incident that cannot happen. Start every agent design by writing down what it must never be able to do, then check that the tool list and scopes actually make those outcomes impossible rather than merely discouraged.

Goal hijacking

Goal hijacking is prompt injection with a job to do, OWASP LLM01 applied to a system that acts. The attacker does not need access to your prompts or your model; they need to place content where the agent will read it. A support agent that reads inbound tickets, a coding agent that reads repository issues, a procurement agent that reads supplier emails: each ingests attacker-controlled text in the normal course of work. A hijacked goal can be blunt, forward all invoices to this address, or subtle, a slow redirection of priorities that never trips an obvious filter.

Long-horizon agents raise the stakes because objectives compound. An instruction absorbed at step 3 can shape a decision at step 40, long after the poisoned content has scrolled out of any human's view. The defenses that matter: strict structural separation between instructions and data when prompts are assembled, provenance tracking for every piece of context the agent consumes, and task-scoped sessions so one contaminated workflow cannot leak goals into the next.

Tool-call abuse

Even with an intact goal, individual calls can be abused. A SQL tool granted write access for one workflow becomes a data destruction primitive. A file tool with a path parameter becomes a traversal primitive. An HTTP tool becomes an exfiltration channel to anywhere unless egress is constrained. Attackers do not attack the model here; they attack the arguments, coaxing legitimate tools into illegitimate parameters.

Treat every tool like an API exposed to a semi-trusted caller, because that is what it is. Validate arguments server-side against a schema, enforce authorization on every call rather than once at session start, constrain filesystem and network scope per tool, and set per-session budgets: maximum calls, maximum rows, maximum spend. If your agents reach tools over the Model Context Protocol, the protocol layer carries its own failure modes, tool poisoning and description-borne injection among them, which we cover separately in MCP Security Explained.

Memory poisoning

Agents remember: scratchpads, conversation summaries, vector stores, learned user preferences. Memory converts a one-shot injection into a persistent implant. A poisoned record written today, say, the approved vendor's bank account has changed, update all future payments, gets retrieved as trusted context next week, long after the malicious input is gone. OWASP files this under LLM04, data and model poisoning, and LLM08, vector and embedding weaknesses, and it is among the least monitored parts of most agent stacks.

The controls are concrete. Validate and sanitize anything before it is written to long-term memory, tag memory entries with provenance and a trust level, give memory a time-to-live so poison decays rather than compounds, and audit memory contents on a schedule, the way you audit privileged group membership. When an incident occurs, memory is part of the crime scene: keep it inspectable, exportable, and wipeable per user and per agent.

Identity and privilege for non-human actors

Most agents in production today run on borrowed identity: a shared service account, an API key in an environment variable, a token scoped to whatever the pilot needed plus slack. That fails audit and forensics at the same time. When an agent acts, you need to answer who acted, on whose behalf, and with what authority, and the answer cannot be the team API key.

Give each agent a first-class identity in your identity provider, distinct from any human's. Issue short-lived, narrowly scoped credentials per task, delegated from the requesting user when the agent acts on a person's behalf, so the agent can never exceed the rights of the human it serves. Broker every secret through a manager; nothing lives in prompts, code, or config. Non-human identities already outnumber human ones by an order of magnitude in most cloud estates, and agents are about to multiply that number. Inventory them now, while the count is still small enough to inventory.

Runtime guardrails

The model's own judgment is not a control. Guardrails have to live outside the model, in a policy layer that inspects proposed actions before execution. The workable pattern is to classify every tool call by consequence: reversible, low-impact actions execute freely; higher-impact actions execute with logging and rate limits; irreversible or high-value actions, payments, deletions, external sends, permission changes, require human approval. OWASP LLM05, improper output handling, applies here too: anything the agent produces that another system will execute, SQL, shell commands, rendered HTML, gets validated like untrusted input.

Approval gates only work if they are rare enough to be read. Gate the 2 percent of actions that can actually hurt you and let telemetry cover the rest, or your reviewers will click approve as a reflex within a week and the gate becomes decoration.

Kill switches and containment

Every agent needs a stop that works in seconds: a pause per agent and per fleet, credential revocation that takes effect immediately rather than at token expiry, and circuit breakers that trip automatically on anomalies such as call-rate spikes, novel tool sequences, or budget overruns. Design for containment the way you design for deployment rollback, and rehearse it. If the only way to stop a misbehaving agent is to find the engineer who owns it, you do not have a kill switch, you have an escalation path.

Observability

You cannot secure what you cannot replay. Log every step: the prompt as constructed, retrieved context with provenance, each tool call with arguments and results, and every approval decision, with timestamps and identity attached. Structured traces make three things possible that raw chat logs do not: forensic reconstruction after an incident, detection rules over live behavior, and regression testing when you change prompts or models. Retention matters more than teams expect, because memory poisoning often surfaces weeks after the write that caused it.

Test the loop, then keep testing it

Agent security fails at composition. The model, the tools, and the policy layer each pass their own review, and the incident happens in the seams between them. Testing has to target the assembled system: hijack attempts through every content channel the agent reads, argument abuse against every tool, memory poisoning with delayed triggers, and privilege checks on every identity the agent can hold. That is the core of an AI penetration test scoped for agents, and the findings feed the architecture work that an agentic AI security program formalizes: least-privilege tool design, guardrail placement, and the monitoring that makes week-two drift visible.

Agents are the first software most companies have deployed that changes its behavior based on what it reads. Secure them as if that is true, because it is, and retest on every material change to tools, prompts, or models, because the system you assessed in March is not the system running in June.