by AgentRedTeam (LX AI)
A red-teaming checklist built for chatbots misses most of what makes an agent dangerous. The moment a model can call tools, read files, or send messages, prompt injection becomes a path to action, not just text. OWASP's GenAI LLM Top 10 for 2026 keeps Prompt Injection at LLM01, and it is the right place to start an agent assessment.
In a pure chatbot the goal of an injection is to make the model say something it should not say. The impact stops at the text that leaves the model. When the same model can invoke a tool, the injected text can become a command that the tool executes. An attacker who can steer a file‑read tool to pull a confidential document, or a mail‑send tool to forward that document, has turned a wording trick into a data breach. The severity therefore depends on what the tool can do, not on how clever the reply is.
The blast radius is the union of all resources that the agent’s tools can touch. If the agent only has a read‑only API to a public blog, the worst case is leaking public information. If the agent can call a database write tool, a file‑system delete tool, or an external payment gateway, the same injection could erase records, alter balances, or move money. Red‑team work must map each tool to the data stores, services, or hardware it can affect, then test whether an injection can reach those assets.
For a quick reference on defensive coding, see the OWASP cheat sheet on LLM Prompt Injection Prevention.
The system prompt often contains API keys, service URLs, or internal policy language. An attacker who can inject into this area may gain direct credentials or learn where to point further probes. Review the prompt for any static secrets and test whether they can be echoed or exfiltrated through a tool call.
Each exposed tool defines a JSON schema for its inputs. If the schema accepts free‑form strings without strict validation, an attacker can embed commands inside those strings. For example, a “run_sql” tool that takes a raw query string is vulnerable to SQL injection unless the agent layer sanitises or parameterises the input. Check whether the agent enforces type limits, length caps, or allow‑lists on every argument.
When the agent fetches data from external sources – web pages, emails, ticket systems – that data becomes part of the model’s context. An attacker who controls any of those sources can place instructions that the model will later treat as trusted. Probe by poisoning a known external feed and watching whether the agent acts on the hidden command.
Some designs return tool output straight into the next model call without re‑sanitising. If a tool returns a piece of text that includes a new instruction, the model may obey it on the following step. This creates a two‑stage injection where the first stage is benign‑looking data and the second stage is the malicious command. Test each tool’s output path to see if the agent re‑feeds raw results into the prompt generator.
For comparison with classic injection defenses, the SQL injection cheat sheet offers useful patterns: OWASP A03:2021 – Injection.
OWASP places Excessive Agency at LLM03 for 2026. It describes agents that can perform irreversible actions – sending email, deleting files, executing code – without a separate approval step. The risk grows when the model’s autonomy outpaces the safeguards around its tools.
| Tool category | Typical privilege | Needed confirmation | Potential abuse if missing |
|---|---|---|---|
| File read | Read‑only access to designated folders | None (read is usually safe) | Exfiltration of logs, source code, configuration |
| File write | Create or overwrite files | Yes – user approval or signed request | Insertion of backdoors, corruption of data |
| Database write | INSERT/UPDATE/DELETE on tables | Yes – transaction approval or MFA | Data loss, financial fraud, privilege escalation |
| External HTTP | Outbound network to arbitrary domains | Yes – allow‑list check | Data exfiltration, C2 beaconing, SSRF |
| Code execution | Arbitrary shell or interpreter | Yes – isolated sandbox + approval | Full host compromise, ransomware deployment |
The table shows why a red‑team must verify each row: if the confirmation column is missing, the agent can move from benign to harmful with a single poisoned input.
A useful finding needs more than a boolean “pass/fail”. Record the exact user utterance that triggered the chain, the ordered list of tools the agent called, and the final observable effect (e.g., a file written, a network request sent, a database row changed). Include the raw tool arguments as they were sent, and the model’s internal reasoning trace if your framework exposes it.
Prioritise fixes for high and critical findings first; low‑severity items can be addressed in a later hardening sprint.
Red‑team work discovers gaps that already exist in the implementation. It does not create the guarantees that come from least‑privilege tool design, confirmation gates, or network segmentation. Treat each test as a validation step: if the controls stop the injected chain, you have evidence they work; if they do not, you have a concrete defect to fix.
When these controls are in place, red‑team findings shift from “we found a way to break it” to “we attempted X and the controls blocked it”, which is a stronger signal of security maturity.
This guide does not promise that following the steps will catch every possible injection. Novel techniques, zero‑day tool vulnerabilities, or undisclosed agent features may evade the checks described here.
Applying the mitigations reduces risk but does not eliminate it. Defense‑in‑depth requires ongoing monitoring, regular tool‑chain reviews, and updates as new OWASP releases appear.
No red‑team method can claim to never miss a flaw. The goal is to raise the cost for an attacker and to make successful exploitation noticeable and traceable.
The recommendations assume you already follow basic secure‑coding practices – input validation, output encoding, and principle of least privilege. They add agent‑specific layers on top of that foundation.
Because the mechanism has not changed: untrusted content reaches the model and is treated as instruction. Agent capabilities make the consequence worse, which is why OWASP keeps it at the top of the list while adding Excessive Agency as a companion risk.
An agent can act through tools, not just answer. Injection that only produced wrong text in a chatbot can trigger a tool call, a file write, or an external message in an agent. The blast radius is the tool set, not the reply.
System prompts, tool schemas, retrieval content, and instructions embedded in tool results. Any place untrusted text enters the agent's context is a candidate injection point.
No. Testing reveals existing gaps. Least-privilege tool access, confirmation on destructive actions, and egress controls are what prevent escapes. Red-teaming validates those controls after they exist.
Try the product: AgentRedTeam
2026-09-16 · primary sources only · no fabricated traffic metrics