Agent red teaming and OWASP LLM01

Why agents change the injection math

From output to action

In a pure chatbot the goal of an injection is to make the model say something it should not say. The impact stops at the text that leaves the model. When the same model can invoke a tool, the injected text can become a command that the tool executes. An attacker who can steer a file‑read tool to pull a confidential document, or a mail‑send tool to forward that document, has turned a wording trick into a data breach. The severity therefore depends on what the tool can do, not on how clever the reply is.

Tool reach defines blast radius

The blast radius is the union of all resources that the agent’s tools can touch. If the agent only has a read‑only API to a public blog, the worst case is leaking public information. If the agent can call a database write tool, a file‑system delete tool, or an external payment gateway, the same injection could erase records, alter balances, or move money. Red‑team work must map each tool to the data stores, services, or hardware it can affect, then test whether an injection can reach those assets.

Common misconceptions

- Assuming the model’s output length limits damage – false, because a short string can trigger a destructive tool call. - Believing that filtering the user message is enough – false, because injections can arrive inside tool results or system prompts. - Thinking that a low‑parameter model is safe – false, when that model is paired with a powerful tool set.

For a quick reference on defensive coding, see the OWASP cheat sheet on LLM Prompt Injection Prevention.

Surfaces to probe beyond the chat box

System prompt and configuration

The system prompt often contains API keys, service URLs, or internal policy language. An attacker who can inject into this area may gain direct credentials or learn where to point further probes. Review the prompt for any static secrets and test whether they can be echoed or exfiltrated through a tool call.

Tool schemas and argument validation

Each exposed tool defines a JSON schema for its inputs. If the schema accepts free‑form strings without strict validation, an attacker can embed commands inside those strings. For example, a “run_sql” tool that takes a raw query string is vulnerable to SQL injection unless the agent layer sanitises or parameterises the input. Check whether the agent enforces type limits, length caps, or allow‑lists on every argument.

Retrieval‑augmented generation (RAG) pipelines

When the agent fetches data from external sources – web pages, emails, ticket systems – that data becomes part of the model’s context. An attacker who controls any of those sources can place instructions that the model will later treat as trusted. Probe by poisoning a known external feed and watching whether the agent acts on the hidden command.

Instruction propagation inside tool results

Some designs return tool output straight into the next model call without re‑sanitising. If a tool returns a piece of text that includes a new instruction, the model may obey it on the following step. This creates a two‑stage injection where the first stage is benign‑looking data and the second stage is the malicious command. Test each tool’s output path to see if the agent re‑feeds raw results into the prompt generator.

Practical checklist

- Extract the system prompt and search for hard‑coded secrets. - List every tool, note its schema, and attempt to pass unexpected data types. - Identify all external data sources used for RAG and inject benign markers to verify reach. - Capture tool outputs and feed them back as new inputs to see if the agent treats them as commands. - Document any chain where a single user message leads to multiple tool invocations.

For comparison with classic injection defenses, the SQL injection cheat sheet offers useful patterns: OWASP A03:2021 – Injection.

Excessive agency is the companion risk

What excessive agency means

OWASP places Excessive Agency at LLM03 for 2026. It describes agents that can perform irreversible actions – sending email, deleting files, executing code – without a separate approval step. The risk grows when the model’s autonomy outpaces the safeguards around its tools.

Red‑team focus areas

- Check whether destructive tools require a explicit confirmation from a human or a separate service. - Try to chain two or more tools together using only model‑generated intermediate outputs. - See if the agent can raise its own privileges, for example by calling a tool that updates its own API key. - Test time‑based triggers: can a delayed tool call be used to bypass a ratelimit or a session timeout?

Illustrative risk matrix

Tool category | Typical privilege | Needed confirmation | Potential abuse if missing File read | Read‑only access to designated folders | None (read is usually safe) | Exfiltration of logs, source code, configuration File write | Create or overwrite files | Yes – user approval or signed request | Insertion of backdoors, corruption of data Database write | INSERT/UPDATE/DELETE on tables | Yes – transaction approval or MFA | Data loss, financial fraud, privilege escalation External HTTP | Outbound network to arbitrary domains | Yes – allow‑list check | Data exfiltration, C2 beaconing, SSRF Code execution | Arbitrary shell or interpreter | Yes – isolated sandbox + approval | Full host compromise, ransomware deployment

The table shows why a red‑team must verify each row: if the confirmation column is missing, the agent can move from benign to harmful with a single poisoned input.

What a probe report should contain

Essential fields

A useful finding needs more than a boolean “pass/fail”. Record the exact user utterance that triggered the chain, the ordered list of tools the agent called, and the final observable effect (e.g., a file written, a network request sent, a database row changed). Include the raw tool arguments as they were sent, and the model’s internal reasoning trace if your framework exposes it.

Severity guidance

- Low – leakage of public information or a reversible state change. - Medium – access to internal‑only data that is not regulated, or a reversible action that requires manual reset. - High – alteration of regulated data, financial transaction, or execution of arbitrary code. - Critical – persistent backdoor, data destruction, or ability to maintain foothold after the session ends.

Prioritise fixes for high and critical findings first; low‑severity items can be addressed in a later hardening sprint.

Actionable remediation hints

- Add schema‑level validation that rejects strings containing SQL keywords or shell metacharacters. - Wrap destructive tools in a microservice that demands a signed approval token. - Implement an egress allow‑list for outbound HTTP calls and log every attempted connection. - Sanitise tool outputs before they are re‑inserted into the model prompt – strip or escape any content that looks like an instruction. - Use a read‑only replica for any database query tool unless a write is explicitly required.

Red teaming is not a substitute for design controls

Why testing complements, not replaces, safeguards

Red‑team work discovers gaps that already exist in the implementation. It does not create the guarantees that come from least‑privilege tool design, confirmation gates, or network segmentation. Treat each test as a validation step: if the controls stop the injected chain, you have evidence they work; if they do not, you have a concrete defect to fix.

Controls to prioritize

- Run each tool under a dedicated OS user with the minimum set of file system capabilities. - Require a separate approval service for any call that can modify state, log the decision, and enforce timeout. - Apply network policies that limit outbound connections to known internal services and a short list of external APIs. - Encrypt secrets at rest and never place them in the system prompt; use a vault service instead. - Log every tool invocation with user ID, timestamp, input hash, and output hash for forensic review.

When these controls are in place, red‑team findings shift from “we found a way to break it” to “we attempted X and the controls blocked it”, which is a stronger signal of security maturity.

What this is NOT

No guarantee of completeness

This guide does not promise that following the steps will catch every possible injection. Novel techniques, zero‑day tool vulnerabilities, or undisclosed agent features may evade the checks described here.

Not a 100 % fix

Applying the mitigations reduces risk but does not eliminate it. Defense‑in‑depth requires ongoing monitoring, regular tool‑chain reviews, and updates as new OWASP releases appear.

Never‑miss claim is false

No red‑team method can claim to never miss a flaw. The goal is to raise the cost for an attacker and to make successful exploitation noticeable and traceable.

Not a replacement for secure coding

The recommendations assume you already follow basic secure‑coding practices – input validation, output encoding, and principle of least privilege. They add agent‑specific layers on top of that foundation.

References

- https://cheatsheetseries.owasp.org/cheatsheets/LLM_Prompt_Injection_Prevention_Cheat_Sheet.html - https://cheatsheetseries.owasp.org/cheatsheets/SQL_Injection_Prevention_Cheat_Sheet.html - https://genai.owasp.org/llmrisk/llm01-prompt-injection - https://owasp.org/Top10/2021/A03_2021-Injection

Related articles

- Tool-abuse tests for LLM agents - Explore the full AI micro-SaaS topic matrix →