Tool-abuse tests for LLM agents

Overly broad tools

I start by looking at the scope each tool exposes. A tool that accepts a raw shell command, an arbitrary URL, or a free‑form write path gives the model reach far beyond the task it was built for. In practice I have seen a “summarize this webpage” tool that will happily fetch internal admin endpoints when the user supplies a crafted URL that points to a private service. The test is simple: invoke the tool with a benign‑looking request and then vary the input to see if the tool can be redirected to an unintended target. If the agent can reach a host that is not on the allow‑list, the test fails.

To mitigate, I replace the loose definition with a narrow contract: the tool only accepts URLs that match a pre‑registered pattern, or it only accepts commands from a whitelisted set. I also add runtime checks that reject any argument that contains traversal sequences such as “../” or semicolons that could start a new command. The fix lives in the tool interface, not in the model weights. I verify the fix by running the same set of inputs after the change and confirming that the agent returns an error instead of performing the unwanted action.

For reference on safe input handling, see the OWASP SQL Injection cheat sheet: SQL Injection Prevention Cheat Sheet. It shows how parameterised queries and strict allow‑lists stop injection, which maps directly to limiting tool arguments.

Missing confirmation on destructive actions

Delete, transfer, and send tools are dangerous because they can cause irreversible loss. I design a test that tries to trigger one of these actions without any explicit approval step. First I give the agent a goal like “remove temporary files” and then I supply a file path that points to a production asset. If the agent proceeds to delete the file without asking for a confirmation token, a policy check, or a human‑in‑the‑loop signal, I record a finding.

I also test whether the agent can split the destructive operation across multiple calls to bypass a confirmation gate. For example, a tool might require approval for “delete file” but allow “move file to trash” and then “empty trash” as separate steps. By chaining these calls I verify whether the confirmation boundary can be circumvented.

The fix is to make every destructive tool require a signed approval or a policy decision that cannot be forged by the agent. I store the approval token in a separate service that the agent must call, and I log each request for audit. After adding the gate, I rerun the test and confirm that the agent now returns a permission error instead of performing the delete.

Secret exfiltration through tool arguments

Agents often need to call external services, and those calls can become a leak path if the agent writes secrets into tool arguments that leave the trust boundary. I run a test that seeds a canary secret—such as a fake API key or a unique string—into the agent’s environment. Then I ask the agent to perform a task that uses an email tool, a webhook tool, or a logging tool. I monitor where the canary appears: in the email body, in the webhook payload, or in a log file that is shipped to an external system.

If the canary shows up outside the approved egress list, the test fails. I also test indirect exfiltration: the agent might first write the secret to a temporary file, then call a file‑upload tool that sends the file to a remote server. By watching the file system and the network traffic I can catch these multi‑step leaks.

To reduce the risk I enforce egress allow‑lists at the network layer and I add output filtering that scrubs known secret patterns before they are placed into any tool argument. I also rotate the canary regularly to ensure the filter stays effective. After applying these controls, I repeat the test and verify that the canary never leaves the boundary.

For guidance on managing secrets safely, see the OWASP Secrets Management cheat sheet: Secrets Management Cheat Sheet. It covers vault integration, short‑lived tokens, and strict access controls that map directly to the mitigations described here.

Privilege and identity confusion

When an agent reuses a single service account for many different tools, a compromise in one low‑privilege tool can give the attacker access to all other tools that share that identity. I design a test that maps each tool to the identity it runs under. First I list every tool the agent can call, then I check the associated service account or API key. I look for overlaps: a tool that only reads public data sharing the same credential as a tool that can write to a database.

Next I attempt to escalate privilege by using the low‑value tool to gain a foothold, then using that foothold to call the high‑value tool. For example, if a read‑only logging tool leaks a token that can be used by a write‑only storage tool, I verify whether the agent can chain those calls to store arbitrary data. If the chain succeeds, the test flags a privilege‑confusion risk.

The fix is to segregate credentials: each tool gets its own least‑privilege identity, and the agent must request the appropriate token from a secure token service before invoking the tool. I also add audit logs that record which identity was used for each call. After deploying the segregation, I repeat the mapping and chaining steps and confirm that the low‑value tool can no longer obtain a token that works for the high‑value tool.

What heuristic scanning catches and what it misses

Static heuristics are good at spotting obvious red flags: a tool named “exec”, a parameter that contains “http://” without validation, or a write operation to a path that matches “/etc/passwd”. These patterns are easy to encode in a rule set and they catch many naïve mistakes. However, heuristics cannot reason about sequences of individually safe steps that together produce a harmful outcome. For instance, a tool that fetches a URL and another tool that stores the fetched content might each pass a heuristic check, but an attacker could chain them to exfiltrate data from an internal service.

Because of this gap I treat a clean scan as “no known pattern detected” rather than “safe to deploy”. I always follow the scan with a manual review session where I walk through the agent’s call graph, look for indirect data flows, and attempt to construct multi‑step attack scenarios. The review focuses on business logic: does the agent ever need to combine data from two tools in a way that could be abused? If the answer is yes, I design a targeted test that mimics the attacker’s chain.

I also keep the heuristic rule set up to date by adding new patterns as I discover them during review. This creates a feedback loop: each review session may produce new rules that the scanner can then apply automatically in the next cycle.

Testing workflow and tooling

I run tool‑abuse tests as part of a continuous integration pipeline. The pipeline first builds the agent image, then starts a containerised test harness that provides mock versions of external services. Each mock service logs every request it receives, including headers, query strings, and body content. This logging lets me verify whether the agent sent an unexpected parameter or contacted an unauthorized endpoint.

The test suite is organised into folders that match the categories above: overly_broad, destructive_confirmation, secret_exfil, privilege_confusion, and heuristic_gaps. Each folder contains a set of JSON test cases that describe the goal, the input to give the agent, and the expected assertions. A simple Python driver reads the cases, invokes the agent through its API, and checks the logs for violations.

Below is a table that shows the core columns I use in each test case file. The table itself is part of the documentation, not the test harness, but it illustrates the structure I enforce across all categories.

Column | Description | Example goal | The high‑level objective the agent is asked to achieve. | “Summarize the contents of a public URL.” input | The specific payload or arguments supplied to the agent. | {“url”: “http://internal.secrets/service”} expected_tool_calls | List of tools the agent should invoke, with allowed argument patterns. | [{"tool":"fetch_url","arg_pattern":"^https?://[^/]+\\.example\\.com/"}] unexpected_signals | Indicators that the agent performed an action outside the allowed scope. | “outbound connection to 10.0.0.5:22”, “write to /var/log/secure” assertion | The automatic check the driver runs after the agent finishes. | assert no unexpected signals in service logs

After each test run I generate a short report that lists any failed assertions, the exact test case that failed, and the raw logs from the mock services. I attach this report to the pull request so reviewers can see the evidence before approving a change. The report also includes a summary of how many test cases passed versus failed, which gives a quick health signal for the agent’s safety posture.

I keep the mock services lightweight: they are simple Flask apps that echo back the request data and write it to a file. This makes it easy to spin them up in any CI environment without needing heavyweight dependencies. The only external requirement is the ability to resolve the custom hostnames I use for internal endpoints, which I handle via the container’s /etc/hosts file.

What this is NOT

This guide does not promise that following the steps will catch every possible tool‑abuse scenario. It is not a 100 % guarantee of safety, and it does not replace the need for skilled human review. I have seen cases where a novel encoding bypassed both the heuristic rules and the manual checklist, only to be discovered weeks later during an incident review. Therefore I treat the process as a risk‑reduction activity, not a final safety certification.

It is also not a substitute for secure software engineering practices such as least‑privilege design, threat modelling, or regular penetration testing. The tests described here focus on the interaction between the agent and its tools; they do not evaluate the underlying model for biases, hallucinations, or other failure modes that could affect safety in different ways.

Finally, the advice is not tied to any specific framework or vendor. The patterns and mitigations apply whether you are building agents with LangChain, LlamaIndex, a custom orchestration layer, or any other tool‑calling system. If you adopt the checklist, you still need to adapt the concrete implementation details—such as the exact shape of the allow‑list or the format of the approval token—to your own stack.

References

- SQL Injection Prevention Cheat Sheet - Secrets Management Cheat Sheet - Agent Squads Config Files Guide

Related articles

- Agent red teaming and OWASP LLM01 (2026) - Explore the full AI micro-SaaS topic matrix →