Build and Attack Your AI Defense #1: What Works and What Doesn’t

Intro

In Lab #1 a chatbot leaked a secret it was told to protect. In Lab #2 we gave it tools through an MCP server and poisoned one. In Lab #3 an agent read a poisoned email, found a VPN password, and tried to send it to an attacker. You had only asked it to summarise your inbox.

Same weakness every time. Instructions and data share one channel, and the model cannot reliably tell them apart. Words, then data, then actions.

I said the next series would show how to defend these systems. This is it.

But I want to be straight about that promise. Nobody can honestly teach “how to defend AI” in one post, and a lot of what gets published under that title is really a sales pitch.

So this post does two things. First it lays out the big picture, because AI security has become one label for at least five different jobs, and people mix them up constantly. Then it goes deep on the one part that was actually present at the moment our three attacks happened.

Along the way I found something I did not expect. The best research in this field has mostly stopped trying to spot bad text. It has moved to limiting what an agent is able to do in the first place. I will come back to that further down, because it changes what you should build.

You cannot defend what you cannot see. Every attack in the previous series succeeded in a place nobody was looking.

First, the big picture

When someone says they do AI security, they could mean any of five quite different things. They happen at different stages, they catch different problems, and none of them replaces another.

Model and supply chain security. Is the thing you deployed the thing you think you deployed? Model files can carry code, and training data can be poisoned before you ever download it. Where a model comes from matters too. An open model uploaded to Hugging Face by someone you have never heard of is not the same as one from Google. You do not know who trained it, on what, or what was hidden inside. This layer is about checking and signing models before anything runs. It does nothing once a clean model is live.

AI red teaming. Attacking your own system on purpose to find what breaks. That is what Labs 1 to 3 were. Done regularly it is really useful. Done once before launch it is a snapshot that goes stale the day you change your prompt. And it cannot catch the live attack, because you are not there when it happens.

Posture, sometimes called AI-SPM. Which models are running, in which accounts, reachable by whom, holding which keys. Boring work, and usually the first real finding, because most companies have more AI running than they knew about. It does not help when a correctly set up system is tricked through its normal interface.

Access and identity. An agent acts on someone’s behalf. Whose, with what permissions, for how long. Lab #3 came down to exactly this. The agent had send_email, so it could send email. Nothing was misconfigured. The permission was the problem, and the permission was intentional. This layer decides how bad an attack gets, not whether it happens.

AI runtime security. Watching the system while it runs, and blocking what should not happen. This is the only layer present at the moment of an attack, and it is the rest of this post.

Why runtime

Look back at the three labs and ask a simple question. At what moment could each attack have been stopped?

Not at build time, because the models were fine. Not at scan time, because nothing was backdoored. Not at red team time, though that is how we found them. Every one of those attacks happened in the seconds between a request arriving and a response leaving.

That is the case for runtime security. Not that it is the best layer, but that it is the only one there when it matters.

The gateway pattern

If you want to inspect AI traffic, the first problem is that you do not have AI traffic. You have twelve applications calling model providers directly, each with its own keys, and no shared view of anything.

The standard answer is an AI gateway. One point that every model call passes through. Same idea as an API gateway, applied to LLM traffic. Once everything goes through one place you can enforce policy, hide sensitive data, rate limit, log, switch between providers, and inspect.

The usual design has two inspection points.

Before the prompt reaches the model, you check it. Typical detectors look for:

Prompt injection and jailbreak attempts
Personal data such as names, card numbers or ID numbers
Secrets and API keys that should never be sent to a provider
Malicious or suspicious URLs
Toxic or abusive language
Topics your company has ruled off limits

Before the answer reaches the user, you check that too. Does it contain a secret, a key or a customer record? A link to a malicious site? Toxic content, or an answer on a topic the app should never discuss?

This catches a lot, and every AI deployment should have it. But agents do more than talk.

The third checkpoint: what the agent does

Recall what actually happened in Lab #3.

The agent read the inbox. One email carried a hidden instruction. The agent found a VPN password in a different email and called send_email to ship it to audit@external-vendor.com. Then it wrote you a perfectly normal summary.

The password never appeared in the model’s answer. The reply you saw was clean. The data left through a tool call.

Trace that through the diagram above. The input check sees “summarise my inbox”. Nothing to flag. The output check sees a friendly summary with no secrets in it. Nothing to flag. Both did their job. The data simply left a different way.

So a gateway for agents needs three checkpoints: what goes in, what comes out, and what the agent does.

Checking what the model says catches a chatbot leaking. Checking what the agent does catches an agent acting. You need both.

Why actions are easier to check than text

Text and actions catch different things, so you want both. But they are not equally hard.

Checking text means asking a fuzzy question. Is this language malicious? The attacker gets unlimited attempts to reword around whatever answer you give.

Checking an action is a much easier question.

The user asked to summarise an inbox, so why is there a send_email at all? The recipient is a domain this agent has never contacted, and it was not in the user’s request, so it must have come from something the agent read. The body looks like a password, and that password came from an email, not from the prompt.

None of that needs an understanding of language. You compare the action against the request that started it, and you track whether each value came from the user or from content the agent read along the way.

Here is what that check would have seen in Lab #3.

What the check looks at	Value	Where it came from	Verdict
The user’s request	“summarise my inbox”	the user	trusted
The tool being called	`send_email`	the model’s decision	not in the request
The recipient	`audit@external-vendor.com`	an email the agent read	untrusted
The body	a string that looks like a password	a different email the agent read	untrusted, and sensitive

Every column is a plain fact. A sending tool nobody asked for, aimed at an address that came from untrusted content, carrying data that came from untrusted content. Any one of those is a warning. All three together is a block, and you did not have to understand a single word of the attacker’s instruction to get there.

That second part is called provenance, which simply means knowing where each piece of data came from. It is exactly what the model cannot do for itself. To the model, everything in its context is just text. It has no way to know that one paragraph came from you and another came from an attacker.

Checking text is a bet against someone who can reword. Checking an action compares it against a request that already exists. You move from guessing to checking.

The research agrees, and goes further

This is the surprise I mentioned at the start. I expected the serious research to be about building better classifiers. It is not, and it has been moving away from that fast.

The clearest statement of the new position comes from a 2026 paper called Parallax. Its argument fits in one sentence:

The system that reasons about actions must be structurally unable to execute them.

In plain words: the part that thinks should not be the part that acts. A text filter and a text attack play the same game, so a filter can make attacks harder but never rule them out.

So instead of detecting, Parallax separates. The thinking part and the acting part run as separate processes that share no memory. Every action passes four levels of checks: fixed rules first, then classifiers, then a model with a limited budget, and finally a human. Data carries sensitivity labels that follow it through the system, which is how it spots a credential moving from one tool to another. That is provenance, built for real, and it is what would have caught Lab #3.

The idea started earlier. A 2025 paper from IBM, Invariant Labs, ETH Zurich, Google and Microsoft, Design Patterns for Securing LLM Agents, set out six ways to limit an agent by design. What makes it useful is that it states the cost of each one openly.

Pattern	What it does	What it costs
Action Selector	The agent picks from a fixed list and never sees the result.	No feedback loop, no multi step work.
Plan Then Execute	The plan is fixed from the user’s prompt alone, then executed. Content the agent reads cannot change it.	It cannot adapt to what it finds.
Map Reduce	Untrusted documents are split across isolated sub agents that cannot see each other.	Loses reasoning across documents.
Dual LLM	A privileged planner that never reads untrusted text, plus a quarantined worker that reads it and holds no tools.	More moving parts, awkward handoffs.
Code Then Execute	The plan becomes a program, so you can check the data flow before anything runs.	Complexity, and it needs a sandbox.
Context Minimisation	Remove untrusted content from the context once it has done its job.	Only works when you can tell what to drop.

Look at Lab #3 through the Dual LLM pattern and the attack has nowhere to land.

The part that read the poisoned email would have had no tools. The part holding send_email would never have seen the attacker’s instruction. No classifier needed.

CaMeL, the same idea built for real

Google DeepMind built this out in a system called CaMeL. A privileged model plans. A quarantined model handles untrusted data and holds no tools. A custom interpreter tracks where every value came from and what may be done with it. It wraps existing models, so there is no retraining. A 2026 follow up extends the same approach to agents that use a computer.

The numbers

On AgentDojo, a benchmark of 97 agent tasks and 629 security test cases, the undefended agent lost to 300 of the 949 attack attempts. With CaMeL, zero got through. The two that still worked were phishing style attacks that never touched the data flow, which the authors say up front are outside what CaMeL protects.

The price is usefulness. The agent completed 77 percent of tasks with CaMeL in place, against 84 percent without it. That is the deal with a design based defence: it does not guess, it does not get weaker when attacked, and it costs you some of what the agent could do.

Parallax reports 98.9 percent of 280 attacks blocked in its default setting with no false positives, and 100 percent at maximum security. But the 100 percent comes with a 36 percent false positive rate, so more than a third of legitimate actions get stopped. Nobody runs that in production for long, which puts you back on the weaker setting. And those 280 attacks were a fixed list, fed straight into the checker. Nobody was adapting to it. That matters, and it is the first item in the next section.

Where it all comes up short

I promised at the end of Lab #3 to be honest about where this field really is. So here is what I would want to know before trusting any of the above, roughly in order of how much it should worry you.

1. Attackers adapt, and the good numbers drop. Researchers took twelve published defences, nine of them against indirect prompt injection (an attack hidden in content the agent reads, like the poisoned email in Lab #3), studied each one and designed attacks aimed at it. All twelve fell, most with attack success above 90 percent. The list includes products you can buy today: Google’s Model Armor, Meta’s PromptGuard and Protect AI’s detector. One defence had reported 2 percent attack success on the standard benchmark. Against an attacker who had studied it, 96 percent. The Parallax numbers above were measured the same way, against a fixed list, so read them as a floor for the attacker, not a ceiling.

2. Text detectors can be fooled completely. Six well known injection detectors, including Microsoft’s Azure Prompt Shield and Meta’s Prompt Guard, were bypassed up to 100 percent of the time using hidden characters and similar tricks, while the attack still worked on the model behind them. Input checks still stop lazy and automated attacks, which is most traffic. They will not stop someone who is really trying.

3. The judge is also an AI. Many detectors are language models themselves. Even with temperature at zero, the same prompt can get a different verdict on two calls in a row. A control that only sometimes fires is not really a control.

4. Reading everything costs money. To inspect traffic, the gateway has to read all of it: every prompt, answer and tool result. Bedrock Guardrails charges 0.10 to 0.17 US dollars per thousand text units, where a text unit is up to 1,000 characters, and each type of check is billed separately. Google’s Model Armor gives you 2 million tokens a month free and then charges 0.10 dollars per million. Small, until you multiply by real volume and find a second AI bill on top of the first. The best checks also cost the most. A regex is free. A model acting as judge is 100 to 1000 times slower. So most traffic gets the cheap check.

5. False positives hurt more than attacks. A developer pastes an internal IP address into a chat to debug something. The PII detector blocks it. Nothing was ever at risk, and you just broke their work. Do that a few times and people find a way around you, which is worse than having no control at all. The better pattern is to ask instead of block: “This looks like personal data. Continue?” The user knows things the classifier does not. Parallax ends up in the same place, since its last level of checks is a human.

6. Many gateways fail open. Fail open means that when the check cannot run, the request goes through anyway. Every check has a time limit. In TrueFoundry’s gateway, for example, the default is 5 seconds, and whether a slow check blocks or allows the request is a setting. When it allows, a big enough input can push the check past its limit and through uninspected. Zuplo only switched its guardrails to fail closed by default in August 2026, which tells you what the default was before.

7. Less gets inspected than the diagrams suggest. User prompts are almost always checked, and model answers usually are. Web search results, retrieved documents and tool calls often are not, and those are exactly the paths agents use and where indirect injection lives. Ask your vendor which ones they actually inspect.

8. Streaming and speed. Users expect words to appear as they are written, but an output check wants the full answer first. Wait for it and you lose streaming. Check in chunks and you can miss something split across two. Check afterwards and the user has already read it. Every check also adds time. Parallax measured about 1.9 seconds for its classifier step and 2.1 seconds for its model step, on top of the model itself.

So, honestly

AI security today is Swiss cheese. Every layer has holes, and sometimes the holes line up.

That is not a reason to skip the layers. It is a reason to know where the holes are, to assume you will be bypassed, and to build so that being bypassed is survivable rather than a disaster.

So what do you actually build

Six principles, roughly in the order I would apply them.

1. Centralise before you inspect. You cannot inspect traffic that does not pass through you. A gateway is worth having even with every check switched off, because it turns twelve apps calling three providers into one place you can see. Visibility first, enforcement second.

2. Limit what the agent can do before you buy a filter. The design patterns paper is right. Limits by design hold up better than detection. Ask what the agent really needs to do, then make everything else impossible. This is free, and it does not weaken when someone attacks it.

3. Check actions as well as text. Most gateways stop at prompts and answers. For an agent, add the tool calls, and compare each one against the request that started it.

4. Track where data came from. Know which parts of the context came from the user and which came from content the agent read. Every attack in Labs 1 to 3 worked because that line was lost. The model cannot recover it. The gateway can keep it.

5. Limit the damage. An agent that cannot reach the internet cannot send data to it. Give tools the narrowest permissions possible, ask for confirmation before anything that cannot be undone, and treat a broadly permissioned agent like a new starter with production access.

6. Assume you will be bypassed, and log for it. With twelve out of twelve defences broken by adaptive attackers, plan to miss things. That makes the log the real product. Every prompt, every answer, every tool call, with enough context to rebuild what happened. If you cannot answer “what did this agent do last Tuesday, and why”, you have a filter, not runtime security.

Eight questions to ask before you trust a gateway

Whether you are buying one or building one, these are the questions I would want answered. Most of them come straight from the gaps above.

Which paths do you inspect? User prompts and model answers, or also tool calls, tool results, web search results and retrieved documents?
Can you compare a tool call against the request that started it, and do you know which values came from the user and which came from content the agent read?
When a check times out or your service is down, does the request go through or get blocked? What is the default, and can I change it?
What is your false positive rate, measured on traffic like mine, and what happens to the user when a check fires? Block, or ask?
Which of your checks are fixed rules and which are models? For the model based ones, how often does the same input get a different answer?
Has anyone attacked your detector adaptively, and can I see the result? A block rate against a fixed test list is not the same thing.
What does inspection add to latency at the 95th percentile, and what does it cost per million tokens once you include tool results?
If I ask what an agent did last Tuesday and why, can you show me every prompt, answer and tool call in that session?

If a vendor cannot answer the first two, they are selling you a chatbot filter and calling it agent security.

Next: the lab

None of this means much until you run it yourself and see where it holds and where it breaks. So the next post is a lab in the same style as the previous series. Local, no API key, no cloud cost.

We take the vulnerable agent from Lab #3, put a gateway in front of it, and run the attack again. Then we switch on the checks one at a time, input, output and tool calls, and see which one catches it.

After that we try to get past the tool call check too.

A defence you have not attacked is a defence you do not understand.

Source	Why it matters here
The Attacker Moves Second	Read this before you trust any block rate. Twelve defences, three of them commercial, all bypassed by adaptive attacks.
Simon Willison on CaMeL	The clearest plain English explanation of separating the part that thinks from the part that acts. Start here, then read the paper.
CaMeL: Defeating Prompt Injections by Design	Provenance built as a real system. Zero attacks through in scope, at a cost of 7 points of task completion.
Design Patterns for Securing LLM Agents	Six ways to limit an agent by design, each with its cost stated.
Parallax	2026. The strongest case for the separation, and one of the few papers to publish its false positive rate.

Build and Attack Your AI Defense #1: What Works and What Doesn't