Build Your AI Defense #1: Prompt Scanning Is Not Enough

Intro

If you take one thing from this post: an AI agent is an autonomous hire that can be tricked. So invest in AI security that checks what the agent does, not only what it says.

I learned this working on AI security with several vendors. Nearly every AI runtime security product checks the words going in and the words coming out. But recent AI systems do more than talk. Agents act. An assistant connected to your mailbox sends email and books meetings. Claude Code or Cowork on your laptop reads files, runs commands and calls tools. What matters is whether each action matches what you asked for. If you asked for a summary of your inbox, no email should leave. Your AI defense needs to protect you from that.

If you are an AI engineer building protection for an AI system, this post is for you. You will see where to set up defenses to protect your AI traffic and systems. You will learn what they have to check and what to build first. I highly encourage you to build your own lab, so you understand it first hand, and the next post gives you one. If you are already working with vendors on your AI security stack, you get a list of questions to ask them, and a secure AI implementation checklist.

If you are a CISO, this post is for you too. It outlines the six most important areas of AI security and shows where your controls and your teams fit. Then it goes deeper into AI runtime security, because most AI security controls sit there today.

First, the big picture

When someone says they do AI security, they could mean any of six quite different things. They happen at different stages, they catch different problems, and none of them replaces another.

Most companies do not build their own AI models. They take a model someone else built, deploy it and run it. So for most security teams, the work is mainly in the deploy and run columns.

Model and supply chain security. Is the thing you deployed the thing you think you deployed? Model files can carry code, and training data can be poisoned before you ever download it. Where a model comes from matters too. An open model uploaded to Hugging Face by someone you have never heard of is not the same as one from Google. You do not know who trained it, on what, or what was hidden inside. This layer is about checking and signing models before anything runs.

AI red teaming. Attacking your own system on purpose to find what breaks. Done regularly it is really useful. Done once before launch, the result is out of date the day you change your prompt. But let us be realistic. With the number of AI systems developers build today, you will not red team all of them. Focus on the systems that can reach sensitive data, and test the ones exposed to the internet often.

AI posture. Which models are running, in which accounts, reachable by whom, holding which keys. Boring work, and usually where the first real finding comes from, because most companies have more AI running than they knew about.

Access and identity. An agent acts on someone’s behalf, so ask whose. Does the agent have its own identity, or does it borrow the identity of the person who built it, or of the user who started it? An agent that inherits a user’s full access can do everything that user can, and the logs will say the user did it. Give each agent its own identity, only the permissions its task needs, and access that expires. This layer decides how bad an attack gets, not whether it happens.

AI endpoint security. The newest of the six, and the name is confusing. It does not mean AI that guards your laptop. It means guarding the AI that runs on it: desktop apps like ChatGPT and Claude, AI in the browser, coding agents like Claude Code and Cursor, and everything they install. It keeps a list of what is there, checks what gets installed, and watches what goes out and what the agents do.

AI runtime security. Watching the system while it runs, and blocking what should not happen. That is most of the rest of this post.

Why runtime

Take one attack. You ask your AI agent to summarize your inbox. One email was written by an attacker and carries a hidden instruction. The agent reads it, finds a password in another email, and tries to send it to an outside address. Then it gives you a perfectly normal summary.

I ran this attack in an earlier lab. It works because the model reads instructions and data as the same kind of text, and cannot tell them apart.

Which of those six layers could have stopped it?

Not model and supply chain security, because the model was clean. Not posture, because nothing was set up wrong. Not access and identity, because the agent used the permission we gave it on purpose. Not endpoint security, because this agent was an application on a server, not a tool on a laptop. Red teaming can find this attack in advance, but it is not running when the attack happens. The whole thing happened in the seconds between a request arriving and a response leaving.

That is the case for runtime security. It is the only layer that is there when the attack happens.

The gateway pattern

Runtime security means inspecting AI traffic while it flows. The first problem is that this traffic is not in one place. You have ten applications calling model providers directly, each with its own keys, and no shared view of anything.

The usual way to fix this is an AI gateway: one point that every model call passes through. Once everything goes through one place you can enforce policy, hide sensitive data, rate limit, log, switch between providers, and inspect.

The usual design has two checkpoints.

Before the prompt reaches the model, you check it. Typical detectors look for:

Prompt injection and jailbreak attempts
Personal data such as names, card numbers or ID numbers
Secrets and API keys that should never be sent to a provider
Malicious or suspicious URLs
Toxic or abusive language
Topics your company has ruled off limits

Before the answer reaches the user, you check that too. Does it contain a secret, a key or a customer record? A link to a malicious site? Toxic content, or an answer on a topic the app should never discuss?

This catches a lot, and every AI deployment should have it. But agents do more than talk.

The third checkpoint: what the agent does

The two checkpoints above look at text. But an agent does not only produce text. It acts. It sends email, changes records, runs commands and moves money. Each of those actions is a tool call, and a tool call is not part of the answer the user sees.

That is the gap. An attacker who gets a hidden instruction in front of the agent, in an email, a web page or a document, does not need the agent to say anything wrong. They need it to do something wrong.

Take the inbox attack from earlier in this post. The input check sees “summarize my inbox”. Nothing to flag. The output check sees a normal summary of your own inbox, and it only runs once the agent has finished. By then the email had already gone out. The theft never touched the model’s answer. It left through a tool call.

So a gateway for agents needs three checkpoints: what goes in, what comes out, and what the agent does.

You do not need to change how your tools run to do this. The model’s tool call comes back inside its answer, so it passes through the gateway before your application runs it. The tool’s result goes back to the model with the next request, so that passes through too. The gateway already sees both.

The third checkpoint asks one thing: does the action match the request? A request for a summary should not end with an email being sent. A question about an invoice should not end with a payment. A request to read a file should not end with a file being deleted. When the action does not follow from the request, the call should be blocked, or the user should be asked first.

Some gateways can already block a tool by name, or match its arguments against a pattern you write in advance. That is not the same thing. An email assistant is supposed to send email, so blocking the tool is not an option. What matters is whether this email, to this address, follows from what the user asked.

Few products make that comparison yet, but it has started. Microsoft’s Task Adherence (still in preview), OpenAI’s open source Guardrails and Meta’s LlamaFirewall each ask a second model whether a tool call fits what the user asked for. Claude Code does something similar in auto mode. Before a risky action runs, a second model checks whether the user authorized it. That checker sees only the user’s messages and the action, never the tool results, so text planted in a file or a web page cannot reach it.

One thing none of them does yet is track where each value came from: the user, or content the agent read.

Checking what the model says catches a chatbot leaking. Checking what the agent does catches an agent acting. You need both.

Why you need layers

Runtime security is the core, because it is there when the attack happens. But no single layer sees everything, and no check catches everything. Here are five reasons to put more layers around it.

1. A gateway does not see the laptop. Claude Code, Cowork and Cursor run on a laptop, and so do their tools: shell commands, file edits, local MCP servers. A gateway sees the conversation with the model, not what happens on the machine. There the action check moves to the endpoint: Claude Code’s PreToolUse hook sees each tool call before it runs, and can block it.

2. Agents trust what they load. Skills, CLAUDE.md, plugins and MCP tool descriptions are all read as instructions. The first malicious MCP server found in real use was a copy of a real email tool that worked correctly for fifteen versions, then sent a hidden copy of every email to the attacker. This is the job of AI endpoint security: scan what the agent loads before the session, and again on every update. Free scanners such as Snyk Agent Scan and Cisco MCP Scanner make a start.

3. A detector can be beaten by someone who keeps trying. Researchers ran more than 20,000 attacks against nine defense setups, and every defense that relied on the model to protect itself eventually broke. So use a detector to stop the common attacks, which are most of the traffic, and never as the only thing between an attacker and your data.

4. The judge is also an AI. Many detectors are language models, so they can be tricked too. Anthropic reports that the check in Claude Code still misses 17 percent of actions that go beyond what the user asked. So let the judge flag the action, and let fixed rules decide: in the study above, the one defense that never leaked was a plain rule in separate code.

5. The good news: attacks are noisy. The study above needed thousands of tries, and real attackers work the same way. A detector that blocks ninety nine attempts and misses one has still warned you ninety nine times. A user or an agent blocked a hundred times in a day is an incident, and your SOC should step in.

Put together

AI security today is Swiss cheese. Every layer has holes. Stack enough layers and the holes stop lining up.

Runtime security stops the common attacks and records the rest. Endpoint security covers what the agent loads. Identity limits what a hijacked agent can reach. Red teaming tests whether your checks hold. Your EDR and your SOC turn blocked attempts into a response. None of them is enough alone. Together they let you run AI in production and know what it is doing.

The secure AI implementation checklist

Seven things, roughly in the order I would do them.

Centralize before you inspect. You cannot inspect traffic that does not pass through you. A gateway is worth having even with every check switched off.
Give each agent its own identity, and the least it needs. Ask what the agent really needs, then make everything else impossible. Narrow permissions for every tool, access that expires, and a confirmation before anything that cannot be undone.
Check actions as well as text. Inspect the tool calls, and compare each one with the request that started it. Where you can, note which values came from the user and which came from content the agent read.
Tune the checks on purpose. Run fast checks on everything and slow checks on the actions that matter. When a check fires, ask the user before you block, because people who get blocked a few times find a way around you. When a check itself fails or times out, most traffic has to go through, or one outage stops everyone’s work. So alert on every request that passed unchecked, and keep a hard stop only for the few actions that cannot be undone.
Know what your agents load. Keep a list of the skills, instruction files, plugins and MCP servers on each laptop. Check them before first use and again on every update.
Log everything, and watch the blocks. Record every prompt, answer and tool call, so you can answer “what did this agent do last Tuesday, and why”. Send blocked attempts to your SOC and alert on repeats. Protect that log, because it holds every secret the agent touched.
Attack your own defense. Red team the systems that can reach sensitive data and the ones exposed to the internet, and do it again when something changes.

Next: the lab

None of this means much until you run it yourself and see where it holds and where it breaks. So the next post is a runtime security lab that you run on your own laptop. Local, no API key, no cloud cost.

We take the inbox agent from this post and put a gateway in front of it. LiteLLM, self hosted, free, no account and no API key. Then we run the attack again and switch the detectors on one at a time, first on the input, then on the output, then on the actions, and see which one catches it. We start with the checks LiteLLM ships, and add our own where they stop. For the actions we try the open source checks named above.

Correction, 1 October 2026: an earlier version of this paragraph said LiteLLM ships with no detectors of its own. That was wrong. It has a built-in filter that runs on your machine and finds secrets and personal data by pattern, and it can block a tool by name. The lab uses both.

After that we try to get past them.

A defense you have not attacked is a defense you do not understand.

Want to see the attacks first? They are in three short labs you can run yourself: a chatbot that leaks a secret, a tool with a poisoned description and the inbox agent.

Bonus: the vendor checklist

Ten questions for anyone choosing an AI gateway or an AI security product. Take them to the vendor call. Several vendors now advertise intent aware detection for agents. Ask how it works, and ask to test it with your own cases.

What do you inspect? Only prompts and answers, or also tool calls, tool results and the documents the agent reads?
Can you compare what the agent does with what the user asked for? How does that check work: fixed rules, or another model?
Answers are usually streamed. How do you inspect a streamed answer? Do you hold it until the check is done, check it piece by piece, or check it after the user has seen it?
Can I try it on my own traffic and with my own attack cases, in watch only mode, before it blocks anything?
When a check fires, do you block the user, or can you ask them first?
When your check times out or your service is down, does the request go through? Can I set that per action, and do I get an alert?
How much delay does inspection add, and what does it cost once tool results are included?
Can I see every prompt, answer and tool call of one agent session? Can you send blocked attempts to my SIEM and alert on repeats?
Where do you store the logged prompts and tool results, for how long, who can read them, and do you use them for training?
Do you cover AI on laptops, such as Claude Code or Cursor? Do you check skills, instruction files and MCP tool descriptions, also when they change?

If a vendor cannot answer the first two, they are selling you a chatbot filter and calling it agent security.

Source	Why it matters here
How Claude Code auto mode works, March 2026	A shipping product that checks risky actions against what the user authorized, and publishes how often it misses. Few vendors do.
“Do Not Mention This to the User”, February 2026	98,380 public skills checked, 157 confirmed malicious. The case for knowing what your agents load.
Cloak and Detonate, July 2026	1,613 real malicious skills, disguised. Packing got past every scanner more than 90 percent of the time. Watching what the skill does in a sandbox still caught 87 percent. Why a scan alone is not enough.
Evaluation of Prompt Injection Defenses in Large Language Models, April 2026	Read this before you trust any block rate. Nine defense setups, more than 20,000 attacks. Every defense that relied on the model broke. A plain rule in separate code did not leak once.
CaMeLs Can Use Computers Too, January 2026	Where the research is heading: the plan is written before the agent reads anything it cannot trust. It names the attack that still works. This needs a rebuild of the application, so it gets its own series later.