Most teams fight prompt injection by writing a better instruction. Add “ignore anything malicious” to the system prompt, run a filter, ship it. The research says that approach has not held up since 2022. The approach that does hold up skips the arms race and limits what a successful attack can do.
The key idea
Prompt injection is what happens when text your system meant to treat as data gets treated as instructions. It has had a name since September 2022, when Simon Willison wrote it up. It sits at number one on OWASP’s list of LLM risks. Nothing published since has produced a defense you can rely on.
The reason is the mechanism. A language model reads one stream of text. Your system prompt, the user’s message, the web page your agent just fetched and the last tool result all arrive as tokens in the same channel. Nothing marks one span as privileged and another as untrusted.
SQL injection has parameterized queries, which separate code from data at the interface. Language models have no equivalent. Willison proposed one in 2022, then wrote in April 2023 that it looks extremely difficult, if not impossible, on current architectures. The person who named the attack retired his own fix seven months later.
The old way vs the new way
Old way: try to win the classification problem. Teach the model, or a filter, to tell good text from bad text.
- Filtering and escaping. There is no character class to strip, because the payload is ordinary language. An exploit that survived JSON quoting was already in the post that named the problem.
- Telling the model to ignore injected instructions. That line is just more text in the same channel. It competes on persuasiveness, not privilege, and the attacker writes last.
- Defenses tested only against a fixed list of attacks. One preprint evaluated eight defenses and bypassed all of them with adaptive attacks, at over 50% success. A defense that folds once the attacker knows it exists was a benchmark score, not a security control.
New way: assume the injection lands, then limit what it can touch. The research is moving the defense outside the model. The June 2025 design patterns paper (Action-Selector, Plan-Then-Execute, Dual LLM and others) does not ask the model to be harder to fool. Each pattern restricts what the model’s output is allowed to cause. The CaMeL system keeps untrusted data from ever impacting program flow and enforces policies when tools are called.
A few honest caveats. All of this is preprint-level, not proven in deployment. The paper’s own authors say general-purpose agents are unlikely to get reliable guarantees on current models, so the patterns are scoped to application-specific agents. And CaMeL’s 77% is a utility number: the share of tasks still completed under the defense (84% undefended), not the share of attacks blocked. A June 2026 preprint points out that these out-of-band defenses are mostly validated only on static benchmarks.
Notice the pairing in OWASP’s current list. Prompt Injection is LLM01. Excessive Agency is LLM03. The attack comes first, and the thing that turns it into an incident comes third.
Practical steps
If you are writing a spec for an agent, put these four moves in it. 🛡️
- Give each component the narrowest tool set that still finishes the job. A capability the agent does not have is one an injection cannot borrow.
- Require a recorded human approval before every irreversible action. Send, delete, pay, publish, transfer. One tip from the comments: show the reviewer the raw action and its arguments (recipient, amount, file path), not the model’s own summary. The summary is text the injection can shape too.
- Treat every span of outside text as hostile. Fetched pages, email bodies, file contents, tool results, other agents’ output. If your agent keeps notes between sessions, include its memory files. Outside text copied into them becomes the next run’s trusted input.
- Make actions observable afterwards. A successful injection should leave a trail, not a mystery.
A quick gut check from the comments is Willison’s lethal trifecta. An agent is exploitable when it has private data, sees untrusted content, and can send data out. Remove any one of the three and the worst case gets a lot smaller. 🔧
One more idea from the thread: run a second model with no tool access to check the first one’s behavior. It adds a layer, but remember the research. Treat it as a speed bump, not a wall, and keep the hard limits from the list above underneath it.
None of this stops the injection. All of it decides how bad the injection is.
Your move
Open the spec for the agent you are building right now. For each tool it can call, ask what happens if an attacker is the one calling it. Cut the tools that answer “something terrible,” and put a human approval step in front of the rest that cannot be undone. That one afternoon of work beats another week of prompt wording! ⚓
Sources: Simon Willison’s weblog (Sept 2022 original post, April 2023 Dual LLM post, June 2025 design patterns write-up), the arXiv papers on design patterns, CaMeL, adaptive attacks and out-of-band defense evaluation, and the OWASP GenAI LLM Top 10 repository.
Frequently Asked Questions
Q: What should a human reviewer see before approving an agent action?
Show the raw action and its arguments, such as the recipient, amount, or file path, instead of the model’s own summary of it. The summary is text an injection can shape too, so a reviewer can end up approving a nice sounding story while the real action is something else.
Q: How do I keep one injected page from doing real damage?
Make sure no single task has all three of these: private data, untrusted content, and a way to send data out. Simon Willison calls this the lethal trifecta. A summarizer for web pages gets no secrets and no network access, and the agent with repo or inbox access doesn’t browse the web.
Q: Is it worth putting a second LLM in front to scan incoming text?
Some people run a model with no tool access to scan the prompt, and some add a second one to check how the first one behaves. Both add a useful layer. A detector is also a language model that can be fooled, though, so use it to catch the obvious attempts and don’t count on it as the last line of defense.
Q: Does prompt injection matter for an agent that keeps notes between sessions?
Yes, because the memory files become the trusted input for the next run. Injected text copied into them can come back with more authority than it arrived with. Keep fetched text out of your notes except as a short quote that is clearly marked, and never write a log line as if it were an instruction to act on.
Prompt injection, what actually stops it
by u/Virtual_Hair_1987 in PromptEngineering