Why AI Security Frameworks Fail in Practice

Most AI security frameworks are competent documents that describe controls for a system with a boundary between code and data. That boundary is what language model applications remove, and the frameworks do not notice, so the checklist stays green while the actual gap widens underneath it.

This is not a criticism of any particular standard. It is a structural mismatch, and recognising it is the difference between security work that transfers and security work that produces documentation.

The assumption that no longer holds

Application security rests on being able to distinguish instructions from input. Parameterised queries work because the query is code and the parameter is data, and the database can tell which is which. Almost every control in a traditional framework inherits that separation somewhere.

A language model has no such separation. The system prompt, the retrieved document, the user's message and the tool response arrive as one undifferentiated sequence of text. You can label the parts, and the model will usually respect the labels, and usually is doing enormous work in that sentence. There is no enforcement mechanism, only a strong tendency.

Every control premised on a trust boundary maps onto the parts of your stack that still have one, and quietly fails to cover the parts the model mediates.

Where the controls stop mapping

Framework control	What it assumes	What actually happens
Validate untrusted input	Untrusted input is identifiable	Instructions arrive inside retrieved documents
Enforce least privilege	Privileges attach to a caller	The agent holds one identity for every task
Log security-relevant events	Events are discrete and typed	The decision was a token sequence
Segregate duties	Steps are performed by parties	One agent plans, approves and executes
Test for known vulnerabilities	Vulnerabilities are deterministic	The same input succeeds intermittently

The privilege row is the one worth dwelling on. An agent typically runs with a single service identity carrying the union of every permission any of its tasks might need. Least privilege is satisfied on paper, because the identity was scoped deliberately, while in practice a request that should touch one record can reach everything the agent can reach. The framework asked whether privileges were scoped. It did not ask whether they were scoped per task, because in a conventional system that distinction rarely arises.

Indirect injection is the part that gets missed

Most threat models treat the user as untrusted and retrieved content as data. That is backwards for agent systems.

If your agent reads web pages, documents, tickets or emails, then anyone who can write to those surfaces can write instructions into your context. They never have to touch your application. A sentence in a document, invisible to a human skimming it, is read by the model with the same weight as your system prompt.

The consequence is that your trust boundary is not your API. It is the union of every source your retrieval layer can reach, and almost nobody has that inventoried. Ask who can write to the knowledge base, and whether that set was ever reviewed as a security question rather than a permissions convenience.

What genuinely transfers

Data governance survives intact, and it is the highest-value work available.

Knowing what data exists, where it lives, who may see it, and what is permitted to leave the boundary is exactly as relevant as it ever was. When a model-layer control fails, and some will, governance is what determines whether the failure is an incident or a catastrophe. An agent that cannot reach the sensitive store cannot leak it, regardless of how thoroughly it was manipulated.

Access control and logging transfer too, on one condition: the inventory has to be extended. Models and their versions, system prompts and their change history, every tool and its permissions, every retrieval source and who can write to it. Teams that maintain a rigorous software bill of materials frequently have no record of when a system prompt last changed, which is a strange asymmetry given the prompt determines the behaviour.

What does not work

Input filtering as a defence against injection. It is a denylist against an unbounded space of phrasings, and it loses to rewording, encoding and indirection. It is worth having as a speed bump and worth nothing as a control.

Point-in-time red-teaming. An adversarial exercise against a model produces a statistical result, this class of input worked some fraction of the time, on that day, against that version, with that prompt. Change the model version and the finding may not reproduce. The report format implies a binary that the underlying reality does not support.

A second model as a guardrail. Screening outputs with another model inherits the failure modes of the first and is itself injectable, since it also reads the untrusted content. It catches unsubtle cases and gives disproportionate confidence, which is the worst combination.

Compliance as a proxy for security. Every control can be satisfied on a system that is trivially manipulable, because the controls were written for a different architecture. A green checklist here is evidence about paperwork, not about the system.

What we do not know

We do not know how to bound injection risk rather than reduce it. Everything credible is mitigation, narrowing the action space, requiring confirmation before consequential steps, keeping the sensitive data out of reach, and none of it is a proof of anything. Anyone offering a solution rather than a mitigation is describing a system that can reliably distinguish instruction from data, which is the open problem.

We are also unsure how to test any of this properly. Because the behaviour is probabilistic and version-dependent, a finding has a shelf life nobody has characterised, and we do not have a defensible answer for how often adversarial testing should be repeated beyond "more often than anyone does".

The practical position

Treat the framework as covering the conventional parts of your stack, which it does well, and assume it says nothing useful about the model-mediated parts, because it does not.

Then do the two things that actually bound the damage: constrain the action space so a successful injection reaches something survivable, and get data governance right so the blast radius is small. Both are architecture decisions, made early, and neither can be retrofitted from a checklist — the same reason supervisability has to be designed in rather than added afterwards. Expect the failures to be quiet, since nothing in the pipeline is positioned to report them.