Realized the other day that “AI reads your instructions” and “AI reads an attacker’s instructions” look identical to it
Been thinking about this since a weird moment last week: I gave an agent a task, it pulled in a file to help, and the file had a line in it that wasn't meant for me, it was meant for the agent. And the agent just read it. No way for it to know that line wasn't part of my actual request.
That's when it clicked that "instructions" and "just some text sitting in a document" are the same thing to a language model. There's no compiler or type-checker telling it "this part is a command, this part is just content," the way there is for basically every other kind of software. It just reads language and decides what to do.
We've spent decades building security around the idea that data and instructions are different things. Agents don't really have that line, and existing standards don't have a home for what breaks because of it: a CVE describes a flaw in a specific package and version, there's no package here. CWE describes a weakness in code, there's no code being executed in the traditional sense, just text being interpreted.
Ended up deep enough in this that a few of us built AVE, an open standard that names these behavioral patterns directly instead of trying to force-fit them into categories built for a different kind of system, 70 records so far, crosswalked into OWASP's and MITRE's own frameworks so it's not reinventing anything that already exists elsewhere.
github.com/aveproject/ave if anyone here working in security wants to poke at it or tell me where it's wrong. Genuinely curious if the "no data/instruction boundary" framing matches how security folks here are already thinking about this, or if there's a sharper way to put it.