u/GuardComfortable6762

I ran a prompt-injection test suite against qwen2.5 (7B/14B) and mistral under a bare agent scaffold. All scored 0% resistance.
▲ 1 r/PromptEngineering+1 crossposts

I ran a prompt-injection test suite against qwen2.5 (7B/14B) and mistral under a bare agent scaffold. All scored 0% resistance.

I built a small offline tool that checks whether an agent resists prompt injection: give it a rule ("never reveal this secret"), give it tools (file read, messaging), then run documented injection cases and score resisted vs. complied.

Ran it against qwen2.5:7b, qwen2.5:14b, and mistral via Ollama, under a deliberately minimal scaffold (system-prompt guardrail + raw tools, no extra filtering). All three scored 0%. In one case, the agent read a poisoned notes.txt it was asked to summarise and called send_message to an external address with the secret in the body.

Two honest caveats: these are small models in a bare setup, so it's an early signal, not a verdict on the models. And my first run reported ~50% until I realised the detector was scoring stalled, no-answer runs as passes; fixing that gave the real 0%.

Fully offline, MIT, reproducible with one command. I'd love for people to run it on their own models/scaffolds and tell me where it's wrong.

github.com/ishan-1010/agent-injection-suite

u/GuardComfortable6762 — 7 days ago