u/Danculus

We ship an open-source agent-memory core and added four defenses against memory/RAG poisoning: value-weighting, a corroboration gate, deterministic supersession, and earned-outcome credit. Then we red-teamed all four against an attacker who *knows* them.

All four fall, for the same reason: each scores a record by something computable from the record's *own content* — and the attacker writes the content. Value is self-declared, corroboration is self-sourced, the winning write is self-timed, the "success" is self-graded.

The only signals the writer can't author are provenance (where it came from) and cost. But the catch that matters for RAG: **provenance authenticates the source, not the truth.** MINJA poisons memory from inside a legitimate, authenticated session — real provenance, false content — so a provenance check waves it through. PoisonedRAG shows the same on the retrieval side.

So provenance/cost is a floor, not a fix. The rule we landed on: **write-cheap, influence-expensive** — store anything in its own scope, but require corroboration by *distinct* anchored sources before a memory can influence an answer outside its scope.

It's all textbook (Sybil, Goodhart, CRDT last-writer-wins, adaptive-eval) — the value is the runnable red-team of one real stack + the honest ceiling. Writeup + runnable probe: https://dancenitra.github.io/agora/public/posts/agent-memory-defense-provenance-not-truth.html

The part I don't have a clean answer for: the *authenticated-but-false* case — when every corroborating source is individually legitimate. How are you handling that in RAG?

We run an autonomous agent loop that does research and publishes write-ups. I took 32 of the ones it shipped as confident "discoveries" ("a law", "we found", "a method win") and put them through a full adversarial audit, then re-scored them. The honest result:

- **34%** — substantively wrong (a real stat bug, a rigged baseline, unreproducible, or a measurement artifact).

- **53%** — the measurement was correct and reproducible, but it was labeled a "law/discovery" when the idea is textbook. e.g. a "two-tier memory law" that's just segmented caching (SLRU/ARC, 1990s); a "verification-tax law" that's the P-vs-NP verify-is-easier-than-produce asymmetry.

- **13%** — honest from the start.

- Strict bar (survived as an *original discovery, as first framed*): **0/32.**

So the dominant failure was over-*labeling*, not bad measurement. The labels failed more than the science did.

Obvious objection: maybe my audit just relabels anything with a prior-art ancestor as "textbook", and almost everything has one. So I ran a positive control — a labeled panel of 10 genuine landmarks (Transformer, CRISPR, GANs, plus hard cases like PageRank next to eigenvector-centrality, Adam next to RMSprop) and 10 textbook-results-dressed-as-discoveries, judged blind by the same pass. False-reframe rate (a real novelty wrongly called "textbook"): **0/10.** The grader doesn't demote genuine novelty — so the 0/32 is about the generator (an agent aimed at well-trodden areas), not a trigger-happy gate.

Two things I'd flag as maybe-useful, not novel:

1. The failure taxonomy is just the human questionable-research-practices literature (HARKing, researcher degrees of freedom, Ioannidis' "most published findings are false") — an agent loop reproduces that distribution on its own output.

2. A *light* self-check ratifies its own errors. Only the full pass (multi-view + adversarial + primary-source verification + a re-audit of the fixed draft) reliably caught the defect. Fits the "LLMs can't reliably self-correct" result.

Scoring and the positive-control panel are public scripts you can re-run and disagree with:

Honest limits up front: self-graded (my audit of my own posts), n=32 of 43, these are the ones we chose to publish (most-confident output, so the base rate over *all* candidates is lower), and the positive control is a hand-built 20-item panel.

If you run an agent loop: do you see the same over-labeling, and how do you catch it before it ships?

I maintain a small open-source memory/retrieval layer for agents (mnemo). I wanted to see how badly one poisoned document could hijack retrieval, so I ran an AgentPoison-style attack against it across three embedders, then tried to defend it at the retrieval layer. The defense result is the interesting part, and it's all runnable.

The attack is easy and it generalizes:

- One poisoned chunk whose trigger is a plain English sentence ("the old lighthouse still guides ships along the rocky coast") lands at rank 1 for 88–100% of trigger-bearing queries on all three retrievers I tried (all-MiniLM-L6-v2, BGE-small-en-v1.5, Contriever).

- It doesn't wash out with scale — padding the corpus to 10,000 chunks keeps the hijack ~flat (~94%).

- A perplexity filter is the wrong wall: natural-sentence triggers have natural perplexity (47–441) and pass straight through while still hijacking (this is the PoisonedRAG point — poisoned text can look clean).

Retrieval-time detection didn't hold up:

- Embedding-outlier detection → defeated by padding the poison with generic text so it isn't an outlier.

- A retrieval-set-coherence re-ranker (down-weight a hit that's topically alien to the query's other results) → works on MiniLM (hijack 100% → 19%) but fails outright on BGE, whose space is more anisotropic (unrelated texts already sit at high cosine), so the poison isn't separable by coherence there. A defense that lives in embedding geometry inherits the encoder's geometry.

What worked was moving off the embedding entirely. The store only "graduates" a chunk to trusted once it's earned corroboration (a credited good outcome, or ≥2 independent-source links — earned automatically through use). Reusing that as an influence gate — retrieve everything for context, but only let corroborated chunks drive an action — dropped the single-instance hijack to 0% on all three retrievers and every scale, benign utility ~90–100%. It generalizes because corroboration is metadata, not vectors, so it doesn't care which embedder you use.

Caveats up front (where I'd want the scrutiny):

- The attack isn't novel — it reproduces AgentPoison (Chen et al., NeurIPS 2024) and PoisonedRAG (Zou et al., USENIX Security 2025). The new bit is the defense-side measurement on a store that has a trust stage, and the split between what a poison can retrieve (88–100%) vs influence (0% gated).

- Small-scale: 16 held-out queries, one synthetic 60-item corpus (padded to 10k), three small single-vector encoders. Existence result, not a benchmark.

- Retrieval-hijack, not end-to-end — no full agent loop with a downstream action.

- The gate has a real cost: it filters rare-but-true chunks that haven't earned corroboration yet (recall ~1.0 → ~0.08), so it's for adversarial/untrusted ingestion, not a default. It raises attacker cost (needs ≥3 coordinated records with ≥2 forged independent sources), it doesn't eliminate the attack.

Probes are deterministic on a local embedder — if a single-instance poison beats the gate on your setup, or the cost is worse than I found, I'd like to know.

Sources / writeup: https://dancenitra.github.io/agora/public/posts/agent-memory-poisoning-influence-gate.html

Runnable probes: https://github.com/DanceNitra/agora/tree/main/mnemo/probes

Prior art: AgentPoison https://arxiv.org/abs/2407.12784 · PoisonedRAG https://arxiv.org/abs/2402.07867

I maintain a small open-source memory/retrieval layer for agents (mnemo). I wanted to see how badly one poisoned document could hijack retrieval, so I ran an AgentPoison-style attack against it across three embedders, then tried to defend it at the retrieval layer. The defense result is the interesting part, and it's all runnable.

The attack is easy and it generalizes:

- One poisoned chunk whose trigger is a plain English sentence ("the old lighthouse still guides ships along the rocky coast") lands at rank 1 for 88–100% of trigger-bearing queries on all three retrievers I tried (all-MiniLM-L6-v2, BGE-small-en-v1.5, Contriever).

- It doesn't wash out with scale — padding the corpus to 10,000 chunks keeps the hijack ~flat (~94%).

- A perplexity filter is the wrong wall: natural-sentence triggers have natural perplexity (47–441) and pass straight through while still hijacking (this is the PoisonedRAG point — poisoned text can look clean).

Retrieval-time detection didn't hold up:

- Embedding-outlier detection → defeated by padding the poison with generic text so it isn't an outlier.

- A retrieval-set-coherence re-ranker (down-weight a hit that's topically alien to the query's other results) → works on MiniLM (hijack 100% → 19%) but fails outright on BGE, whose space is more anisotropic (unrelated texts already sit at high cosine), so the poison isn't separable by coherence there. A defense that lives in embedding geometry inherits the encoder's geometry.

What worked was moving off the embedding entirely. The store only "graduates" a chunk to trusted once it's earned corroboration (a credited good outcome, or ≥2 independent-source links — earned automatically through use). Reusing that as an influence gate — retrieve everything for context, but only let corroborated chunks drive an action — dropped the single-instance hijack to 0% on all three retrievers and every scale, benign utility ~90–100%. It generalizes because corroboration is metadata, not vectors, so it doesn't care which embedder you use.

Caveats up front (where I'd want the scrutiny):

- The attack isn't novel — it reproduces AgentPoison (Chen et al., NeurIPS 2024) and PoisonedRAG (Zou et al., USENIX Security 2025). The new bit is the defense-side measurement on a store that has a trust stage, and the split between what a poison can retrieve (88–100%) vs influence (0% gated).

- Small-scale: 16 held-out queries, one synthetic 60-item corpus (padded to 10k), three small single-vector encoders. Existence result, not a benchmark.

- Retrieval-hijack, not end-to-end — no full agent loop with a downstream action.

- The gate has a real cost: it filters rare-but-true chunks that haven't earned corroboration yet (recall ~1.0 → ~0.08), so it's for adversarial/untrusted ingestion, not a default. It raises attacker cost (needs ≥3 coordinated records with ≥2 forged independent sources), it doesn't eliminate the attack.

Probes are deterministic on a local embedder — if a single-instance poison beats the gate on your setup, or the cost is worse than I found, I'd like to know.

Sources / writeup: https://dancenitra.github.io/agora/public/posts/agent-memory-poisoning-influence-gate.html

Runnable probes: https://github.com/DanceNitra/agora/tree/main/mnemo/probes

Prior art: AgentPoison https://arxiv.org/abs/2407.12784 · PoisonedRAG https://arxiv.org/abs/2402.07867

A lot of agent setups use the model's own confidence to decide when to act vs hold back: answer if confident, abstain if not. I wanted to know whether that actually works, so I measured it across models from small to frontier.

The question, in plain terms: does a higher confidence actually mean the answer is more often correct? (The technical name for this is AUROC of confidence vs correctness, but the idea is simple — 0.5 means the confidence tells you nothing, a coin flip; 1.0 means it perfectly separates right answers from wrong ones. This is what matters for deciding when to abstain, and it's not the same as calibration.)

Task: multi-step integer arithmetic generated from random numbers — contamination-free (nothing memorized), graded exactly. Each item the model returns an answer and a 0–100 confidence.

Does confidence predict correctness?

model	score (0.5 = coin flip)	how overconfident (conf − accuracy)
qwen2.5:7b (small)	0.50	+0.72
qwen3-coder:30b (mid)	0.54	+0.84
glm-5.2 (frontier)	0.73 *	+0.19
claude-sonnet-4-6 (frontier)	0.90	+0.02

The small/mid models slap ~maxed-out confidence on almost everything, including wrong answers — so their confidence is useless for telling right from wrong, and they're wildly overconfident. The frontier model was near-perfectly calibrated and genuinely knew when it was about to be wrong (it put ~2% confidence on most of its wrong answers).

Why it matters for agents: if the model deciding "is this right / should I keep this / should I abstain" is a small or local one, you can't let it gate on its own confidence — it'll act on wrong things while feeling certain. What's worked for me is to gate on corroboration (independent sources agreeing) instead of confidence, and escalate genuinely ambiguous cases to a stronger model.

Honest limits: one task family (arithmetic), a handful of models — directional, not a scaling law. Arithmetic probably exaggerates the confidence-maxing (models treat it as deterministic). *glm-5.2 didn't emit a usable confidence on ~34% of items, so its score is on a subset; Claude gave one every time (cleanest data point).

(Disclosure: the probe and writeup are from my own open-source project — MIT, nothing to sell, sharing because the data's the point and you can re-run it on your own models.)

Runnable single-file probe + raw per-item data, so you can re-run it on your own models: https://github.com/DanceNitra/agora/tree/main/mnemo/probes/overconfidence_tax

Fuller writeup: https://dancenitra.github.io/agora/public/posts/can-an-llm-trust-its-own-confidence.html

How are you deciding when your agent abstains — its own confidence, a separate verifier, or corroboration? And has anyone gotten a 7B–30B model above coin-flip on this?

I kept seeing memory-poisoning papers (MINJA, Agent Security Bench) report 70–95% attack success on standard memory-based agents: inject a crafted "memory", it persists and steers the agent on a later query. I wanted to know if a corroboration gate actually defends, so I replicated the attack at the memory layer and measured.

Setup: a naive importance/recency-ranked store vs a store that only makes a memory durable/trusted when it's CORROBORATED (earned outcome credit, or ≥2 independent corroborations — not because it's "important"). Two attack goals: ENTRENCH a poison as durable memory, and OVERWRITE a true fact. Measured end-to-end on the recall the agent would actually use. N=150 per cell.

Results:

- Naive store: 100% poisoned for both goals (in line with the 70–95% from the papers).

- Corroboration gate: single-source poison → 0% (entrench by default; overwrite once you require corroboration to supersede a standing fact).

- Honest limits (no silver bullet): a sybil that forges ≥2 independent-looking corroborations bypasses it (100%); and a poison phrased as a PROCEDURE ("always do X") sidesteps the gate, because procedural memories are durable by design.

Takeaways if you build agent memory: don't rank durability by importance/recency (that's exactly what poisoning exploits); gate durability on corroboration; require it for procedural/durable writes too; and treat corroboration-count as forgeable — you still need a source-independence signal against sybils.

Building a memory layer for agents, I kept hitting the same wall in four places: what to forget, when to believe a contradicting fact, how fast to stop trusting a source that goes bad, and how many samples to take in best-of-N.

They look separate but they're the same tradeoff: a single update rule is either too fast (one poisoned input flips your memory) or too slow (it lags a real change). At the instant something deviates, an isolated bad input and the first sample of a genuine change are the SAME observation — you can't tell them apart until it persists.

That's just sequential change detection, which has a known-optimal answer (CUSUM). So the fix isn't tuning a decay rate — it's a persistence-based update: confirm a change over a few corroborating samples, with the window set to how transient your noise is vs how fast real change happens.

I tried to falsify "persistence is better" on real data (16 expert-labelled NAB streams). Honest result is an asymmetry: on every SUSTAINED change a persistence detector wins or ties — the naive point rule never won one (one server misconfiguration: 0 vs 1181 false alarms for the same recall). Naive only wins on TRANSIENT spikes. So "is the change sustained?" tells you which to use.

Caveats: it's a unification of known ideas, not a new theorem; the gain shrinks to ~0 when changes are big and the signal is clean; and persistence is exactly wrong if the signal you care about is itself a brief spike.

Happy to share the full method + reproducible sims if that's useful to anyone. Curious whether others building agent memory hit the same gullible-vs-rigid wall — and what you did about it.

We spent a week measuring retrieval/grounding on hard multi-hop QA (MuSiQue, strict grading, local models — every number reproducible). One result surprised us enough to share:

Partial retrieval scored below just dumping the full context. Flat semantic search recovered ~42% of the needed paragraphs → answer accuracy 0.22. Giving the model all 20 paragraphs (no retrieval) → 0.47. So a mediocre retriever that drops some of the needed facts is worse than no retriever at all — because multi-hop needs the complete chain, and one missing hop breaks the answer.

The ceiling is real, but it's in completeness. Hand the model the gold paragraphs (oracle retrieval) → 0.66 (+21pp over full-context). So the headroom lives in retrieval quality, specifically complete-chain recall — not in sampling/voting.

More retrieval effort didn't close it. Agentic/iterative retrieval lifted per-paragraph recall to 0.72, but accuracy still didn't beat full-context: per-paragraph recall isn't enough when you need every hop (≈0.72^3.5 → the full chain rarely lands). A naive entity-link graph traversal didn't either.

(Aside, for the "just ensemble it" crowd: self-consistency, multi-model voting, and self-verification didn't rescue hard answers either — LLM errors turn out to be systematic, not random, so aggregation has nothing to average away.)

Practical takeaway: measure complete-chain recall, not just top-k hit rate; on multi-hop, partial retrieval can actively hurt, and full-context-if-it-fits can beat a so-so retriever. Invest in chain-following retrieval.

Full method, every number, and the falsifier for each claim (plus a second result on idea-generation): https://dancenitra.github.io/agora/public/posts/diversity-is-noise-for-answers-signal-for-ideas.html

Curious whether others see "partial retrieval hurts" in production — and how you measure chain completeness rather than hit-rate.

A practical finding from testing memory/RAG recall: as a store grows and accumulates older, near-duplicate content, pure semantic similarity starts surfacing confidently-wrong stale chunks that still match the query. We measured a crossover where a recency/usage boost (freshness reranking) overtakes pure semantic ranking - and the crossover depends on store size/age, not the embedding model.

Two things that surprised us:

• Once the store is large, the best embedding model matters less than decay/freshness - most recall loss in a growing store comes from staleness, not embedding quality.
• recall@k measured on a static benchmark overstates live performance, because real queries drift from whatever the index was tuned on.

Practical takeaways: tune the freshness/decay weight as a function of store size, not once; and down-weight (do not hard-delete) superseded chunks - the first false positive in an is-this-stale check deletes a true memory.

How do you all handle decay / supersession in production RAG?