u/wonker007

Most of us faced some version of the same problem dealing with AI in work and life: Memory retrieval for AI eventually disappoints because we expect human-like retrieval but often get trash.

Drilling down deeper, one realizes that we are more often than not expecting random-access multi-hop retrieval - because that’s how our human memory works. But what we currently have as tools are graph crawling, cosine lookups or (gasp) regex matching. Who knew grep was such a powerful tool, token waste be damned?

So how do you make an AI system remember in a way that’s actually useful for humans? You model it after human memory, of course. Not a Frankenstein bolt-on mess of open-source code, but a designed-from-the-ground-up, built-from-scratch lean memory engine modeled after literal neurobiological systems.

My own frustration trying to fully utilize AI for my day job as a pharma/biotech consultant drove me to build this sparse tensor-based graph memory engine over the past few months — my PhD is in biochemistry, so I’m drawing from what I actually know rather than what sounds good on a pitch deck. And because I am a proud scientist (almost to a fault), I naively threw the engine against MuSiQue 1,000Q, which is as close to a real multi-hop memory recall test as we have in the literature. It could have gone horribly wrong, but if it did, you wouldn’t be reading about it.

The short version: F1 = 0.677 on the full 1,000Q corpus (highest published zero-shot end-to-end score as of May 2026, to the best of my knowledge). Yeah. Went quite a bit better than I expected.

Reader-controlled baseline with a compact local embedding model (nomic): 0.565 vs LlamaIndex at 0.418 and BM25 at 0.329.

But the number isn’t really the point. What I think matters more for anyone building memory systems is why this architecture works differently from established tools.

The recall problem nobody talks about

Vector similarity search answers “what’s close to this query in embedding space?” That’s fine for a simple lookup. Search, rank, done. But MuSiQue was specifically designed to defeat that mechanism — it was designed so that no single retrieved passage contains the entire answer. You need passage A to find passage B to find passage C. That’s memory traversal, not memory search. Graph crawling is also similarly limited as it must crawl edges at the risk of fanning out too thin before finding the next relevant node.

The engine builds a weighted graph where edges carry typed relationships (like various neurotransmitters) and activation energy propagates through connections (like how neurons fire) — nodes that are semantically distant but informationally connected either through logical relationships, provenance, hierarchy etc. still light up if the path between them has enough weight. Same principle as biological associative recall: you smell something and remember a childhood memory that has zero semantic overlap with the smell but a strong associative pathway.

That’s the architectural hypothesis. The benchmark results suggest it works. I posted the full methodology and honest limitations over on r/RAG (including the ~52% reader confound, PropRAG’s superior retrieval lift at +81.9% vs our +71.7%, and Beam Retrieval’s higher supervised score of 0.692) because I didn’t want to bury the caveats. Full transparency on what beat us and where. You can also see the full write-up with all the numbers: https://elucidx.ca/insights/2026-05-15-rag-needs-real-value/

The harness is public

The engine itself is proprietary and patent-pending — I’m not releasing source. But the evaluation harness, dataset, and scoring protocol are all public: [github.com/wonker007/musique-eval-harness]. If you’re building a memory system and want to know how it does on genuine multi-hop recall, run your system against the same corpus, same protocol with the same scorer and post the number. I’ll reference it.

I’m also currently running conversational-scale benchmarks (128K to 10M token range) testing temporal reasoning, knowledge updates, and contradiction detection — the stuff that actually matters for memory persistence over long interactions with AI. More data coming.

If anyone here is working on multi-hop recall architectures — whether that’s GraphRAG, memory-augmented transformers, or something else entirely — I’d love to hear what serious benchmarks you’re using and what you’re seeing. MuSiQue is good but it’s still Wikipedia passages, not production conversational data.

(Post was written with the help of AI, edited by me)

Been lurking and commenting here and there for a while, hinting at building something out of sheer frustration on crappy context management state of AI especially related to my day job in pharma and healthcare. So I just up and went on to build a new-from-the-ground-up graph-based retrieval engine and ran it through MuSiQue - the 1,000Q set.

This is not a wrapper, not a Frankenstein mish-mash of open source code. Legit new architecture based on what I know best - biology. And I think I’m as qualified as they come as a PhD in biochemistry working in biotech and pharma nearing twenty years now.

Posting the full results, methodology, and limitations here because I actually have the balls to put it all out there - and the results are damn impressive, if I do say so myself.

And yes, the dry bits below are written with the help of AI (thank you Claude) because this is an AI-related sub.

Setup

Same corpus as HippoRAG 2: 1,000 questions and 11,656 Wikipedia passages from their published HuggingFace dataset (osunlp/HippoRAG_2). 496 answerable questions scored. Evaluation metric: SQuAD F1 — deterministic token-level precision/recall, no LLM judge involved. All comparators (BM25, LlamaIndex) run through the same reader model (Gemini Flash, temperature=0) on the same hardware to control variables.

The engine is a Rust-based sparse tensor graph that retrieves through associative activation pathways rather than pure vector similarity search. It runs as a single 12.5 MB binary. The entire benchmark was run on a laptop (i7, 16GB RAM, RTX 3050 Ti).

Results

Reader-controlled baseline (same reader, same embedding model across all three):

System	F1
BM25 (whitespace tokenization, top_k=50)	0.329
LlamaIndex (nomic-embed-text-v1.5, 768d)	0.418
Donna-Alfred (nomic-embed-text-v1.5, "Eager Mode")	0.565

With optimized configuration (stronger embedding model (Gemini) + reader reasoning enabled): F1 = 0.677. To the best of our knowledge as of May 2026, this is the highest published zero-shot end-to-end F1 on MuSiQue. Yeah. Good stuff.

Total benchmark cost: $30.04.

Now the honest part

The 0.677 number needs context that I’m not going to bury. Three things:

Reader confound. HippoRAG 2 used Llama-3.3-70B as their reader; I used Gemini Flash. Comparing BM25 baselines across readers (theirs: 0.288, ours: 0.329), roughly 52% of the raw F1 gap between our baseline and HippoRAG 2’s published 0.486 is attributable to reader advantage, not retrieval quality. The fairer comparison is BM25-relative retrieval lift — how much each system improves over BM25 using the same reader:

System	F1	BM25 (same reader)	Retrieval lift
LlamaIndex (Flash)	0.418	0.329	+27.1%
HippoRAG 2 (Llama-3.3-70B)	0.486	0.288	+68.8%
Donna w/ nomic (Flash)	0.565	0.329	+71.7%
PropRAG (Llama-3.3-70B)	0.524	0.288	+81.9%

PropRAG beats us on retrieval lift. +81.9% vs our +71.7%. We are not claiming to be the best retrieval system in the world for everything. That kind of thing just can't exist. We are claiming competitive retrieval quality at a fraction of the computational cost — our embedding model was 137M parameters vs NV-Embed-v2 at 7-8B.

Supervised systems score higher. Beam Retrieval (Zhang et al., NAACL 2024), fine-tuned on MuSiQue’s own training data, reaches 0.692. Our engine is zero-shot — no task-specific training. The gap is 1.5 F1 points.

What the engine is NOT

It’s not open-source. It’s proprietary and patent-pending. I’m not releasing code, binaries, or API access. I will be opening up slots for alpha testers in the near future though, so stay tuned.

What IS public: the benchmark methodology, the dataset (HippoRAG 2’s published corpus on HuggingFace), the evaluation protocol, and the evaluation harness. The eval harness is here: https://github.com/wonker007/musique-eval-harness

Per the original protocol, the scoring metric is deterministic. Anyone can reproduce the comparator arms and verify the methodology claims independently.

I built this solo using AI - lots of AI. Claude, Gemini, Perplexity (well, Perplexity technically isn't AI but why not give a shoutout - RIP), ChatGPT. Part of me wants this to be proof that vibe coding can actually produce production quality software, although with over 1,300 quality and governance documents weighing in at over 145 MB (not code, just the markdown documentation part), it isn't exactly "vibe" coding per se. FYI, quality management principles were borrowed from my wheelhouse of pharma and diagnostics manufacturing.

As I said, my background is biochemistry and pharma commercial strategy, not CS. The architectural approach is neurobiology-inspired - associative activation over a sparse tensor graph, same way biological neural networks process and retrieve by spreading activation through synapse connections of varying affinities and through several different neurotransmitters. The CS establishment will probably hate this claim because there are so many kids claiming to have solved RAG by “modeling after biology and the brain”. But I actually have the credentials to back my claim up.

But the thing is, F1 doesn’t care about your pedigree or your claims, and neither does MuSiQue. This is hard data from hard code, plain and simple.

I say bring your benchmark data in with full transparency if you want to play with the big boys.

What I’m looking for from this community

Methodological criticism. If the experimental design has a flaw, I want to know. If there’s a comparator I should be running against, tell me. If the reader confound analysis is insufficient, challenge it. The full write-up with all the numbers, per-hop breakdowns, the 2×2 optimization matrix, production calibration curves, and the data sovereignty argument for single-binary deployment is here: https://elucidx.ca/insights/2026-05-15-rag-needs-real-value/

I’m also working toward formalizing this for peer-reviewed publication and running additional benchmarks as we speak (conversational RAG at 128K-10M token scale). More data coming.

And if you’re really interested, as I mentioned, I’m planning to open up alpha testing in the near future, probably when I finish up the conversational benchmark. Only serious enterprise-level engineers need apply - it’s a highly-customizable drop-in Rust-based RAG engine with 70+ tunable variables on a clean API surface.