We ran a 1,655 person blind study on AI memory. The results changed how we think about the problem.
We’re building KAPEX (getkapex.ai), memoryware for AI applications. Two co-founders, bootstrapped, patent pending. I wanted to share some of what we’ve learned because the discourse in this space keeps circling the same assumptions and I think a few of them are wrong.
The study: 1,655 participants interacted with AI systems with and without our memory layer. Blind setup, they didn’t know which condition they were in.
The finding that mattered most: first-session preference was around 65%. Not bad, but not a clear signal. After 20+ sessions, preference climbed past 80% and kept rising. The longer people used it, the wider the gap.
That trajectory is the insight. Not the final number. The trajectory.
Here’s why that matters for anyone building in this space:
Most AI memory tools are optimized for first impressions. Demo well, retrieve fast, show the user you remembered their name. That’s fine. But it means the entire evaluation framework for memory (including the benchmarks everyone cites) is testing the wrong thing. LongMemEval and LoCoMo test whether you can find what was said. They don’t test whether the system knows what still matters.
Retrieval and relevance are different problems. The industry has spent two years building better retrieval. Almost nobody is building relevance governance: what stays important, what fades, what gets superseded, and whether the user can see and correct what the system believes.
Three things we learned the hard way:
1. Clean store beats fancy retrieval. Every time. If your memory layer lets stale context accumulate without governance, no amount of reranking or hybrid search fixes the degradation over time. The capture and maintenance side is where the leverage actually is.
2. Memory without transparency is a black box. If developers can’t see why the agent believes something, and users can’t see what the system thinks it knows about them, then memory becomes a liability rather than a feature. Inspectability isn’t a nice-to-have. It’s what makes correctability possible.
3. The value of memory is invisible in short sessions. This is why benchmarks miss it. A 5-turn evaluation can’t distinguish between a system with real governance and one that just retrieved the right vector. The difference only shows up after sustained use, which is also when it matters most.
Our approach treats relevance as something that should be handled continuously by the architecture, not at query time by the retrieval layer. Context that stops being reinforced through usage naturally loses priority. Not deleted, just deprioritized. That’s the principle. Can’t share more on implementation for IP reasons.
Curious what others here are seeing. Is anyone else finding that the retrieval-first paradigm breaks down over time? And is anyone working on evaluation frameworks that test sustained-use performance rather than single-session recall?
getkapex.ai if you want to follow along. Still pre-launch but opening access soon.