I've been building TokenMizer, a local proxy that sits between your editor/CLI and whatever model you're using (Claude, GPT, Ollama) and handles two things I kept re-solving by hand: session checkpoint/resume, and a graph-based memory instead of a flat transcript.

The problem: once a long agent session hits the context limit, the usual fix is summarization, and summaries lose the reasoning behind a decision, not just the decision itself. I'd see a summary saying "switched to Argon2" with no trace of why bcrypt was rejected, so the agent would re-litigate the same tradeoff two sessions later. Flat transcripts have the opposite problem: everything is kept, but nothing is prioritized, so retrieval is just recency-biased keyword luck.

What TokenMizer does differently: instead of one growing text blob, decisions, constraints, and open questions are stored as nodes with edges (this decision depends on that constraint, this question was resolved by that decision). Checkpointing snapshots that graph plus a resumable session state, so you can kill a session and pick it back up without replaying the whole history through the model again.

Where it's rough: there's no eval harness yet comparing retrieval quality against a naive flat-transcript baseline, so right now my evidence is anecdotal (my own sessions), not benchmarked. I also learned the hard way that benchmarking your own memory system by asking it questions only it can answer is circular, so I'm holding off on publishing numbers until I have an honest comparison.

Repo: github.com/Shweta-Mishra-ai/tokenmizer (I'm the author). It's a Python project, MIT licensed. If you've hit the same summarization-loses-reasoning problem, I'd be interested in how you're handling it, and PRs/issues on the eval-harness gap would genuinely help.

I hear a lot of people argue about the need for agent memory vs simply using skills, files context, state management systems, vector DBs, etc.

And I have been writing the pitfalls of each of these approaches individually in our research paper, blogs and more.

But the single most important way to look at agent memory is not in a single dimension of accuracy, cost or speed alone; but looking at them together.

The counters to agent memory are akin to: you give a case to a lawyer and ask them to refer to the case papers at every argument in court. Or a surgeon referring to the case papers at every step of the surgery. Or a therapist reading through notes before every response to the patient.

Memory is the approach that's needed to jump to the right approach, quickly and least wastefully; because an agent needs to do it several 10s of times every turn and it needs to be accurate, fast and efficient.

Hey everyone,

Like many of you, I've been building and experimenting with AI Agents. But as a software architect who hates unnecessary overhead, watching the community spin up heavy Vector DBs and complex graph frameworks just to store conversation context felt like massive over-engineering.

So, I built a lean alternative.

👉 Repo is here if you want to skip the text and jump straight into the code:https://github.com/phucphungbk/lean-ai-memory

The core problem: We often use massive systems to solve small problems. I wanted an AI memory system that is zero-cost, serverless, and completely transparent.

The Lean Approach:

1. Git as the Core Engine: We already use Git for version control. It turns out it's absolutely perfect for managing conversation history. You can easily track, diff, and rollback an AI's "thought process" just like reverting a commit.
2. Markdown as the Storage Format: It’s lightweight, humans can read and debug it instantly, and LLMs parse it perfectly without needing complex embedding pipelines.
3. Zero-cost & Portable: It can be packaged as an independent module and integrated directly into internal automation tools without incurring any DB maintenance costs.

I’m open-sourcing this with a completely open mindset. Instead of optimizing it in a silo, I want to see how this framework holds up in the wild. I'm highly anticipating the community bringing their own battle-tested custom rules into the system to push its boundaries.

I’d love for you guys to clone it, tear the architecture apart, test it, and drop your feedback or PRs. Let me know what you think!

EDIT 2 (14 Aug): Numbers below have changed. Following the judge-model point raised in the comments, I confirmed the 0.831 run was judged by gpt-4.1-mini, not the gpt-4o-mini the paper uses. Re-judged with the paper's judge, the figure is 0.8175. I also found my per-category labels were wrong — LoCoMo's categories are multi-hop / temporal / open-domain / single-hop, and I'd labelled them with LongMemEval's vocabulary, so what I called "multi-session" is temporal reasoning. Separately, 8 answers lost to an OpenAI outage mid-run had never made it into the saved output file; they're restored and the data now reproduces its own scores. Everything below reflects the corrected numbers, and the run data is now published so you can check it.

EDIT (13 Aug 22:41 GMT): A commenter correctly identified an error in the headline comparison. The 0.801 figure I used for Memora comes from Table 3 of the paper — a component build-up ablation row ("primary abstraction, with update"), not the published system result. Memora's actual scores from Table 1 are 0.849 (semantic retriever) and 0.863 (policy retriever). My 0.831 sits below both of those, so the headline as written overstates the comparison.

--

The numbers were run using Memora's (Microsoft Research, arXiv:2602.03315) open-source benchmark harness. The eval and scoring code is unmodified — I added a Recordari adapter for the memory backend. Same LoCoMo dataset, same category-exclusion convention (adversarial excluded, 1,540 scored questions), and the same models the paper specifies: gpt-4.1-mini for memory curation and answering (§5.1), gpt-4o-mini as the LLM judge (Appendix B).

Judge-matched, Recordari scores 0.8175 end to end against Memora's published 0.849 (semantic retriever) and 0.863 (policy retriever). It is below both.

Per category is the more interesting read. Against Memora S: multi-hop 0.801 vs 0.784, temporal 0.863 vs 0.851, open-domain 0.594 vs 0.594, single-hop 0.831 vs 0.900. Weighted by question count, the entire 0.032 gap is single-hop detail recall — extraction paraphrases specifics away ("salads, sandwiches and homemade desserts" becomes "dinner") where raw verbatim storage keeps them. Parity or better on everything else.

The internal before/after: raw batch storage 0.8065 → extraction 0.8175 overall, and on temporal questions 0.757 → 0.863. That gain comes from resolving relative dates to absolute at write time. Worth noting the aggregate difference (+0.011) is far smaller than the category one, and the judge model alone is worth 0.010–0.017, so I'd treat anything under ~0.004 as noise.

BLEU/F1 move the other way in Phase 2 (0.370/0.440 vs 0.464/0.547). Token-overlap metrics reward verbatim storage, so paraphrasing into clean facts costs surface overlap by design — flagging it here rather than leaving it in the README.

All results public (MIT): https://github.com/corbym/locomo-recordari — including the raw run data, both judges' per-question verdicts, and the score files, so you can re-score without paying for a run.

The harness runs against the prod API - exactly the MCP search and recall api that a real agent would use. Not a mocked backend, the real API, with a configuration of top_k=30, and a 1 hop edge expansion.

Recordari works with Claude, Claude Code, ChatGPT and pretty much any agent that can connect to MCP and reason.

If you want to run an agent against it to run the LoCoMo benchmark yourself, you can anon login at:

https://admin.recordar.io/start

Grab your personal key to use in the harness.

If you don't want to run the harness, just have a play with the sandbox memories, one click from the dashboard sets it up. Just remember to add the Full Skill from the Connect page, and then connect your agent.

--

What is recordari?

Recordari is a multi tenanted memory graph accessible by MCP. The graph can be used by teams, memories stay in the graph when disagreements happen, and resolve using type edges rather than being removed.

References and Further reading:

• Recordari documentation: https://docs.recordar.io/
• Recordari commercial site: https://recordar.io/
• Recordari Admin (register/sign in page): https://admin.recordar.io/
• Recordari Anon login (team tier): https://admin.recordar.io/start
• Microsoft blog: https://www.microsoft.com/en-us/research/blog/memora-a-harmonic-memory-representation-balancing-abstraction-and-specificity/
• Memora github: https://github.com/microsoft/Memora
• Memora paper: https://arxiv.org/abs/2602.03315

Things to ask the agent when running the demo sandbox:

• Why was VTIR created?
• Describe what was next in the project and why?
• Why must Pascal fixtures be written before the Rust port?
• What is TurboSound and why does it need special handling?
• What is the current state of the project?
• Why does WASM file I/O work differently from native?
• What are the standing rules for the AY chip port?

Song to download and try on VTIR

https://corbym.github.io/vtir/  (live web demo)

• https://zxart.ee/tune/325740  (download pt3 original file)

I decided to pull all my collected lessons learned, research, project documentation regarding Agent Memory into a singular open source repository.

A field guide to governed memory for autonomous and agentic systems.

Agent Memory is about more than retrieving old context. It defines what becomes memory, what remains uncertain, what may influence future behavior, who may change durable state, and how retained state can be corrected or forgotten.

I eagerly welcome Discussions, Contributions or Stars openly.

https://github.com/MythologIQ-Labs-LLC/agent-memory

If you're new to Agent Memory, the wiki is built to make the knowledge accessible and easy to understand.

I had Claude Opus 5 analyze several repos for what does the code do, how does the memory work in a particular repo. Initially I was aware of only a few memory systems, that I wanted to study for my own memory system. It turned out, there are lots of agent harnesses that have memory, libraries with memory.

https://neoneye.github.io/agent-memory-atlas/

I have not analyzed closed source repos, since I don't have access.

Claude being the judge wether a repo is relevant or not.

Looking at the impl instead of the marketing material.

Hello people, I've been working on this short-term memory.

Do you know when you are hearing a story and as you listen to it you start to create a "Scene" in your mind? Well, unless you have aphantasia that's what would probably happen:

- You hear some facts: the first image is formed in your mind. Let's say "I was in Lisbon and had 10 dollars in his pocket"

- Next, the story goes: "Then, I traveled to Porto and sold a hat for 50 bucks"

- Then, as the story unfolds, the state of the elements of the story will get new states.

For most of us, it's not hard to keep a clear image of the current "scene" of the story. But if we wanted, we could also take a single element, and trace back how it got there.

Nowadays, to my knowledge, the closest we get to that is the LLM's context. But as it grows, it gets hard and expensive to track down when facts happened during the evolution of the context. Even using CoT, in the end, LLMs are probabilistic machines and so, when it comes to precision recall, noise can be added to the output. Then you plug in some sort of external memory, *DBs, MD files, etc. These are great solutions for the "Big memory", but not necessarily great to fix context growth, or to understand the order of events.

So decided to play around and try to find a naive solution that would allow traceability and increase precision, while reducing the context of the conversation. Right now, I've been experimenting with Ontology triples following the RDF Standard.

The idea

Let's say that you tell it you moved from Lisbon to Porto. Ask "where do I live?" and you either get Lisbon, or you get Porto and the fact that you ever lived in Lisbon is gone. Overwriting loses the history; appending loses the present.

So I built a different shape and measured it properly. Everything runs local through Ollama.

Every asserted fact becomes a (subject, relation, object) triple filed under a canonical key:

user | location   
t1  lisbon    superseded   
t3  porto     current

Paraphrases land in the same slot without embeddings: "where I live", "my city", "my residence" canonicalize to the same key. A slot keeps every value it ever had, in logical time order. Newest is current, the rest are superseded. Nothing is deleted; facts get invalidated, not forgotten. So "where do I live?" reads the current value and "where did I live before?" reads the history, out of the same structure, with no separate archive.

Writing is immutable: each turn produces a new scene, so a failure mid-turn never leaves memory half-written.

The result that made me keep going

Same model (gemma4:12b), two different inputs:

• reading the raw sessions, ~9,000 words: baseline
• reading the compressed scene, ~550 words: +0.102 accuracy

16x less input, and it does better. Compression isn't the price you pay here. The noise the scene strips out duplicate facts, stale values, updates scattered across sessions, is exactly what was confusing the reader. On the clean scene, a small local model matched a much stronger reader working on raw text.

The benchmark, with the caveats attached

477/500 (95.4%) on LongMemEval-S under the official judging protocol, above Mastra's published per-indicator numbers on all six indicators. One reproducible pass over all 500 questions, checked against a canonical state file whose guard refuses to write if anything drifts.

The part I care about more than the score: every mechanism went in with a prediction committed to git before measuring, and an explicit bar for what would falsify it. Ten arms failed and are published as prominently as the ones that worked, plus one retraction. If you read one thing in the repo, read finding 23 in the findings log the same finding got written three times in one day, because the first two drafts concluded from small n and the third had to retract both.

Caveats that matter to this sub specifically. The router triggers and absence gates are regexes calibrated on LongMemEval's English corpus; in another language they don't fire without recalibration. Two indicators sit at their measured oracle ceiling, so further progress there needs a stronger reader model, not better retrieval. And the comparison against Mastra is against their published numbers, not a head-to-head rerun on my hardware.

What's still broken

Seven open findings, all written up with repros:

Reported speech becomes a plain fact. "They said there was gold" gets stored as there being gold. The negation survives only as a string inside the value.

A question in quotes gets read as an assertion. Typing "Does Lucas use Go?" ingested it as a claim, and it overwrote the correct value in that slot.

Partial names spawn parallel entities. "Lucas" and "Lucas Almeida" become different subjects, so half of what the scene knows about him is disconnected from the other half.

Three ways in, if you want to poke at it

Break the demo. There's a live chat with the scene inspector beside it. You watch slots being born, updated and superseded turn by turn, and each answer prints the facts that produced it. Talk to it for twenty minutes with facts that change. 24 findings so far came out of exactly this, 15 already fixed upstream. Highest-yield thing anyone can do here.

Take an open finding. #18 is the tractable one: deterministic repro, both code paths fail, and the fix is obvious. #23 already has its acceptance gate written down in advance, so you'd know immediately whether your fix worked.

Replicate where the numbers don't claim to hold. Another language, another model, another domain. The limits section is a list of things nobody has measured. A clean negative result gets published as one.

MIT, and CONTRIBUTING.md has the open findings in a table with what the work looks like for each.

Repo: https://github.com/natanloterio/scene-memory

Happy to take questions, including hostile ones about the benchmark. Those are the useful kind.