Looking for contributors for Short-Term memory project
Hello people, I've been working on this short-term memory.
Do you know when you are hearing a story and as you listen to it you start to create a "Scene" in your mind? Well, unless you have aphantasia that's what would probably happen:
- You hear some facts: the first image is formed in your mind. Let's say "I was in Lisbon and had 10 dollars in his pocket"
- Next, the story goes: "Then, I traveled to Porto and sold a hat for 50 bucks"
- Then, as the story unfolds, the state of the elements of the story will get new states.
For most of us, it's not hard to keep a clear image of the current "scene" of the story. But if we wanted, we could also take a single element, and trace back how it got there.
Nowadays, to my knowledge, the closest we get to that is the LLM's context. But as it grows, it gets hard and expensive to track down when facts happened during the evolution of the context. Even using CoT, in the end, LLMs are probabilistic machines and so, when it comes to precision recall, noise can be added to the output. Then you plug in some sort of external memory, *DBs, MD files, etc. These are great solutions for the "Big memory", but not necessarily great to fix context growth, or to understand the order of events.
So decided to play around and try to find a naive solution that would allow traceability and increase precision, while reducing the context of the conversation. Right now, I've been experimenting with Ontology triples following the RDF Standard.
The idea
Let's say that you tell it you moved from Lisbon to Porto. Ask "where do I live?" and you either get Lisbon, or you get Porto and the fact that you ever lived in Lisbon is gone. Overwriting loses the history; appending loses the present.
So I built a different shape and measured it properly. Everything runs local through Ollama.
Every asserted fact becomes a (subject, relation, object) triple filed under a canonical key:
user | location
t1 lisbon superseded
t3 porto current
Paraphrases land in the same slot without embeddings: "where I live", "my city", "my residence" canonicalize to the same key. A slot keeps every value it ever had, in logical time order. Newest is current, the rest are superseded. Nothing is deleted; facts get invalidated, not forgotten. So "where do I live?" reads the current value and "where did I live before?" reads the history, out of the same structure, with no separate archive.
Writing is immutable: each turn produces a new scene, so a failure mid-turn never leaves memory half-written.
The result that made me keep going
Same model (gemma4:12b), two different inputs:
- reading the raw sessions, ~9,000 words: baseline
- reading the compressed scene, ~550 words: +0.102 accuracy
16x less input, and it does better. Compression isn't the price you pay here. The noise the scene strips out duplicate facts, stale values, updates scattered across sessions, is exactly what was confusing the reader. On the clean scene, a small local model matched a much stronger reader working on raw text.
The benchmark, with the caveats attached
477/500 (95.4%) on LongMemEval-S under the official judging protocol, above Mastra's published per-indicator numbers on all six indicators. One reproducible pass over all 500 questions, checked against a canonical state file whose guard refuses to write if anything drifts.
The part I care about more than the score: every mechanism went in with a prediction committed to git before measuring, and an explicit bar for what would falsify it. Ten arms failed and are published as prominently as the ones that worked, plus one retraction. If you read one thing in the repo, read finding 23 in the findings log the same finding got written three times in one day, because the first two drafts concluded from small n and the third had to retract both.
Caveats that matter to this sub specifically. The router triggers and absence gates are regexes calibrated on LongMemEval's English corpus; in another language they don't fire without recalibration. Two indicators sit at their measured oracle ceiling, so further progress there needs a stronger reader model, not better retrieval. And the comparison against Mastra is against their published numbers, not a head-to-head rerun on my hardware.
What's still broken
Seven open findings, all written up with repros:
Reported speech becomes a plain fact. "They said there was gold" gets stored as there being gold. The negation survives only as a string inside the value.
A question in quotes gets read as an assertion. Typing "Does Lucas use Go?" ingested it as a claim, and it overwrote the correct value in that slot.
Partial names spawn parallel entities. "Lucas" and "Lucas Almeida" become different subjects, so half of what the scene knows about him is disconnected from the other half.
Three ways in, if you want to poke at it
Break the demo. There's a live chat with the scene inspector beside it. You watch slots being born, updated and superseded turn by turn, and each answer prints the facts that produced it. Talk to it for twenty minutes with facts that change. 24 findings so far came out of exactly this, 15 already fixed upstream. Highest-yield thing anyone can do here.
Take an open finding. #18 is the tractable one: deterministic repro, both code paths fail, and the fix is obvious. #23 already has its acceptance gate written down in advance, so you'd know immediately whether your fix worked.
Replicate where the numbers don't claim to hold. Another language, another model, another domain. The limits section is a list of things nobody has measured. A clean negative result gets published as one.
MIT, and CONTRIBUTING.md has the open findings in a table with what the work looks like for each.
Repo: https://github.com/natanloterio/scene-memory
Happy to take questions, including hostile ones about the benchmark. Those are the useful kind.
