I thought this might be of interest to share with the forum before anyone goes down a rabbit hole again.

Had a random thought this morning about context rot in agentic coding.

What if an agent could detect when its context is getting dangerously full, take the older \~50% of the conversation, compress the meaning into a compact machine optimized representation, and save it as a md checkpoint?

Something like:
conversation > semantic compression > [checkpoint.md](http://checkpoint.md) \> reinject into context

The raw history could still be archived separately in case the agent needs to retrieve something later.

Another part is making the agent aware of its own context state, so it decides when to checkpoint instead of blindly summarizing at a fixed token limit.

Im calling the compressed representation “gibberlink” for lack of a better name lol, although I know actual gibberlink/ggwave is an audio protocol, not a semantic language.

This probably isn’t a new idea in principle, but I havent seen this exact combination applied to coding agents.

So, thoughts?

Cursor, Claude Code, Codex, Copilot, etc. can handle a lot now, but I'm curious what developers still find unnecessarily manual.

Could be anything—context, debugging, reviewing generated code, managing prompts/rules, testing, documentation, switching between tools, whatever.

What's something you find yourself doing repeatedly that you wish the AI/tool handled better?

tl;dr Don't inject custom context basis user query/RAG/etc. into prompt, make agent search it with a tool with params (query, filter, search_type, temporal, limit). It was the single biggest lever to bring control on using the agent..

I build AI agents at my company, and initially, our context layer was obsidian stlye skills markdown files folders, cross-links. We would initially do vector search / RAG and inject the context alongside prompts. We used to see repetitive challenges there and we went down rabbit hole trying to fix it.. What kept breaking:

• Context poisoning / digression. Once we were past ~50 markdown files, the agent would wander between docs and pick up instructions that had nothing to do with the task. We tried building explicit navigation paths and interlinking everything, but it didn't help much.
• No source proof. As the knowledge base grew, we couldn't reliably say which piece of context drove a given action. Users won't trust an agent that can't show its work.

What actually worked for us:

1. Structured docs instead of markdown. We moved context into JSON / structured documents. Agents navigate way better when things look like code. We had about 10-12 document types and then each type had 5-8 fields within them
2. Make the agent search, don't spoon-feed it. Instead of pre-injecting context, we gave it meta-info about what context exists and made it responsible for searching and discovering the right pieces (tool-based search capability for the agent rather than us running RAG/prompt expansion upstream).

For search, we created a tool search_resources that would run queries on the opensearch index in which the structured docs were stored - the tool we created had 5 parameters:

* query - Select the query it wants to run

* filter - Filter by specific type of documents

* search_type - Define search type (semantic / syntactic)

* temporal - Add temporal True/False if your data has time based staleness

* limit - number of responses it receives in return

If you're doing something similar, what's your experience been?

What else is working great for you?

EDIT: TL;DR up top, fair complaint about the length. 2 minute read.

Loaded 3 years of chat history (3,629 conversation chunks, 23M chars, 36k extracted memories) into self hosted Hindsight. Ran it as a shadow import for 10 hours while my old memory provider kept serving, then benchmarked both on the same 45 case eval set and switched.

Hindsight vs the provider I replaced

Same 45 questions, same eval harness, both arms measured identically:

old provider	Hindsight
recall	0.514
context tokens per query	1,724
questions fully answered	23/40
stale facts leaked	0
p95 latency	9ms

Per category, where it actually moved:

category	old	Hindsight
episodic recall	0.80	1.00
recency	0.60	1.00
cross session	0.50	0.80
temporal	0.21	0.26
procedure	0.40	0.40
supersession	0.53	0.53
profile lookup	1.00	0.83

Short version: it is much better at "what happened in that conversation months ago" and level or slightly worse at everything else. Cross session recall was the whole reason I did this, and it went from coin flip to 0.8.

The cost result, which I did not expect

Memory injects ~3,600 extra tokens into every turn, so I assumed the bill goes up. Ran 8 complex questions through the real agent, once while memory was silently broken and once after fixing it:

memory broken	memory working
wall time	739s
input tokens	1,513k
tool calls	36

It adds 3,600 tokens per turn and still cuts half a million tokens off the total. When the agent already knows the answer it stops going on a file reading expedition. If you run an agent with tools and think memory is a cost center, measure it.

Key takeaways

Compare at equal token budget, not equal result count. At top_k=8 both providers looked tied. But the old one stores long paragraphs and Hindsight stores short facts, so 8 vs 8 handed one side 4x the context. That single mistake almost made me abandon the project.

Retrieval depth is not linear. k=48 scored worse than k=32 while costing 1,000 more tokens. Saturates completely at k=96, and k=128 buys 399 extra tokens for exactly zero recall.

Good answers do not prove memory works. 8 real sessions, all 8 answers correct. Then I checked what was actually injected: 7 of 8 got zero memory. The agent had just read files with tools instead. Cold start was eating the entire retrieval budget. I nearly shipped a dead memory system with a clean test report.

Four config values turn cutover into a silent no-op. Wrong mode (spawns its own instance and never touches your bank), wrong bank id, recall_types pointing at a type I had 300 of instead of the 36,000 I had, and auto recall off. Everything reports healthy and memory does nothing.

A broken service and an idle one log the same line. My catch-up scan had a flag meaning "only messages with id at or below zero". It ran on schedule, reported success, inserted nothing, forever. Identical to "no new conversations". Found it by removing the flag and watching 2 episodes appear instantly.

Score thresholds cannot make it say "I don't know." Made up questions score 1.039, real ones 1.077. No separation, so no threshold works. It abstains fine at the model layer anyway, just not at retrieval.

Consolidation is real but expensive. 100 memories per 50 minutes, so two weeks for my bank, and while running it pushed live query latency from 1.1s to 10.7s. Turned it off and kept the 4 stale facts.

Current state: 36,000 memories, live as main provider, ~1.2s recall, 8/8 on my hard question set, token bill down.

Full detail and the rest of the bugs below.

---------------------------------------------------------------------------------------------------

That was the part I did not expect, so I am putting it first.

Adding a memory layer means you inject extra context into every single message. Mine adds about 3,600 tokens per turn. So I assumed the bill goes up and I would be trading money for quality.

I measured it with the same 8 complex questions, once with memory actually reaching the model and once without:

                    memory broken    memory working
sessions with memory      1/8             8/8
total wall time          739s            416s     44% faster
total input tokens     1,513k          1,011k     33% fewer
tool calls                 36              26     28% fewer

It adds 3,600 tokens per turn and still saves half a million tokens overall. Reason is dumb in hindsight: when the agent already knows the answer it stops going on a file reading expedition. Ten fewer tool round trips paid for the memory many times over.

If you are running an agent with tools and you think memory is a cost center, measure it. Mine is a savings.

What this was

Three years of conversation history spread across two different agent stacks, two SQLite databases, hundreds of session JSON files, agent JSONL logs, and a pile of markdown notes. Nothing was searchable in any useful way. Ask the agent something from four months ago and it had no idea.

Goal was to get all of it into Hindsight (open source agent memory, MIT, self hosted with Docker) and then decide, with actual numbers, whether it beats the simple memory provider I was already using.

Total: 3,629 episodes, 23.2 million characters submitted, 35,800 extracted memories. Took about 10 hours of processing.

I ran the whole thing as a shadow import first. Real provider untouched, agent kept serving normally, nothing in the live prompt path. Only flipped the switch after the numbers came in.

The finding that changed the whole evaluation

First A/B I ran said Hindsight was barely better than what I already had. 0.514 recall versus 0.528. Basically noise. I almost stopped there.

Then I noticed the token column. Same top_k of 8 for both arms:

old provider:  0.514 recall, 1,724 context tokens
Hindsight:     0.528 recall,   462 context tokens

The old provider stores long paragraphs. Hindsight stores short precise facts. Eight of each is not a fair fight. I was giving one side four times the context budget and then concluding it knew more.

So I swept retrieval depth:

k=8    0.500 recall,   448 tokens
k=16   0.500 recall,   883 tokens
k=24   0.583 recall, 1,308 tokens   <- beats baseline on BOTH axes
k=32   0.597 recall, 1,590 tokens
k=48   0.583 recall, 2,613 tokens   <- goes DOWN
k=64   0.639 recall, 3,562 tokens
k=96   0.694 recall, 5,089 tokens
k=128  0.694 recall, 5,488 tokens   <- zero gain, pure cost

Three things fall out of that table.

At k=24 it wins on both axes at once. More recall for fewer tokens than the thing I was replacing.

The curve is not monotonic. k=48 scores worse than k=32 while costing a thousand more tokens. More context does not reliably mean better ranking.

It saturates hard at k=96. Going to 128 buys 399 extra tokens and exactly zero recall. The ceiling is what is in the bank, not how deep you dig.

Final numbers at full data, k=24:

                    old provider    Hindsight
recall                    0.514        0.597    +16%
context tokens            1,724        1,450    -16%
fully covered cases       23/40        27/40
stale facts returned          0            4
p95 latency                 9ms      2,099ms

Per category, biggest wins were episodic recall (0.8 to 1.0), recency (0.6 to 1.0) and cross session (0.5 to 0.8). Procedure and supersession came out level. Profile lookups got slightly worse.

Worth saying: my metric checks whether the correct fact is present in the returned context. It does not check whether the model then used it. At k=96 the right answer sits inside 5,000 tokens where attention dilution is real and my metric is blind to it. That asymmetry is why I picked the smaller operating point.

The 3 second question, and why the obvious fix was wrong

Old provider answers in 9ms. Hindsight takes about 1.2 seconds. That is a 130x regression on paper.

My first instinct was to tune it. Threads, budget, batch size, the usual. Then I read the integration code and found the actual problem was not speed at all.

The plugin warms a recall at the end of each turn and serves it on the next one. But it was ignoring the query argument entirely. So the memory injected into your current question was retrieved using your previous question. Ask about your project list, get memories about whatever you said before that.

Latency was never the bug. Relevance was.

Fixed it by tracking which question a warm result belongs to. If it matches, serve instantly at zero cost. If it does not, recall for the current question inside a bounded budget and fall back to the warm result if the budget expires.

Then I broke it in a new way, which was educational.

I set the budget to 2.5 seconds based on my own measurement of 1.1 second recalls. Ran the scenario suite. 8 out of 10 returned nothing, and every single one took exactly 2,501 ms.

Turns out the client funnels everything through one event loop. My timed out threads were abandoned but still holding that loop, so every following call queued behind a corpse and hit the ceiling too. A budget that is too tight does not make things fast, it creates a pileup.

Raised it past the real p95 and it went from 2/10 to 8/10 with p50 at 1.2 seconds. Same code, one number.

The bug that would have quietly ruined everything

At the very end I ran 8 real sessions through the actual agent, fresh session each time, and asked hard questions. All 8 answers were correct and detailed. Looked like a clean win.

Then I checked how much memory was actually injected into each one.

session 1: 12,765 chars
session 2:  1,192 chars
session 3:  1,192 chars
...
session 8:  1,192 chars

That 1,192 is a fixed header. Seven out of eight sessions got zero memory. The answers were good because the agent went and read files with tools instead. It worked for it.

If I had judged by answer quality alone I would have shipped a memory system that was not being used and never known.

Cause was cold start. First recall in a fresh process spent the whole budget building the HTTP client and returned nothing. I added a warmup at session init, which then raced against the real query on that same single event loop and made it worse. Fixed it by making the query wait for the warmup instead of competing with it.

After that, 8/8 sessions with 12,000 to 14,000 characters of memory each, and the numbers at the top of this post.

Lesson I keep relearning: a good output is not proof the thing you built is what produced it.

Things that did not work, so you do not have to try them

Score thresholds cannot make it say "I don't know." The API takes a min_scores parameter. I swept it from 0.2 to 0.65 and got byte identical results every time. Looked at the raw scores:

made up questions:  final score max 1.039 to 1.067
real questions:     final score max 1.077 to 1.100

There is no separation. The final score saturates near 1.0 for everything, semantic overlaps, keyword has a bit of signal but still overlaps. Multilingual embeddings put every well formed sentence in roughly the same neighborhood. Abstention is not solvable at the retrieval layer, full stop.

The funny part: end to end it works fine anyway. I asked it what coffee I drank in a city in 1987 and it said it had no record of that, and added that I would not have been born yet based on my age in memory. The model handles it even though retrieval hands it 40 irrelevant facts. I was pessimistic about the wrong layer.

Consolidation is real but expensive. It merges duplicate and contradictory memories into synthesized observations, and it is the only mechanism that fixes stale facts. I measured it at 100 memories per 50 minutes. For 36,000 memories that is roughly two weeks of background processing. Worse, while it runs it competes with live recall for the LLM and the DB, and pushed my query latency from 1.1 to 10.7 seconds. Turned it off. Left the four stale facts. Not worth it right now.

prefer_observations and type filters did nothing measurable.

Integration gotchas that cost me real time

Four config values were wrong in a way that would have made the cutover a silent no-op:

• Plugin was in embedded mode, which spawns its own separate instance and never touches your bank
• bank_id pointed at a different bank entirely
• recall_types was set to observation, and observations only exist after consolidation. I had 36,000 world and experience memories and roughly 300 observations. It would have returned almost nothing
• auto_recall was false, so nothing gets injected at all

All four look harmless in a config file. Together they mean you flip the switch, everything reports healthy, and memory silently does nothing.

Other things worth knowing:

operation_id must be a UUID. I was using a truncated sha256 for deterministic idempotency and got a 422. Switched to UUID5 over a fixed namespace, which keeps determinism and satisfies the validator.

Watch container memory. Mine was sitting at 980MB against a 1GB limit before any load, and the cgroup had already hit its ceiling 1,513 times. There is a closed upstream issue about API memory growth on older versions. I raised the limit and cut the DB pool. Anonymous RSS turned out to be a stable 855MB baseline, not a leak, but a multi day import would have OOM looped on the original setting.

Check your worker slot math. Mine had 2 slots with 1 reserved for consolidation, which was disabled. So the import ran at exactly one concurrent extraction and I wondered why it was slow. Freeing that slot and raising the count took throughput from 157 to 630 episodes an hour.

Rate limits are real on the heavy tail. The last third of my queue was the long multi turn conversations, 30 to 40 extracted facts each, 3 to 6 minutes apiece. Provider started returning 429s. Circuit breaker plus durable retry handled it with zero lost work, but my ETA went from 3 hours to 9.

Bugs I found in my own pipeline before they did damage

Writing this part because the pipeline bugs were nastier than the integration ones and every single one was found by measuring, not by reading code.

336 real notes were being silently excluded. My coverage rules classified anything outside a memory/ directory as a workspace working file. That swept up identity documents, an ideas folder, reports and findings. All genuine user authored content. Caught it by auditing the exclusion list instead of trusting the residual count, which was happily reporting zero.

11,138 false redactions from file paths. I built the secret scanner to register configured secret values from env and config files. It also registered anything long and high entropy, which includes filesystem paths. Paths appear constantly in developer conversations. Every occurrence got replaced with a redaction marker. My memory would have gone in full of holes.

Then config identifiers did the same thing. After the path fix it was still firing 14,565 times. The model id, the bank name, ordinary lowercase-with-dashes strings sitting under keys named token or auth. Added a shape test asking whether a real credential could plausibly look like this. False hits went from 111 per 364k characters down to 2.

Split secret detection was masking entire episodes. If a credential appeared in fragments, the code masked from the first fragment to the last. In a long conversation that is the whole thing. Changed it to mask only the fragments.

A circuit breaker that never used its configured cooldown. Off by one on the exponent, so the first trip always waited double. Found by a test that asserted the documented contract.

The catch-up service could never have worked. I passed --hermes-max-message-id 0 to the periodic rescan, meaning "only messages with id at or below zero". No new conversation would ever have been captured. The scan reported success every time it ran, inserting nothing, which looks identical to "nothing new to do". Only caught it because I removed the flag and immediately saw 2 new episodes appear.

That last one is my favorite failure mode. A broken thing and a correctly idle thing produce the same log line.

What I would tell someone starting this

Run it as a shadow first. Mine ran 10 hours against a live agent that never noticed, on a separate bank and volume, with the old provider still serving. Cutover was one config line after the numbers were in.

Compare at equal token budget, not equal result count. This flipped my entire conclusion.

Verify the plumbing separately from the output. Answer quality told me everything was fine while seven of eight sessions were getting no memory at all.

Measure the thing you actually run. My clean 1.1 second recalls came from hitting the API directly. The real integration path had a cold start that cost the first message of every session its entire memory, and I would not have seen it from the outside.

Idempotency is worth building on day one. A full rescan of 3,626 items created zero new identities, so I could rerun the scanner whenever I wanted without thinking about it. Content addressed IDs, deterministic operation IDs, insert or ignore.

Do not trust a zero. Residual count zero, dead letter count zero, inserted zero. Each of those meant something was working right in one place and something was silently broken in another. Zero is a claim, go check what produced it.

Current state: 36,000 memories, live as the main provider, roughly 1.2 second recall, 8 out of 8 on my hard question set, and a token bill that went down. Consolidation off, four known stale facts, abstention working at the model layer despite retrieval offering no help. Rollback is one config value and I have verified snapshots of everything.

Happy to answer questions about any of it.

I’ve been using ChatGPT, Claude, Cursor and a few other AI tools pretty heavily, and one thing started bothering me more and more.

Every time I switch tools, I basically have to start over.

One tool knows something I told it last week. Another has a completely different understanding of what I’m working on. I end up copying old conversations, pasting context, explaining projects again, or keeping notes somewhere just so the AI can catch up.

And the more I use AI, the more ridiculous this feels.

We already have AI that can reason, code, research and work across huge amounts of information, but the context around me is still stuck inside individual products.

That’s actually the problem we started exploring with innernet.

The idea is pretty simple: what if your context lived independently from the AI tool?

So instead of ChatGPT having one version of you, Claude having another, and Cursor having basically none, your context could follow you across all of them.

We’re still figuring out what the right architecture for this looks like, but I keep coming back to the same thought:

AI should remember you, not the app you happen to be talking to.

Curious if anyone else has run into this while switching between AI tools.

Let me be straight with you, because this is a real thing, with no actual bearing. That's why I have not done anything yet. Note... and this is huge... none of this has actually been documented. Per rule 3.a.56 subsection 4.1, I created a shadow gear test to confirm exactly this. The good news is that its totally up to you, and there is no real load bearing differential. If you would like, just say the word, and I will proceed.

Most large language model (LLM) inference framework comparisons begin with a leaderboard. One framework posts the highest tokens per second on a standard benchmark, and that number quietly becomes the reason a team adopts it.

The trouble is that the conditions that produce a clean benchmark result rarely resemble the conditions a model faces in production.

Synthetic benchmarks tend to use fixed prompt lengths, steady request rates, and a single model on familiar hardware. Production traffic does none of that.

This article is written for engineering leaders who are choosing an inference framework and want a way to reason about that choice beyond the headline numbers.

It covers why a benchmark winner can underperform once real traffic arrives, three tradeoff axes that usually decide the outcome, and a practical evaluation process you can run before you commit.

Hey y'all. I work out of an AI lab and wanted to share a new method of reranking that I think has a lot of potential (all open source).

A bit of background: Relational Transformers is a new transformers architecture that is trained on relational data. You load the context with typed database cells, and then it can do prediction or classification tasks. Since the model is tiny (less than 100M parameters), it can be run very quickly over a large result set.

In this case we are trying to predict the ranking of search results. We load the context with a bunch of data like consumer preferences, buy signals, typed product data like floats for price, and condition the network to try to learn its rank. We can then rerun that conditioned network to discover what actually contributes to the reranking performance, so we can keep our context extremely lean.

But one caveat, a large part of the performance gains came from converting the query to possible database cells (to supplement the context), since RT is trained on database cells. I just have a small LLM do a single pass over it to convert it to json and then load that in as context. They don't need to be real database cells, just approximate the names and the model will figure it out. Adding some schema hints edged out some extra performance numbers. But you're not passing the entire candidate result set into an LLM so this part remains cheap, relatively speaking.

End-to-end, this gets near LLM level performance at the cost of a mid-sized reranking model.

https://relativedb.com/research/relational-reranking

If you experiment with this, let me know!

The first time I heard the time offloading context was around September 2025, by a langchain talk that Lance Martin gave, it was one of those concepts that it made immediately a lot of sense.

The problem with skills

What are skills? Skills are basically long prompts that someone created and that are very good at transmitting the information to the AI to do a particular task.

It’s possible that you have had the change of creating you skills, maybe for creating components in your project, or to deploy a specific thing, or to create a changelog for your app…

But skills are long, way too long, does it need to be like that?

I see most skills like 2000+ text code lines, a bunch of information grouped together, maybe once I need the first part, maybe then I need the other, the question that comes to me constantly when working with them is … where did we left building things with a single purpose.

Going further than skills

Imagine a skill that it’s broken down into many small pieces, and for some task a small piece might fit and for another tasks another pieces is better suited. This is what I’ve been doing for the past months, building a kind of Tree of Skills or Context Tree, where just the right information is loaded in the AI for the task at hand.

This is how I do this

/modules
  /index.md // Basic descriptions and references to other files/folders in same level
  file1.md // File with info
  folder1 // Folder with other files
    index.md // Basic description of folder 1 and refernece to files and folders
    file2.md // File with info

This can grow exponentially, in some cases creating playbooks we’ve created at MAAT up until 60+ files in the same folder, and the AI is able to find the right one thanks to the reference. For the index.md files we use something similar to the llms.txt

Where we point the file name and a brief description of it, this may look like this

file1.md: This file is to do X and Y
folder: This folder contains files that can do Z

This way the AI doesn’t have to read all the file before getting to the interesting part, thus keeping the context thin.

A local MCP to context offload

At the beginning I was doing this context offloading manually, but it was tedious to always have the skill that indicated what is the llms.txt and I found it that I had to repeat it for every single project where I want it, this is what it lead me to finally create gcontext.ai, I am not so sure how to call it, context management system, framework to create context agents or a simple context offloader.

You can read more about it here: https://github.com/bleak-ai/gcontext

Would love to hear any feedback that you might have on this

So there are more and more "context layer" products in the market now, have you had experience with any of them? What do you think are the meaningful differences between a proper context layer and simply relying on MD files/git repo?

Keen to hear what people in this sub think!