r/coding

I've been building a small memory layer you bolt onto any frozen model — GPT-2, Llama, Qwen, from 124M to 7B. It's not a new model and it doesn't beat anyone's model. It adds one thing the base model structurally can't do — remember and revise facts beyond its context window — and it does it at a cost that changes the math.

What you can reproduce locally (clone, run — no cloud, no API): graft onto Qwen2.5-7B (4-bit) → the 7B fits in ~4.7 GB, the memory adds megabytes; facts beyond the window go from 0/10 (model alone) → 10/10. Store 30,000 real Wikipedia facts → 100% recalled, 0 forgotten, 14.7 MB.

The cost metric — why this matters The expensive, unsustainable part of AI at scale usually isn't the model running once — it's holding and recalling memory, and how that scales. Three hard, measured facts:

1. O(N²) → flat. This is the big one. A transformer fed a growing stream is O(N²) in attention and O(N) in KV memory — it OOMs a 24 GB machine at ~79,000 tokens (measured). The same model on a fixed window + this memory ran a 20-billion-token stream at 0 bytes of memory growth — the process RSS held one constant value the entire run (measured, byte-for-byte). That's the difference between "can't stay on" and "runs forever at flat cost."

2. Memory-delivery cost is structurally ~$0 — vs a per-query token tax or a per-user database. For per-user memory at ~1M users (modeled, red-teamed):

Long-context (re-send each user's facts every query): token bill scales as facts × queries → on the order of $3M/month, and it OOMs past the window. RAG (a hot vector index): ~$45k–165k/month at ~1B vectors — a permanent infra line growing with users × facts, and non-deterministic. This memory: no extra context tokens, no vector DB, O(1)/token → ~$0 marginal, or literally $0 server cost on-device. I stress-tested this: the ordering (this ≪ RAG ≪ long-context) survives a 10× swing in any price, because this memory's per-query token cost is structurally exactly zero. (Absolute dollars are estimates; the ordering is not.)

3. Known queries cost 0 model compute. The confidence gate routes: a fact it knows is answered in ~0.1 µs with 0 model forwards (counted). So you pay for model inference only on the fraction that actually needs the model.

Why it's important: per-user personalization is the line item making AI economics brutal today — RAG bills you a database per user forever, long-context bills you re-sent tokens per query forever. This takes the memory layer to near-zero, deterministic and flat, so it can live on edge/consumer devices instead of a GPU data center. For a memory-heavy or always-on product, that's the cost that's been eating the margins.

The honest boundary (so this stays true): this is the cost of the memory/context layer, not the base model's compute. When the model actually runs — to generate or reason — you still pay for the model. What collapses to ~0 is holding, revising, and recalling facts, plus the always-on footprint. For personalization/streaming workloads, that's exactly the part that doesn't scale today.

Also true (measured) Deterministic & auditable: integer-only memory, byte-exact. Two runs → identical memory; you can track every write to the byte over the 20B-token run. The base model is still a black box, but an auditable memory beside it is a real benefit for regulated use. Zero forgetting: each fact in its own row → a new fact can't disturb an old one (the opposite of a fine-tune). What it's NOT (so nobody wastes time) Not a replacement for the big models, not "beating" the big labs — it's an add-on that needs a base model. It's a fact memory, not reasoning. Only tested on real public Wikipedia. 7B tested at 4-bit. Architecture demos, not product benchmarks.

Prior art it lives near: kNN-LM, Memorizing Transformers, RAG, ROME/MEMIT. What I couldn't find in one system is the combination — deterministic + clean-revision + zero-forget + online-on-CPU. Know one that does all four? Send the link.

Repo (AGPL-3.0, one command, fixed seeds): https://github.com/devkancheti4-design/living-fused — clone it, break it,

CTOs, engineering managers, and staff engineers are rushing to deploy autonomous AI agents across their businesses – either through their own volition or because of the clamor of demand from rank-and-file workers. However, they should think twice, a new study shows.

Enterprise large language model (LLM) agents are likely leaking company secrets, and throwing more compute at the problem is only making it worse, the study finds.

In part, that’s because of the AI’s ability to retrieve and synthesize vast amounts of internal data, from Slack messages to board transcripts, to automate tasks. By gathering that information, they also create issues with contextual integrity.

When retrieving dense corporate data, these agents routinely fail to disentangle essential task data from sensitive, contextually inappropriate information. Higher task completion rates often directly correlate with increased privacy violations.

Read the full story: https://leaddev.com/ai/frontier-ai-models-haemorrhage-sensitive-data

Token usage is the lines-of-code metric of the AI era. The industry knows it. It just hasn't agreed on what comes next....

Meta built a leaderboard ranking engineers by how many AI tokens they consumed. It has since been taken down, but the impulse behind it hasn't gone away.

Across engineering organizations, there is enormous pressure to prove that the millions being spent on AI tooling are paying off. When that pressure mounts, leaders reach for the easiest number available.

Token usage is objective, automated, and scalable. It's also easy to game and almost entirely disconnected from whether AI is actually making engineers more productive.

As one engineering leader put it: "I wouldn't be surprised if we see the opposite trend next year, aiming for efficient usage of tokens as opposed to celebrating burning them at expensive rates."

So what should organizations be measuring instead?

Full article available here: https://leaddev.com/ai/tokenmaxxing-and-the-search-for-ai-metrics-that-matter

I have been writing software for ten years and I still stalled on interview problems. Not on the code. On the first ten minutes, working out what the problem even was.

Every course I bought put the pattern in the heading. "Sliding window: maximum sum subarray." You read it already knowing the answer.

Then the interview hands you the same problem with the heading gone and you sit there listing everything it might be.

So I built the exercise that was missing, for me.

Problems with the label removed, ninety seconds each, and you answer with the pattern, the signals that gave it away, and the target complexity.

After ten of those it shows you a confusion matrix: which patterns you swap for which, and how often. Mine said sliding window for two pointers, eleven times. I had no idea.

It works on me. That is the only evidence I have, which is why I am here.

Free test first, no account, fifteen problems, about seven minutes: https://algopath.pro/placement

If it lands for you, I am taking beta testers. Sign up before end of the August.

What I want back is honest feedback: which questions felt ambiguous, where you got bored, what you expected to be there and was not.

6 languages run in your browser, nothing executes on my server. English and Russian.

The technical interview is evolving as AI-assisted coding becomes the norm.

Over the past decade, a burgeoning industry formed around the promise of helping software developers pass technical interviews and nail exhaustive multi-round interviews at desirable, but elusive, tech firms.

Now with AI reshaping the entire software development industry, the traditional technical interview – heavy on LeetCode style tests and algorithmic questions which test developers’ coding skills and practical knowledge – is becoming redundant. However, the coaching firms who built their reputation helping developers pass these tests aren’t feeling the heat.

https://leaddev.com/hiring/think-the-technical-interview-is-dead-think-again