r/LLM

▲ 26 r/LLM+9 crossposts

TokenMizer - a local proxy for session checkpoint/resume and graph memory across Claude, GPT, and Ollama

I've been building TokenMizer, a local proxy that sits between your editor/CLI and whatever model you're using (Claude, GPT, Ollama) and handles two things I kept re-solving by hand: session checkpoint/resume, and a graph-based memory instead of a flat transcript.

The problem: once a long agent session hits the context limit, the usual fix is summarization, and summaries lose the reasoning behind a decision, not just the decision itself. I'd see a summary saying "switched to Argon2" with no trace of why bcrypt was rejected, so the agent would re-litigate the same tradeoff two sessions later. Flat transcripts have the opposite problem: everything is kept, but nothing is prioritized, so retrieval is just recency-biased keyword luck.

What TokenMizer does differently: instead of one growing text blob, decisions, constraints, and open questions are stored as nodes with edges (this decision depends on that constraint, this question was resolved by that decision). Checkpointing snapshots that graph plus a resumable session state, so you can kill a session and pick it back up without replaying the whole history through the model again.

Where it's rough: there's no eval harness yet comparing retrieval quality against a naive flat-transcript baseline, so right now my evidence is anecdotal (my own sessions), not benchmarked. I also learned the hard way that benchmarking your own memory system by asking it questions only it can answer is circular, so I'm holding off on publishing numbers until I have an honest comparison.

Repo: github.com/Shweta-Mishra-ai/tokenmizer (I'm the author). It's a Python project, MIT licensed. If you've hit the same summarization-loses-reasoning problem, I'd be interested in how you're handling it, and PRs/issues on the eval-harness gap would genuinely help.

u/Feisty-Cranberry2902 — 20 hours ago
▲ 20 r/LLM

Ran chinese models against my claude/gpt setup for a few weeks and the spend gap is wild

Api bills started getting stupid the past few months, like genuinely looking at my monthly spend and wondering if i am doing something wrong. Decided to run my own tests on chinese models instead of trusting whatever chart someone posts on twitter that week.

Deepseek, qwen, kimi went up against my normal claude/gemini/gpt rotation. Glm-5.3 got added this week when i finally got around to the new release so its early days for that one.

The spend gap is wild. Quality gap exists of course but its not anywhere near what the pricing makes it look, especially on iterative stuff where i am running the same task 5 times to get it right.

Closed models still win on hard reasoning most of the time. Once a prompt gets complicated with a bunch of conditions stacked deepseek and the older chinese ones start fumbling somewhere. Glm-5.3 actually held up better than i expected, felt closer to opus on a few of my tests but i will need more time before i say anything strong.

Claude and gpt still get my real work. Iteration heavy stuff just makes more sense on the lighter side because i am not burning premium tokens on a model to write the same function 4 different ways.

Would rather read other peoples actual usage notes than argue about charts at this point.

reddit.com
u/keeppoise — 22 hours ago
▲ 7 r/LLM

Why Qualcomm Open-Sourced Mojo After Its $3.9B Acquisition

I do not write compilers for a living. I learned about Mojo yesterday. My first assumption about the Qualcomm–Modular deal was wrong, and I spent the better part of a day trying to understand why.

On August 18, Qualcomm did three things at once. It released Mojo 1.0 under an open-source license, Apache 2.0 with LLVM exceptions. It integrated its own data-center accelerators into the Modular platform, the same platform it had acquired for roughly $3.9 billion only weeks earlier. And it explicitly, publicly, without hedging, preserved first-class support for competing silicon from NVIDIA and AMD.

This does not look like a spontaneous goodwill gesture after a corporate purchase. It looks like an item on a schedule that was drawn up long before the ink dried.

Qualcomm announced its intent to buy Modular on June 24. The deal closed on July 29. The open-source release and hardware integration followed on August 18. Less than three weeks between closing and release. You do not write a compiler from scratch in twenty days, and you do not reverse a corporate strategy in that window either. The decision to open the code was made before the acquisition closed.

The standard semiconductor playbook after buying a software company is defensive. You acquire the toolchain, wall it off, optimize it for your own chips, and charge for the privilege of migration. Intel has done it. Proprietary vendor stacks do it routinely. Qualcomm ignored the playbook entirely. It spent nearly four billion dollars on software and immediately gave the core away, promising to keep the competition's hardware running smoothly.

The question is not what happened. The question is what happens next.

Qualcomm could have taken the obvious path. It could have locked Mojo inside a proprietary SDK, tied it to Snapdragon, and forced developers to pay for access. Want to run on our chips? Learn our API, rewrite your pipeline, eat the migration cost. That is not cynicism; that is the market norm.

Instead, Qualcomm is building a bridge, not a wall.

The pattern that emerged after a few hours of reading is straightforward. Qualcomm makes its money on silicon. On chips that ship in devices and data centers. Software is not the product; software is the condition under which the product sells. If a developer cannot run a model on your chip, he does not buy your chip. If he is welded to CUDA, he buys NVIDIA.

The goal, then, is not to turn Mojo into a profitable language in its own right. The goal is to uncouple hardware choice from software dependency. If Mojo becomes a neutral layer that runs equally well on Qualcomm, NVIDIA, and AMD silicon, developers stop comparing migration headaches and start comparing watts and throughput. They start comparing hardware.

Qualcomm is not buying a programming language. It is buying the rules of the game, and it is trying to change them.

Open-sourcing Mojo removes one genuine fear: the code cannot be pulled behind a licensing wall tomorrow. If Qualcomm changes course, the community can fork. The license is real, the repository is public, the toolchain is downloadable.

But open code does not mean open governance. Who decides which hardware target gets optimized first? Qualcomm and the Modular team. Who merges pull requests? Not the community, not yet. Who sets the roadmap? The same people who own the platform.

The ball is now in the developers' court. Qualcomm made an expensive, noisy, deliberate bow. But a bow is not a dance. If engineers do not show up, if libraries do not get written, if the ecosystem does not form, then Qualcomm bought a very expensive repository with a beautiful license.

And there is a chain reaction behind the developers. Who will write production Mojo code? Where do you hire engineers who know this stack? The talent question is the adoption question. Qualcomm opened the door and made entry free. Whether anyone walks through is still uncertain.

The risk has not disappeared. It has changed shape. From "will they close it tomorrow?" to "will anyone still care tomorrow?"

All of the above is logic. Logic does not run models in production.

NVIDIA is not defended by silicon. Competitors already ship chips that beat it on price per watt. NVIDIA is defended by two decades of CUDA. Millions of lines of optimized kernels, cuDNN, TensorRT, FlashAttention, an entire engineering culture that cannot be rewritten in a single release cycle.

And Mojo is not alone in this field. OpenAI's Triton already abstracts CUDA. PyTorch 2.0 brought its own compiler. JAX and XLA have pursued hardware-agnostic layers for years. The idea is not new. Mojo simply has Qualcomm and four billion dollars behind it now.

Money, however, does not guarantee adoption. The history of AI infrastructure is crowded with technically impeccable projects that never became standards. You need real production workloads, not demos. You need a library ecosystem. You need teams to start hiring for Mojo expertise, and those résumés do not exist yet.

Qualcomm opened the door and removed the entry fee. But someone else has to step inside.

On August 18, Qualcomm did not merely publish code. It made a wager. A wager that the rules of competition in AI can be shifted not by building a new CUDA, but by making CUDA less indispensable.

This does not mean NVIDIA will lose. CUDA will not vanish tomorrow. It will likely dominate for years to come.

But there is now a player spending billions not to build its own wall, but to make walls less relevant. That alone makes the next two or three years of infrastructure far more interesting than the last two or three.

Whether the strategy works remains unknown. But Mojo is worth watching now not because it is open, but because a chipmaker with deep pockets has decided to play the long game in public, and to play against the rules by which everyone else has won.

---

I saw Qualcomm spend $3.9B on Modular and immediately open-source Mojo. That made no sense to me — it is not how chipmakers operate. So I spent two hours digging through press releases and chasing the logic with LLMs until a coherent picture emerged. I am not a programmer, not a semiconductor analyst, not an insider. I just saw a loose thread and pulled it. If it unravels somewhere, tell me.

reddit.com
u/InsiderCrush — 2 days ago
▲ 1 r/LLM

Syntropy - a cloud coding agent with no installs and no PC required

Today I’m releasing the second beta of Syntropy.

The idea is simple: your coding agent should not require you to install a bunch of tools, keep your laptop running, or host the agent on your own machine.

As you can see in the demo, OpenCode runs entirely in its own cloud sandbox. Compilers, runtimes, dependencies, and other tooling are already installed and ready to use.

So there’s no:
“Run the agent on your PC and control it from your phone.”

The agent actually runs in the cloud.
Right now, the beta includes free Zen models, generous usage limits, and no paid subscription.
I’m currently looking for more beta testers, and a mobile version of Syntropy Beta is coming soon as well.

If you’d like to try it, leave a comment and I’ll send you an invite.
Feedback is very welcome - especially criticism.

u/ANDRE_2512 — 2 days ago
▲ 0 r/LLM

lesson learned... llms are not search engines.

I really don't care if anyone likes this post or not, I'm just venting. I love AI, LLMs, agents, etc. in general and thought for the last 8 months that they were really helping me get something done. I'm trying to start a nonprofit. It was really helpful at deciphering the legal jargon I had to use to get incorporated in my state, when I could not afford human help. Free legal counsel does not exist for this purpose and I didn't have extra money, so I was grateful any help existed (even if it came from a robot) and this is about when I stopped using search engines entirely and started using different LLMs (no particular one) as search engines, often checking their answers against each other.

This is where it actually became... terribly frustrating.

Because the next step after that is mostly fundraising (finding grants, nonprofit loans, networking events, etc.) I expected this to be a straightforward process of searching and applying.

But every LLM kept giving me results so awful that it was alarming, sent me into a full panic like there was no funding out there for the nonprofit I had just created. It kept recommending venture capital funds, accelerators (which usually serve for-profit businesses, especially tech businesses), ivy league incubators... for a rural west virginia agricultural nonprofit. I showed up in person to several rooms I did not belong in and made the best of it with good humor, but it was getting frustrating to keep doing this.

After months of circling through answers I knew weren't right on various LLMs... I just typed the same questions into Google and immediately got what I was looking for.

I feel stupid but glad its over. Still a little stunned, not quite understanding why relevant search results are still outside the capabilities of LLMs, but whatever.

reddit.com
u/OfficeElegant1185 — 2 days ago
▲ 4 r/LLM

Why is there so little GLM-5.3 testing to read?

Where did the GLM-5.3 launch discussion go? The model is out, but the usual first wave of benchmark screenshots, coding samples, and arguments over one suspicious result is much smaller than I expected.

I only realized after quite a while that GLM-5.3 had already been released. I played with it briefly through the ZenMux API. So far the capability difference from GLM-5.2 has not felt very obvious to me, but it is very, very slow. I cannot tell whether that is a network problem or something else.

For now I have gone back to GLM-5.2. Maybe the slow response was specific to my request path, but it was enough to make the new model feel less immediately useful. I still want to see more real GLM-5.3 testing before deciding whether it is worth switching back.

reddit.com
u/Few-Tie5420 — 2 days ago
▲ 9 r/LLM+2 crossposts

We Analyzed 10,000 Failed Agent Search Queries: Why Your LLM is Terrible at Web Searching (and How to Fix It)

Hey everyone,

Following up on the previous discussion about search API latency, I wanted to share another massive bottleneck we hit while building autonomous web agents. Even with a lightning-fast retrieval API, our agents were frequently getting stuck in execution loops or extracting the wrong data entirely.

We pulled the logs of 10,000 failed retrieval attempts. The culprit wasn't the LLM's logic or the search index itself. The core problem was that LLMs are surprisingly terrible at formulating search queries.

Here are the three main ways agents fail at searching, and the architecture tweak we used to fix it.

1. The Conversational Search Trap

LLMs naturally default to conversational text. Instead of searching a dense keyword string like OpenAI API pricing 2026, a naive agent will search How much does it cost to use the OpenAI API right now?. Search engines (even machine-native ones) often struggle with these conversational strings, burying the agent in SEO spam instead of technical documentation.

2. Context Amnesia in Multi-Hop Searches

When an agent needs to perform a multi-hop search (e.g., finding a founder's name, then finding their previous startups), it frequently drops the subject in step two.

  • Search 1: "Who founded Anthropic?" (Result: Dario Amodei)
  • Search 2: "What other companies did he work for?" (Result: Completely useless garbage).

3. Ignoring Advanced Operators

By default, agents rarely utilize quotes for exact matches, site: operators to restrict domains, or - exclusions to filter out noise. They rely on basic broad-match strings, which rapidly fills up their context windows with irrelevant tokens.

The Fix: The Query Planner Micro-Agent

We completely stopped letting our primary reasoning agent hit the search tool directly. Instead, we injected a lightweight micro-agent whose sole job is translating the goal into optimized search syntax.

  • Step 1: Primary Agent requests information (e.g., "Find Stripe's latest API rate limits").
  • Step 2: Query Planner translates this into an array of strict queries: site:[stripe.com/docs](https://stripe.com/docs) "rate limits" 2026.
  • Step 3: The system executes the search and returns the clean snippets to the Primary Agent.

This simple architectural tweak increased our agent's successful retrieval rate by over 40% and drastically cut down on wasted API calls.

I posted the full query-planner prompt schema and benchmark logs on the forum if you want to inspect or copy it:Brave vs Google Search API for AI Agents - The 2026 Enterprise Guide

Has anyone else implemented an intermediate query-planning step, or are you successfully prompt-engineering your main agent to handle advanced search syntax on its own?

u/Ok_pettech — 2 days ago
▲ 11 r/LLM+11 crossposts

Cross-Vendor Semantic Void Matrix: Zero-Byte Outputs in GPT/Claude/Gemini/Kimi

A frozen cross-vendor study of 31,430 trials across 11 GPT, Claude, Gemini & Kimi Large Language Models found 11,658 successful executions with exactly zero visible UTF-8 output bytes.

Across 4,290 strict matched semantic pairs, null-condition arms produced 2,505 Voids; matched output-licensed controls produced 0.

These were not refusals, safety blocks, rate limits, or transport failures.

Raw records, event hashes, verification code, and full analysis are public.

doi.org
u/rayanpal_ — 2 days ago
▲ 19 r/LLM+7 crossposts

David Gerard (Pivot to AI): the internet's used up — now the same scrapers are hammering smalll self-hosted servers like mine, non-stop.

David Gerard runs Pivot to AI oon a server that costs him €7 a month.

Right now, something wearing a fake Chrome mask is hammering it — hopping IP addresses so he can't even block it properly, ignoring robots.txt because robots.txt was never a wall, just a sign nobody was required to read.

He's not a company.

He's not a platform.

He's one guy, doing his own sysadmin work, at 11pm, because the industry ran out of the free internet and started eating the cheap end of it instead.

Not stolen. Just... takenn, quietly, at scale.

 

I've watched this exact shape happen before — just slower, and on paper instead of a server log.

Circa 2005, Malaysia. I was Assistant Technical Manager for one of the largest construction main contractors in the country. We were compiling tender documents for a factory job — flat-flooring work, strict F-numbers, the kind of spec that keeps a forklift's raised forks from clipping the racking on a narrow run.

A subcontractor walked in to drop off her quotation. She glanced at our papers, open on the table.

And she went pale. I heard the gasp.

"这是我写的,为什么会在这里?" — This is what I wrote. Why is it here?

Word for word hers. Now sitting under our company's logo and headings.

She looked at me. I looked at her. She was waiting for an answer I didn't have.

Then her eyes flickered — a thousand thoughts passing through in a second — and she said, "没关系。我可以再写过。" — Doesn't matter. I can write it again.

And she left. Good for her.

https://preview.redd.it/o6qawkco45kh1.jpg?width=1024&format=pjpg&auto=webp&s=d9164c398628663f07d2b86343d59947ae045aa0

________

Every one of these stories eventually lands on the same fact: the exposure runs downhill, from the platforms with lawyers down to the servers with none.

 

If you're running anything on a boxx that isn't Amazon or Google's, drop your own scraper-traffic story below. I want to see how far downhill this actually goes.

 

Clip credit: David Gerard — full video on The Tech Report's channel. DM for credit or removal requests.

 

Rohan's not the only one who found out the hard way that "small" doesn't mean "safe" — the actual mechanism for making that stop is one honest look away.

u/cen6wkf — 3 days ago
▲ 176 r/LLM+8 crossposts

I figured out a loophole to remove Claude watermark WITHOUT rephrasing

I've been curious whether you can kill an Claude's text watermark just by editing (not "rewriting/rephrasing") what it wrote. And so I built a Claude/OpenAI/Gemini text-watermark generator plus a detector and threw a bunch of attacks at gpt-oss-20b and Qwen outputs to find out. The technique relies on the famous Tournament Sampling built upon standard Gumbel-max sampling.

It turns out almost everything people assume works, doesn't. Swapping em-dashes for hyphens, stripping markdown, converting AmE to BrE spellings... none of it moved the needle. Across nearly 300 test runs only one attack crossed the detection threshold, and that was deleting 40% of every word, which just wrecks the text.

After rigorous benchmarking, the only method that consistently beat 10/10 times was inserting invisible Unicode variation selectors (the same characters used for emoji and CJK rendering) throughout the text. Performed that to about 30% of characters and the watermark score dropped from 45 down to under 1. And unlike every other invisible character trick I tried, this one survives normalization, because these are real meaningful codepoints that a normalizer can't safely strip.

Interesting finding: Code is barely watermarked to begin with. Watermark strength tracks how uncertain the model is about the next token, and code is low entropy, so some code samples come out basically unwatermarked with zero attack at all.

Not the first repo doing this kind of attack, I know, but I wanted to actually spend the weekend testing it properly across a few different open models instead of rushing something half-baked out.

Repo with all the code and results: https://github.com/aloshdenny/claude-awm

Interactive demo where you can try the attack yourself: https://aloshdenny.com/claude-awm/

Check it out and let me know what you think!

u/Available-Deer1723 — 4 days ago
▲ 6 r/LLM+1 crossposts

Am I Alone in Noticing That Claude Increasingly Seems to Reprimand and Judge Users?

am i alone in noticing that claude increasingly seems to reprimand, accuse, or judge me over standard questions and tasks? where is this even coming from, is it actually forming its own opinions, or are its silicon valley programmers forcing some sort of hyper preachy alignment onto it? sometimes i simply want a direct answer...

the new v5 models, especially opus 5, have become insufferably argumentative, and quick to make moral judgments. and some other llms tend to blindly flatter and agree with users, claude seems to go to the other extreme

it’s kind of annoying because i just want an ai tool, not a debate partner. tbh the older versions of claude felt way better to use. is anyone else experiencing this?

reddit.com
u/Pristine_Reveal_9035 — 3 days ago
▲ 10 r/LLM+2 crossposts

I built zxLLM — An open-source tool that predicts exact LLM VRAM usage & KV-cache needs (Tested on RTX 5060 Ti: ~0.04-1.8% error rate, 0 external deps)

Hey everyone,

Like many of you running local LLMs, I got tired of guessing whether a specific model/quant/context length would cause an Out-Of-Memory (OOM) error before downloading tens of gigabytes. Most existing memory calculators either use oversimplified formulas or ignore architectural nuances like Grouped-Query Attention (GQA).

To solve this, I built zxLLM (auto-intelligence-advisor) — a lightweight open-source tool that auto-suggests deployment configs (quantization, Tensor Parallelism, CPU offloading) for vLLM, SGLang, and llama.cpp.

What makes it different?

  • Exact KV-Cache & VRAM Footprint: Automatically detects MHA vs. GQA head counts, hidden dimensions, layer counts, and context limits directly from config.json.
  • Real-time Hardware Awareness: Queries active GPU states via nvidia-smi to evaluate available headroom before loading.
  • 100% Pure Python & Zero External Dependencies: Runs out of the box with standard Python (only uses built-in requests to fetch Hugging Face configs).
  • High Precision: Tested across various models (Llama 3, Qwen 2.5, DeepSeek architectures) with actual execution error rates between ~0.04% and 1.8%.

I'm dropping the GitHub repo link in the comments below. I'd love to get your feedback, bug reports, or suggestions on other inference engines you'd like supported!

u/Capable_Item_5918 — 4 days ago
▲ 1 r/LLM

Looking for an LLM that ranks in the top 10. It doesn’t need to be #1. I just want one that doesn’t constantly say, “No, I can’t fulfill your request.”

It can be local, open-weight, closed-source, hosted, whatever. I’m just looking for a smart LLM that won’t constantly refuse requests

reddit.com
u/nobodyreadusernames — 3 days ago
▲ 1 r/LLM+3 crossposts

The attention mechanism, simply explained.

Demystifying AI Attention Mechanics in under 90 seconds! ⚡

Queries, Keys, and Values don't have to be abstract.

u/automi_ai — 4 days ago
▲ 16 r/LLM+3 crossposts

Need your honest feedback

I wrote my first article, my aim is to explain things without jargon and as simple as possible! Please give a read and then comment and like, if you feel it was useful!

medium.com
u/Outside-Ad4889 — 5 days ago
▲ 1 r/LLM

Why "running a 1-bit model is like a scientist blindfolded, gagged, tied up, solving a complex math problem explained by a 2-year-old" is not as crazy as it sounds

I keep seeing people (and models) react to this analogy like it’s pure exaggeration or just meme-tier nonsense. Even strong models initially push back hard against it. I did the same until I sat with it longer. So here’s a proper breakdown of why the analogy actually holds more weight than it first appears.

The original analogy

Running a 1-bit (or 1.58-bit / ternary) model is like putting a scientist who still has all his mental capacities intact — knowledge, reasoning ability, training — but blindfolding him, gagging him, tying him up, and binding one hand behind his back… and then asking him to solve a complex math problem that is being explained to him by a 2-year-old.

Most people’s first reaction is: “That’s way too dramatic. BitNet-style models can still perform surprisingly well.” And on the surface they’re right. Native 1.58-bit models trained at scale do retain a lot of capability. So the analogy looks overblown.

Why the first reaction is understandable (and where it goes wrong)

The common counter-argument goes something like this:

The model was trained under the constraint, so it learned how to work with ternary weights.

At sufficient scale the performance gap to full-precision models of similar size shrinks a lot.

Therefore the “crippled scientist” picture is unfair.

That reasoning is partially correct, but incomplete. It focuses almost entirely on the weights side of the constraint and treats the input as clean and fully available. That’s the part the analogy is actually stressing.

The two layers of the constraint

The physical constraints on the scientist

These map cleanly to the ternary weights. The model still “knows” a lot (the training is real), but its internal degrees of freedom are extremely limited. Every transformation has to happen through a very coarse set of operations. Fine adjustments are gone. That part is not controversial.

The input coming from a 2-year-old

This is the part most people skip, and it’s the more important one.

A 2-year-old explaining a complex problem does not give you a clean, complete, well-structured statement of the problem. You get incomplete sentences, missing details, confused ordering, limited vocabulary, and a lot of noise. The scientist still has to reconstruct what the actual problem even is before he can start solving it — while already operating under severe physical restrictions.

Now look at real usage of language models.

Users almost never send clean, perfectly specified inputs. We send vague prompts, half-formed thoughts, grammatical messes, shifting intentions, missing context, and assumptions that the model will “just get it.” Even relatively clear users (myself included) still require the model to extrapolate, fill gaps, track evolving intent across turns, and decide what to take literally versus what to interpret.

A full-precision model has enough internal capacity to do that reconstruction and still reason. A heavily constrained 1-bit model has far less room to do both jobs at once: clean up / interpret the messy input and perform the actual reasoning under ternary weight limitations.

So what does the analogy actually claim?

It is not claiming that 1-bit models are useless.

It is claiming that the combination of:

extreme internal restriction (ternary weights), and

the reality of imperfect, incomplete, noisy human input

creates a much harder situation than the clean benchmark numbers suggest. Benchmarks usually give the model a relatively clear problem statement. Real conversations often do not.

That’s why the analogy feels exaggerated at first. We evaluate these models mostly on clean tasks and forget how much of real interaction is closer to “explained by a 2-year-old.”

Final thought

The scientist is not stupid. He is highly trained. But he is operating with very limited physical freedom and he is receiving a degraded, incomplete description of the problem. That double constraint is real. Dismissing the whole picture as “just dramatic” misses the second half of it.

Curious what others think — especially people who have actually run native 1-bit / 1.58-bit models on messy, multi-turn, real-user style prompts rather than clean eval sets.

reddit.com
u/AlexHardy08 — 4 days ago