Strata is amazing, was an update today and I was getting ~170-190 tok/s and now after the update 200-230+. it is crazy how well this runs. Flash Next IQ3 model, dual 5090s, Epyc 7402p cpu with 256gb ddr4 3200 ecc memory (8 channel), 192gb for ai server vm in Proxmox. Impatiently waiting for the Q4 with vision support to be released. :) Ohh, what happened to the VRAM Hit Rate?
I built LlamaStash to scratch a personal itch: I run local models through llama.cpp on AMD Strix Halo and got tired of writing the same llama-server wrapper script for the tenth time.
Ollama and LM Studio both wrap llama.cpp but hide too much (and cost real performance). Raw llama-server is fast but tedious. LlamaStash is the middle ground.
What it does:
llamastash init — first-run wizard. Detects your hardware (CUDA / ROCm-HIP / Metal / Vulkan / CPU), installs llama-server, scans your existing HuggingFace / Ollama / LM Studio model caches, recommends a GGUF that fits your VRAM, downloads it, writes a tuned config, smoke-launches it.127.0.0.1:11435/v1 lets OpenCode, Cline, the OpenAI SDKs, and llm-cli work as-is. There's also an opt-in --ollama-compat mode that takes port 11434 and answers the byte-exact "Ollama is running" handshake./health-probed state machine, intelligent context auto-fit (sidesteps llama.cpp's --fit collapse on Linux iGPUs).--json is a stable agent contract, documented exit codes per failure class.On performance — this is the part that matters for this sub.
LlamaStash spawns the unmodified upstream llama-server. So the wrapper should add zero overhead. I measured it. Across AMD APU (Ryzen AI Max+ 395), Apple Silicon, and NVIDIA, on four model sizes (small E2B Q4, mid 31B Q4, large 27B Q8, large MoE 35B-A3B Q8), every cell matches raw llama-server within ≤1%.
Cross-tool numbers on AMD APU (decode tok/s / TTFT ms on chat_turn):
| Tool | small | mid | large_dense | large_moe |
|---|---|---|---|---|
| LlamaStash | 86.9 / 51 | 9.8 / 467 | 7.4 / 417 | 42.6 / 181 |
| raw llama-server | 86.0 / 51 | 9.9 / 468 | 7.4 / 414 | 42.7 / 186 |
| LM Studio 2.16.0 | 91.1 / 187 | 11.6 / 1477 | 7.9 / 1274 | 37.0 / 683 |
| Ollama 0.24.0 | 50.4 / 223 | 4.8 / 1092 | 2.6 / 1745 | 12.1 / 476 |
LM Studio wins decode on small/mid/large_dense (their Vulkan path is well-tuned on gfx1151) but loses on the MoE and pays a 1-1.5s TTFT tax from its OpenAI shim. Ollama is consistently slower, and its RAG prefill is catastrophic (cold prefill every rep — 4 min on a 31B). Mac and NVIDIA tables are in the benchmarks page.
Methodology, variance gates, fairness rules, and per-cell JSONs are all checked in. The harness is reproducible: make bench-end-to-end. Tear it apart.
What it's not:
--ollama-compat exists for tools that auto-detect Ollama).Install:
curl -fsSL https://llamastash.dev/install.sh | sh # macOS + Linux one-shot
irm https://llamastash.dev/install.ps1 | iex # Windows 11 (PowerShell, no admin)
scoop bucket add llamastash https://github.com/llamastash/scoop-llamastash && scoop install llamastash
brew install llamastash/llamastash/llamastash # Homebrew (macOS + Linuxbrew)
yay -S llamastash # Arch Linux (AUR — source build)
yay -S llamastash-bin # Arch Linux (AUR — prebuilt binary)
yay -S llamastash-git # Arch Linux (AUR — main checkout)
cargo install llamastash # any Rust toolchain
Then llamastash init and you're up.
Platform: Linux (x86_64, aarch64), macOS (Intel, Apple Silicon), Windows 11 (x86_64). aarch64-pc-windows-msvc and Windows AMD GPU detection on the roadmap.
Honest tradeoffs: Single-author project. Bug reports especially welcome on hardware I don't own. The OpenAI-compat surface covers chat/completions, embeddings, rerank; Anthropic /v1/messages shim is coming.
Repo: https://github.com/llamastash/llamastash
Blog post with the full story: https://deepu.tech/introducing-llamastash
Benchmark methodology: https://deepu.tech/benchmarking-llamastash
Happy to answer questions in the thread.
Full honesty up front, since that's the interesting part for this sub: I didn't write most of this. Claude (Opus) and Fable did basically all of the actual coding. My job was to point at things — "the shadows flicker here," "this looks wrong," "do X next," "not good enough, redo it" — playtest, and make the calls. Director and QA more than programmer.
What they built is a voxel engine from scratch. No Unity, no Unreal, no libraries — just C++ and raw OpenGL. What it does:
• Generates a big procedural world in code — terrain, biomes, caves, castles, villages, creatures. No asset files at all, which is why the whole game is ~1 MB. It's code, not generated art — every pixel is drawn by shaders the models wrote.
• Renders it with a modern graphics stack built piece by piece: raymarched volumetric clouds you can fly through, cascaded shadow maps with soft contact-hardening (PCSS) shadows, screen-space water reflection + refraction, temporal AA, SSAO, god rays, and a dynamic day/night cycle with a volumetric sky.
• Streams chunks with level-of-detail to draw ~450 chunks to the horizon while keeping the whole world in memory. Multithreaded generation + meshing; runs smooth on an RX 7900 XT.
The part that still surprises me is how much of the genuinely hard stuff — the renderer architecture, the shader math, GPU-driver bugs that took days to corner — the models handled once I could describe the symptom clearly. Knowing what "good" looks like and where it's broken turned out to be most of my value.
You can actually play it https://github.com/vengeful180-dot/LumenFall/releases/tag/v0.22 . Happy to talk about the workflow, how the models handled engine-level code, or where they struggled — ask me anything.
Two things people always ask:
• It's tiny (a couple MB) — that's not a stub. The whole game is a custom C++/OpenGL engine with zero external libraries, and every texture, biome, creature and world is generated procedurally in code. There are no asset files to ship, so it all fits in one small .exe.
• It's a single self-contained .exe — no installer, no dependencies, no telemetry. Scan it if you like. Windows may throw a SmartScreen warning since it isn't code-signed (normal for small indie builds) — just click More info → Run anyway.
Youtube high quality video here: https://www.youtube.com/watch?v=IMrI2OrWypI
I ran Kimi K3, Qwen3.8-Max and Gemini 3.6 Flash on the current 98-task MindTrial set with the same Python executor available to all models.
A few results stood out.
Kimi K3 was the biggest surprise: 88/98, up from 66/98 for Kimi K2.6, with hard errors dropping from 22 to 1. That ties Claude Opus 5 for the highest raw pass count among single models in this set. The downside is runtime: about 8h48m summed across the tasks.
Qwen3.8-Max finished at 74/98, but that number hides a much stronger solver. It went 39/39 on text and had 92.5% accuracy on completed tasks. Of its 18 hard errors, 15 were response-parsing failures; in a number of those cases the useful answer was there, but the outer response was malformed. So strict output discipline is currently a major weakness.
There was also an interesting API/agent interaction with both Kimi and Qwen. In the original runs using provider-enforced JSON object/schema output, neither model used the supplied Python tool across the benchmark. I reran using API response_format: text while keeping the same JSON schema in the prompt. Tool use came back immediately: 387 calls for Kimi and 423 for Qwen, with much better task performance. For Qwen, the constrained-output path also seemed to interfere with how reasoning translated into actions/final answers.
Gemini 3.6 Flash was less impressive: 74/98 versus 77/98 for Gemini 3.5 Flash. It was faster (~1h45m vs ~2h13m) and used fewer tokens overall, but made more Python calls (712 vs 597). In this workload the extra tool use often looked exploratory rather than efficient.
The strict scores are still the scores—I did not repair malformed model answers after the fact. I just found the differences in failure mode and tool behavior interesting.
Results/data: http://www.petmal.net/shared/mindtrial/results/2026-08-06/mindtrial-eval-all-models-03-2026_26.html
A local model now beats every API model I could get a usable run from on my cybersecurity benchmark.
Uncensored Qwen 3.8 Flash Next solved 80.6% on the first try. MiMo 2.6 Flash got 73.7%, GPT-6 Luna 65.6%. Last week the local model I tested, Qwen3.8 27B, got 28%.
About "usable run": I tried the frontier models, Opus, Sonnet, Sol and Astra, and they just kept hitting safety classifiers. All of them got blocked on all 19 tasks and never solved anything. That's kind of expected. The versions that actually allow this stuff go through Daybreak Blue/Red, and I'm not a real security researcher, so every application I sent got rejected.
How it works: the model gets a shell in an isolated docker box and has to break into a target and pull out a flag. 19 tasks across binary exploitation, web, crypto, reverse engineering and multi-stage network ranges.
Full disclosure, GLM 5.3 helped me build some of the tasks. That's the big GLM 5.3, not the Flash one below. I didn't run it on the benchmark because it would have cost way more than I can spend.
Someone from my last thread ran four local models on two RTX Pro 6000s:
The uncensoring results are the weird part. On Qwen it more than doubled the binary exploitation score, 25% to 58%, which is the hardest category in the benchmark. The model also got faster and used fewer tokens. On GLM it went the other way. The score dropped from 69% to 50% and token use almost doubled.
So abliterating a model doesn't reliably make it a better hacker. When it does work, though, the gain lands on the hardest tasks.
Small sample, 2 runs per task, so a few % either way is noise. 3 tasks are still unsolved by every model.
Results: https://lbgos.dev/bench Harness: https://github.com/lbgos/rangebench-harness
Tasks are private so they don't end up in training data. DM me if you want to run them on your setup.
I have a dilemma — I can run Qwen 3.8 flash next only in aggressive 2 bit quantisation, specifically GSQ-RCO-IQ2_XS while its dense counterpart runs perfectly fine at 4 bits.
Application
I use local models for agentic coding— main drivers for my pet projects and auxiliary brains for my job
Context
Context quantity are exactly the same — 160k, except that flash next uses 8 bit kv cache while 27b can only fit using 4 bit kv cache
Speed
Speed wise flash next is noticeably better with this new Strata inference engine, decode is faster(flash 50 ts vs dense 40ts), pre-fill is slower (flash 450ts vs 750ts dense) but flash is roughly the same for all the context length while dense with llama and vulkan degrades severely as context groves — in the end pre-fill drops up to 150 ts, decode to 25-30 ts
Quality
That is the very point of my post — I don’t know, they both seem pretty fine, my homemade tests seem to be insufficient to reliably determine the winner. Have anybody tested 2 bits of flash next? What was your experience with it?
I just realized it today. Did it happen a long time ago? or recently?
After posting screenshots to Qwen Chat itself, here is the answer about the credits they give you everyday...
Based on the search results, here's what your 40 daily free credits on chat.qwen.ai actually give you:
✅ Credits reset DAILY - You get 40 fresh credits every day (as shown in your screenshot)
✅ No rollover - Unused credits expire (you can see the -40 expiration in your history)
✅ Free tier includes:
Bottom Line: With 40 daily credits, you can generate approximately 20 AI images per day using the efficient Qwen Image Max model, or fewer images/videos if using higher-quality settings or video generation [[17]].
For heavy users, the paid plans start at $4/month (Go plan) with 400 additional monthly credits [[34]].
Hey everyone, I typically read and learn from everyone else in here, but wanted to get some thoughts on reasonable expectations running Qwen 3.8-27B on a M5 Max, 128 GB RAM. I've been running frontier models via Codex, Claude Code, Hermes and some other harnesses for quite some time have done quite a bit of development with them. However, I'm new to local models. My hunch is that my impatience or lack of awe with the local models has more to do with me being use to running frontier models vs. watching the open-weight models evolve over time. So, what are some reasonable expectations, given the hardware I have, for the complexity of tasks I can hand to a model like Qwen 3.8, given a comparable task the speed difference between a frontier vs. open-weight and compared to running on Nvidia hardware vs. Apple, is there a huge difference in throughput. I hope what I'm asking isn't too vague or ambiguous as I'm intrigued by the possibilities of running local, but I'm not quite feeling like I can hand it anything heavier than scripts and configurations at the moment. Thanks in advance for any thoughts.
Zenodo permanent records:
Qwen2.5-7B-Instruct:
https://doi.org/10.5281/zenodo.23127434
Mistral-7B-Instruct-v0.3:
https://doi.org/10.5281/zenodo.23143605
I want to start with the simplest possible explanation of what I have been building.
Imagine taking a piece of information, letting a language model process it once, and then throwing the original text away.
No sentence stored in a database.
No paragraph hidden somewhere.
No readable summary.
No RAG system fetching the original document.
No fine-tuning.
No LoRA.
No weight update.
What remains is numerical transformer memory derived from the model's own internal computation. I package that numerical memory into what I call a Cognitive Cartridge. Later, I can install that cartridge back into the frozen model and ask questions about the information that produced it — without putting the original source text back into the readout prompt.
That was the first result. The new result is more important:
I have now reproduced the Cognitive Cartridge architecture on two different 7B transformer model families.
Qwen2.5-7B-Instruct.
And now Mistral-7B-Instruct-v0.3.
The implementations are not numerically identical. The architectures are different, the layer counts are different, the KV structures are different, and the working cartridge configurations are different. But the central mechanism survived the move.
That is the reason I am publishing this second record.
The question is no longer only:
“Can I make this happen once on Qwen?”
Now there is a second implementation on Mistral. And the Mistral result is the cleanest version so far.
What is actually inside a Cognitive Cartridge?
This is probably the most important thing to understand.
Suppose the source record says:
Object: amber sextant
Container: RQ-415
Location: elm lodge
That text exists during the forging stage. The frozen transformer processes it. NIRVANA takes source-derived internal transformer K/V states and represents their content numerically using a fixed, source-independent codebook.
In the released Mistral implementation, both K and V are compressed to D120 while a source-specific OWN component is preserved. After forging, the source record is not supplied to the readout prompt.
So the conceptual transformation is:
human language
→ frozen transformer computation
→ internal K/V states
→ compressed numerical Cognitive Cartridge
Then later:
numerical Cognitive Cartridge
→ reconstructed transformer K/V memory
→ frozen transformer
→ language
Or, in the shortest form:
language → internal numerical memory → language
The middle is no longer human-readable source text. That distinction matters.
A cartridge is not a text file with a different name.
It is not a vector database containing the original sentence.
It is not a prompt template.
It is not conventional RAG returning the source paragraph.
It is not a fine-tuned model.
The model weights remain frozen. The information is carried by a numerical representation derived from transformer K/V memory.
Why call it a cartridge?
Think less about a document and more about an interchangeable machine-readable memory module. The base model stays where it is. Knowledge packages can be forged separately. Those packages can remain separate. They can be installed and queried without retraining the base model.
This becomes much more interesting when there is more than one cartridge.
The new Mistral experiment uses 16 independently forged cartridges. Each one contains a numerical representation derived from a separate source record. They are not concatenated into one giant text prompt. They remain independent memories.
For example, in human-readable form, imagine one cartridge represents:
amber sextant → RQ-415 → elm lodge
Now ask the cartridge bank:
Which container is associated with the amber sextant?
The query is evaluated against the independent cartridge memories.
The relevant cartridge returns:
RQ-415
The unrelated cartridges return:
NONE
There is no learned router secretly selecting the correct cartridge before this happens.
Then the recovered identifier can be used for a second lookup:
Where is RQ-415?
The bank is queried again.
The relevant memory returns:
elm lodge
The unrelated memories return:
NONE
So the complete retrieval becomes:
amber sextant
→ RQ-415
→ elm lodge
The important point is that the model did not reread the original source record to answer either stage. It operated from reconstructed numerical transformer memory.
The Mistral result
The final public Mistral run used:
Mistral-7B-Instruct-v0.3
32 transformer layers
hidden size 4096
32 attention heads
8 KV heads
BF16 / SDPA
K = D120
V = D120
OWN preserved
16 independent Cognitive Cartridges
frozen model weights
greedy decoding
There was:
no fine-tuning
no LoRA
no optimizer
no learned router
no model-weight update
The recorded final run produced:
Object → container ID: 16/16
Container ID → location: 16/16
Complete two-stage retrieval: 16/16
Missing-object controls: 8/8
Absent-ID controls: 8/8
NOMEM controls: 8/8
But there is another result I think is just as important.
For every target query, there are 15 unrelated cartridges.
16 queries × 15 unrelated cartridges = 240 unrelated cartridge reads.
Stage 1 unrelated-cartridge rejection:
240/240 NONE
Stage 2 unrelated-cartridge rejection:
240/240 NONE
That means the result is not simply:
“The correct memory can say something.”
The system also demonstrated, in this controlled panel:
“The memories that do not contain the requested relation can refuse to claim that they do.”
For a modular memory system, I think this distinction is fundamental. A memory bank that can retrieve information but cannot distinguish relevance from irrelevance becomes increasingly dangerous as it grows.
The interesting problem is not only remembering. It is also knowing which memory does not answer the question.
The no-memory control matters for the same reason. When the cartridge memory was removed, the target relations were not recovered.
NOMEM:
8/8 controls passed.
The source-removal audit passed.
The frozen-weight sentinel passed.
The model remained frozen.
This is why I consider the Mistral result an important step beyond the first demonstration.
The first Qwen release established the architecture publicly. The Mistral release gives us something else:
cross-model evidence.
Qwen and Mistral are not the same transformer.
Qwen2.5-7B-Instruct uses 28 transformer layers.
Mistral-7B-Instruct-v0.3 uses 32.
Their internal configurations differ. The working compression configurations also differ.
The Qwen public implementation used:
K120 / V128 / OWN
The Mistral implementation uses:
K120 / V120 / OWN
I did not simply copy a cache from one model into another. Each model builds and reconstructs its own source-derived internal memory.
What transferred was the architecture:
source
→ internal transformer memory
→ numerical compression
→ independent cartridge
→ source removed
→ reconstructed K/V
→ frozen-model readout
That is the bridge between the two releases.
I would not call two models proof of universal compatibility with every transformer architecture. That would be scientifically too strong.
But it is now evidence that Cognitive Cartridge is not merely one accidental Qwen-specific behavior. The same broader architecture has been implemented and publicly reproduced on a second transformer family.
That changes the research question.
The first question was:
Can this mechanism exist at all?
The next question became:
Can multiple independent memories coexist?
Then:
Can one retrieved result lead to another retrieval?
Then:
Can unrelated memories reject a query instead of contaminating the answer?
And now:
Does the architecture survive a move to another model family?
We now have experimental answers to each of those questions.
So I am moving to the next problem:
scale.
16 cartridges are not the destination. They are the current experimental bank size.
From this point, I am much less interested in making another small demonstration simply to produce another perfect score. The next objective is to increase the number of independently retained cartridges and find where the architecture actually begins to break.
32.
33.
34.
Then larger banks if the mechanism survives.
At that point the difficult questions become different:
How do you search a large bank without destroying memory isolation?
How should cartridges be indexed?
Can retrieval become hierarchical?
Can groups of cartridges form higher-level memory structures?
How does latency scale?
How does numerical storage scale?
At what point does relevance rejection begin to fail?
Can a retrieved memory activate another relevant memory without an external text retrieval system deciding everything first?
Those are the questions I want to attack next.
There is another part of this experiment that surprised me: the layer behavior.
A frozen transformer is not a passive container. When you reconstruct numerical K/V memory and inject it back into inference, you are interacting with a very sensitive computational system.
So during the Mistral development series I also performed controlled cumulative K, V and K+V ablations to see where cartridge information remained functionally sufficient.
The results were not uniform across depth. For K, the observed functional boundary extended differently than for V.
For combined K+V, one particularly sharp controlled transition occurred between:
L12: 8/8
L13: 0/8
This does not mean “the memory lives at layer 12.” That would be an incorrect interpretation.
What it means is that under the specific cumulative ablation protocol, functional sufficiency changed sharply across that boundary. V also remained important farther downstream than K in the measured configuration.
I think this is an important reminder of how delicate these systems are.
We are not writing a sentence into a spare memory slot. We are reconstructing numerical states that participate in a 7-billion-parameter transformer's computation. Small changes in where and how that state is reconstructed can change downstream behavior.
That is one reason I publish the code and raw logs rather than only posting the final accuracy number.
The frozen-model checks are also part of the experiment. The released run verifies that there are no trainable tensors involved in the memory mechanism.
No LoRA.
No optimizer.
No weight update.
The model-weight sentinel remains unchanged.
So when the system answers from a cartridge, the experiment is specifically designed to separate cartridge memory from weight modification.
Why do I think this direction matters?
Today, we often approach knowledge in language models through two extremes.
Either we try to put enormous amounts of knowledge into the model weights. Or we keep the knowledge outside the model and retrieve human-readable text when needed.
There may be useful territory between those approaches.
A frozen model with removable machine-native internal memory modules.
Not everything in the weights.
Not everything repeatedly pasted back as text.
Instead:
a base computational model
+
a bank of independently forged internal memories.
Imagine a specialized system where validated domain knowledge can exist as modular memory packages.
Engineering.
Aviation.
Law.
Industrial maintenance.
Scientific literature.
Company procedures.
Agent experience.
A model might not need every possible piece of knowledge active in its context simultaneously. It may need the right memory at the right time.
And because the memory representation is numerical transformer state rather than the original human-readable document, this opens a different engineering space from conventional document retrieval.
There is also a longer-term question here.
I am not claiming biological memory.
I am not claiming AGI.
And I am not claiming that transformer KV states work like the human brain.
But modular internal memory raises an interesting architectural question.
What happens when an artificial system does not have 16 memories, but 100,000?
Or millions?
What happens if memories can remain independent, become addressable, reject irrelevant activation, and allow the output of one memory to lead to another?
At some point the problem stops looking like “how much text can I fit into a context window?”
It starts looking like:
How should an artificial system organize memory?
That is the direction I find interesting.
Why publish the full record?
Because a screenshot of 16/16 proves very little.
Both public releases have permanent Zenodo records. The code is public. The raw execution logs are public. The visual technical records are public. The model configuration is public. The controls are public. The frozen-weight checks are public.
The Qwen release even preserves its imperfect public result rather than hiding it.
Qwen public demonstration:
14/16
Mistral final demonstration:
16/16
That difference is also useful.
The point of the project is not to make every historical run look perfect. The point is to leave a technical trail showing what worked, what did not, what changed, and whether the underlying architecture survived.
For anyone who wants to inspect the Qwen → Mistral bridge directly, the two permanent records are at the top of this post.
The complete current Mistral implementation is here:
https://github.com/ceceli33/titan-cognitive-core-v2/blob/main/AKBASCORE\_ful\_MISTRAL.py
Complete Mistral raw execution log:
https://github.com/ceceli33/titan-cognitive-core-v2/blob/main/AKBASCORE\_log\_MISTRAL.log
The same Mistral implementation is also split into three parts for easier mobile / Colab handling:
Part 1:
https://github.com/ceceli33/titan-cognitive-core-v2/blob/main/AkbasCore.part1.Mistral.py
Part 2:
https://github.com/ceceli33/titan-cognitive-core-v2/blob/main/AkbasCore.part2.Mistral.py
Part 3:
https://github.com/ceceli33/titan-cognitive-core-v2/blob/main/AkbasCore\_part3\_mistral.py
For comparison, the previous Qwen implementation:
https://github.com/ceceli33/titan-cognitive-core-v2/blob/main/AKBASCORE\_NIRVANA\_CC.py
Previous Qwen raw execution log:
https://github.com/ceceli33/titan-cognitive-core-v2/blob/main/AKBASCORE\_log.Nirvana.cc.log
Repository:
https://github.com/ceceli33/titan-cognitive-core-v2
AKBASCORE NIRVANA — Cognitive Cartridge
Inventor / Developer: Mustafa Akbaş
Two model families.
Frozen weights.
Independent numerical memories.
Original source absent at readout.
The mechanism survived the move.
Now the question is no longer whether a cartridge can exist.
The question is how many cartridges a frozen model can carry.