r/huggingface

A large language model can only use the text that fits in its context window, and it recomputes its internal key-value (KV) state for a prompt every time the prompt is sent. We test a memory layer, the public package galahad-kv, that saves the KV state of each block of about 16,000 tokens to encrypted local NVMe disk and loads it back later, byte-exact, without recomputing it. We ran it on 50,000,000 tokens of real public text, served through vLLM on one NVIDIA H100, with Gemma 4 12B and Gemma 4 31B

https://huggingface.co/papers/2610.10845

Stack

• Strata NVFP4 fork: github.com/sergqwer/strata-nvfp4
• Model: jpezzulli/OrcaRouter-Qwen3.8-Flash-Next-Uncensored-ModelOpt-NVFP4 Hugging Face model
• NVFP4 routed experts (~63.3 GiB), separate FP8 PLE, INT8 KV cache, MTP speculative decoding
• W4A8 prefill on Blackwell

Main engine flags:

./build/strata \
  --pack packs/orca-nvfp4 \
  --native models/orca-nvfp4.gguf \
  --native-dense-gguf models/orca-nvfp4.gguf \
  --ple-gguf models/ple-fp8.gguf \
  --mtp mtp-orca/rt \
  --spec 4 --spec-min-p 0.5 \
  --prefill auto \
  --expert-profile data/expert-profile.bin \
  --expert-cache auto \
  --resident-budget-gib 40 \
  --max-context 200000 \
  --kv int8

I serve it through Strata's OpenAI-compatible server.

Results so far:

• ~50k context: up to ~80 tok/s
• ~188k warm context: ~60–67 tok/s
• cold 189k full prompt: ~1,680 tok/s prefill, ~53 tok/s decode

Pretty impressive for a single 32GB GPU + only 64GB system RAM.Running Qwen3.8-Flash-Next NVFP4 with Strata on a single RTX PRO 4500 32GB + 64GB DDR5.StackStrata NVFP4 fork: github.com/sergqwer/strata-nvfp4

Model: jpezzulli/OrcaRouter-Qwen3.8-Flash-Next-Uncensored-ModelOpt-NVFP4 Hugging Face model

NVFP4 routed experts (~63.3 GiB), separate FP8 PLE, INT8 KV cache, MTP speculative decoding

W4A8 prefill on BlackwellMain engine flags:./build/strata \
--pack packs/orca-nvfp4 \
--native models/orca-nvfp4.gguf \
--native-dense-gguf models/orca-nvfp4.gguf \
--ple-gguf models/ple-fp8.gguf \
--mtp mtp-orca/rt \
--spec 4 --spec-min-p 0.5 \
--prefill auto \
--expert-profile data/expert-profile.bin \
--expert-cache auto \
--resident-budget-gib 40 \
--max-context 200000 \
--kv int8I serve it through Strata's OpenAI-compatible server.Results so far:~50k context: up to ~80 tok/s

~188k warm context: ~60–67 tok/s

cold 189k full prompt: ~1,680 tok/s prefill, ~53 tok/s decodePretty impressive for a single 32GB GPU + only 64GB system RAM.

Face-hugger is a new project I've been working on to improve Hugging Face's search engine which could be described as terrible at best.

• Attempts to analyze files to verify their model (1.2M+ so far)
• Fetches training metadata when available
• Filter by model family or sub-type
• Other filter options: download/like count, date range, gated filter
• Easily download files directly from HF
• Add repos, files, or authors to your favorites list to easily find later
• Hide repos or authors to avoid seeing cloned repos
• Badge color customization to easily distinguish models when scrolling
• No account creation needed, everything is stored locally in your browser
• It's a work in progress so if you find bugs/have suggestions there's a Discord link in the About section

Introducing Agr, an open decision model by Command Code.

• Agr (31B) and Agr-flash (360M)
• 58.15 on Decision Index 0.2.1
• Focus on: tool calls & routing
• TypeSafe SDK support

Agr is best-in-class on coding agent tool calls and routing. Live on Hugging Face.

Send state as text or JSON with typed questions. Agr skips text generation and returns typed values with a probability for each option.

We tested Agr against the complete public Decision Index suite. Our focus with these training runs has been on accuracy for tool calls and coding-task routing, while we train a much broader general variant.

Agr model family is available on Hugging Face and GitHub.

We’ll work closely with the open-source community on inference support and explore more deployment options. Planning a large scale deployment? Let’s talk.

Link here: https://huggingface.co/CommandCode/agr