I’ve been testing how far I can push long-context serving on a single A100 before the KV cache becomes the thing that kills concurrency.

Here’s the latest result:

9 concurrent users
~128K context per user
1× A100 80GB
Needle checks run independently for every user
FP8 holds the first 5 users

One technical detail I also want to correct from some of the earlier shorthand:

The V tail is tiered 4/3/2-bit bit-plane. It is not plain INT4.

Some of the recipe labels are stale. The actual pool sizes line up with the tiered 4/3/2-bit representation, so calling the tail “INT4” would describe the old label rather than what is actually being stored.

Method

I don’t keep the entire KV cache at one fixed precision.

The cache is divided into regions. Newer or more sensitive KV stays at higher precision, while older regions progressively move into cheaper representations.

As the context grows, the cache footprint can keep falling without forcing the same quantization level across the entire cache.

The 9-user run is currently the capacity end of the curve.

I’m also rerunning the 2-user and 4-user points using the same V2 tiered recipe. The July numbers came from the earlier V1 recipe family, and I want the next throughput curve to be completely apples-to-apples.

So far, a single A100 is keeping 9 users at ~128K context resident, with the per-user retrieval checks still passing.

There are a lot of knobs here: concurrency, context length, per-user throughput, precision, and memory.

I want to make those knobs much more flexible so long-context serving isn’t immediately dictated by the KV-memory wall. At scale, that can have a very real impact on how much useful inference you can get out of the same hardware.

Happy to share more of the pool math, the 4/3/2-bit layout, or the vLLM implementation if anyone wants to dig into it.

I’ve been testing bounded tool-calling on-device to see how small I can push local models before tool reliability breaks down.

A common issue with local agents is memory allocation—loading a 2.5 GB model into active phone RAM often leads to OS background terminations or thermal throttling during generation.

To test this, I built a small offline test pipeline using llama.cpp (b10075) with Metal and mmap to keep active memory low, then connected it strictly to local iOS calendar actions via EventKit.

On-Device Run Metrics

• Model Artifact: Gemma-2B quantized GGUF (SmartEdge-IQ3XXS.gguf)
• Disk Footprint: 2.45 GB
• Active Resident RAM (RSS): ~516 MB (leveraging mmap to page weights from disk rather than keeping the whole file in active memory)
• Decode Speed: ~12 tok/s interactive / 21.6 tok/s in a 256-token greedy benchmark
• Environment: Tested in airplane mode on iOS

Tool Execution Flow

The local LLM is restricted entirely to intent extraction and structured tool output; it does not execute actions directly.

1. User Input: "Find some time on Thursday for VC meeting."
2. Model: Extracts parameters and outputs a structured tool call.
3. App: Swift code validates the schema, queries local EventKit, and writes the event directly to the device calendar.

This avoids routing routine calendar edits through remote inference APIs or exposing local schedule data to external endpoints.

Quantization & Loss Comparisons

I also generated two calibration-aware quants to evaluate KLD degradation against the original bf16 baseline:

• Hi-Fi Q4_K_M: Equivalent size to standard Q4_K_M, with 36.0% lower code/math KLD and 27.2% lower general KLD against the original model.
• Hi-Fi Phone (2.86 GB): 17.5% smaller than the Q4_K_M baseline with 31.8% lower code/math KLD and 26.7% lower general KLD.

Limitations

• The full KLD matrix for the 2.45 GB SmartEdge build is still completing; current validation relies on on-device behavior logs, SHA-256 app receipts, and benchmark outputs. (The 2.86 GB and Q4_K_M builds have complete KLD data logged in the repo).
• Small models are prone to schema degradation if the prompt complexity scales beyond simple parameter extraction.
• Tool failure recovery still requires strict system-level guards or fallback routing.

Weights & Benchmarks

The GGUF weights, imatrix, SHA-256 hashes, evaluation slices, and Wikitext-2 perplexity loss are available on Hugging Face:
https://huggingface.co/fraQtl/Gemma-4-E2B-it-Hi-Fi-GGUF

If anyone tests this on other iOS hardware or Apple Silicon, I'd be curious to see your RSS memory usage and sustained decode rates.

Running Gemma 2B locally on iPhone for offline calendar actions (~516 MB active RAM, 21.6 tok/s, GGUF weights)

I’ve been testing bounded tool-calling on-device to see how small I can push local models before tool reliability breaks down.

A common issue with local agents is memory allocation—loading a 2.5 GB model into active phone RAM often leads to OS background terminations or thermal throttling during generation.

To test this, I built a small offline test pipeline using llama.cpp (b10075) with Metal and mmap to keep active memory low, then connected it strictly to local iOS calendar actions via EventKit.

On-Device Run Metrics

• Model Artifact: Gemma-2B quantized GGUF (SmartEdge-IQ3XXS.gguf)
• Disk Footprint: 2.45 GB
• Active Resident RAM (RSS): ~516 MB (leveraging mmap to page weights from disk rather than keeping the whole file in active memory)
• Decode Speed: ~12 tok/s interactive / 21.6 tok/s in a 256-token greedy benchmark
• Environment: Tested in airplane mode on iOS

Tool Execution Flow

The local LLM is restricted entirely to intent extraction and structured tool output; it does not execute actions directly.

1. User Input: "Find some time on Thursday for VC meeting."
2. Model: Extracts parameters and outputs a structured tool call.
3. App: Swift code validates the schema, queries local EventKit, and writes the event directly to the device calendar.

This avoids routing routine calendar edits through remote inference APIs or exposing local schedule data to external endpoints.

Quantization & Loss Comparisons

I also generated two calibration-aware quants to evaluate KLD degradation against the original bf16 baseline:

• Hi-Fi Q4_K_M: Equivalent size to standard Q4_K_M, with 36.0% lower code/math KLD and 27.2% lower general KLD against the original model.
• Hi-Fi Phone (2.86 GB): 17.5% smaller than the Q4_K_M baseline with 31.8% lower code/math KLD and 26.7% lower general KLD.

Limitations

• The full KLD matrix for the 2.45 GB SmartEdge build is still completing; current validation relies on on-device behavior logs, SHA-256 app receipts, and benchmark outputs. (The 2.86 GB and Q4_K_M builds have complete KLD data logged in the repo).
• Small models are prone to schema degradation if the prompt complexity scales beyond simple parameter extraction.
• Tool failure recovery still requires strict system-level guards or fallback routing.

Weights & Benchmarks

The GGUF weights, imatrix, SHA-256 hashes, evaluation slices, and Wikitext-2 perplexity loss are available on Hugging Face:
https://huggingface.co/fraQtl/Gemma-4-E2B-it-Hi-Fi-GGUF

If anyone tests this on other iOS hardware or Apple Silicon, I'd be curious to see your RSS memory usage and sustained decode rates.

I have been working on KV cache compression for long context inference.

The hard part was not making the cache smaller. That is easy to claim.

The hard part was making the memory savings turn into real serving concurrency in vLLM, without decode speed collapsing and without breaking retrieval.

I made the repro card public here:

https://huggingface.co/fraQtl/qwen3-4b-instruct-2507-kv-sidecars

Setup:

Qwen3 4B Instruct 2507
vLLM 0.20.2
one A100 80GB
128K context per user
1024 decode tokens per user
preemption free concurrent users
needle in a haystack gated runs

At each arm’s clean max:

fp16 KV: 2 users, 66.6 tok/s
fp8 KV: 5 users, 105.0 tok/s
compressed KV: 6 users, 140.9 tok/s

The main point is not just smaller KV cache.

It is:

same GPU
more long context users
still retrieval correct

Technically, the runtime stores the KV cache in a compressed representation instead of evicting tokens. The logical context stays present. The attention backend reads the compressed pages directly, so there is no separate decompress to fp16 step before attention.

That was the failure mode I kept running into earlier: if compression saves memory but slows the read path, the capacity win does not become useful serving throughput.

Every number is measured three ways:

fp16 KV
fp8 KV
compressed KV

Same command, same engine, same GPU.

Every arm has to pass retrieval before timing counts. I care about this because throughput without long context correctness is not very meaningful.

Current scope:

A100 SM80 only for this release
vLLM plugin path
no 256K claim
chunked prefill is still disabled for the compressed arm at 128K
compression is lossy, so the standard is retrieval verification, not lossless claims
calibration and factory code are not public, but the sidecars and repro kit are

Runtime wheel:

https://huggingface.co/fraQtl/fraqtl-sm80-runtime

Let me know what you think as it s been a lot of tears an frustration to get there lol

I published a Hugging Face repo for fraQtl D2 on Mistral-7B-Instruct-v0.3:

https://huggingface.co/fraQtl/fraqtl-d2-mistral-7b-instruct-v0.3

Clarification: this is not a full standalone Mistral checkpoint and not a fine-tune. It is a D2 KV-cache compression sidecar release for a patched llama.cpp setup.

The repo contains two D2 sidecar files, result images, README, and checksums.

The main result:

128K KV memory:
fp16:  22.1 GB
Q8:    15.1 GB
D2:    12.4 GB

So D2 is about -44% vs fp16 and -18% vs Q8 at 128K.

Retrieval result on paired 32K–128K needle eval:

fp16: 63/63 exact
Q8:   63/63 exact
D2:   60/63 exact
D2:   62/63 content-correct

The three D2 exact-match misses are disclosed in the README: one genuine digit error and two cosmetic hyphen-splits where all digits are correct.

So I am not claiming D2 beats Q8 on retrieval. The claim is more specific:

near-Q8 retrieval quality
below-Q8 KV memory
full 128K context
no retraining / no fine-tune
patched llama.cpp sidecars

There is also a small LongBench QA u/64K table in the README. On the disclosed 16-longest-samples/subset eval, D2 matches or beats fp16 on 3 of 4 subsets and beats Q8 on qasper.

I’m sharing this here because it may be useful to people experimenting with:

• Mistral-7B-Instruct-v0.3
• long-context inference
• KV-cache compression
• llama.cpp runtime modifications
• needle / retrieval evals

Feedback welcome, especially on:

1. whether the repo layout is clear
2. what additional receipts would make this easier to trust
3. whether there are better Mistral long-context evals I should run
4. whether the sidecar format is understandable to other llama.cpp users

released a reproducible fraQtl D1 patch bundle for Mistral-7B-Instruct-v0.3. ("D1" = the first released fraQtl drop — a pinned llama.cpp patch plus precomputed V/K sidecars, receipts, QUICKSTART, and a smoke script.)

https://github.com/fraqtl-ai/fraqtl-mistral-d1

The narrow claim

Mistral-7B-Instruct-v0.3, Q4_K_M weights, 128K context:

KV mode	Live VRAM	NIAH @ 128K
fp16 KV	22,657 MiB	5/5
q8_0 KV	15,437 MiB	1/5
q4_0 KV	11,287 MiB	0/5
fraQtl D1	13,261 MiB	5/5

The whole story is in those four rows:

• fp16 KV works but is expensive
• q8_0 / q4_0 KV save VRAM but lose long-context retrieval on this setup
• D1 keeps retrieval (5/5) while saving ~9.4 GiB vs fp16 So the point is not "smallest possible KV cache" — q4_0 is smaller and gets 0/5. The point is preserving long-context retrieval where normal KV quantization fails here. Those are the standard llama.cpp KV quant types (q8_0, q4_0) as representative comparison points, not a cherry-picked subset.

The VRAM gap also widens with context — the two curves diverge as you scale up:

On reproducibility

The sidecars are fixed artifacts. The smoke script verifies them against the receipts independent of any calibration code, so you can reproduce the VRAM and NIAH numbers without the (closed) sidecar generator. NIAH uses standard needle-in-a-haystack probes; harness details are in the repo.

What's public

• llama.cpp patch against a pinned upstream commit
• V/K sidecars
• receipts (command lines, VRAM peak traces, NIAH logs)
• QUICKSTART
• smoke script Sidecars are mirrored on Hugging Face: https://huggingface.co/datasets/fraQtl/Mistral-7B-v0.3-fraqtl-sidecars

What is not public yet

• calibration code that generates the sidecars
• allocator / scoring pipeline
• other model work This is not an upstream llama.cpp PR yet. I wanted the Mistral result reproducible first, then decide whether/how to upstream pieces cleanly.

If anyone runs the smoke script on A100 / H100 / 4090-class hardware, I'd be interested in your VRAM peak and whether your result lands near the receipt.

Hey guys,

I patched llama.cpp CUDA with post-hoc KV-cache compression for GQA models.
This is Mistral-7B v0.3 Q4_K_M on A100-80GB.

Measurements are live process VRAM from nvidia-smi, not theoretical KV bytes.
Same recipe at every context.

Results:

Context	fp16 KV	fraQtl	Δ MiB	Δ %	Quality
8K	6,095	5,793	−302	−5.0%	PPL parity
16K	7,215	6,305	−910	−12.6%	PPL parity
32K	9,327	7,201	−2,126	−22.8%	PPL parity
64K	13,695	9,137	−4,558	−33.3%	PPL parity
128K	22,657	13,235	−9,422	−41.6%	PPL drift ≤0.004
256K	40,497	21,601	−18,896	−46.7%	PPL drift ≤0.004

At 128K, fp16 KV uses 22,657 MiB while the compressed runtime uses 13,235 MiB. That is 9,422 MiB saved, or 41.6% lower live VRAM, with max PPL drift ≤0.004.

To note :)

• Both rows use Q4_K_M weights.
• The delta is KV cache + allocator + kernel layer, not weight quantization.
• llama.cpp proof only, not vLLM/SGLang yet.
• NIAH/passkey and tok/s are pending.
• Fork is private while I decide open-source vs binary release.

Ask: Happy to share methodology, receipt JSONs, or run requested context tiers. Independent measurement / bug reports welcome.

 Hey guys,
Just released fraQtl Apex, a Q4_K_M-class GGUF for Qwen 3.6 35B-A3B:

https://huggingface.co/fraQtl/Apex-Qwen3.6-35B-A3B

The goal was to keep a practical llama.cpp deployment footprint while preserving more of the model’s output behavior through calibration-aware per-tensor allocation.

Measured against a Q8 teacher on held-out slices:

Code/math:

  - Apex KLD: 0.02034

  - public Q4_K_M baseline: 0.02900

  - ~29.9% lower KLD

  - top-1 agreement: 97.22%

General chat/tool/long-form:

  - Apex KLD: 0.04852

  - public Q4_K_M baseline: 0.07166

  - ~32.3% lower KLD

  - top-1 agreement: 93.16%

  Two things changed vs stock Q4_K_M:

  - calibration-aware per-tensor protection

  - imatrix budget tuned to a measured optimum, 256K tokens on this packet

Interesting side result:

More calibration was not always better. A 384K budget produced worse KLD than the 256K build on the same slice.

The imatrix is included for reproducibility.

 Would genuinely love feedback from people running GGUFs locally, especially Qwen users.

I’ve been working on long-context inference/compression, and I kept running into a dumb but important problem:

It is easy to run a 64K context test that is not actually a clean 64K benchmark.

A model may have a native RoPE context of 32K, but you ask for 64K. Now the result depends on whether YaRN / rope scaling is configured correctly, whether the backend supports it, and whether you actually measured peak VRAM and retrieval behavior instead of just assuming it worked.

So I built a small diagnostic command that prints a “model context receipt” before I treat anything as a benchmark.

Example:

fraqtl inspect Qwen/Qwen2.5-7B-Instruct --context 65536

For Qwen2.5-7B at 64K, it flags things like:

• native context is 32,768
• requested context is 65,536
• YaRN / rope scaling is required
• YaRN is not configured
• estimated FP16 KV cache at 64K is about 3.76 GB
• peak VRAM still needs to be measured
• retrieval still needs to be tested

The point is not “this model works at 64K.”

The point is the opposite:

Before claiming anything, I want a receipt that says what is known, what is assumed, and what still needs to be tested.

I’m thinking of adding:

• perplexity
• needle-in-a-haystack / passkey retrieval
• decode tok/sec
• prefill tok/sec
• peak VRAM
• batch concurrency
• backend-specific notes for llama.cpp / vLLM / Transformers

Question for people doing inference or long-context evals:

What else would you want in this receipt before trusting a long-context run?