Sharing some results from my own PC running [Strata](https://github.com/Niko1221/Strata) with the HIP backend. The model is **Qwen3.8-Flash-Next-GSQ-RCO-IQ3_XXS**. These are engine timings from real chat sessions, including generated reasoning tokens.

My setup:

• AMD Radeon RX 7800 XT, 16 GB VRAM (`gfx1101`)
• Ryzen 7 5800X3D, 8 cores / 16 threads, AVX2
• 64 GB DDR4 RAM
• Ubuntu 26.04.1 LTS, kernel 7.0.0-38-generic
• Strata 0.1.42, built locally for HIP; system HIP 7.15 and hipBLASLt 1.4.1

In my latest session, the prompt/context grew from **41,709 to 75,100 tokens** across **54 completed requests**. The engine generated **18,806 tokens in 289.772 seconds of decode time**:

| Measurement | Result |

| Aggregate decode throughput, total tokens / total decode time | **64.9 tok/s** |

| Median throughput per request | **66.4 tok/s** |

| Per-request decode range | **56.1–82.8 tok/s** |

| Fresh 41,709-token prompt, zero reused tokens | **985.8 tok/s prefill** |

Most later turns reused the conversation prefix, so their prompt timings should not be treated as full-context prefill benchmarks. The expert cache was also warmed by the conversation.

Other fresh-prompt measurements with this setup were **912.9 tok/s at 6,731 tokens**, **1,004.6 tok/s at 76,599 tokens**, and **791.0 tok/s at 116,392 tokens**. I also had a slower 38,790-token read at **509.8 tok/s**. Across all 61 completed tail-skip-enabled requests in today's log, including the slower sessions and much longer generations, aggregate decode was **52.7 tok/s**. The ~65 figure refers specifically to the latest 54-request session.

The settings used for these runs:

--expert-cache auto
--prefill auto
--max-context 131072
--kv int8
--spec 4
--spec-min-p 0.70
--pcie-frac 0.35
--adapt-every 1
--adapt-swaps 160
--adapt-decay 0.97

MTP was enabled. The default CPU pool used 7 workers plus the host thread. The PCIe probe measured about 27.9 GB/s, but decode calibration selected `--pcie-frac 0.35` rather than the automatic 0.55. The engine loaded **39.97 GiB of expert weights into host RAM** and cached **5,048 experts / 8.19 GiB** in VRAM at startup.

I also used:

export STRATA_SH_STREAM=1
export STRATA_ROUTE_TAIL_SKIP=7
export STRATA_HIPBLASLT_TUNING="/path/to/strata-gfx1101-hipblaslt.txt"

The hipBLASLt table was generated locally for **gfx1101 + hipBLASLt 1.4.1**, with 26 rows. The log confirms that it was loaded. Those solution IDs are tied to that architecture and library version.

The project's **experimental speed projection** was also enabled at strength 1.0, layers 4–44, using `--cvec-mode project --cvec-dir per-layer`.

One informal comparison on the same accounting/Python prompt gave **50.8 tok/s without route tail skip** and **56.7 tok/s with it**. Both visible answers were correct, but the generated lengths and expert-cache hit rates differed, so this is not a controlled estimate of the switch's benefit. Tail skip changes the computation and can change the answer; I have not run a comprehensive quality benchmark.

I hit one watchdog stall during an earlier stock-config run. The latest 54-request session completed without a reported stall. This is a report of my own measurements, rather than a claim that these settings will be fastest or stable on every AMD setup.

Between the EU Chat Act and not wanting to hand my data to LLM vendors, I moved my team’s chat, our documents and the agents’ memory onto a local machine we run ourselves. We’ve been running it for about half a year now and couldn’t be happier, no issues or hiccups, so I wanted to write it up in case it helps someone doing the same.

We run Synapse, and the agents are just Matrix users in our rooms. Mention one and it replies, usually from a local model. Nothing’s exposed to the internet. Phones and laptops get in over Tailscale.

Every agent goes through a tool gateway. The rule is that an agent on a hosted model never gets a tool that reads local data. The gateway refuses to start if the config allows it, and checks again on every call. It also caps each agent’s tokens per day, which you’ll want once a hosted model is in the mix. Hosted models are opt-in, and one only sees the rooms you invite it to.

There’s also a scribe running in the background on a local model. Once a day it reads the rooms and files the day into a local Qdrant index, so agents can search old conversations without anything leaving the machine. It indexes the links people share but doesn’t fetch them unless you turn that on.

The repo is docs, specs and compose templates. We run our own builds of the gateway and the scribe, but the repo has them as specs only, so you can build your own or swap in something that exists. It’s Linux first. Only the homeserver chapter has been tested on a clean machine, and there’s nothing on backups or hardening. The README also lists the metadata that leaves anyway: device info on the overlay, certificate transparency logs, and phone notifications, which go through Apple’s and Google’s push services (they see that a message arrived, but not what it says).

https://github.com/nphardorworse/matrix-agent-mesh

Microsoft's Oct 7 "hybrid intelligence" announcement for Windows is mostly framed around Copilot getting access to local context, local actions and local models. For this sub, I think the developer side matters more. There are new experimental Windows ML capabilities for running models of your choosing locally, including through open-source tools, and GitHub Copilot is adding local/cloud routing plus explicit local-model selection.

One line from their own technical explanation stuck with me: "local inference does not make the session offline." Running the weights on your machine doesn't settle what the app around them sends out or depends on.

So here are the tests I'd apply before calling a setup actually yours:

1. Can you swap models without losing your accumulated knowledge and context?
2. Can you move to a different app and take your data with you?
3. Can you decide what leaves your device?
4. Can you plug into shared or networked tools without handing over your whole history?

Does anything on the Windows side pass these yet, or is a llama.cpp / Ollama / LM Studio stack still the only way to be sure? And if you already run local, does Windows ML change anything for you?

I wrote a longer piece on why I think Apple, Microsoft and Google are positioning for what comes after the model wars, and on a framework I call POND (Personal & Private, On-device & On-premise, Nodal & Networked, Distributed & Decentralized): https://www.thesingularityproject.ai/p/microsoft-moves-beyond-the-model

I built LlamaStash to scratch a personal itch: I run local models through llama.cpp on AMD Strix Halo and got tired of writing the same llama-server wrapper script for the tenth time.

Ollama and LM Studio both wrap llama.cpp but hide too much (and cost real performance). Raw llama-server is fast but tedious. LlamaStash is the middle ground.

What it does:

• llamastash init — first-run wizard. Detects your hardware (CUDA / ROCm-HIP / Metal / Vulkan / CPU), installs llama-server, scans your existing HuggingFace / Ollama / LM Studio model caches, recommends a GGUF that fits your VRAM, downloads it, writes a tuned config, smoke-launches it.
• TUI + CLI + daemon + OpenAI-compatible proxy in one Rust binary. The proxy at 127.0.0.1:11435/v1 lets OpenCode, Cline, the OpenAI SDKs, and llm-cli work as-is. There's also an opt-in --ollama-compat mode that takes port 11434 and answers the byte-exact "Ollama is running" handshake.
• Multi-model concurrency with per-model port allocation, /health-probed state machine, intelligent context auto-fit (sidesteps llama.cpp's --fit collapse on Linux iGPUs).
• Agent-friendly CLI: every TUI capability has a CLI subcommand, --json is a stable agent contract, documented exit codes per failure class.
• In-TUI HuggingFace browser with search, sort, paginate, per-file hardware fit, download with cancel.

On performance — this is the part that matters for this sub.

LlamaStash spawns the unmodified upstream llama-server. So the wrapper should add zero overhead. I measured it. Across AMD APU (Ryzen AI Max+ 395), Apple Silicon, and NVIDIA, on four model sizes (small E2B Q4, mid 31B Q4, large 27B Q8, large MoE 35B-A3B Q8), every cell matches raw llama-server within ≤1%.

Cross-tool numbers on AMD APU (decode tok/s / TTFT ms on chat_turn):

Tool	small	mid	large_dense	large_moe
LlamaStash	86.9 / 51	9.8 / 467	7.4 / 417	42.6 / 181
raw llama-server	86.0 / 51	9.9 / 468	7.4 / 414	42.7 / 186
LM Studio 2.16.0	91.1 / 187	11.6 / 1477	7.9 / 1274	37.0 / 683
Ollama 0.24.0	50.4 / 223	4.8 / 1092	2.6 / 1745	12.1 / 476

LM Studio wins decode on small/mid/large_dense (their Vulkan path is well-tuned on gfx1151) but loses on the MoE and pays a 1-1.5s TTFT tax from its OpenAI shim. Ollama is consistently slower, and its RAG prefill is catastrophic (cold prefill every rep — 4 min on a 31B). Mac and NVIDIA tables are in the benchmarks page.

Methodology, variance gates, fairness rules, and per-cell JSONs are all checked in. The harness is reproducible: make bench-end-to-end. Tear it apart.

What it's not:

• Not an Ollama fork or replacement (though --ollama-compat exists for tools that auto-detect Ollama).
• Not a model hub.
• Not a llama.cpp fork. Same upstream binary.
• Not a hosted service. Loopback-only in 0.0.2. LAN + auth + TLS are on the roadmap.

Install:

curl -fsSL https://llamastash.dev/install.sh | sh   # macOS + Linux one-shot
irm https://llamastash.dev/install.ps1 | iex        # Windows 11 (PowerShell, no admin)
scoop bucket add llamastash https://github.com/llamastash/scoop-llamastash && scoop install llamastash
brew install llamastash/llamastash/llamastash       # Homebrew (macOS + Linuxbrew)
yay -S llamastash                                   # Arch Linux (AUR — source build)
yay -S llamastash-bin                               # Arch Linux (AUR — prebuilt binary)
yay -S llamastash-git                               # Arch Linux (AUR — main checkout)
cargo install llamastash                            # any Rust toolchain

Then llamastash init and you're up.

Platform: Linux (x86_64, aarch64), macOS (Intel, Apple Silicon), Windows 11 (x86_64). aarch64-pc-windows-msvc and Windows AMD GPU detection on the roadmap.

Honest tradeoffs: Single-author project. Bug reports especially welcome on hardware I don't own. The OpenAI-compat surface covers chat/completions, embeddings, rerank; Anthropic /v1/messages shim is coming.

Repo: https://github.com/llamastash/llamastash

Blog post with the full story: https://deepu.tech/introducing-llamastash

Benchmark methodology: https://deepu.tech/benchmarking-llamastash

Happy to answer questions in the thread.

This whole project actually started with me just trying to get Strata working on my Intel Arc GPUs. I wasn't originally planning on making a fork or adding a bunch of features. I just wanted to get the engine running on my hardware.

But as I got things working and started benchmarking, I noticed something interesting. A single request was generating around 40–45 tokens/sec, while multiple concurrent requests could make use of a lot more of the server's processing capacity.

That got me thinking. Since I'm the only one using my server, why not try putting those extra batch slots to work on a single complicated request?

That's basically how Strata Void started. It's an experimental fork of Niko1221's Strata, with a feature I've been working on called Task-Parallel Requests.

The idea is pretty simple. Instead of having the model work through one big question sequentially, it can break suitable requests into smaller independent tasks, process them concurrently using the same loaded model, and then combine everything into one final response.

You can enable it with "task_parallel": "auto". Simple questions go through normally, while more complicated requests can be split up if the planner thinks it would help.

The results so far

I ran 10 different complex tasks with a 32K context window, using cold runs with one test per task and mode.

• Normal request: 181 seconds median
• Four parallel subtasks + synthesis: 68 seconds median
• AUTO mode: 78 seconds median

The important distinction is that this doesn't actually increase single-stream token generation speed. That's still around 40–45 tok/s on my setup. What improves is the total time it takes to finish a complicated request.

For reference, my setup is:

• Intel Arc Pro B70 (32GB) + B65 (32GB) + B60 (24GB)
• 88GB total VRAM
• Ryzen 9 9950X / 64GB DDR5
• Ubuntu Server 26.04
• Swift 1.5 Qwen3.8 Flash-Next, IQ4_XS

All of the model's expert weights are loaded into VRAM, so there's no RAM spilling involved in these benchmarks.

I've also been experimenting with faster prompt processing on Intel Arc, shared-context caching between subtasks, and larger context configurations up to 128K.

A few caveats

This isn't some magical free performance boost. Splitting a request means additional planning, processing, and synthesis, so it uses more total compute. It's also not useful for every prompt, and I've seen cases where parallel tasks make mistakes or disagree on numerical results.

The benchmarks are preliminary and all come from one machine and model, so I'm definitely not claiming everyone will see the same improvements.

Also, credit where it's due: the underlying engine comes from Niko1221 and the Strata contributors. I worked on the direction of this fork and its evaluation, with Claude assisting heavily with implementation, debugging, and benchmarking. Task parallelism itself isn't a new concept; I wanted to see how well it could work integrated directly into Strata.

I'm sharing this because I'd genuinely love to see how it performs on other hardware, especially different Intel Arc configurations or NVIDIA CUDA setups.

If anyone feels like experimenting with it, I'd love to hear what works, what breaks, or whether the performance improvements hold up on your system. Contributions and bug reports are welcome too.

GitHub: https://github.com/Jumbomuffin777/Strata-Void

The repository has the setup instructions, benchmark methodology, and raw results if anyone wants to dig into the details.

Flash-Next Q4, q8 kv, 262k context

4x P100 on PCIe3 x8 slots (4x16GB=64GB total)

128GB DDR4 at 2133MHz

E5-2683 v4 (16c/32t at 2.6GHz boost)

On average it's about twice as fast as 27B and five times faster than Flash-Next using a customized llama.cpp just to get it to load at all.

512K context on a single RTX 4090 within the low budget RAM

total vram + sys ram = 50GB

Active strata contributor, working k8v4. Ask whatever you want

Stack

• Strata NVFP4 fork: github.com/sergqwer/strata-nvfp4
• Model: jpezzulli/OrcaRouter-Qwen3.8-Flash-Next-Uncensored-ModelOpt-NVFP4 Hugging Face model
• NVFP4 routed experts (~63.3 GiB), separate FP8 PLE, INT8 KV cache, MTP speculative decoding
• W4A8 prefill on Blackwell

Main engine flags:

./build/strata \
  --pack packs/orca-nvfp4 \
  --native models/orca-nvfp4.gguf \
  --native-dense-gguf models/orca-nvfp4.gguf \
  --ple-gguf models/ple-fp8.gguf \
  --mtp mtp-orca/rt \
  --spec 4 --spec-min-p 0.5 \
  --prefill auto \
  --expert-profile data/expert-profile.bin \
  --expert-cache auto \
  --resident-budget-gib 40 \
  --max-context 200000 \
  --kv int8

I serve it through Strata's OpenAI-compatible server.

Results so far:

• ~50k context: up to ~80 tok/s
• ~188k warm context: ~60–67 tok/s
• cold 189k full prompt: ~1,680 tok/s prefill, ~53 tok/s decode

Pretty impressive for a single 32GB GPU + only 64GB system RAM.Running Qwen3.8-Flash-Next NVFP4 with Strata on a single RTX PRO 4500 32GB + 64GB DDR5.StackStrata NVFP4 fork: github.com/sergqwer/strata-nvfp4

Model: jpezzulli/OrcaRouter-Qwen3.8-Flash-Next-Uncensored-ModelOpt-NVFP4 Hugging Face model

NVFP4 routed experts (~63.3 GiB), separate FP8 PLE, INT8 KV cache, MTP speculative decoding

W4A8 prefill on BlackwellMain engine flags:./build/strata \
--pack packs/orca-nvfp4 \
--native models/orca-nvfp4.gguf \
--native-dense-gguf models/orca-nvfp4.gguf \
--ple-gguf models/ple-fp8.gguf \
--mtp mtp-orca/rt \
--spec 4 --spec-min-p 0.5 \
--prefill auto \
--expert-profile data/expert-profile.bin \
--expert-cache auto \
--resident-budget-gib 40 \
--max-context 200000 \
--kv int8I serve it through Strata's OpenAI-compatible server.Results so far:~50k context: up to ~80 tok/s

~188k warm context: ~60–67 tok/s

cold 189k full prompt: ~1,680 tok/s prefill, ~53 tok/s decodePretty impressive for a single 32GB GPU + only 64GB system RAM.

mrdefaultuser@team-green:~$ /srv/ai/llama.cpp/build/bin/llama-server \
 -m /srv/ai/models/qwen/Qwen3.8-Flash-Next-Q8_0/Qwen3.8-Flash-Next-Q8_0-00001-of-00002.gguf \
 -c 262144 \
 -np 1 \
 -ngl 64 \
 --cpu-moe \
 --lazy-mode on \
 --load-mode auto \
 --agent \
 --tools all \
 --host 0.0.0.0 \
 --port 8080

Every 2.0s: free -h; echo; nvidia-smi --query-gpu=memory.used,memory.total,utilization.gpu,power.draw,temperature.gpu --format=csv,noheader                                                                                                                                                                                          team-green: Sun Oct  4 16:20:08 2026

total        used        free      shared  buff/cache   available
Mem:           125Gi       3.5Gi       855Mi       416Mi       122Gi       122Gi
Swap:          127Gi       3.5Gi       124Gi

14206 MiB, 32768 MiB, 48 %, 95.85 W, 49

prompt processing ~52 tokens per second
generation ~10 tokens per second