Login

u/peonist-ai

r/ROCm r/StrixHalo
Image 1 — halogen-flash-server 0.16.0: Strix Halo NPU now serves embeddings, reranking and classifiers beside Qwen3.8-Flash-Next. Nearly 10,000 tok/s
Image 2 — halogen-flash-server 0.16.0: Strix Halo NPU now serves embeddings, reranking and classifiers beside Qwen3.8-Flash-Next. Nearly 10,000 tok/s
Image 3 — halogen-flash-server 0.16.0: Strix Halo NPU now serves embeddings, reranking and classifiers beside Qwen3.8-Flash-Next. Nearly 10,000 tok/s
Image 4 — halogen-flash-server 0.16.0: Strix Halo NPU now serves embeddings, reranking and classifiers beside Qwen3.8-Flash-Next. Nearly 10,000 tok/s
▲ 175 r/ROCm+1 crossposts

halogen-flash-server 0.16.0: Strix Halo NPU now serves embeddings, reranking and classifiers beside Qwen3.8-Flash-Next. Nearly 10,000 tok/s

I want to thank everyone for the kind words, trust, and encouragement in the development of this project. I am doing my best to make the halogen UX amazing for you. I also want to thank AMD's Jeremy Fowers. u/jfowers_amd has been nothing but supportive and sent me a development Strix Halo machine which has been taking a beating over the last couple weeks.

halogen-flash-server 0.16.0 is out.

It runs Qwen3.8-Flash-Next on the Strix Halo GPU, and this release puts the chip's other accelerator to work too.

The Ryzen AI NPU now serves small models beside the Flash model, on the same port, while the GPU keeps generating.

What's new

Small models on the NPU, with one variable. Set HALOGEN_NPU_MODELS and the server runs them on the NPU. Leave it out and nothing changes.

  • decider-0.8b makes decisions. Give it a text, a question and 2 to 10 options, and it returns a probability for every option from one pass.
    • Use it as a guard, a router or a classifier.
  • qwen3-embedding-0.6b answers `/v1/embeddings`, in the OpenAI request shape.
  • qwen3-reranker-0.6b answers `/v1/rerank`, in the Cohere and Jina request shape.

Your own fine-tunes. Put a fine-tune of any of the three on the models volume and name its path. The server converts it for the NPU at the first start and serves it under the directory's name.

A smaller checkpoint, opt-in. qwen38-flash-next-ht43.hgn pins about 8 GiB less memory than the default checkpoint. Prefill is several percent slower, and decode with the default drafter is a few percent slower. The default checkpoint does not change. Select it with HALOGEN_CHECKPOINT=/models/qwen38-flash-next-ht43.hgn.

96 GB: On a 96 GB machine, run ht43 with HALOGEN_MAX_TOK=16384 and HALOGEN_CTX=131072. That leaves room for one conversation of 131k tokens or four of 32k.

Fixes. A long prefill can be cancelled now (#122, thanks u/semidark). The draft counters count every drafted token (#124, thanks u/wszgrcy).

How fast is the NPU?

  • A decision with Qwen3.5-0.8B (the architecture `decider-0.8b` uses) takes 49 ms at 64 tokens, 84 ms at 256 and 183 ms at 1k. Eight decisions of 256 tokens in one pass take 41 ms each.
  • Batched embeddings run at about 9,400 tokens a second through the server, eight texts of 512 tokens at a time. One text of 512 takes about 97 ms.
  • The reranker scores 19 pairs a second, eight pairs of 512 tokens at a time.

The NPU and the GPU together

Something worth knowing even if you never run halogen. On Strix Halo, GPU work and NPU work at the same time can hang the whole machine while the GPU's fabric clock changes speed. Holding that clock at its top level stops it, at a cost of about a watt. The server holds it itself when it runs as root with /sys mounted. For rootless podman, a small systemd unit in the repo holds it at boot. If you run an NPU program next to a GPU program and have seen freezes, this may be why.

The Flash model runs somewhat slower while the NPU works, since the two share the memory and the power budget. With the NPU idle there is no cost.

Setup

The host needs the amdxdna driver, XRT with its NPU plugin, the NPU firmware, and the IOMMU on. If you added amd_iommu=off for GPU speed, it also turns the NPU off. iommu=ptkeeps it. The full setup, every request shape and the fine-tune rules are in https://github.com/peonist-ai/halogen-flash-server/blob/main/docs/NPU.md

podman pull ghcr.io/peonist-ai/halogen-flash-server:0.16.0

Repo: https://github.com/peonist-ai/halogen-flash-server

Weights: https://huggingface.co/peonist-ai/halogen-qwen3.8-flash-next

Discord: https://discord.gg/bcm6QknaV6

How to read this. FastFlowLM's numbers are from its own published benchmark pages. They were measured on a different chip (Ryzen AI 7 350) with older versions than its current one, in its default power mode. FastFlowLM lists no embedding model, so its Qwen3-0.6B row (the same backbone) stands in for it. Ours are medians from our own testing on Ryzen AI Max+ 395 machines, and where we measured two machines, the slower one is shown. This is not a same-machine test. If you run both on one machine, we would like to see your numbers.

u/peonist-ai — 13 days ago