slotpin: cache-maxing scheduler and dashboard for llama-server and Gufo
I've been running an always-on assistant at home on a couple of 128GB Strix Halo boxes for a few months, and struggled with getting solid cache hits. I tried two serving slots, with one for interactive and one for background, but didn't have a reliable way to route requests.
I ended up with a little proxy that sits in front of llama-server and deals with that. This then grew to a request scheduler, metrics dashboard and now added support for the Gufo engine (a newer engine aimed at Strix Halo: https://github.com/gufo-org/gufo ) too! It's currently running on both my boxes, one with a single-slot Gufo/Qwen3.8 Flash-Next and one with a four-slot Gufo/Qwen3.8 27b.
I've just tidied it up and made it public in case it's useful to anyone else: https://github.com/headbouyJB/slotpin
What it does:
- With llama-server: Pins each conversation to a slot, so the next turn lands where its cache already is.
- Sorts requests into lanes with regexes (my chat messages vs cron jobs and other background stuff), so the interactive lane is protected and background work waits its turn without being starved.
- With llama-server: Saves the interactive slot's KV to disk and restores it if something evicted it. On my box a returning conversation went from an 88 second cold turn to 0.7 seconds to restore, in a direct test of exactly that case.
- Writes one line of metrics per turn: time to first token, decode and prefill speed, cache hit, queue wait, memory, and optionally energy from a smart plug, so you get Wh per request. No prompt or reply text is recorded anywhere.
- Has a dashboard (screenshot below) with live slots, performance, cache, health and system views, plus a list of recent turns you can filter down to the slow ones or the cache misses.
- Works in front of Gufo as well as llama.cpp. Gufo does its own caching, so in that mode the proxy steps back and just does lanes, scheduling and metrics.
- Has an optional "fail fast" mode. I had an engine stall for 55 hours while still answering health checks and sending keep-alive pings, so my client never fell back to anything else. The proxy can now spot "no output and no token progress" and return a proper error instead of hanging. That one’s brand new... I’m only just turning it on myself!
Fair warning:
- It's alpha. Config keys and metric fields might still change.
- It's been run on exactly two machines, both mine, both Strix Halo on Linux. The test suite is decent and CI is green, but I'd expect rough edges on other setups.
- There's no authentication. It listens on localhost by default and the README says how to expose it sensibly.
- I'm not a professional developer. Most of the code was written with Claude, with me testing it on real hardware and my real serving environment.
If you just want to see the dashboard without setting anything up, there's a demo mode with made-up data:
git clone https://github.com/headbouyJB/slotpin && cd slotpin
uv run python tools/demo_dashboard.py
What I’m curious about is what everyone else puts between their client and their server. Are you scheduling or prioritising requests, routing different kinds of work to different slots or models, or just letting everything hit llama-server and living with it? That middle layer is where most of my pain was, and I can’t tell if I’ve solved a common problem or just my own.