Cachy-Router an OpenAI-compatible router for llama.cpp with shared KV-cache
Disclaimer: all code was written by Codex, I am not a coder so please don't be too harsh. This post is written 100% by me. I don't know how it works and this is the first thing I have ever put on GitHub. It isn't perfect but it actually works for me. The performance of how much less time work was being done got me so excited I wanted to share it.
https://github.com/JCFrags/Cachy-Router
The problem I wanted solved was making effective use of multiple Strix Halo PCs, running llama.cpp on both meant entering 2 APIs into an agent and effective coordination of the machines for sub-agent use was a nightmare. It also meant that agent id is set to a specific PC creating idle time waiting for the right one to become available or forcing long prompt processing.
The inspiration came from seeing CachyLLama ( https://github.com/fewtarius/CachyLLama#MIT-1-ov-file ) saving KV-cache to disk and I had a thought that files on disk can be shared between computers.
CachyLLama is a fork of llama.cpp, this tries to imitate the caching ideas while being an independent router layer so you can bring any version of llama.cpp that you want.
Strix Halo is great but if you have a large system prompt it can easily take 1-2 minutes just to say "Hi", now it is processed once and future messages begin generating almost immediately. That change alone felt magical. Now with this if you are using sub-agents it doesn't matter which one finishes first because the KV-cache is shared between them.
Cachy-Router is designed to reduce idle time on a multi PC setup when work is queued and focus on token generation instead of re-processing the same things over and over.
I had Codex SSH into the Strix Halo PCs to optimize and set everything up. Now I am running Step-3.7-Flash Q4_K_S with MTP at sustained 32 t/s and now with Cachy-Router it actually feels usable and competitive with mid-range cloud options.