u/Specific_Hand_7474

Hi everyone,

I'm trying to run the Muse-Glimmer model locally on Apple Silicon using the MLX framework for an agentic workflow. However, I’ve hit a massive roadblock: persistent KV caching is not working.

As a result, every single new follow-up request from the agent triggers a full prompt re-evaluation. On longer context windows, this makes the performance unusable.

Here is what I have already tried and the results:

1. LM Studio: No luck. I couldn't find a compatible model format/architecture that would even boot up there.
2. ExecuTorch (Meta's official method): It runs, but KV caching is missing. Every single step forces a full context recalculation.
3. mlx-vlm (standard run): The model starts up fine, but context caching does not work.
4. llama.cpp: The only framework where everything works perfectly and the KV cache is properly retained. However, there is a massive downside — the prompt processing speed is painfully slow compared to native MLX performance.
5. mtplx: Didn't even bother trying, as I previously failed to get KV caching working on this stack even for standard Qwen models.

My question to the community:
Has anyone successfully managed to get KV caching working for Muse-Glimmer on pure MLX ormlx-vlm?

Any code snippets, fork links, or ideas would be greatly appreciated!