LLMs Gateway – Model Management for llama.cpp
LLMs Gateway – Model Management for llama.cpp
I got tired of manually juggling GGUF downloads, symlinks, and llama-server restarts every time I wanted to swap models. So I built LLMs Gateway – a lightweight CLI + REST API that sits on top of llama.cpp and handles the entire model lifecycle.
What My Project Does
LLMs Gateway simplifies local LLM management by providing a single interface for discovering, installing, validating, activating, and serving GGUF models.
Features:
- Search Hugging Face repositories directly from the CLI
- Inspect model metadata before downloading
- Download and install GGUF models with a single command
- Maintain a local JSON-based model registry
- Validate downloaded files using hashes
- Activate models through symlink switching
- Automatically restart llama-server when a model changes
- Expose all functionality through both a CLI and REST API
Example workflow:
docker compose up -d
modelctl search llama
modelctl inspect unsloth/gemma-4-E2B-it-qat-GGUF
modelctl install unsloth/gemma-4-E2B-it-qat-GGUF model.gguf
modelctl activate <model-id>
Once activated, llama-server automatically picks up the new model without manual intervention.
Target Audience
LLMs Gateway is designed for:
- Developers running local LLMs with llama.cpp
- Self-hosted AI enthusiasts
- Homelab users
- Teams building local AI services or internal tooling
- Anyone managing multiple GGUF models on a single machine
The project is intended to be production-capable for small to medium deployments while remaining lightweight enough for personal use.
Comparison
Unlike tools such as Ollama that manage their own model ecosystem and runtime, LLMs Gateway focuses on model lifecycle management for llama.cpp.
Key differences:
- Works directly with GGUF repositories from Hugging Face
- Keeps a transparent local JSON registry instead of a hidden database
- Provides explicit control over installed artifacts
- Uses symlink-based activation to switch models
- Integrates directly with existing llama.cpp deployments
- Combines model management and serving orchestration in a single workflow
The goal is not to replace llama.cpp, but to make operating multiple local models on top of llama.cpp significantly easier.
Architecture
Stack:
- Python monorepo (uv workspace)
- FastAPI
- llama.cpp
- Single Docker image
Two services, one image.
The coolest part is the container entrypoint. It watches for model activation changes and seamlessly restarts llama-server with the selected weights. No manual process management, no PID hunting, and no server reconfiguration.
GitHub: https://github.com/regisx001/llms-gateway
I'm interested in hearing how others manage local models today. Are you using symlinks, Ollama, custom scripts, or something else?