I made an open-source tool that shrinks your LLM context before it hits the model — feedback and contributors wanted
I've been working on SlimToken, an open-source token compressor that minifies your LLM context before it's sent to the model, so you fit more into your context window and pay for fewer tokens. It started as a fix for hitting context limits and ballooning token costs, and it's in a state I'm happy to share. It's MIT licensed at github.com/greyok00/slimtoken, and it ships with orjson, xxhash, and tiktoken so its token counts are real (cl100k), not guesses.
Three ways to run it (same code, same wins)
- Proxy (default) — an always-on server in front of Anthropic / OpenAI / Ollama. slimtoken install wires ANTHROPIC_BASE_URL to it, so every request is minified automatically and reversibly (slimtoken uninstall restores everything).
- MCP server — exposes the pipeline as 8 tools any MCP agent can call on demand.
- Agent Skill + CLI + library — a packaged skill for Claude Code / Codex / OpenCode, a slimtoken optimize CLI, and import slimtoken as a plain Python library. The always-on minify pipeline (runs on every request by default)
- 🧰 tools — drops $comment/title/examples from schemas, keeps structure and enums, compresses descriptions to their first example.
- 📋 system — collapses repeated banner lines and whitespace outside code fences; preserves <tag> markers and fenced code byte-for-byte.
- 💬 messages — collapses blank-line runs and trailing whitespace in text; passes tool/image blocks through untouched.
- 🔄 dedup — collapses repeated tool_result contents; latest copy kept verbatim, older copies stubbed.
- 📝 distill — truncates old assistant prose beyond the last N turns (fence-aware, no model call).
- 🎯 budget — a hard token cap that drops a leading prefix pair-safely when you're over.
- 🌐 DOM prune (opt-in) — strips script/style/svg, nav/footer, and layout attrs from large HTML tool results.
- 🗜️ tool_compress — type-specific reduction of big tool results (directory listings, git output, logs, JSON, source).
Safety guarantees: fenced code blocks preserved byte-identical, pruning is pair-safe (a tool_result is never orphaned from its tool_use), unchanged content is returned zero-copy, and the grammar field is stripped.
Output filter (filler-strip on by default) — drops lead-in filler like "Sure!" / "Here is the code:" from the streamed head, plus opt-in output token cap and stop sequences, all applied to the stream so a runaway completion is cut off at the source.
Prompt reframe — when the user prompt itself is the problem: a pure-CPU (~1 ms) rewriter with no model roundtrip that turns a rambling 200-word request into a tight ~25-word instruction while preserving intent by construction. Five stages (classify_domain → reframe → shrink → minify → build_system), available as Python API, CLI, and an MCP server.
Measured savings, not marketing — slimtoken presets --measure recomputes the tables on your machine. On a bloated coding session the pipeline drops ~57–64% of tokens; on HTML dumps ~83%. The honest caveat is in the README: a clean short session gets ~0% — it removes redundancy, never invents savings.
Backends — Anthropic (identity), OpenAI, and Ollama, with bodies normalized to a canonical form so no optimization logic is duplicated. Pair-safety holds across the round trip.
One config, no profiles — the full pipeline is always on; every stage and knob is a raw SLIMTOKEN_* env switch (turn off a single lossy stage, or SLIMTOKEN_MINIFY=0 for raw passthrough).
Local-model helpers
- VRAM presets (4/8/16 GB) for common local models, each with a usable-context recommendation and live-measured reduction.
- High-context dense + MoE table — effective raw-token capacity after compression (e.g. ~898k effective on a 128k MoE row), because compression effectively multiplies your context window.
- Config optimizer — inspects your GPU VRAM and model size and recommends ready-to-paste llama-server flags (full offload, flash attention, KV-quant, ubatch, --kv-unified) that typically
give 2–4× decode speedup and ~2× context capacity.
Observability — stats.json and a /metrics endpoint with cumulative token counts and per-request latency buckets, so you can see exactly what you're saving.
What I'm asking for:
- Recommendations — which backends or integrations would make you reach for it? Better compression heuristics, a web UI, more model families?
- Contributors — it's a solo project; help with testing across backends, packaging, docs, and new compressor strategies is very welcome. Issues, PRs, and "this is over-engineered, just do X" are all gold.
Even a quick honest "here's where it breaks" is hugely useful. Thanks for reading! 🙌