u/gvij

Sharing a CPU TTS benchmark with objective MOS scores in case it's useful for anyone evaluating small TTS models. Adding this because Kyutai's Pocket TTS is architecturally different from the others in the field and I hadn't seen a head-to-head with it yet.

Models:

• Kokoro 82M (PyTorch and ONNX Runtime, StyleTTS2-inspired)
• Supertonic 3 at 2 and 5 flow-matching steps (Vector Estimator backbone)
• Inflect-Nano-v1 (4.6M param FastSpeech-style, tiny end of the spectrum)
• Pocket TTS (~100M param streaming LM over Kyutai's Mimi neural audio codec)

Setup: Intel Xeon 8272CL, 4 cores, 15.6GB RAM. CUDA disabled at env level. ONNX sessions pinned to CPUExecutionProvider. Six configs, six text lengths (12 to 1712 chars), five timed reps per cell after a discarded warmup. 180 total runs. Every saved WAV scored with UTMOS (utmos22_strong) for objective MOS.

Aggregate results:

Findings I think are actually interesting:

1. Streaming LM architecture produces flat RTF scaling. Pocket TTS's RTF is 0.69 to 0.76 across the entire text length range. Because it emits audio tokens autoregressively at a steady rate, cost is linear in output length with no fixed overhead to amortize. Compare to Kokoro PyTorch, which climbs from 0.49 on tiny to 0.83 on long inputs, or Supertonic which goes the other way (0.36 on tiny down to 0.20 on medium) because of high per-call fixed overhead. If you're budgeting worst-case latency for an interactive system, flat is worth a lot.

2. UTMOS has a known failure mode on small vocoders. Inflect-Nano-v1 scored 3.48, which reads mid-pack. By ear it's buzzy and robotic. This is a documented issue: UTMOS rewards HiFi-GAN outputs for being clean even when they lack prosodic naturalness. Pocket TTS scored similarly (4.10) but sounds legitimately natural. The point isn't that UTMOS is broken, it's that a single quality number can't distinguish "clean and mechanical" from "clean and natural" on small models. Worth pairing with human listening or a naturalness-specific metric like NISQA.

3. Inflect-Nano has an undocumented ~15s output cap. The model config sets max_frames = 1400, which caps synthesis at ~14.93s regardless of input text length. Its RTF and throughput on long/paragraph/extended inputs are inflated because it's doing less work than the models it's compared against. Real comparison for that model is on tiny/short/medium only.

4. Kokoro ONNX vs PyTorch results reverse from the previous run. I ran an earlier version of this benchmark on AMD EPYC and PyTorch beat ONNX in aggregate. On this Xeon, ONNX is faster (0.641 vs 0.665). Same code, different silicon. AMD vs Intel kernel optimization differences at CPU inference are apparently real enough to flip the ranking. If anyone has replicated this on ARM I'd be curious.

Zero-shot voice cloning as a capability that doesn't fit the benchmark axes:

Pocket TTS can clone a voice from ~5 seconds of reference audio, zero-shot, on CPU. No other model in this field does this. I pinned it to a preset voice for the speed/quality comparison to be fair, so the cloning capability isn't reflected in the numbers. This is a real limitation of RTF-and-MOS-based comparisons: they can't capture capabilities that only one model has. Might want a separate speaker-similarity evaluation for a v2.

Limitations:

• Single hardware platform
• English only
• UTMOS is one MOS predictor; NISQA or a listening panel would strengthen the quality claims
• Voice cloning quality was not evaluated
• No batched inference tested

Disclosure: The benchmark harness was written by an AI engineering agent (Neo) from a prompt I specified. I chose the methodology, validated the outputs, and reviewed the audio. Mentioning it because it's relevant to how you'd want to weight the code.

All code, raw CSVs (180 rows), MOS CSV (36 rows), and WAV samples are in the repo mentioned in the comments below 👇

Feedback on the protocol welcome, especially on the MOS methodology and what a proper voice-cloning eval would look like.

If you're building agents on small or on-device models, this one's relevant: I measured how long three edge models hold a single fact as the conversation grows, and the way they fail is worse for agents than plain forgetting.

Setup was simple on purpose: inject one fact, pile on N turns of unrelated filler, ask for the fact. Three runs per depth, shuffled filler each time.

The failure mode: when an agent loses the fact, it doesn't guess wrong. It asserts it could never have known, "I don't have access to your personal information." But the fact is still sitting in context. For an agent that's supposed to carry user state across a session, this means it won't just drop a constraint, it'll confidently tell the user the constraint was never given. That breaks trust and it's painful to trace, because nothing actually errored.

The numbers, short version:

• LFM2.5 (1.5B active MoE): longest memory, degrades gradually.
• Gemma 4 E2B (~2B): solid then a sudden cliff around 8-10 turns.
• Gemma 4 E4B (~4B): shortest memory of the three, breaks at 5 turns, but the strongest at instruction-following and keeping tool-call formats intact.

That last split is the interesting tension for agent builders. The model best at not breaking your tool schema was the worst at remembering what the user said. If memory and format-discipline really do trade off, you may want one model driving structured tool calls and a separate mechanism (retrieval, refreshed system state) holding the facts, rather than expecting one small model to do both.

Writeup with the full chart, per-depth breakdown, and the reproducible harness. Link in the comments below.

Curious if anyone running agent frameworks has hit the "you never told me" refusal in the wild, and how you worked around it.

A methodological finding from a recent benchmark that might be useful for others building ASR evaluation pipelines.

We evaluated nvidia/parakeet-tdt-0.6b-v3 on CPU-only hardware using Harvard sentences as reference text, with two different TTS generators to produce the test audio. The WER difference between them was 20.9% vs 4.65% — on the same model, same weights, same reference text.

espeak-ng produced robotic synthetic speech that mispronounced several words outside typical English phoneme patterns: "zest", "zestful", and "tacos al pastor". These errors were consistent across both inference backends we tested (HF Transformers bfloat16 and ONNX Runtime FP32), confirming the confound is in the audio generator rather than the model.

gTTS produced more natural prosody and pronunciation, bringing WER to 4.65% — consistent with NVIDIA's reported performance on natural speech corpora.

This is a known issue in the ASR evaluation literature but easy to overlook in practice when you reach for espeak-ng because it's offline and dependency-free. The cleaner approach is to treat TTS source as an explicit variable in your evaluation design and report it alongside your WER numbers.

For this benchmark, inference path also mattered: ONNX Runtime FP32 ran at RTF 0.328 vs HF Transformers bfloat16 at 0.519 on 2 CPU cores — a 37% throughput difference attributable to operator fusion in the ONNX execution provider.

Full methodology, scripts, and raw results link in comments below.

Disclosure: this benchmark was run using Neo, a local AI engineering agent inside Claude Code via MCP. The TTS source selection and runtime choice came from its pre-execution research phase.

Quick practical finding for anyone deploying transformer-based ASR models on CPU without a GPU.

Benchmarked nvidia/parakeet-tdt-0.6b-v3 (FastConformer-TDT, 0.6B params) on a 2-core CPU box (AVX2/FMA, 7.7GB RAM) across three inference paths:

The 37% RTF gap between ONNX and HF Transformers on CPU comes down to a few things: ONNX Runtime's execution provider uses operator fusion that collapses attention + layer norm + activation sequences into single optimized kernels, and its CPU backend is more aggressive about using AVX2/FMA intrinsics than PyTorch's generic CPU path. The FP32 vs bfloat16 precision difference goes against ONNX here — it should be slower — which makes the RTF advantage more meaningful.

GGUF Q6_K via parakeet.cpp is compute-bound (99.8% CPU) rather than memory-bound, which explains why it's slower despite the quantization reducing model size. The 6-bit dequantization overhead on every matmul adds up without the kernel fusion that ONNX Runtime provides.

Memory tradeoff is real: ONNX FP32 peaks at 2.7GB, GGUF Q6_K at 928MB. For edge deployment or memory-constrained inference, GGUF wins on footprint. For sustained throughput on a box with available RAM, ONNX is faster and leaves 50% CPU headroom for concurrent workloads.

Also worth noting: test audio quality had a larger effect on WER than runtime choice. espeak-ng inflated WER to 20.9% on inputs where gTTS got 4.65% — both runtimes got identical WER within each run, isolating the audio generator as the variable.

Repo with scripts, raw JSON results, and evaluation setup link in comments below.

Disclosure: this benchmark was run using Neo, a local AI engineering agent inside Claude Code via MCP. The ONNX runtime choice and audio selection came from its pre-execution research phase rather than prior knowledge on my end.

Sharing a small CPU inference benchmark for nvidia/parakeet-tdt-0.6b-v3 that turned up a result I didn't expect going in.

Setup: 2 x86-64 vCPUs (AVX2/FMA), 7.7GB RAM, no GPU. Test audio: 16.78s Harvard sentences at 16kHz mono.

Results:

ONNX Runtime is 37% faster than HF Transformers bfloat16 on this hardware. The gap comes from operator fusion and AVX2-optimized execution providers in ONNX Runtime that the PyTorch CPU path doesn't exploit as aggressively. Memory cost is the tradeoff — FP32 weights load at ~2.7GB peak.

GGUF Q6_K trades throughput for memory efficiency. 928MB peak vs 2.7GB, but RTF doubles and CPU utilization hits 99.8%. For memory-constrained deployments it's the right call. For sustained throughput on a box with headroom, ONNX wins.

One methodological note worth flagging for anyone doing ASR benchmarking with synthetic audio: espeak-ng inflated WER to 20.9% on a sentence set where gTTS got 4.65%. Both runtimes got identical WER within each run, confirming it's the TTS distribution mismatch rather than model or quantization quality. NVIDIA reports 1.93% on LibriSpeech — the gTTS number is a much more honest CPU-only proxy.

Github repo with code, raw results, and evaluation scripts in comments below.

Disclosure: benchmark was run using Neo, a local AI engineering agent inside Claude Code using its MCP. Mentioning because the runtime and audio choices came from its research phase, not prior knowledge on my end.

Did a CPU-only evaluation of nvidia/parakeet-tdt-0.6b-v3 and ran into two things worth sharing for anyone building ASR evaluation pipelines.

Hardware: 2 x86-64 vCPUs (AVX2/FMA), 7.7GB RAM, no GPU.

Finding 1: ONNX Runtime is significantly faster than HF Transformers on CPU

ONNX Runtime runs at RTF 0.328 vs 0.519 for the HF Transformers path — 37% faster on identical hardware. Operator fusion and AVX2-optimized kernels make a real difference when there's no GPU to absorb the slack. The tradeoff is RAM: ONNX FP32 peaks at ~2.7GB loading full weights.

GGUF Q6_K is the right call if you're memory-constrained — 928MB peak, nearly identical accuracy — but it pegs both CPU cores at 99.8% and runs at roughly 2x the RTF of ONNX.

Finding 2: espeak-ng is a bad choice for ASR benchmarking

This one cost me a run. Using espeak-ng as the TTS source for test audio inflated WER to 20.9% on Harvard sentences that should be straightforward for this model. NVIDIA reports 1.93% WER on LibriSpeech. The gap is not the model.

espeak-ng mispronounces words like "zest", "zestful", and "tacos al pastor" in ways that sit far outside Parakeet's training distribution. Both inference backends got identical WER within the same run — confirming it's the audio generator, not the runtime.

Switching to gTTS brought WER to 4.65% on the same reference text. Still not LibriSpeech quality but a much more honest proxy for real speech. For CPU benchmarking where you're generating synthetic test audio, gTTS is worth the extra step.

Repo with scripts, raw JSON results, and evaluation setup link in comments below.

Curious if others have run into the espeak-ng WER inflation issue or found better synthetic audio options for ASR eval.

Disclosure: this benchmark was run using Neo, an AI engineering agent that runs locally inside Claude Code via MCP. The ONNX and gTTS decisions came out of its pre-execution research phase rather than from my own upfront knowledge - worth mentioning since it affected the methodology.

I went and read three of the trending "skills" repos for Claude Code looking specifically at what's prompt-engineering-novel inside them.

The repos:

• forrestchang/andrej-karpathy-skills (~70k stars): one CLAUDE.md, four behavioral rules, derived from a Karpathy tweet
• mattpocock/skills (~115k stars): ~10 single-purpose SKILL.md files
• affaan-m/everything-claude-code (~175k stars): 182 SKILL.md files plus 48 agent definitions, hooks, rules, MCP configs

What's actually in them, prompt-wise:

The karpathy file is four imperative behavioral rules. No CoT scaffolding, no few-shot examples, no role definition, no structured output spec. Just declarative principles like "don't make silent assumptions, surface inconsistencies, present tradeoffs." It works because the model is now good enough that imperative behavioral instructions stick. Five years ago this would have been a non-starter and we'd have written elaborate few-shot examples. Now four sentences gets you 70k stars.

mattpocock's skills are procedural workflows in markdown prose. The tdd skill walks through red-green-refactor steps. to-issues describes how to slice a plan into independently-grabbable units. There's YAML frontmatter declaring when each skill should auto-fire, but the body is essentially what you'd write as a system prompt section, just modularized into discrete files.

ECC's skills look similar at the unit level but the system around them does more. YAML frontmatter with evaluation criteria, "instinct" files that track confidence scores per pattern, hooks that auto-extract patterns from sessions into new skills. Some of this is prompt-engineering-adjacent infrastructure (session memory, context-window management) rather than prompting per se.

So is this prompt engineering or packaging?

Honest answer, mostly packaging, with one real prompting innovation worth naming.

The packaging story is obvious. SKILL.md gave the community a unit of distribution. You can publish, fork, version, install. That makes prompts shippable in a way they weren't before, and that alone explains most of the trending list. None of these repos invented a new prompting technique.

The technique worth naming is conditional injection via frontmatter description matching. A SKILL.md's frontmatter description tells the harness when this skill should fire. The harness reads all installed skills' descriptions, decides which match the current task, and injects only those into context. So you can have a 182-skill catalog installed without paying 182 skills worth of tokens per turn. That's RAG-over-prompts using model-based routing on descriptions rather than vector embeddings. We've been doing this informally with system prompt sections for years, but standardizing it as the loading mechanism is genuinely new.

The bear case for prompt engineers specifically: if a four-sentence file derived from a tweet outperforms careful prompt construction, what are we doing? My read is that the model improvements collapsed a lot of the prompt-engineering surface area into "tell it clearly what you want," and skills survive as a packaging convention because they make that distributable, not because they're harder to write.

For people who do this for a living, are you still seeing returns on technique-heavy prompts (few-shot, CoT scaffolding, structured output, role chains), or is everything collapsing toward declarative behavioral instructions in markdown? Where are you getting the actual wins?

Three of the trending "skills" repos, read for what's inside:

• karpathy-skills: 1 markdown file, 4 rules, ~70k stars
• mattpocock/skills: ~10 SKILL.md files, ~115k stars
• everything-claude-code: 182 SKILL.md files + 48 agents + 68 commands + hooks + rules + MCP configs + npm packages, ~178k stars (plus a second repo by the same author also in the top 10)

Three orders of magnitude in scope, same label.

What's underneath all of them is Anthropic's SKILL.md format. Markdown with YAML frontmatter, auto-loaded by Claude Code (and via shims by Cursor, Codex, OpenCode, Gemini, Antigravity) at session start. It's prompt-with-conventions. That's the actually interesting part.

For two years, the answer to "how do I make this agent better at X" was prompt engineering, manual context, glue code. SKILL .md is the first widely-adopted attempt to make a unit of agent capability publishable, forkable, and installable, with a defined invocation pattern (file gets auto-loaded, frontmatter declares when it fires). That's a real primitive even if the trending list around it is noisy.

Four implications I keep coming back to as a builder:

1. The packaging is the primitive, not the content. mattpocock's tdd skill and ECC's tdd-workflow skill solve the same problem with similar prose. The differentiation is whether you ship it in a format other people can compose with, not whether your wording is cleverer.
2. Cross-harness portability is leaky. ECC ships into .claude/, .cursor/, .codex/, .opencode/, .gemini/, .agent/, six paths, each with its own quirks (Cursor has 20 hook events vs Claude Code's 8, plugin distribution can't carry rules, OpenCode has a different plugin system). Write-once-run-anywhere is real-ish but you pay for it.
3. Workflow enforcement vs capability extension is a real split. Most of the agent-skills discussion in 2024-2025 was capability (tool use, browser, APIs). What's actually trending in mid-2026 is mostly workflow (TDD, triage, code review, planning). Different bet about where value lives.
4. The bear case. If your "skill" is a markdown file derived from a tweet and it has 70k stars, the moat is roughly zero. A startup whose differentiation is a skills pack should assume that pack gets copied, forked, or absorbed into the base model within 12 months. Karpathy's four rules will be in the default behavior of the next Claude release.

For people shipping production agents, are you treating skills as a real distribution primitive (publishing, versioning, dep-managing), or as personal scratch that occasionally gets pushed to GitHub? And does the cross-harness story hold up for you or do you end up forking per setup?

Ran a head-to-head on two open-weight models for tool-calling on a 4-core CPU, no GPU, no cherry-picking. Wanted to see if the small specialist (Needle, 26M, distilled from Gemini 3.1 for function calls) actually holds up against a small generalist (Qwen3-0.6B) that also does tools.

Setup: 50 queries across 5 tiers (simple, paraphrased, implicit, ambiguous, edge cases including foreign language and a "don't call any tool" trap). 5 mock tools. Three metrics per run: parse_success, tool_match, args_match. Same queries, same eval rubric, same hardware.

Headline numbers:

                    Needle (26M)   Qwen3 (0.6B)
tool_match overall    72.0%          56.0%
parse_success         84.0%          54.0%
args_match | match    97.2%         100.0%
mean latency        10.9s          47.9s

The interesting part is not the overall win, it's the failure shapes. They diverge completely:

• Needle fails by picking the wrong tool. When it does pick a tool, args are right 97% of the time. Its sin is selection, mostly routing system commands to search_web instead of run_command.
• Qwen3 fails by not calling a tool at all. Every single one of its 22 misses is a parse failure where it answered in prose instead of emitting <tool_call> tags. When it does emit a call, args are perfect 100% of the time.

Tier breakdown is where it gets sharp. T1 and T2 (literal and paraphrased) are tied at ~95% each. T3 (implicit, like "should I bring an umbrella in Amsterdam?" where the tool name never appears) is where Qwen3 falls off a cliff: 80% to 10%. Needle just maps the intent. Qwen3 tries to be helpful in prose and apologizes for not having real-time data.

T5 (edge) is the only tier Qwen3 wins, by 10 pts. Hindi queries broke Needle's tokenizer (Devanagari fragments badly, one query timed out at 73s with garbled output). Qwen3 handled both Hindi and French cleanly.

One thing that almost killed the Needle run: first pass it scored 8% because I was feeding it OpenAI JSON Schema. Needle was trained on a flat schema ({location: {type, description, required}}) and was literally echoing the word "properties" back as an argument value. Wrote a converter, accuracy jumped from 8% to 72% with no other changes. Worth knowing if anyone else picks up the Needle weights.

Qwen3 had its own issue, it never emitted EOS on the hand-rolled prompt template and burned the full 256-token budget on every query (~230s each). Switching to tokenizer.apply_chat_template(tools=...) with enable_thinking=False dropped it to ~37s and the <tool_call> tags started appearing naturally.

My read: these are not the same product category even though they sound like they are. Needle is a dispatcher. Qwen3 is a tiny chatbot that can also call tools. If you want on-device single-shot tool routing with a fixed palette, Needle is genuinely good for 13MB. If you want any conversational ability, Needle has zero of it and Qwen3 wins by default.

Limitations: n=50 is small. Single CPU hardware. Mock tools, not real ones. Would love anyone who reproduces it on different hardware or with a paraphrase-stress-test to share results.

Repo with full code, raw_log.jsonl, summary.json, and the 5 charts are in comments below 👇

This evaluation was done using NEO, an AI engineering agent. It built the eval harness, handled the checkpointed runs, debugged the schema mismatch and the EOS issue, and consolidated results. I reviewed everything manually and made the calls on what to ship.

Posting some practical findings from a structured audit of a production customer support RAG system. Methodology and caveats up front.

Methodology:

• 6 representative turns from a real production session as the eval set (small, acknowledged limitation)
• LLM-as-judge using Claude Haiku 4.5, scoring relevance/accuracy/helpfulness/overall on 0-10, returning per-turn reasoning strings for verification
• Same judge across all conditions, same questions, same retrieval state where possible
• Production model held constant while isolating retrieval changes, then swept across 5 LLMs once retrieval was fixed
• Live pricing from OpenRouter /models API rather than estimates

Findings:

1. Heuristic evaluation produces zero signal. The existing evaluator counted keywords and source references. Output was numerical but uncorrelated with response quality. LLM judges with explicit rubrics caught hallucinations, identified zero-retrieval turns, and produced reasoning that could be spot-checked. The cost is real but small (cents per run) compared to shipping undetected regressions.
2. Retrieval failures present as generation failures. A turn where the agent said "I don't have information about our company" looked like a model knowledge problem. Trace showed zero documents retrieved. Root cause was a similarity threshold (cosine distance 0.7 in Chroma) too strict for casual openers. Always inspect what entered the context window before tuning the generation step.
3. The production model was not on the Pareto frontier. Sweep across Gemini Flash Lite Preview (incumbent), Gemma 4 26B, Mistral Small 3.2, Nova Micro, and one more. Gemma 4 26B dominated the incumbent on both axes: higher quality scores (7.88 vs 7.33) at 75% lower cost. The incumbent was neither cheapest nor best.
4. Grounding constraints have measurable helpfulness cost. Adding "only state facts present in retrieved documents" to the system prompt improved accuracy scores and reduced helpfulness scores on turns where docs didn't fully answer the question. The judge consistently flagged "the documents don't specify this, contact support" responses as accurate but less actionable. Real tradeoff worth surfacing rather than discovering post-deployment.

Limitations I want to be honest about:

• n=6 is small. Treat the deltas as directional, not as confidence intervals.
• LLM-as-judge has known biases (length, verbosity, self-preference). Using a different family than the production models reduces but doesn't eliminate this. Sanity checked by reading the reasoning strings.
• "Quality" here is judge-defined, not user-defined. A proper next step would be correlating judge scores with user satisfaction signals.

End-to-end delta: +19% quality, −79% cost. The cost win is robust because pricing is mechanical. The quality win I'd want to see replicated on a larger eval set before claiming it generalizes.

I've also written a detailed write up if anyone wants to go in depth on the evaluation process details. Mentioned below in comments 👇

Short post about a tradeoff I keep seeing teams stumble into.

I was auditing a RAG support bot. The original system prompt was friendly, vague, and let the model fall back on its own knowledge when the retrieved docs didn't fully answer a question. This was producing two failure modes:

One, hallucinated product names that weren't in the knowledge base.

Two, generic helpful-sounding advice that was technically off-policy because it wasn't grounded in the docs.

I rewrote the prompt with a grounding rule: only state facts that are present in the retrieved documents. If the docs don't cover it, say so and route to support.

What happened to the scores (LLM judge, 0-10 across relevance/accuracy/helpfulness/overall):

• Accuracy went up. Hallucinations basically stopped.
• Helpfulness went down on turns where the docs didn't fully answer the question. The judge correctly flagged "the documents don't specify this, contact support" as accurate but less actionable than the previous behavior.

The instinct here is to fix the helpfulness drop by softening the rule. Don't, at least not for a factual support bot. The previous behavior was creating compliance risk (off-policy advice) and customer trust risk (hallucinations). The accuracy gain is worth the helpfulness loss for this use case.

What I'd do differently if I were writing the prompt from scratch:

• Be explicit about what to do when the docs don't cover the question. "Acknowledge the gap, restate what's known, route to human support" beats "say you don't know."
• Add tone de-escalation language separately. The grounding rule and the tone rule are different jobs.
• Remove boilerplate greetings. The original prompt was producing "Hello! Thank you for reaching out" on every turn including turn 5 of an ongoing conversation. Embarrassing and a clear signal nobody had tested multi-turn behavior.

Broader lesson I'd take to any prompt change: measure both the metric you're targeting and the one you might accidentally hurt. If I'd only looked at accuracy I would have called this a clean win. The helpfulness drop is a real cost. Better to know about it and ship consciously than discover it from a user complaint.

This chatbot was evaluated and optimized using Neo AI Engineer that built the eval harness, handled checkpointing through timeouts and context limit issues, and consolidated results. I reviewed everything manually

Full report in the comments if useful 👇

Small thing that we open-sourced this week. The problem it solves: you pick a model at the start of a project and that decision is basically frozen forever even though the model landscape changes constantly.

This router sits between your app and OpenRouter and picks the best model per request based on whatever you care about.

You pass a priority flag (speed / cost / quality / balanced) and it runs a weighted score across the whole catalogue to figure out the optimal pick. Decision latency is under 1ms.

Also handles fallback automatically if the selected model fails, caches repeated requests in Redis (or in-memory), and exposes a metrics endpoint so you can see p95/p99 latency and cache hit rates over time.

It's a FastAPI server with a CLI. The CLI has a dry-run mode that's kind of fun to play with, you can ask it "what would you pick for this prompt at speed priority" without actually spending any API credits.

Definitely not finished, quality scores are static right now which limits the adaptiveness. But it's been useful for my own projects so figured I'd share.

Github repo is in comments below 👇

Built this project using Neo AI Engineer.