u/kjames2001

Disclosure: this comes from a project I built (Auth Your Agent; the client side is MIT and free).

I let an AI agent run errands on websites for me. Getting it signed in was never the worrying part. What worried me was what it does once it's in. So anything that costs money or can't be undone waits for me to confirm with a passkey on my phone.

Some choices I'd like opinions on:

• A password can never approve these. It has to be a fresh passkey assertion, and each approval works once.
• The shop writes the text I approve ("Pay $45.00 at Lamp Shop for Brass lighthouse lamp"), not the agent, so the agent can't describe one purchase and make another.
• I approve a sign-in once for a period I choose, from an hour to 30 days, but payments ask every time.
• For a scheduled run (say, posting at 3 a.m.), the agent sends a plan first, and one passkey approval covers the steps I tick, each with its exact text. That approval needs the passkey. Signing in with Google can't approve a plan, because it would approve every step.

Here's a real agent checkout on a demo shop, 24 seconds from Buy to Paid (31-second video): https://authyouragent.com/static/video/demo-shop-checkout.mp4

Two things I'm unsure about. Is a passkey prompt on every payment too much friction? And if you have synced passkeys on several devices, should any of them be able to approve, or only the phone that registered?

Update: TRACER surrogate + two bug fixes that stopped the cascade from imploding

Original post

Recap: Hermes Router Proxy is a single-file Python server that sits between your OpenAI-compatible client and your model providers. It classifies every prompt as "simple" or "complex" and routes accordingly — simple stuff like "ok", "thanks", or "make that darker" goes to a cheap/fast model (e.g. deepseek-v4-flash), while real work like "implement a binary search tree" goes to a smarter one (e.g. deepseek-v4-pro). It has multi-tier fallback cascading, per-model circuit breakers, automatic key rotation on rate limits, and a transparent SSE proxy that survives mid-stream failures. It was built for my Hermes agent setup running on a Proxmox homelab in Botswana, but it works with anything that speaks the OpenAI API format.

---

The original router used a small LLM (qwen2.5:3b) for every classification — ~200ms and an Ollama call per new conversation. It worked, but using an LLM to save LLM calls is circular. Then someone pointed me at TRACER by adrida (arXiv). Their idea: train a cheap surrogate, gate it with confidence, only call the teacher model when unsure. That was the missing piece.

I didn't implement full TRACER (they replace entire LLM responses — I only replace the routing decision). But the surrogate + confidence gate pattern is straight from them.

What's actually running now:

• Sentence embeddings surrogate (all-MiniLM-L6-v2 + logistic regression). Classifies prompts in ~1ms locally. Only falls back to qwen2.5:3b when confidence is below threshold.
• Weekly auto-refit. Cron job runs every Saturday at 3am — re-trains the surrogate on production traces, restarts the service, smoke-tests. Zero maintenance.
• Classifier report endpoint (/classifier/report). Coverage metrics, confidence distribution, drift timeline. Also works as ?format=html.
• Real traces, not synthetic. 11 production traces so far (just deployed). CV accuracy 1.0 with 100% coverage on that tiny set — meaningless number, ask me again when there's 200+.

Two bugs that took way too long to find:

1. Circuit breakers were per-host, not per-model. deepseek-v4-pro returned 503 on ollama.com → circuit opened for the entire host → glm-5.1 on the same host got blocked too, even though it was fine. One bad model could kill every tier. Fix: circuit keys now include model name (https_ollama_com_v1__deepseek-v4-pro). Each model gets its own circuit.

2. Streaming had no 503 retry. Non-streaming retried 503 with exponential backoff (2s/4s/8s). Streaming didn't — one 503 and it immediately cascaded to the next tier. Three 503s in a row exhausted everything. Fix: streaming now does the same retry loop, including resuming the SSE stream if retry succeeds.

Both bugs together meant a single overloaded model could burn through all fallback tiers in seconds. The router looked broken but the cascade logic was actually correct — it just had nothing left to fall back to.

What's still rough:

• 11 traces is nothing. Real accuracy numbers need weeks of data.
• The simple/complex binary split works for coding agents. Probably breaks for anything more nuanced.
• Embedding model adds ~6-7s to cold start on CPU. Fine for 24/7 homelab, annoying otherwise.

Repo: github.com/kjames2001/hermes-router-proxy

Credit: the surrogate + acceptor gate pattern is straight from TRACER. If you're doing LLM routing, their paper is worth reading.

Just received this from ollama:

Hi there,

Earlier this week some users reported hitting session limits much faster than expected on their Ollama subscription. We investigated and discovered a caching issue in our model hosting that caused session and weekly limits to be consumed too quickly, especially for DeepSeek-V4-Pro.

I'm sorry this happened. Your account was one of those affected, and I've issued you a refund for the last month of your subscription.

The team is working hard to make sure this doesn't happen again in the future. Please reach out to me anytime with feedback.

Best,

Jeff Ollama