u/mesolimbic-bliss

I wrote a local, tiny 8-task benchmark test so I can compare all of the new HF models and quants that fit on my 24GB VRAM rig and then I post the results.

I use several harnesses for each model, that way I can see if the model failed or if the harness failed.

Cline inside VsCode hasnt succeeded even once. But it succeeds using top cloud models, which kinda defeats the point.

All of the local models failed and couldnt finish the 8 tasks and submit a score. I saw tool call nightmares, loops, crazy hallucinations, etc.

I love Cline. I used it to win a stock trading competition a few months ago (using an older GPT model).

Am I doing something wrong? Is there setting I need to tweak? Why are the local models failing only on Cline while succeeding on Pi and Hermes? Do they perform more hand-holding?

EDIT: Here's an example summarized by Gemini
"This log snippet provides a clear view of why the agent got stuck. It reveals a compounding storm of shell escaping traps, PowerShell syntax errors, and tool-formatting failures while trying to debug a regex script"

Rig: 2 x 3060s = 24GB VRAM, 64GB Ram, 1TB NVMe, Ryzean 5500, llama.cpp

Various coding and agentic tasks. After the benchmark ran, I started using Hermes for agents and Oh My Pi with desktop GUI for coding.

Model	Quant/profile	Encode TPS	Decode TPS	Full Mode-A accepted/hour	Notes
CoderX 27B-A3B + MTP	Q4_K_M, 64K, Q8 KV, 55/45	~539	~82–84	6.32	Clear coding-worker winner; best repeatability
Nex-N2.5-mini	IQ3_XS	~516	~59.6	3.18	Second-best throughput; solid agentic candidate
Qwen3.8-27B	UD-IQ4_XS	~215	~19.3	2.38	Stronger dense baseline, but much slower
Ornith-1.5-35B-A3B	AD-Q4_K-IQ4_XS	~547	~58.1	1.34	Fast encode, but weakest accepted-tasks/hour result

The full benchmark was 136 runs, ranking by tasks/hour

1. CoderX — 6.32
2. Nex — 3.18
3. Qwen3.8 — 2.38
4. Ornith — 1.34

My daily use cases:

CoderX (coding and agent work)
Nex (mostly agent work if CoderX doesnt work)
Qwen3.8 (coding tasks overnight)
Ornith (backup to the backup, but still honestly decent on alot of little agent tasks)

Is anyone running Claude Code desktop app with Ori to use Openrouter models through the Openrouter API?