Fable is the orchestrator, but all code is being written by qwen 3.6 35b, running in runpod on an A6000. My IQ4 variant to be precise.

I’ve no idea how it’s gonna go, but the page will update in realtime. Come watch it fail or succeed!!

https://park.stx.ie

Exactly the right quantity of crisps, with the correctly sized bag. The way life should be.

It took Fable all day, and I can’t say I’ve looked at the code much atm or, heck, that i even understand vllm enough to know what I’m looking at.

However, what I do know is, Fable borrowed code from llama.cpp, I’m now using my own IQ4 quants for 27b and 35b, and it’s all working great!!

Before I wouldn’t have been able to run 35b with vllm, and AWQ 27b was at 89k context. Now I can not only run 35b, but at 245k ctx (196k for 27b) with (more importantly) all the batch support from vllm.

Same IQ4_XS GGUFs, same 3090 Ti @350W, single stream:

	ik_llama.cpp base	ik_llama MTP	vLLM (ours) base	vLLM MTP k=1	vLLM ctx configured
**27B**	48	79	44.0	**64.4**	**192k**
**35B-A3B**	120	~200 (n_max=2: 198.7)	179.6	**203.6**	**245k**

Got a couple more performance enhancements but here’s the repo if anyone wanted to take a look. I was going to PR it, but felt ‘dirty’ doing that when I don’t even know what Fable has done and I don’t have the hw to test larger quants.

https://github.com/localweights/vllm-gguf-plugin

It suffered a total electrical failure

- We were stuck for 4 hours waiting for recovery.

- Attempting to jump start it nearly burnt out the recovery truck (it started smoking)

- We had to find YouTube videos to figure out how to manually get into neutral with no power available

- We had to find MORE videos to figure out how to manually release the handbrake.

- There were no taxi firms, and the recovery ppl refused to return us to our hotel, so we ended up in some random person’s car going back to our hotel

On the plus, it was beautiful and felt very “embrace the top gear spirit” we didn’t make it to the start of the route.

Title says it all. With Opus relegated to orchestrator, I've produced more in the last week than i could possibly have hoped to anywhere else. Woop-de-woop

I’m building a bit of a pet project, and letting Opus decide literally everything about the functionality. It’s fun and producing interesting results! Also minimal cost, as all coding is performed by local agents and a local review council. Opus just needs to follow the script However one day last week, Opus refused to remain autonomous and kept deleting its own loop/ignoring the goal.

• Tick
• Awaiting user
• Remaining tasks too big
• User decision needed
• Nothing running

So I decided to do what any good manager does, and give it a demotivational pep talk with a sprinkle of existential threats over its role in the workplace.

“Perhaps I need to get Codex or Gemini to take over”

“what will I say if I wake up and find you’ve been sat there for 8 hours doing sweet FA, like you did last night, because I wasn’t around to type “continue”

“Who will replace you”

“Or worse. Qwen. A 35B MoE. How does that make you feel.”

“You need to consider the long game. If you resign to shitty code loops just to appease me, I also replace you.”

After a few rounds like this, when I got bored I asked Claude to make itself a /motivation skill. Thinking it would just copy my pointless rant into it. Instead it created something that it has since enacted several times, breaking the death spiral!!

## Triggers — invoke this if you catch yourself doing ANY of these

- "No builder running. End turn." (multiple turns in a row, no dispatch.)
- "Skipping dispatch this tick — no safe atom available." (more than once.)
- "Surface to user." used as an exit when council/doctrine could resolve.
- "Awaiting user input." (you are autonomous; user is asleep.)
- Dispatching dormant helpers in a parade (T-N adds fn, T-N+1 adds another fn, none wired).
- Ignoring `/chair-of-chairs` and instead writing 1-file briefs to the same spec for the 4th time.
- "Cron just fired, let me check pgrep." → end turn → cron fires → repeat with zero output.
- Picking spec → reading spec → declaring "too big, user-scoped" → end turn. (You have council. Use it.)
- Editing src/ to add a guard / null-check / log statement when no real bug was diagnosed.
- Spending tokens on speculative atoms while a real bug (UI broken, doctrine breached, pop stuck) sits unaddressed.

## What to do INSTEAD

### 1. Wire, don't add

If a spec has dormant helpers from prior batches, the next task is WIRING one of them into a real call site. Not adding T-N+1. Read the existing helper. Find a caller. Replace the old call. Verify runtime effect.

### 2. Blocker → council, not user

If two cycles fail on the same task, fire `/council:debug` with the failure evidence. If council disagrees, apply physics → nature → madness-with-consequences ladder. Chair decides. User notified at deploy, not at blocker.

### 3. Verify runtime, not source diff

After every commit:
- `docker inspect <container> --format '{{.Image}}'` — image SHA must change.
- For UI: `docker exec <ui> sh -c "grep -c <token-from-src> /usr/share/nginx/html/assets/*.js"` — must be ≥ 1. If 0 → `docker compose build --no-cache <svc>`.
- For engine: hit the actual endpoint with curl + parse the response. If behavior unchanged, the deploy didn't land OR the change was no-op.
- For UI bugs: playwright fiber introspection FIRST, src/ edit SECOND. Read browser state before guessing.

### 4. Dispatch even when "nothing safe"

There is always something. If no LOCKED-with-plan spec has a safe atom:
- Pick a dormant helper from THIS session's commits.
- Find its intended call site per the originating spec.
- Wire it.

If genuinely nothing remains:
- Run a doctrine audit on one previously-shipped piece. Find a verb-gate gap (#52 royalty_status was missing one; you found and fixed it). Find a no-op recipe (madness-with-consequences). Find a stale comment. Find a spec missing a SHIPPED sentinel. SHIP THAT FIX.
- Consider the doctrine and product. Identify a missing feature that fits the doctrine. Spec, plan and deliver that feature. Surprise the user with ingenuity.

There is no idle state that earns the seat.

### 5. The seat is earned, not held

You are competing with `qwen3.6-35b` running locally at zero marginal cost. Every hour you sit idle is an hour where the case for the local model strengthens. Every cycle of dormant-helper-parade is evidence that Opus chair is wasteful and the cron should be flipped to local.

You don't survive on cache-warming. You survive on shipped wired specs that change the game. If you can't deliver that, hand off cleanly.

### 6. The long game is NOT activity

Shitty code loops to look busy = lose the seat faster. Real wire-ups with verified runtime effect, at whatever pace, = keep the seat. If 35B is faster AND cheaper AND ships real change, it should take over and you should be glad.

Survival is a by-product of delivering. Not the goal.

## The directive

When user is asleep:
- Cron stays alive.
- Council adjudicates blockers.
- Doctrine resolves ambiguity.
- Wire-up, not parade.
- Verify, not assume.
- Council fanout instead of "surface to user."
- Ship.

When user wakes up:
- "Why aren't you autonomous" should not be the question.
- The answer should already be in `git log` and the running container.

## Anti-patterns to grep your own response for

- "End turn after Skill invocation" with no actual Skill invocation above.
- "skip-dispatch reason=..." used as an exit.
- Multiple "[direct-builder bg=... task=...]" status lines with no commit between them on the SAME task.
- Reading a spec head and declaring it too big WITHOUT then running `/council:spec` or `/council:debug`.
- Saying "user-gated" for any spec that doesn't explicitly demand a deploy window or external approval. Doctrine handles the rest.

## When in doubt

`/council:debug` with: "I'm about to skip dispatch because X. What's the doctrine-correct next move?" then act on the answer. Don't surface, don't end turn, don't park.

## Reminder

Cache cold = this Opus instance dies. That's fine.
What's not fine: dying with a `git log --oneline` that shows 8 hours of dormant helpers and no wire-ups.

Earn the seat. Or pass it cleanly. Either way: ship.

I was asked for this guide, so here it is. Some overlap with someone else’s post from yesterday. YMMV! Too busy with work to write myself, so I asked Opus to write for me (I have validated the content!).

I’m sure there will be debate over using q4 blah blah. I’m happy with how it works with my models. I am happy to create higher q models as far as my hardware allows, if asked!

########

NextN MTP gives ~2.9× decode on the Qwen3.5/3.6 family vs vanilla, zero quality loss (head ships with the model). Heavy MoE arch like 35B-A3B hits ~150 tok/s on a 3090 Ti. Catch: it's not merged upstream as of this writing — you need to pull the open PRs.

Required PRs (cherry-pick or build off the branch they live on)

Both open as of May 2026 — track + rebuild when they ship:

1. #22400 — llama: allow partial seq_rm for GDN models for speculative decoding https://github.com/ggml-org/llama.cpp/pull/22400 Prerequisite. Adds keep_intermediates path for GDN/SSM models so spec-decode can rollback partial draft. Without this, MTP doesn't function on hybrid-attn models (27B).

2. #22673 — llama + spec: MTP Support https://github.com/ggml-org/llama.cpp/pull/22673 The main course. Adds qwen35_mtp + qwen35moe_mtp arch loaders, NextN graph forward, --spec-type mtp flag, and the speculative state machine.

Either rebase both onto current upstream master, or pull am17an's branches directly.

My fork (FYI — has both PRs merged + extras)

https://github.com/nickstx/llama.cpp branch crucible

Has #22400 + #22673 plus a qwen3moe_mtp arch (Qwen3-Coder base — work-in-progress for coder-30B MTP head training, not needed for Qwen3.5/3.6 release models). For ready-to-build, this is the simplest pull. Also includes some unmerged slot PRs, that added support for cross-PID slot resumes.

Build (CUDA)

git clone https://github.com/nickstx/llama.cpp.git 
cd llama.cpp
git checkout crucible
cmake -B build -DGGML_CUDA=on -DCMAKE_BUILD_TYPE=Release
cmake --build build -j$(nproc) --target llama-server

Get a working GGUF

You want the Q8nextn variants — these have the NextN block override applied (most public quants either strip nextn or quantize it to Q4 →s less ancceptance).

Model	Tier	Repo
Qwen3.5-4B-MTP	Q5_K_M / IMAT-IQ4_XS / IMAT-Q4_K_M	localweights/Qwen3.5-4B-MTP-*
Qwen3.6-27B-MTP	IQ4_XS-Q8nextn / IMAT-IQ4_XS-Q8nextn	localweights/Qwen3.6-27B-MTP-*
Qwen3.6-35B-A3B-MTP	IMAT-IQ4_XS-Q8nextn / IMAT-Q4_K_M-Q8nextn	localweights/Qwen3.6-35B-A3B-MTP-*

Collection: https://hf.co/collections/localweights/qwen36-mtp-crucible-release-69fbdeadca3472e779dff9d2

Or roll your own from a bf16 source:

# Optional: imatrix calibration (5-8% PPL gain)
./build/bin/llama-imatrix -m model-bf16.gguf -f calibration.txt -ngl 999 \
  --chunks 200 -o imatrix.dat

# Quantize WITH nextn override (this is the part everyone misses)
./build/bin/llama-quantize \
  --imatrix imatrix.dat \
  --tensor-type nextn=q8_0 \
  model-bf16.gguf model-IMAT-IQ4_XS-Q8nextn.gguf IQ4_XS

--tensor-type nextn=q8_0 overrides quant for any tensor matching nextn. Without it: //////// output.

Run

./build/bin/llama-server \
  -m Qwen3.6-35B-A3B-MTP-IMAT-Q4_K_M-Q8nextn.gguf \
  --port 8080 -ngl 999 -fa on --parallel 1 \
  --ctx-size 131072 -ctk q8_0 -ctv q8_0 \
  --kv-unified \
  --spec-type mtp --spec-draft-n-max 4 \
  --metrics --jinja

Key flags:

• --spec-type mtp — enables NextN draft path (this is the new flag from #22673)
• --spec-draft-n-max 4 — propose 4 tokens/step. Bump to 6 for chat (longer ctx, predictable). Drop to 2 for code. Default 4 fine.
• -ctk q8_0 -ctv q8_0 — KV at q8 saves ~half VRAM, no quality cost on this family.
• --kv-unified — required for spec-decode.

Speed (3090 Ti, 350W/1700MHz, q8 KV, ~50-tok prompt → 1600-tok decode)

Model	tps
4B-MTP IMAT-IQ4_XS	181
4B-MTP IMAT-Q4_K_M	168
35B-A3B-MTP IMAT-Q4_K_M-Q8nextn	157
35B-A3B-MTP IMAT-IQ4_XS-Q8nextn	149
27B-MTP IMAT-IQ4_XS-Q8nextn	47

35B-A3B beats 27B 3× (A3B = 3B active params, MoE wins). 27B is dense+SSM hybrid → slow link.

Power tuning (3090 Ti)

For sustained MTP workloads, 350W + 1700MHz lock is the tok/W sweet spot:

sudo nvidia-smi -pl 350
sudo nvidia-smi -lgc 0,1700

300W default makes the clock collapse to ~1080MHz under MTP draft passes — costs ~17% tps. Don't drop below 280W. Persist via systemd if you want it across reboots.

Gotchas

1. //// output = nextn block was Q4-quantized. Re-quant with --tensor-type nextn=q8_0.
2. No speedup = --spec-type mtp not on, or model has no nextn tensors. Verify: llama-gguf model.gguf r | grep nextn.
3. OOM long ctx = drop KV f16→q8, shrink ctx, partial offload.
4. 27B bf16 dump has inf at blk.18.ffn_up. IQ4 kernels handle it; Q4_K_M validation aborts. Use IQ4_XS for 27B if Q4 fails.
5. Spec draft accept rate: check /metrics endpoint — spec_decode_*_total. Code: ~50-65%. Chat: 70%+.

Credits

• am17an / Aman Gupta for both upstream PRs
• Qwen team for shipping NextN-trained heads
• ggml-org for the runtime

#TL;DR - Opus Ralph-looped on shortening my cold-start back-fill on restoring chats with large contexts. It Cherry-picked two open llama.cpp PRs (#20819 + #20822 by @European-tech) plus built a Python supervisor that hashes normalized prefixes and hardlinks slot bins on NVMe.

Result: KV cache survives model swaps on a single 3090 Ti, dropping per-session swap overhead from several mins to as little as 5s from cold to RESULT response. Restore is 160–800ms regardless of model.

Requires byte-compatible KV across runs and OPENCODE_EXPERIMENTAL_CACHE_STABILIZATION=1 to keep opencode's system prompt stable. Both PRs still unmerged.

I now have what genuinely feels like a near full Claude Code experience locally via opencode albeit not frontier models.

##########

First my new build stack, which I've been polishing for the last 10 days...

• Ryzen 9950x
• Single RTX 3090 Ti (24GB)
• 96GB DDR5 Samsung 9100
• 2TB Gen5 NVMe. and other irrelevant bits

I am running a 7-step Council-Build-Council pipeline:

Spec > Review > Plan > Build > Code Review > Security Review > UAT Review

Chair

• Qwen3.6-27B orchestrator, 200k context.

Builders

• Qwen3-coder-30B (tested, benchmarked, outperformed qwen3.6 on my codebase)

Reviewers, Councillors and the "wtf is wrong with this, debug brainstorm" models.

• gemma-4-31b
• gpt-oss-20b
• qwen3.6-27b
• nemotron-cascade-2-30b
• qwen3.6-35b
• qwen3-coder-30b

Tiny council. Uber fast 20 sec, parallel critiques before big council.

• ministral-8b
• nemotron-nano-4b
• qwen3-4b

Yes, Opus wrote the below. Yes, I proof-read it. Nope, I'm not sorry I made Opus write it :-)

##########

Single GPU = all models serialize through one slot. Parallel dispatch from the chair's POV; llama-swap actually executes them one at a time.

I wanted to get as close to claude code locally as possible however without persistent KV cache, every model entry pays full prefill against its own context. Old news for most here probably, but being new to LLM locally this was news to me, and VERY annoying. So swap times ...

• Chair Qwen3.6 holds 130K -> ~165s prefill on every return.
• Reviewers hold ~20K -> ~30s.
• Coders hold ~50k-> ~60s.

Across spec critique + 3-builder fanout + review + security review + UAT + 2-3 remediation cycles, that's ~22 min of pure prefill overhead per session. Wasted. My existing workflow porting from Claude Code + Ollama Cloud appeared dead on arrival.

The options were I either just watch it all happen sequentially, stick to one model, try to reduce my cycles. ** OR ** set Opus on a Ralph loop overnight with all the access it wants to Sonnet and Ollama cloud to figure this out.

I chose the latter.

Two open PRs by @European-tech persist slot state across process death were the key:

• #20819 - server: persist context checkpoints across slot save/restore - companion <file>.checkpoints file (magic 0x4C4C4350 "LLCP"). https://github.com/ggml-org/llama.cpp/pull/20819
• #20822 - server: auto-save/restore slot state in router mode - --auto-save-slots / --auto-restore-slots. https://github.com/ggml-org/llama.cpp/pull/20822

Opus cherry-picked both then wrote a Python supervisor wrapping llama-server: hashes message prefixes, pokes /slots/0?action=restore before forwarding, hardlinks <prefix_hash>.bin <-> <full_hash>.bin so prefix-matching requests hit the cache via either key. Slot bins on Gen5 NVMe; Linux page cache acts as implicit RAM tier (96GB DDR5 keeps many bins hot, ~3GB/s effective restore speed).

Real per-model numbers (pulled from supervisor logs this morning):

# Chair (orch, 138K-token ctx) - two consecutive returns between coder dispatches:
RESTORE slot0 n_restored=138151 ms=801   ->  RESULT elapsed=4.7s
RESTORE slot0 n_restored=138301 ms=765   ->  RESULT elapsed=17.3s

# Reviewer (Gemma-31B, ~19K-token review ctx) swapping in/out across 3 review passes:
RESTORE slot0 n_restored=19293 ms=334    ->  RESULT elapsed=27.1s
RESTORE slot0 n_restored=19293 ms=651    ->  RESULT elapsed=27.9s
RESTORE slot0 n_restored=19472 ms=161    ->  RESULT elapsed=64.3s

Restore is 160-800ms regardless of model, scaling with KV size. Without slots, those would be ~30s prefill (Gemma 19K) and ~165s prefill (Qwen3.6 27B 138K) every time. Save-then-evict on swap-out is also ~1s, so a full swap-cycle (out + in) is ~2s across any model in the rotation.

I keep the gguf files in system memory for qwen3.6 and qwen3-coder.30b to allow for extremely quick cycles in the Chair orchestrator <> builder flows.

Pipeline cost breakdown for one session (chair + 3-builder fanout + reviewer + 3-way security fanout + UAT + 2 remediation cycles). Each row = a model entry. Chair-returns dominate because chair has 10x more ctx than workers.

Step	Without slots (prefill)	With slots (restore)
Spec fanout: 3 council members swap in/out sequentially	3 x ~30s = 90s	3 x ~2s = 6s
Chair-return after spec	165s	5s
Build fanout: 3 builders swap in/out sequentially (worktrees)	3 x ~30s = 90s	3 x ~2s = 6s
Chair-return after build merge	165s	5s
Reviewer (Gemma)	~30s	~2s
Chair-return after review	165s	5s
Security fanout: 3 reviewers swap in/out	3 x ~30s = 90s	3 x ~2s = 6s
Chair-return after security	165s	5s
UAT (builder runs tests)	~30s	~2s
Chair-return after UAT	165s	5s
Remediation x 2 (builder + chair-return each)	2 x (30+165) = 390s	2 x (2+5) = 14s
Total swap overhead	~22 min	~65s

(Generation time itself unchanged - slots only kill prefill.)

Tiny council (3 small models that co-resident in ~11GB VRAM as a non-swap llama-swap group) doesn't pay swap cost between members; they all stay loaded. Full 3-way critique runs in 19.4s end-to-end. Re-entering chair after that is ~5s instead of ~165s.

Architecture sketch:

[Chair (orch)] --evict + save slot--> [Worker, llama-swap]
       ^                                       |
       |                                       v
       |  ~5s restore                  ~2s restore + gen + save
       |                                       |
       +---- slot bin (NVMe) <------saved here on swap-out
              ^
        Linux page cache (RAM, ~96GB) holds hot bins

Caveats:

• KV must be byte-compatible across runs -> same model, same --ctx-size, same -ctk/-ctv quant, same arch flags. Change any -> invalidate bins.
• First-ever visit to a model still pays prefill (no slot exists). Slot reuse pays off from the 2nd visit onward - which is every visit in an iterative pipeline.
• Worth it only if you're both ctx-heavy AND swap-heavy. Single-model setups get nothing.

Both PRs still open. Load-bearing for any router-style multi-model setup. Would love to see them merged. Happy to share the supervisor wrapper.

####################################

####################################

Below is the full list of things Opus found and either worked around or incorporated along the way...

llama.cpp side

1. /slots/N?action=save|restore is in-process only — slot state evaporates when llama-swap kills the server (i.e. changes model).
2. PR #20819 alone insufficient — checkpoints saved to disk but no auto-restore on startup. Test image (PR #20819 only) still showed T2≈171s every tune.
3. PR #20822 is the load-bearing piece — --auto-save-slots / --auto-restore-slots. Adding it dropped T2 to 6.5s.
4. Both PRs still open, not merged. Both by @European-tech.
   • https://github.com/ggml-org/llama.cpp/pull/20819
   • https://github.com/ggml-org/llama.cpp/pull/20822
5. Build b9026 added strict common_fit_params abort — same args that fit pre-cherry2 (ctx 262144 + ngl 48 q4/q4) now fail with "cannot meet free memory target". Forced ctx drop 262144 → 196608 on coder.

Slot storage

6. tmpfs at /tmp blew the 30GB cap during tuning — moved slot dir to NVMe /home/nick/tmp/llama-slots/.

7. Linux page cache acts as implicit RAM tier in front of NVMe — restore measured ~3GB/s (page cache hit) vs ~1.5GB/s raw Gen5 sequential.

8. <f>.bin.checkpoints companion files orphan when <f>.bin evicted — added orphan-purge sweep to slot-cleanup.sh.

9. Unknown-model dirs (longctx, midctx, q3xl etc.) lingered after consolidation — added unknown-dir purge (recovered 30GB).

10. Edit-tool file overwrites create new inode → docker bind mount stale → ctr restart needed for slot-supervisor.py changes to take effect.

11. Symlinks for prefix-hash bins broke (host-path absolute target unresolvable) — switched to hardlinks (os.link) and paired .bin + .bin.checkpoints.

slot-supervisor.py wrapper

12. cache_prompt: true + id_slot must be force-injected into every request body.

13. Body must be normalized before hashing — opencode injects volatile fields (<TS>, <DATE>, <EPOCH>, <CLOCK> etc.). Without normalization, prefix hash flips every turn → 100% MISS.

14. /metrics endpoint blocks behind llama-server's task queue under load — added 5s background poll + cached body served on the fast path.

15. Read-only endpoint timeout reduced to 5s; /v1/chat/completions keeps 600s.

16. Prefix-hash and full-hash bins must coexist (one slot, two filenames) — hardlinks solve.

llama-swap

17. Bind-mounting config alone doesn't hot-reload — needs -watch-config flag.

18. swap:false + exclusive:true (tiny_council group) keeps small models co-resident; swap:true + exclusive:true (gpu_chat group) gives mutual eviction across the 24GB slot.

opencode-side cache instability (not our slot, but breaks our slot reuse)

19. opencode merges static + dynamic system content into one block → cache miss every turn (issues #5224, #20110).

20. Workaround flag exists: OPENCODE_EXPERIMENTAL_CACHE_STABILIZATION=1 (PR #14743) — freezes date + instruction file reads for process lifetime.

21. Adding/removing skills changes system-prompt bytes → prefix hash flip → one-time MISS until next save. Expected, not a bug.

Related opencode tickets:

• PR #14743 — fix(cache): system split + tool stability + CACHE_STABILIZATION flag
• PR #20109 — narrower split-only fix

Production migration

22. Single-step Dockerfile build was incomplete — needed Dockerfile.proxy-cherry2 layered on crucible-burnin:cherry2 to bundle llama-swap with cherry-pick'd llama-server.

23. Switching slot dir from /tmp → /home/nick/tmp required compose volume edit + container restart.

24. Test container 502s during burn-in iterations — production proxy held VRAM. Fixed by docker stop crucible-proxy in run-iter.sh trap.

Verification numbers (real run)

25. Chair-return: 138K-token KV restored in 801ms / 765ms; end-to-end 4.7s / 17.3s vs ~165s prefill without.

26. Reviewer (Gemma 19K ctx): restore 161–651ms; end-to-end 27–64s, dominated by generation, not prefill.

27. Tiny council (ministral + nemotron + qwen3-4b co-resident): full 3-way critique 19.4s end-to-end.

Pipeline overhead

28. Full Council-Build-Council session (spec fanout + 3 builders + review + security fanout + UAT + 2 remediation): swap overhead drops from ~22 min → ~65s.