u/Zacharytackary

got banned for broadening the conversation outside of just labor oops

posting specifically to be a petty asshole because i was obviously jorking it and made a comment that is clearly not possible to enact given my clearly observable status as someone without significant finances.

lmk what edits i need to make to constrain myself to be within ruleposting guidelines idk what the formal barriers are tysm

Repo can be found here.

I would like to start this off with: I hate github but forgejo won't send me the confirmation email so I'm stuck send help

anyways

Multi-token-Prediction has recently been refined in vanilla Llama.cpp; and, with aggressive quantization, the model at significantly large context windows can be loaded onto compute hardware that most would deem near-useless for inference.

I'm running an Nvidia 3060 (12Gb VRAM) & a 12th gen i7 (16 GB CPU RAM). Pre-build; and I can cleanly get about 95/100% maximum context out of Jackrong's Qwopus 3.5 9B Q5_K_S model. Before MTP; I was getting about 20tps out for 95% GPU utilization and ~50% CPU utilization.

MTP speeds it up to around ~32tps at the cost of some extra memory overhead, but it ends up leaving my CPU almost entirely alone. As such, I can run inference workloads on one monitor, play light-moderate-weight Indie games on my main monitor, and actively 'work' on software development tasks by pausing my game and steering the model out of a hallucination loop every 25-45 minutes.

For example, the merge you see before you.

My specific motive for doing this was I wanted access to Jackrong's Qwopus 3.5 9B Coder MTP model, and did not want to give up my turboquantization *or* switch to another branch. So I made my own, and spent the next three days trying to figure out why the fuck Cmake wasn't recognizing my CUDA compiler while also immediately saying it was recognizing my CUDA compiler (what the absolute fuck windows??).

I am glad to say the inference backend works well enough that I notice literally no issues (test-T=~8-12 hrs inference no crashes). And that the Coder MTP model is a wildcard that performs near-perfectly if you - instead of starting it with a fresh context - start the chat off with the more stable but less good at coding original verison until you have properly created the spec or identified the problem, and then let the coder model rip on the actual coding tasks itself, with heavy steering required due to the coder model's slightly degraded pivoting and social/semantic inquiry tasks; it is quite nice. Definitely would not reccommend as a generalized agent, however...

People seem to be hating on the local models, but the near-complete detachment from the megaconglomerate ecosystem is amazing to work within, especially if you're new to software development and don't want to spend subscription pricings on revokable access that overcharges you $200 for having a hermes commit in your repo history.

Lmk if I broke it!