Disclaimer: Any* means any model that exposes its CoT without it being censored.

Hey guys - half a tutorial/guide, and half an I built this, so I went for resources. This is something that I created for myself recently when I couldn't find any good existing solution. I wrote this post myself, no AI!

Probably a fair number of you have seen my posts about fine-tuning AliceAI 80B A3B according to my own synthetic datasets. If you did, I'm still fine tuning it on a live stream right now - check out https://figure-bios-expect-cio.trycloudflare.com/ -- it'll let you inspect any and all of the training data that I generated with this engine. If you have any interest, it's pretty neat! unfortunately that link is optimized for desktop only and I'd have to kill the run to reset it, so u may want to rotate the phone.

That thread was at https://www.reddit.com/r/LocalLLaMA/comments/1wslgkw/comment/pcw4kgw/?context=1&screen_view_count=1

That run is using off-policy distillation, and that's I made this for. my training data for that project with this repo, and just customized it for an OSS release. Basically, you create a "curriculum" for your goal - e.g. if I was training an agentic model, I'd need things like tool calls, bug fixing, working in a workspace, tracing errors, etc. You define your curriculum in a yaml file, then an LLM creates tasks based on the curriculum you defined, and the chosen LLM you're distilling from then solves each task, leaving you with a full Q/A set that encompasses your fine tune goals.

I used qwen 3.8 27b on medium to generate the tasks - I'd recommend avoiding anything any weaker than that.

I forked my private repo of this that I've been using into SFTMill, which is basically just the same thing with great documentation and a few steps added to get anyone onboarded rapidly. I created it [and my original version] because I couldn't find any existing pieces of software made with this design, for this purpose.

I release it because I enjoy contributing to the community, and there's a vague hope someone will eventually see one of my pieces of work and want to hire me (if you're reading this and you like the project and you need a software/ml engineer remote or in NYC, let me know <3). It makes me happy when my software helps others so I'd love if you let me know if it helped you. Cheers!

Shoutout u/FullOf_Bad_Ideas for helping me with my alice train in areas I wasn't experienced enough in - I threw a Multi-Turn Hybrid-Reasoning (user <> assistant) section in the readme just for you bud, hope it helps.

https://github.com/jackjusko/sftmill

8.8 years POWERED ON and counting with 0 errors

https://preview.redd.it/68cqyv7u5mnh1.png?width=3472&format=png&auto=webp&s=6d649fa96f36a7748a446efb2d3f68b8a890f96d

Xeon E5-2697v4 with 128gb ddr4, in quad channel

https://preview.redd.it/8ofj4uhw5mnh1.png?width=3472&format=png&auto=webp&s=075b41523b59fe77f3196c92b2555e23ab9f5636

https://preview.redd.it/o2hz3l6y5mnh1.png?width=4624&format=png&auto=webp&s=7d663dbf74b13c4a261defb68da0c9cbad5e68cb

https://preview.redd.it/t41pa76z5mnh1.png?width=3472&format=png&auto=webp&s=45e244a4ff39e271339b49dac5cbf92a595add44

One tesla P40 (24 gb)

Two 32gb V100's in NVLink on a, externally powered breakout carrier board through a PEX PCie bridge

Motherboard was a gigabyte x99 ud4p board - took me roughly two full days to get it to map the memory associated with all of the gpus to map, thank god for LLMs for doing bios mods because I probably flashed 16-18 different iterations before one worked with all this different shit. Eventually I managed to create a modified QFlash image (after spoofing the checksums on the modified one to appear as an original bios). It is just barely technically possible with a few weird quirks that can be worked around.

I've also got a driver mod for the p40 or the v100s that lets you enable WDDM mode and stream games throug ha different gpu, it's pretty cool to play games on them.

I made my reddit post title so long and exact in case anyone is every trying this on this motherboard again can find it on google (I think it's a pretty cool AI build design so I imagine people might consider) - so they can find the bios image. The exact QFlash image is archived at https://archive.org/details/x-99-ud-4-p.-23c, along with a full start to finish guide on how to use it. Good luck, future soldiers!

I didn't have enough space to put the 4th SAS drive next to the others so I added an anti vibration sock.

UPDATE: I've had to rethink this substantially - the final product will be different, but viable. I'm attempting upscaling now directly on the V100 in WDDM mode, and that'll pipe through the igpu. Should mostly kill latency but will be complicated. This post still serves as a proof of concept, and I'll continue development and updating. The goalpost has shifted towards a true temporal upscaler directly on the WDDM card that takes advantage of its capacity. The goal is to get a high performance upscaler taking advantage of tensor cores to run natively on either a v100 16gb or a titan v. In theory, a V100 card absolutely blows the rtx 20xx line out of the water in both memory bandwidth (huge for ML speed) and raw compute, but the difficulty is going to be getting it to work, and actually compete.

I am going to continue to add development updates to this post until it's time for a new thread

Disclaimer: I'm still working on holding framerate always above 60 and still improving the upscaling quality (playing around with a bunch of different neural net implementations) - so take this as a working demo to back up the rest of this post and that I'm not just talking out my ass. I've only been working on this for a few days and have a long way to go. I think reddit is cranking down the res, so the youtube video is at https://www.youtube.com/watch?v=6-y3H98Y7hg&feature=youtu.be

TL-DR: I've developed a way to use an old, cheap nvidia datacenter card to run machine learning accelerated upscaling on any game you already own, without replacing your GPU or your setup.

Hi everyone,

I'm a software developer who's currently focused on AI/ML workloads. I tend to be thrifty and use hardware that's a little less mainstream to get extra performance.

I own a Tesla V100 - a nvidia chip from 2017/2018 that was the first chip to ever include tensor cores. The 16gb PCIe edition is currently $230 on ebay. I only own it in the first place for running and training LLMs.

I'm developing an AMD-FSR esque temporal upscaling solution that uses a small neural network on the V100's. I actually already have a working (alpha, buggy of course) implementation running. This began as a personal hobbyist project, as I actually own TWO v100s - I wanted to see if I could put the extra one to work doing upscaling while the main one does the gaming workflow. The way I have the display setup is funky and hacky - one v100 has had its driver modified to enable wddm mode, and I stream the output from that through an iGPU.

The realization I've come to is that what I'm developing should already work as a drop in with any video card, provided you have an extra PCIe x16 lane on your machine that's not in use. Essentially, the V100 intercepts the frames being rendered in real time, upscales them, and hands them back off to your main GPU.

I feel fairly confident there'll be interest in a fully developed version of this, considering any card that actually has AI upscaling is still crazy expensive. As I mention, a 16gb V100 can be bought for about $230 and dropped into any existing rig. This should also work with any existing graphics card you already have installed.

I'm not ready to release this yet, just gauging interest in a fully developed version. I'd be remiss if I only told you about it though, so here's an early demo of upscaling tomb raider 2013 720p to 1080p.

Hey! Some of you may have seen the post when I started this build, which was at https://www.reddit.com/r/LocalLLM/comments/1vxjpni/meet_bbprime_my_2k_104gb_vram_256gb_ram_extremely/

Forgive the absolutely batshit setup - this thing needs two PCIE power ports, two CPU power ports, and a full ATX power connector. Running both pcie power connectors on the same PSU wire didn't work, so each PSU has one CPU and one GPU/PCIE connector powered each. The V100s only use 600W max, and I keep them capped total around 450 because I don't have my finished cooling setup yet, and these ghetto ass fans won't keep them under 80C unless I slightly power throttly them. Barely moves performance though.

Unfortunately I didn't realize that poweredge didn't support AVX2, so I had to go a different direction. However, after MUCH trial and error, I have two V100s (32gb each) running in NVLink, tensor split mode with a total of 64gb VRAM! I've been hyping up these gpus like crazy lately in various threads, and here's the reason why! (the hardware for just the v100s, necessary adapters included, cost me roughly $1500 - that includes the V100s, the carrier board, the fans, the SlimSAS rig, etc - everything).

so NOW, WHAT YOUV'E BEEN WAITING FOR: THE BENCHMARK!

Setup: Qwen 3.8 27b at Q5_K_M, with 256k context window, and KV cache at full FP16. MTP Enabled with max prediction length 3, temp 0.8

~24k context window used

Prefill: roughly 1,050 tps

Decode: probably average 55 tps, bounces between 40 and 70

128k context window used and near 200k context used:

I was getting roughly 600 prefill and 50tps at 128k

check back later - I'll have these up within 24 hours, I have to go to a party right now and I'm already late, and qwen won't shut the fuck up long enough for me to get a prefill benchmark. CUSOON!

update: benched with the Q8_0 version, all other settings kept

bruh can't send my screenshots, but 900 prefill at 64k context, 730 prefill at 126k, highest I've gone so far is 161k in which I'm getting 656.

At 160kish context the average tps for decode is is still about 55! I might have to recheck my earlier bench, but I am absolutely sure I am still getting 55 tokens per sec at Q8, 160k context

jk i was a little hasty on that part, a better estimate that's fair is probably 40-45, with some times consistently peaking above, almost like a cpu turbo in sections

I'm pretty impressed, I didn't really know what performance was going to be before I bought this, but for the age of the cards and the overall price I paid it's damn fast.

It's hard.