vLLM ROCm/HiP, 4 bit compressed-tensors (int4)
Not a fair comparison, but Qwen-122b on the most optimized format possible I have run (rocmFP4) does not touch Ling in speed.
https://x.com/ciruai/status/2085996633267777554?s=46
Tool call is broken in certain harnesses. It works well with pi-type harnesses (omp, feynman). Has anyone noticed this?
It has respectable numbers for 1 stream, but the decode goes down severely with multiple streams. Using the ROCmFP4 quant with the appropriate Llama.cpp fork.
Here is the prefill performance in an M2 Ultra with 192GB of RAM.
For decode, at the following depth:
Start: 28 t/s
45k: 23.5 t/s
192k: 18 t/s
That speed is maintained with 8k token output at those depths.
Perfect for Strix Halo in quant form if it holds up!
I’m not a coder, but I run local models. I gave in to agent hype (I was building my own, but there is so much to do) and installed Hermes. Running with Qwen-397b out of a 2 spark cluster.
So…I asked Hermes today to tally the token count, and the result…200 million tokens. In 5 days.
At this rate, using an agent for tasks like installing software and debugging things I want to try out, what is the cost I am saving? Artificial Analysis says the price is about 1.25 dollars per million tokens on average from providers. At current pricing per Artificial Analysis, that gives me about 1250 dollars per month, and my sparks will pay themselves by 6 months.
So, caveats of course I bought them at cheaper prices than today, but it’s a simple estimate that there is some valid reasons to go local.
Like I said, I am not programming and I know there are programmers that easily triple my token count in the same time. That implies that if you use 100 million tokens per day, the return on investment is still there today, even with crazy computer prices.
To me, local AI is about the desire to utilize a cool technology without the strings attached that threaten individual privacy and intellectual property. But knowing that my investment is not just purely hobbyism gives me more conviction that local AI is the future.
I know I am preaching to the choir…So the question is, has anyone else felt their rig is becoming more sustainable now than 6 months ago, price wise? Would love to hear!!
