Buy now or wait on a local LLM box during the memory crunch? How I'd read it after running a Strix Halo daily for 6 months
I get asked some version of "should I buy a local AI box now or wait for the next one" a lot, and my answer changed this year, so I wrote up how I currently think about it. The short version here, full writeup linked at the bottom.
As I'm sure you can all relate, the usual instinct is to wait, because hardware normally gets cheaper and better. Right now that's kind of backwards for this class of machine, for two reasons.
One, the memory is soldered (at least for the non-DIY boxes, I am not talking about multi-gpu self-built setups here) and it's getting more expensive, not cheaper. LPDDR5X jumped 89% in a single quarter and the analysts I've read don't expect real relief before late 2027. So for these, the capacity you pick is permanent, and you're picking it in a bad market. Concretely: the same box I paid under $2,200 for is about $2,800 now.
Two, for token generation these boxes are all bandwidth-bound, and the announced successors mostly add memory at the same bandwidth. A 192GB or 768GB successor lets you load a bigger model but runs it at more or less the same speed. So waiting for "the bigger one" only helps if you're actually running out of capacity, not if you want more speed.
One caveat so I'm not oversimplifying: prompt processing (time to first token) is a separate, compute-bound story, and there the picture flips. The Spark is much faster at it, Apple and Strix Halo are weaker. The M5 Mac is interesting because it is reported to target exactly that, and it's the one case where I'd actually think about waiting, with the honest caveat that Apple just raised prices and the release date is still an expectation, not a certainty.
I'm running a Strix Halo box myself, so I'm not neutral, but I tried to be fair to the Spark and the Mac in the full post. So I'm curious what you'd do: if someone came to you today and asked whether to buy now or wait, what would you tell them? Especially keen to hear from people actually running a Spark or a Mac Studio, and how prompt processing feels with daily use.