TL;DR: I used a Ryzen AI Max+ 395 mini PC (128 GB unified memory) as an actual training box to convert Qwen2.5-7B into 1.58-bit ternary (BitNet b1.58). Three takeaways: (1) the 128 GB unified memory is the real feature: teacher+student distillation at 7B peaks at 87 GB, which no consumer discrete GPU fits; (2) stock ROCm segfaults on gfx1151 on first dispatch, so use AMD's TheRock wheel; (3) bf16 is mandatory, because the fp32 matrix path on RDNA 3.5 is a ~100× trap. Caveat: every run here is budget-limited (~0.5M–4M tokens vs the ~10B a real recovery needs), so this is a direction-and-scaling result, not a quality claim.

A short field report from using the Nimo AI Mini PC (AMD Ryzen AI Max+ 395) as an actual ML training machine: the ROCm bring-up, one honest 7B result, a recipe comparison from 360M up to 7B, and where the 128 GB really pays off.

The rig

Nimo AI Mini PC (Ryzen AI Max+ 395):

• 16 cores / 32 threads, up to 5.1 GHz
• Radeon 8060S iGPU (RDNA 3.5, 40 CU, gfx1151)
• 128 GB LPDDR5X-8000, 256-bit (~256 GB/s), unified between CPU and iGPU
• dual M.2 PCIe 4.0
• running Linux + ROCm

(Silicon figures are AMD's published specs for the Ryzen AI Max+ 395; the box itself is the Nimo build.) That unified 128 GB is the whole reason it's on my desk for this.

The workload

I'm building TernForge, a pipeline that converts full-precision LLMs into 1.58-bit ternary (BitNet b1.58: weights become {−1, 0, +1} × a scale, activations int8). The important part for this audience: it's not inference. It's retraining: surgery to replace every linear layer, then quantization-aware training (QAT) to heal the model back to something usable. That means holding the model, its full-precision "latent" weights, optimizer state, and activations in memory at once. Memory-hungry by design, which is exactly why the Nimo AI Mini PC is interesting.

Why this box

The 128 GB unified memory is the headline. Converting Qwen2.5-7B requires the model, its full-precision latent weights, the optimizer state, and checkpointed activations to be resident at once, and this takes up 67 GB in my run. That simply doesn't fit a consumer 8/16/24 GB discrete GPU, but it sits comfortably on the Nimo box with headroom to spare. Under the hood, that footprint is fp32 latent weights + fp32 gradients with a memory-frugal Adafactor optimizer (factored second moments and no momentum, the trick that sidesteps Adam's 2×-params optimizer state), a bf16-autocast forward, and gradient checkpointing, about 10–12 bytes/param. That's why the teacher-free 7B run peaks at 67 GB; adding the frozen bf16 teacher for distillation pushes it to 87 GB.

That ~256 GB/s of unified bandwidth is generous for a mini-PC but modest next to a datacenter GPU: plenty of room, moderate speed. The Nimo AI Mini PC is memory-rich and compute-modest, a great match for workloads gated by capacity rather than raw FLOPs. You can run real 7B-scale training experiments on a desk overnight. It gets better: the recipe that scales needs even more memory, so more on that at the end.

The bring-up (the part worth sharing)

Getting gfx1151 to run real training surfaced a few things you'll probably hit too:

1. Stock ROCm segfaults on gfx1151: use AMD's TheRock wheel. Both the stock PyTorch ROCm7.0 wheel and the system ROCm7.1 runtime crashed on the first kernel dispatch (inside libhsa-runtime64.so.1). I reproduced it with a plain native-HIP program built by the system hipcc, which pins it on the system HSA runtime, not PyTorch—the fix: the TheRock gfx1151-specific wheel (torch 2.10.0+rocm7.13), which bundles its own gfx1151 ROCr. GEMM and backward, then ran clean. Don't fight stock ROCm on this silicon.

2. fp32 matmul is ~100× slower than bf16. Raw 4096³ GEMM measured 0.38 TFLOP/s in fp32 vs 37.5 TFLOP/s in bf16. gfx1151 (RDNA 3.5) has fast bf16/fp16 matrix (WMMA) units but no fast fp32 matrix path: WMMA takes fp16/bf16/int8 inputs, so fp32 GEMM falls back to the vector ALUs. The architecture is why bf16 wins; the ~100× magnitude is mostly a software artifact, not a hardware ratio. The RDNA 3.5 peak-FLOP gap is single-digit, and 0.38 TFLOP/s is low enough to point at an immature, unoptimized fp32 GEMM path on gfx1151 rather than a 100× silicon deficit. My FP32 training step ran ~170 s/step; BF16 autocast (keeping master weights and the quant math in FP32) dropped it to ~21.6 s/step, about 8× end-to-end. I validated bf16 against the fp32 loss curve on a small model first, so I knew it wasn't quietly changing the result. On the Nimo AI Mini PC, bf16 is not optional: it's the difference between "overnight" and "next week."

3. Mind the disk. A full-precision 7B latent checkpoint is ~30 GB; a near-full root partition will bite you mid-run. Stage outputs on a secondary drive.

The first real answer, honestly

The first thing I ran at 7B scale was the simplest recipe, teacher-free quantization-aware training, where the model heals from its own loss with no help. The hardware and pipeline worked: 67 GB peak, stable, recovered from the ternarization shock, ran overnight. The quality result was a deliberate, pre-committed NO-GO. At a tiny ~4M-token budget, the model didn't come close to recovering (student perplexity 14,418 vs the full-precision teacher's 8.82). That was a research finding, not a hardware problem, and it led me to look for a better recipe. The box did its job. It let me get a real, gated answer at 7B scale on a desk.

The experiment: two recipes, 360M → 7B

So I compared the simple recipe against a heavier one, at two sizes, each judged honestly against the original full-precision model (perplexity ratio and top-1 agreement on held-out text, not against a copy of itself):

• Recipe A: teacher-free. Just quantization-aware training; the model heals from its own loss.
• Recipe B: distillation. A frozen full-precision teacher rides along, and the ternary student learns to match its outputs.

I started small on 360M (that baseline is tiny enough to run on an 8 GB laptop GPU, 4.5 GB peak), then used the Nimo AI Mini PC for the distillation comparison and the full 7B runs:

model	recipe	perplexity vs teacher	top-1 agreement
360M	A (teacher-free)	192×	9.08%
360M	B (distillation)	186×	9.88%
7B	A (teacher-free)	1,635×	4.0%
7B	B (distillation)	550×	8.66%

Two things stood out. At 360M, the recipes are close (both are undertrained at this tiny budget). At 7B, they diverge hard: teacher-free QAT gets worse as the model grows (top-1 falls 9.08% → 4.0%), while distillation holds roughly flat (9.88% → 8.66%) and lands ~3× better on perplexity. In other words, the naive recipe has a negative size-scaling problem, and the teacher fixes it, which is the whole reason to bother with the heavier setup. This isn't a new claim: Microsoft's BitNet Distillation targets the same scale-dependent gap between finetuned full-precision and 1.58-bit models, and fixes it with distillation (plus continual pretraining) on off-the-shelf models like Qwen. What I'm adding isn't the direction. It's that the whole loop runs end-to-end at 7B on a desk.

Honest caveat: these are all budget-limited runs (~0.5M–4M tokens vs the ~10B a real recovery needs), so none of these models is actually good yet. This was a direction-and-scaling result ("distillation is the right path as size grows"), not a quality claim.

Where the 128 GB actually earns its keep

Here's the Nimo-specific payoff. Distillation means holding two 7B models resident at once: the frozen full-precision teacher and the ternary student (plus its fp32 latent weights, optimizer state, and the distillation machinery). That peaked at 87 GB. On any consumer GPU, that's a non-starter; you'd either have to shard across multiple cards or rent a datacenter GPU. On the Nimo box, it just… fits, with room to spare. The recipe that scales is the one that needs the memory this box has—a clean fit, not a coincidence.

Takeaways

• 128 GB unified memory is the real ML feature. It runs training footprints (especially teacher+student distillation) that no consumer discrete GPU fits.
• Install the TheRock gfx1151 wheel. Stock ROCm currently segfaults on first dispatch.
• Use bf16. The fp32 matrix path is a ~100× trap on RDNA 3.5.
• A capable ML dev box for memory-bound work, if you handle the ROCm bring-up.

Happy to answer questions about the setup, especially the ROCm bring-up notes. These were single-seed runs on one box; your mileage may vary, but the TheRock + bf16 lessons should generalize. Not sponsored: I bought the box myself.

Saw this setup shared by someone in a Discord community and it got me thinking. If you were starting from scratch today, would you go with a gaming laptop or build a desktop? Feels like both sides have gotten way better lately, so curious what everyone would pick.

With new GPUs being expensive, I keep seeing more people go second-hand (especially last gen high end cards). And i noticed that many used GPUs have sold on Amazon and AE or anything else. On paper it makes sense, like it have better price/performance, and maybe still strong raw power, or that has tons of listings after upgrades. But also have these questions like the mining history concerns/ unknown wear/ no warranty etc. And tbh, would you buy a used high end GPU today or only go new?