Does anyone with an RK3588 board want to help me find out if a distributed LLM inference board is worth building? (one script, ~15 min, no root)
I'm designing an open-hardware board that runs frontier-scale Mixture-of-Experts models by streaming expert weights off NVMe instead of holding them in RAM. Design stage , nothing has been built, and I'm trying hard not to build the wrong thing.
The whole design rests on one number I can't get without hardware: real NVMe read throughput on RK3588, at large block sizes, with O_DIRECT. My architecture assumes ~3.2 GB/s on a PCIe 3.0 x4 link. If that's actually 1.5 GB/s in practice, most of my timing figures are wrong and I'd rather find out now than after ordering PCBs.
I don't own an RK3588 board yet. Plenty of you do.
The ask: run one Python script on your board and paste the output. It:
- is a single file, stdlib-only , no pip install, no repo clone
- needs no root (it'll tell you what root would additionally buy)
- is read-only apart from one temp file it creates and deletes
- takes ~15 minutes, most of which is a sustained thermal test
- prints one block you can paste back
What it measures:
- Whether your CPU has SDOT (
asimddp) , decides if llama.cpp's fast quantized kernels work - NVMe sequential + random read at 16 MB blocks, with and without O_DIRECT
- PCIe link width and speed actually negotiated (I especially want to know if anyone's board reports x1 when it should be x4)
- Sustained CPU frequency and temperature over 10 minutes, not burst , I care about the throttled number, not the marketing number
What's in it for you: you get a straight comparison of your board's real storage and thermal behaviour against everyone else's, which is annoyingly hard to find. I'll post the collated results back here regardless of what they say, including if they kill the idea.
Boards I'd love numbers from: Orange Pi 5 / 5 Plus / 5 Max, Radxa Rock 5B / 5C, FriendlyElec NanoPC-T6 / CM3588, ArmSoM, Banana Pi ,and honestly any RK3588 variant, plus non-RK3588 boards for comparison.
Repo (all design docs, MIT/CERN-OHL-P): https://github.com/Techburst36/swarm (Apologies in advance as the repo is a bit of a mess right now and might still have a few conflicting figures while the docs catch up.)
Script: swarm_bench.py in the repo root, plus a README explaining exactly what it does before you run it. Please do read it first , you shouldn't run a stranger's script on faith, and it's short enough to skim.
Happy to answer anything about the design in the comments.
Edit: made some changes as per u/fortean and u/12345myluggage feedback. The test had some real bugs that needed catching and some improvements to be made. Thank you!