u/soflgolf

**The setup**
4× Mac Studio M3 Ultra (Mac15,14), 256 GB each
Fully connected Thunderbolt 5 mesh — all 6 cables, verified
macOS 26.4.1 on all nodes
RDMA enabled via Recovery (rdma\_ctl enable), confirmed with ibv\_devices
MLX 0.31.2, mlx-lm with the JACCL backend
Goal: tensor-parallel inference of a \~600 GB 8-bit MoE model (1T-param class) sharded across the four nodes
**What works**
mlx.distributed\_config --backend jaccl --over thunderbolt --auto-setup completes cleanly, assigns all the point-to-point IPs, writes a full-mesh hostfile
ibv\_devinfo shows 3× PORT\_ACTIVE per node — full mesh, no bridge0, no EXO virtual interfaces
Model converted to MLX format, synced to all 4 nodes, paths verified
GPU wired limit raised (iogpu.wired\_limit\_mb=229376) so the shard fits
**What fails**
JACCL init never connects. Both the model launch *and* a trivial 10-element all\_sum smoke test die identically:
\[jaccl\] Connection attempt 0 waiting 1000 ms
... (backs off 1s/2s/4s/8s)
RuntimeError: \[jaccl\] Couldn't connect (error: 60)
error: 60 = ETIMEDOUT. It fails at mx.distributed.init(backend="jaccl") — before any weights load, on a zero-memory-pressure test. So it's not OOM, not the model, not the hostfile, not paths. It's the RDMA queue-pair handshake itself.
**What we tried (in order)**
Regenerated the hostfile from scratch with --auto-setup — clean output, full mesh, same errno 60
Found one TB link (en4, the c2↔c3 pair) had come up PORT\_DOWN after a reboot — reseated the cable, confirmed all 3 ports PORT\_ACTIVE on every node — still errno 60
Suspected Protection Domain exhaustion (per mlx-lm #955: PDs leak across init/teardown cycles, only a reboot reclaims them) — full reboot of all 4 nodes — still errno 60
Found EXO auto-starting and creating virtual Thunderbolt interfaces that pollute the topology (per exo #1843) — disabled io.exo.networksetup and booted out the EXO launch agents, confirmed 0 EXO interfaces — still errno 60
Clean-boot 10-element all\_sum over JACCL, every variable controlled — **still errno 60**
That last one is the kicker: a clean reboot, no leaked PDs, no EXO, full active mesh, freshly generated hostfile, ten floats — and JACCL still can't bring up queue pairs.
**What we found researching it**
JACCL's reliability across macOS 26.x looks *inconsistent*, not a clean regression:
errno 60 RTR failures on 26.4.1 — open, unresolved (exo #1973)
RTR errno 22 failures on **26.2** (exo #1390, build 25C56)
RDMA-won't-init reports on 26.2 (exo #1004, mlx #2944)
But a documented **success** on 26.3.1 (705 GB transferred at 3.8 GB/s, mlx #3207)
So it's not "version X broke it" — it reads like an immature feature with config/timing-sensitive init failures that some people clear and others don't. We considered an IPSW rollback (26.2 is still signed for Mac15,14) but backed off, since 26.2 has its own JACCL failure reports — rolling back doesn't look like a reliable fix.
**Where we landed (tentative)**
Ring backend (TCP over Thunderbolt) is the fallback we're about to validate — same mesh, same cabling, just higher latency than RDMA. Treating JACCL as "revisit when it stabilizes."
**The ask**
Is anyone running JACCL **stably** on 4 nodes, and on which exact macOS build? Trying to find a known-good version for Mac15,14.
Has anyone gotten *past* errno 60 at init — and if so, what actually fixed it? (Coordinator port/firewall? A specific MLX version? An init barrier? Something in the RDMA enable step?)
Is errno 60 at init (vs. the errno 22 / RTR-state errors others report) a distinct failure mode with a distinct cause?
Anyone confirm JACCL working — or still broken — on 26.5.x?
Happy to share full logs, hostfile, and ibv\_devinfo output. Hardware and config are verified sound; the blocker is purely JACCL RDMA init.