I read that quite a few people have issues with running linux on their macpro for local inference. I can't comment on Ubuntu or other distros because all my machines run Nixos but since it works flawlessly, I thought I'd share my repo in case that inspires anyone to try something similar.

For those who don't know, Nixos allows you to configure your computer in a deterministic way. You write your config (in the nix language), referencing nix-packages. Nix-packages have sets of options that you use in your config files. There are other benefits to Nixos but this isn't the topic here. What I think is the main benefit is that I can comment out a line in my config file, change that option to something else and leverage git for version control. If I break something, I can choose a previous (working) generation of the system at boot.

In this setup, I use llama-swap to let me manage models on the fly, SearchXNG module for web search, OpenWeb UI for chat and user friendly automation/agents, Nixos MCP so my coding agents can manage my config files accurately.

You can see the models I'm currently running llama-swap.nix file.

Link to repo

PS: I only serve my LAN so security is tailored to that, meaning it's not hardened as much as it could be.

--------------------------------

Extract from the Readme (written by Qwen}:

NixOS configuration for donnager, a headless Mac Pro 7,1 (T2) running as a local LLM inference server.

Hardware

• Mac Pro 7,1 (2019), T2 chip — T2-patched kernel via nixos-hardware apple-t2
• AMD Radeon Pro Vega II (Vulkan/RADV compute for llama.cpp)
• Wired 10GbE, behind a NAT router (the LAN is the trust boundary)

Services

Service	Port	Notes
SSH	22	keys only, no root login
open-webui	3000	browser UI, password auth, web search via searxng
mcp-nixos	8001	NixOS MCP server (HTTP), for pi on the LAN
searxng	8888	private metasearch; secret key via agenix, limiter off
llama-swap	9292	model router for llama-server (Vulkan); OpenAI-compatible

Models live in /var/lib/llama/models/ (not in git — see .gitignore). llama-swap unloads models after 15 min idle to free VRAM; each model pins its own context size / quantization / chat template (Qwen uses the pinned froggeric fixed chat template, fetched by hash).

Fans are driven by t2fanrd (the Vega II is passively cooled; T2 case fans are the only cooling).NixOS configuration for donnager, a headless Mac Pro 7,1 (T2) running as a
local LLM inference server.
Hardware
Mac Pro 7,1 (2019), T2 chip — T2-patched kernel via nixos-hardware apple-t2
AMD Radeon Pro Vega II (Vulkan/RADV compute for llama.cpp)
Wired 10GbE, behind a NAT router (the LAN is the trust boundary)
Services
Service Port Notes
SSH 22 keys only, no root login
open-webui 3000 browser UI, password auth, web search via searxng
mcp-nixos 8001 NixOS MCP server (HTTP), for pi on the LAN
searxng 8888 private metasearch; secret key via agenix, limiter off
llama-swap 9292 model router for llama-server (Vulkan); OpenAI-compatible
Models live in /var/lib/llama/models/ (not in git — see .gitignore).
llama-swap unloads models after 15 min idle to free VRAM; each model pins its
own context size / quantization / chat template (Qwen uses the pinned
froggeric fixed chat template, fetched by hash).
Fans are driven by t2fanrd (the Vega II
is passively cooled; T2 case fans are the only cooling).

80% GPU offloaded across 3 AMD GPUs (56GB VRAM), remainder CPU/RAM offloaded. RX7900XTX RX6900XT internal and RX6800XT eGPU thunderbot. Bootiting Bazzite from an Acasis TB501 pro. I am just tinkering around and wanted to see if I could get all three of these working.

https://preview.redd.it/ut400ivmf7jh1.png?width=490&format=png&auto=webp&s=c6a6be0deaaa06a7bed266e9ce9262b64352912c

Native vLLM + ROCm 7.15 Runtime for RX 6000 (RDNA2) on Windows 11 — 25.9 TFLOPS FP16, 54.2 tok/s, No WSL2 [RX 6750 XT gfx1031 Verified]

I built a native vLLM + ROCm 7.15 runtime for AMD RX 6000 Series on Windows 11.

AMD lists RX 6750 XT / 6700 XT / 6600 XT as "Runtime only" on Windows with HIP SDK excluded. I built rocBLAS binaries for gfx1031 via ROCm/TheRock to close that gap.

This runs native Windows HIP and ROCm directly, no WSL2 wrapper.

Tested on AMD Radeon RX 6750 XT 12GB (gfx1031) — Windows 11 Native — August 2026

Verification — Real terminal logs

1. Environment:

   torch.compile disabledtorch 2.12.0+rocm7.15.0a20260728 | cuda_avail True | dev AMD Radeon RX 6750 XTAvailable plugins for group vllm.platform_plugins:- windows_rocm -> vllm_windows_rocm:registerPlatform plugin windows_rocm is activated

2. rocBLAS Benchmark — 25.9 TFLOPS FP16:

   Device ID 0 : AMD Radeon RX 6750 XT gfx1031with 12.9 GB memory, max. SCLK 2495 MHz, max. MCLK 1125 MHzrocBLAS version: 5.7.0.67811f1ee52transA,transB,M,N,K,alpha,lda,beta,ldb,ldc,cold_iters,hot_iters,rocblas-Gflops,usN,N,4096,4096,4096,1,4096,0,4096,4096, 2, 10, 25977.3, 5290.73=> 25.97 TFLOPS in 5.29ms

3. vLLM Inference — FIRST_TOKEN_OK:

   Loading model: facebook/opt-125mAvailable KV cache memory: 5.47 GiB / 159,264 tokens / 311x concurrencyPROMPT: 'Hello, my name is'OUTPUT: ' J.C. and I am a student at the University of California, Berkeley. I am a graduate...'FIRST_TOKEN_OKInput: 10.16 tok/s Output: 54.20 tok/s | Init 1.84s

Full logs in benchmarks/ and screenshots in assets/ on GitHub.

How it works

1. TheRock builds clr (HIP) and rocBLAS with Tensile kernels for gfx1031
2. HSA_OVERRIDE_GFX_VERSION=10.3.1 forces HIP to recognize RX 6750 XT
3. PyTorch 2.12.0+rocm7.15 links against TheRock runtime => torch.cuda.is_available() True
4. vLLM plugin vllm_windows_rocm bypasses vllm._C dependency and registers WinRocmAwqGemvKernel with TRITON_ATTN
5. vLLM engine loads with enforce_eager=True and runs native

Quick Start

Prerequisites: Windows 11 23H2+, RX 6600-6750 XT (gfx1030/gfx1031), ROCm 7.15 TheRock at C:\TheRock, Python 3.11+ TheRock venv, Adrenalin 24.x+

Release Zip (708MB minimal runtime):

1. Download ROCm_VLLM_Runtime_RDNA2_Windows.zip from GitHub Releases and extract to C:\TheRock\
2. Run setup.bat as Administrator
3. Run run.bat — Expected: FIRST_TOKEN_OK

Fixes in v1.1.0

• Fixed zmq.error.ZMQError: Protocol not supported (ipc://...) on Windows — forced VLLM_ENABLE_V1_MULTIPROCESSING=0 to use TCP
• Fixed socket error 10049 — set MASTER_ADDR=127.0.0.1 MASTER_PORT=29500
• Repo cleaned to <5MB main + 708MB release

Known Issues — Please note this may have bugs

This is an experimental reference implementation. It works on my RX 6750 XT but could present bugs on other hardware.

• Other RDNA2 cards (6600/6600 XT/6700 XT) not yet tested — may need 10.3.0 vs 10.3.1 or rocBLAS rebuild for gfx1030
• enforce_eager=True required — torch.compile disabled on RDNA2 Windows
• FP8 / AWQ not tested yet, multi-GPU not tested
• If you test it on your RDNA2 card, please open an Issue with GPU model and logs — contributions welcome

GitHub Repo:

https://github.com/sebastianmechno-sys/vllm-rocm-windows-rdna2

Includes setup.bat, run.bat, inference.py, assets with screenshots, benchmarks logs, docs/BUILD_ROCBLAS.md

Let me know if you test it on other RDNA2 cards.

Built on ROCm/TheRock, PyTorch ROCm, vLLM. Not affiliated with AMD. License Apache 2.0