I read that quite a few people have issues with running linux on their macpro for local inference. I can't comment on Ubuntu or other distros because all my machines run Nixos but since it works flawlessly, I thought I'd share my repo in case that inspires anyone to try something similar.

For those who don't know, Nixos allows you to configure your computer in a deterministic way. You write your config (in the nix language), referencing nix-packages. Nix-packages have sets of options that you use in your config files. There are other benefits to Nixos but this isn't the topic here. What I think is the main benefit is that I can comment out a line in my config file, change that option to something else and leverage git for version control. If I break something, I can choose a previous (working) generation of the system at boot.

In this setup, I use llama-swap to let me manage models on the fly, SearchXNG module for web search, OpenWeb UI for chat and user friendly automation/agents, Nixos MCP so my coding agents can manage my config files accurately.

You can see the models I'm currently running llama-swap.nix file.

Link to repo

PS: I only serve my LAN so security is tailored to that, meaning it's not hardened as much as it could be.

--------------------------------

Extract from the Readme (written by Qwen}:

NixOS configuration for donnager, a headless Mac Pro 7,1 (T2) running as a local LLM inference server.

Hardware

• Mac Pro 7,1 (2019), T2 chip — T2-patched kernel via nixos-hardware apple-t2
• AMD Radeon Pro Vega II (Vulkan/RADV compute for llama.cpp)
• Wired 10GbE, behind a NAT router (the LAN is the trust boundary)

Services

Service	Port	Notes
SSH	22	keys only, no root login
open-webui	3000	browser UI, password auth, web search via searxng
mcp-nixos	8001	NixOS MCP server (HTTP), for pi on the LAN
searxng	8888	private metasearch; secret key via agenix, limiter off
llama-swap	9292	model router for llama-server (Vulkan); OpenAI-compatible

Models live in /var/lib/llama/models/ (not in git — see .gitignore). llama-swap unloads models after 15 min idle to free VRAM; each model pins its own context size / quantization / chat template (Qwen uses the pinned froggeric fixed chat template, fetched by hash).

Fans are driven by t2fanrd (the Vega II is passively cooled; T2 case fans are the only cooling).NixOS configuration for donnager, a headless Mac Pro 7,1 (T2) running as a
local LLM inference server.
Hardware
Mac Pro 7,1 (2019), T2 chip — T2-patched kernel via nixos-hardware apple-t2
AMD Radeon Pro Vega II (Vulkan/RADV compute for llama.cpp)
Wired 10GbE, behind a NAT router (the LAN is the trust boundary)
Services
Service Port Notes
SSH 22 keys only, no root login
open-webui 3000 browser UI, password auth, web search via searxng
mcp-nixos 8001 NixOS MCP server (HTTP), for pi on the LAN
searxng 8888 private metasearch; secret key via agenix, limiter off
llama-swap 9292 model router for llama-server (Vulkan); OpenAI-compatible
Models live in /var/lib/llama/models/ (not in git — see .gitignore).
llama-swap unloads models after 15 min idle to free VRAM; each model pins its
own context size / quantization / chat template (Qwen uses the pinned
froggeric fixed chat template, fetched by hash).
Fans are driven by t2fanrd (the Vega II
is passively cooled; T2 case fans are the only cooling).