I’m building a fully local RAG system for ~2000 complex documents, with a target of 3–5 concurrent users initially and eventually 5–10, using Open WebUI, Docling/custom PDF parsing, pgvector, BGE-M3/reranking, and local LLM serving. I’ve ruled out Apple/MLX as the main production path because concurrency is weak. I’m building a 3×AMD Instinct MIx ROCm server because the HBM capacity/bandwidth per dollar is much better than current NVIDIA workstation cards.

What I’m trying to understand from people who have actually run production-ish local LLM/RAG workloads on Instinct MI200/ROCm is: how big is the real “engineering friction” penalty versus CUDA today—model compatibility, vLLM/SGLang stability, quantization/kernel support, RCCL/multi-GPU issues, upgrades breaking things, etc.—and is that trade-off reasonable if the goal is sustained 5–10 user concurrency without spending RTX PRO 6000-class money?

My fallback is a capable but slow 2xDGX Spark/GB10 mainly as a CUDA-compatible reference and compact inference box**. Will appreciate your feedback/experiences before investing $20K-ish into this ROCm custom build.** 

Thanks you!

Before I get thrashed by people for asking a seemingly stupid question. I am new to this part of A.I. All the A.I stuff I've been doing is text-to-text (mostly coding). So be gentle please.

Recently I got Qwen-Image-Edit on my A.I server. It generates images fine without downloading ComfyUI. I set it up with a simple python server and it works fine.

This leads me to ask, why should I download ComfyUI? What does it do that can't be done with just a simple python script? Is it just the ability to visually connect lines between parts of the workflow? Note that I am a software engineer, and actually prefer command line interfaces for making stuff in most cases. Is there some other benefit I am missing?

Running Linux on dual Xeon system with 512gb of memory. How far have the tools for the 9700s come?

Help me decide between two cards for local AI messin':

• More common: AMD Radeon Pro card (RDNA 2.0) with 32GB of VRAM
  • Pros: Common, supported by a lot of software. More VRAM, cheaper.
  • Cons: Older architecture. Will likely sound like a jet engine strapping a fan to it.
• Less common: Tenstorrent p100a with 28GB of VRAM
  • Pros: New, mission dedicated local LLM card. Good vram, interesting alternative design. Quieter. Open Source.
  • Cons: New, less developed software ecosystem. Expensive.

The Tenstorrent card is a *LOT* more, but a lot more interesting to me. The Radeon Pro is more common and likely supported by more software... but a bit more boring.

Don't mention NVIDIA. I run Linux and NVIDIA cards always suck to use.

Looking for something that will be useful for 3-5 more years. No worried about graphics use.. this is a dedicated AI LLM card. (I currently have an RX 6700 XT which is fine... but most models don't fit in 12GB)

Buying my first dedicated inference GPU for an existing Proxmox homelab. I've narrowed it to three cards and keep going in circles, so I'd rather hear from people actually running them. I am very new to local AI and am mostly interested in this purchase to learn. Budget is up to $1500 for the card.

The box

Ryzen 7 9800X3D / ASUS ROG STRIX X870E-E / 32GB DDR5-6000 (two slots free) / Corsair HX1000i 1000W / PCIe 5.0 x16 available / Proxmox VE 9.2, kernel 7.0.14-11-pve.

Two things that matter more than the specs:

• It's on 24/7 and already runs my whole homelab, so idle power and driver stability are important.
• Everything runs in unprivileged LXC containers (today), not VMs. So passthrough for me means driver on the Proxmox host (correct me if I'm wrong when it comes to having a dedicated GPU here). The iGPU currently handles Jellyfin transcoding this way.

What I want to run

Mainly local LLM inference - general chat, coding help, and a homelab agent that monitors/reads logs and configs and suggests fixes. Secondary: Immich CLIP search and face recognition and possibly implementing one of the Paperless AI solutions.

Speed is not super important to me, but I'd like enough responsiveness that I'm not losing patience for a simple inquiry. The vast majority of my use case is monitoring services across Proxmox, my UNRAID NAS, a backup NAS, and my Unifi-based network. I'm assuming this is a fairly light load that smaller models can handle, but also want an LLM to be able to help me diagnose and troubleshoot.

The options

Used RTX 3090	RX 7900 XTX	Radeon AI PRO R9700
VRAM	24GB	24GB
Bandwidth	~936 GB/s	~960 GB/s
TDP	350W	355W
Price	~$1,000–1,100 used	~$700–900
Stack	CUDA	ROCm

My questions

1. Is 32GB at 640 GB/s actually better than 24GB at ~940 GB/s here? This is the whole decision. The R9700 fits a bigger model; the other two are faster on whatever fits. For a mixed agent + photo-ML workload, which way does that fall in practice?

2. How much CUDA-only pain am I avoiding by paying the NVIDIA tax?

3. Anyone running ROCm on a current Proxmox kernel? I'm on 7.0.14-11-pve.

4. Idle power, real numbers? With a model loaded and sitting there vs unloaded.

One AMD-specific worry: my iGPU is /dev/dri/renderD128 and that path is passed into my Jellyfin container. A second AMD card means two render nodes with no guaranteed stable numbering. A udev rule pinned to the PCI address might fix it, but is that a non-issue in practice, or a recurring annoyance?

Thank you in advance. I'm happy to answer questions about the setup.

Look, I'm probably the idiot here, but I really want to be able to use things like Kimi K3 with hardware I actually own. I suspect there may be other people that are also dreaming of this day, but ultimately have a single GPU dangling by a thread from their machine. So, I wonder if a group of us could put our GPUs into a cluster and essentially time-share our much larger cluster to actually run frontier-level models as a group? The cost is simple: you get your portion of the cluster's compute for free. If you want more, you essentially beg, borrow, or buy it.

Why hasn't this been done? Have I missed something in the market?

5090's used price been skyrocketing, then there's taobao sellers claiming to have the MI210 for just under RMB 20,000 ($3k) Is the MI210 good or it has to be bridged with quad cards to actually see some benefits?

As in the title. I came across a post here that referenced the blog post below :

https://blog.hellas.ai/blog/thunderbolt-ibverbs/

Are there any vLLM (or llama.cpp etc.) forks or at least early implementation attempts/ plans that make use of USB4STREAM?
Basically , it should allow to drastically reduce cross node latency(so improve inference) vs ordinary USB 4 (or non RDMA Ethernet network) as it removes the whole network stack overhead .

So e.g. with two PCs , equipped with USB 4 40Gbit ports you’d just need to get a cheap passive cable for pipeline parallelism to be viable solution.

llama.cpp RPC can be used for such , but standard Ethernet latency / overhead makes it far from ideal.

I have an M3 Ultra 512gb ram 1Tb, up for sale. Would accept a m5 max 128gb laptop + cash as well. DM to chat. No longer need cause I have 2 rtx pros for inference and prefill is mid on m3 ultra where TTFT is too slow for me to actually find usability in local models. I don’t have a firm asking price in mind, nothing as absurd as 25k , but enough to get an m5 max 128 Mbp and have some cash left over so maybe like 17k?