AMD/ROCm vs Nvidia/Cuda: how big a trade-off for a local RAG build?
I’m building a fully local RAG system for ~2000 complex documents, with a target of 3–5 concurrent users initially and eventually 5–10, using Open WebUI, Docling/custom PDF parsing, pgvector, BGE-M3/reranking, and local LLM serving. I’ve ruled out Apple/MLX as the main production path because concurrency is weak. I’m building a 3×AMD Instinct MIx ROCm server because the HBM capacity/bandwidth per dollar is much better than current NVIDIA workstation cards.
What I’m trying to understand from people who have actually run production-ish local LLM/RAG workloads on Instinct MI200/ROCm is: how big is the real “engineering friction” penalty versus CUDA today—model compatibility, vLLM/SGLang stability, quantization/kernel support, RCCL/multi-GPU issues, upgrades breaking things, etc.—and is that trade-off reasonable if the goal is sustained 5–10 user concurrency without spending RTX PRO 6000-class money?
My fallback is a capable but slow 2xDGX Spark/GB10 mainly as a CUDA-compatible reference and compact inference box**. Will appreciate your feedback/experiences before investing $20K-ish into this ROCm custom build.**
Thanks you!