u/Shuck-in-jive

Dual GPU Passthrough, Host is claiming one of the two even tho I have blacklisted drivers

My background: I followed DigitalSpaceport's 24gb home AI server(dual rtx3060s) build but with an dual processor hp640(build used hp440). I installed nvidia driver on the host and lxc container, no problems. Super fun, both GPU's passed to LXC! Just messing around with llama3.2-vision I was getting 57/s tokens, and that's with V3 Xeon cpu's. Just upgrading to V4 will improve my memory speed.

So now I want to mess with some video gen with VM's I know I can't combine GPUs for video generation(like I did in llama language models...) so I wanted to create two VMs, and pass each GPU to each... then just do parallel seperate encoding with the two VMs... Well, the host keeps claiming one of the GPUs and it something stoopid simple... I've only done the dual gpu passthrough once with the digitalspaceport guide so I'm new to this. I have added some screenshots of commands, and there's one screen of my previous successful lxc dual passthrough too.

  • I created a blacklist file(/etc/modprobe.d/blacklist.conf see pic) with no success, all the tutorials(dated 2023-2025) online say to blacklist the nvidia drivers so the host doesn't claim one of the gpu's... then on fb someone recently said I don't need to blacklist... only as a last resort. I'm confused.
  • each gpu is in its own iommu group(see pics)
  • I vaguely remember adding the gpu id as a Options parameter? But now I can't find that in the original tutes I followed. Do I need to register the gpu id's somewhere?
  • I'm using nvtop & lspci -v(kernel driver in use:) to verify passthrough... are the correct modules loading?(kernel modules:).

The solution is something simple and its driving me nuts.
Please help.

EDIT: sorry, some images aren't uploading... reddit you suck.

https://preview.redd.it/k1mwkdh8fc1h1.jpg?width=677&format=pjpg&auto=webp&s=066234aa86179b4be16f57d5589ba35eafcdb206

https://preview.redd.it/mtph67h8fc1h1.jpg?width=1351&format=pjpg&auto=webp&s=849ee4dc13a9082a6e9a09f879cf241c21993ec5

https://preview.redd.it/seb3z4h8fc1h1.jpg?width=701&format=pjpg&auto=webp&s=bdb10f9f2ebfc559fdbbe008eecd3bf0118875b8

https://preview.redd.it/311g54h8fc1h1.jpg?width=1324&format=pjpg&auto=webp&s=b3b5ab65f182cdc197b28ddfdf3d5fc39c714198

https://preview.redd.it/o1py35h8fc1h1.jpg?width=1113&format=pjpg&auto=webp&s=32b1f7905f0ca5fcd667d336f6da2f26fbb89465

reddit.com
u/Shuck-in-jive — 3 months ago