r/VFIO

I've got debian running in virt manager, and it seems to only run with spice audio.

Whenever I try to enable pipewire pass through with this https://wiki.archlinux.org/title/PCI_passthrough_via_OVMF#Passing_audio_from_virtual_machine_to_host_via_PipeWire_directly it just tells me the host is down, but I know it's up when I check with sysemctl.

I'm going a bit nuts, if someone knows what I'm doing wrong I'd jump for joy.

Not sure why but my Nvidia 4070 says I am using a simple frame buffer
In the past I was able to use my hooks to do gpu passthrough, after bricking my build I started over with arch Linux zen kernel 7.* and I am not able to use the hooks I kept because it says I’m using simple frame buffer.

I tried to install the nvidia driver from AUR for my drive and it confirmed that I have switched however I still can’t passthrough anymore.

Hello; I Fortnite on KVM with single GPU passthrough using a patched QEMU and EDK II. However I’m getting a lot of lag and only around 40–60 FPS. How could I optimize my XML configuration or anything else ?

<domain type="kvm">

<name>win11</name>
<uuid>01c59c89-26d0-446f-89b6-fc38fa447e1b</uuid>
  <metadata>

<libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
<libosinfo:os id="http://microsoft.com/win/11"/>
</libosinfo:libosinfo>
  </metadata>

<memory unit="KiB">16775168</memory>
<currentMemory unit="KiB">16775168</currentMemory>
<vcpu placement="static">4</vcpu>
  <cputune>

<vcpupin vcpu="0" cpuset="2"/>

<vcpupin vcpu="1" cpuset="3"/>

<vcpupin vcpu="2" cpuset="4"/>

<vcpupin vcpu="3" cpuset="5"/>

<emulatorpin cpuset="0-1"/>

  </cputune>

  <sysinfo type="smbios">

<bios>

<entry name="vendor">American Megatrends Inc.</entry>
<entry name="version">F31o</entry>
<entry name="date">12/03/2020</entry>
</bios>

<system>

<entry name="manufacturer">Gigabyte Technology Co., Ltd.</entry>
<entry name="product">X570 AORUS ULTRA</entry>
<entry name="version">Rev 1.0</entry>
<entry name="serial">163058293563</entry>
<entry name="uuid">01c59c89-26d0-446f-89b6-fc38fa447e1b</entry>
<entry name="sku">To be filled by O.E.M.</entry>
<entry name="family">X570 MB</entry>
</system>

  </sysinfo>

  <os>

<type arch="x86\_64" machine="pc-q35-11.0">hvm</type>
<loader readonly="yes" secure="yes" type="pflash" format="raw">/home/kaizaki/edk2/Build/OvmfX64/DEBUG_GCC5/FV/OVMF_CODE.fd</loader>
<nvram template="/home/kaizaki/edk2/Build/OvmfX64/DEBUG\_GCC5/FV/OVMF\_VARS.fd" templateFormat="raw" format="raw">/var/lib/libvirt/qemu/nvram/win11_VARS.fd</nvram>
<boot dev="hd"/>

  </os>

  <features>

<acpi/>

<apic/>

<hyperv mode="custom">

<relaxed state="on"/>

<vapic state="on"/>

<spinlocks state="on" retries="8191"/>

<vpindex state="on"/>

<synic state="on"/>

<stimer state="on"/>

<frequencies state="on"/>

</hyperv>

<kvm>

<hidden state="on"/>

</kvm>

<vmport state="off"/>

<smm state="on"/>

  </features>

  <cpu mode="host-passthrough" check="none" migratable="on">

<topology sockets="1" dies="1" clusters="1" cores="4" threads="1"/>

<feature policy="disable" name="smep"/>

<feature policy="disable" name="hypervisor"/>

<feature policy="require" name="vmx"/>

<feature policy="require" name="invtsc"/>

<feature policy="require" name="aes"/>

  </cpu>

  <clock offset="localtime">

<timer name="rtc" tickpolicy="catchup"/>

<timer name="pit" tickpolicy="delay"/>

<timer name="hpet" present="no"/>

<timer name="hypervclock" present="yes"/>

<timer name="tsc" present="yes" mode="native"/>

  </clock>

<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
  <pm>

<suspend-to-mem enabled="no"/>

<suspend-to-disk enabled="no"/>

  </pm>

  <devices>

<emulator>/home/kaizaki/qemu/build/qemu-system-x86_64</emulator>
<disk type="file" device="disk">

<driver name="qemu" type="raw"/>

<source file="/home/kaizaki/KVM/win11.img"/>

<target dev="sda" bus="sata"/>

<address type="drive" controller="0" bus="0" target="0" unit="0"/>

</disk>

<disk type="file" device="cdrom">

<driver name="qemu" type="raw"/>

<target dev="sdb" bus="sata"/>

<readonly/>

<address type="drive" controller="0" bus="0" target="0" unit="1"/>

</disk>

<disk type="file" device="disk">

<driver name="qemu" type="raw"/>

<source file="/mnt/bcache/Jeu/fortnite.img"/>

<target dev="sdc" bus="sata"/>

<address type="drive" controller="0" bus="0" target="0" unit="2"/>

</disk>

<controller type="usb" index="0" model="qemu-xhci" ports="15">

<address type="pci" domain="0x0000" bus="0x02" slot="0x00" function="0x0"/>

</controller>

<controller type="pci" index="0" model="pcie-root"/>

<controller type="pci" index="1" model="pcie-root-port">

<model name="pcie-root-port"/>

<target chassis="1" port="0x10"/>

<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x0" multifunction="on"/>

</controller>

<controller type="pci" index="2" model="pcie-root-port">

<model name="pcie-root-port"/>

<target chassis="2" port="0x11"/>

<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x1"/>

</controller>

<controller type="pci" index="3" model="pcie-root-port">

<model name="pcie-root-port"/>

<target chassis="3" port="0x12"/>

<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x2"/>

</controller>

<controller type="pci" index="4" model="pcie-root-port">

<model name="pcie-root-port"/>

<target chassis="4" port="0x13"/>

<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x3"/>

</controller>

<controller type="pci" index="5" model="pcie-root-port">

<model name="pcie-root-port"/>

<target chassis="5" port="0x14"/>

<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x4"/>

</controller>

<controller type="pci" index="6" model="pcie-root-port">

<model name="pcie-root-port"/>

<target chassis="6" port="0x15"/>

<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x5"/>

</controller>

<controller type="pci" index="7" model="pcie-root-port">

<model name="pcie-root-port"/>

<target chassis="7" port="0x16"/>

<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x6"/>

</controller>

<controller type="pci" index="8" model="pcie-root-port">

<model name="pcie-root-port"/>

<target chassis="8" port="0x17"/>

<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x7"/>

</controller>

<controller type="pci" index="9" model="pcie-root-port">

<model name="pcie-root-port"/>

<target chassis="9" port="0x18"/>

<address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x0" multifunction="on"/>

</controller>

<controller type="pci" index="10" model="pcie-root-port">

<model name="pcie-root-port"/>

<target chassis="10" port="0x19"/>

<address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x1"/>

</controller>

<controller type="pci" index="11" model="pcie-root-port">

<model name="pcie-root-port"/>

<target chassis="11" port="0x1a"/>

<address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x2"/>

</controller>

<controller type="pci" index="12" model="pcie-root-port">

<model name="pcie-root-port"/>

<target chassis="12" port="0x1b"/>

<address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x3"/>

</controller>

<controller type="pci" index="13" model="pcie-root-port">

<model name="pcie-root-port"/>

<target chassis="13" port="0x1c"/>

<address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x4"/>

</controller>

<controller type="pci" index="14" model="pcie-root-port">

<model name="pcie-root-port"/>

<target chassis="14" port="0x1d"/>

<address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x5"/>

</controller>

<controller type="sata" index="0">

<address type="pci" domain="0x0000" bus="0x00" slot="0x1f" function="0x2"/>

</controller>

<controller type="virtio-serial" index="0">

<address type="pci" domain="0x0000" bus="0x03" slot="0x00" function="0x0"/>

</controller>

<interface type="network">

<mac address="52:54:00:f0:93:fd"/>

<source network="default"/>

<model type="e1000e"/>

<address type="pci" domain="0x0000" bus="0x01" slot="0x00" function="0x0"/>

</interface>

<serial type="pty">

<target type="isa-serial" port="0">

<model name="isa-serial"/>

</target>

</serial>

<console type="pty">

<target type="serial" port="0"/>

</console>

<input type="tablet" bus="usb">

<address type="usb" bus="0" port="1"/>

</input>

<input type="mouse" bus="ps2"/>

<input type="keyboard" bus="ps2"/>

<sound model="ich9">

<address type="pci" domain="0x0000" bus="0x00" slot="0x1b" function="0x0"/>

</sound>

<audio id="1" type="none"/>

<hostdev mode="subsystem" type="pci" managed="yes">

<source>

<address domain="0x0000" bus="0x03" slot="0x00" function="0x0"/>

</source>

<rom file="/usr/share/vgabios/rx-6600.rom"/>

<address type="pci" domain="0x0000" bus="0x05" slot="0x00" function="0x0"/>

</hostdev>

<hostdev mode="subsystem" type="pci" managed="yes">

<source>

<address domain="0x0000" bus="0x03" slot="0x00" function="0x1"/>

</source>

<address type="pci" domain="0x0000" bus="0x06" slot="0x00" function="0x0"/>

</hostdev>

<hostdev mode="subsystem" type="usb" managed="yes">

<source startupPolicy="mandatory">

<vendor id="0x1e7d"/>

<product id="0x2cf1"/>

</source>

<address type="usb" bus="0" port="2"/>

</hostdev>

<hostdev mode="subsystem" type="usb" managed="yes">

<source startupPolicy="mandatory">

<vendor id="0x03f0"/>

<product id="0x4941"/>

</source>

<address type="usb" bus="0" port="3"/>

</hostdev>

<hostdev mode="subsystem" type="usb" managed="yes">

<source startupPolicy="mandatory">

<vendor id="0x1b1c"/>

<product id="0x1bc7"/>

</source>

<address type="usb" bus="0" port="4"/>

</hostdev>

<watchdog model="itco" action="reset"/>

<memballoon model="virtio">

<address type="pci" domain="0x0000" bus="0x04" slot="0x00" function="0x0"/>

</memballoon>

  </devices>

</domain>

Not sure if this is more related to r/VFIO or not, but I will post in both subreddits just in case. I do however feel like this is more of a Hackintosh issue.

So, I manages to install macOS Sequoia (but I checked everything from Big Sur onwards) in a VM, and everything works ok, but I want to pass through a GPU in order to enable video acceleration.

Now, the problem is that my GPU is an AMD E9173, basically an embedded version of the Lexa RX 550, with the GPU ID of 699F. I have found success stories with this GPU, by spoofing it to 67FF, so I decided to try it, and it worked, but not quite...

In my current scenario, if my GPU is not spoofed, it displays an output just fine, but with no graphics acceleration at all, which means that the pass through is working properly. However, once I spoof it through OpenCore Device Properties (but I have also tried with an SSDT), it no longer shows any output on the display once the drivers load in macOS. Basically, I see the Apple logo loading, and half way through, when the GPU drivers would load, the screen would simply go black and the monitor would show no signal. I can connect remotely to the VM using AnyDesk or Rustdesk, and I can see that the GPU is properly recognized as the spoofed 67FF and has full graphics acceleration enabled. And in benchmarks it does perform as expected, for the performance of the GPU.

I do however need a display output or, at the very least, a way to make macOS think there is one connected. I tried with "radcodec", "agdpmod=pikera", I tried adding the "no-gpu-spoof" parameter in Device Properties, and I even tried remapping the framebuffer, but nothing helped. I eventually installed Better Displays in order to create a virtual display to fix my resolution, but without a proper display connected, macOS seems to disable hardware acceleration for some apps...

Specifically, Parsec doesn't work at all right now, and I would need that, because both AnyDesk and RustDesk have a lot of latency and it's not comfortable at all to use the system like that. I tried installing Sunshine and it failed, and it seems very finicky on macOS, so I would like to avoid that, if possible. Right now, on Sequoia, Parsec crashes outright when I try to connect to it, and on Monterey and older I can connect, but it defaults to software encoding, which makes it even more unusable than AnyDesk or Rustdesk. My understanding is that I would need to at least make macOS think there is a display connected, but the GPU disables all outputs as soon as the macOS Graphics Driver loads...

So, what can I do? Is this a known issue, or is there something I am missing about this? Is there something I can do to make the spoofed GPU have proper display output once the driver loads? Or, at the very least, is it possible to make macOS think there is a real display connected, so Parsec could work properly? Thanks.

TL;DR AMD 699F GPU spoofed through OpenCore to 67FF drops all display outputs as soon as the drivers load. I can confirm that the drivers loaded properly by connecting remotely. I want to fix the display output or, at least, make Parsec work.

I had access to a few machines with different CPU architectures and GPU configurations and ran benchmarks on several KVM/libvirt GPU passthrough VMs to see how much vCPU pinning and guest NUMA topology actually matter.


TL;DR: vCPU pinning was usually the safest starting point. Exposing NUMA topology to the guest could unlock much higher CPU-side memory bandwidth on multi-NUMA hosts, but it was not a guaranteed win. In some multi-GPU cases it made collective bandwidth worse when traffic had to cross NUMA nodes through host memory.


Tested systems: H200, MI350X, RTX PRO 6000, RTX 4090, RTX 4080, RTX 5090.
Compared:
- default-ish libvirt/QEMU layout
- vCPU pinning only
- vCPU pinning + guest NUMA topology
- vCPU pinning + guest NUMA + SMT siblings


Takeaways:
- Guest NUMA exposure made a huge difference for CPU-side memory bandwidth. STREAM improved by roughly 2x to 7x on some multi-NUMA hosts.
- vCPU pinning alone helped latency/noise in some cases, but did not unlock the big memory-bandwidth gains by itself.
- Exposing SMT siblings mostly made the guest look bigger without making it much faster. CPU throughput only improved around 2 to 3 percent in the larger tests.
- GPU compute benchmarks, like FP16 matmul, barely changed across CPU/NUMA configs.
- NCCL was the tricky part. Plain vCPU pinning helped a lot on some systems, while guest NUMA hurt badly on others.
- NCCL was tested with `NCCL_P2P_DISABLE=1`, so these results emphasize host-memory / PCIe behavior rather than direct GPU-to-GPU links.
- One RTX 5090 setup was clearly misconfigured or platform-limited: PCIe bandwidth looked like Gen 2, so I treated those results as diagnostic rather than representative.


The i9-14900K desktop baseline did very well in some CPU benchmarks, especially single-thread/sysbench, where its boost behavior helped it beat older EPYC Zen 2 in certain cases.


In a nutshell, for GPU-heavy passthrough VMs, I’d start with `vcpu_pin` and keep the guest single-NUMA. I’d only expose guest NUMA after checking where the GPUs physically sit and confirming the workload actually benefits from the extra host memory bandwidth. I plan to write a follow-up post about algorithm of choosing the right vCPU/NUMA config based on the hardware and workload that takes into account other tenants on the host and hugepages.


Full write-up:
https://medium.com/itnext/gpu-vm-performance-do-vcpu-pinning-and-numa-topology-really-matter-1b2093f4b45a


Benchmark scripts:
https://github.com/6erun/vcpu_benchmarks