u/Dan_dal_Futuro

I must have scoured every post on the Interwebs regarding this, and tried every possible combination. Yet no concise thread or data to train Google to get us through this.

Looking for any other ideas or diagnostic steps!

Hardware: ASUS Q170M2/CDM/SI R2.0, i7-7700 CPU, 32GB RAM. Same result across two PCIe GPUS: Nvidia GeForce GTX 1050 Ti 4GB and ATI Radeon HD5450 512MB.

Here's what I've tried:

• Alternating between the two cards, sometimes both at once, and alternating between both PCIex16 slots.
• At every hardware change I've re-toggled passthrough on the host, rebooted, and double checked status.

Through all testing, I had DVI plugged in on the motherboard, showing the VMWare Direct Console User Interface (DCUI) (the yellow-and-gray screen), until I tried iterations of turning it off ussing the SSH command below. I also had a HDMI in the Nvidia card and in the Radeon to see if one of them came on.

Host motherboard BIOS options tried:

• Advanced > CPU Configuration > set Intel(VMX) Virtualization Technology to Enabled.
• Advanced > System Agent (SA) Configuration > VT-d > set to Enabled.
• Advanced > System Agent (SA) Configuration > Above 4G Decoding > set to Enabled.
• Advanced > System Agent (SA) Configuration > Graphics Configuration > Primary Display > PCIE
• Advanced > System Agent (SA) Configuration > Graphics Configuration > iGPU Multi-Monitor > Disabled
• Boot > CSM (Compatibility Support Module) > Set Launch CSM to Disabled.
• Set Secure Boot to "Other OS".

ESXi host options tried:

• SSH and "esxcli system settings kernel set -s vga -v FALSE"

Guest is set up as UEFI (not BIOS) although a few early tests I did try on a BIOS install.

Options tried in guest setup/configuration in ESXi:

• Locked all guest memory (2GB).
• Configuration Keys: | Value:
  • hypervisor.cpuid.v0 | FALSE
  • pciHole.start | 1200 / 2048 / 2048
  • pciHole.end | 2200 / 3072 / 6144
  • svga.present | FALSE / TRUE
  • svga.autodetect | FALSE / TRUE
  • svga.vramSize | 4194304 / 16777216
  • Install VM setup with ESXi 7.2 compatibility.
  • Install VM setup with ESXi 6.5 compatibility.
  • Ensure the VMware Video Card settings do not have 3D Support enabled.

Options tried within Windows 8.1 guest:

• Activated Windows before doing anything, just in case there's some AARD code stopping things.
• Disabling the VMWare SVGA driver.
• Driver install methods. When switching between these methods, I used Display Driver Uninstaller (DDU) and rebooted first:
  • Let Windows automatically install drivers.
  • Manually install vendor driver packages (driver 472.12 on the NVidia card, 15.7.1 for the Radeon).
  • Use Browse my computer for driver software, and pointing it to the extracted driver files (versions above).

I've performed multiple reinstalls. Once I've got the base activated VM running using RDP, I switch to a VNC session (as the VMWare SVGA is disabled, along with svga.present and svga.autodetect to FALSE) while playing around with graphics drivers.

In *almost* no configuration could I get GPU-Z to display the BIOS Versions, however most other fields are propagated.

I have an NEC PCI USB card on which passthrough works fine, so it's not a general issue with passthrough. I removed it for the sake of the troubleshooting above.

The closest partial-successes I had:

• In a few configurations the Radeon card didn't get a Code 43 after I reinstalled Windows 8.1 with UEFI, but with ESX 6.5 compatibility. The card showed no errors in Device Manager for a number of test iterations, BUT it wouldn't show in display properties so I couldn't actually turn another display on.
• The Code 43 cleared on BOTH cards while installed at the same time during the Radeon software install process while they were installed at the same time. BUT, neither of them actually worked on multiple displays. I read in a post that "the active hardware detection process by the AMD Catalyst installer causes the driver to realize it has direct access to the GPU, thus resolving the error." But it doesn't stay resolved, or detect the outputs.

Please help before I go postal!

Are there any pros out there? I just copied the pciHole parameters from the Interwebs, but don't know how we actually calculate these.

If anyone can find any flaws or anything I've missed, it would be greatly appreciated!