u/Even_Professional953

My Proxmox host (PVE 9.2, Gigabyte Z390 AORUS ELITE, i9-9900KF, BIOS F10, Seasonic 750W 80+ Gold) keeps freezing because the boot NVMe drops off the PCIe bus. Root fs goes read-only, host limps in RAM until I power-cycle. Caught the whole thing via an off-box kernel log streamer (journald dies with the disk, so on-box logs never show the cause):

nvme nvme0: controller is down; will reset: CSTS=0xffffffff, PCI_STATUS=0x10
nvme nvme0: Does your device have a faulty power saving mode enabled?
nvme nvme0: Try "nvme_core.default_ps_max_latency_us=0 pcie_aspm=off pcie_port_pm=off" and report a bug
nvme nvme0: Disabling device after reset failure: -19
EXT4-fs (dm-1): I/O error while writing superblock
EXT4-fs (dm-1): Remounting filesystem read-only

Here's the kicker: those three kernel parameters were already active when it crashed. Verified in the captured cmdline and via get-feature 0x0c (APSTE: Disabled on the drive itself).

What I've eliminated over 5 incidents in 2 weeks:

• The drive. Happened on a Samsung 970 EVO Plus 500GB, so I swapped in a completely different drive (Samsung PM9A1 1TB, different controller gen, fresh PVE install). Dropped identically 6 days later.
• The slot/port. Happened in both M.2 slots (different PCH root ports, 00:1b.0 and 00:1d.0).
• APST / ASPM / PCIe port PM. All disabled (kernel cmdline + BIOS ASPM off). Still drops.
• Thermal. SMART lifetime counters: zero warning/critical composite temp time, zero throttle events, ~49°C at failure.
• Drive health. 0 media errors, 100% spare, error log literally empty after the event. The drive never sees its own death.
• Workload. Has died under sustained backup reads, under bursty docker writes, and now completely idle (19h of kernel silence before the drop).
• Kernel. Two different PVE kernels.
• RAM OC. Dropped to JEDEC 2133.
• PSU. Quality Seasonic 750W Gold with roughly 2x headroom over worst-case system draw (~425W with both GPUs and CPU maxed), full protection suite. And the latest drop happened at idle, not under load.
• No AER/DPC precursors in the latest event. The controller just reads all-ones out of nowhere. Only a cold power-cycle brings the drive back; warm reset/reboot doesn't.

Has anyone run into this and found an actual fix? Any idea I haven't tried is welcome.