r/ProxmoxVE

Proxmox VE 9.x keeps getting SATA/I/O errors and remounting filesystem as read-only

My Proxmox server keeps experiencing random filesystem errors, often overnight. The root filesystem becomes read-only, causing the web UI and services to stop working properly.

Hardware:

- Intel i7-4790

- 32GB RAM

- 240GB second-hand SATA SSD

- Proxmox VE 9.x

Key errors from logs:

ata3: host bus error, interface fatal error

ata3: hard resetting link

ata3: SATA link up 1.5 Gbps

failed command: WRITE FPDMA QUEUED

ICRC ABRT

I/O error, dev sdb

EXT4-fs: Detected aborted journal

EXT4-fs: Remounting filesystem read-only

EXT4-fs: potential data loss!

pveproxy: Read-only file system

Troubleshooting already done:

- Tried multiple SATA cables.

- Switched between different motherboard SATA ports.

- Installed a separate SATA expansion card, but the problem persists.

- SSD passes SMART tests without reporting any issues.

- Recovered the filesystem multiple times, but the errors keep returning.

- Previously ran Windows 10 on this SSD for about a year without similar issues.

I'm struggling to identify the actual cause since changing cables, ports, and even the SATA controller hasn't resolved it. The SSD also passes SMART testing, making it harder to determine whether the drive itself is faulty.

Could this be a failing SSD despite passing SMART, a power delivery issue, or something specific to Linux/Proxmox?

Has anyone experienced something similar or have suggestions on what to check next?

I'm looking for a bit of inspiration I have built an home lab running proxmox on a dell vostro with home assistant, tailscale, and immich, I just spent half the day today installing both seerr and ryot hoping to centralized all my tracking from MAL and Trakt to find out doesn't sync automatically and add I use stremio on my tvs it all turned out to be a waste of time this was all from suggestions via ai which made me think I need to do some real life research instead into what I should add to my proxmox to improve my life and also things that are fun or give off cool tech vibes.

So first step all for some inspiration here

We run a 6 node PVE 9.2.20 cluster with HA and two dedicated corosync links on separate NICs (Intel E810, link 0 and link 1 each on their own port, nothing else on those interfaces). Default config, so link_mode passive.
Today a single port started flapping on one node. The NIC counts mac_local_faults, the switch side shows zero CRC errors, physical link only went down once, but knet kept marking link 0 down and up every few seconds towards all peers.

[KNET ] link: host: 4 link: 0 is down
[KNET ] host: host: 4 (passive) best link: 1 (pri: 1)
[KNET ] rx: host: 4 link: 0 is up
[KNET ] host: host: 4 (passive) best link: 0 (pri: 1)
pve-ha-lrm: loop took too long (40 seconds)
pvescheduler: cfs-lock 'file-jobs_cfg' error: got lock request timeout
watchdog-mux: client watchdog expired - disable watchdog updates

Questions for people running larger HA clusters

1. Is anyone running link_mode: active in production?Does it actually make a single flapping link harmless?
2. Or do you stay on passive and raise knet_pong_count (or ping timeout/interval) to get some hysteresis?
3. Any other best practice to make sure one bad port or DAC can never cause a node to fence?

Root cause on the hardware side is being handled separately (DAC and switch). I am mainly interested in how to make corosync resilient against this in general.
Thanks!