Just a reminder to make sure your drive monitoring solution works (and a sad story)
I just learned a very expensive lesson, and I would be remiss not to share this story with those who may benefit from my mishap.
Like any good data hoarder with a NAS, I keep tabs on the SMART status of my drives so I can prepare to replace them or get an RMA if needed. I use TrueNAS and the geniuses at iXsystems decided to get rid of the built-in scheduled SMART checks in a recent version. Luckily I noticed this in the changelog before upgrading, so I set up scrutiny to watch the drive status. It recognizes all my drives just fine and they're all green. Lovely! I set up a notification target in my config so that I get notified as soon as a drive starts failing. I go on with my day.
Fast forward several months later... Yesterday I was checking scrutiny to get a drive's serial number, and I notice one is failing. WHAT! 22 REALLOCATED SECTORS! Why the hell didn't I get a notification about this?! I look in my config file and... there was a typo in the target address. When I set up scrutiny notifications I neglected to notice the mechanism to actually test them, so the failure continued silently without me knowing.
The drive had been failing for several months now. At least since April 2026, but probably even earlier since scrutiny only retains a certain number of scans.
So I go to request an RMA from ServerPartDeals... and they tell me I'm 23 days out of the warranty period. At this point I'm kicking myself for 1) not testing the notifications and 2) only noticing this shortly after the warranty period expired.
I bought this drive in 2024, which of course means it's literally tripled in price since I bought it.
Anyway, I've asked SPD support for pity and offered repentance for my sins so we'll see how that goes.
So: Just as we all know untested backups are worth just as much as no backups at all... an untested monitoring solution is as well. Go make sure yours is set up and working. Right now!