I'm considering building a TrueNas or a similar system on older windows server hardware.

I've read a lot about data recovery from failed drives, but I can't find anything about taking healthy drives from a failed system/ motherboard or whatever and pluging those drives into another system to get the data back.
I've read that ZFS is/can be used as an open standard so I would guess another system could see the data?

I've had a consumer NAS hardware fail before. At that time I didn't understand the drive file format details and couldn't get the drives to mount on another system.

What can I expect with ZFS?

I heard ZFS is the best, I want to switch to it atleast for my backup USB HDD (Kopia).

• I don't understand all it's features, but it seems like it is it's own OS.

1. I want to able to access the disk from any OS (Win,Linux,macOS) currently using exFAT.
2. How is support on Windows (for basic features equivalent to normal FS like exFAT etc.)

1. What Filesystem to use for each purpose ?

1. Internal Disk - OpenZFS/ext4/NTFS ? for all OS (Win,Lin,mac)
2. External Disk - Is exFAT OK ? Else OpenZFS/ext4 ? for all OS
3. Backup Disk - guy says exFAT is bad for backup as it corrupts. so ***ext4/***OpenZFS ?
4. SOHO Server/NAS - ZFS

2. OpenZFS vs BtrFS?

3.1 Is ZFS on Win/mac mature enough ?

• https://github.com/openzfsonwindows/openzfs
• https://openzfsonosx.org/

3.2 If not Is Ext4 on Win/mac good ?

• https://github.com/bobranten/Ext4Fsd
• https://github.com/macfuse/macfuse , https://github.com/gerard/ext4fuse (no update)

https://preview.redd.it/2ve13nr4qw1h1.png?width=3799&format=png&auto=webp&s=80dba86577f90155fa4b30a5b56a6947e960afbf

The zfs pool:

                                                            capacity     operations     bandwidth 
pool                                                      alloc   free   read  write   read  write
--------------------------------------------------------  -----  -----  -----  -----  -----  -----
Exos                                                      23.7T  41.8T     21     40  1.22M  3.39M
  raidz1-0                                                23.7T  41.8T     21     40  1.22M  3.39M
    ata-ST24000NM001H-3KS113_ZYD5B1W9                         -      -      7     13   414K  1.13M
    ata-ST24000NM001H-3KS113_ZYD5AV8T                         -      -      7     13   420K  1.13M
    ata-ST24000NM001H-3KS113_ZYD5B1DT                         -      -      7     13   419K  1.13M
cache                                                         -      -      -      -      -      -
  nvme-Lexar_SSD_NM1090_PRO_2TB_QC6614R000385P3200-part1   364G   590G     21     10  2.66M  1.16M
--------------------------------------------------------  -----  -----  -----  -----  -----  -----

kernel: 6.19.13+deb14-amd64

installed package:

zfsutils-linux:
  Installed: 2.4.1-1
  Candidate: 2.4.1-1
  Version table:
     2.4.2-2 100
        100 http://deb.debian.org/debian sid/contrib amd64 Packages
 *** 2.4.1-1 500
        500 http://deb.debian.org/debian testing/contrib amd64 Packages
        100 /var/lib/dpkg/status

System Memory: 128GB

I limited my ARC to 8GB of total system memory, as it's needed elsewhere. At first that was fine, up to about 700GB of L2ARC where it just stopped taking in more data. When I researched the issue I thought that l2arc_meta_percent might be limiting me, but after increasing it to 50, nothing changed.

Then one day I had some very non-sequential read/write workload run overnight and when I woke up, my L2ARC was drained to only 424G and the header size (l2_hdr_size) decreased to tiny amounts (822MiB from around 2GB). I also noticed l2_abort_lowmem = 254671. I never saw this number being more than zero and it hasn't increased since (1 day). Note that nothing then was RAM-heavy, I still had like good 60GB free (not accounting for linux page cache).

The trend of my L2ARC decreasing in size continued and currently it's at 364G.

Only non-default settings are:

options zfs zfs_arc_max=8589934592
options zfs l2arc_write_max=16777216
options zfs l2arc_meta_percent=50

The evicted data wasn't cold or stale, before the event my L2ARC hit rate was well above 90% in the 95% regions, Now it's nowhere near that and I see my HDDs being utilized way more. Here is a link to arcstats just after the anomaly: https://pastebin.com/raw/cxnY82e3, and another one after a day: https://pastebin.com/raw/79Dfx7hu

This Nas is my off-site backup. I want to backup my BTRFS system to the NAS and have snapshot history as well so if anything breaks on my Linux pc and I somehow can't use the snapshots on the pc, I can also do a rollback from my NAS. What's the best way. I'm getting pretty confused by the articles online and chatgpt.

I recently migrated my 3x2tb RAID Z1 pool (named storage) from a Truenas VM to the Proxmox host itself so i can free up some RAM. Last night a power fluctuation rebooted my server (I know, I need a UPS). I forgot that I had left the TrueNAS VM set to auto-start at boot. So after reboot, both the Proxmox host (which had the pool imported) and the TrueNAS VM (which also tried to import the pool) were writing to the same disks at the same time.
Now the pool is corrupted. Here's what I've tried and the current state.

zpool import
  pool: storage
    id: 4106071641955219047
 state: FAULTED
status: The pool metadata is corrupted.
action: The pool cannot be imported due to damaged devices or data.
        The pool may be active on another system, but can be imported using the '-f' flag.
config:
        storage            FAULTED  corrupted data
          raidz1-0         ONLINE
            disk1          ONLINE
            disk2          ONLINE
            disk3          ONLINE

What I've attempted while searching for solutions:

• zpool import -f storage > I/O error
• zpool import -F -n storage > nothing
• zpool import -FX -o readonly=on storage > cannot import: one or more devices currently unavailable
• zdb -e -AAA storage > shows configuration, then zdb: can't open 'storage': Input/output error
• zdb -e -y storage > same I/O error
• Forced import with -T using a known good txg from labels (all three disks show txg 1673045 in zdb -l): zpool import -d /dev/disk/by-id -o readonly=on -f -T 1673045 storage > one or more devices is currently unavailable
• zpool import -d /dev/disk/by-id -o readonly=on -f -F storage > I/O error, destroy and re-create from backup
• zdb -l on each disk shows the same pool GUID, same txg=1673045, and all labels appear intact.
• Smart data on all three disks is clean (no reallocated or pending sectors).

Is there anything I'm missing? Is the data gone for good?

For the past few years, my homelab server has been rock solid. It's a mini PC with a TB4 eGPU and a 9-bay Terramaster USB 3.2 gen2 enslosure with 12TB disks running raidz2 in debian.

My first and only issue really was the random i/o and reset errors on the usb drives/enclosure during heavy read/write usage. Usually during a scrub, i'd see these errors which would put my pool in a degraded state. Went through customer support emails, many USB cables, and power supplies.The solution has been to disable the UAS driver and use the old school BOT usb-storage. It has been fantastic with no issues at all servering production applications and media needs.

UAS/UASP is the newer USB storage protocol that lets a drive behave more like a real SATA/SCSI device over USB: it supports command queuing, multiple commands in flight, better full-duplex behavior, lower CPU overhead, and usually better random I/O and multi-drive enclosure performance. usb-storage is the older BOT mode: simpler and often more stable with flaky USB bridges, but it handles one command at a time and can bottleneck harder when ZFS is reading/writing many blocks across drives. For your ZFS/DAS use, UAS is “better” when the enclosure/controller is stable because it allows higher throughput and responsiveness, but usb-storage can be safer if UAS causes resets, disconnects, or I/O timeouts.

This week, I started planning to use ZFS's expand feature, followed by numerous scrubs and then a rewrite. So far, so good, but it's taking a lot of time as usual. This got me thinking about the newer, more efficient UAS driver that I haven't been using. Theoretically, this driver would be much more ideal while I'm doing all of this ZFS rework and using the storage simultaneously. So I started switching back to UAS and trying to Linux-quirk my way out of this USB issue.

Not much info out there other than to just disable UAS and use usb-storage, but that's not what i want to do at all. So far, since adding the following grub commands, it has been stable and giving me better parallel processing speeds!

>GRUB_CMDLINE_LINUX_DEFAULT="quiet splash usbcore.autosuspend=-1 pci=noaer nvidia-drm.modeset=1 iommu=pt"
GRUB_CMDLINE_LINUX="usb-storage.quirks=0bda:9201:rg

The g flag in the usb-storage quirk is what solved my issue. I also enabled IOMMU passthrough, and I'm going ahead and adding the r flag to disregard anything questionable that this USB-to-SATA controller is sending. Hopefully, it isn't anything meaningful. Here is the documentation on the flags and their uses. Here is the documentation on the flags and their uses.

The g flag caps the maximum number of 512-byte sectors per single I/O command at 240 sectors, which isn't ideal, but it still gives me huge improvements over the other driver.

Anyways, I hope this helps someone in a similar situation. I'll also post the i/o and USB errors I was getting below. Would love to hear the elite folks' thoughts on what I found and if there would be any other solutions to this. Just know I've tried quite a bit!

[  363.554936] sd 2:0:0:0: [sdc] tag#26 uas_eh_abort_handler 0 uas-tag 2 inflight: CMD IN
[  363.554949] sd 2:0:0:0: [sdc] tag#26 CDB: Read(16) 88 00 00 00 00 00 83 68 64 00 00 00 01 b0 00 00
[  363.555103] sd 2:0:0:0: [sdc] tag#9 uas_eh_abort_handler 0 uas-tag 3 inflight: CMD IN
[  363.555109] sd 2:0:0:0: [sdc] tag#9 CDB: Read(16) 88 00 00 00 00 00 83 68 63 b8 00 00 00 48 00 00
[  363.555193] sd 2:0:0:0: [sdc] tag#6 uas_eh_abort_handler 0 uas-tag 1 inflight: CMD IN
[  363.555198] sd 2:0:0:0: [sdc] tag#6 CDB: Read(16) 88 00 00 00 00 00 83 68 62 e0 00 00 00 d8 00 00
[  363.555281] usb 4-1.1.3: stat urb: no pending cmd for uas-tag 1
[  363.590943] scsi host2: uas_eh_device_reset_handler start
[  363.667172] usb 4-1.1.3: reset SuperSpeed USB device number 8 using xhci_hcd
[  363.807651] scsi host2: uas_eh_device_reset_handler success
[  374.010991] sd 2:0:0:0: [sdc] tag#26 uas_eh_abort_handler 0 uas-tag 1 inflight: CMD
[  374.011005] sd 2:0:0:0: [sdc] tag#26 CDB: Test Unit Ready 00 00 00 00 00 00
[  374.011012] scsi host2: uas_eh_device_reset_handler start
[  374.087185] usb 4-1.1.3: reset SuperSpeed USB device number 8 using xhci_hcd
[  374.226044] scsi host2: uas_eh_device_reset_handler success
[  374.226053] sd 2:0:0:0: Device offlined - not ready after error recovery
[  374.226067] sd 2:0:0:0: Device offlined - not ready after error recovery
[  374.226071] sd 2:0:0:0: Device offlined - not ready after error recovery
[  374.226083] sd 2:0:0:0: [sdc] tag#6 FAILED Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK cmd_age=41s
[  374.226089] sd 2:0:0:0: [sdc] tag#6 CDB: Read(16) 88 00 00 00 00 00 83 68 62 e0 00 00 00 d8 00 00
[  374.226092] I/O error, dev sdc, sector 2204656352 op 0x0:(READ) flags 0x0 phys_seg 27 prio class 2
[  374.226127] zio pool=tank vdev=/dev/disk/by-id/usb-ST12000N_T001--0:0-part1 error=5 type=1 offset=1128783003648 size=110592 flags=2148533424
[  374.226145] sd 2:0:0:0: [sdc] tag#9 FAILED Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK cmd_age=41s
[  374.226149] sd 2:0:0:0: [sdc] tag#9 CDB: Read(16) 88 00 00 00 00 00 83 68 63 b8 00 00 00 48 00 00
[  374.226152] I/O error, dev sdc, sector 2204656568 op 0x0:(READ) flags 0x0 phys_seg 9 prio class 2
[  374.226177] zio pool=tank vdev=/dev/disk/by-id/usb-ST12000N_T001--0:0-part1 error=5 type=1 offset=1128783114240 size=36864 flags=3145904
[  374.226190] sd 2:0:0:0: [sdc] tag#26 FAILED Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK cmd_age=41s
[  374.226194] sd 2:0:0:0: [sdc] tag#26 CDB: Read(16) 88 00 00 00 00 00 83 68 64 00 00 00 01 b0 00 00
[  374.226196] I/O error, dev sdc, sector 2204656640 op 0x0:(READ) flags 0x0 phys_seg 54 prio class 2
[  374.226220] zio pool=tank vdev=/dev/disk/by-id/usb-ST12000N_T001--0:0-part1 error=5 type=1 offset=1128783151104 size=221184 flags=2148533424
[  374.226284] sd 2:0:0:0: rejecting I/O to offline device
[  374.226284] I/O error, dev sdc, sector 2204657072 op 0x0:(READ) flags 0x4000 phys_seg 128 prio class 2
[  374.226297] I/O error, dev sdc, sector 2204659088 op 0x0:(READ) flags 0x4000 phys_seg 128 prio class 2
[  374.226326] I/O error, dev sdc, sector 2204658096 op 0x0:(READ) flags 0x0 phys_seg 124 prio class 2
[  374.226339] I/O error, dev sdc, sector 2204660112 op 0x0:(READ) flags 0x0 phys_seg 124 prio class 2
[  374.226358] zio pool=tank vdev=/dev/disk/by-id/usb-ST12000N_T001--0:0-part1 error=5 type=1 offset=1128783372288 size=1032192 flags=2148533424
[  374.226364] I/O error, dev sdc, sector 2204661104 op 0x0:(READ) flags 0x4000 phys_seg 128 prio class 2
[  374.226373] zio pool=tank vdev=/dev/disk/by-id/usb-ST12000N_T001--0:0-part1 error=5 type=1 offset=1128784404480 size=1032192 flags=2148533424
[  374.226374] I/O error, dev sdc, sector 2204662128 op 0x0:(READ) flags 0x0 phys_seg 124 prio class 2
[  374.226378] zio pool=tank vdev=/dev/disk/by-id/usb-ST12000N_T001--0:0-part1 error=5 type=1 offset=1128785436672 size=1032192 flags=2148533424
[  374.226430] I/O error, dev sdc, sector 2576 op 0x0:(READ) flags 0x0 phys_seg 2 prio class 2
[  374.226459] zio pool=tank vdev=/dev/disk/by-id/usb-ST12000N_T001--0:0-part1 error=5 type=1 offset=270336 size=8192 flags=1245377
[  374.226476] zio pool=tank vdev=/dev/disk/by-id/usb-ST12000N_T001--0:0-part1 error=5 type=1 offset=12000127623168 size=8192 flags=1245377
[  374.226490] zio pool=tank vdev=/dev/disk/by-id/usb-ST12000N_T001--0:0-part1 error=5 type=1 offset=12000127885312 size=8192 flags=1245377
[  374.226556] zio pool=tank vdev=/dev/disk/by-id/usb-ST12000N_T001--0:0-part1 error=5 type=1 offset=1128786468864 size=1032192 flags=2148533424
[  374.226695] zio pool=tank vdev=/dev/disk/by-id/usb-ST12000N_T001--0:0-part1 error=5 type=1 offset=1128787501056 size=1032192 flags=2148533424

Hi all,

Current setup: ESXI (so hardware raid), VMFS on top, File server with Debian/ext4 on the virtual disk. 3x8tb in a hardware raid 5 with a local 16tb jbod for backup. Host is UPS backed and has 192gb ecc ram. OS/VMs are on a hardware raid1 of 2x4tb sata ssd. There is a single 1tb nvme ssd that I use as a network swap drive (just for moving shit around quickly as I have 10gb lan). This setup has been stable for about 5 years, but its time to move away from ESXI and im out of storage.

Proposed plan: Borrow 20TB of storage just to temporarily throw everything on to, flatten the current server, install Debian or Rocky (yet undecided) on the 2x 4tb ssd (likely in an md raid 1), buy 3x more 8TB drives (so 6x total), then raidz2 of the 6x8tb's.

Single vdev, single pool, maybe 3-4 datasets, no zvols.

Am I right in thinking I dont want a special due to single point of failure, and little benefit to a SLOG/L2 given ram quantity? Am I best off selling the 1tb nvme to offset the cost of the 3x new 8tbs?

Any drivers to prefer debian or rocky from a ZFS perspective?

Also inherited an LTO 6 drive so interested to learn my options for flushing zfs snapshots to tape.

i have 100% full zpool, which from what i read means for zfs "hoplesly dead, delete and start from scratch"
It is bricked effectively - can't delete file/vdev/snapshot can't zfs send
I can copi all files from it, but there are also some vdev's with raw vps images. How can i backup them, given "zfs send" fails with "out of space"?

If you use ZFS and encrypt your data, you likely have the same problem: do you keep the key in plaintext on the server for unattended reboots (convenience), or do you make yourself SSH into the server to type it every time (security)?

For the last few years I've been using my own Revaulter, which allows the server to request to decrypt the volumes' keys, and I can approve them with a web app on my phone. Revaulter v2, released a few days ago, uses Passkeys to encrypt data safely, replacing the old Azure Key Vault needed by v1.

This blog explains exactly how I use Revaulter to unlock the ZFS volumes of my servers when they boot up, keeping the keys safe.

Hello! recently I been asked to help create an archive, the computer have 3 drives, 2 of 16tb and one of 12tb. I was thinking about make a pool in raidz2 thru 3 drives (because possible future expansion) and mirror in the free space of the 16tb disk (it's possible right?) for snapshots or another drive. Do you have some suggestions? could have some problems in the future?

You may need to roll back to 6.17.0-22. Not sure how this happened, I'm not supposed to be on an experimental kernel.

I have a server in a dorm with a pool of zfs drives that are spun down most of the time. I have carefully organized a schedule of file syncs and other maintenance tasks to ensure that drives spin down only once every day and some specific days of the month 2 times to run things like smart tests and scrubs and I have scheduled them at times when noise wont disturb anyone. Now that all works perfectly except that I cant find a way to disable the default zfs scrub job and the drives just spin up at 00:24h and it is very loud. How the hell do I disable this, I have disabled zfs zed, I have checked the /etc/cron.d/zfsutils-linux and completely deleted the lines for the scrub jobs and I have also set the zpool periodic scrub properties that chatgpt told me to do so I dont know if those make any difference anyway, but still the drives just spin up and scrubs start anyway. This started happening after an update to 2.4.1 but it seems that no matter what I do I cant find a way to disable this stupid schedule. Can anyone help me fix this because its extremely annoying and making noise in the middle of the night for no reason at all. I am running zfs version 2.4.1 on debian 13 and it was installed from backports.

Hardware:

• Dell PowerEdge R730 (Ubuntu 24.04) — 10× 1.2TB SAS
• Chenbro chassis (TrueNAS Scale) — 10× 3TB internal
• NetApp DS2246 — 24× 600GB SAS (connected to TrueNAS via LSI SAS9207-8e HBA)
• EMC JBOD — 24× 900GB SAS (daisy chained through DS2246)
• Incoming: NetApp DS4246 — 24× 4TB (will daisy chain into existing shelf chain)

Current setup: The R730 runs Ubuntu with a local ZFS pool (1.2TB drives). The Chenbro runs TrueNAS and manages everything else through the HBA and shelf chain. Both servers use MergerFS on the R730 to present a unified /data path to Plex and the arr stack. Downloads land on an Intel Optane P4800X as a scratch device, with an Oracle F80 as SLOG on the TrueNAS pool.

The question: With drives split across multiple shelves in four different sizes, what's the best ZFS pool architecture for Plex media storage?

Specifically I'm trying to figure out:

1. Should I run one big pool with lots of vdevs, or separate pools per drive size group?
2. Within each drive size group, should I split into multiple smaller vdevs or one wide vdev?
3. Is dRAID worth it here? The 24-drive groups seem like good dRAID candidates.
4. I keep hearing "if one vdev fails the whole pool is gone" does that change the calculus on how wide to make each vdev?

Primary use case is Plex streaming. TrueNAS box has 296GB ECC RAM for ARC.

Happy to hear any opinions on the architecture.