r/zfs

I'm trying to set up ZFS snapshot replication with Sanoid/Syncoid. I have a backup server connect to the production server and pull the snapshots with Syncoid. Snapshots are pruned on the backup server with Sanoid.

I'm considering a scenario where production is compromised, creating bogus snapshots to force a prune on the backup server upon next pull.

I haven't managed to determine what the best practices are to avoid this, how's everyone else handling this? Am I stuck rolling my own service that manages snapshot holds on the backup server or is this a solved problem?

Thanks!

I have two computers with two pools, one as a main and one as a backup. Each is running zfs on linux, and the user on the backup is not a root user, so it does not have mount permissions. Also, I set its zfs permissions via zfs allow, and it does not have destroy permissions.

When I send snapshots via send and receive, I noticed that datasets that I destroyed on my main pool still existed on my backup pool, even when snapshots sent correctly. My snapshots are sent with the -R flag and I also add flags to send them unmounted and read only.

I also set up a local AI and I asked it why datasets destroyed on my main pool don't get destroyed once I sent snapshots, and it first said it was likely because the datasets were mounted and in use on the backup. I then thought about it myself, and came to the conclusion that it was probably because my backup user doesn't have destroy permissions, so when it runs the receive command, it can't include destroying datasets. I asked the AI if that was the case, and it said "you have 100% hit the nail on the head, that is exactly correct . . ." etc etc etc.

I wanted to check with real people if I'm actually right about that though. Is it really a permissions issue?

I've got 9 drives of 20 TB, Was thinking of doing raidz2, with 8 of them, and keeping 1 spare.

Does this make more sense than 3x3 raidz1 (no spare), or 2x4 raidz1 (1 spare)? Any other sensible options?

Typed out this post to ask for advice but then the resilver completed right before I hit post lol. So now I'm just posting this because I could only find one other person mentioning this from like 10 years ago and it may be useful knowledge for other people seeing this behavior.

On Saturday, in a pool that spans 5 JBODs, zfs started reporting drives from one of the JBODs as degraded and automatically started resilvering to available hotspares. I am not sure what caused that JBOD to start marking these drives degraded but smartctl showed no errors (except for non-medium errors). I exported the pool over the weekend to avoid any more issues showing up before I could dig in further. Today, I power-cycled the JBOD and moved SAS cables to different ports, and then reimported the pool. I ran zpool clear on each drive after checking its SMART status. Then I detached each hotspare that had started resilvering.

It has been a few hours now with the drives all showing ONLINE and data seems to be all present and accounted for. But the resilver progress continues anyway finished, well past 100%.

zpool status -v output:

root@aq-zfs1:~# zpool status -v zpool1
  pool: zpool1
 state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Sat Jun 27 11:20:54 2026
1.08P scanned at 180G/s, 16.0E issued at 0B/s, 1.14P total
0B resilvered, 1440707.37% done, no estimated completion time
config:

NAME                                        STATE     READ WRITE CKSUM
zpool1                                      ONLINE       0     0     0
 raidz2-0                                  ONLINE       0     0     0
   ZRS1AKA2_ST16000NM004J_16TB_SAS1_1      ONLINE       0     0     0
   ZRS1AK8Y_ST16000NM004J_16TB_SAS1_3      ONLINE       0     0     0
   ZRS1AJHY_ST16000NM004J_16TB_SAS2_1      ONLINE       0     0     0
   ZRS1AK07_ST16000NM004J_16TB_SAS2_3      ONLINE       0     0     0
   ZRS18J06_ST16000NM004J_16TB_SAS3_1      ONLINE       0     0     0
   ZRS18HQ1_ST16000NM004J_16TB_SAS3_3      ONLINE       0     0     0
   ZRS18J6M_ST16000NM004J_16TB_SAS4_1      ONLINE       0     0     0
   ZRS18J0T_ST16000NM004J_16TB_SAS4_3      ONLINE       0     0     0
   ZRS18HQV_ST16000NM004J_16TB_SAS5_1      ONLINE       0     0     0
   ZRS18HNB_ST16000NM004J_16TB_SAS5_3      ONLINE       0     0     0
 raidz2-1                                  ONLINE       0     0     0
   ZRS1AJHC_ST16000NM004J_16TB_SAS1_5      ONLINE       0     0     0
   ZRS1AJVH_ST16000NM004J_16TB_SAS1_15     ONLINE       0     0     0
   ZRS18J63_ST16000NM004J_16TB_SAS2_5      ONLINE       0     0     0
   ZRS18HKV_ST16000NM004J_16TB_SAS2_15     ONLINE       0     0     0
   ZRS18HNC_ST16000NM004J_16TB_SAS3_5      ONLINE       0     0     0
   ZRS18HVJ_ST16000NM004J_16TB_SAS3_15     ONLINE       0     0     0
   ZRS1AJQF_ST16000NM004J_16TB_SAS4_5      ONLINE       0     0     0
   ZRS1AK00_ST16000NM004J_16TB_SAS4_15     ONLINE       0     0     0
   ZRS18HWR_ST16000NM004J_16TB_SAS5_5      ONLINE       0     0     0
   ZRS1AJK6_ST16000NM004J_16TB_SAS5_15     ONLINE       0     0     0
 raidz2-2                                  ONLINE       0     0     0
   ZRS18J74_ST16000NM004J_16TB_SAS1_17     ONLINE       0     0     0
   ZRS18HMY_ST16000NM004J_16TB_SAS1_19     ONLINE       0     0     0
   ZRS18J1M_ST16000NM004J_16TB_SAS2_17     ONLINE       0     0     0
   ZRS1AK38_ST16000NM004J_16TB_SAS2_19     ONLINE       0     0     0
   ZR611S70_ST16000NM004J_16TB_SAS3_17     ONLINE       0     0     0
   ZR6123GN_ST16000NM004J_16TB_SAS3_19     ONLINE       0     0     0
   ZRS1AKAC_ST16000NM004J_16TB_SAS4_17     ONLINE       0     0     0
   ZRS1AK7K_ST16000NM004J_16TB_SAS4_19     ONLINE       0     0     0
   ZRS18J4F_ST16000NM004J_16TB_SAS5_17     ONLINE       0     0     0
   ZRS1AJPE_ST16000NM004J_16TB_SAS5_19     ONLINE       0     0     0
 raidz2-3                                  ONLINE       0     0     0
   ZRS1AJY9_ST16000NM004J_16TB_SAS1_29     ONLINE       0     0     0
   ZRS1AJMJ_ST16000NM004J_16TB_SAS1_31     ONLINE       0     0     0
   ZRS18J20_ST16000NM004J_16TB_SAS2_29     ONLINE       0     0     0
   ZRS18HYL_ST16000NM004J_16TB_SAS2_31     ONLINE       0     0     0
   ZR6123GK_ST16000NM004J_16TB_SAS3_29     ONLINE       0     0     0
   ZR611RQQ_ST16000NM004J_16TB_SAS3_31     ONLINE       0     0     0
   ZRS1AK3Z_ST16000NM004J_16TB_SAS4_29     ONLINE       0     0     0
   ZRS1AKC2_ST16000NM004J_16TB_SAS4_31     ONLINE       0     0     0
   ZRS18HZY_ST16000NM004J_16TB_SAS5_29     ONLINE       0     0     0
   ZRS1AK2H_ST16000NM004J_16TB_SAS5_31     ONLINE       0     0     0
 raidz2-4                                  ONLINE       0     0     0
   ZRS18J0S_ST16000NM004J_16TB_SAS1_33     ONLINE       0     0     0
   ZRS18HT9_ST16000NM004J_16TB_SAS1_43     ONLINE       0     0     0
   ZRS18HSZ_ST16000NM004J_16TB_SAS2_33     ONLINE       0     0     0
   ZRS1AJMW_ST16000NM004J_16TB_SAS2_43     ONLINE       0     0     0
   ZR612386_ST16000NM004J_16TB_SAS3_33     ONLINE       0     0     0
   ZR611Q1G_ST16000NM004J_16TB_SAS3_43     ONLINE       0     0     0
   ZRS1AJSF_ST16000NM004J_16TB_SAS4_33     ONLINE       0     0     0
   ZRS1AJJD_ST16000NM004J_16TB_SAS4_43     ONLINE       0     0     0
   ZRS1AJGQ_ST16000NM004J_16TB_SAS5_33     ONLINE       0     0     0
   ZRS1AK9S_ST16000NM004J_16TB_SAS5_43     ONLINE       0     0     0
 raidz2-5                                  ONLINE       0     0     0
   ZRS1AJLW_ST16000NM004J_16TB_SAS1_45     ONLINE       0     0     0
   ZRS18J5A_ST16000NM004J_16TB_SAS1_47     ONLINE       0     0     0
   ZRS18HLM_ST16000NM004J_16TB_SAS2_45     ONLINE       0     0     0
   ZRS18HLS_ST16000NM004J_16TB_SAS2_47     ONLINE       0     0     0
   ZR611PPV_ST16000NM004J_16TB_SAS3_45     ONLINE       0     0     0
   ZR611PRP_ST16000NM004J_16TB_SAS3_47     ONLINE       0     0     0
   ZRS1AJS5_ST16000NM004J_16TB_SAS4_45     ONLINE       0     0     0
   ZRS1AK5C_ST16000NM004J_16TB_SAS4_47     ONLINE       0     0     0
   ZRS1AK8D_ST16000NM004J_16TB_SAS5_45     ONLINE       0     0     0
   ZRS1AK0A_ST16000NM004J_16TB_SAS5_47     ONLINE       0     0     0
 raidz2-6                                  ONLINE       0     0     0
   ZRS1AK6M_ST16000NM004J_16TB_SAS1_57     ONLINE       0     0     0
   ZRS1AKAS_ST16000NM004J_16TB_SAS1_59     ONLINE       0     0     0
   ZRS18J75_ST16000NM004J_16TB_SAS2_57     ONLINE       0     0     0
   ZRS1AJTE_ST16000NM004J_16TB_SAS2_59     ONLINE       0     0     0
   ZR611AG5_ST16000NM004J_16TB_SAS3_57     ONLINE       0     0     0
   ZR611PYT_ST16000NM004J_16TB_SAS3_59     ONLINE       0     0     0
   ZRS1AJTH_ST16000NM004J_16TB_SAS4_57     ONLINE       0     0     0
   ZRS1AKC3_ST16000NM004J_16TB_SAS4_59     ONLINE       0     0     0
   ZRS18J18_ST16000NM004J_16TB_SAS5_57     ONLINE       0     0     0
   ZRS1AJK0_ST16000NM004J_16TB_SAS5_59     ONLINE       0     0     0
 raidz2-7                                  ONLINE       0     0     0
   ZRS1AJRX_ST16000NM004J_16TB_SAS1_61     ONLINE       0     0     0
   ZRS1AJXY_ST16000NM004J_16TB_SAS1_71     ONLINE       0     0     0
   ZRS18HMN_ST16000NM004J_16TB_SAS2_61     ONLINE       0     0     0
   ZRS1AJZ2_ST16000NM004J_16TB_SAS2_71     ONLINE       0     0     0
   ZRS1AJX9_ST16000NM004J_16TB_SAS3_61     ONLINE       0     0     0
   ZRS1AK23_ST16000NM004J_16TB_SAS3_71     ONLINE       0     0     0
   ZRS1AK2M_ST16000NM004J_16TB_SAS4_61     ONLINE       0     0     0
   ZRS1AK0Q_ST16000NM004J_16TB_SAS4_71     ONLINE       0     0     0
   ZRS18J0K_ST16000NM004J_16TB_SAS5_61     ONLINE       0     0     0
   ZRS18HZW_ST16000NM004J_16TB_SAS5_71     ONLINE       0     0     0
 raidz2-8                                  ONLINE       0     0     0
   ZRS1AKBL_ST16000NM004J_16TB_SAS1_73     ONLINE       0     0     0
   ZRS1AJXA_ST16000NM004J_16TB_SAS1_75     ONLINE       0     0     0
   ZRS1AKCZ_ST16000NM004J_16TB_SAS2_73     ONLINE       0     0     0
   ZRS18HLW_ST16000NM004J_16TB_SAS2_75     ONLINE       0     0     0
   ZRS1AKDL_ST16000NM004J_16TB_SAS3_73     ONLINE       0     0     0
   ZRS1AJVC_ST16000NM004J_16TB_SAS3_75     ONLINE       0     0     0
   ZRS18HTY_ST16000NM004J_16TB_SAS4_73     ONLINE       0     0     0
   ZRS1AJSG_ST16000NM004J_16TB_SAS4_75     ONLINE       0     0     0
   ZRS1AJW5_ST16000NM004J_16TB_SAS5_73     ONLINE       0     0     0
   ZRS1AK8M_ST16000NM004J_16TB_SAS5_75     ONLINE       0     0     0
special
 mirror-9                                  ONLINE       0     0     0
   24212X803118_WD_BLACK_4TB_NVME_da       ONLINE       0     0     0
   24212X801125_WD_BLACK_4TB_NVME_9b       ONLINE       0     0     0
   24212X801132_WD_BLACK_4TB_NVME_dc       ONLINE       0     0     0
 mirror-10                                 ONLINE       0     0     0
   24212X804611_WD_BLACK_4TB_NVME_ed       ONLINE       0     0     0
   24212X801675_WD_BLACK_4TB_NVME_9a       ONLINE       0     0     0
   24233N805374_WD_BLACK_4TB_NVME_ee       ONLINE       0     0     0
logs
 mirror-11                                 ONLINE       0     0     0
   24212X800463_WD_BLACK_4TB_NVME_d9       ONLINE       0     0     0
   24212X801126_WD_BLACK_4TB_NVME_99       ONLINE       0     0     0
spares
 ZR611RTT_ST16000NM004J_16TB_SAS5_2_SPARE  AVAIL   
 ZR611RNN_ST16000NM004J_16TB_SAS4_2_SPARE  AVAIL   
 ZR611R4J_ST16000NM004J_16TB_SAS1_2_SPARE  AVAIL   
 ZR5EZEZW_ST16000NM004J_16TB_SAS2_2_SPARE  AVAIL   
 ZR6123NX_ST16000NM004J_16TB_SAS3_2_SPARE  AVAIL   

errors: No known data errors

I have rebooted several times, exported and reimported, and ran zpool clear a few times. Nothing so far as stopped the resilver. Anyone have any idea what's going on? resolved, see top of post

root@aq-zfs1:~# zfs version
zfs-2.1.5-1ubuntu6~22.04.6
zfs-kmod-2.2.2-0ubuntu9.4

Could some one please confirm that Broadcom 9500-8i is a good choice for ZFS (using Ubuntu)? (i.e) it defaults to HBA mode? Thanks

I'd like all of the drives in the array to be known by connection type, model number and then serial. It just makes it eaiser for me to reference them.

So I intend to zpool export them and then use "zpool import -d /dev/disk/by-id" to bring them back in but named that way.

I always get nervous about doing this kind of thing in ZFS so I thought I'd make sure I wasn't about to make a castrophic error and lose data

Hi guys, i am building a new server with 14x 28TB Exos drives, mainly as media server. Access is only 1-2 users and mainly streaming. Files are big (2G-10G).

1. Recordsize: I am planing to use at least 4M, any suggestions for lower or higher?

2a) I am worried about resilver performance when a drive failures… any real world examples from you guys with similar setups? I am hoping wirh big recordsize and large files it would be not so bad?

2b) During resilvering, is the maximum limit the sequential write of one drive since only on the new drive is written and on all other drives is reading? If yes, the 13x drives are almost idle‘ing because rhey are waiting for one drive writes (the new one) ?

3. Any zfs parameters to optimize for this setup? Any recommendations?

4. FYI: I know 2x raidz2 is am option but i need the additional space and do not need more IO.

I’ve been with init7 for years.
Hubrid7 for a while then 3 years ago moved and there was init7 pop.

I was looking around for a backup solution for my homelab and run into: rsync.net

ZFS send capable, I was likening this.

The cherry on top then: their DC in ZH has init7 connectivity!!

I’m in love.

backup server is 4 hops away and for the first ZFS send of about 5TB.

10g connection saturated for the whole 5TB transfer from my TrueNAS instance at home.

It was impressive to see.

As a nerd if init7 ever makes bundles and or deals with these rsync.net guys I would probably propose to Fredy

TLDR: init7 rocks, init7 + rsync.net rocks my socks

I'm jumping into the world of ZFS on Proxmox and would like some advice. I don't need RAID to protect drive failures since the data will be mirrored on each host and isn't critical data.

I will have two pve nodes (and a third as witness) with a 512GB SSD on host 1 mirrored two a 512GB SSD on host 2.

I also have an 8TB and 18TB HDD. I want to put the 18TB on host 1 and the 8tb on host 2. From what I read, I can create an 8TB and 10TB partition to use the extra 10TB for extra data, right?

I may one day match the drives, but I'm saving money for now. I may even sell the 18TB to get 2x 8TB, or similar.

Do you have any suggestions or corrections for this?

Thank you

Hey there. So I've been running a media server and I went to add a drive to it at the end of May and long story short, it turns out that one of the drives in my array had been down for a while without me realizing. I know. I have since educated myself about better error notifications.

Okay. Fine. I'll have to resilver the drive that's been offline. This is my punishment.

So one month of resilvering later, I'm ready to add the new drive. Aaaand I've somehow messed this up. The drive that was resilvering is now missing its label and the pool wants nothing to do with it. As far as I can tell it's healthy and has all the data, it's just missing its ZFS label.

I'm at my wit's end here. I don't want to start over, but unless I can figure this out, I'm going to have to start over. Can anyone tell me how I might fix the missing label from my resilvered drive and get my pool to come back?

Okay. Big writeup time. I hope I don't miss anything important.

Server:

• Debian Linux
• OpenZFS 2.3.2
• Pool "poool": 3-wide RAIDZ1 across three 8TB Seagate drives (ST8000DM004)
• Pool was originally created with plain /dev/sd* paths

The basic problem:
My pool is UNAVAIL and won't import due to insufficient replicas. One of the three drives (ZR163NBH) has no readable ZFS labels, even though it successfully completed a resilver less than 24 hours ago.

Technical breakdown of what happened:
Drive ZR163NBH was UNAVAIL due to a cable issue. I used zpool replace -f with its by-id path to kick off a resilver, which completed successfully after 23 days (6.07T resilvered, 0 errors). After the resilver completed, the pool showed DEGRADED with a second drive (ZR165FYD) now FAULTED, also due to the drives re-enumerating between reboots (the pool was created with /dev/sd* paths, so drives shuffling causes ZFS to lose track of them).

While attempting to fix the FAULTED drive, I ran wipefs, sgdisk --zap-all, and dd commands targeting what I believed was ZR165FYD. However, due to the /dev/sd* enumeration shuffling between reboots, I believe I accidentally hit ZR163NBH instead at some point, wiping its ZFS labels.

Current state:
sudo zpool import -d /dev shows:

pool: poool
  id: 3794980518434859057
state: UNAVAIL
status: One or more devices contains corrupted data.
action: The pool cannot be imported due to damaged devices or data.
config:
        poool                                UNAVAIL  insufficient replicas
          raidz1-0                           UNAVAIL  insufficient replicas
            sdb                              ONLINE
            13851791371351368010             OFFLINE
            ata-ST8000DM004-2U9188_ZR163NBH  UNAVAIL

sudo zdb -l /dev/sdd (where ZR163NBH currently lives) returns:

failed to unpack label 0
failed to unpack label 1
failed to unpack label 2
failed to unpack label 3

However, sudo zdb -l /dev/sdb1 (ZR164YVN, the most up-to-date drive) shows a clean pool config at txg 2521027 with ZR163NBH listed as a direct vdev child (no longer in a replacing vdev), confirming the resilver completed successfully:

txg: 2521027
guid: 5504141216371227705
children[2]:
    type: 'disk'
    guid: 3892473579986994429
    path: '/dev/disk/by-id/ata-ST8000DM004-2U9188_ZR163NBH-part1'
    devid: 'ata-ST8000DM004-2U9188_ZR163NBH-part1'
    DTL: 691

What I've tried:

• zpool import -f -m -F -d /dev 3794980518434859057 returns "no such pool or dataset"
• zpool import -d /dev -f -m -T 2479927 poool gives me "no such pool or dataset"
• Various combinations of -d /dev/sda -d /dev/sdb -d /dev/sdd give me "no such pool available"

So I guess it all comes down to the questions down below:

1. Is there a way to restore or reconstruct the ZFS labels on /dev/sdd so the pool can recognize it and import?
2. Is there a way to force import a RAIDZ1 pool with one UNAVAIL member, treating it as degraded rather than un-importable?
3. The data on /dev/sdd should still be physically intact since we only wiped labels/partition tables, not the actual data blocks. Is there a recovery path here or am I going to have to start over?

All services are stopped and I am not writing anything to these drives while I wait for advice. Any help would be amazing.

EDIT: Seems like the answer isn't the one I was hoping for. I guess it might be time to rebuild the media library. Oh well. Learning experience, I guess.

Hi–Like the title says, I am experiencing my first faulted drive on a five wide RAIDz2 pool. I do plan on replacing the drive but HDDs of this size are painfully expensive right now.

Is there a 'zfs repair' or 'zpool repair' type command that I can run on the drive? Or am I limited to 'zpool replace'?

This is on Debian 13.

zpool status
pool: tank
state: DEGRADED
status: One or more devices are faulted in response to persistent errors.
    Sufficient replicas exist for the pool to continue functioning in a degraded state.
action: Replace the faulted device, or use 'zpool clear' to mark the device repaired.
scan: scrub in progress since Wed Jun 24 09:12:29 2026
    37.7T / 37.7T scanned, 36.2T / 37.7T issued at 960M/s
    0B repaired, 95.91% done, 00:28:05 to go
expand: expanded raidz2-0 copied 29.1T in 23:59:29, on Thu May  7 13:22:23 2026
config:

NAME                        STATE     READ WRITE CKSUM
tank                        DEGRADED     0     0     0
  raidz2-0                  DEGRADED     0     0     0
    wwn-0x5000c500b55f7fbb  ONLINE       0     0     0
    wwn-0x5000c500c6b2ce49  FAULTED     37     0     0  too many errors
    wwn-0x5000c500dcbfd120  ONLINE       0     0     0
    wwn-0x5000c500e9d074f8  ONLINE       0     0     0
    wwn-0x5000c500e9d07d11  ONLINE       0     0     0

Still new managing these types of arrays. Based on the information below, would you suggest the drive be replaced?

root@bci-server11:~# zpool status
  pool: storage
 state: ONLINE
status: One or more devices has experienced an unrecoverable error.  An
	attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
	using 'zpool clear' or replace the device with 'zpool replace'.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-9P
  scan: scrub repaired 0B in 00:31:53 with 0 errors on Sun Jun 14 00:55:56 2026
config:

	NAME                                      STATE     READ WRITE CKSUM
	storage                                   ONLINE       0     0     0
	  raidz2-0                                ONLINE       0     0     0
	    2d5fad83-9063-49fb-b367-75bb904fa5e2  ONLINE       0     0     0
	    137a6807-abdc-4ef4-a46f-85ec22bf4a83  ONLINE       0     0     0
	    065b84e7-2607-4caf-acf4-cca197b9dbb9  ONLINE       0     0     0
	    4f2abdd6-2473-4f3b-b795-c0a662f7089e  ONLINE       0     0     0
	    3d81de5a-c0ca-4c08-9c0b-c21f68c811a3  ONLINE       4     0     0
	    916601cb-88f1-4bd7-bc3b-648b2c6c9034  ONLINE       0     0     0
	    1ab2cbfa-022d-431f-ba30-6007f63f7c7a  ONLINE       0     0     0
	    f3430d58-2888-4ac6-b0cf-b8f38261a571  ONLINE       0     0     0

errors: No known data errors

---

smartctl long test on the drive showing four read errors:

Error counter log:
           Errors Corrected by           Total   Correction     Gigabytes    Total
               ECC          rereads/    errors   algorithm      processed    uncorrected
           fast | delayed   rewrites  corrected  invocations   [10^9 bytes]  errors
read:          0      266         0       266   10139173    1127549.436           4
write:         0        0         0         0     410244      11190.469           0
verify:        0      420         0       420     987492    1112156.765           0

Non-medium error count:        0

SMART Self-test log
Num  Test              Status                 segment  LifeTime  LBA_first_err [SK ASC ASQ]
     Description                              number   (hours)
# 1  Background long   Failed in segment -->       7   34502         134237248 [0x3 0x5d 0x1]

---

related errors in dmesg output

root@bci-server11:~# dmesg | grep 3d81de5a
[1660905.551809] zio pool=storage vdev=/dev/disk/by-partuuid/3d81de5a-c0ca-4c08-9c0b-c21f68c811a3 error=61 type=1 offset=68819107840 size=4096 flags=1573040
[1660912.901714] zio pool=storage vdev=/dev/disk/by-partuuid/3d81de5a-c0ca-4c08-9c0b-c21f68c811a3 error=61 type=1 offset=149270634496 size=4096 flags=1573040
[1660921.618321] zio pool=storage vdev=/dev/disk/by-partuuid/3d81de5a-c0ca-4c08-9c0b-c21f68c811a3 error=61 type=1 offset=167512334336 size=4096 flags=1573040
[1660924.309933] zio pool=storage vdev=/dev/disk/by-partuuid/3d81de5a-c0ca-4c08-9c0b-c21f68c811a3 error=61 type=1 offset=176097927168 size=4096 flags=1573040