▲ 19 r/zfs
Resilver made it to 1440707.03% before completing
Typed out this post to ask for advice but then the resilver completed right before I hit post lol. So now I'm just posting this because I could only find one other person mentioning this from like 10 years ago and it may be useful knowledge for other people seeing this behavior.
On Saturday, in a pool that spans 5 JBODs, zfs started reporting drives from one of the JBODs as degraded and automatically started resilvering to available hotspares. I am not sure what caused that JBOD to start marking these drives degraded but smartctl showed no errors (except for non-medium errors). I exported the pool over the weekend to avoid any more issues showing up before I could dig in further. Today, I power-cycled the JBOD and moved SAS cables to different ports, and then reimported the pool. I ran zpool clear on each drive after checking its SMART status. Then I detached each hotspare that had started resilvering.
It has been a few hours now with the drives all showing ONLINE and data seems to be all present and accounted for. But the resilver progress continues anyway finished, well past 100%.
zpool status -v output:
root@aq-zfs1:~# zpool status -v zpool1
pool: zpool1
state: ONLINE
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Sat Jun 27 11:20:54 2026
1.08P scanned at 180G/s, 16.0E issued at 0B/s, 1.14P total
0B resilvered, 1440707.37% done, no estimated completion time
config:
NAME STATE READ WRITE CKSUM
zpool1 ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
ZRS1AKA2_ST16000NM004J_16TB_SAS1_1 ONLINE 0 0 0
ZRS1AK8Y_ST16000NM004J_16TB_SAS1_3 ONLINE 0 0 0
ZRS1AJHY_ST16000NM004J_16TB_SAS2_1 ONLINE 0 0 0
ZRS1AK07_ST16000NM004J_16TB_SAS2_3 ONLINE 0 0 0
ZRS18J06_ST16000NM004J_16TB_SAS3_1 ONLINE 0 0 0
ZRS18HQ1_ST16000NM004J_16TB_SAS3_3 ONLINE 0 0 0
ZRS18J6M_ST16000NM004J_16TB_SAS4_1 ONLINE 0 0 0
ZRS18J0T_ST16000NM004J_16TB_SAS4_3 ONLINE 0 0 0
ZRS18HQV_ST16000NM004J_16TB_SAS5_1 ONLINE 0 0 0
ZRS18HNB_ST16000NM004J_16TB_SAS5_3 ONLINE 0 0 0
raidz2-1 ONLINE 0 0 0
ZRS1AJHC_ST16000NM004J_16TB_SAS1_5 ONLINE 0 0 0
ZRS1AJVH_ST16000NM004J_16TB_SAS1_15 ONLINE 0 0 0
ZRS18J63_ST16000NM004J_16TB_SAS2_5 ONLINE 0 0 0
ZRS18HKV_ST16000NM004J_16TB_SAS2_15 ONLINE 0 0 0
ZRS18HNC_ST16000NM004J_16TB_SAS3_5 ONLINE 0 0 0
ZRS18HVJ_ST16000NM004J_16TB_SAS3_15 ONLINE 0 0 0
ZRS1AJQF_ST16000NM004J_16TB_SAS4_5 ONLINE 0 0 0
ZRS1AK00_ST16000NM004J_16TB_SAS4_15 ONLINE 0 0 0
ZRS18HWR_ST16000NM004J_16TB_SAS5_5 ONLINE 0 0 0
ZRS1AJK6_ST16000NM004J_16TB_SAS5_15 ONLINE 0 0 0
raidz2-2 ONLINE 0 0 0
ZRS18J74_ST16000NM004J_16TB_SAS1_17 ONLINE 0 0 0
ZRS18HMY_ST16000NM004J_16TB_SAS1_19 ONLINE 0 0 0
ZRS18J1M_ST16000NM004J_16TB_SAS2_17 ONLINE 0 0 0
ZRS1AK38_ST16000NM004J_16TB_SAS2_19 ONLINE 0 0 0
ZR611S70_ST16000NM004J_16TB_SAS3_17 ONLINE 0 0 0
ZR6123GN_ST16000NM004J_16TB_SAS3_19 ONLINE 0 0 0
ZRS1AKAC_ST16000NM004J_16TB_SAS4_17 ONLINE 0 0 0
ZRS1AK7K_ST16000NM004J_16TB_SAS4_19 ONLINE 0 0 0
ZRS18J4F_ST16000NM004J_16TB_SAS5_17 ONLINE 0 0 0
ZRS1AJPE_ST16000NM004J_16TB_SAS5_19 ONLINE 0 0 0
raidz2-3 ONLINE 0 0 0
ZRS1AJY9_ST16000NM004J_16TB_SAS1_29 ONLINE 0 0 0
ZRS1AJMJ_ST16000NM004J_16TB_SAS1_31 ONLINE 0 0 0
ZRS18J20_ST16000NM004J_16TB_SAS2_29 ONLINE 0 0 0
ZRS18HYL_ST16000NM004J_16TB_SAS2_31 ONLINE 0 0 0
ZR6123GK_ST16000NM004J_16TB_SAS3_29 ONLINE 0 0 0
ZR611RQQ_ST16000NM004J_16TB_SAS3_31 ONLINE 0 0 0
ZRS1AK3Z_ST16000NM004J_16TB_SAS4_29 ONLINE 0 0 0
ZRS1AKC2_ST16000NM004J_16TB_SAS4_31 ONLINE 0 0 0
ZRS18HZY_ST16000NM004J_16TB_SAS5_29 ONLINE 0 0 0
ZRS1AK2H_ST16000NM004J_16TB_SAS5_31 ONLINE 0 0 0
raidz2-4 ONLINE 0 0 0
ZRS18J0S_ST16000NM004J_16TB_SAS1_33 ONLINE 0 0 0
ZRS18HT9_ST16000NM004J_16TB_SAS1_43 ONLINE 0 0 0
ZRS18HSZ_ST16000NM004J_16TB_SAS2_33 ONLINE 0 0 0
ZRS1AJMW_ST16000NM004J_16TB_SAS2_43 ONLINE 0 0 0
ZR612386_ST16000NM004J_16TB_SAS3_33 ONLINE 0 0 0
ZR611Q1G_ST16000NM004J_16TB_SAS3_43 ONLINE 0 0 0
ZRS1AJSF_ST16000NM004J_16TB_SAS4_33 ONLINE 0 0 0
ZRS1AJJD_ST16000NM004J_16TB_SAS4_43 ONLINE 0 0 0
ZRS1AJGQ_ST16000NM004J_16TB_SAS5_33 ONLINE 0 0 0
ZRS1AK9S_ST16000NM004J_16TB_SAS5_43 ONLINE 0 0 0
raidz2-5 ONLINE 0 0 0
ZRS1AJLW_ST16000NM004J_16TB_SAS1_45 ONLINE 0 0 0
ZRS18J5A_ST16000NM004J_16TB_SAS1_47 ONLINE 0 0 0
ZRS18HLM_ST16000NM004J_16TB_SAS2_45 ONLINE 0 0 0
ZRS18HLS_ST16000NM004J_16TB_SAS2_47 ONLINE 0 0 0
ZR611PPV_ST16000NM004J_16TB_SAS3_45 ONLINE 0 0 0
ZR611PRP_ST16000NM004J_16TB_SAS3_47 ONLINE 0 0 0
ZRS1AJS5_ST16000NM004J_16TB_SAS4_45 ONLINE 0 0 0
ZRS1AK5C_ST16000NM004J_16TB_SAS4_47 ONLINE 0 0 0
ZRS1AK8D_ST16000NM004J_16TB_SAS5_45 ONLINE 0 0 0
ZRS1AK0A_ST16000NM004J_16TB_SAS5_47 ONLINE 0 0 0
raidz2-6 ONLINE 0 0 0
ZRS1AK6M_ST16000NM004J_16TB_SAS1_57 ONLINE 0 0 0
ZRS1AKAS_ST16000NM004J_16TB_SAS1_59 ONLINE 0 0 0
ZRS18J75_ST16000NM004J_16TB_SAS2_57 ONLINE 0 0 0
ZRS1AJTE_ST16000NM004J_16TB_SAS2_59 ONLINE 0 0 0
ZR611AG5_ST16000NM004J_16TB_SAS3_57 ONLINE 0 0 0
ZR611PYT_ST16000NM004J_16TB_SAS3_59 ONLINE 0 0 0
ZRS1AJTH_ST16000NM004J_16TB_SAS4_57 ONLINE 0 0 0
ZRS1AKC3_ST16000NM004J_16TB_SAS4_59 ONLINE 0 0 0
ZRS18J18_ST16000NM004J_16TB_SAS5_57 ONLINE 0 0 0
ZRS1AJK0_ST16000NM004J_16TB_SAS5_59 ONLINE 0 0 0
raidz2-7 ONLINE 0 0 0
ZRS1AJRX_ST16000NM004J_16TB_SAS1_61 ONLINE 0 0 0
ZRS1AJXY_ST16000NM004J_16TB_SAS1_71 ONLINE 0 0 0
ZRS18HMN_ST16000NM004J_16TB_SAS2_61 ONLINE 0 0 0
ZRS1AJZ2_ST16000NM004J_16TB_SAS2_71 ONLINE 0 0 0
ZRS1AJX9_ST16000NM004J_16TB_SAS3_61 ONLINE 0 0 0
ZRS1AK23_ST16000NM004J_16TB_SAS3_71 ONLINE 0 0 0
ZRS1AK2M_ST16000NM004J_16TB_SAS4_61 ONLINE 0 0 0
ZRS1AK0Q_ST16000NM004J_16TB_SAS4_71 ONLINE 0 0 0
ZRS18J0K_ST16000NM004J_16TB_SAS5_61 ONLINE 0 0 0
ZRS18HZW_ST16000NM004J_16TB_SAS5_71 ONLINE 0 0 0
raidz2-8 ONLINE 0 0 0
ZRS1AKBL_ST16000NM004J_16TB_SAS1_73 ONLINE 0 0 0
ZRS1AJXA_ST16000NM004J_16TB_SAS1_75 ONLINE 0 0 0
ZRS1AKCZ_ST16000NM004J_16TB_SAS2_73 ONLINE 0 0 0
ZRS18HLW_ST16000NM004J_16TB_SAS2_75 ONLINE 0 0 0
ZRS1AKDL_ST16000NM004J_16TB_SAS3_73 ONLINE 0 0 0
ZRS1AJVC_ST16000NM004J_16TB_SAS3_75 ONLINE 0 0 0
ZRS18HTY_ST16000NM004J_16TB_SAS4_73 ONLINE 0 0 0
ZRS1AJSG_ST16000NM004J_16TB_SAS4_75 ONLINE 0 0 0
ZRS1AJW5_ST16000NM004J_16TB_SAS5_73 ONLINE 0 0 0
ZRS1AK8M_ST16000NM004J_16TB_SAS5_75 ONLINE 0 0 0
special
mirror-9 ONLINE 0 0 0
24212X803118_WD_BLACK_4TB_NVME_da ONLINE 0 0 0
24212X801125_WD_BLACK_4TB_NVME_9b ONLINE 0 0 0
24212X801132_WD_BLACK_4TB_NVME_dc ONLINE 0 0 0
mirror-10 ONLINE 0 0 0
24212X804611_WD_BLACK_4TB_NVME_ed ONLINE 0 0 0
24212X801675_WD_BLACK_4TB_NVME_9a ONLINE 0 0 0
24233N805374_WD_BLACK_4TB_NVME_ee ONLINE 0 0 0
logs
mirror-11 ONLINE 0 0 0
24212X800463_WD_BLACK_4TB_NVME_d9 ONLINE 0 0 0
24212X801126_WD_BLACK_4TB_NVME_99 ONLINE 0 0 0
spares
ZR611RTT_ST16000NM004J_16TB_SAS5_2_SPARE AVAIL
ZR611RNN_ST16000NM004J_16TB_SAS4_2_SPARE AVAIL
ZR611R4J_ST16000NM004J_16TB_SAS1_2_SPARE AVAIL
ZR5EZEZW_ST16000NM004J_16TB_SAS2_2_SPARE AVAIL
ZR6123NX_ST16000NM004J_16TB_SAS3_2_SPARE AVAIL
errors: No known data errors
I have rebooted several times, exported and reimported, and ran zpool clear a few times. Nothing so far as stopped the resilver. Anyone have any idea what's going on? resolved, see top of post
root@aq-zfs1:~# zfs version
zfs-2.1.5-1ubuntu6~22.04.6
zfs-kmod-2.2.2-0ubuntu9.4
u/VergilTheHuragok — 6 days ago