Replacing a Failed Disk in a zpool
In my backup server I have a zpool called tank which consists of four mirrored vdevs. Each vdev has two 4TB drives. The other day I started getting email alerts from smartmon-tools telling me that one of my drives has 8 pending sectors. Sometimes this happens and will correct itself and every once in a while that number will increase but will never go below 8. Additionally, you can see the kernel is reporting errors from the drive as well.
smartctl -A /dev/sdf
smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.10.0-29-amd64] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF READ SMART DATA SECTION ===
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000b 100 100 016 Pre-fail Always - 0
2 Throughput_Performance 0x0005 137 137 054 Pre-fail Offline - 79
3 Spin_Up_Time 0x0007 126 126 024 Pre-fail Always - 641 (Average 588)
4 Start_Stop_Count 0x0012 100 100 000 Old_age Always - 85
5 Reallocated_Sector_Ct 0x0033 100 100 005 Pre-fail Always - 0
7 Seek_Error_Rate 0x000b 100 100 067 Pre-fail Always - 0
8 Seek_Time_Performance 0x0005 119 119 020 Pre-fail Offline - 35
9 Power_On_Hours 0x0012 096 096 000 Old_age Always - 32646
10 Spin_Retry_Count 0x0013 100 100 060 Pre-fail Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 85
192 Power-Off_Retract_Count 0x0032 099 099 000 Old_age Always - 1411
193 Load_Cycle_Count 0x0012 099 099 000 Old_age Always - 1411
194 Temperature_Celsius 0x0002 076 076 000 Old_age Always - 78 (Min/Max 22/88)
196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 0
197 Current_Pending_Sector 0x0022 100 100 000 Old_age Always - 8
198 Offline_Uncorrectable 0x0008 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x000a 200 200 000 Old_age Always - 1
dmesg
...snip...
[1066524.427216] ata6.00: configured for UDMA/133
[1066524.427278] sd 5:0:0:0: [sdf] tag#24 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=6s
[1066524.427287] sd 5:0:0:0: [sdf] tag#24 Sense Key : Medium Error [current]
[1066524.427293] sd 5:0:0:0: [sdf] tag#24 Add. Sense: Unrecovered read error - auto reallocate failed
[1066524.427301] sd 5:0:0:0: [sdf] tag#24 CDB: Read(16) 88 00 00 00 00 01 3c 2f fd 68 00 00 08 00 00 00
[1066524.427308] blk_update_request: I/O error, dev sdf, sector 5304745320 op 0x0:(READ) flags 0x700 phys_seg 16 prio class 0
[1066524.427390] zio pool=tank vdev=/dev/disk/by-id/wwn-0x5000cca23dede80e-part1 error=5 type=1 offset=2716028555264 size=1048576 flags=40080cb0
[1066524.427416] ata6: EH complete
[1066530.718364] ata6.00: exception Emask 0x0 SAct 0x3800 SErr 0x0 action 0x0
[1066530.718432] ata6.00: irq_stat 0x40000008
[1066530.718490] ata6.00: failed command: READ FPDMA QUEUED
[1066530.718554] ata6.00: cmd 60/00:58:68:45:30/08:00:3c:01:00/40 tag 11 ncq dma 1048576 in
res 51/40:80:e8:45:30/00:07:3c:01:00/40 Emask 0x409 (media error)
[1066530.718640] ata6.00: status: { DRDY ERR }
[1066530.718694] ata6.00: error: { UNC }
[1066530.961239] ata6.00: configured for UDMA/133
[1066530.961279] sd 5:0:0:0: [sdf] tag#11 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=3s
[1066530.961288] sd 5:0:0:0: [sdf] tag#11 Sense Key : Medium Error [current]
[1066530.961294] sd 5:0:0:0: [sdf] tag#11 Add. Sense: Unrecovered read error - auto reallocate failed
[1066530.961301] sd 5:0:0:0: [sdf] tag#11 CDB: Read(16) 88 00 00 00 00 01 3c 30 45 68 00 00 08 00 00 00
[1066530.961308] blk_update_request: I/O error, dev sdf, sector 5304763752 op 0x0:(READ) flags 0x700 phys_seg 16 prio class 0
[1066530.961390] zio pool=tank vdev=/dev/disk/by-id/wwn-0x5000cca23dede80e-part1 error=5 type=1 offset=2716037992448 size=1048576 flags=40080cb0
[1066530.961432] ata6: EH complete
[1066534.314389] ata6.00: exception Emask 0x0 SAct 0x5c8000 SErr 0x0 action 0x0
[1066534.314454] ata6.00: irq_stat 0x40000008
[1066534.314511] ata6.00: failed command: READ FPDMA QUEUED
[1066534.314574] ata6.00: cmd 60/00:90:68:4d:30/08:00:3c:01:00/40 tag 18 ncq dma 1048576 in
res 51/40:48:20:4e:30/00:07:3c:01:00/40 Emask 0x409 (media error)
[1066534.314692] ata6.00: status: { DRDY ERR }
[1066534.314745] ata6.00: error: { UNC }
...snip...
Looking at the zpool status you can see that one of the drives is FAULTED leaving mirror-1 in a DEGRADED state. In this case the mirror-1 vdev has lost redundancy, but if I had three or more mirrored drives, this would be considered a warning instead of a critical warning. A faulted component or device in the vdev means that it’s inaccessible, but still there. If the device or component was marked as REMOVED or UNAVAIL then it’s completely gone. Since the drive is FAULTED I could try resetting it but given that there is something more than a minor hiccup won’t fix the problem.
zpool status
pool: tank
state: DEGRADED
status: One or more devices are faulted in response to persistent errors.
Sufficient replicas exist for the pool to continue functioning in a
degraded state.
action: Replace the faulted device, or use 'zpool clear' to mark the device
repaired.
scan: scrub repaired 0B in 07:12:43 with 0 errors on Thu Jun 13 22:51:16 2024
config:
NAME STATE READ WRITE CKSUM
tank DEGRADED 0 0 0
mirror-0 ONLINE 0 0 0
wwn-0x5000cca22be4ad8b ONLINE 0 0 0
wwn-0x5000cca22bcaac7e ONLINE 0 0 0
mirror-1 DEGRADED 0 0 0
wwn-0x5000cca23de313c1 ONLINE 0 0 0
wwn-0x5000cca23dede80e FAULTED 15 0 21 too many errors
mirror-2 ONLINE 0 0 0
wwn-0x5000c5007b4ef77f ONLINE 0 0 0
wwn-0x5000c5008cea8033 ONLINE 0 0 0
mirror-3 ONLINE 0 0 0
wwn-0x5000cca249d4044b ONLINE 0 0 0
wwn-0x5000cca25dde5aaf ONLINE 0 0 0
mirror-4 ONLINE 0 0 0
wwn-0x5000c5007b5416c0 ONLINE 0 0 0
wwn-0x5000cca249c3500f ONLINE 0 0 0
errors: No known data errors
Finding the Failed Disk
The Manual Way
If it’s easy enough to take the side off your computer or you keep excellent records of which drive is in which bay, then you can get the information using the lsblk command then cross reference it with your records.
lsblk -I 8 -d -o PATH,WWN,VENDOR,MODEL,SERIAL | grep 0x5000cca23dede80e
/dev/sdf 0x5000cca23dede80e ATA Hitachi_HUS724040ALE641 PBK7ZM7S
ledctl
There is a package that called ledmon which allows you to cause the LED for the device to blink. The catch is that the drive, controller, and storage backplane controller need to support SES-2 and SMP, LED messages over SGPIO, or VMD and NPEM.
ledctl locate=/dev/sdf
ledctl locate_off=/dev/sdf
dd or pv
If ledctl doesn’t work and you have a case with indivual bays that have lights on them, then you can run the use the dd or pv command.
NOTE: The
ddcommand is already installed whereas thepvcommand has to be installed manually.
dd if=/dev/sdf of=/dev/null bs=2048
pv -pberta /dev/sdf > /dev/null
smartctl
If dd or pv doesn’t work then the last option is to use smartctl
while :; do smartctl -a /dev/sdf > null ; done ;
Replacing the Failed Drive
There are two ways that we can go about this. The first is replacing the failed drive with a spare that’s already in the system while the other is removing and replacing the drive. These two methods follow the same procedure:
- Find failed drive
- Take failed drive offline
- Physically replace the failed drive
- Replace (zpool) the drive
- Wait
zpool offline tank /dev/disk/by-id/wwn-0x5000cca23dede80e
zpool status
pool: tank
state: DEGRADED
status: One or more devices are faulted in response to persistent errors.
Sufficient replicas exist for the pool to continue functioning in a
degraded state.
action: Replace the faulted device, or use 'zpool clear' to mark the device
repaired.
scan: scrub repaired 0B in 07:12:43 with 0 errors on Thu Jun 13 22:51:16 2024
config:
NAME STATE READ WRITE CKSUM
tank DEGRADED 0 0 0
mirror-0 ONLINE 0 0 0
wwn-0x5000cca22be4ad8b ONLINE 0 0 0
wwn-0x5000cca22bcaac7e ONLINE 0 0 0
mirror-1 DEGRADED 0 0 0
wwn-0x5000cca23de313c1 ONLINE 0 0 0
wwn-0x5000cca23dede80e FAULTED 15 0 21 too many errors
mirror-2 ONLINE 0 0 0
wwn-0x5000c5007b4ef77f ONLINE 0 0 0
wwn-0x5000c5008cea8033 ONLINE 0 0 0
mirror-3 ONLINE 0 0 0
wwn-0x5000cca249d4044b ONLINE 0 0 0
wwn-0x5000cca25dde5aaf ONLINE 0 0 0
mirror-4 ONLINE 0 0 0
wwn-0x5000c5007b5416c0 ONLINE 0 0 0
wwn-0x5000cca249c3500f ONLINE 0 0 0
errors: No known data errors
Remove the drive and replace the drive.
zpool status
pool: tank
state: DEGRADED
status: One or more devices has been removed by the administrator.
Sufficient replicas exist for the pool to continue functioning in a
degraded state.
action: Online the device using zpool online' or replace the device with
'zpool replace'.
scan: scrub repaired 0B in 07:12:43 with 0 errors on Thu Jun 13 22:51:16 2024
config:
NAME STATE READ WRITE CKSUM
tank DEGRADED 0 0 0
mirror-0 ONLINE 0 0 0
wwn-0x5000cca22be4ad8b ONLINE 0 0 0
wwn-0x5000cca22bcaac7e ONLINE 0 0 0
mirror-1 DEGRADED 0 0 0
wwn-0x5000cca23de313c1 ONLINE 0 0 0
wwn-0x5000cca23dede80e REMOVED 0 0 0
mirror-2 ONLINE 0 0 0
wwn-0x5000c5007b4ef77f ONLINE 0 0 0
wwn-0x5000c5008cea8033 ONLINE 0 0 0
mirror-3 ONLINE 0 0 0
wwn-0x5000cca249d4044b ONLINE 0 0 0
wwn-0x5000cca25dde5aaf ONLINE 0 0 0
mirror-4 ONLINE 0 0 0
wwn-0x5000c5007b5416c0 ONLINE 0 0 0
wwn-0x5000cca249c3500f ONLINE 0 0 0
errors: No known data errors
zpool replace tank /dev/disk/by-id/wwn-0x5000cca23dede80e /dev/disk/by-id/wwn-0x5000cca24cf56668
zpool status
pool: tank
state: DEGRADED
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Fri Jun 14 01:34:40 2024
3.81T scanned at 24.7G/s, 2.91T issued at 18.9G/s, 12.4T total
502M resilvered, 23.51% done, 00:08:34 to go
config:
NAME STATE READ WRITE CKSUM
tank DEGRADED 0 0 0
mirror-0 ONLINE 0 0 0
wwn-0x5000cca22be4ad8b ONLINE 0 0 0
wwn-0x5000cca22bcaac7e ONLINE 0 0 0
mirror-1 DEGRADED 0 0 0
wwn-0x5000cca23de313c1 ONLINE 0 0 0
replacing-1 DEGRADED 0 0 0
wwn-0x5000cca23dede80e REMOVED 0 0 0
wwn-0x5000cca24cf56668 ONLINE 0 0 0 (resilvering)
mirror-2 ONLINE 0 0 0
wwn-0x5000c5007b4ef77f ONLINE 0 0 0
wwn-0x5000c5008cea8033 ONLINE 0 0 0
mirror-3 ONLINE 0 0 0
wwn-0x5000cca249d4044b ONLINE 0 0 0
wwn-0x5000cca25dde5aaf ONLINE 0 0 0
mirror-4 ONLINE 0 0 0
wwn-0x5000c5007b5416c0 ONLINE 0 0 0
wwn-0x5000cca249c3500f ONLINE 0 0 0
errors: No known data errors
To watch the drive resilver:
watch zpool status
Bonus
During the resilvering process the replacement drive failed so I was left with this:
pool: tank
state: DEGRADED
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Fri Jun 14 01:34:40 2024
3.45T scanned at 9.62G/s, 2.67T issued at 7.46G/s, 12.4T total
0B resilvered, 21.59% done, 00:22:12 to go
config:
NAME STATE READ WRITE CKSUM
tank DEGRADED 0 0 0
mirror-0 ONLINE 0 0 0
wwn-0x5000cca22be4ad8b ONLINE 0 0 0
wwn-0x5000cca22bcaac7e ONLINE 0 0 0
mirror-1 DEGRADED 0 0 0
wwn-0x5000cca23de313c1 ONLINE 0 0 0
replacing-1 UNAVAIL 0 0 0 insufficient replicas
wwn-0x5000cca23dede80e REMOVED 0 0 0
wwn-0x5000cca24cf56668 REMOVED 0 0 0
mirror-2 ONLINE 0 0 0
wwn-0x5000c5007b4ef77f ONLINE 0 0 0
wwn-0x5000c5008cea8033 ONLINE 0 0 0
mirror-3 ONLINE 0 0 0
wwn-0x5000cca249d4044b ONLINE 0 0 0
wwn-0x5000cca25dde5aaf ONLINE 0 0 0
mirror-4 ONLINE 0 0 0
wwn-0x5000c5007b5416c0 ONLINE 0 0 0
wwn-0x5000cca249c3500f ONLINE 0 0 0
errors: No known data errors
The original drive that failed was wwn-0x5000cca23dede80e so I replaced the original drive again with a new one.
zpool replace tank /dev/disk/by-id/wwn-0x5000cca23dede80e /dev/disk/by-id/wwn-0x5000c500e8c81e6f
zpool status
pool: tank
state: DEGRADED
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Fri Jun 14 01:34:40 2024
4.39T scanned at 4.35G/s, 3.49T issued at 3.46G/s, 12.4T total
41.4G resilvered, 28.17% done, 00:43:56 to go
config:
NAME STATE READ WRITE CKSUM
tank DEGRADED 0 0 0
mirror-0 ONLINE 0 0 0
wwn-0x5000cca22be4ad8b ONLINE 0 0 0
wwn-0x5000cca22bcaac7e ONLINE 0 0 0
mirror-1 DEGRADED 0 0 0
wwn-0x5000cca23de313c1 ONLINE 0 0 0
replacing-1 DEGRADED 0 0 0
wwn-0x5000cca23dede80e REMOVED 0 0 0
wwn-0x5000cca24cf56668 REMOVED 0 0 0
wwn-0x5000c500e8c81e6f ONLINE 0 0 0 (resilvering)
mirror-2 ONLINE 0 0 0
wwn-0x5000c5007b4ef77f ONLINE 0 0 0
wwn-0x5000c5008cea8033 ONLINE 0 0 0
mirror-3 ONLINE 0 0 0
wwn-0x5000cca249d4044b ONLINE 0 0 0
wwn-0x5000cca25dde5aaf ONLINE 0 0 0
mirror-4 ONLINE 0 0 0
wwn-0x5000c5007b5416c0 ONLINE 0 0 0
wwn-0x5000cca249c3500f ONLINE 0 0 0
errors: No known data errors