ZFS on Linux: Changing device IDs
When I created my zpool on FreeBSD, I went to great lengths to use volume names that matched the serial numbers on the disks. I then made sure the last 7 digits of the serial numbers were visible while the disks were in-situ, so that if one failed I could immediately tell which. This all went out the damn window when I moved the pool to Linux, because ZFS-on-Linux imported them as standard Linux disks (/dev/sd[bcd]).
That’s a pain in the arse, because the ZFS disks are in the higher numbered drive slots, so if I add or remove an extra disk and reboot the machine, Linux re-letters them, and ZFS needs help to figure it out as it seems to give up if nothing matches what’s in the cache file. The cache file, by the way, is binary in part, so editing it by hand didn’t seem like much fun. This is all not to mention that my original intention of knowing exactly which disk to pull was still not working.
For work today, we ran into a situation where we had unusually named disks on one machine, and I figured I’d test it at home, scratching my itch at the same time. What follows is a log of that.
Here’s the situation:
root@ghast:~# zpool status
pool: zeefus
state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
the pool may no longer be accessible by software that does not support
the features. See zpool-features(5) for details.
scan: scrub repaired 0B in 9h34m with 0 errors on Sun Oct 13 09:58:47 2019
config:
NAME STATE READ WRITE CKSUM
zeefus ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
sdc ONLINE 0 0 0
sdd ONLINE 0 0 0
sdb ONLINE 0 0 0
errors: No known data errors
So we stop LXD and export the pool to offline it so that we can work on it:
root@ghast:~# snap stop lxd
Stopped.
root@ghast:~# zpool export zeefus
Now to work out which devices I want to use: the ones in /dev/disk/by-id/ look useful enough:
root@ghast:~# ls /dev/disk/by-id/
ata-FUJITSU_MHZ2080BH_G1_K63RT8732CD3 ata-WDC_WD40EFRX-68N32N0_WD-WCC7K5PDYKAL-part3 ata-WDC_WD40EFRX-68N32N0_WD-WCC7K6VCNC9V-part1 wwn-0x50014ee210055975-part1 wwn-0x50014ee2bab02242
ata-FUJITSU_MHZ2080BH_G1_K63RT8732CD3-part1 ata-WDC_WD40EFRX-68N32N0_WD-WCC7K6STX1EZ ata-WDC_WD40EFRX-68N32N0_WD-WCC7K6VCNC9V-part9 wwn-0x50014ee210055975-part2 wwn-0x50014ee2bab02242-part1
ata-FUJITSU_MHZ2080BH_G1_K63RT8732CD3-part2 ata-WDC_WD40EFRX-68N32N0_WD-WCC7K6STX1EZ-part1 wwn-0x500000e04258de81 wwn-0x50014ee210055975-part3 wwn-0x50014ee2bab02242-part2
ata-WDC_WD40EFRX-68N32N0_WD-WCC7K5PDYKAL ata-WDC_WD40EFRX-68N32N0_WD-WCC7K6STX1EZ-part2 wwn-0x500000e04258de81-part1 wwn-0x50014ee26596e35a wwn-0x50014ee2bab02242-part3
ata-WDC_WD40EFRX-68N32N0_WD-WCC7K5PDYKAL-part1 ata-WDC_WD40EFRX-68N32N0_WD-WCC7K6STX1EZ-part3 wwn-0x500000e04258de81-part2 wwn-0x50014ee26596e35a-part1
ata-WDC_WD40EFRX-68N32N0_WD-WCC7K5PDYKAL-part2 ata-WDC_WD40EFRX-68N32N0_WD-WCC7K6VCNC9V wwn-0x50014ee210055975 wwn-0x50014ee26596e35a-part9
root@ghast:~# zpool import -d /dev/disk/by-id
pool: zeefus
id: 7993881214442468791
state: ONLINE
status: Some supported features are not enabled on the pool.
action: The pool can be imported using its name or numeric identifier, though
some features will not be available without an explicit 'zpool upgrade'.
config:
zeefus ONLINE
raidz1-0 ONLINE
wwn-0x50014ee26596e35a ONLINE
wwn-0x50014ee210055975 ONLINE
wwn-0x50014ee2bab02242 ONLINE
Aww, fuck - it imported by the wrong ones. Those are predictable, sure (at least, to some extent), but they don’t scratch my first itch. After several attempts at working out how to coax ZFS to use the other ones, I finally just rage-quit and removed the offending symlinks temporarily:
root@ghast:~# zpool export zeefus
root@ghast:/dev# cd disk/by-id
root@ghast:/dev/disk/by-id# mkdir /root/disks
root@ghast:/dev/disk/by-id# mv wwn-0x50014ee210055975* /root/disks/
root@ghast:/dev/disk/by-id# mv wwn-0x50014ee26596e35a* /root/disks/
root@ghast:/dev/disk/by-id# mv wwn-0x50014ee2bab02242* /root/disks/
root@ghast:/dev/disk/by-id# zpool import -d /dev/disk/by-id -a
cannot mount '/zeefus': directory is not empty
root@ghast:/dev/disk/by-id# zpool status
pool: zeefus
state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
the pool may no longer be accessible by software that does not support
the features. See zpool-features(5) for details.
scan: scrub canceled on Thu Oct 31 03:01:42 2019
config:
NAME STATE READ WRITE CKSUM
zeefus ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
ata-WDC_WD40EFRX-68N32N0_WD-WCC7K6VCNC9V ONLINE 0 0 0
ata-WDC_WD40EFRX-68N32N0_WD-WCC7K5PDYKAL ONLINE 0 0 0
ata-WDC_WD40EFRX-68N32N0_WD-WCC7K6STX1EZ ONLINE 0 0 0
errors: No known data errors
root@ghast:/dev/disk/by-id# mv /root/disks/* ./
A quick reboot to make sure it comes back:
root@ghast:/dev/disk/by-id# shutdown -r now
...
fwaggle@ghast:~$ zpool status
pool: zeefus
state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
the pool may no longer be accessible by software that does not support
the features. See zpool-features(5) for details.
scan: scrub canceled on Thu Oct 31 03:01:42 2019
config:
NAME STATE READ WRITE CKSUM
zeefus ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
ata-WDC_WD40EFRX-68N32N0_WD-WCC7K6VCNC9V ONLINE 0 0 0
ata-WDC_WD40EFRX-68N32N0_WD-WCC7K5PDYKAL ONLINE 0 0 0
ata-WDC_WD40EFRX-68N32N0_WD-WCC7K6STX1EZ ONLINE 0 0 0
errors: No known data errors
Hopefully I haven’t broken anything long-term!