[BUG]: Unable to upgrade if /boot/efi is setup as an mdadm RAID1

[ramereth]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

I was trying to upgrade an aarch64 node from AlmaLinux 8 to AlmaLinux 9 when I encountered an issue during the upgrade process in the booted initramfs.

While this was on aarch64, I believe this likely affects x86_64 or any other architecture. Here's the layout of the system and fstab for a reference:

Block devices:

NAME                      MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINTS
nvme0n1                   259:0    0 894.3G  0 disk  
|-nvme0n1p1               259:1    0     1G  0 part  
| `-md127                   9:127  0  1022M  0 raid1 /boot
|-nvme0n1p2               259:2    0   600M  0 part  
| `-md125                   9:125  0 599.9M  0 raid1 /boot/efi
`-nvme0n1p3               259:3    0 892.7G  0 part  
  `-md126                   9:126  0 892.5G  0 raid1 
    |-almalinux_arm1-root 253:0    0 764.5G  0 lvm   /
    `-almalinux_arm1-swap 253:1    0   128G  0 lvm   [SWAP]
nvme1n1                   259:4    0 931.5G  0 disk  
|-nvme1n1p1               259:5    0     1G  0 part  
| `-md127                   9:127  0  1022M  0 raid1 /boot
|-nvme1n1p2               259:6    0   600M  0 part  
| `-md125                   9:125  0 599.9M  0 raid1 /boot/efi
`-nvme1n1p3               259:7    0 892.7G  0 part  
  `-md126                   9:126  0 892.5G  0 raid1 
    |-almalinux_arm1-root 253:0    0 764.5G  0 lvm   /
    `-almalinux_arm1-swap 253:1    0   128G  0 lvm   [SWAP]

fstab

/dev/mapper/almalinux_arm1-root /                       ext4    defaults        1 1
UUID=ef8fe825-99ef-4c01-aab1-96a46b642e82 /boot                   ext4    defaults        1 2
UUID=3D34-9948          /boot/efi               vfat    umask=0077,shortname=winnt 0 2
/dev/mapper/almalinux_arm1-swap none                    swap    defaults        0 0

Unfortunately I don't have any error output as I've already worked around it, however it's likely reproducible. The problem seems to be the fact that the systems doesn't know how to reinstall grub since it tries to use the RAID device instead of each disk.

I think this might be an issue with grub-install, however I know the AlmaLinux installer allowed this configuration (at least that's my recollection).

The work around was to:

1. Break the RAID1 of /boot/efi and reconfigure the system to use one disk
2. After the upgrade, rebuild the RAID1 again manually by making a copy of the contents of /boot/efi

Expected Behavior

I expected to leapp at least send me a warning before doing the upgrade with the suggested fix.

If you want, I'll see if I can replicate this inside of a VM to make sure it just wasn't some kind of one-off.

Steps To Reproduce

No response

Anything else?

No response

Search terms

mdadm efi

[yuravk]

This issue is currently being tracked by upstream in RHEL-58913

[mmiller7]

I'm also hitting this issue going from Alma 8 to Alma 9 using elevate/leapp. And after it fails once, it seems there is no way to "start over" to retry without kernel panics.

Strangely the CentOS 7 to Alma 8 Elevate/Leapp upgrade with this exact same configuration worked properly.

I think I'm also seeing a similar issue with my /home area which is LVM on MDADM RAID1 where it appears to not try and activate the RAID arrays for EFI nor for my data LVMs and then complains they can't be found.

Hoping there's a solution soon, the software I need to use doesn't support EL8 due to library conflicts so I'm stuck at EL7 until the EL9 upgrade works properly. My best current "guess" is rearranging the system to a non-RAID config for the problem partitions and then re-rearranging it back to RAID post-upgrade but I am still not successfully worked thru all the problem partitions...and have to start the multi-hour process from the beginning every time.

UPDATE - Workaround:
If this helps, it was painful but got me thru my upgrade. Its a heck of a ride and took me ALL day to work out, but I got thru the one system I really needed to get updated successfully. Basically I'm taking the RAID out of the equation for the update and putting it back after.

This is intentionally a bit vague because its a very dangerous procedure writing/overwriting block devices with your data. If you don't understand the commands, you may wish to wait for the bugfix or do a clean reinstall. Mistakes WILL hose your system, data, or both. Have backups of EVERYTHING as usual.

Also note I had to restore my system from backup images before I could follow this - if I got to the point where Elevate/leapp failed, everything got worse and worse trying to fix and retry without rolling the whole system back. I had made a full disk image of my EFI, boot, and rootfs partitions immediately before the upgrade so I rolled back by simply restoring those images to their respective places.

1. ensure your /etc/fstab uses UUID to find the EFI partition
2. boot from a recovery media or live media (e.g. Fedora live)
3. use dd to back up the /dev/md### partition to somewhere safe (you'll need this to restore the disk partition signature later)
4. connect a spare disposable USB drive which will be your boot drive during the procedure
5. use dd to copy the /dev/md### partiton on top of the USB drive partition (ignore size mismatch, seems fine as long as the USB is bigger)
6. use dd to ERASE the /dev/md### EFI partiton by writing /dev/zero over top of the /dev/md### so the EFI partition can't be accidentally detected on the RAID-1 (but don't mess with the underlying /dev/sdX partitions, you don't want to wipe out the RAID array itself)
7. set up "root" user with valid login password (and if needed PermitRootLogin=yes SSH)
8. edit /etc/fstab to comment out everything except: efi, boot, root, swap. This includes no home areas.
9. run the upgrade procedure per elevate/leapp, verify completion
10. edit /etc/fstab to uncomment your various mountpoints
11. reboot and make sure it all works
12. review lsblk or similar to find the md### your EFI RAID was supposed to be
13. use dd to copy your backed up image file (which will be the right size for the original RAID) back to /dev/md### where the EFI RAID belongs. This will restore the filesystem and disk/partition signature that was erased with zeros
14. mount /dev/md### on /media or wherever, and delete all the old files that were in it
15. move all the files from /boot/efi/* to /media or wherever you mounted the md###, this will restore the "new" boot files onto the "original" RAID array
16. unmount /boot/efi
17. unmount /media (or wherever)
18. unplug the temporary USB drive
19. run mount -a and it should find the /dev/md### EFI partition and mount it at /boot/efi by its UUID.
20. verify /boot/efi/* contains files
21. reboot and verify the system boots properly off the RAID EFI area
22. use gparted, HP Disk Format Tool, dd/fdisk, etc. to totally wipe/repartition the temporary USB drive to erase the EFI partition disk signature so you can use the USB drive in the future without confusing conflicts