Recovering boot of LUKS encrypted system after kernel update

uConsole
,
1

I have Debian Linux installed on my uConsole with root partition encryption using LUKS.

Encryption process was done by good tutorial by mclbn: Updated guide for Bookworm : encrypted root partition on uConsole

Yesterday I’ve updated my system by apt update && apt upgrade and after reboot got an error:

Cannot initialize device-mapper. Is dn_mod kernel module loaded?

It is that moment, when you realize that something go wrong, because you didn’t think ahead. New kernel was installed and initramfs wasn’t updated or was updated incorrectly, resulting in absence of module, needed to unlock encrypted root fs.

I’ve tried to chroot to this card from x86-64 Manjaro Linux with no luck. qemu-static + systemd-binfmt just refused to work, so I gave up.

Then I’ve made second uSD card with the same Debian for uConsole, booted from it and connected my main uSD using card reader. After some straightforward operations I’ve generated correct initramfs and now device continues to work.

For those to have a similar problem, I’ve provided complete instruction of this repair.

We are starting when uConsole already booted from fresh uSD card and uSD to repair is connected to the USB port using card reader. All operations needed root privileges so you can prefix every operation with sudo or make sudo su - at start.

At first we need to check, what device name our external card got:
lsblk

In my case it is /dev/sda. If in your case it will be different name, change some commands accordingly.

We need to ensure that all dependencies installed on the host OS:
apt install busybox cryptsetup initramfs-tools cryptsetup-initramfs

Then open encrypted fs:
cryptsetup -v luksOpen /dev/sda2 sdcard

And mount this fs with all additional to make correct environment for chroot:
mount /dev/mapper/sdcard /mnt
mount /dev/sda1 /mnt/boot/firmware
mount -o bind /dev /mnt/dev
mount -o bind /dev/pts /mnt/dev/pts
mount -t sysfs none /mnt/sys
mount -t proc none /mnt/proc

After this we can chroot into our environment:
chroot /mnt /bin/bash

Optionally we can update the system:
apt update && apt upgrade

Now we need to temporarily change device name in crypttab from mmcblk0p2 to sda2
nano /etc/crypttab

Or:
sed -i 's/mmcblk0p/sda/g' /etc/crypttab

Now the main part - update our initramfs:
update-initramfs -c -k $(uname -r)
mkinitramfs -o /boot/firmware/initramfs8 $(uname -r)

And revert changes to crypttab (manually or by sed):
sed -i 's/sda/mmcblk0p/g' /etc/crypttab

Now we can safely logout from chroot, unmount crypted uSD card, turn off the uConsole and install fixed card as main:
logout
sync
umount /mnt/{dev/pts,dev,sys,proc,boot/firmware} /mnt
cryptsetup -v luksClose sdcard
halt -p

2 Likes
2

Hi, thanks a lot for the detailed info! Did you identify what went wrong during the kernel update in the first place?

The script at /etc/kernel/postinst.d/initramfs-rebuild should take care of the operation you had to do manually. Maybe the script isn’t triggered after a kernel upgrade or it has not properly been made executable… That may imply there are missing steps in my tutorial :-S

Could you provide the output of ls -ld /etc/kernel/postinst.d/initramfs-rebuild on your LUKS-enabled sd-card?

I have not updated mine for a while, but now I am little bit afraid to do so :smiley:

3

It looks correct, but somehow failed. So for a while I will update initramfs manually after OS upgrade.

$ ls -ld /etc/kernel/postinst.d/initramfs-rebuild 
-rwxr-xr-x 1 root root 608 Jun 17 18:13 /etc/kernel/postinst.d/initramfs-rebuild

$ cat /etc/kernel/postinst.d/initramfs-rebuild 
#!/bin/sh -e

version="$1"
current_version="$(uname -r)"
boot_initramfs_path="/boot/firmware/initramfs8"

if [ "$1" = "$current_version" ]; then
    echo "Initramfs needs rebuilding."
else
    echo "Not current kernel version, initramfs needs no rebuilding."
    exit 0
fi

if [ -e "$boot_initramfs_path" ]; then
   echo "Backing up previous initramfs..."
   cp "$boot_initramfs_path" "$boot_initramfs_path".bak
   echo "Done."
fi

# Don't know if both paths are needed. Just in case...
echo "Rebuilding..."
update-initramfs -c -k "$version"
mkinitramfs -o "$boot_initramfs_path" "$version"
echo "All done."
1 Like
5

Thanks. I’ll keep an eye on it when I do my next apt upgrade…