[Solved] Unable to change CM4 kernel in config.txt

DevTerm
1

I’m trying to install multiple kernels and switch between them by changing config.txt on my CM4 DevTerm. However, I can’t get it to boot if I change the kernel to anything besides the default. Here’s what I’ve tried:

  1. Add the line kernel=kernel8.img to config.txt. This works fine.
  2. Copy kernel8.img to kernel_copy.img. Change config.txt to kernel=kernel_copy.img. This fails to boot.
  3. Try scenario 2 again, but this time also copy initramfs8 to initramfs_copy.img, just in case it autodetects the name like that. No change.
  4. I’ve also tried compiling my own kernel and installing it to kernel_mine.img, as suggested in the Raspberry Pi kernel docs. It fails in the same way as 2, of course.

I have UART logging enabled for the bootloader and kernel. Here’s what the logs look like for scenario 1:

RPi: BOOTLOADER release VERSION:4fd8f1f3 DATE: 2023/05/11 TIME: 07:26:03
BOOTMODE: 0x06 partition 0 build-ts BUILD_TIMESTAMP=1683786363 serial 6cc3ffaa boardrev c03140 stc 478856
PM_RSTS: 0x00001000
part 00000000 reset_info 00000000
uSD voltage 3.3V
Initialising SDRAM 'Samsung' 16Gb x2 total-size: 32 Gbit 3200
DDR 3200 1 0 32 152

Boot mode: SD (01) order f2564
SD HOST: 200000000 CTL0: 0x00800000 BUS: 400000 Hz actual: 390625 HZ div: 512 (256) status: 0x1fff0000 delay: 276
SD HOST: 200000000 CTL0: 0x00800f00 BUS: 400000 Hz actual: 390625 HZ div: 512 (256) status: 0x1fff0000 delay: 276
OCR c0ff8000 [23]
CID: 006f0303434241445310aa0000830173
CSD: 400e00325b590000ea6d7f800a404000
SD: bus-width: 4 spec: 2 SCR: 0x02b58003 0x00000000
SD HOST: 200000000 CTL0: 0x00800f04 BUS: 50000000 Hz actual: 50000000 HZ div: 4 (2) status: 0x1fff0000 delay: 2
MBR: 0x00002000, 1048576 type: 0x0c
MBR: 0x00102000,60397568 type: 0x83
MBR: 0x00000000,       0 type: 0x00
MBR: 0x00000000,       0 type: 0x00
Trying partition: 0
type: 32 lba: 8192 oem: 'mkfs.fat' volume: ' bootfs     '
rsc 32 fat-sectors 2040 c-count 261116 c-size 4
root dir cluster 2 sectors 0 entries 0
FAT32 clusters 261116
Trying partition: 0
type: 32 lba: 8192 oem: 'mkfs.fat' volume: ' bootfs     '
rsc 32 fat-sectors 2040 c-count 261116 c-size 4
root dir cluster 2 sectors 0 entries 0
FAT32 clusters 261116
Read config.txt bytes      337 hnd 0x292e
gpio_cmd: '10=ip,np' pins: 10-10 drive: -1 fsel: 0 term: 0
Read start4.elf bytes  2254208 hnd 0x4ca5
Read fixup4.dat bytes     5403 hnd 0x129
0x00c03140 0x00000000 0x00001fff
MEM GPU: 76 ARM: 948 TOTAL: 1024
Firmware: 03dc77429335caee083e22ddc8eec09c07f12a7a Aug 10 2023 15:33:38
Starting start4.elf @ 0xfec00200 partition 0
+
[    0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd083]
[    0.000000] Linux version 5.10.17-v8+ (cpi@cpi-X79) (aarch64-linux-gnu-gcc (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0, 3
[    0.000000] Machine model: Raspberry Pi Compute Module 4 Rev 1.0
[    0.000000] efi: UEFI not found.
[    0.000000] Reserved memory: created CMA memory pool at 0x0000000016c00000, size 384 MiB
[    0.000000] OF: reserved mem: initialized node linux,cma, compatible id shared-dma-pool
[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x0000000000000000-0x000000003fffffff]
[    0.000000]   DMA32    [mem 0x0000000040000000-0x00000000fbffffff]
[    0.000000]   Normal   empty

[...and this continues on and shows the whole boot sequence up to the login screen...]

And here is scenario 2. It seems like the bootloader passes off to the kernel, which never prints its first log.

RPi: BOOTLOADER release VERSION:4fd8f1f3 DATE: 2023/05/11 TIME: 07:26:03
BOOTMODE: 0x06 partition 0 build-ts BUILD_TIMESTAMP=1683786363 serial 6cc3ffaa boardrev c03140 stc 478856
PM_RSTS: 0x00001000
part 00000000 reset_info 00000000
uSD voltage 3.3V
Initialising SDRAM 'Samsung' 16Gb x2 total-size: 32 Gbit 3200
DDR 3200 1 0 32 152                                                        
                                                                           
Boot mode: SD (01) order f2564                                             
SD HOST: 200000000 CTL0: 0x00800000 BUS: 400000 Hz actual: 390625 HZ div: 512 (256) status: 0x1fff0000 delay: 276
SD HOST: 200000000 CTL0: 0x00800f00 BUS: 400000 Hz actual: 390625 HZ div: 512 (256) status: 0x1fff0000 delay: 276
OCR c0ff8000 [18]
CID: 006f0303434241445310aa0000830173
CSD: 400e00325b590000ea6d7f800a404000
SD: bus-width: 4 spec: 2 SCR: 0x02b58003 0x00000000
SD HOST: 200000000 CTL0: 0x00800f04 BUS: 50000000 Hz actual: 50000000 HZ div: 4 (2) status: 0x1fff0000 delay: 2
MBR: 0x00002000, 1048576 type: 0x0c
MBR: 0x00102000,60397568 type: 0x83
MBR: 0x00000000,       0 type: 0x00
MBR: 0x00000000,       0 type: 0x00
Trying partition: 0
type: 32 lba: 8192 oem: 'mkfs.fat' volume: ' bootfs     '
rsc 32 fat-sectors 2040 c-count 261116 c-size 4
root dir cluster 2 sectors 0 entries 0
FAT32 clusters 261116
Trying partition: 0
type: 32 lba: 8192 oem: 'mkfs.fat' volume: ' bootfs     '
rsc 32 fat-sectors 2040 c-count 261116 c-size 4
root dir cluster 2 sectors 0 entries 0
FAT32 clusters 261116
Read config.txt bytes      317 hnd 0x9
gpio_cmd: '10=ip,np' pins: 10-10 drive: -1 fsel: 0 term: 0
Read start4.elf bytes  2254208 hnd 0x4ca5
Read fixup4.dat bytes     5403 hnd 0x129
0x00c03140 0x00000000 0x00001fff
MEM GPU: 76 ARM: 948 TOTAL: 1024
Firmware: 03dc77429335caee083e22ddc8eec09c07f12a7a Aug 10 2023 15:33:38
Starting start4.elf @ 0xfec00200 partition 0
+

Diff of the two logs (up to the point where the kernel boots):

@@ -9,7 +9,7 @@
 Boot mode: SD (01) order f2564
 SD HOST: 200000000 CTL0: 0x00800000 BUS: 400000 Hz actual: 390625 HZ div: 512 (256) status: 0x1fff0000 delay: 276
 SD HOST: 200000000 CTL0: 0x00800f00 BUS: 400000 Hz actual: 390625 HZ div: 512 (256) status: 0x1fff0000 delay: 276
-OCR c0ff8000 [23]
+OCR c0ff8000 [18]
 CID: 006f0303434241445310aa0000830173
 CSD: 400e00325b590000ea6d7f800a404000
 SD: bus-width: 4 spec: 2 SCR: 0x02b58003 0x00000000
@@ -28,7 +28,7 @@
 rsc 32 fat-sectors 2040 c-count 261116 c-size 4
 root dir cluster 2 sectors 0 entries 0
 FAT32 clusters 261116
-Read config.txt bytes      337 hnd 0x292e
+Read config.txt bytes      317 hnd 0x9
 gpio_cmd: '10=ip,np' pins: 10-10 drive: -1 fsel: 0 term: 0
 Read start4.elf bytes  2254208 hnd 0x4ca5
 Read fixup4.dat bytes     5403 hnd 0x129

Any ideas on what could be going wrong? Maybe something needs to be configured with the initramfs images as well?

2

Ah, I just learned about an option in config.txt: uart_2ndstage=1. This shows some extra logs via UART that adds some details on what happens between the bootloader starting and the kernel starting. This has some things that I was expecting to see, like which kernel image file is being loaded. Here’s what my 2nd stage bootloader logs look like when booting a kernel copied from kernel8.img to kernel_copy.img:

MESS:00:00:02.805003:0: arasan: arasan_emmc_open
MESS:00:00:02.806664:0: arasan: arasan_emmc_set_clock C0: 0x00800000 C1: 0x000e0047 emmc: 200000000 actual: 390625 di5
MESS:00:00:02.926893:0: arasan: arasan_emmc_set_clock C0: 0x00800000 C1: 0x000e0047 emmc: 200000000 actual: 390625 di5
MESS:00:00:02.939763:0: arasan: arasan_emmc_set_clock C0: 0x00800f00 C1: 0x000e0047 emmc: 200000000 actual: 390625 di5
MESS:00:00:02.976737:0: arasan: arasan_emmc_set_clock C0: 0x00800f06 C1: 0x000e0207 emmc: 200000000 actual: 50000000 1
MESS:00:00:03.128051:0: brfs: File read: /mfs/sd/config.txt
MESS:00:00:03.130964:0: brfs: File read: 353 bytes
MESS:00:00:03.156246:0: HDMI0:EDID error reading EDID block 0 attempt 0
MESS:00:00:03.160750:0: HDMI0:EDID giving up on reading EDID block 0
MESS:00:00:03.177659:0: HDMI1:EDID error reading EDID block 0 attempt 0
MESS:00:00:03.182166:0: HDMI1:EDID giving up on reading EDID block 0
MESS:00:00:03.188015:0: brfs: File read: /mfs/sd/config.txt
MESS:00:00:03.192948:0: gpioman: gpioman_get_pin_num: pin LEDS_PWR_OK not defined
MESS:00:00:03.694670:0: gpioman: gpioman_get_pin_num: pin LEDS_PWR_OK not defined
MESS:00:00:03.700309:0: *** Restart logging
MESS:00:00:03.702951:0: brfs: File read: 353 bytes
MESS:00:00:03.712925:0: hdmi: HDMI0:EDID error reading EDID block 0 attempt 0
MESS:00:00:03.717951:0: hdmi: HDMI0:EDID giving up on reading EDID block 0
MESS:00:00:03.728573:0: hdmi: HDMI0:EDID error reading EDID block 0 attempt 0
MESS:00:00:03.733602:0: hdmi: HDMI0:EDID giving up on reading EDID block 0
MESS:00:00:03.739200:0: hdmi: HDMI:hdmi_get_state is deprecated, use hdmi_get_display_state instead
MESS:00:00:03.752994:0: hdmi: HDMI1:EDID error reading EDID block 0 attempt 0
MESS:00:00:03.758024:0: hdmi: HDMI1:EDID giving up on reading EDID block 0
MESS:00:00:03.768645:0: hdmi: HDMI1:EDID error reading EDID block 0 attempt 0
MESS:00:00:03.773670:0: hdmi: HDMI1:EDID giving up on reading EDID block 0
MESS:00:00:03.779267:0: hdmi: HDMI:hdmi_get_state is deprecated, use hdmi_get_display_state instead
MESS:00:00:03.788031:0: HDMI0: hdmi_pixel_encoding: 300000000
MESS:00:00:03.793505:0: HDMI1: hdmi_pixel_encoding: 300000000
MESS:00:00:03.799725:0: gpioman: gpioman_get_pin_num: pin CAMERA_0_I2C_PORT not defined
MESS:00:00:03.810955:0: dtb_file 'bcm2711-rpi-cm4.dtb'
MESS:00:00:03.820120:0: brfs: File read: /mfs/sd/bcm2711-rpi-cm4.dtb
MESS:00:00:03.823368:0: Loaded 'bcm2711-rpi-cm4.dtb' to 0x100 size 0xd84f
MESS:00:00:03.843303:0: brfs: File read: 55375 bytes
MESS:00:00:03.856498:0: brfs: File read: /mfs/sd/overlays/overlay_map.dtb
MESS:00:00:03.885857:0: brfs: File read: 4839 bytes
MESS:00:00:03.888249:0: brfs: File read: /mfs/sd/config.txt
MESS:00:00:03.892937:0: dtparam: audio=on
MESS:00:00:03.900866:0: brfs: File read: 353 bytes
MESS:00:00:03.905574:0: brfs: File read: /mfs/sd/overlays/dwc2.dtbo
MESS:00:00:03.914814:0: Loaded overlay 'dwc2'
MESS:00:00:03.916057:0: dtparam: dr_mode=host
MESS:00:00:03.931408:0: brfs: File read: 801 bytes
MESS:00:00:03.945031:0: brfs: File read: /mfs/sd/overlays/vc4-kms-v3d-pi4.dtbo
MESS:00:00:04.011080:0: Loaded overlay 'vc4-kms-v3d-pi4'
MESS:00:00:04.013279:0: dtparam: cma-384=true
MESS:00:00:04.168239:0: brfs: File read: 3913 bytes
MESS:00:00:04.174360:0: brfs: File read: /mfs/sd/overlays/devterm-pmu.dtbo
MESS:00:00:04.188306:0: Loaded overlay 'devterm-pmu'
MESS:00:00:04.249053:0: brfs: File read: 2875 bytes
MESS:00:00:04.251835:0: brfs: File read: /mfs/sd/overlays/devterm-panel.dtbo
MESS:00:00:04.264990:0: Loaded overlay 'devterm-panel'
MESS:00:00:04.296461:0: brfs: File read: 1469 bytes
MESS:00:00:04.299261:0: brfs: File read: /mfs/sd/overlays/devterm-misc.dtbo
MESS:00:00:04.318253:0: Loaded overlay 'devterm-misc'
MESS:00:00:04.368665:0: brfs: File read: 2129 bytes
MESS:00:00:04.371452:0: brfs: File read: /mfs/sd/overlays/audremap.dtbo
MESS:00:00:04.383328:0: Loaded overlay 'audremap'
MESS:00:00:04.384919:0: dtparam: pins_12_13=true
MESS:00:00:04.389529:0: dtparam: spi=on
MESS:00:00:04.418889:0: brfs: File read: 1155 bytes
MESS:00:00:04.421834:0: brfs: File read: /mfs/sd/cmdline.txt
MESS:00:00:04.426070:0: Read command line from file 'cmdline.txt':
MESS:00:00:04.431941:0: 'console=serial0,115200 console=tty1 root=PARTUUID=7788c428-02 rootfstype=ext4 fsck.repair=ye'
MESS:00:00:04.559176:0: brfs: File read: 148 bytes
MESS:00:00:06.368074:0: brfs: File read: /mfs/sd/kernel_copy.img
MESS:00:00:06.371589:0: Loaded 'kernel_copy.img' to 0x8000 size 0x1460200
MESS:00:00:06.378723:0: Device tree loaded to 0x2eff1b00 (size 0xe449)
MESS:00:00:06.387038:0: uart: Set PL011 baud rate to 103448.300000 Hz
MESS:00:00:06.394030:0: uart: Baud rate change done...
MESS:00:00:06.396048:0:

Here’s the diff compared to loading kernel8.img; this could indicate a problem!

@@ -62,10 +62,9 @@
 Read command line from file 'cmdline.txt':
 'console=serial0,115200 console=tty1 root=PARTUUID=7788c428-02 rootfstype=ext4 fsck.repair=ye'
 brfs: File read: 148 bytes
-brfs: File read: /mfs/sd/kernel8.img
-Loaded 'kernel8.img' to 0x80000 size 0x1460200
-Kernel relocated to 0x200000
+brfs: File read: /mfs/sd/kernel_copy.img
+Loaded 'kernel_copy.img' to 0x8000 size 0x1460200
 Device tree loaded to 0x2eff1b00 (size 0xe449)
 uart: Set PL011 baud rate to 103448.300000 Hz
 uart: Baud rate change done...
1 Like
3

Kernel relocation is normal. (I haven’t said anything in the thread because I don’t know the answer; apologies.)

1 Like
4

I scoured the docs for information about how the bootloader chooses the kernel address and found this:

kernel_address is the memory address to which the kernel image should be loaded. 32-bit kernels are loaded to address 0x8000 by default, and 64-bit kernels to address 0x200000. If kernel_old is set, kernels are loaded to the address 0x0.

Since my copied kernel was starting at 0x8000, it seems like it wasn’t being detected as a 64-bit kernel. I added arm_64bit=1 to my config.txt. That worked!

1 Like