Can't boot linux-lts with strip=true

[John-Gee]

Hello,

I wonder if this is somewhat related to #280 .
I usually use regular kernels not lts but to debug an issue I switched to linux-lts and couldn't boot. Out of curiosity I also tried linux-cachyos-lts and also had the same issue, whereas linux and linux-cachyos worked fine. All with same booster and systemd boot configs being the same of course.

The issue I face on the LTS kernels is:
"/dev/btrfs-control cannot be opened"

which would make it seems like a btrfs module wasn't loaded, even though I explicitly mention it. Also weird that it works with regular kernels but not lts, maybe it has to do with how booster look up the needed modules during regen and I regen the images from regular kernels?

Happy to provide whatever information is needed to fix this, as failing to boot kernels when trying to debug stuff is bad...

Thanks!

[craftfever]

I can't boot linux-lts with booster initramfs with strip: false either

[John-Gee]

I can't boot linux-lts with booster initramfs with strip: false either

False works fine for me hmmm.
Have you tried turning universal on?

[craftfever]

I can't boot linux-lts with booster initramfs with strip: false either

False works fine for me hmmm. Have you tried turning universal on?

I found the source of bug. If i generate image within booted linux-lts kernel, it's booting flawlessly. If I generate booster initramfs from linux stable (6.17/6.18) linux-lts refuse to boot with that particular initramfs, cause it was generated from different version of kernel (but this does not supposed to be happen, 'cause booster have to handle corresponding and correct modules versions for kernels regardless of what kernel is running, when I generating image). "universal: true" workarounds the problem, but this is eliminating purpose of booster, as initramfs image is bigger than mkinitcpio's and generating time is approx a minute instead of 16s for all installed kernels. I reverted to mkinitcpio.

[anatol]

If i generate image within booted linux-lts kernel, it's booting flawlessly. If I generate booster initramfs from linux stable (6.17/6.18) linux-lts refuse to boot with that particular initramfs, cause it was generated from different version of kernel (but this does not supposed to be happen, 'cause booster have to handle corresponding and correct modules versions for kernels regardless of what kernel is running, when I generating image).

I am not sure what are you trying to do here, but booster has never allowed to boot initramfs for incompatible kernel. If you try to mismatch kernel<->initramfs versions then the boot process exit with a big fat error message https://github.com/anatol/booster/blob/master/init/main.go#L846

In fact none of the initramfs generators allow you to do it. And it is by design - Linux modules interface does not support stable API.

[craftfever]

If i generate image within booted linux-lts kernel, it's booting flawlessly. If I generate booster initramfs from linux stable (6.17/6.18) linux-lts refuse to boot with that particular initramfs, cause it was generated from different version of kernel (but this does not supposed to be happen, 'cause booster have to handle corresponding and correct modules versions for kernels regardless of what kernel is running, when I generating image).

I am not sure what are you trying to do here, but booster has never allowed to boot initramfs for incompatible kernel. If you try to mismatch kernel<->initramfs versions then the boot process exit with a big fat error message https://github.com/anatol/booster/blob/master/init/main.go#L846

In fact none of the initramfs generators allow you to do it. And it is by design - Linux modules interface does not support stable API.

But I rechecked multiple times, that I set correct initramfs image, corresponding to kernel version. I say that booster generates different images, depending on what kernel it doing it, if "universal: true" is not set (Or "universal: false" set). If I generate image on linux -lts for this version, LTS boot successfully, if I generate LTS-initramfs on linux stable (6.17/6.18), linux-lrs won't boot, despite correct iniramfs version setting. I don't know, why this happening. There is no error messages, indicating incorrect version, it's just bunch of udev messages and stuck.

[anatol]

universal: true does not affect what version is selected for initramfs generation. universal: true config option says to include a lot of kernel modules with a hope it will be needed during the boot process.

The selected kernel version happens in the distributive's install script. Here is the example of one for Arch Linux https://github.com/anatol/booster/blob/master/packaging/arch/regenerate_images#L13 pay attention to --kernel-version flag.

If the problem you see is with cachyos then check their package install logic and find out what --kernel-version flag they use.

[John-Gee]

I can't boot linux-lts with booster initramfs with strip: false either

False works fine for me hmmm. Have you tried turning universal on?

I found the source of bug. If i generate image within booted linux-lts kernel, it's booting flawlessly. If I generate booster initramfs from linux stable (6.17/6.18) linux-lts refuse to boot with that particular initramfs, cause it was generated from different version of kernel (but this does not supposed to be happen, 'cause booster have to handle corresponding and correct modules versions for kernels regardless of what kernel is running, when I generating image). "universal: true" workarounds the problem, but this is eliminating purpose of booster, as initramfs image is bigger than mkinitcpio's and generating time is approx a minute instead of 16s for all installed kernels. I reverted to mkinitcpio.

Yeah I was wondering about that. But never got to trying it, thank you for trying!

@anatol you may have misunderstood him, I believe he says generating the initramfs for lts while running the OS with regular gives him a problematic one whereas generating for lts while running it works fine, which makes some sense since the issue seems related to modules not correctly added to the initramfs.

[anatol]

@John-Gee thank you for the clarification.

To help to answer the problem I need two log files for the error case scenario:

1. booster build log, run booster as before but add --verbose flag
2. booster failed boot log - add booster.log=debug,console kernel parameter. The boot will likely get into busybox shell where you can get the kernel logs using dmesg. The most interesting part will be closer to the end.

see https://man.archlinux.org/man/extra/booster/booster.1.en for more help with these config options.

Please attach logs as files.

[craftfever]

@John-Gee thank you for the clarification.

To help to answer the problem I need two log files for the error case scenario:

1. booster build log, run booster as before but add --verbose flag
2. booster failed boot log - add booster.log=debug,console kernel parameter. The boot will likely get into busybox shell where you can get the kernel logs using dmesg. The most interesting part will be closer to the end.

see https://man.archlinux.org/man/extra/booster/booster.1.en for more help with these config options.

Please attach logs as files.

Ok, I found the case, why linux-lts does not boot, when generating initramfs from recent linux version. The reason in different modules structure, in this case, in crc32c module (without it, it can't mount root partition, and hence, boot successfully) In lunux-lts 6.12 it's loadable separate module, in recent linux kernel it's built-in kernel monolite part. Solution is to add "crc32c" in the module list, but there is another problem, for recent kernel versions, it says, that module does not find and refuse to build initramfs for it. So, it necessary to create different configs for linux-lts and new linux versions. Is there way to ignore "invalid" module names for certain linux versions for having only one unified config?

[John-Gee]

I am not sure what changed, but I got back to testing this yesterday and it seemed fine now.
Maybe it is as @craftfever said, and there's no such difference between latest lts and current kernels? If so, and if nothing else changed in the code base we may still to look into this?

[craftfever]

I am not sure what changed, but I got back to testing this yesterday and it seemed fine now. Maybe it is as @craftfever said, and there's no such difference between latest lts and current kernels? If so, and if nothing else changed in the code base we may still to look into this?

Cause LTS got 6.18 update, that eliminated differences. As I said, the difference was in crc32c driver,that was as module in 6.12 and built-in in 6.18+, hence Booster needed for support for different configs for different kernels, that's what I wanted to say to @anatol

[John-Gee]

I'm attaching the booster log:
regen.log

but I didn't get to a shell on boot so I couldn't really do anything and the log isn't in systemctl -b unless I am missing it (-1 is not for the lts boot but the one before that).

Is there anything I can do? @anatol

[anatol]

The generator log you provided looks fine.

So the problem is somewhere else. It could be that the module file is corrupted somehow.

The best next step is to enable boot debug log and the emergency shell. The boot your system until the failure, login to the shell and inspect the logs. It will provide further information about what is going on there.

[John-Gee]

As I wrote I added the debug log parameters but I just saw a lot of verbosity without any shell I could use... Maybe I need different options while generating the image?

I think @craftfever 's explanation is more likely that the module getting corrupted only for lts kernels.

[anatol]

As I wrote I added the debug log parameters but I just saw a lot of verbosity without any shell I could use

Hi, I was talking about boot time debug log and emergency shell. See https://github.com/anatol/booster/blob/master/docs/manpage.md#debugging

And if you add busybox to the extra_files you'll get the emergency shell in case of the boot failure. The shell is a really useful tool to inspect the broken system.