ISO Boot Failure on Supermicro Servers in UEFI Mode

[zohourih]

Problem description

ISOs generated by kiwi-ng with firmware="efi" are not detected as bootable in UEFI mode when mounted over IPMI on Supermicro servers, and it is impossible to boot from them. The same ISO boots fine in Legacy mode. The same ISO boots fine on HPE and DELL servers, but Supermicro informed us that is due to an extra emulation layer they have implemented for the virtual CDROM. Standard Linux ISOs from Ubuntu/SLES/etc. boot fine on Supermicro servers in both UEFI and Legacy mode. The issue is reproducible on all generations of Supermicro servers.

Per Supermicro, the structure of the ISOs generated by kiwi-ng is missing certain components that are required for correct boot in UEFI mode. A similar problem existed in Mint 21.2 ISO which was corrected in Mint 21.3 ISO. The following information is provided to us by Supermicro:

1. Mint 21.2 – Cannot boot up in UEFI mode through IPMI Virtual CDROM on X12DDW-A6.
2. Mint 21.3 – Boot up fine in UEFI mode through IPMI Virtual CDROM.

Mint 21.2 UEFI in X12DDW-A6: No FS device.

Compare the two ISOs:
Mint 21.2 doesn’t have the efi.img file, that’s the file for EFI boot information.
Mint fixed the issue in ISO 21.3

Per Supermicro, the following steps are required to correct images generated by kiwi-ng for correct UEFI boot (copied verbatim from Supermicro's response):

Original ISO:

Modified ISO:

Modification steps:

1. Create an efiboot.img for EFI boot
2. added isolinux.bin and libraries for Legacy boot.

Generate new efiboot.img (I changed from 2880 to 5760 so bootx64.efi can fit in):

dd if=/dev/zero of=EFI/BOOT/efiboot.img bs=512 count=5760

Format the boot image with F 12:

mkfs.msdos -F 12 -n 'KLA-BOOT-ISO' EFI/BOOT/efiboot.img

Create Directory structure for the efiboot image:

mmd -i EFI/BOOT/efiboot.img ::EFI
mmd -i EFI/BOOT/efiboot.img ::EFI/BOOT
mcopy -i EFI/BOOT/efiboot.img EFI/BOOT/bootx64.efi ::EFI/BOOT/bootx64.efi

Generate new ISO using xorriso to include EFI platform support and EFI boot image:

xorriso -as mkisofs -no-emul-boot -boot-load-size 4 -boot-info-table -iso-level 4 -b isolinux/isolinux.bin -eltorito-alt-boot -eltorito-platform 0xEF -volid "KLA-BOOT-ISO" -eltorito-boot EFI/BOOT/efiboot.img -no-emul-boot -o /tmp/klaboot.iso isoinfo .

isohybrid -u /tmp/klaboot.iso:

Is there some configuration we can change in config.xml or kiwi config.yml that would allow us to build ISOs that follow the structure mentioned above?

Expected behaviour

ISOs created using kiwi-ng with firmware="efi" should be detected and bootable in UEFI mode when mounted on Supermicro servers over IPMI.

Steps to reproduce the behaviour

1. Download the config file below and replace *server* with target repository.
   config.zip
2. Use the following kiwi config file and build the ISO:

xz:
  - options: '--threads=8'
container:
  - compress: xz
mapper:
  - part_mapper: kpartx

3. Procure any Supermicro X11 or newer server that supports UEFI boot, and set boot mode to UEFI in the BIOS.
4. Mount the ISO over IPMI and try to boot from it.
5. The UEFI virtual CDROM will never show up among boot options as the ISO is not detected as bootable in UEFI mode.

OS and Software information

• KIWI version: 9.24.55
• Operating system host version: SLES 15 SP3
• Operating system target version: SLES 15 SP3
• Open Build Service version (N/A if not using OBS): N/A
• Koji version (N/A if not using Koji): N/A

[schaefi]

Thanks much for the very detailed and professional report 👍

As a side note, we dropped support for isolinux (see #2439 for details). Thus legacy boot is handled through grub for
quite some time. As we are talking about EFI boot this information is JFI and I only mentioned it because your fix procedure
added isolinux.bin and friends for the legacy boot. The isolinux project is EOL since some time.

but Supermicro informed us that is due to an extra emulation layer

Yeah every system that has EFI boot capabilities and allows to boot from iso9660 requires a FAT based loader structure as part of the ISO layer. We do that in kiwi too which makes me wonder why it did not boot on Supermicro. I looked at one of our integration tests to see the differences. For example: https://build.opensuse.org/package/live_build_log/Virtualization:Appliances:Images:Testing_x86:tumbleweed/test-image-live:Secure/images/x86_64

In there if you search for mcopy you find the instructions:

[   71s] [ DEBUG   ]: 19:18:15 | EXEC: [qemu-img create /usr/src/packages/KIWI-iso/efi-loader.2w3kejdz 20M]
[   71s] [ DEBUG   ]: 19:18:15 | EXEC: [mkdosfs -n BOOT /usr/src/packages/KIWI-iso/efi-loader.2w3kejdz]
[   71s] [ DEBUG   ]: 19:18:15 | EXEC: [mcopy -Do -s -i /usr/src/packages/KIWI-iso/efi-loader.2w3kejdz /usr/src/packages/KIWI-iso/live-media.caxzpjnr/EFI ::]

The FAT image efi-loader.2w3kejdz is a temporary file used in the later xorriso call and not put as a file on the ISO. In your case you created this as efiboot.img and placed it also as a file to the ISO filesystem.

Further down the log you can search for xorriso and you will find this

[  157s] [ DEBUG   ]: 19:19:40 | EXEC: [/usr/bin/xorriso -application_id 0x6be39c0e -publisher SUSE LINUX GmbH -preparer_id KIWI - https://github.com/OSInside/kiwi -volid CDROM -joliet on -padding 0 -outdev /usr/src/packages/KIWI-iso/kiwi-test-image-live.x86_64-1.42.3.iso -map /usr/src/packages/KIWI-iso/live-media.caxzpjnr / -chmod 0755 / -- -boot_image grub bin_path=boot/x86_64/loader/eltorito.img -boot_image grub grub2_mbr=/usr/src/packages/KIWI-iso/live-media.caxzpjnr/boot/x86_64/loader/boot_hybrid.img -boot_image grub grub2_boot_info=on -boot_image any partition_offset=16 -boot_image any cat_path=boot/x86_64/boot.catalog -boot_image any cat_hidden=on -boot_image any boot_info_table=on -boot_image any platform_id=0x00 -boot_image any emul_type=no_emulation -boot_image any load_size=2048 -append_partition 2 0xef /usr/src/packages/KIWI-iso/efi-loader.2w3kejdz -boot_image any next -boot_image any efi_path=--interval:appended_partition_2:all:: -boot_image any platform_id=0xef -boot_image any emul_type=no_emulation]

As you can see the FAT image is used as an any loader of the form

-boot_image any load_size=2048 -append_partition 2 0xef /usr/src/packages/KIWI-iso/efi-loader.2w3kejdz

Which I believe is very similar to your -eltorito-boot instructions

Ok, so from that perspective I have several questions:

1. I see your FAT image is created as a FAT12 filesystem. Our procedure creates a FAT16 (default) filesystem. Is that an issue for Supermicro ?
2. Can you please compare your xorriso build instructions with the provided kiwi log and share your thoughts if you agree with me that kiwi basically does the same thing ?
3. You placed your FAT image file as EFI/BOOT/efiboot.img into the ISO filesystem. Is this a requirement for Supermicro ? From my perspective it's not needed to place the FAT image as a file to the ISO, kiwi does not do this.

Would be great if you can respond to my findings. Currently I only think that the FAT format could be an issue which things are not booting on Supermicro. I don't have this hardware at hand so I'm afraid I can't test it.

Thanks

[zohourih]

@schaefi Thank you for looking into this. I am afraid apart from the problem statement part, all the technical information I have put in the issue were provided by the Supermicro engineer who worked on the case, and, unfortunately, I don't have the answer to your questions. I will reach out to the Supermicro engineer and see if they are willing to participate in this discussion.

The original feedback I got from Supermicro was that unless the isohybrid test succeeds on the ISO, it won't boot using Supermicro's virtual CDROM over UEFI. The test certainly fails on the original ISO, but succeeds on the modified ISO. I am not quite sure what the success criteria is in that case, and it might be possible to pass that criteria in ways other than the one suggested by Supermicro. I think it could also be valuable to study the Linux Mint 21.2 to 21.3 ISO change, as well, and determine why that change was implemented, as this is a very recent case.

I have access to several Supermicro servers and if there is anything in particular you want to be tested, I can make the change and rebuild the ISO and test and report the results.

[zohourih]

@schaefi I got more information from Supermicro about this case. The person who provided the instructions to correct the ISO said they followed the steps detailed here and studied the format of Rocky Linux ISO and replicated the same format. Here is a verbatim quote regarding their requirement:

To boot ISO in EFI using ATEN virtual CD, then the ISO should include an efiboot.img (it can be named for any name) that contains the EFI boot directory “/EFI/BOOT” and boot images “bootx64.efi and grubx64.efi” which you can find in all standard Linux ISOs.

Is there any standard Linux ISO (Ubuntu, SLES, Rocky, etc.) that follows the same EFI boot format as the ISOs generated by kiwi? If there is such standard ISO, it should fails to boot using Supermicro's virtual CDROM in UEFI mode, and we might then be able to compel them to resolve the problem on their side; otherwise, they are not going to allocate resources to resolve a problem that doesn't occur with standard Linux ISOs. It is worth noting that we prefer if we can find a solution at kiwi level since, even if Supermicro fixes the problem on their side, we will need to update the BIOS/BMC FW of thousands of servers scattered across several countries to be able to get the fix, which is not practical.

[schaefi]

There is no standard in designing an ISO layout, it's just a filesystem with data. However, there is a clear EFI standard and that has to be followed in order to boot via EFI. Of course kiwi follows this as all others do too. For booting an ISO via EFI an embedded FAT image is required because the EFI standard requires a FAT signature plus EFI binaries below EFI/BOOT. kiwi also does that as shown in the integration test log. At this point, even though there is a standard, many hardware vendors already deviate and add stuff to change the behavior.

I'm happy to help and fix what is needed to boot on supermicro. I also believe that it cannot be a big issue because technically kiwi does what is required to boot an ISO from EFI. I'm afraid but the comparison to others does not help here because the design for an ISO layout from a content perspective varies between the distributions and is also not an issue as long as the technical requirements to boot are met.

Thus it's required to understand the real cause of the boot issue. In order to find that we need to clarify on the open questions that I still have:

• The embedded fat image on supermicro is FAT12 kiwi creates FAT16 (the EFI standard btw). So why is fat12 used for supermicro and is it a requirement or not ?

• I provided our xorriso command and I don't see any regressions compared to supermicro. A short look from a supermicro engineer would be enough to identify any ISO layer issues. I don't see any

• the FAT image file is placed as file EFI/BOOT/efiboot.img into the ISO filesystem. This is imho not needed because the file is passed to xorriso and is embedded into the ISO header at a certain offset. The binary blob at this offset is the only place where this data becomes effective during boot. Thus it's not needed as a plain file on the ISO filesystem. Is it required to have the FAT image file in the ISO filesystem ? and why ?

We can only continue with a test and patch when we have clarity on the above questions and I believe this can only be done together with a supermicro engineer contributing to this thread. That's because I have a number of devices here on my table but no supermicro system ;)

Thanks

[zohourih]

@schaefi I did actually ask the person who worked on the case to participate in this thread, the response I got was that he is not a "build engineer" and only followed the steps I mentioned earlier to "fix" the ISO we provided them with (albeit, there is no proof the ISO was broken to begin with). He is not sure what the boot requirements of Supermicro's UEFI is, either. It is quite difficult to get Supermicro to co-operate on matters that involve custom solutions, and as I said, unless the issue also happens with ISOs from a standard distribution, they are not going to engage their BIOS/UEFI engineers.

Nonetheless, I will pass your questions to Supermicro and see if they can provide an answer. I can predict with high confidence that the problem is exactly what you raised in your last question: Supermicro's bootloader probably cannot read the EFI image from the ISO header, and only looks for it inside the ISO filesystem.

[zohourih]

Verbatim copy of Supermicro's response:

1) As I told you before, I just follow the link from this github to modify the ISO image who happened to use FAT 12  (Fat12 is for floppy image).  I compared your ISO versus the standard Rocky Linux and Mint Linux before I did the modification.  
https://github.com/syzdek/efibootiso

2) Is FAT16 work?  I don’t know because I couldn’t find any information regarding ISO and FAT16.
https://www.file-recovery.com/recovery-understanding-file-system-fat.htm

3) I believe you can put the efiboot.img in any directory you want as long as you specified in the directory structure where to read the bootx64.efi in the ISO
# mmd -i EFI/BOOT/efiboot.img ::EFI
# mmd -i EFI/BOOT/efiboot.img ::EFI/BOOT
# mcopy -i EFI/BOOT/efiboot.img EFI/BOOT/bootx64.efi ::EFI/BOOT/bootx64.efi

4) Xorriso ISO image creator
https://unix.stackexchange.com/questions/283846/why-can-my-custom-build-iso-not-boot-in-efi
https://forums.centos.org/viewtopic.php?t=78987

FYI, we have been using the same BMC ATEN virtual media for ISO mount in all supermicro products for years (X9, X10,X11,X12,X13 ..) This is the first time I learned about the EFI ISO boot issue.  Mint Linux fixed their 21.2 ISO boot EFI issue with the release of 21.3 ISO if you want to compare between the two.

@schaefi Do you know of any standard distribution that embeds the EFI image in the ISO header, rather than including it as part of the ISO file system? Looks like Linux Mint was one, but they went back to putting the image in the ISO filesystem in 21.3 and the changelog of the 21.3 release does not actually mention why they did that. I am pretty sure this was not to restore compatibility with Supermicro's virtual CDROM, though (😆), and the 21.2 structure probably worked everywhere else as is. Unfortunately, Supermicro is going to refer to this case until the end of time, as "proof" that this is the only correct structure, even though we know this is not the case, and unfortunately, there is no official standard or spec to prove them wrong.

[zohourih]

@schaefi I checked several standard Linux ISOs to see if I can find any that doesn't work on Supermicro's servers. The Linux Mint 21.2 ISO that doesn't boot on Supermicro's servers in UEFI mode does actually have an efi.img file, but it is under /boot/grub/. In 21.3 there are two different efi.img files in the ISO: one under /boot/grub/, and one directly in the root of the ISO. I am not sure why they did that. Other difference between the two ISOs is that the 21.2 ISO fails the isohybrid test, but 21.3 passes.

On the other hand, the Ubuntu 22.04 ISOs neither pass the isohybrid test, nor have any efi.img file anywhere inside the ISO, yet they boot fine on Supermicro's servers. I guess this means the problem is not actually related to whether efi.img is in the ISO header or inside the ISO, as there are working ISOs with both structures.

The remaining point is regarding FAT 12 vs. FAT 16. Is there any way I can confirm if the EFI partition is FAT12 or FAT16 in an existing ISO such as Ubuntu's ISO?

[Conan-Kudo]

Did you try using firmware="uefi" instead? That tells kiwi to pull in all the normal resources from the distribution (e.g. shim and friends) and set up a more "conventional" UEFI ESP that you'd see from distro images.

Additionally, firmware="efi" means that if your system has a verified boot process using the Secure Boot feature, it will fail. firmware="uefi" is required for compatibility with Secure Boot.

[zohourih]

@Conan-Kudo Yes, we have tried "uefi", as well. Results are the same. We are not using secure boot in our environment.

[Conan-Kudo]

We probably just fixed this with the merge of #2695. Could you try setting efiparttable="gpt" and building with kiwi 10.2.4?

[zohourih]

@Conan-Kudo Thank you for the update. I updated kiwi to 10.2.5 and tried adding efiparttable="gpt" to config.xml and rebuilding my minimal test ISO. I did confirm that after adding that option, fdisk -l does report the ISO as gpt instead of msdos. However, unfortunately, this did not change anything with respect to booting in UEFI mode on Supermicro's Virtual CDROM, and the ISO is still not detected as bootable in UEFI mode, and does not show up among the boot options (while it does show up as a legacy boot option if BIOS is set to Legacy or DUAL). I also tried adding gpt_hybrid_mbr="true" and confirmed that the option was getting enabled by looking at the log, but again, the results were the same.

For reference, here is the content of my updated config.xml:

<?xml version="1.0" encoding="utf-8"?>

<image schemaversion="7.4" name="TEST">
    <description type="system">
        <author>TEST</author>
        <contact></contact>
        <specification>TEST</specification>
    </description>
    <profiles>
        <profile name="Disk" description="Expandable Disk image"/>
        <profile name="SLES" description="SLES based image"/>
    </profiles>
    <preferences>
        <version>1.0.0</version>
        <packagemanager>zypper</packagemanager>
        <locale>en_US</locale>
        <keytable>us</keytable>
        <timezone>America/Los_Angeles</timezone>
        <rpm-excludedocs>false</rpm-excludedocs>
        <rpm-check-signatures>false</rpm-check-signatures>
        <bootsplash-theme>SLE</bootsplash-theme>
        <bootloader-theme>SLE</bootloader-theme>
    </preferences>
    <preferences profiles="Disk">
        <type image="oem" filesystem="ext4" initrd_system="dracut" installiso="true" kernelcmdline="splash" firmware="efi" efiparttable="gpt" gpt_hybrid_mbr="true">
            <bootloader name="grub2" console="gfxterm" timeout="30"/>
            <oemconfig>
                <oem-boot-title>Unity Common Host</oem-boot-title>
                <oem-swapsize>2048</oem-swapsize>
                <oem-swap>true</oem-swap>
                <oem-device-filter>/dev/ram</oem-device-filter>
                <oem-multipath-scan>false</oem-multipath-scan>
            </oemconfig>
        </type>
    </preferences>
    <users>
        <user password="admin" pwdformat="plain" home="/root" name="root" groups="root" shell="/bin/bash"/>
    </users>

    <repository type="rpm-md" alias="SLE-Product-15.3" imageinclude="true">
        <source path="http://*local_repo*/repo/SUSE/Products/SLE-Product-SLES/15-SP3/x86_64/product/"/>
    </repository>
    <repository type="rpm-md" alias="SLE-Module-Basesystem-15.3" imageinclude="true">
        <source path="http://*local_repo*/repo/SUSE/Products/SLE-Module-Basesystem/15-SP3/x86_64/product/"/>
    </repository>
    <repository type="rpm-md" alias="SLE-Module-Development-Tools-15.3" imageinclude="true">
        <source path="http://*local_repo*/repo/SUSE/Products/SLE-Module-Development-Tools/15-SP3/x86_64/product/"/>
    </repository>
    <repository type="rpm-md" alias="SLE-Module-Server-Applications-15.3" imageinclude="true">
        <source path="http://*local_repo*/repo/SUSE/Products/SLE-Module-Server-Applications/15-SP3/x86_64/product/"/>
    </repository>
    <repository type="rpm-md" alias="SLE-Module-Public-Cloud-15.3" imageinclude="true">
        <source path="http://*local_repo*/repo/SUSE/Products/SLE-Module-Public-Cloud/15-SP3/x86_64/product/"/>
    </repository>
    <repository type="rpm-md" alias="SLE-Module-Python2-15.3" imageinclude="true">
        <source path="http://*local_repo*/repo/SUSE/Products/SLE-Module-Python2/15-SP3/x86_64/product/"/>
    </repository>
    <repository type="rpm-md" alias="SLE-15-Backports-15.3" imageinclude="true">
        <source path="http://*local_repo*/repo/SUSE/Backports/SLE-15-SP3_x86_64/standard/"/>
    </repository>

    <packages type="image">
        <package name="ca-certificates"/>
        <package name="ca-certificates-mozilla"/>
        <package name="coreutils"/>
        <package name="iputils"/>
        <package name="netcfg"/>
        <package name="suse-build-key"/>
        <package name="timezone"/>
        <package name="rpm"/>
        <package name="kernel-default"/>
        <package name="dracut-kiwi-oem-dump"/>
        <package name="dracut-kiwi-oem-repart"/>
        <package name="grub2"/>
        <package name="grub2-x86_64-efi" arch="x86_64"/>
        <package name="iproute"/>
        <package name="dhcp-client"/>
        <package name="wicked"/>
    </packages>

    <packages type="bootstrap">
        <package name="ca-certificates"/>
        <package name="cracklib-dict-full"/>
        <package name="filesystem"/>
        <package name="glibc-locale"/>
        <package name="udev"/>
    </packages>
</image>

If you see anything amiss and can think of any other options for me to test, please let me know. We are still looking for a solution to this issue, and Supermicro is not helpful as their Virtual CDROM works fine with standard Linux ISOs, several of which I personally tested while I working on this issue back on April.

P.S. I also tested FAT12 by modifying kiwi's code and changing F16 to F12 in fat16.py on the old kiwi build we are using in production, to see if that might fix the issue, but it didn't.