Partition issue with Linux Distribution iso image

Partition issue with Linux Distribution iso image

[Gao, Zhichao]

[-- Attachment #1: Type: text/plain, Size: 1545 bytes --]

Hi All,

Some Linux Distribution ISO image would contain both Eltorito(iso 9660) volume info and MBR table at the same time. That would cause the partition check pass the MBR and ignore the UDF (Eltorito compatible) check.
There is a bug in the partition child handler: https://bugzilla.tianocore.org/show_bug.cgi?id=2843. It would make the MBR transfer the wrong device size. That cause the FAT driver fail to install the FAT       protocol onto the device.
But when I solve the issue, the EFI partition can be found as MBR FAT device. That would cause another issue. Boot from the MBR FAT from ISO image is different with boot from CD FAT.
Here is the difference:
Boot from MBR FAT: go to the grub terminal window.
Boot from CD FAT: go to the installation window.
I don't know why the same grub image act different behaviors. But I believe "Boot from CD FAT" is the right behavior. That means the ISO 9660 should not be treated as MBR device in UEFI BIOS.

So I would like to add a logic: the ISO image that contain both ISO 9660 volume info and MBR partition, the partition driver would skip the MBR check and continue to do the ISO 9660 check. There is a easy way to do this, i.e. put the UDF (ISO 9660 compatible) before the MBR check. It means the partition routine child handle would have a priority and the UDF (ISO 9660) has high priority than the MBR. That would solve all the Linux Distribution ISO image issue in my thought.

I want to get some suggestion from the community before I sent the patch.

Thanks,
Zhichao

[-- Attachment #2: Type: text/html, Size: 3926 bytes --]

Re: [edk2-devel] Partition issue with Linux Distribution iso image

[Andrew Fish]

[-- Attachment #1: Type: text/plain, Size: 5209 bytes --]

Zhichao,

The discovery algorithm for partitions is defined in the UEFI Spec. 

13.3.2 Partition Discovery
...

The following is the order in which a block device must be scanned to determine if it contains partitions.
When a check for a valid partitioning scheme succeeds, the search terminates.

1) Check for GUID Partition Table Headers.
2) Follow ISO-9660 specification to search for ISO-9660 volume structures on the magic LBA. 
3) Check for an “El Torito” volume extension and follow the “El Torito” CD-ROM specification. 
4) If none of the above, check LBA 0 for a legacy MBR partition table.
5) No partition found on device.

If a disk contains a recognized RAID structure (e.g. DDF structure as defined in The Storage Networking Industry Association Common RAID Disk Data Format Specification--see Glossary), the data on the disk must be ignored, unless the driver is using the RAID structure to produce a logical RAID volume.
EFI supports the nesting of legacy MBR partitions, by allowing any legacy MBR partition to contain more legacy MBR partitions. This is accomplished by supporting the same partition discovery algorithm on every logical block device. It should be noted that the GUID Partition Table does not allow nesting of GUID Partition Table Headers. Nesting is not needed since a GUID Partition Table Header can support an arbitrary number of partitions (the addressability limits of a 64-bit LBA are the limiting factor).

13.3.2.1 ISO-9660 and El Torito
IS0-9660 is the industry standard low level format used on CD-ROM and DVD-ROM. The CD-ROM format is completely described by the “El Torito” Bootable CD-ROM Format Specification Version 1.0. To boot from a CD-ROM or DVD-ROM in the boot services environment, an EFI System partition is stored in a “no emulation” mode as defined by the “El Torito” specification. A Platform ID of 0xEF indicates an EFI System Partition. The Platform ID is in either the Section Header Entry or the Validation Entry of the Booting Catalog as defined by the “El Torito” specification. EFI differs from “El Torito” “no emulation” mode in that it does not load the “no emulation” image into memory and jump to it. EFI interprets the “no emulation” image as an EFI system partition. EFI interprets the Sector Count in the Initial/Default Entry or the Section Header Entry to be the size of the EFI system partition. If the value of Sector Count is set to 0 or 1, EFI will assume the system partition consumes the space from the beginning of the “no emulation” image to the end of the CD-ROM.
A DVD-ROM image formatted as required by the UDF 2.0 specification (OSTA Universal Disk Format Specification, Revision 2.0) shall be booted by UEFI if:
• the DVD-ROM image conforms to the "UDF Bridge" format defined in the UDF 2.0 specification, and
• the DVD-ROM image contains exactly one ISO-9660 file system, and
• the ISO-9660 file system conforms to the "El Torito" Bootable CD-ROM Format Specification.
Booting from a DVD-ROM that satisfies the above requirements is accomplished using the same methods as booting from a CD-ROM: the ISO-9660 file system shall be booted.
Since the EFI file system definition does not use the same Initial/Default entry as a legacy CD-ROM it is possible to boot personal computers using an EFI CD-ROM or DVD-ROM. The inclusion of boot code for personal computers is optional and not required by EFI.

Thanks,

Andrew Fish

> On Aug 9, 2020, at 6:51 AM, Gao, Zhichao <zhichao.gao@intel.com> wrote:
> 
> Hi All,
>
> Some Linux Distribution ISO image would contain both Eltorito(iso 9660) volume info and MBR table at the same time. That would cause the partition check pass the MBR and ignore the UDF (Eltorito compatible) check.
> There is a bug in the partition child handler: https://bugzilla.tianocore.org/show_bug.cgi?id=2843 <https://bugzilla.tianocore.org/show_bug.cgi?id=2843>. It would make the MBR transfer the wrong device size. That cause the FAT driver fail to install the FAT       protocol onto the device.
> But when I solve the issue, the EFI partition can be found as MBR FAT device. That would cause another issue. Boot from the MBR FAT from ISO image is different with boot from CD FAT.
> Here is the difference:
> Boot from MBR FAT: go to the grub terminal window.
> Boot from CD FAT: go to the installation window.
> I don’t know why the same grub image act different behaviors. But I believe “Boot from CD FAT” is the right behavior. That means the ISO 9660 should not be treated as MBR device in UEFI BIOS.
>
> So I would like to add a logic: the ISO image that contain both ISO 9660 volume info and MBR partition, the partition driver would skip the MBR check and continue to do the ISO 9660 check. There is a easy way to do this, i.e. put the UDF (ISO 9660 compatible) before the MBR check. It means the partition routine child handle would have a priority and the UDF (ISO 9660) has high priority than the MBR. That would solve all the Linux Distribution ISO image issue in my thought.
>
> I want to get some suggestion from the community before I sent the patch.
>
> Thanks,
> Zhichao
> 


[-- Attachment #2: Type: text/html, Size: 8658 bytes --]

Re: [edk2-devel] Partition issue with Linux Distribution iso image

[Gao, Zhichao]

[-- Attachment #1: Type: text/plain, Size: 5879 bytes --]

Andrew,

Thanks for your info. I didn’t aware the partition driver is part of the UEFI spec. And the order is mismatch between the code implementation and spec description.
The order now is GPT > MBR > UDF( ISO 9660). I would send a patch to correct it as it would both follow the spec and fix our issue.

Thanks,
Zhichao

From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Andrew Fish via groups.io
Sent: Monday, August 10, 2020 5:15 AM
To: devel@edk2.groups.io; Gao, Zhichao <zhichao.gao@intel.com>
Cc: Ni, Ray <ray.ni@intel.com>; Laszlo Ersek <lersek@redhat.com>; Gary Lin <glin@suse.com>; Wu, Hao A <hao.a.wu@intel.com>
Subject: Re: [edk2-devel] Partition issue with Linux Distribution iso image

Zhichao,

The discovery algorithm for partitions is defined in the UEFI Spec.

13.3.2 Partition Discovery
...

The following is the order in which a block device must be scanned to determine if it contains partitions.
When a check for a valid partitioning scheme succeeds, the search terminates.

1) Check for GUID Partition Table Headers.
2) Follow ISO-9660 specification to search for ISO-9660 volume structures on the magic LBA.
3) Check for an “El Torito” volume extension and follow the “El Torito” CD-ROM specification.
4) If none of the above, check LBA 0 for a legacy MBR partition table.
5) No partition found on device.

If a disk contains a recognized RAID structure (e.g. DDF structure as defined in The Storage Networking Industry Association Common RAID Disk Data Format Specification--see Glossary), the data on the disk must be ignored, unless the driver is using the RAID structure to produce a logical RAID volume.
EFI supports the nesting of legacy MBR partitions, by allowing any legacy MBR partition to contain more legacy MBR partitions. This is accomplished by supporting the same partition discovery algorithm on every logical block device. It should be noted that the GUID Partition Table does not allow nesting of GUID Partition Table Headers. Nesting is not needed since a GUID Partition Table Header can support an arbitrary number of partitions (the addressability limits of a 64-bit LBA are the limiting factor).

13.3.2.1 ISO-9660 and El Torito
IS0-9660 is the industry standard low level format used on CD-ROM and DVD-ROM. The CD-ROM format is completely described by the “El Torito” Bootable CD-ROM Format Specification Version 1.0. To boot from a CD-ROM or DVD-ROM in the boot services environment, an EFI System partition is stored in a “no emulation” mode as defined by the “El Torito” specification. A Platform ID of 0xEF indicates an EFI System Partition. The Platform ID is in either the Section Header Entry or the Validation Entry of the Booting Catalog as defined by the “El Torito” specification. EFI differs from “El Torito” “no emulation” mode in that it does not load the “no emulation” image into memory and jump to it. EFI interprets the “no emulation” image as an EFI system partition. EFI interprets the Sector Count in the Initial/Default Entry or the Section Header Entry to be the size of the EFI system partition. If the value of Sector Count is set to 0 or 1, EFI will assume the system partition consumes the space from the beginning of the “no emulation” image to the end of the CD-ROM.
A DVD-ROM image formatted as required by the UDF 2.0 specification (OSTA Universal Disk Format Specification, Revision 2.0) shall be booted by UEFI if:
• the DVD-ROM image conforms to the "UDF Bridge" format defined in the UDF 2.0 specification, and
• the DVD-ROM image contains exactly one ISO-9660 file system, and
• the ISO-9660 file system conforms to the "El Torito" Bootable CD-ROM Format Specification.
Booting from a DVD-ROM that satisfies the above requirements is accomplished using the same methods as booting from a CD-ROM: the ISO-9660 file system shall be booted.
Since the EFI file system definition does not use the same Initial/Default entry as a legacy CD-ROM it is possible to boot personal computers using an EFI CD-ROM or DVD-ROM. The inclusion of boot code for personal computers is optional and not required by EFI.

Thanks,

Andrew Fish


On Aug 9, 2020, at 6:51 AM, Gao, Zhichao <zhichao.gao@intel.com<mailto:zhichao.gao@intel.com>> wrote:

Hi All,

Some Linux Distribution ISO image would contain both Eltorito(iso 9660) volume info and MBR table at the same time. That would cause the partition check pass the MBR and ignore the UDF (Eltorito compatible) check.
There is a bug in the partition child handler: https://bugzilla.tianocore.org/show_bug.cgi?id=2843. It would make the MBR transfer the wrong device size. That cause the FAT driver fail to install the FAT       protocol onto the device.
But when I solve the issue, the EFI partition can be found as MBR FAT device. That would cause another issue. Boot from the MBR FAT from ISO image is different with boot from CD FAT.
Here is the difference:
Boot from MBR FAT: go to the grub terminal window.
Boot from CD FAT: go to the installation window.
I don’t know why the same grub image act different behaviors. But I believe “Boot from CD FAT” is the right behavior. That means the ISO 9660 should not be treated as MBR device in UEFI BIOS.

So I would like to add a logic: the ISO image that contain both ISO 9660 volume info and MBR partition, the partition driver would skip the MBR check and continue to do the ISO 9660 check. There is a easy way to do this, i.e. put the UDF (ISO 9660 compatible) before the MBR check. It means the partition routine child handle would have a priority and the UDF (ISO 9660) has high priority than the MBR. That would solve all the Linux Distribution ISO image issue in my thought.

I want to get some suggestion from the community before I sent the patch.

Thanks,
Zhichao



[-- Attachment #2: Type: text/html, Size: 11422 bytes --]

Re: [edk2-devel] Partition issue with Linux Distribution iso image

[Laszlo Ersek]

On 08/10/20 05:04, Gao, Zhichao wrote:
> Andrew,
> 
> Thanks for your info. I didn’t aware the partition driver is part of the UEFI spec. And the order is mismatch between the code implementation and spec description.
> The order now is GPT > MBR > UDF( ISO 9660). I would send a patch to correct it as it would both follow the spec and fix our issue.

Sounds good, thanks.
Laszlo

> 
> Thanks,
> Zhichao
> 
> From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Andrew Fish via groups.io
> Sent: Monday, August 10, 2020 5:15 AM
> To: devel@edk2.groups.io; Gao, Zhichao <zhichao.gao@intel.com>
> Cc: Ni, Ray <ray.ni@intel.com>; Laszlo Ersek <lersek@redhat.com>; Gary Lin <glin@suse.com>; Wu, Hao A <hao.a.wu@intel.com>
> Subject: Re: [edk2-devel] Partition issue with Linux Distribution iso image
> 
> Zhichao,
> 
> The discovery algorithm for partitions is defined in the UEFI Spec.
> 
> 13.3.2 Partition Discovery
> ...
> 
> The following is the order in which a block device must be scanned to determine if it contains partitions.
> When a check for a valid partitioning scheme succeeds, the search terminates.
> 
> 1) Check for GUID Partition Table Headers.
> 2) Follow ISO-9660 specification to search for ISO-9660 volume structures on the magic LBA.
> 3) Check for an “El Torito” volume extension and follow the “El Torito” CD-ROM specification.
> 4) If none of the above, check LBA 0 for a legacy MBR partition table.
> 5) No partition found on device.
> 
> If a disk contains a recognized RAID structure (e.g. DDF structure as defined in The Storage Networking Industry Association Common RAID Disk Data Format Specification--see Glossary), the data on the disk must be ignored, unless the driver is using the RAID structure to produce a logical RAID volume.
> EFI supports the nesting of legacy MBR partitions, by allowing any legacy MBR partition to contain more legacy MBR partitions. This is accomplished by supporting the same partition discovery algorithm on every logical block device. It should be noted that the GUID Partition Table does not allow nesting of GUID Partition Table Headers. Nesting is not needed since a GUID Partition Table Header can support an arbitrary number of partitions (the addressability limits of a 64-bit LBA are the limiting factor).
> 
> 13.3.2.1 ISO-9660 and El Torito
> IS0-9660 is the industry standard low level format used on CD-ROM and DVD-ROM. The CD-ROM format is completely described by the “El Torito” Bootable CD-ROM Format Specification Version 1.0. To boot from a CD-ROM or DVD-ROM in the boot services environment, an EFI System partition is stored in a “no emulation” mode as defined by the “El Torito” specification. A Platform ID of 0xEF indicates an EFI System Partition. The Platform ID is in either the Section Header Entry or the Validation Entry of the Booting Catalog as defined by the “El Torito” specification. EFI differs from “El Torito” “no emulation” mode in that it does not load the “no emulation” image into memory and jump to it. EFI interprets the “no emulation” image as an EFI system partition. EFI interprets the Sector Count in the Initial/Default Entry or the Section Header Entry to be the size of the EFI system partition. If the value of Sector Count is set to 0 or 1, EFI will assume the system partition consumes the space from the beginning of the “no emulation” image to the end of the CD-ROM.
> A DVD-ROM image formatted as required by the UDF 2.0 specification (OSTA Universal Disk Format Specification, Revision 2.0) shall be booted by UEFI if:
> • the DVD-ROM image conforms to the "UDF Bridge" format defined in the UDF 2.0 specification, and
> • the DVD-ROM image contains exactly one ISO-9660 file system, and
> • the ISO-9660 file system conforms to the "El Torito" Bootable CD-ROM Format Specification.
> Booting from a DVD-ROM that satisfies the above requirements is accomplished using the same methods as booting from a CD-ROM: the ISO-9660 file system shall be booted.
> Since the EFI file system definition does not use the same Initial/Default entry as a legacy CD-ROM it is possible to boot personal computers using an EFI CD-ROM or DVD-ROM. The inclusion of boot code for personal computers is optional and not required by EFI.
> 
> Thanks,
> 
> Andrew Fish
> 
> 
> On Aug 9, 2020, at 6:51 AM, Gao, Zhichao <zhichao.gao@intel.com<mailto:zhichao.gao@intel.com>> wrote:
> 
> Hi All,
> 
> Some Linux Distribution ISO image would contain both Eltorito(iso 9660) volume info and MBR table at the same time. That would cause the partition check pass the MBR and ignore the UDF (Eltorito compatible) check.
> There is a bug in the partition child handler: https://bugzilla.tianocore.org/show_bug.cgi?id=2843. It would make the MBR transfer the wrong device size. That cause the FAT driver fail to install the FAT       protocol onto the device.
> But when I solve the issue, the EFI partition can be found as MBR FAT device. That would cause another issue. Boot from the MBR FAT from ISO image is different with boot from CD FAT.
> Here is the difference:
> Boot from MBR FAT: go to the grub terminal window.
> Boot from CD FAT: go to the installation window.
> I don’t know why the same grub image act different behaviors. But I believe “Boot from CD FAT” is the right behavior. That means the ISO 9660 should not be treated as MBR device in UEFI BIOS.
> 
> So I would like to add a logic: the ISO image that contain both ISO 9660 volume info and MBR partition, the partition driver would skip the MBR check and continue to do the ISO 9660 check. There is a easy way to do this, i.e. put the UDF (ISO 9660 compatible) before the MBR check. It means the partition routine child handle would have a priority and the UDF (ISO 9660) has high priority than the MBR. That would solve all the Linux Distribution ISO image issue in my thought.
> 
> I want to get some suggestion from the community before I sent the patch.
> 
> Thanks,
> Zhichao
> 
> 
> 
> 
>