Creating EFI System Partition

Creating EFI System Partition

[Robinson, Herbie]

I have been tasked with implementing EFI boot in our VOS operating system (in a panic, nobody bothered to tell us ahead of time that legacy boot was being dropped from the BIOS on our latest hardware).  I think we have a pretty good handle on writing the bootloader, but I can't find any solid information on creating an empty EFI System Partition bearing the correct flavor of FAT32 to put the bootloader in.  I need to end up with a binary image file (which I can process into an object module and embed into our OS code for initializing disks).

I found a thread on submitting a "GPT" EFI shell application on this list, but it seems to have trailed off to nowhere about two years ago.

Did that end up somewhere that I haven't found?

Herbie Robinson
Software Architect
Stratus Technologies | www.stratus.com
5 Mill and Main Place, Suite 500 | Maynard, MA 01754
T: +1-978-461-7531 | E: Herbie.Robinson@stratus.com
[Stratus Technologies]<http://go.stratus.com/US>

Re: Creating EFI System Partition

[Andrew Fish]

> On Jun 14, 2018, at 5:35 PM, Robinson, Herbie <Herbie.Robinson@stratus.com> wrote:
> 
> I have been tasked with implementing EFI boot in our VOS operating system (in a panic, nobody bothered to tell us ahead of time that legacy boot was being dropped from the BIOS on our latest hardware).  I think we have a pretty good handle on writing the bootloader, but I can't find any solid information on creating an empty EFI System Partition bearing the correct flavor of FAT32 to put the bootloader in.  I need to end up with a binary image file (which I can process into an object module and embed into our OS code for initializing disks).
> 

Herbie,

It is "bog standard" FAT3", see Microsoft Extensible Firmware Initiative FAT32 File System Specification <https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&ved=0ahUKEwi_g6P8ttTbAhUCE3wKHQMHC24QFggpMAA&url=https%3A%2F%2Fdownload.microsoft.com%2Fdownload%2F1%2F6%2F1%2F161ba512-40e2-4cc9-843a-923143f3456c%2Ffatgen103.doc&usg=AOvVaw2NwPT7yLI0sIcY1mEmzrbe>

The OS loader is just an EFI Executable in file. So any old FAT32 tools will do. 

So are you looking for EFI tools to do GPT partitioning and FAT32 file system creation? 

Thanks,

Andrew Fish

> I found a thread on submitting a "GPT" EFI shell application on this list, but it seems to have trailed off to nowhere about two years ago.
> 
> Did that end up somewhere that I haven't found?
> 
> Herbie Robinson
> Software Architect
> Stratus Technologies | www.stratus.com
> 5 Mill and Main Place, Suite 500 | Maynard, MA 01754
> T: +1-978-461-7531 | E: Herbie.Robinson@stratus.com
> [Stratus Technologies]<http://go.stratus.com/US>
> 
> _______________________________________________
> edk2-devel mailing list
> edk2-devel@lists.01.org
> https://lists.01.org/mailman/listinfo/edk2-devel

Re: Creating EFI System Partition

[Rod Smith]

On 06/14/2018 08:35 PM, Robinson, Herbie wrote:
> I have been tasked with implementing EFI boot in our VOS operating
> system (in a panic, nobody bothered to tell us ahead of time that
> legacy boot was being dropped from the BIOS on our latest hardware).
> I think we have a pretty good handle on writing the bootloader, but I
> can't find any solid information on creating an empty EFI System
> Partition bearing the correct flavor of FAT32 to put the bootloader
> in.

The EFI spec has some quirky CYA-type wording about its filesystem, but
AFAIK, any common tool for creating a FAT32 filesystem should work. I
generally do it with mkdosfs in Linux, but equivalent tools in macOS,
Windows, or the BSDs also work. In practice, FAT16 and FAT12 usually
work, too, although the EFI spec does explicitly say at one point that
the filesystem should be FAT32, and I know of at least one
implementation that gets a little flaky with FAT16, so I'd stick with
FAT32. An exception is if you need a very small filesystem, as on a
bootable optical disc, in which case FAT16 or FAT12 might be required.
Also, I've seen reports of problems with filesystems smaller than 512MiB
on some EFIs, so I recommend making it at least that large, at least if
it will be on a hard disk.

Ideally, the EFI System Partition (ESP) should be identified by the
correct partition table type code. On an MBR disk, this is 0xEF; and on
a GPT disk, it's C12A7328-F81F-11D2-BA4B-00A0C93EC93B, which is
identified in various ways depending on the partitioning software (type
EF00 in gdisk; a "boot flag" or "esp flag" set in parted; etc.). GPT is
preferable, particularly for installations on hard disks, since some
OSes tie the boot mode to the partition table type. (Of course, this
might not be an issue for you -- it depends on what you're preparing.)
In practice, I'm not aware of any EFI that actually requires the
partition type code to be set correctly; every one I've tried will boot
fine even if the type code is set to something other than an official
ESP type code. That said, I'd set the type code correctly just to be
sure it works on an oddball system, and to avoid confusion if/when users
look at the disk.

If the boot medium is an optical disc, you'll need a particular variant
of an El Torito boot image. If you need help with this, please say so; I
can dig up the details, or at least point you to a sample mkisofs command.

> I need to end up with a binary image file (which I can process
> into an object module and embed into our OS code for initializing
> disks).

GPT places data structures at both the beginning and the end of the
disk, which might create some complications, depending on your exact
needs. If you need to write an image to a blank disk of unknown size,
several options occur to me:

* You can use MBR, which does not rely on data structures at the
  end of the disk. As noted above, though, that's a little iffy in
  some cases -- but it might be fine for your case (it's hard to
  tell without more details).
* You can prepare a virtual image of a small disk and write it out
  to a larger disk, leaving the backup GPT data structures before
  the end of the disk; then either:
  * Leave it that way, which will effectively limit the size of the
    disk. This might be OK for a USB flash drive.
  * Use a disk partitioning tool to relocate the backup GPT data
    structures to the end of the disk. The sgdisk "-e" option will
    do this, for instance. IIRC, parted will do it automatically,
    or at least prompt that it be done, if any operation is performed
    on the disk.
* You can create an image of the FAT32 ESP filesystem ONLY (without
  partition table), then have your wrapper tool use sgdisk, parted,
  or some other tool to prepare a disk with GPT and an ESP. You'd
  then write out the image to the ESP on the disk you've just
  partitioned.

Caveat/full disclosure: I mostly use Linux, so I've referenced Linux
commands. I'm also the author of GPT fdisk (gdisk, sgdisk, and cgdisk);
but of course, other tools on many platforms can do what you need.

> I found a thread on submitting a "GPT" EFI shell application on this
> list, but it seems to have trailed off to nowhere about two years
> ago.
> 
> Did that end up somewhere that I haven't found?
> 
> Herbie Robinson Software Architect Stratus Technologies |
> www.stratus.com 5 Mill and Main Place, Suite 500 | Maynard, MA 01754 
> T: +1-978-461-7531 | E: Herbie.Robinson@stratus.com [Stratus
> Technologies]<http://go.stratus.com/US>
> 
> _______________________________________________ edk2-devel mailing
> list edk2-devel@lists.01.org 
> https://lists.01.org/mailman/listinfo/edk2-devel
> 


-- 
Rod Smith
rodsmith@rodsbooks.com
http://www.rodsbooks.com

Re: Creating EFI System Partition

[Andrew Fish]

> On Jun 15, 2018, at 6:17 AM, Rod Smith <rodsmith@rodsbooks.com> wrote:
> 
> On 06/14/2018 08:35 PM, Robinson, Herbie wrote:
>> I have been tasked with implementing EFI boot in our VOS operating
>> system (in a panic, nobody bothered to tell us ahead of time that
>> legacy boot was being dropped from the BIOS on our latest hardware).
>> I think we have a pretty good handle on writing the bootloader, but I
>> can't find any solid information on creating an empty EFI System
>> Partition bearing the correct flavor of FAT32 to put the bootloader
>> in.
> 
> The EFI spec has some quirky CYA-type wording about its filesystem,

Rob,

Thanks for all this info. 

The CYA is the UEFI Spec referencing the FAT spec I cross linked. Given a specification needs to be the arbiter of truth some times it is required to be pedantic. In this case the UEFI Spec needed a definition of FAT32, and that is the one and only FAT spec. contributed by Microsoft. 

> but
> AFAIK, any common tool for creating a FAT32 filesystem should work. I
> generally do it with mkdosfs in Linux, but equivalent tools in macOS,
> Windows, or the BSDs also work. In practice, FAT16 and FAT12 usually
> work, too, although the EFI spec does explicitly say at one point that
> the filesystem should be FAT32, and I know of at least one
> implementation that gets a little flaky with FAT16, so I'd stick with
> FAT32. An exception is if you need a very small filesystem, as on a
> bootable optical disc, in which case FAT16 or FAT12 might be required.
> Also, I've seen reports of problems with filesystems smaller than 512MiB
> on some EFIs, so I recommend making it at least that large, at least if
> it will be on a hard disk.
> 

I seem to remember that the FAT32 spec also defined FAT16 and FAT12. it also defines when FAT32, FAT16, or FAT12 should be used for media.

If the edk2 FAT driver has an issue with media that conform to the FAT32 spec we should fix that. If the issue is non conferment, then we need to decide if the fix can be made that will follow the FAT Spec. See that CYA was useful after all :). 

Thanks,

Andrew Fish

> Ideally, the EFI System Partition (ESP) should be identified by the
> correct partition table type code. On an MBR disk, this is 0xEF; and on
> a GPT disk, it's C12A7328-F81F-11D2-BA4B-00A0C93EC93B, which is
> identified in various ways depending on the partitioning software (type
> EF00 in gdisk; a "boot flag" or "esp flag" set in parted; etc.). GPT is
> preferable, particularly for installations on hard disks, since some
> OSes tie the boot mode to the partition table type. (Of course, this
> might not be an issue for you -- it depends on what you're preparing.)
> In practice, I'm not aware of any EFI that actually requires the
> partition type code to be set correctly; every one I've tried will boot
> fine even if the type code is set to something other than an official
> ESP type code. That said, I'd set the type code correctly just to be
> sure it works on an oddball system, and to avoid confusion if/when users
> look at the disk.
> 
> If the boot medium is an optical disc, you'll need a particular variant
> of an El Torito boot image. If you need help with this, please say so; I
> can dig up the details, or at least point you to a sample mkisofs command.
> 
>> I need to end up with a binary image file (which I can process
>> into an object module and embed into our OS code for initializing
>> disks).
> 
> GPT places data structures at both the beginning and the end of the
> disk, which might create some complications, depending on your exact
> needs. If you need to write an image to a blank disk of unknown size,
> several options occur to me:
> 
> * You can use MBR, which does not rely on data structures at the
>  end of the disk. As noted above, though, that's a little iffy in
>  some cases -- but it might be fine for your case (it's hard to
>  tell without more details).
> * You can prepare a virtual image of a small disk and write it out
>  to a larger disk, leaving the backup GPT data structures before
>  the end of the disk; then either:
>  * Leave it that way, which will effectively limit the size of the
>    disk. This might be OK for a USB flash drive.
>  * Use a disk partitioning tool to relocate the backup GPT data
>    structures to the end of the disk. The sgdisk "-e" option will
>    do this, for instance. IIRC, parted will do it automatically,
>    or at least prompt that it be done, if any operation is performed
>    on the disk.
> * You can create an image of the FAT32 ESP filesystem ONLY (without
>  partition table), then have your wrapper tool use sgdisk, parted,
>  or some other tool to prepare a disk with GPT and an ESP. You'd
>  then write out the image to the ESP on the disk you've just
>  partitioned.
> 
> Caveat/full disclosure: I mostly use Linux, so I've referenced Linux
> commands. I'm also the author of GPT fdisk (gdisk, sgdisk, and cgdisk);
> but of course, other tools on many platforms can do what you need.
> 
>> I found a thread on submitting a "GPT" EFI shell application on this
>> list, but it seems to have trailed off to nowhere about two years
>> ago.
>> 
>> Did that end up somewhere that I haven't found?
>> 
>> Herbie Robinson Software Architect Stratus Technologies |
>> www.stratus.com 5 Mill and Main Place, Suite 500 | Maynard, MA 01754 
>> T: +1-978-461-7531 | E: Herbie.Robinson@stratus.com [Stratus
>> Technologies]<http://go.stratus.com/US>
>> 
>> _______________________________________________ edk2-devel mailing
>> list edk2-devel@lists.01.org 
>> https://lists.01.org/mailman/listinfo/edk2-devel
>> 
> 
> 
> -- 
> Rod Smith
> rodsmith@rodsbooks.com <mailto:rodsmith@rodsbooks.com>
> http://www.rodsbooks.com <http://www.rodsbooks.com/>
> _______________________________________________
> edk2-devel mailing list
> edk2-devel@lists.01.org <mailto:edk2-devel@lists.01.org>
> https://lists.01.org/mailman/listinfo/edk2-devel <https://lists.01.org/mailman/listinfo/edk2-devel>

Re: Creating EFI System Partition

[Andrew Fish]

Sorry Rod, called you Rob. I blame pre coffee emailing. My bad.

Thanks,

Andrew Fish

> On Jun 15, 2018, at 8:01 AM, Andrew Fish <afish@apple.com> wrote:
> 
> 
> 
>> On Jun 15, 2018, at 6:17 AM, Rod Smith <rodsmith@rodsbooks.com <mailto:rodsmith@rodsbooks.com>> wrote:
>> 
>> On 06/14/2018 08:35 PM, Robinson, Herbie wrote:
>>> I have been tasked with implementing EFI boot in our VOS operating
>>> system (in a panic, nobody bothered to tell us ahead of time that
>>> legacy boot was being dropped from the BIOS on our latest hardware).
>>> I think we have a pretty good handle on writing the bootloader, but I
>>> can't find any solid information on creating an empty EFI System
>>> Partition bearing the correct flavor of FAT32 to put the bootloader
>>> in.
>> 
>> The EFI spec has some quirky CYA-type wording about its filesystem,
> 
> Rob,
> 
> Thanks for all this info. 
> 
> The CYA is the UEFI Spec referencing the FAT spec I cross linked. Given a specification needs to be the arbiter of truth some times it is required to be pedantic. In this case the UEFI Spec needed a definition of FAT32, and that is the one and only FAT spec. contributed by Microsoft. 
> 
>> but
>> AFAIK, any common tool for creating a FAT32 filesystem should work. I
>> generally do it with mkdosfs in Linux, but equivalent tools in macOS,
>> Windows, or the BSDs also work. In practice, FAT16 and FAT12 usually
>> work, too, although the EFI spec does explicitly say at one point that
>> the filesystem should be FAT32, and I know of at least one
>> implementation that gets a little flaky with FAT16, so I'd stick with
>> FAT32. An exception is if you need a very small filesystem, as on a
>> bootable optical disc, in which case FAT16 or FAT12 might be required.
>> Also, I've seen reports of problems with filesystems smaller than 512MiB
>> on some EFIs, so I recommend making it at least that large, at least if
>> it will be on a hard disk.
>> 
> 
> I seem to remember that the FAT32 spec also defined FAT16 and FAT12. it also defines when FAT32, FAT16, or FAT12 should be used for media.
> 
> If the edk2 FAT driver has an issue with media that conform to the FAT32 spec we should fix that. If the issue is non conferment, then we need to decide if the fix can be made that will follow the FAT Spec. See that CYA was useful after all :). 
> 
> Thanks,
> 
> Andrew Fish
> 
>> Ideally, the EFI System Partition (ESP) should be identified by the
>> correct partition table type code. On an MBR disk, this is 0xEF; and on
>> a GPT disk, it's C12A7328-F81F-11D2-BA4B-00A0C93EC93B, which is
>> identified in various ways depending on the partitioning software (type
>> EF00 in gdisk; a "boot flag" or "esp flag" set in parted; etc.). GPT is
>> preferable, particularly for installations on hard disks, since some
>> OSes tie the boot mode to the partition table type. (Of course, this
>> might not be an issue for you -- it depends on what you're preparing.)
>> In practice, I'm not aware of any EFI that actually requires the
>> partition type code to be set correctly; every one I've tried will boot
>> fine even if the type code is set to something other than an official
>> ESP type code. That said, I'd set the type code correctly just to be
>> sure it works on an oddball system, and to avoid confusion if/when users
>> look at the disk.
>> 
>> If the boot medium is an optical disc, you'll need a particular variant
>> of an El Torito boot image. If you need help with this, please say so; I
>> can dig up the details, or at least point you to a sample mkisofs command.
>> 
>>> I need to end up with a binary image file (which I can process
>>> into an object module and embed into our OS code for initializing
>>> disks).
>> 
>> GPT places data structures at both the beginning and the end of the
>> disk, which might create some complications, depending on your exact
>> needs. If you need to write an image to a blank disk of unknown size,
>> several options occur to me:
>> 
>> * You can use MBR, which does not rely on data structures at the
>> end of the disk. As noted above, though, that's a little iffy in
>> some cases -- but it might be fine for your case (it's hard to
>> tell without more details).
>> * You can prepare a virtual image of a small disk and write it out
>> to a larger disk, leaving the backup GPT data structures before
>> the end of the disk; then either:
>> * Leave it that way, which will effectively limit the size of the
>>   disk. This might be OK for a USB flash drive.
>> * Use a disk partitioning tool to relocate the backup GPT data
>>   structures to the end of the disk. The sgdisk "-e" option will
>>   do this, for instance. IIRC, parted will do it automatically,
>>   or at least prompt that it be done, if any operation is performed
>>   on the disk.
>> * You can create an image of the FAT32 ESP filesystem ONLY (without
>> partition table), then have your wrapper tool use sgdisk, parted,
>> or some other tool to prepare a disk with GPT and an ESP. You'd
>> then write out the image to the ESP on the disk you've just
>> partitioned.
>> 
>> Caveat/full disclosure: I mostly use Linux, so I've referenced Linux
>> commands. I'm also the author of GPT fdisk (gdisk, sgdisk, and cgdisk);
>> but of course, other tools on many platforms can do what you need.
>> 
>>> I found a thread on submitting a "GPT" EFI shell application on this
>>> list, but it seems to have trailed off to nowhere about two years
>>> ago.
>>> 
>>> Did that end up somewhere that I haven't found?
>>> 
>>> Herbie Robinson Software Architect Stratus Technologies |
>>> www.stratus.com 5 Mill and Main Place, Suite 500 | Maynard, MA 01754 
>>> T: +1-978-461-7531 | E: Herbie.Robinson@stratus.com [Stratus
>>> Technologies]<http://go.stratus.com/US>
>>> 
>>> _______________________________________________ edk2-devel mailing
>>> list edk2-devel@lists.01.org 
>>> https://lists.01.org/mailman/listinfo/edk2-devel
>>> 
>> 
>> 
>> -- 
>> Rod Smith
>> rodsmith@rodsbooks.com <mailto:rodsmith@rodsbooks.com> <mailto:rodsmith@rodsbooks.com <mailto:rodsmith@rodsbooks.com>>
>> http://www.rodsbooks.com <http://www.rodsbooks.com/> <http://www.rodsbooks.com/ <http://www.rodsbooks.com/>>
>> _______________________________________________
>> edk2-devel mailing list
>> edk2-devel@lists.01.org <mailto:edk2-devel@lists.01.org> <mailto:edk2-devel@lists.01.org <mailto:edk2-devel@lists.01.org>>
>> https://lists.01.org/mailman/listinfo/edk2-devel <https://lists.01.org/mailman/listinfo/edk2-devel> <https://lists.01.org/mailman/listinfo/edk2-devel <https://lists.01.org/mailman/listinfo/edk2-devel>>
> _______________________________________________
> edk2-devel mailing list
> edk2-devel@lists.01.org <mailto:edk2-devel@lists.01.org>
> https://lists.01.org/mailman/listinfo/edk2-devel <https://lists.01.org/mailman/listinfo/edk2-devel>

Re: Creating EFI System Partition

[Rod Smith]

On 06/15/2018 11:01 AM, Andrew Fish wrote:
> 
>> On Jun 15, 2018, at 6:17 AM, Rod Smith <rodsmith@rodsbooks.com 
>> <mailto:rodsmith@rodsbooks.com>> wrote:
>> 
>> but AFAIK, any common tool for creating a FAT32 filesystem should
>> work. I generally do it with mkdosfs in Linux, but equivalent tools
>> in macOS, Windows, or the BSDs also work. In practice, FAT16 and
>> FAT12 usually work, too, although the EFI spec does explicitly say
>> at one point that the filesystem should be FAT32, and I know of at
>> least one implementation that gets a little flaky with FAT16, so
>> I'd stick with FAT32.
>> 
> 
> I seem to remember that the FAT32 spec also defined FAT16 and FAT12.
> it also defines when FAT32, FAT16, or FAT12 should be used for
> media.
> 
> If the edk2 FAT driver has an issue with media that conform to the
> FAT32 spec we should fix that. If the issue is non conferment, then
> we need to decide if the fix can be made that will follow the FAT
> Spec. See that CYA was useful after all :).

No, I didn't mean to imply that the EDK2 FAT driver gets flaky with 12-
or 16-bit FAT filesystems. The EFI in question was an early
EFI-over-BIOS implementation on a Gigabyte motherboard from 2011 or
2012. That thing was a horror, and I wrote up my experiences at the time:

http://www.rodsbooks.com/gb-hybrid-efi/

I no longer have that motherboard, so I can't do any more tests with it
or double-check my findings from 2012. From that page, though:

: A FAT-16 ESP, on the other hand, seems problematic. Ubuntu 11.04 (and
: 11.10) in EFI mode creates a dinky FAT-16 ESP, and after my test
: install of Ubuntu 11.04, the board hung on reboot until I reworked
: the ESP as FAT-32. Thus, if you plan to install Ubuntu, or any other
: OS that creates a FAT-16 ESP, be prepared to fix it, preferably
: before the system reboots!

Note that Ubuntu no longer creates a "dinky FAT-16 ESP;" it now creates
a 512MiB FAT-32 ESP. The experience remains a relevant cautionary tale,
though, for anybody who's trying to write an OS installer, particularly
if the system must be installable on some random computer -- systems
from that period are still in use today, so a FAT-16 ESP could cause
problems in the real world. That said, I've not encountered this problem
on any modern (say, 2014 or later) EFI.

-- 
Rod Smith
rodsmith@rodsbooks.com
http://www.rodsbooks.com