EXT FS support

EXT FS support

[Marcin Wojtas]

Hello everyone,

Is there any plan of adding EXT filesystem support to EDK2? If not
officially is there a chance that it exists on some old branch, or in
any side project?

I'd be very grateful for any information.

Best regards,
Marcin

Re: EXT FS support

[Pete Batard]

Hi Marcin,

I can't speak for EDK2 integration plans, but as far as side-projects 
are concerned, I have been porting the various GRUB *read-only* file 
systems into standalone UEFI drivers. This includes an ext2/3/4 driver 
if you are interested (and you can live without write support).

Please see http://efi.akeo.ie/ as well as the github project at 
https://github.com/pbatard/efifs.

Regards,

/Pete

On 2016.11.22 15:25, Marcin Wojtas wrote:
> Hello everyone,
>
> Is there any plan of adding EXT filesystem support to EDK2? If not
> officially is there a chance that it exists on some old branch, or in
> any side project?
>
> I'd be very grateful for any information.
>
> Best regards,
> Marcin
> _______________________________________________
> edk2-devel mailing list
> edk2-devel@lists.01.org
> https://lists.01.org/mailman/listinfo/edk2-devel
>

Re: EXT FS support

[Rebecca Cran]

On 11/22/16 8:25 AM, Marcin Wojtas wrote:
> Hello everyone,
>
> Is there any plan of adding EXT filesystem support to EDK2? If not
> officially is there a chance that it exists on some old branch, or in
> any side project?

The rEFInd boot manager (http://www.rodsbooks.com/refind) has drivers 
for several filesystems including EXT and NTFS. From 
http://www.rodsbooks.com/refind/drivers.html :

"As described in the next section, Selecting an EFI Driver 
[http://www.rodsbooks.com/refind/drivers.html#selecting], drivers for 
ext2fs, ext3fs, ext4fs, ReiserFS, Btrfs, and a few non-native 
filesystems come with rEFInd."


-- 
Rebecca

Re: EXT FS support

[Rebecca Cran]

On 11/22/16 9:19 AM, Pete Batard wrote:
> Hi Marcin,
>
> I can't speak for EDK2 integration plans, but as far as side-projects
> are concerned, I have been porting the various GRUB *read-only* file
> systems into standalone UEFI drivers. This includes an ext2/3/4 driver
> if you are interested (and you can live without write support).
>
> Please see http://efi.akeo.ie/ as well as the github project at
> https://github.com/pbatard/efifs.

Pete,

Have you seen the UEFI filesystem drivers from the rEFInd project, at 
https://sourceforge.net/p/refind/code/ci/master/tree/filesystems/ ?

-- 
Rebecca

Re: EXT FS support

[Michael Zimmermann]

I wouldn't call this a stable implementation atm but I've ported all
filesystems supported by the linux kernel(ntfs, ext2/3/4, f2fs, ...)
including write support to UEFI using the LKL(Linux Kernel Library)
project.

https://github.com/efidroid/uefi_apps_LKL

Thanks
Michael

On Tue, Nov 22, 2016 at 5:41 PM, Rebecca Cran <rebecca@bluestop.org> wrote:
> On 11/22/16 9:19 AM, Pete Batard wrote:
>>
>> Hi Marcin,
>>
>> I can't speak for EDK2 integration plans, but as far as side-projects
>> are concerned, I have been porting the various GRUB *read-only* file
>> systems into standalone UEFI drivers. This includes an ext2/3/4 driver
>> if you are interested (and you can live without write support).
>>
>> Please see http://efi.akeo.ie/ as well as the github project at
>> https://github.com/pbatard/efifs.
>
>
> Pete,
>
> Have you seen the UEFI filesystem drivers from the rEFInd project, at
> https://sourceforge.net/p/refind/code/ci/master/tree/filesystems/ ?
>
> --
> Rebecca
>
> _______________________________________________
> edk2-devel mailing list
> edk2-devel@lists.01.org
> https://lists.01.org/mailman/listinfo/edk2-devel

Re: EXT FS support

[Marcin Wojtas]

Thank you all,

I also found this:
https://sourceforge.net/p/cloverefiboot/code/HEAD/tree/FileSystems/VBoxFsDxe/

I'll specify more my needs. I use ARMv8 platform on top of the newest
OpenPlatformPackage (linaro/master) in order to build and work as a
part of EDK2 tianocore/master. There's no GRUB, platform simply boots
to the Shell.

I'd be grateful also for pointing the easiest way to port one of the
EXT solutions. I went over all url's provided and I must say porting
seems not trivial. Write is nice-to-have feature, but we can go with
read-only, unless it's as fast as FAT.

Best regards,
Marcin



2016-11-22 17:59 GMT+01:00 Michael Zimmermann <sigmaepsilon92@gmail.com>:
> I wouldn't call this a stable implementation atm but I've ported all
> filesystems supported by the linux kernel(ntfs, ext2/3/4, f2fs, ...)
> including write support to UEFI using the LKL(Linux Kernel Library)
> project.
>
> https://github.com/efidroid/uefi_apps_LKL
>
> Thanks
> Michael
>
> On Tue, Nov 22, 2016 at 5:41 PM, Rebecca Cran <rebecca@bluestop.org> wrote:
>> On 11/22/16 9:19 AM, Pete Batard wrote:
>>>
>>> Hi Marcin,
>>>
>>> I can't speak for EDK2 integration plans, but as far as side-projects
>>> are concerned, I have been porting the various GRUB *read-only* file
>>> systems into standalone UEFI drivers. This includes an ext2/3/4 driver
>>> if you are interested (and you can live without write support).
>>>
>>> Please see http://efi.akeo.ie/ as well as the github project at
>>> https://github.com/pbatard/efifs.
>>
>>
>> Pete,
>>
>> Have you seen the UEFI filesystem drivers from the rEFInd project, at
>> https://sourceforge.net/p/refind/code/ci/master/tree/filesystems/ ?
>>
>> --
>> Rebecca
>>
>> _______________________________________________
>> edk2-devel mailing list
>> edk2-devel@lists.01.org
>> https://lists.01.org/mailman/listinfo/edk2-devel

Re: EXT FS support

[Pete Batard]

Hi Rebecca,

On 2016.11.22 16:41, Rebecca Cran wrote:
> Have you seen the UEFI filesystem drivers from the rEFInd project, at
> https://sourceforge.net/p/refind/code/ci/master/tree/filesystems/ ?

I did.

But I had some concerns over the licensing at the time (which I shared 
with Rod Smith) and this is why I preferred to go with port of the GPLv3 
GRUB drivers instead.

Regards,

/Pete

Re: EXT FS support

[Pete Batard]

On 2016.11.22 17:16, Marcin Wojtas wrote:
> There's no GRUB, platform simply boots to the Shell.

I think there may be some confusion.

All the drivers we linked you to are pure UEFI drivers. It doesn't 
matter if they were a port from GRUB or something else, the code was 
converted to work as a standalone UEFI driver. Especially you don't need 
to have GRUB running or anything. You can just use the "load" command in 
the shell to load the driver, and then access the file system.

I believe you should be able to download any of the ext binary drivers 
mentioned and use them in your shell right away to access your file 
system from it.

> I'd be grateful also for pointing the easiest way to port one of the
> EXT solutions. I went over all url's provided and I must say porting
> seems not trivial.

I don't think there's anything to port. The drivers have already been 
ported for UEFI (though, in the case of efifs, they rely on gnu-efi for 
compilation rather than EDK2 - but this has the benefit of being able to 
use a regular Visual Studio 2015 solution).

Regards,

/Pete

Re: EXT FS support

[Rod Smith]

On 11/22/2016 01:25 PM, Pete Batard wrote:
> On 2016.11.22 17:16, Marcin Wojtas wrote:
>> There's no GRUB, platform simply boots to the Shell.
> 
> I think there may be some confusion.
> 
> All the drivers we linked you to are pure UEFI drivers. It doesn't
> matter if they were a port from GRUB or something else, the code was
> converted to work as a standalone UEFI driver. Especially you don't need
> to have GRUB running or anything. You can just use the "load" command in
> the shell to load the driver, and then access the file system.
> 
> I believe you should be able to download any of the ext binary drivers
> mentioned and use them in your shell right away to access your file
> system from it.

Yes, but it sounds like Marcin may want to embed the ext4fs driver in a
custom EFI build. AFAIK, none of the drivers in question were designed
with that in mind; however, the VirtualBox project has incorporated
ISO-9660 and HFS+ drivers, both of which are built on the same framework
(rEFIt's) as rEFInd's drivers, into its own firmware. Thus, Marcin might
be able to look at the VirtualBox code and use whatever techniques or
glue it uses to incorporate something else. (I can't point to specific
files, though.) The rEFInd drivers might be easiest to build in this
way, but that's just a guess.

Note, however, that all of the drivers referenced so far in this thread
are licensed under the GPL. Thus, building an EFI in this way would
cause the EFI as a whole to be GPL-licensed. This might or might not be
an issue, depending on what the point of the exercise is.

Of course, a standalone driver might be perfectly acceptable, too. I've
seen options in some EFIs to load drivers at start time, but I've never
gotten them to work. (I haven't tried very hard.) If nothing else, a
small driver-loading program could be written and set as the first boot
option.

Marcin wrote:

> I also found this:
> https://sourceforge.net/p/cloverefiboot/code/HEAD/tree/FileSystems/VBoxFsDxe/

FWIW, that's a fork of the rEFInd code. I'd not seen it before now; the
Clover developers haven't bothered to upstream their changes. (I
maintain rEFInd.)

It's still not clear to me why you want this driver, Marcin. If you want
to load a Linux kernel directly, without using GRUB, rEFInd, or some
other tool, you can simply put the kernel(s) on a FAT filesystem. This
is a common approach among Arch Linux users; they mount the ESP at /boot
and it works pretty well. Some distributions assume that /boot supports
links or other features that aren't available with FAT, though, so maybe
this wouldn't work as well for you, but it's worth considering.

-- 
Rod Smith
rodsmith@rodsbooks.com
http://www.rodsbooks.com

Re: EXT FS support

[Laszlo Ersek]

On 11/22/16 22:31, Rod Smith wrote:
> On 11/22/2016 01:25 PM, Pete Batard wrote:
>> On 2016.11.22 17:16, Marcin Wojtas wrote:
>>> There's no GRUB, platform simply boots to the Shell.
>>
>> I think there may be some confusion.
>>
>> All the drivers we linked you to are pure UEFI drivers. It doesn't
>> matter if they were a port from GRUB or something else, the code was
>> converted to work as a standalone UEFI driver. Especially you don't need
>> to have GRUB running or anything. You can just use the "load" command in
>> the shell to load the driver, and then access the file system.
>>
>> I believe you should be able to download any of the ext binary drivers
>> mentioned and use them in your shell right away to access your file
>> system from it.
> 
> Yes, but it sounds like Marcin may want to embed the ext4fs driver in a
> custom EFI build. AFAIK, none of the drivers in question were designed
> with that in mind; however, the VirtualBox project has incorporated
> ISO-9660 and HFS+ drivers, both of which are built on the same framework
> (rEFIt's) as rEFInd's drivers, into its own firmware. Thus, Marcin might
> be able to look at the VirtualBox code and use whatever techniques or
> glue it uses to incorporate something else. (I can't point to specific
> files, though.) The rEFInd drivers might be easiest to build in this
> way, but that's just a guess.

Any valid EFI binary can be built into edk2 platforms easily. As an example, I recommend looking at

  EdkShellBinPkg/FullShell/FullShell.inf

and the following portions from OvmfPkg/OvmfPkgX64.fdf:

INF  RuleOverride = BINARY EdkShellBinPkg/FullShell/FullShell.inf
...

[Rule.Common.UEFI_DRIVER.BINARY]
  FILE DRIVER = $(NAMED_GUID) {
    DXE_DEPEX DXE_DEPEX Optional      |.depex
    PE32      PE32                    |.efi
    UI        STRING="$(MODULE_NAME)" Optional
    VERSION   STRING="$(INF_VERSION)" Optional BUILD_NUM=$(BUILD_NUMBER)
  }
[Rule.Common.UEFI_APPLICATION.BINARY]
  FILE APPLICATION = $(NAMED_GUID) {
    PE32      PE32                    |.efi
    UI        STRING="$(MODULE_NAME)" Optional
    VERSION   STRING="$(INF_VERSION)" Optional BUILD_NUM=$(BUILD_NUMBER)
  }

> 
> Note, however, that all of the drivers referenced so far in this thread
> are licensed under the GPL. Thus, building an EFI in this way would
> cause the EFI as a whole to be GPL-licensed.

Yes.

Also, as far as I'm aware, OpenSSL and GPL don't mix, so a firmware binary that combined edk2's OpenSSL-based Secure Boot stack and GPL drivers might be undistributable. (It would be fine for in-house use.)

https://people.gnome.org/~markmc/openssl-and-the-gpl.html

There's another option (and while it requires initial setup on the end-user's side, it doesn't require constant fiddling): copy the stand-alone, GPL-licensed UEFI_DRIVER binary to a small, separate VFAT filesystem, then create a Driver#### option that points to it.

(At least with edk2 and OVMF, the driver options are dispatched between BeforeConsole and AfterConsole, and OVMF's AfterConsole calls EfiBootManagerConnectAll(). The result is that filesystems recognized by the drivers loaded from the small VFAT partition would all be bound, and would become available for booting off of.)

This VFAT + Driver#### setup could even be done by customized OS installers, without user interaction.

Now I understand that the point might be to eliminate a VFAT partition altogether, but after the initial driver installation, it could become read-only from the OS runtime POV as well, and the real boot loader stuff could reside on ext2.

Separately, a small note on ext4 (because you mention it above). I seem to recall a filesystem expert colleague of mine advise *against* using journaled filesystems for booting with e.g. grub2. The argument goes (if I recall right), XFS is considered to be in clean state if the data has made it to the final location *or* the persistent journal. When you cleanly unmount (or remount r/o) and shut down, the journal will be flushed to the final location, so a boot loader that doesn't know about the journal will read consistent data. However, if you crash *without* data loss, then part of the data might be in the journal only, and only clients that can read the journal will see consistent data.

> This might or might not be
> an issue, depending on what the point of the exercise is.
> 
> Of course, a standalone driver might be perfectly acceptable, too. I've
> seen options in some EFIs to load drivers at start time, but I've never
> gotten them to work. (I haven't tried very hard.) If nothing else, a
> small driver-loading program could be written and set as the first boot
> option.

Newer firmware might also support SysPrep#### options for such applications (although firmware that supports SysPrep#### would arguably get Driver#### right as well).

Invoking the same logic / app from a regular boot option, and exiting with success, might or might not chain to the next boot option automatically. For example, in "MdeModulePkg/Universal/BdsDxe/BdsEntry.c", edk2 has 

    //
    // If the boot via Boot#### returns with a status of EFI_SUCCESS, platform firmware
    // supports boot manager menu, and if firmware is configured to boot in an
    // interactive mode, the boot manager will stop processing the BootOrder variable and
    // present a boot manager menu to the user.
    //

I guess invariably exiting with failure could ensure chaining.

> 
> Marcin wrote:
> 
>> I also found this:
>> https://sourceforge.net/p/cloverefiboot/code/HEAD/tree/FileSystems/VBoxFsDxe/
> 
> FWIW, that's a fork of the rEFInd code. I'd not seen it before now; the
> Clover developers haven't bothered to upstream their changes. (I
> maintain rEFInd.)
> 
> It's still not clear to me why you want this driver, Marcin. If you want
> to load a Linux kernel directly, without using GRUB, rEFInd, or some
> other tool, you can simply put the kernel(s) on a FAT filesystem.

Yes, if the kernel was built with the EFI stub, it can be launched as a UEFI application.

Furthermore, arm and arm64 kernels parse the "initrd=..." command line parameter in the EFI stub, and can load the initrd from the same filesystem (using the UEFI drivers) where the kernel image was loaded from. The simplest (maybe only?) use case is to provide just a filename (no pathname separators) with "initrd=", and then the file will be loaded from the same directory as the kernel, IIRC.

This enables a development style where one
- sets up one boot option, pointing to the kernel image (with the EFI stub) as the binary to launch,
- stores "initrd=..." in the boot option's LoadOptions field,
- boots the system, modifies and rebuilds the kernel,
- installs the kernel and the initrd in the exact same spot as before (under the same pathnames on the ESP),
- reboots.

> This
> is a common approach among Arch Linux users; they mount the ESP at /boot
> and it works pretty well. Some distributions assume that /boot supports
> links or other features that aren't available with FAT, though, so maybe
> this wouldn't work as well for you, but it's worth considering.
> 

Thanks
Laszlo

Re: EXT FS support

[Michael Zimmermann]

Laszlo,

can you explain the problems with putting GPL code in the fdf?
afaik every efi file is like a small self-contained application and
shouldn't the UEFI API's be excluded from derived work like the linux
kernel's system call table?

Thanks
Michael

On Wed, Nov 23, 2016 at 8:50 PM, Laszlo Ersek <lersek@redhat.com> wrote:
> On 11/22/16 22:31, Rod Smith wrote:
>> On 11/22/2016 01:25 PM, Pete Batard wrote:
>>> On 2016.11.22 17:16, Marcin Wojtas wrote:
>>>> There's no GRUB, platform simply boots to the Shell.
>>>
>>> I think there may be some confusion.
>>>
>>> All the drivers we linked you to are pure UEFI drivers. It doesn't
>>> matter if they were a port from GRUB or something else, the code was
>>> converted to work as a standalone UEFI driver. Especially you don't need
>>> to have GRUB running or anything. You can just use the "load" command in
>>> the shell to load the driver, and then access the file system.
>>>
>>> I believe you should be able to download any of the ext binary drivers
>>> mentioned and use them in your shell right away to access your file
>>> system from it.
>>
>> Yes, but it sounds like Marcin may want to embed the ext4fs driver in a
>> custom EFI build. AFAIK, none of the drivers in question were designed
>> with that in mind; however, the VirtualBox project has incorporated
>> ISO-9660 and HFS+ drivers, both of which are built on the same framework
>> (rEFIt's) as rEFInd's drivers, into its own firmware. Thus, Marcin might
>> be able to look at the VirtualBox code and use whatever techniques or
>> glue it uses to incorporate something else. (I can't point to specific
>> files, though.) The rEFInd drivers might be easiest to build in this
>> way, but that's just a guess.
>
> Any valid EFI binary can be built into edk2 platforms easily. As an example, I recommend looking at
>
>   EdkShellBinPkg/FullShell/FullShell.inf
>
> and the following portions from OvmfPkg/OvmfPkgX64.fdf:
>
> INF  RuleOverride = BINARY EdkShellBinPkg/FullShell/FullShell.inf
> ...
>
> [Rule.Common.UEFI_DRIVER.BINARY]
>   FILE DRIVER = $(NAMED_GUID) {
>     DXE_DEPEX DXE_DEPEX Optional      |.depex
>     PE32      PE32                    |.efi
>     UI        STRING="$(MODULE_NAME)" Optional
>     VERSION   STRING="$(INF_VERSION)" Optional BUILD_NUM=$(BUILD_NUMBER)
>   }
> [Rule.Common.UEFI_APPLICATION.BINARY]
>   FILE APPLICATION = $(NAMED_GUID) {
>     PE32      PE32                    |.efi
>     UI        STRING="$(MODULE_NAME)" Optional
>     VERSION   STRING="$(INF_VERSION)" Optional BUILD_NUM=$(BUILD_NUMBER)
>   }
>
>>
>> Note, however, that all of the drivers referenced so far in this thread
>> are licensed under the GPL. Thus, building an EFI in this way would
>> cause the EFI as a whole to be GPL-licensed.
>
> Yes.
>
> Also, as far as I'm aware, OpenSSL and GPL don't mix, so a firmware binary that combined edk2's OpenSSL-based Secure Boot stack and GPL drivers might be undistributable. (It would be fine for in-house use.)
>
> https://people.gnome.org/~markmc/openssl-and-the-gpl.html
>
> There's another option (and while it requires initial setup on the end-user's side, it doesn't require constant fiddling): copy the stand-alone, GPL-licensed UEFI_DRIVER binary to a small, separate VFAT filesystem, then create a Driver#### option that points to it.
>
> (At least with edk2 and OVMF, the driver options are dispatched between BeforeConsole and AfterConsole, and OVMF's AfterConsole calls EfiBootManagerConnectAll(). The result is that filesystems recognized by the drivers loaded from the small VFAT partition would all be bound, and would become available for booting off of.)
>
> This VFAT + Driver#### setup could even be done by customized OS installers, without user interaction.
>
> Now I understand that the point might be to eliminate a VFAT partition altogether, but after the initial driver installation, it could become read-only from the OS runtime POV as well, and the real boot loader stuff could reside on ext2.
>
> Separately, a small note on ext4 (because you mention it above). I seem to recall a filesystem expert colleague of mine advise *against* using journaled filesystems for booting with e.g. grub2. The argument goes (if I recall right), XFS is considered to be in clean state if the data has made it to the final location *or* the persistent journal. When you cleanly unmount (or remount r/o) and shut down, the journal will be flushed to the final location, so a boot loader that doesn't know about the journal will read consistent data. However, if you crash *without* data loss, then part of the data might be in the journal only, and only clients that can read the journal will see consistent data.
>
>> This might or might not be
>> an issue, depending on what the point of the exercise is.
>>
>> Of course, a standalone driver might be perfectly acceptable, too. I've
>> seen options in some EFIs to load drivers at start time, but I've never
>> gotten them to work. (I haven't tried very hard.) If nothing else, a
>> small driver-loading program could be written and set as the first boot
>> option.
>
> Newer firmware might also support SysPrep#### options for such applications (although firmware that supports SysPrep#### would arguably get Driver#### right as well).
>
> Invoking the same logic / app from a regular boot option, and exiting with success, might or might not chain to the next boot option automatically. For example, in "MdeModulePkg/Universal/BdsDxe/BdsEntry.c", edk2 has
>
>     //
>     // If the boot via Boot#### returns with a status of EFI_SUCCESS, platform firmware
>     // supports boot manager menu, and if firmware is configured to boot in an
>     // interactive mode, the boot manager will stop processing the BootOrder variable and
>     // present a boot manager menu to the user.
>     //
>
> I guess invariably exiting with failure could ensure chaining.
>
>>
>> Marcin wrote:
>>
>>> I also found this:
>>> https://sourceforge.net/p/cloverefiboot/code/HEAD/tree/FileSystems/VBoxFsDxe/
>>
>> FWIW, that's a fork of the rEFInd code. I'd not seen it before now; the
>> Clover developers haven't bothered to upstream their changes. (I
>> maintain rEFInd.)
>>
>> It's still not clear to me why you want this driver, Marcin. If you want
>> to load a Linux kernel directly, without using GRUB, rEFInd, or some
>> other tool, you can simply put the kernel(s) on a FAT filesystem.
>
> Yes, if the kernel was built with the EFI stub, it can be launched as a UEFI application.
>
> Furthermore, arm and arm64 kernels parse the "initrd=..." command line parameter in the EFI stub, and can load the initrd from the same filesystem (using the UEFI drivers) where the kernel image was loaded from. The simplest (maybe only?) use case is to provide just a filename (no pathname separators) with "initrd=", and then the file will be loaded from the same directory as the kernel, IIRC.
>
> This enables a development style where one
> - sets up one boot option, pointing to the kernel image (with the EFI stub) as the binary to launch,
> - stores "initrd=..." in the boot option's LoadOptions field,
> - boots the system, modifies and rebuilds the kernel,
> - installs the kernel and the initrd in the exact same spot as before (under the same pathnames on the ESP),
> - reboots.
>
>> This
>> is a common approach among Arch Linux users; they mount the ESP at /boot
>> and it works pretty well. Some distributions assume that /boot supports
>> links or other features that aren't available with FAT, though, so maybe
>> this wouldn't work as well for you, but it's worth considering.
>>
>
> Thanks
> Laszlo

Re: EXT FS support

[Laszlo Ersek]

On 11/23/16 21:01, Michael Zimmermann wrote:
> Laszlo,
> 
> can you explain the problems with putting GPL code in the fdf?

No. I'm not a lawyer. I just repeated what I heard / read elsewhere, and
provided a link. Personally I wouldn't try distributing such
combinations, just to be on the safe side. For work, I would contact RH
Legal if I wanted to to this, and follow their instructions.

> afaik every efi file is like a small self-contained application and
> shouldn't the UEFI API's be excluded from derived work like the linux
> kernel's system call table?

No clue. :)

Thanks
Laszlo

> On Wed, Nov 23, 2016 at 8:50 PM, Laszlo Ersek <lersek@redhat.com> wrote:
>> On 11/22/16 22:31, Rod Smith wrote:
>>> On 11/22/2016 01:25 PM, Pete Batard wrote:
>>>> On 2016.11.22 17:16, Marcin Wojtas wrote:
>>>>> There's no GRUB, platform simply boots to the Shell.
>>>>
>>>> I think there may be some confusion.
>>>>
>>>> All the drivers we linked you to are pure UEFI drivers. It doesn't
>>>> matter if they were a port from GRUB or something else, the code was
>>>> converted to work as a standalone UEFI driver. Especially you don't need
>>>> to have GRUB running or anything. You can just use the "load" command in
>>>> the shell to load the driver, and then access the file system.
>>>>
>>>> I believe you should be able to download any of the ext binary drivers
>>>> mentioned and use them in your shell right away to access your file
>>>> system from it.
>>>
>>> Yes, but it sounds like Marcin may want to embed the ext4fs driver in a
>>> custom EFI build. AFAIK, none of the drivers in question were designed
>>> with that in mind; however, the VirtualBox project has incorporated
>>> ISO-9660 and HFS+ drivers, both of which are built on the same framework
>>> (rEFIt's) as rEFInd's drivers, into its own firmware. Thus, Marcin might
>>> be able to look at the VirtualBox code and use whatever techniques or
>>> glue it uses to incorporate something else. (I can't point to specific
>>> files, though.) The rEFInd drivers might be easiest to build in this
>>> way, but that's just a guess.
>>
>> Any valid EFI binary can be built into edk2 platforms easily. As an example, I recommend looking at
>>
>>   EdkShellBinPkg/FullShell/FullShell.inf
>>
>> and the following portions from OvmfPkg/OvmfPkgX64.fdf:
>>
>> INF  RuleOverride = BINARY EdkShellBinPkg/FullShell/FullShell.inf
>> ...
>>
>> [Rule.Common.UEFI_DRIVER.BINARY]
>>   FILE DRIVER = $(NAMED_GUID) {
>>     DXE_DEPEX DXE_DEPEX Optional      |.depex
>>     PE32      PE32                    |.efi
>>     UI        STRING="$(MODULE_NAME)" Optional
>>     VERSION   STRING="$(INF_VERSION)" Optional BUILD_NUM=$(BUILD_NUMBER)
>>   }
>> [Rule.Common.UEFI_APPLICATION.BINARY]
>>   FILE APPLICATION = $(NAMED_GUID) {
>>     PE32      PE32                    |.efi
>>     UI        STRING="$(MODULE_NAME)" Optional
>>     VERSION   STRING="$(INF_VERSION)" Optional BUILD_NUM=$(BUILD_NUMBER)
>>   }
>>
>>>
>>> Note, however, that all of the drivers referenced so far in this thread
>>> are licensed under the GPL. Thus, building an EFI in this way would
>>> cause the EFI as a whole to be GPL-licensed.
>>
>> Yes.
>>
>> Also, as far as I'm aware, OpenSSL and GPL don't mix, so a firmware binary that combined edk2's OpenSSL-based Secure Boot stack and GPL drivers might be undistributable. (It would be fine for in-house use.)
>>
>> https://people.gnome.org/~markmc/openssl-and-the-gpl.html
>>
>> There's another option (and while it requires initial setup on the end-user's side, it doesn't require constant fiddling): copy the stand-alone, GPL-licensed UEFI_DRIVER binary to a small, separate VFAT filesystem, then create a Driver#### option that points to it.
>>
>> (At least with edk2 and OVMF, the driver options are dispatched between BeforeConsole and AfterConsole, and OVMF's AfterConsole calls EfiBootManagerConnectAll(). The result is that filesystems recognized by the drivers loaded from the small VFAT partition would all be bound, and would become available for booting off of.)
>>
>> This VFAT + Driver#### setup could even be done by customized OS installers, without user interaction.
>>
>> Now I understand that the point might be to eliminate a VFAT partition altogether, but after the initial driver installation, it could become read-only from the OS runtime POV as well, and the real boot loader stuff could reside on ext2.
>>
>> Separately, a small note on ext4 (because you mention it above). I seem to recall a filesystem expert colleague of mine advise *against* using journaled filesystems for booting with e.g. grub2. The argument goes (if I recall right), XFS is considered to be in clean state if the data has made it to the final location *or* the persistent journal. When you cleanly unmount (or remount r/o) and shut down, the journal will be flushed to the final location, so a boot loader that doesn't know about the journal will read consistent data. However, if you crash *without* data loss, then part of the data might be in the journal only, and only clients that can read the journal will see consistent data.
>>
>>> This might or might not be
>>> an issue, depending on what the point of the exercise is.
>>>
>>> Of course, a standalone driver might be perfectly acceptable, too. I've
>>> seen options in some EFIs to load drivers at start time, but I've never
>>> gotten them to work. (I haven't tried very hard.) If nothing else, a
>>> small driver-loading program could be written and set as the first boot
>>> option.
>>
>> Newer firmware might also support SysPrep#### options for such applications (although firmware that supports SysPrep#### would arguably get Driver#### right as well).
>>
>> Invoking the same logic / app from a regular boot option, and exiting with success, might or might not chain to the next boot option automatically. For example, in "MdeModulePkg/Universal/BdsDxe/BdsEntry.c", edk2 has
>>
>>     //
>>     // If the boot via Boot#### returns with a status of EFI_SUCCESS, platform firmware
>>     // supports boot manager menu, and if firmware is configured to boot in an
>>     // interactive mode, the boot manager will stop processing the BootOrder variable and
>>     // present a boot manager menu to the user.
>>     //
>>
>> I guess invariably exiting with failure could ensure chaining.
>>
>>>
>>> Marcin wrote:
>>>
>>>> I also found this:
>>>> https://sourceforge.net/p/cloverefiboot/code/HEAD/tree/FileSystems/VBoxFsDxe/
>>>
>>> FWIW, that's a fork of the rEFInd code. I'd not seen it before now; the
>>> Clover developers haven't bothered to upstream their changes. (I
>>> maintain rEFInd.)
>>>
>>> It's still not clear to me why you want this driver, Marcin. If you want
>>> to load a Linux kernel directly, without using GRUB, rEFInd, or some
>>> other tool, you can simply put the kernel(s) on a FAT filesystem.
>>
>> Yes, if the kernel was built with the EFI stub, it can be launched as a UEFI application.
>>
>> Furthermore, arm and arm64 kernels parse the "initrd=..." command line parameter in the EFI stub, and can load the initrd from the same filesystem (using the UEFI drivers) where the kernel image was loaded from. The simplest (maybe only?) use case is to provide just a filename (no pathname separators) with "initrd=", and then the file will be loaded from the same directory as the kernel, IIRC.
>>
>> This enables a development style where one
>> - sets up one boot option, pointing to the kernel image (with the EFI stub) as the binary to launch,
>> - stores "initrd=..." in the boot option's LoadOptions field,
>> - boots the system, modifies and rebuilds the kernel,
>> - installs the kernel and the initrd in the exact same spot as before (under the same pathnames on the ESP),
>> - reboots.
>>
>>> This
>>> is a common approach among Arch Linux users; they mount the ESP at /boot
>>> and it works pretty well. Some distributions assume that /boot supports
>>> links or other features that aren't available with FAT, though, so maybe
>>> this wouldn't work as well for you, but it's worth considering.
>>>
>>
>> Thanks
>> Laszlo

Re: EXT FS support

[Rebecca Cran]

On 11/23/16 1:11 PM, Laszlo Ersek wrote:

>>> Separately, a small note on ext4 (because you mention it above). I seem to recall a filesystem expert colleague of mine advise *against* using journaled filesystems for booting with e.g. grub2. The argument goes (if I recall right), XFS is considered to be in clean state if the data has made it to the final location *or* the persistent journal. When you cleanly unmount (or remount r/o) and shut down, the journal will be flushed to the final location, so a boot loader that doesn't know about the journal will read consistent data. However, if you crash *without* data loss, then part of the data might be in the journal only, and only clients that can read the journal will see consistent data.
>>>
>>>> This might or might not be
>>>> an issue, depending on what the point of the exercise is.


The problem of course is that "reset" (reboot), "reset -s" (shutdown) 
etc. don't have any hooks a driver can use to flush data, so there will 
sometimes (depending on how long ago data was written) be an unclean 
unmount. At least when booting an OS there's an ExitBootServices event 
available to make sure any data is flushed before boot services ends.

Though I've seen one person talking about how there at least used to be 
a bug that means ExitBootServices isn't/wasn't called for one OS.

-- 
Rebecca

Re: EXT FS support

[Laszlo Ersek]

On 11/24/16 02:03, Rebecca Cran wrote:
> On 11/23/16 1:11 PM, Laszlo Ersek wrote:
> 
>>>> Separately, a small note on ext4 (because you mention it above). I
>>>> seem to recall a filesystem expert colleague of mine advise
>>>> *against* using journaled filesystems for booting with e.g. grub2.
>>>> The argument goes (if I recall right), XFS is considered to be in
>>>> clean state if the data has made it to the final location *or* the
>>>> persistent journal. When you cleanly unmount (or remount r/o) and
>>>> shut down, the journal will be flushed to the final location, so a
>>>> boot loader that doesn't know about the journal will read consistent
>>>> data. However, if you crash *without* data loss, then part of the
>>>> data might be in the journal only, and only clients that can read
>>>> the journal will see consistent data.
>>>>
>>>>> This might or might not be
>>>>> an issue, depending on what the point of the exercise is.
> 
> 
> The problem of course is that "reset" (reboot), "reset -s" (shutdown)
> etc. don't have any hooks a driver can use to flush data, so there will
> sometimes (depending on how long ago data was written) be an unclean
> unmount. At least when booting an OS there's an ExitBootServices event
> available to make sure any data is flushed before boot services ends.

Related: https://bugzilla.tianocore.org/show_bug.cgi?id=224

> Though I've seen one person talking about how there at least used to be
> a bug that means ExitBootServices isn't/wasn't called for one OS.
> 

Re: EXT FS support

[Carsey, Jaben]

> -----Original Message-----
> From: edk2-devel [mailto:edk2-devel-bounces@lists.01.org] On Behalf Of
> Laszlo Ersek
> Sent: Wednesday, November 23, 2016 11:50 AM
> To: Rod Smith <rodsmith@rodsbooks.com>
> Cc: Michael Zimmermann <sigmaepsilon92@gmail.com>; edk2-
> devel@ml01.01.org
> Subject: Re: [edk2] EXT FS support
> Importance: High
> 
> On 11/22/16 22:31, Rod Smith wrote:
> > On 11/22/2016 01:25 PM, Pete Batard wrote:
> >> On 2016.11.22 17:16, Marcin Wojtas wrote:
> >>> There's no GRUB, platform simply boots to the Shell.
> >>
> >> I think there may be some confusion.
> >>
> >> All the drivers we linked you to are pure UEFI drivers. It doesn't
> >> matter if they were a port from GRUB or something else, the code was
> >> converted to work as a standalone UEFI driver. Especially you don't need
> >> to have GRUB running or anything. You can just use the "load" command
> in
> >> the shell to load the driver, and then access the file system.
> >>
> >> I believe you should be able to download any of the ext binary drivers
> >> mentioned and use them in your shell right away to access your file
> >> system from it.
> >
> > Yes, but it sounds like Marcin may want to embed the ext4fs driver in a
> > custom EFI build. AFAIK, none of the drivers in question were designed
> > with that in mind; however, the VirtualBox project has incorporated
> > ISO-9660 and HFS+ drivers, both of which are built on the same framework
> > (rEFIt's) as rEFInd's drivers, into its own firmware. Thus, Marcin might
> > be able to look at the VirtualBox code and use whatever techniques or
> > glue it uses to incorporate something else. (I can't point to specific
> > files, though.) The rEFInd drivers might be easiest to build in this
> > way, but that's just a guess.
> 
> Any valid EFI binary can be built into edk2 platforms easily. As an example, I
> recommend looking at
> 
>   EdkShellBinPkg/FullShell/FullShell.inf

I would recommend ShellBinPkg over the EdkShellBinPkg.  One is the UEFI Shell Spec compliant, the other is not.

> 
> and the following portions from OvmfPkg/OvmfPkgX64.fdf:
> 
> INF  RuleOverride = BINARY EdkShellBinPkg/FullShell/FullShell.inf
> ...
> 
> [Rule.Common.UEFI_DRIVER.BINARY]
>   FILE DRIVER = $(NAMED_GUID) {
>     DXE_DEPEX DXE_DEPEX Optional      |.depex
>     PE32      PE32                    |.efi
>     UI        STRING="$(MODULE_NAME)" Optional
>     VERSION   STRING="$(INF_VERSION)" Optional
> BUILD_NUM=$(BUILD_NUMBER)
>   }
> [Rule.Common.UEFI_APPLICATION.BINARY]
>   FILE APPLICATION = $(NAMED_GUID) {
>     PE32      PE32                    |.efi
>     UI        STRING="$(MODULE_NAME)" Optional
>     VERSION   STRING="$(INF_VERSION)" Optional
> BUILD_NUM=$(BUILD_NUMBER)
>   }
> 
> >
> > Note, however, that all of the drivers referenced so far in this thread
> > are licensed under the GPL. Thus, building an EFI in this way would
> > cause the EFI as a whole to be GPL-licensed.
> 
> Yes.
> 
> Also, as far as I'm aware, OpenSSL and GPL don't mix, so a firmware binary
> that combined edk2's OpenSSL-based Secure Boot stack and GPL drivers
> might be undistributable. (It would be fine for in-house use.)
> 
> https://people.gnome.org/~markmc/openssl-and-the-gpl.html
> 
> There's another option (and while it requires initial setup on the end-user's
> side, it doesn't require constant fiddling): copy the stand-alone, GPL-licensed
> UEFI_DRIVER binary to a small, separate VFAT filesystem, then create a
> Driver#### option that points to it.
> 
> (At least with edk2 and OVMF, the driver options are dispatched between
> BeforeConsole and AfterConsole, and OVMF's AfterConsole calls
> EfiBootManagerConnectAll(). The result is that filesystems recognized by the
> drivers loaded from the small VFAT partition would all be bound, and would
> become available for booting off of.)
> 
> This VFAT + Driver#### setup could even be done by customized OS
> installers, without user interaction.
> 
> Now I understand that the point might be to eliminate a VFAT partition
> altogether, but after the initial driver installation, it could become read-only
> from the OS runtime POV as well, and the real boot loader stuff could reside
> on ext2.
> 
> Separately, a small note on ext4 (because you mention it above). I seem to
> recall a filesystem expert colleague of mine advise *against* using journaled
> filesystems for booting with e.g. grub2. The argument goes (if I recall right),
> XFS is considered to be in clean state if the data has made it to the final
> location *or* the persistent journal. When you cleanly unmount (or remount
> r/o) and shut down, the journal will be flushed to the final location, so a boot
> loader that doesn't know about the journal will read consistent data.
> However, if you crash *without* data loss, then part of the data might be in
> the journal only, and only clients that can read the journal will see consistent
> data.
> 
> > This might or might not be
> > an issue, depending on what the point of the exercise is.
> >
> > Of course, a standalone driver might be perfectly acceptable, too. I've
> > seen options in some EFIs to load drivers at start time, but I've never
> > gotten them to work. (I haven't tried very hard.) If nothing else, a
> > small driver-loading program could be written and set as the first boot
> > option.
> 
> Newer firmware might also support SysPrep#### options for such
> applications (although firmware that supports SysPrep#### would arguably
> get Driver#### right as well).
> 
> Invoking the same logic / app from a regular boot option, and exiting with
> success, might or might not chain to the next boot option automatically. For
> example, in "MdeModulePkg/Universal/BdsDxe/BdsEntry.c", edk2 has
> 
>     //
>     // If the boot via Boot#### returns with a status of EFI_SUCCESS, platform
> firmware
>     // supports boot manager menu, and if firmware is configured to boot in an
>     // interactive mode, the boot manager will stop processing the BootOrder
> variable and
>     // present a boot manager menu to the user.
>     //
> 
> I guess invariably exiting with failure could ensure chaining.
> 
> >
> > Marcin wrote:
> >
> >> I also found this:
> >>
> https://sourceforge.net/p/cloverefiboot/code/HEAD/tree/FileSystems/VBo
> xFsDxe/
> >
> > FWIW, that's a fork of the rEFInd code. I'd not seen it before now; the
> > Clover developers haven't bothered to upstream their changes. (I
> > maintain rEFInd.)
> >
> > It's still not clear to me why you want this driver, Marcin. If you want
> > to load a Linux kernel directly, without using GRUB, rEFInd, or some
> > other tool, you can simply put the kernel(s) on a FAT filesystem.
> 
> Yes, if the kernel was built with the EFI stub, it can be launched as a UEFI
> application.
> 
> Furthermore, arm and arm64 kernels parse the "initrd=..." command line
> parameter in the EFI stub, and can load the initrd from the same filesystem
> (using the UEFI drivers) where the kernel image was loaded from. The
> simplest (maybe only?) use case is to provide just a filename (no pathname
> separators) with "initrd=", and then the file will be loaded from the same
> directory as the kernel, IIRC.
> 
> This enables a development style where one
> - sets up one boot option, pointing to the kernel image (with the EFI stub) as
> the binary to launch,
> - stores "initrd=..." in the boot option's LoadOptions field,
> - boots the system, modifies and rebuilds the kernel,
> - installs the kernel and the initrd in the exact same spot as before (under the
> same pathnames on the ESP),
> - reboots.
> 
> > This
> > is a common approach among Arch Linux users; they mount the ESP at
> /boot
> > and it works pretty well. Some distributions assume that /boot supports
> > links or other features that aren't available with FAT, though, so maybe
> > this wouldn't work as well for you, but it's worth considering.
> >
> 
> Thanks
> Laszlo
> _______________________________________________
> edk2-devel mailing list
> edk2-devel@lists.01.org
> https://lists.01.org/mailman/listinfo/edk2-devel