[edk2-devel] efi and ext4 and case sensitive file names

[edk2-devel] efi and ext4 and case sensitive file names

[Gerd Hoffmann]

  Hi,

I've noticed that the edk2 ext4 driver does case-insensitive filename
matching.  I know the fat filesystem is case-insensitive, and the uefi
spec describing the fat filesystem also explicitly says it is
case-insensitive.  On a quick scan I can't find anything in the uefi
spec requiring *all* filesystem drivers being case-insensitive though.

So I'm wondering whenever the ext4 driver behavior is correct.  It
certainly is different than the linux kernel's behavior which is (by
default) case-sensitive.

Also note that the linux kernel ext4 driver recently got support for
case-insensitive file names, which must be explicitly enabled for both
filesystems (EXT4_FEATURE_INCOMPAT_CASEFOLD) and directories
(EXT4_CASEFOLD_FL).

On practical terms I've ran into actual problems due to Fedora mounting
the ESP at /boot/efi[1] and UKIs (unified kernel images) should be
placed in EFI/Linux on either ESP or XBOOTLDR partition, which on fedora
translates to /boot/efi/EFI/Linux (ESP) or /boot/EFI/Linux (XBOOTLDR).
So I have both /boot/efi and /boot/EFI ...

take care,
  Gerd

[1] Yes, a bad choice for a number of reasons, but changing that isn't
    easy as this is hard-coded in a unknown number of places :(



-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#109121): https://edk2.groups.io/g/devel/message/109121
Mute This Topic: https://groups.io/mt/101615699/7686176
Group Owner: devel+owner@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [rebecca@openfw.io]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [edk2-devel] efi and ext4 and case sensitive file names

[Laszlo Ersek]

On 9/27/23 14:09, kraxel at redhat.com (Gerd Hoffmann) wrote:
>   Hi,
> 
> I've noticed that the edk2 ext4 driver does case-insensitive filename
> matching.  I know the fat filesystem is case-insensitive, and the uefi
> spec describing the fat filesystem also explicitly says it is
> case-insensitive.  On a quick scan I can't find anything in the uefi
> spec requiring *all* filesystem drivers being case-insensitive though.
> 
> So I'm wondering whenever the ext4 driver behavior is correct.  It
> certainly is different than the linux kernel's behavior which is (by
> default) case-sensitive.
> 
> Also note that the linux kernel ext4 driver recently got support for
> case-insensitive file names, which must be explicitly enabled for both
> filesystems (EXT4_FEATURE_INCOMPAT_CASEFOLD) and directories
> (EXT4_CASEFOLD_FL).
> 
> On practical terms I've ran into actual problems due to Fedora mounting
> the ESP at /boot/efi[1] and UKIs (unified kernel images) should be
> placed in EFI/Linux on either ESP or XBOOTLDR partition, which on fedora
> translates to /boot/efi/EFI/Linux (ESP) or /boot/EFI/Linux (XBOOTLDR).
> So I have both /boot/efi and /boot/EFI ...
> 
> take care,
>   Gerd
> 
> [1] Yes, a bad choice for a number of reasons, but changing that isn't
>     easy as this is hard-coded in a unknown number of places :(

I don't think this is fixable until / unless the underlying filesystem
is genuinely case-insensitive -- which is just a way to say that *all*
consumers of the filesystem must agree on case-insensitive
interpretation of pathnames.

I've encountered this issue from the opposite direction: with virtiofs.
VirtioFsDxe is case-sensitive. If you share a case-sensitive host-side
filesystem through VirtioFsDxe with guest UEFI, then in some cases that
produces unexpected results in UEFI. UEFI apps expect case-insensitive
handling, while Linux apps expect case-sensitive handling. At best this
conflict can be resolved at the deepest level, in the underlying
filesystem's specification.

So my point is, even if you flipped Ext4Pkg to case-sensitive behavior,
you'd still see issues, namely between UEFI apps and Linux apps. Those
would only go away if the underlying ext4 filesystem used both of those
new flags that you mention (at which point Ext4Pkg could be
case-sensitive or -insensitive, it wouldn't matter).

Laszlo



-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#109142): https://edk2.groups.io/g/devel/message/109142
Mute This Topic: https://groups.io/mt/101615699/7686176
Group Owner: devel+owner@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/leave/12367111/7686176/1913456212/xyzzy [rebecca@openfw.io]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [edk2-devel] efi and ext4 and case sensitive file names

[Pedro Falcato]

On Wed, Sep 27, 2023 at 1:09 PM Gerd Hoffmann <kraxel@redhat.com> wrote:
>
>   Hi,
>
> I've noticed that the edk2 ext4 driver does case-insensitive filename
> matching.  I know the fat filesystem is case-insensitive, and the uefi
> spec describing the fat filesystem also explicitly says it is
> case-insensitive.  On a quick scan I can't find anything in the uefi
> spec requiring *all* filesystem drivers being case-insensitive though.
>
> So I'm wondering whenever the ext4 driver behavior is correct.  It
> certainly is different than the linux kernel's behavior which is (by
> default) case-sensitive.

Ok, so this is a funny problem. When I was originally writing it, I
noticed a good few consumers would expect case insensitiveness, so I
ended up needing case insensitive string comparison.

As an example, in my current system:
Boot0001* ARCHLINUX
HD(1,GPT,7e1b2e20-3013-4683-b8ab-bef1f9bfb1c8,0x800,0x32000)/File(\EFI\ARCHLINUX\GRUBX64.EFI)
whereas the GRUB tooling installed itself as:
/boot/efi/EFI/ARCHLINUX/grubx64.efi

I treated some good bits of the filesystem driver as "should remain
similar to FAT" due to compatibility reasons. I really really wouldn't
be surprised if lots of EFI code out there relied on case
insensitiveness or other FAT-ish stuff, instead of UNIX semantics.
(funnily enough, EFI also has the restriction where every filename
must be valid unicode, which is not the case on most UNIX systems out
there, that take the file name as "bag of bytes - '/'")

>
> Also note that the linux kernel ext4 driver recently got support for
> case-insensitive file names, which must be explicitly enabled for both
> filesystems (EXT4_FEATURE_INCOMPAT_CASEFOLD) and directories
> (EXT4_CASEFOLD_FL).

Ugh, this is annoying, they didn't even bother documenting it...
(https://www.kernel.org/doc/html/latest/filesystems/ext4/globals.html#super-incompat)
This leaves me in the awkward spot where figuring the behavior out
would require me to read the fs/ext4 code and thus be legally dubious,
yay :^)

>
> On practical terms I've ran into actual problems due to Fedora mounting
> the ESP at /boot/efi[1] and UKIs (unified kernel images) should be
> placed in EFI/Linux on either ESP or XBOOTLDR partition, which on fedora
> translates to /boot/efi/EFI/Linux (ESP) or /boot/EFI/Linux (XBOOTLDR).
> So I have both /boot/efi and /boot/EFI ...

Oh boy, that seems fun. So Ext4Dxe can only open one of the two dirs, right?
Now that I think of it, there should be fun behavior when doing an EFI
readdir, where you could find two dirents with "different" names but
then opening both will lead to the same dirent being open twice...
yuck

-- 
Pedro


-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#109173): https://edk2.groups.io/g/devel/message/109173
Mute This Topic: https://groups.io/mt/101615699/7686176
Group Owner: devel+owner@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [rebecca@openfw.io]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [edk2-devel] efi and ext4 and case sensitive file names

[Marvin Häuser]

> On Sep 28, 2023, at 19:57, Pedro Falcato <pedro.falcato@gmail.com> wrote:
> 
> On Wed, Sep 27, 2023 at 1:09 PM Gerd Hoffmann <kraxel@redhat.com> wrote:
>> 
>>  Hi,
>> 
>> I've noticed that the edk2 ext4 driver does case-insensitive filename
>> matching.  I know the fat filesystem is case-insensitive, and the uefi
>> spec describing the fat filesystem also explicitly says it is
>> case-insensitive.  On a quick scan I can't find anything in the uefi
>> spec requiring *all* filesystem drivers being case-insensitive though.
>> 
>> So I'm wondering whenever the ext4 driver behavior is correct.  It
>> certainly is different than the linux kernel's behavior which is (by
>> default) case-sensitive.

Maybe when Linux starts adhering the spec for file names (the spec clearly defines e.g. BOOTx64.EFI, while at least some distros/images use bootx64.efi), this can be discussed. :) Let's not break various GRUB setups...

> 
> Ok, so this is a funny problem. When I was originally writing it, I
> noticed a good few consumers would expect case insensitiveness, so I
> ended up needing case insensitive string comparison.
> 
> As an example, in my current system:
> Boot0001* ARCHLINUX
> HD(1,GPT,7e1b2e20-3013-4683-b8ab-bef1f9bfb1c8,0x800,0x32000)/File(\EFI\ARCHLINUX\GRUBX64.EFI)
> whereas the GRUB tooling installed itself as:
> /boot/efi/EFI/ARCHLINUX/grubx64.efi
> 
> I treated some good bits of the filesystem driver as "should remain
> similar to FAT" due to compatibility reasons.

Definitely must keep it that way. I think true user-facing case-sensitivity is mostly a Linux thing, even recent macOS APFS is still only case-preserving (UX-wise). There certainly is no real use-case for UEFI itself beyond things like Linux interoperability.

> I really really wouldn't
> be surprised if lots of EFI code out there relied on case
> insensitiveness or other FAT-ish stuff, instead of UNIX semantics.
> (funnily enough, EFI also has the restriction where every filename
> must be valid unicode, which is not the case on most UNIX systems out
> there, that take the file name as "bag of bytes - '/'")
> 
>> 
>> Also note that the linux kernel ext4 driver recently got support for
>> case-insensitive file names, which must be explicitly enabled for both
>> filesystems (EXT4_FEATURE_INCOMPAT_CASEFOLD) and directories
>> (EXT4_CASEFOLD_FL).
> 
> Ugh, this is annoying, they didn't even bother documenting it...
> (https://www.kernel.org/doc/html/latest/filesystems/ext4/globals.html#super-incompat)
> This leaves me in the awkward spot where figuring the behavior out
> would require me to read the fs/ext4 code and thus be legally dubious,
> yay :^)

All hail GPL...

> 
>> 
>> On practical terms I've ran into actual problems due to Fedora mounting
>> the ESP at /boot/efi[1] and UKIs (unified kernel images) should be
>> placed in EFI/Linux on either ESP or XBOOTLDR partition, which on fedora
>> translates to /boot/efi/EFI/Linux (ESP) or /boot/EFI/Linux (XBOOTLDR).
>> So I have both /boot/efi and /boot/EFI ...
> 
> Oh boy, that seems fun. So Ext4Dxe can only open one of the two dirs, right?
> Now that I think of it, there should be fun behavior when doing an EFI
> readdir, where you could find two dirents with "different" names but
> then opening both will lead to the same dirent being open twice...
> yuck

Yes, but this often is not a concern due to case-preservation. There also isn't much of a "right or wrong", as supporting case-insensitivity here could change the semantics of existing desync'd sensitivity setups. However, case-insensitivity would at least be more predictable.

The only real options are:

1) We assume the host OS understands that UEFI is inherently case-sensitive and doesn't do weird things, then the current behaviour makes sense. It's efficient, doesn't touch unnecessary data, but it's not so predictable.

2) We assume the host OS understands this, but we consider this a risk to security. Then it might make sense to reject FSes that feature this. Obviously Secure Boot and such should be taking care of only invoking trusted binaries, but someone might be trying to invoke ambiguous FS driver behaviour because they are aware of an exploit vector. Nevertheless, this seems unnecessary and overkill, also it doesn't fix the broken UX. It will actually completely break that Linux use-case, which might be a bug or a feature. :) But it's predictable!

3) We assume the host OS does not do its job, but we are being nice anyway. Then it might make sense to have opportunistic case-sensitivity, where equality is preferred, but insensitive-equality is used as a fallback. This fixes the issue when the OS is at least consistent in its insensitivity-violation. However, as Pedro showed an example of a desync'd device path and FS name, there is yet another loophole to (accidentally) abuse the ambiguity. Not even some obviously crazy stuff like ranking by editing distances can save you here. And it's also limited in predictability when there is no exact match.

I don't think any option is particularly nice, but 1) is the easiest. :)

Best regards,
Marvin

> 
> -- 
> Pedro



-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#109190): https://edk2.groups.io/g/devel/message/109190
Mute This Topic: https://groups.io/mt/101615699/7686176
Group Owner: devel+owner@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [rebecca@openfw.io]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [edk2-devel] efi and ext4 and case sensitive file names

[Pedro Falcato]

On Fri, Sep 29, 2023 at 10:47 AM Marvin Häuser <mhaeuser@posteo.de> wrote:
>
>
> > On Sep 28, 2023, at 19:57, Pedro Falcato <pedro.falcato@gmail.com> wrote:
> >
> > On Wed, Sep 27, 2023 at 1:09 PM Gerd Hoffmann <kraxel@redhat.com> wrote:
> >>
> >>  Hi,
> >>
> >> I've noticed that the edk2 ext4 driver does case-insensitive filename
> >> matching.  I know the fat filesystem is case-insensitive, and the uefi
> >> spec describing the fat filesystem also explicitly says it is
> >> case-insensitive.  On a quick scan I can't find anything in the uefi
> >> spec requiring *all* filesystem drivers being case-insensitive though.
> >>
> >> So I'm wondering whenever the ext4 driver behavior is correct.  It
> >> certainly is different than the linux kernel's behavior which is (by
> >> default) case-sensitive.
>
> Maybe when Linux starts adhering the spec for file names (the spec clearly defines e.g. BOOTx64.EFI, while at least some distros/images use bootx64.efi), this can be discussed. :) Let's not break various GRUB setups...

Agreed.
>
> >
> > Ok, so this is a funny problem. When I was originally writing it, I
> > noticed a good few consumers would expect case insensitiveness, so I
> > ended up needing case insensitive string comparison.
> >
> > As an example, in my current system:
> > Boot0001* ARCHLINUX
> > HD(1,GPT,7e1b2e20-3013-4683-b8ab-bef1f9bfb1c8,0x800,0x32000)/File(\EFI\ARCHLINUX\GRUBX64.EFI)
> > whereas the GRUB tooling installed itself as:
> > /boot/efi/EFI/ARCHLINUX/grubx64.efi
> >
> > I treated some good bits of the filesystem driver as "should remain
> > similar to FAT" due to compatibility reasons.
>
> Definitely must keep it that way. I think true user-facing case-sensitivity is mostly a Linux thing, even recent macOS APFS is still only case-preserving (UX-wise). There certainly is no real use-case for UEFI itself beyond things like Linux interoperability.
>
> > I really really wouldn't
> > be surprised if lots of EFI code out there relied on case
> > insensitiveness or other FAT-ish stuff, instead of UNIX semantics.
> > (funnily enough, EFI also has the restriction where every filename
> > must be valid unicode, which is not the case on most UNIX systems out
> > there, that take the file name as "bag of bytes - '/'")
> >
> >>
> >> Also note that the linux kernel ext4 driver recently got support for
> >> case-insensitive file names, which must be explicitly enabled for both
> >> filesystems (EXT4_FEATURE_INCOMPAT_CASEFOLD) and directories
> >> (EXT4_CASEFOLD_FL).
> >
> > Ugh, this is annoying, they didn't even bother documenting it...
> > (https://www.kernel.org/doc/html/latest/filesystems/ext4/globals.html#super-incompat)
> > This leaves me in the awkward spot where figuring the behavior out
> > would require me to read the fs/ext4 code and thus be legally dubious,
> > yay :^)
>
> All hail GPL...
>
> >
> >>
> >> On practical terms I've ran into actual problems due to Fedora mounting
> >> the ESP at /boot/efi[1] and UKIs (unified kernel images) should be
> >> placed in EFI/Linux on either ESP or XBOOTLDR partition, which on fedora
> >> translates to /boot/efi/EFI/Linux (ESP) or /boot/EFI/Linux (XBOOTLDR).
> >> So I have both /boot/efi and /boot/EFI ...
> >
> > Oh boy, that seems fun. So Ext4Dxe can only open one of the two dirs, right?
> > Now that I think of it, there should be fun behavior when doing an EFI
> > readdir, where you could find two dirents with "different" names but
> > then opening both will lead to the same dirent being open twice...
> > yuck
>
> Yes, but this often is not a concern due to case-preservation. There also isn't much of a "right or wrong", as supporting case-insensitivity here could change the semantics of existing desync'd sensitivity setups. However, case-insensitivity would at least be more predictable.
>
> The only real options are:
>
> 1) We assume the host OS understands that UEFI is inherently case-sensitive and doesn't do weird things, then the current behaviour makes sense. It's efficient, doesn't touch unnecessary data, but it's not so predictable.

Status quo, kind of works but breaks in Gerd's use case.
>
> 2) We assume the host OS understands this, but we consider this a risk to security. Then it might make sense to reject FSes that feature this. Obviously Secure Boot and such should be taking care of only invoking trusted binaries, but someone might be trying to invoke ambiguous FS driver behaviour because they are aware of an exploit vector. Nevertheless, this seems unnecessary and overkill, also it doesn't fix the broken UX. It will actually completely break that Linux use-case, which might be a bug or a feature. :) But it's predictable!

Probably not a way forward, yeah.
>
> 3) We assume the host OS does not do its job, but we are being nice anyway. Then it might make sense to have opportunistic case-sensitivity, where equality is preferred, but insensitive-equality is used as a fallback. This fixes the issue when the OS is at least consistent in its insensitivity-violation. However, as Pedro showed an example of a desync'd device path and FS name, there is yet another loophole to (accidentally) abuse the ambiguity. Not even some obviously crazy stuff like ranking by editing distances can save you here. And it's also limited in predictability when there is no exact match.
>

I'm relatively ok with this, aka doing an exact search first and
*only* then doing a case insensitive search (if CASEFOLD_FL, we would
skip the exact search). It should solve all compatibility problems
with UNIXes, although at the moment I'm wary of any possible security
concerns.

> I don't think any option is particularly nice, but 1) is the easiest. :)

1) is the easiest, but it also has weird edge cases.
3) has the nice property of being compatible with both
EFI/windows-style users (that expect insensitivity) and UNIX-style
users (that don't expect it), but it also has weird edge cases (true
insensitive behavior should make sure sHell.EFI and Shell.efi both
resolve to the same file in every situation).

Pick your poison :/

-- 
Pedro


-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#109193): https://edk2.groups.io/g/devel/message/109193
Mute This Topic: https://groups.io/mt/101615699/7686176
Group Owner: devel+owner@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [rebecca@openfw.io]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [edk2-devel] efi and ext4 and case sensitive file names

[Michael Brown]

On 29/09/2023 10:47, Marvin Häuser wrote:
> Maybe when Linux starts adhering the spec for file names (the spec clearly defines e.g. BOOTx64.EFI, while at least some distros/images use bootx64.efi), this can be discussed. :) Let's not break various GRUB setups...

Seems slightly unfair to blame Linux when EDK2 also apparently gets it 
wrong (in a different way)... :)

//
// EFI File location to boot from on removable media devices
//
...
#define EFI_REMOVABLE_MEDIA_FILE_NAME_X64 L"\\EFI\\BOOT\\BOOTX64.EFI"

Is this something worth fixing in UefiSpec.h?

Thanks,

Michael



-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#109194): https://edk2.groups.io/g/devel/message/109194
Mute This Topic: https://groups.io/mt/101615699/7686176
Group Owner: devel+owner@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [rebecca@openfw.io]
-=-=-=-=-=-=-=-=-=-=-=-

Re: [edk2-devel] efi and ext4 and case sensitive file names

[Marvin Häuser]

> On Sep 29, 2023, at 12:58, Michael Brown <mcb30@ipxe.org> wrote:
> 
> On 29/09/2023 10:47, Marvin Häuser wrote:
>> Maybe when Linux starts adhering the spec for file names (the spec clearly defines e.g. BOOTx64.EFI, while at least some distros/images use bootx64.efi), this can be discussed. :) Let's not break various GRUB setups...
> 
> Seems slightly unfair to blame Linux when EDK2 also apparently gets it wrong (in a different way)... :)

I’m perfectly happy with any blame cast on Linux and EDK II, preferably both. :) Seems like this is my lucky day…

> 
> //
> // EFI File location to boot from on removable media devices
> //
> ...
> #define EFI_REMOVABLE_MEDIA_FILE_NAME_X64 L"\\EFI\\BOOT\\BOOTX64.EFI"
> 
> Is this something worth fixing in UefiSpec.h?

At this point I’m never sure whether to fix EDK II or fix the spec… I wouldn’t make any sudden moves without checking OS behaviours.

Best regards,
Marvin

> 
> Thanks,
> 
> Michael
> 



-=-=-=-=-=-=-=-=-=-=-=-
Groups.io Links: You receive all messages sent to this group.
View/Reply Online (#109195): https://edk2.groups.io/g/devel/message/109195
Mute This Topic: https://groups.io/mt/101615699/7686176
Group Owner: devel+owner@edk2.groups.io
Unsubscribe: https://edk2.groups.io/g/devel/unsub [rebecca@openfw.io]
-=-=-=-=-=-=-=-=-=-=-=-