Side effects of enabling PML5 in EFI

Side effects of enabling PML5 in EFI

[Pedro Falcato]

Hi all,

(+CC people vaguely related to the EFI spec, the PML5 implementation
and kernel EFI boot code)

As a result of the latest 5-level paging patches, I've been looking
into how tiano supports PML5.
This raised a question: Doesn't enabling PML5 in-firmware break
compatibility with non-PML5-aware bootloaders and kernels?

>From an architectural point of view:
- PML5 is enabled in CR4.LA57, but may only be toggled when not in
IA32e mode (so, only in 32-bit)
- Trying to mindlessly write to CR4 will #GP, and loading a 4-level
page tables will crash with probable page faults or #GPs

>From an EFI spec point of view:
- Whereas other architectures (arm64 for instance) specify the MMU
state in detail, the x64 bits do not specify anything beyond "Paging
enabled" (see 2.3.4). Which pre-PML5, was obviously well defined.
- When under boot services, this is likely not a problem as page
tables are owned by boot services. Unless they touch them as defined
in "2.3.4.3. Enabling Paging or Alternate Translations in an
Application", which may run into problems.

>From an OS kernel/bootloader point of view:
- A PML5 aware kernel/bootloader will likely correctly identify the
PML5 capability and enable LA57, load 5-level page tables. As such,
this scenario always works.
- A non-PML5-aware one may incorrectly overwrite LA57 (and #GP), or
just load a 4-level paging structure into CR3, and thus disastrously
crash.

So, how is any of this supposed to work?

-- 
Pedro

Re: [edk2-devel] Side effects of enabling PML5 in EFI

[Gerd Hoffmann]

On Tue, May 09, 2023 at 06:24:03PM +0100, Pedro Falcato wrote:
> Hi all,
> 
> (+CC people vaguely related to the EFI spec, the PML5 implementation
> and kernel EFI boot code)
> 
> As a result of the latest 5-level paging patches, I've been looking
> into how tiano supports PML5.
> This raised a question: Doesn't enabling PML5 in-firmware break
> compatibility with non-PML5-aware bootloaders and kernels?

kernels yes, bootloaders probably not (unless the bootloader not the
kernel calls exitbootservices).

> So, how is any of this supposed to work?

edk2 is relatively late to the party, so with everybody else supporting
5-level paging already it might not be much of an issue in practice even
though there is no compatibility handling (I know of) in edk2.

When running your operating system museum in virtual machines you have
the option to just turn off 5-level paging support in the virtual cpu.

take care,
  Gerd

Re: [edk2-devel] Side effects of enabling PML5 in EFI

[Ni, Ray]

Firmware chooses to use 5-level paging when the platform using this firmware claims to boot 5-level paging OS only.

Usually, firmware uses 4-level paging to keep maximum OS compability.

Thanks,
Ray

> -----Original Message-----
> From: Gerd Hoffmann <kraxel@redhat.com>
> Sent: Wednesday, May 10, 2023 5:18 PM
> To: devel@edk2.groups.io; pedro.falcato@gmail.com
> Cc: Andrew Fish <afish@apple.com>; Kinney, Michael D
> <michael.d.kinney@intel.com>; Ni, Ray <ray.ni@intel.com>; Ard Biesheuvel
> <ardb@kernel.org>
> Subject: Re: [edk2-devel] Side effects of enabling PML5 in EFI
> 
> On Tue, May 09, 2023 at 06:24:03PM +0100, Pedro Falcato wrote:
> > Hi all,
> >
> > (+CC people vaguely related to the EFI spec, the PML5 implementation
> > and kernel EFI boot code)
> >
> > As a result of the latest 5-level paging patches, I've been looking
> > into how tiano supports PML5.
> > This raised a question: Doesn't enabling PML5 in-firmware break
> > compatibility with non-PML5-aware bootloaders and kernels?
> 
> kernels yes, bootloaders probably not (unless the bootloader not the
> kernel calls exitbootservices).
> 
> > So, how is any of this supposed to work?
> 
> edk2 is relatively late to the party, so with everybody else supporting
> 5-level paging already it might not be much of an issue in practice even
> though there is no compatibility handling (I know of) in edk2.
> 
> When running your operating system museum in virtual machines you have
> the option to just turn off 5-level paging support in the virtual cpu.
> 
> take care,
>   Gerd

Re: Side effects of enabling PML5 in EFI

[Ard Biesheuvel]

On Tue, 9 May 2023 at 19:24, Pedro Falcato <pedro.falcato@gmail.com> wrote:
>
> Hi all,
>
> (+CC people vaguely related to the EFI spec, the PML5 implementation
> and kernel EFI boot code)
>
> As a result of the latest 5-level paging patches, I've been looking
> into how tiano supports PML5.
> This raised a question: Doesn't enabling PML5 in-firmware break
> compatibility with non-PML5-aware bootloaders and kernels?
>
> From an architectural point of view:
> - PML5 is enabled in CR4.LA57, but may only be toggled when not in
> IA32e mode (so, only in 32-bit)
> - Trying to mindlessly write to CR4 will #GP, and loading a 4-level
> page tables will crash with probable page faults or #GPs
>
> From an EFI spec point of view:
> - Whereas other architectures (arm64 for instance) specify the MMU
> state in detail, the x64 bits do not specify anything beyond "Paging
> enabled" (see 2.3.4). Which pre-PML5, was obviously well defined.

We actually have a related problem on ARM: the size of the virtual
address space is not mandated by the spec, but it does require that
all memory is mapped 1:1.

This means that, if a system has any memory that is outside of the
48-bit physical range, it must enable 5 level paging to map it 1:1 in
the 52-bit virtual range.

Given that EDK2's page allocator allocates from top down from the end
of the address space, we might end up with allocations for ACPI tables
etc that cannot be mapped by kernels that do not implement support for
5 level paging.

I imagine a similar issue might exist on x86 as well, and this
suggests that using 5 level paging in the firmware is only sensible if
it is guaranteed that the OS and loader can deal with it (IOW, running
the firmware with 5 level pages and switching back to 4 in
ExitBootServices() may result in other issues)

> - When under boot services, this is likely not a problem as page
> tables are owned by boot services. Unless they touch them as defined
> in "2.3.4.3. Enabling Paging or Alternate Translations in an
> Application", which may run into problems.
>
> From an OS kernel/bootloader point of view:
> - A PML5 aware kernel/bootloader will likely correctly identify the
> PML5 capability and enable LA57, load 5-level page tables. As such,
> this scenario always works.
> - A non-PML5-aware one may incorrectly overwrite LA57 (and #GP), or
> just load a 4-level paging structure into CR3, and thus disastrously
> crash.
>
> So, how is any of this supposed to work?
>

I don't think the firmware should ever use 5 level paging unless it is
strictly needed for a particular use case. And even then, it should
avoid allocating memory from the region that is only 1:1 accessible
when 5 level paging is enabled.

Re: [edk2-devel] Side effects of enabling PML5 in EFI

[Pedro Falcato]

On Wed, May 10, 2023 at 10:18 AM Gerd Hoffmann <kraxel@redhat.com> wrote:
>
> On Tue, May 09, 2023 at 06:24:03PM +0100, Pedro Falcato wrote:
> > Hi all,
> >
> > (+CC people vaguely related to the EFI spec, the PML5 implementation
> > and kernel EFI boot code)
> >
> > As a result of the latest 5-level paging patches, I've been looking
> > into how tiano supports PML5.
> > This raised a question: Doesn't enabling PML5 in-firmware break
> > compatibility with non-PML5-aware bootloaders and kernels?
>
> kernels yes, bootloaders probably not (unless the bootloader not the
> kernel calls exitbootservices).

Bootloaders can theoretically enable "alternative translations" as
they call it in-spec. No need to EBS() to use your own page tables.
I have no idea how prevalent this is, but if someone told me GRUB does
this I would be 0% shocked.

> > So, how is any of this supposed to work?
>
> edk2 is relatively late to the party, so with everybody else supporting
> 5-level paging already it might not be much of an issue in practice even
> though there is no compatibility handling (I know of) in edk2.
>
> When running your operating system museum in virtual machines you have
> the option to just turn off 5-level paging support in the virtual cpu.

Ok, went looking. Linux supports this since ~2017. Darwin (from macOS)
does not support this.
FreeBSD supports LA57 from 2020 onwards. NetBSD does not support it.
Fuchsia does not support it.

And, more glaringly, Windows 11 does not support it (no source, went
testing with qemu -cpu +la57).

So, this is far from an operating system museum. I don't think this
can be safely enabled at all. And especially not,
when you're not running under a VM (presumably, this will get deployed
on real hardware).

-- 
Pedro

Re: [edk2-devel] Side effects of enabling PML5 in EFI

[Pedro Falcato]

On Wed, May 10, 2023 at 10:34 AM Ni, Ray <ray.ni@intel.com> wrote:
>
> Firmware chooses to use 5-level paging when the platform using this firmware claims to boot 5-level paging OS only.
>
> Usually, firmware uses 4-level paging to keep maximum OS compability.

Hi Ray,

So, what happens if I don't enable LA57, have a gazillion TB of memory
(such that I go over the 128TB 47-bit AS limit). Will EFI never try to
access memory up there and page fault?

What happens to the OS/bootloader? If it asks for memory up there (for
KASLR, etc)? Does the memory map pre-reserve those upper regions of
memory that are not accessible using 4-level paging?

I really fail to see the advantages of PML5 support in EFI fw at the
moment, particularly as you can simply choose to start all your
allocations lower down the memory map (where you can indeed access
things), reserve the upper, inaccessible bits, and things Should Work?
Unless you're keeping some MMIO ranges up there, in which case, the
solution is probably hard.

-- 
Pedro

Re: Side effects of enabling PML5 in EFI

[Pedro Falcato]

On Wed, May 10, 2023 at 10:41 AM Ard Biesheuvel <ardb@kernel.org> wrote:
>
> On Tue, 9 May 2023 at 19:24, Pedro Falcato <pedro.falcato@gmail.com> wrote:
> >
> > Hi all,
> >
> > (+CC people vaguely related to the EFI spec, the PML5 implementation
> > and kernel EFI boot code)
> >
> > As a result of the latest 5-level paging patches, I've been looking
> > into how tiano supports PML5.
> > This raised a question: Doesn't enabling PML5 in-firmware break
> > compatibility with non-PML5-aware bootloaders and kernels?
> >
> > From an architectural point of view:
> > - PML5 is enabled in CR4.LA57, but may only be toggled when not in
> > IA32e mode (so, only in 32-bit)
> > - Trying to mindlessly write to CR4 will #GP, and loading a 4-level
> > page tables will crash with probable page faults or #GPs
> >
> > From an EFI spec point of view:
> > - Whereas other architectures (arm64 for instance) specify the MMU
> > state in detail, the x64 bits do not specify anything beyond "Paging
> > enabled" (see 2.3.4). Which pre-PML5, was obviously well defined.
>
> We actually have a related problem on ARM: the size of the virtual
> address space is not mandated by the spec, but it does require that
> all memory is mapped 1:1.
>

Yes, your ARM problem sounds similar, but you do have the advantage
that T0SZ (et al) have been there since forever,
while for x86_64 this is just a surprise CR4 bit that you had no idea
existed for the past 20 years.

> This means that, if a system has any memory that is outside of the
> 48-bit physical range, it must enable 5 level paging to map it 1:1 in
> the 52-bit virtual range.
>
> Given that EDK2's page allocator allocates from top down from the end
> of the address space, we might end up with allocations for ACPI tables
> etc that cannot be mapped by kernels that do not implement support for
> 5 level paging.

Do we need to have all the memory mapped? I know the EFI spec demands
identity mapping,
but it does sound a lot like we could reserve everything from
2^(va_size) onwards with some (new or old) EFI memory type and keep
going?
Right? I imagine 128/256 TiB of DRAM will work just fine for EFI firmware...

Basically, EFI allocations and usage would be restricted to commonly
accessible ranges of memory (47-bit for x86, 48 for ARM64, etc...)

>
> I imagine a similar issue might exist on x86 as well, and this
> suggests that using 5 level paging in the firmware is only sensible if
> it is guaranteed that the OS and loader can deal with it (IOW, running
> the firmware with 5 level pages and switching back to 4 in
> ExitBootServices() may result in other issues)
>

Right. Sadly, I don't think you can even pull an EBS() trick here, as
in theory you *can* set up alternate translations, even when under
boot services (see 2.3.4.3).

> > - When under boot services, this is likely not a problem as page
> > tables are owned by boot services. Unless they touch them as defined
> > in "2.3.4.3. Enabling Paging or Alternate Translations in an
> > Application", which may run into problems.
> >
> > From an OS kernel/bootloader point of view:
> > - A PML5 aware kernel/bootloader will likely correctly identify the
> > PML5 capability and enable LA57, load 5-level page tables. As such,
> > this scenario always works.
> > - A non-PML5-aware one may incorrectly overwrite LA57 (and #GP), or
> > just load a 4-level paging structure into CR3, and thus disastrously
> > crash.
> >
> > So, how is any of this supposed to work?
> >
>
> I don't think the firmware should ever use 5 level paging unless it is
> strictly needed for a particular use case. And even then, it should
> avoid allocating memory from the region that is only 1:1 accessible
> when 5 level paging is enabled.

I agree. But I would like it if the more EFI "big whigs" (Mike K,
Andrew, etc) could give a heads up as to what should happen. And then
possibly put wording into the spec.

-- 
Pedro

Re: [edk2-devel] Side effects of enabling PML5 in EFI

[Ni, Ray]

> -----Original Message-----
> From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Pedro
> Falcato
> Sent: Thursday, May 11, 2023 12:39 AM
> To: Ni, Ray <ray.ni@intel.com>
> Cc: Gerd Hoffmann <kraxel@redhat.com>; devel@edk2.groups.io; Andrew Fish
> <afish@apple.com>; Kinney, Michael D <michael.d.kinney@intel.com>; Ard
> Biesheuvel <ardb@kernel.org>
> Subject: Re: [edk2-devel] Side effects of enabling PML5 in EFI
> 
> On Wed, May 10, 2023 at 10:34 AM Ni, Ray <ray.ni@intel.com> wrote:
> >
> > Firmware chooses to use 5-level paging when the platform using this firmware
> claims to boot 5-level paging OS only.
> >
> > Usually, firmware uses 4-level paging to keep maximum OS compability.
> 
> Hi Ray,
> 
> So, what happens if I don't enable LA57, have a gazillion TB of memory
> (such that I go over the 128TB 47-bit AS limit). Will EFI never try to
> access memory up there and page fault?

No. UEFI firmware on x86 doesn't try to create non-1:1 mapping page table to let
firmware code access these high physical addresses.

> 
> What happens to the OS/bootloader? If it asks for memory up there (for
> KASLR, etc)? Does the memory map pre-reserve those upper regions of
> memory that are not accessible using 4-level paging?

That depends on how the EFI_HOB_CPU is produced from the firmware code.
The UEFI memory map data is from GCD database in DxeCore.
GCD database is built based on the physical address size value in EFI_HOB_CPU.

> 
> I really fail to see the advantages of PML5 support in EFI fw at the
> moment, particularly as you can simply choose to start all your
> allocations lower down the memory map (where you can indeed access
> things), reserve the upper, inaccessible bits, and things Should Work?
> Unless you're keeping some MMIO ranges up there, in which case, the
> solution is probably hard.

I agree that 5-level paging is only needed when very high MMIO is needed for
silicon init.

> 
> --
> Pedro
> 
> 
> 
>