Question regarding MMIO address space exposed from GetMemoryMap

Question regarding MMIO address space exposed from GetMemoryMap

[jon]

I am working on sorting out a failure on test 605 of the SBSA test.
The test is "Where a memory access is to an unpopulated part of the
addressable memory space, accesses must be terminated in a manner that
is presented to the PE as either a precise Data Abort or that causes a
system error interrupt or an SPI or LPI interrupt to be delivered to
the GIC."  The issue is that the random test address that was chosen
0x04200000 falls directly in the middle of the Qoriq configuration,
control, and status register (CCSR) address space.  This is an area in
the memory space that provides CPU access to the device registers.

An entry is added for this region in the VirtualMemoryMap, and
registered with the attribute, ARM_MEMORY_REGION_ATTRIBUTE_DEVICE.
However only a handful of devices are being registered so only a
couple of ranges are showing up in memmap as MMIO.

The SBSA test in question gets the memorymap and starts at the address
mentioned above and looks for the first chunk of memory that is free
and then attempts to access the memory and is looking for a Data
Abort.  Of course a data abort is not generated because 0x04200000 is
a valid address.  If you offset this to 0x42000000 then a DA is
generated as expected and the test passes.

There is a very old thread started by Ard, "MMIO regions in
GetMemoryMap ()", in which he questions if all the MMIO regions should
be reported by GetMemoryMap() or only the ones being used by
initialized devices, which is what the current implementation does.
The end result of the thread was the current implementation is
correct.  That leaves me with the question, "What is the proper
solution for the current implementation that I am working with?"

To me it seems like I need a special Library that on boot goes through
and claims and maps every valid MMIO slot in this region, and then
have the drivers use this library for proxying requests to
gDS->AddMemorySpace (), gDS->SetMemorySpaceAttributes () etc.   If
there is a standardized way to deal with this configuration I would
much rather follow that.

Thanks for any input,
Jon

Re: [edk2-devel] Question regarding MMIO address space exposed from GetMemoryMap

[Andrew Fish]

Jon,

Its a little confusing but gBS->GetMemoryMap () only returns information about DRAM and any address that requires a kernel virtual address mapping in EFI. The OS calls EFI Runtime Services from a kernel virtual mapping so the memory map is also involved in the hand shake to communicate the virtual address mappings to EFI. Thus any MMIO region in the EFI Memory Map should always have  the EFI_MEMORY_RUNTIME attribute set. The most common MMIO region to be mapped is the NOR FLASH to support the EFI Runtime Variable services. 

On a UEFI system MMIO regions are described to the OS via ACPI. Processing the ACPI tables requires an interpreter, and source to an interpreter is available at the ACPI CA site [1]. 

The PI (Platform Initialization) Spec has a concept called the Global Coherency Domain (GCD) [2]. The GCD is the map for who owns the CPU address and IO (x86 in/out instructions not MMIO) space. You can dump the GCD info via gDS->GetMemorySpaceMap(). The idea is the DRAM controller would carve out a EfiGcdMemoryTypeSystemMemory range, and the PCI Host Bridge would carve out a EfiGcdMemoryTypeMemoryMappedIo range, etc. 

Note: A UEFI implementation is not required to be constructed out of PI Spec components, but the EDKII UEFI implementation is constructed using the PI Specification. 

[1] https://acpica.org/downloads/uefi-support
[2] Sorry I could not make up a better name at the time. 

Thanks,

Andrew Fish

> On Oct 31, 2019, at 1:32 AM, Jon Nettleton <jon@solid-run.com> wrote:
> 
> I am working on sorting out a failure on test 605 of the SBSA test.
> The test is "Where a memory access is to an unpopulated part of the
> addressable memory space, accesses must be terminated in a manner that
> is presented to the PE as either a precise Data Abort or that causes a
> system error interrupt or an SPI or LPI interrupt to be delivered to
> the GIC."  The issue is that the random test address that was chosen
> 0x04200000 falls directly in the middle of the Qoriq configuration,
> control, and status register (CCSR) address space.  This is an area in
> the memory space that provides CPU access to the device registers.
> 
> An entry is added for this region in the VirtualMemoryMap, and
> registered with the attribute, ARM_MEMORY_REGION_ATTRIBUTE_DEVICE.
> However only a handful of devices are being registered so only a
> couple of ranges are showing up in memmap as MMIO.
> 
> The SBSA test in question gets the memorymap and starts at the address
> mentioned above and looks for the first chunk of memory that is free
> and then attempts to access the memory and is looking for a Data
> Abort.  Of course a data abort is not generated because 0x04200000 is
> a valid address.  If you offset this to 0x42000000 then a DA is
> generated as expected and the test passes.
> 
> There is a very old thread started by Ard, "MMIO regions in
> GetMemoryMap ()", in which he questions if all the MMIO regions should
> be reported by GetMemoryMap() or only the ones being used by
> initialized devices, which is what the current implementation does.
> The end result of the thread was the current implementation is
> correct.  That leaves me with the question, "What is the proper
> solution for the current implementation that I am working with?"
> 
> To me it seems like I need a special Library that on boot goes through
> and claims and maps every valid MMIO slot in this region, and then
> have the drivers use this library for proxying requests to
> gDS->AddMemorySpace (), gDS->SetMemorySpaceAttributes () etc.   If
> there is a standardized way to deal with this configuration I would
> much rather follow that.
> 
> Thanks for any input,
> Jon
> 
> 
> 

Re: [edk2-devel] Question regarding MMIO address space exposed from GetMemoryMap

[Jon Nettleton]

On Thu, Oct 31, 2019 at 5:44 PM Andrew Fish via Groups.Io
<afish=apple.com@groups.io> wrote:
>
> Jon,
>
> Its a little confusing but gBS->GetMemoryMap () only returns information about DRAM and any address that requires a kernel virtual address mapping in EFI. The OS calls EFI Runtime Services from a kernel virtual mapping so the memory map is also involved in the hand shake to communicate the virtual address mappings to EFI. Thus any MMIO region in the EFI Memory Map should always have  the EFI_MEMORY_RUNTIME attribute set. The most common MMIO region to be mapped is the NOR FLASH to support the EFI Runtime Variable services.
>
> On a UEFI system MMIO regions are described to the OS via ACPI. Processing the ACPI tables requires an interpreter, and source to an interpreter is available at the ACPI CA site [1].
>
> The PI (Platform Initialization) Spec has a concept called the Global Coherency Domain (GCD) [2]. The GCD is the map for who owns the CPU address and IO (x86 in/out instructions not MMIO) space. You can dump the GCD info via gDS->GetMemorySpaceMap(). The idea is the DRAM controller would carve out a EfiGcdMemoryTypeSystemMemory range, and the PCI Host Bridge would carve out a EfiGcdMemoryTypeMemoryMappedIo range, etc.
>
> Note: A UEFI implementation is not required to be constructed out of PI Spec components, but the EDKII UEFI implementation is constructed using the PI Specification.
>
> [1] https://acpica.org/downloads/uefi-support
> [2] Sorry I could not make up a better name at the time.
>
> Thanks,
>
> Andrew Fish
>
> > On Oct 31, 2019, at 1:32 AM, Jon Nettleton <jon@solid-run.com> wrote:
> >
> > I am working on sorting out a failure on test 605 of the SBSA test.
> > The test is "Where a memory access is to an unpopulated part of the
> > addressable memory space, accesses must be terminated in a manner that
> > is presented to the PE as either a precise Data Abort or that causes a
> > system error interrupt or an SPI or LPI interrupt to be delivered to
> > the GIC."  The issue is that the random test address that was chosen
> > 0x04200000 falls directly in the middle of the Qoriq configuration,
> > control, and status register (CCSR) address space.  This is an area in
> > the memory space that provides CPU access to the device registers.
> >
> > An entry is added for this region in the VirtualMemoryMap, and
> > registered with the attribute, ARM_MEMORY_REGION_ATTRIBUTE_DEVICE.
> > However only a handful of devices are being registered so only a
> > couple of ranges are showing up in memmap as MMIO.
> >
> > The SBSA test in question gets the memorymap and starts at the address
> > mentioned above and looks for the first chunk of memory that is free
> > and then attempts to access the memory and is looking for a Data
> > Abort.  Of course a data abort is not generated because 0x04200000 is
> > a valid address.  If you offset this to 0x42000000 then a DA is
> > generated as expected and the test passes.
> >
> > There is a very old thread started by Ard, "MMIO regions in
> > GetMemoryMap ()", in which he questions if all the MMIO regions should
> > be reported by GetMemoryMap() or only the ones being used by
> > initialized devices, which is what the current implementation does.
> > The end result of the thread was the current implementation is
> > correct.  That leaves me with the question, "What is the proper
> > solution for the current implementation that I am working with?"
> >
> > To me it seems like I need a special Library that on boot goes through
> > and claims and maps every valid MMIO slot in this region, and then
> > have the drivers use this library for proxying requests to
> > gDS->AddMemorySpace (), gDS->SetMemorySpaceAttributes () etc.   If
> > there is a standardized way to deal with this configuration I would
> > much rather follow that.
> >
> > Thanks for any input,
> > Jon
> >
> >
> >
>
>
> 
>

Andrew,

Thanks for clearing this up.  I definitely understand the
relationships much better now.  I am still uncertain of the proper way
to fix this issue regarding the SBSA peripheral test.  The test is
using gBS->GetMemoryMap in order to look for available memory
segments, which according to the earlier thread and your confirmation
"gBS->GetMemoryMap () only returns information about DRAM and any
address that requires a kernel virtual address mapping in EFI.", which
means that this test will never work on this platform because the
address that was randomly chosen just happens to fall right into the
address range for the device addresses that are valid so won't produce
a DA, but aren't being used by EFI therefore don't require a virtual
address mapping in EFI.

Is this something I should bring up with the SBSA group?

-Jon

Re: [edk2-devel] Question regarding MMIO address space exposed from GetMemoryMap

[Jon Nettleton]

On Fri, Nov 1, 2019 at 6:57 AM Jon Nettleton via Groups.Io
<jon=solid-run.com@groups.io> wrote:
>
> On Thu, Oct 31, 2019 at 5:44 PM Andrew Fish via Groups.Io
> <afish=apple.com@groups.io> wrote:
> >
> > Jon,
> >
> > Its a little confusing but gBS->GetMemoryMap () only returns information about DRAM and any address that requires a kernel virtual address mapping in EFI. The OS calls EFI Runtime Services from a kernel virtual mapping so the memory map is also involved in the hand shake to communicate the virtual address mappings to EFI. Thus any MMIO region in the EFI Memory Map should always have  the EFI_MEMORY_RUNTIME attribute set. The most common MMIO region to be mapped is the NOR FLASH to support the EFI Runtime Variable services.
> >
> > On a UEFI system MMIO regions are described to the OS via ACPI. Processing the ACPI tables requires an interpreter, and source to an interpreter is available at the ACPI CA site [1].
> >
> > The PI (Platform Initialization) Spec has a concept called the Global Coherency Domain (GCD) [2]. The GCD is the map for who owns the CPU address and IO (x86 in/out instructions not MMIO) space. You can dump the GCD info via gDS->GetMemorySpaceMap(). The idea is the DRAM controller would carve out a EfiGcdMemoryTypeSystemMemory range, and the PCI Host Bridge would carve out a EfiGcdMemoryTypeMemoryMappedIo range, etc.
> >
> > Note: A UEFI implementation is not required to be constructed out of PI Spec components, but the EDKII UEFI implementation is constructed using the PI Specification.
> >
> > [1] https://acpica.org/downloads/uefi-support
> > [2] Sorry I could not make up a better name at the time.
> >
> > Thanks,
> >
> > Andrew Fish
> >
> > > On Oct 31, 2019, at 1:32 AM, Jon Nettleton <jon@solid-run.com> wrote:
> > >
> > > I am working on sorting out a failure on test 605 of the SBSA test.
> > > The test is "Where a memory access is to an unpopulated part of the
> > > addressable memory space, accesses must be terminated in a manner that
> > > is presented to the PE as either a precise Data Abort or that causes a
> > > system error interrupt or an SPI or LPI interrupt to be delivered to
> > > the GIC."  The issue is that the random test address that was chosen
> > > 0x04200000 falls directly in the middle of the Qoriq configuration,
> > > control, and status register (CCSR) address space.  This is an area in
> > > the memory space that provides CPU access to the device registers.
> > >
> > > An entry is added for this region in the VirtualMemoryMap, and
> > > registered with the attribute, ARM_MEMORY_REGION_ATTRIBUTE_DEVICE.
> > > However only a handful of devices are being registered so only a
> > > couple of ranges are showing up in memmap as MMIO.
> > >
> > > The SBSA test in question gets the memorymap and starts at the address
> > > mentioned above and looks for the first chunk of memory that is free
> > > and then attempts to access the memory and is looking for a Data
> > > Abort.  Of course a data abort is not generated because 0x04200000 is
> > > a valid address.  If you offset this to 0x42000000 then a DA is
> > > generated as expected and the test passes.
> > >
> > > There is a very old thread started by Ard, "MMIO regions in
> > > GetMemoryMap ()", in which he questions if all the MMIO regions should
> > > be reported by GetMemoryMap() or only the ones being used by
> > > initialized devices, which is what the current implementation does.
> > > The end result of the thread was the current implementation is
> > > correct.  That leaves me with the question, "What is the proper
> > > solution for the current implementation that I am working with?"
> > >
> > > To me it seems like I need a special Library that on boot goes through
> > > and claims and maps every valid MMIO slot in this region, and then
> > > have the drivers use this library for proxying requests to
> > > gDS->AddMemorySpace (), gDS->SetMemorySpaceAttributes () etc.   If
> > > there is a standardized way to deal with this configuration I would
> > > much rather follow that.
> > >
> > > Thanks for any input,
> > > Jon
> > >
> > >
> > >
> >
> >
> >
> >
>
> Andrew,
>
> Thanks for clearing this up.  I definitely understand the
> relationships much better now.  I am still uncertain of the proper way
> to fix this issue regarding the SBSA peripheral test.  The test is
> using gBS->GetMemoryMap in order to look for available memory
> segments, which according to the earlier thread and your confirmation
> "gBS->GetMemoryMap () only returns information about DRAM and any
> address that requires a kernel virtual address mapping in EFI.", which
> means that this test will never work on this platform because the
> address that was randomly chosen just happens to fall right into the
> address range for the device addresses that are valid so won't produce
> a DA, but aren't being used by EFI therefore don't require a virtual
> address mapping in EFI.
>
> Is this something I should bring up with the SBSA group?
>
> -Jon
>

I have followed up with ARM and they agree it is an issue with the
test and not a configuration error.  They will provide a waiver and
look into refining the test in the future.

Thanks for your help.