How do I access Prefetchable Memory region through EFI_PCI_IO_PROTOCOL

How do I access Prefetchable Memory region through EFI_PCI_IO_PROTOCOL

[Kirkendall, Garrett]

Hopefully someone can help me understand.

I assume I should be able to access MMIO pointed to by a PCI devices Prefetchable Memory BAR.  I would think I would use EFI_PCI_IO_PROTOCOL.Mem.Read or Write.  This in turn will send the request up to EFI_PCI_ROOT_BRIDGE_IO_PROTOCOL.Mem.Read or Write.  

MdeModulePkg/Bus/Pci/PciHostBridgeDxe/PciRootBridgeIo.c: RootBridgeIoMemRead (Write) calls RootBridgeIoCheckParameter() to verify that the request falls within the confines of the PCI root bridge.  The below code verifies the address against non-prefetchable MMIO regions and skips the prefetchable memory regions.  Is this correct, and if so what is the method to read/write prefetchable memory regions?  If I force EFI_PCI_HOST_BRIDGE_COMBINE_MEM_PMEM, then I can access prefetchable memory regions because they are forced within Mem.* and MemAbove4G.*



EFI_STATUS
RootBridgeIoCheckParameter (
  IN EFI_PCI_ROOT_BRIDGE_IO_PROTOCOL        *This,
  IN OPERATION_TYPE                         OperationType,
  IN EFI_PCI_ROOT_BRIDGE_IO_PROTOCOL_WIDTH  Width,
  IN UINT64                                 Address,
  IN UINTN                                  Count,
  IN VOID                                   *Buffer
  )
{
...
  } else if (OperationType == MemOperation) {
    //
    // Allow Legacy MMIO access
    //
    if ((Address >= 0xA0000) && (Address + MultU64x32 (Count, Size)) <= 0xC0000) {
      if ((RootBridge->Attributes & EFI_PCI_ATTRIBUTE_VGA_MEMORY) != 0) {
        return EFI_SUCCESS;
      }
    }
    //
    // By comparing the Address against Limit we know which range to be used
    // for checking
    //
    if (Address + MultU64x32 (Count, Size) <= RootBridge->Mem.Limit + 1) {
      Base = RootBridge->Mem.Base;
      Limit = RootBridge->Mem.Limit;
    } else {
      Base = RootBridge->MemAbove4G.Base;
      Limit = RootBridge->MemAbove4G.Limit;
    }
  } else {
...

GARRETT KIRKENDALL
SMTS Firmware Engineer | CTE
7171 Southwest Parkway, Austin, TX 78735 USA 
AMD   facebook  |  amd.com

Re: How do I access Prefetchable Memory region through EFI_PCI_IO_PROTOCOL

[Laszlo Ersek]

Adding Ray, and a comment at the bottom:

On 09/14/18 20:24, Kirkendall, Garrett wrote:
> Hopefully someone can help me understand.
> 
> I assume I should be able to access MMIO pointed to by a PCI devices Prefetchable Memory BAR.  I would think I would use EFI_PCI_IO_PROTOCOL.Mem.Read or Write.  This in turn will send the request up to EFI_PCI_ROOT_BRIDGE_IO_PROTOCOL.Mem.Read or Write.  
> 
> MdeModulePkg/Bus/Pci/PciHostBridgeDxe/PciRootBridgeIo.c: RootBridgeIoMemRead (Write) calls RootBridgeIoCheckParameter() to verify that the request falls within the confines of the PCI root bridge.  The below code verifies the address against non-prefetchable MMIO regions and skips the prefetchable memory regions.  Is this correct, and if so what is the method to read/write prefetchable memory regions?  If I force EFI_PCI_HOST_BRIDGE_COMBINE_MEM_PMEM, then I can access prefetchable memory regions because they are forced within Mem.* and MemAbove4G.*
> 
> 
> 
> EFI_STATUS
> RootBridgeIoCheckParameter (
>   IN EFI_PCI_ROOT_BRIDGE_IO_PROTOCOL        *This,
>   IN OPERATION_TYPE                         OperationType,
>   IN EFI_PCI_ROOT_BRIDGE_IO_PROTOCOL_WIDTH  Width,
>   IN UINT64                                 Address,
>   IN UINTN                                  Count,
>   IN VOID                                   *Buffer
>   )
> {
> ...
>   } else if (OperationType == MemOperation) {
>     //
>     // Allow Legacy MMIO access
>     //
>     if ((Address >= 0xA0000) && (Address + MultU64x32 (Count, Size)) <= 0xC0000) {
>       if ((RootBridge->Attributes & EFI_PCI_ATTRIBUTE_VGA_MEMORY) != 0) {
>         return EFI_SUCCESS;
>       }
>     }
>     //
>     // By comparing the Address against Limit we know which range to be used
>     // for checking
>     //
>     if (Address + MultU64x32 (Count, Size) <= RootBridge->Mem.Limit + 1) {
>       Base = RootBridge->Mem.Base;
>       Limit = RootBridge->Mem.Limit;
>     } else {
>       Base = RootBridge->MemAbove4G.Base;
>       Limit = RootBridge->MemAbove4G.Limit;
>     }
>   } else {
> ...
> 

I'm not a PCI expert, but I think you are right; the code should permit
access to the PMem / PMemAbove4G apertures as well. According to the PCI
spec (--> fair use citation below),

"A PCI Express Function requesting memory resources through a BAR must
set the BAR's Prefetchable bit unless the range contains locations with
read side effects or locations in which the Function does not tolerate
write merging."

IOW "prefetchable" means "there are no side effects to reading, and the
Function tolerates write merging".

I don't see why that's reason for RootBridgeIoCheckParameter() to reject
the access.

Ray, what's your take?

Thanks
Laszlo

Re: How do I access Prefetchable Memory region through EFI_PCI_IO_PROTOCOL

[Kirkendall, Garrett]

Ray, did you get a chance to look at the code and see how Prefetchable Memory regions should be handled?

GARRETT KIRKENDALL
SMTS Firmware Engineer | CTE
7171 Southwest Parkway, Austin, TX 78735 USA 
AMD   facebook  |  amd.com

-----Original Message-----
From: Laszlo Ersek <lersek@redhat.com> 
Sent: Sunday, September 16, 2018 5:20 AM
To: Kirkendall, Garrett <Garrett.Kirkendall@amd.com>
Cc: edk2-devel@lists.01.org; Ruiyu Ni <ruiyu.ni@intel.com>
Subject: Re: [edk2] How do I access Prefetchable Memory region through EFI_PCI_IO_PROTOCOL

Adding Ray, and a comment at the bottom:

On 09/14/18 20:24, Kirkendall, Garrett wrote:
> Hopefully someone can help me understand.
> 
> I assume I should be able to access MMIO pointed to by a PCI devices Prefetchable Memory BAR.  I would think I would use EFI_PCI_IO_PROTOCOL.Mem.Read or Write.  This in turn will send the request up to EFI_PCI_ROOT_BRIDGE_IO_PROTOCOL.Mem.Read or Write.  
> 
> MdeModulePkg/Bus/Pci/PciHostBridgeDxe/PciRootBridgeIo.c: 
> RootBridgeIoMemRead (Write) calls RootBridgeIoCheckParameter() to 
> verify that the request falls within the confines of the PCI root 
> bridge.  The below code verifies the address against non-prefetchable 
> MMIO regions and skips the prefetchable memory regions.  Is this 
> correct, and if so what is the method to read/write prefetchable 
> memory regions?  If I force EFI_PCI_HOST_BRIDGE_COMBINE_MEM_PMEM, then 
> I can access prefetchable memory regions because they are forced 
> within Mem.* and MemAbove4G.*
> 
> 
> 
> EFI_STATUS
> RootBridgeIoCheckParameter (
>   IN EFI_PCI_ROOT_BRIDGE_IO_PROTOCOL        *This,
>   IN OPERATION_TYPE                         OperationType,
>   IN EFI_PCI_ROOT_BRIDGE_IO_PROTOCOL_WIDTH  Width,
>   IN UINT64                                 Address,
>   IN UINTN                                  Count,
>   IN VOID                                   *Buffer
>   )
> {
> ...
>   } else if (OperationType == MemOperation) {
>     //
>     // Allow Legacy MMIO access
>     //
>     if ((Address >= 0xA0000) && (Address + MultU64x32 (Count, Size)) <= 0xC0000) {
>       if ((RootBridge->Attributes & EFI_PCI_ATTRIBUTE_VGA_MEMORY) != 0) {
>         return EFI_SUCCESS;
>       }
>     }
>     //
>     // By comparing the Address against Limit we know which range to be used
>     // for checking
>     //
>     if (Address + MultU64x32 (Count, Size) <= RootBridge->Mem.Limit + 1) {
>       Base = RootBridge->Mem.Base;
>       Limit = RootBridge->Mem.Limit;
>     } else {
>       Base = RootBridge->MemAbove4G.Base;
>       Limit = RootBridge->MemAbove4G.Limit;
>     }
>   } else {
> ...
> 

I'm not a PCI expert, but I think you are right; the code should permit access to the PMem / PMemAbove4G apertures as well. According to the PCI spec (--> fair use citation below),

"A PCI Express Function requesting memory resources through a BAR must set the BAR's Prefetchable bit unless the range contains locations with read side effects or locations in which the Function does not tolerate write merging."

IOW "prefetchable" means "there are no side effects to reading, and the Function tolerates write merging".

I don't see why that's reason for RootBridgeIoCheckParameter() to reject the access.

Ray, what's your take?

Thanks
Laszlo

Re: How do I access Prefetchable Memory region through EFI_PCI_IO_PROTOCOL

[Ni, Ruiyu]

Garrett,
I think it's a real bug in the code. Thanks for reporting that.
I just submitted a Bz 1196 for this issue.
Patches are also sent out.

Thanks/Ray

> -----Original Message-----
> From: edk2-devel <edk2-devel-bounces@lists.01.org> On Behalf Of
> Kirkendall, Garrett
> Sent: Wednesday, September 19, 2018 9:53 AM
> To: Laszlo Ersek <lersek@redhat.com>
> Cc: Ni, Ruiyu <ruiyu.ni@intel.com>; edk2-devel@lists.01.org
> Subject: Re: [edk2] How do I access Prefetchable Memory region through
> EFI_PCI_IO_PROTOCOL
> 
> Ray, did you get a chance to look at the code and see how Prefetchable
> Memory regions should be handled?
> 
> GARRETT KIRKENDALL
> SMTS Firmware Engineer | CTE
> 7171 Southwest Parkway, Austin, TX 78735 USA AMD   facebook  |  amd.com
> 
> -----Original Message-----
> From: Laszlo Ersek <lersek@redhat.com>
> Sent: Sunday, September 16, 2018 5:20 AM
> To: Kirkendall, Garrett <Garrett.Kirkendall@amd.com>
> Cc: edk2-devel@lists.01.org; Ruiyu Ni <ruiyu.ni@intel.com>
> Subject: Re: [edk2] How do I access Prefetchable Memory region through
> EFI_PCI_IO_PROTOCOL
> 
> Adding Ray, and a comment at the bottom:
> 
> On 09/14/18 20:24, Kirkendall, Garrett wrote:
> > Hopefully someone can help me understand.
> >
> > I assume I should be able to access MMIO pointed to by a PCI devices
> Prefetchable Memory BAR.  I would think I would use
> EFI_PCI_IO_PROTOCOL.Mem.Read or Write.  This in turn will send the
> request up to EFI_PCI_ROOT_BRIDGE_IO_PROTOCOL.Mem.Read or Write.
> >
> > MdeModulePkg/Bus/Pci/PciHostBridgeDxe/PciRootBridgeIo.c:
> > RootBridgeIoMemRead (Write) calls RootBridgeIoCheckParameter() to
> > verify that the request falls within the confines of the PCI root
> > bridge.  The below code verifies the address against non-prefetchable
> > MMIO regions and skips the prefetchable memory regions.  Is this
> > correct, and if so what is the method to read/write prefetchable
> > memory regions?  If I force
> EFI_PCI_HOST_BRIDGE_COMBINE_MEM_PMEM, then
> > I can access prefetchable memory regions because they are forced
> > within Mem.* and MemAbove4G.*
> >
> >
> >
> > EFI_STATUS
> > RootBridgeIoCheckParameter (
> >   IN EFI_PCI_ROOT_BRIDGE_IO_PROTOCOL        *This,
> >   IN OPERATION_TYPE                         OperationType,
> >   IN EFI_PCI_ROOT_BRIDGE_IO_PROTOCOL_WIDTH  Width,
> >   IN UINT64                                 Address,
> >   IN UINTN                                  Count,
> >   IN VOID                                   *Buffer
> >   )
> > {
> > ...
> >   } else if (OperationType == MemOperation) {
> >     //
> >     // Allow Legacy MMIO access
> >     //
> >     if ((Address >= 0xA0000) && (Address + MultU64x32 (Count, Size)) <=
> 0xC0000) {
> >       if ((RootBridge->Attributes & EFI_PCI_ATTRIBUTE_VGA_MEMORY) != 0)
> {
> >         return EFI_SUCCESS;
> >       }
> >     }
> >     //
> >     // By comparing the Address against Limit we know which range to be
> used
> >     // for checking
> >     //
> >     if (Address + MultU64x32 (Count, Size) <= RootBridge->Mem.Limit + 1) {
> >       Base = RootBridge->Mem.Base;
> >       Limit = RootBridge->Mem.Limit;
> >     } else {
> >       Base = RootBridge->MemAbove4G.Base;
> >       Limit = RootBridge->MemAbove4G.Limit;
> >     }
> >   } else {
> > ...
> >
> 
> I'm not a PCI expert, but I think you are right; the code should permit access
> to the PMem / PMemAbove4G apertures as well. According to the PCI spec
> (--> fair use citation below),
> 
> "A PCI Express Function requesting memory resources through a BAR must
> set the BAR's Prefetchable bit unless the range contains locations with read
> side effects or locations in which the Function does not tolerate write
> merging."
> 
> IOW "prefetchable" means "there are no side effects to reading, and the
> Function tolerates write merging".
> 
> I don't see why that's reason for RootBridgeIoCheckParameter() to reject the
> access.
> 
> Ray, what's your take?
> 
> Thanks
> Laszlo
> _______________________________________________
> edk2-devel mailing list
> edk2-devel@lists.01.org
> https://lists.01.org/mailman/listinfo/edk2-devel