Re: [RFC] [PATCH 3/3] MdeModulePkg/PciBus: Add IOMMU support.

["Ni, Ruiyu" <ruiyu.ni@intel.com>]

#1. If PciHostBridge driver doesn't consume IOMMU protocol, will the change in PciBus driver cause any
       negative impact?
#2. Please check the PciHostBridgeDxe driver (searching “Maps”). It contains the code to manage the mapping in a list.
       Actually you modified the Map/Unmap code, you should have seen the code if you look a bit more. 😊

Thanks/Ray

From: Yao, Jiewen
Sent: Monday, March 27, 2017 2:16 PM
To: Ni, Ruiyu <ruiyu.ni@intel.com>; edk2-devel@lists.01.org
Cc: Leo Duran <leo.duran@amd.com>; Brijesh Singh <brijesh.singh@amd.com>
Subject: RE: [RFC] [PATCH 3/3] MdeModulePkg/PciBus: Add IOMMU support.

Answer below:

From: Ni, Ruiyu
Sent: Monday, March 27, 2017 1:40 PM
To: Yao, Jiewen <jiewen.yao@intel.com<mailto:jiewen.yao@intel.com>>; edk2-devel@lists.01.org<mailto:edk2-devel@lists.01.org>
Cc: Leo Duran <leo.duran@amd.com<mailto:leo.duran@amd.com>>; Brijesh Singh <brijesh.singh@amd.com<mailto:brijesh.singh@amd.com>>
Subject: RE: [RFC] [PATCH 3/3] MdeModulePkg/PciBus: Add IOMMU support.

#1. If PciHostBridge driver doesn't consume IOMMU protocol, will the change in PciBus driver cause any
       negative impact? For example, causing system hang? Do we need to introduce a bit to let PciHostBridge
       report the capability bit about IOMMU through PciHostBridge.GetAllocationAttribute()?
[Jiewen] That is a very good question.

I thought it before, but I finally give up the idea to define capability bit, because I want to make things simple.
I assume the system integration engineer should carefully pick up all drivers and validate the combination.

If the system integration engineer choose to integrate IOMMU driver, he/she should also choose a PciHostBridge which IOMMU capability.

I do not deny the goodness of such capability. But I want to add them later until we see the real usage.



#2, Can you add a Maps field into the PciIoDevice structure? So that the PciIoMapInfo can be inserted to
       the list and we can check whether the Mapping is invalid by checking the existence in the list.
       This is better than checking the memory signature (PciIoMapInfo->Signature).
[Jiewen] I am not clear about this.
Can you provide some sample code to show your idea?
Maybe just submit a patch based upon current code. If it is good, I can integrate the patch directly.


#3.  I saw PciIo.Map/Allocate sets the IOMMU attribute and Unmap/Free clears the IOMMU attribute.
        Do we have the case that the original IOMMU attribute is not 0? If that may happen, we may need to
        have an additional interface in IOMMU protocol to get the original attribute.
[Jiewen] Good question again.
I added below word in IOMMU protocol header file.

Allocation success mean no one use this memory before, so that it should be DMA blocked.


+/**

+  Set IOMMU attribute for a system memory.

+

+  If the IOMMU protocol exists, the system memory cannot be used

+  for DMA by default.

+

+  When a device requests a DMA access for a system memory,

+  the device driver need use SetAttribute() to update the IOMMU

+  attribute to request DMA access (read and/or write).





Thanks/Ray

> -----Original Message-----
> From: Yao, Jiewen
> Sent: Saturday, March 25, 2017 5:29 PM
> To: edk2-devel@lists.01.org<mailto:edk2-devel@lists.01.org>
> Cc: Ni, Ruiyu <ruiyu.ni@intel.com<mailto:ruiyu.ni@intel.com>>; Leo Duran <leo.duran@amd.com<mailto:leo.duran@amd.com>>;
> Brijesh Singh <brijesh.singh@amd.com<mailto:brijesh.singh@amd.com>>
> Subject: [RFC] [PATCH 3/3] MdeModulePkg/PciBus: Add IOMMU support.
>
> The responsibility of PciBus driver is to set IOMMU attribute, because only
> PciBus knows which device submits DMA access request.
>
> PciBus driver assumes that PciHostBridge driver can allocate IOMMU page
> aligned memory, if IOMMU protocol exists.
>
> Cc: Ruiyu Ni <ruiyu.ni@intel.com<mailto:ruiyu.ni@intel.com>>
> Cc: Leo Duran <leo.duran@amd.com<mailto:leo.duran@amd.com>>
> Cc: Brijesh Singh <brijesh.singh@amd.com<mailto:brijesh.singh@amd.com>>
> Contributed-under: TianoCore Contribution Agreement 1.0
> Signed-off-by: Jiewen Yao <jiewen.yao@intel.com<mailto:jiewen.yao@intel.com>>
> ---
>  MdeModulePkg/Bus/Pci/PciBusDxe/PciBus.c      |  12 +++
>  MdeModulePkg/Bus/Pci/PciBusDxe/PciBus.h      |  10 ++
>  MdeModulePkg/Bus/Pci/PciBusDxe/PciBusDxe.inf |   1 +
>  MdeModulePkg/Bus/Pci/PciBusDxe/PciIo.c       | 100
> ++++++++++++++++++++
>  4 files changed, 123 insertions(+)
>
> diff --git a/MdeModulePkg/Bus/Pci/PciBusDxe/PciBus.c
> b/MdeModulePkg/Bus/Pci/PciBusDxe/PciBus.c
> index f3be47a..c9ee4de 100644
> --- a/MdeModulePkg/Bus/Pci/PciBusDxe/PciBus.c
> +++ b/MdeModulePkg/Bus/Pci/PciBusDxe/PciBus.c
> @@ -42,6 +42,8 @@ UINT64                                        gAllZero             = 0;
>
>  EFI_PCI_PLATFORM_PROTOCOL                     *gPciPlatformProtocol;
>  EFI_PCI_OVERRIDE_PROTOCOL                     *gPciOverrideProtocol;
> +EDKII_IOMMU_PROTOCOL                          *gIoMmuProtocol;
> +UINTN                                         mIoMmuPageSize = 1;
>
>
>  GLOBAL_REMOVE_IF_UNREFERENCED
> EFI_PCI_HOTPLUG_REQUEST_PROTOCOL mPciHotPlugRequest = { @@ -
> 256,6 +258,16 @@ PciBusDriverBindingStart (
>    }
>
>    gBS->LocateProtocol (
> +        &gEdkiiIoMmuProtocolGuid,
> +        NULL,
> +        (VOID **) &gIoMmuProtocol
> +        );
> +  if (gIoMmuProtocol != NULL) {
> +    gIoMmuProtocol->GetPageSize (gIoMmuProtocol, &mIoMmuPageSize);
> +    ASSERT ((mIoMmuPageSize & (mIoMmuPageSize - 1)) == 0);  }
> +
> +  gBS->LocateProtocol (
>           &gEfiIncompatiblePciDeviceSupportProtocolGuid,
>           NULL,
>           (VOID **) &gIncompatiblePciDeviceSupport diff --git
> a/MdeModulePkg/Bus/Pci/PciBusDxe/PciBus.h
> b/MdeModulePkg/Bus/Pci/PciBusDxe/PciBus.h
> index 39ba8b9..6f96696 100644
> --- a/MdeModulePkg/Bus/Pci/PciBusDxe/PciBus.h
> +++ b/MdeModulePkg/Bus/Pci/PciBusDxe/PciBus.h
> @@ -32,6 +32,7 @@ WITHOUT WARRANTIES OR REPRESENTATIONS OF ANY
> KIND, EITHER EXPRESS OR IMPLIED.
>  #include <Protocol/IncompatiblePciDeviceSupport.h>
>  #include <Protocol/PciOverride.h>
>  #include <Protocol/PciEnumerationComplete.h>
> +#include <Protocol/IoMmu.h>
>
>  #include <Library/DebugLib.h>
>  #include <Library/UefiDriverEntryPoint.h> @@ -304,6 +305,13 @@ struct
> _PCI_IO_DEVICE {
>    CR (a, PCI_IO_DEVICE, LoadFile2, PCI_IO_DEVICE_SIGNATURE)
>
>
> +#define PCI_IO_MAP_INFO_SIGNATURE  SIGNATURE_32 ('p', 'm', 'a', 'p')
> +typedef struct {
> +  UINT32                                    Signature;
> +  UINTN                                     NumberOfBytes;
> +  EFI_PHYSICAL_ADDRESS                      DeviceAddress;
> +  VOID                                      *PciRootBridgeIoMapping;
> +} PCI_IO_MAP_INFO;
>
>  //
>  // Global Variables
> @@ -319,6 +327,8 @@ extern UINT64                                       gAllOne;
>  extern UINT64                                       gAllZero;
>  extern EFI_PCI_PLATFORM_PROTOCOL                    *gPciPlatformProtocol;
>  extern EFI_PCI_OVERRIDE_PROTOCOL                    *gPciOverrideProtocol;
> +extern EDKII_IOMMU_PROTOCOL                         *gIoMmuProtocol;
> +extern UINTN                                        mIoMmuPageSize;
>  extern BOOLEAN                                      mReserveIsaAliases;
>  extern BOOLEAN                                      mReserveVgaAliases;
>
> diff --git a/MdeModulePkg/Bus/Pci/PciBusDxe/PciBusDxe.inf
> b/MdeModulePkg/Bus/Pci/PciBusDxe/PciBusDxe.inf
> index a3ab11f..5da094f 100644
> --- a/MdeModulePkg/Bus/Pci/PciBusDxe/PciBusDxe.inf
> +++ b/MdeModulePkg/Bus/Pci/PciBusDxe/PciBusDxe.inf
> @@ -95,6 +95,7 @@
>    gEfiPciRootBridgeIoProtocolGuid                 ## TO_START
>    gEfiIncompatiblePciDeviceSupportProtocolGuid    ##
> SOMETIMES_CONSUMES
>    gEfiLoadFile2ProtocolGuid                       ## SOMETIMES_PRODUCES
> +  gEdkiiIoMmuProtocolGuid                         ## SOMETIMES_CONSUMES
>
>  [FeaturePcd]
>    gEfiMdeModulePkgTokenSpaceGuid.PcdPciBusHotplugDeviceSupport      ##
> CONSUMES
> diff --git a/MdeModulePkg/Bus/Pci/PciBusDxe/PciIo.c
> b/MdeModulePkg/Bus/Pci/PciBusDxe/PciIo.c
> index f72598d..01786c1 100644
> --- a/MdeModulePkg/Bus/Pci/PciBusDxe/PciIo.c
> +++ b/MdeModulePkg/Bus/Pci/PciBusDxe/PciIo.c
> @@ -967,6 +967,8 @@ PciIoMap (
>  {
>    EFI_STATUS    Status;
>    PCI_IO_DEVICE *PciIoDevice;
> +  PCI_IO_MAP_INFO       *PciIoMapInfo;
> +  UINT64                IoMmuAttribute;
>
>    PciIoDevice = PCI_IO_DEVICE_FROM_PCI_IO_THIS (This);
>
> @@ -999,6 +1001,46 @@ PciIoMap (
>        );
>    }
>
> +  if (gIoMmuProtocol != NULL) {
> +    if (!EFI_ERROR(Status)) {
> +      PciIoMapInfo = AllocatePool (sizeof(*PciIoMapInfo));
> +      if (PciIoMapInfo == NULL) {
> +        PciIoDevice->PciRootBridgeIo->Unmap (PciIoDevice->PciRootBridgeIo,
> *Mapping);
> +        return EFI_OUT_OF_RESOURCES;
> +      }
> +
> +      PciIoMapInfo->Signature              = PCI_IO_MAP_INFO_SIGNATURE;
> +      PciIoMapInfo->NumberOfBytes          = *NumberOfBytes;
> +      PciIoMapInfo->DeviceAddress          = *DeviceAddress;
> +      PciIoMapInfo->PciRootBridgeIoMapping = *Mapping;
> +      *Mapping = PciIoMapInfo;
> +
> +      switch (Operation) {
> +      case EfiPciIoOperationBusMasterRead:
> +        IoMmuAttribute = EDKII_IOMMU_ATTRIBUTE_READ;
> +        break;
> +      case EfiPciIoOperationBusMasterWrite:
> +        IoMmuAttribute = EDKII_IOMMU_ATTRIBUTE_WRITE;
> +        break;
> +      case EfiPciIoOperationBusMasterCommonBuffer:
> +        IoMmuAttribute = EDKII_IOMMU_ATTRIBUTE_READ |
> EDKII_IOMMU_ATTRIBUTE_WRITE;
> +        break;
> +      default:
> +        ASSERT(FALSE);
> +        return Status;
> +      }
> +      //
> +      // PciHostBridge should map IOMMU page aligned HostAddress.
> +      //
> +      gIoMmuProtocol->SetAttribute (
> +                        gIoMmuProtocol,
> +                        PciIoDevice->Handle,
> +                        PciIoMapInfo->DeviceAddress,
> +                        ALIGN_VALUE(PciIoMapInfo->NumberOfBytes,
> mIoMmuPageSize),
> +                        IoMmuAttribute
> +                        );
> +    }
> +  }
>    return Status;
>  }
>
> @@ -1021,9 +1063,19 @@ PciIoUnmap (
>  {
>    EFI_STATUS    Status;
>    PCI_IO_DEVICE *PciIoDevice;
> +  PCI_IO_MAP_INFO *PciIoMapInfo;
>
>    PciIoDevice = PCI_IO_DEVICE_FROM_PCI_IO_THIS (This);
>
> +  PciIoMapInfo = NULL;
> +  if (gIoMmuProtocol != NULL) {
> +    PciIoMapInfo = Mapping;
> +    if (PciIoMapInfo->Signature != PCI_IO_MAP_INFO_SIGNATURE) {
> +      return EFI_INVALID_PARAMETER;
> +    }
> +    Mapping = PciIoMapInfo->PciRootBridgeIoMapping;
> +  }
> +
>    Status = PciIoDevice->PciRootBridgeIo->Unmap (
>                                            PciIoDevice->PciRootBridgeIo,
>                                            Mapping @@ -1037,6 +1089,22 @@ PciIoUnmap (
>        );
>    }
>
> +  if (gIoMmuProtocol != NULL) {
> +    if (!EFI_ERROR(Status)) {
> +      //
> +      // PciHostBridge should map IOMMU page aligned HostAddress.
> +      //
> +      gIoMmuProtocol->SetAttribute (
> +                        gIoMmuProtocol,
> +                        PciIoDevice->Handle,
> +                        PciIoMapInfo->DeviceAddress,
> +                        ALIGN_VALUE(PciIoMapInfo->NumberOfBytes,
> mIoMmuPageSize),
> +                        0
> +                        );
> +      FreePool (PciIoMapInfo);
> +    }
> +  }
> +
>    return Status;
>  }
>
> @@ -1073,6 +1141,7 @@ PciIoAllocateBuffer (  {
>    EFI_STATUS    Status;
>    PCI_IO_DEVICE *PciIoDevice;
> +  UINTN         Size;
>
>    if ((Attributes &
>        (~(EFI_PCI_ATTRIBUTE_MEMORY_WRITE_COMBINE |
> EFI_PCI_ATTRIBUTE_MEMORY_CACHED))) != 0){ @@ -1102,6 +1171,21 @@
> PciIoAllocateBuffer (
>        );
>    }
>
> +  if (gIoMmuProtocol != NULL) {
> +    if (!EFI_ERROR(Status)) {
> +      //
> +      // PciHostBridge should allocate IOMMU page aligned HostAddress.
> +      //
> +      Size = EFI_PAGES_TO_SIZE(Pages);
> +      gIoMmuProtocol->SetAttribute (
> +                        gIoMmuProtocol,
> +                        PciIoDevice->Handle,
> +                        (UINTN)*HostAddress,
> +                        ALIGN_VALUE(Size, mIoMmuPageSize),
> +                        EDKII_IOMMU_ATTRIBUTE_READ |
> EDKII_IOMMU_ATTRIBUTE_WRITE
> +                        );
> +    }
> +  }
>    return Status;
>  }
>
> @@ -1127,6 +1211,7 @@ PciIoFreeBuffer (
>  {
>    EFI_STATUS    Status;
>    PCI_IO_DEVICE *PciIoDevice;
> +  UINTN         Size;
>
>    PciIoDevice = PCI_IO_DEVICE_FROM_PCI_IO_THIS (This);
>
> @@ -1144,6 +1229,21 @@ PciIoFreeBuffer (
>        );
>    }
>
> +  if (gIoMmuProtocol != NULL) {
> +    if (!EFI_ERROR(Status)) {
> +      //
> +      // PciHostBridge should allocate IOMMU page aligned HostAddress.
> +      //
> +      Size = EFI_PAGES_TO_SIZE(Pages);
> +      gIoMmuProtocol->SetAttribute (
> +                        gIoMmuProtocol,
> +                        PciIoDevice->Handle,
> +                        (UINTN)HostAddress,
> +                        ALIGN_VALUE(Size, mIoMmuPageSize),
> +                        0
> +                        );
> +    }
> +  }
>    return Status;
>  }
>
> --
> 2.7.4.windows.1