[PATCH 0/4] Improve IntelVTd performance.

[PATCH 0/4] Improve IntelVTd performance.

[Jiewen Yao]

This series patch replace WBINVD with CLFlush
in IntelVTdDxe driver, which improves boot performance.

This patch is validated on Intel Kabylake generation.

Jiewen Yao (4):
  MdePkg/CpuLib.h: Add CacheLineFlush function.
  MdePkg/BaseCpuLib: Add CacheLineFlush function.
  IntelSiliconPkg/dsc: Add CpuLib.
  IntelSiliconPkg/IntelVTdDxe: Improve performance.

 IntelSiliconPkg/IntelSiliconPkg.dsc                |   1 +
 IntelSiliconPkg/IntelVTdDxe/DmaProtection.c        |   6 +
 IntelSiliconPkg/IntelVTdDxe/DmaProtection.h        |  61 ++---
 IntelSiliconPkg/IntelVTdDxe/IntelVTdDxe.c          |  10 +
 IntelSiliconPkg/IntelVTdDxe/IntelVTdDxe.inf        |   4 +
 IntelSiliconPkg/IntelVTdDxe/PciInfo.c              |  12 +
 IntelSiliconPkg/IntelVTdDxe/TranslationTable.c     |  58 +++-
 IntelSiliconPkg/IntelVTdDxe/TranslationTableEx.c   |   2 +
 IntelSiliconPkg/IntelVTdDxe/VtdReg.c               | 283 +++++++-------------
 MdePkg/Include/Library/CpuLib.h                    |  12 +-
 MdePkg/Library/BaseCpuLib/BaseCpuLib.inf           |   5 +-
 MdePkg/Library/BaseCpuLib/Ebc/CpuSleepFlushTlb.c   |  16 +-
 MdePkg/Library/BaseCpuLib/Ia32/CacheLineFlush.nasm |  37 +++
 MdePkg/Library/BaseCpuLib/X64/CacheLineFlush.nasm  |  37 +++
 14 files changed, 318 insertions(+), 226 deletions(-)
 create mode 100644 MdePkg/Library/BaseCpuLib/Ia32/CacheLineFlush.nasm
 create mode 100644 MdePkg/Library/BaseCpuLib/X64/CacheLineFlush.nasm

-- 
2.7.4.windows.1

[PATCH 1/4] MdePkg/CpuLib.h: Add CacheLineFlush function.

[Jiewen Yao]

This function will be used by IntelVTd driver.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Star Zeng <star.zeng@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.0
Signed-off-by: Jiewen Yao <jiewen.yao@intel.com>
---
 MdePkg/Include/Library/CpuLib.h | 12 +++++++++++-
 1 file changed, 11 insertions(+), 1 deletion(-)

diff --git a/MdePkg/Include/Library/CpuLib.h b/MdePkg/Include/Library/CpuLib.h
index 100020a..1b2f7e8 100644
--- a/MdePkg/Include/Library/CpuLib.h
+++ b/MdePkg/Include/Library/CpuLib.h
@@ -7,7 +7,7 @@
   PAL Calls require PEI and DXE specific mechanisms to look up PAL Entry Point.
   As a result, these services could not be defined in the Base Library.
 
-Copyright (c) 2006 - 2008, Intel Corporation. All rights reserved.<BR>
+Copyright (c) 2006 - 2017, Intel Corporation. All rights reserved.<BR>
 This program and the accompanying materials
 are licensed and made available under the terms and conditions of the BSD License
 which accompanies this distribution.  The full text of the license may be found at
@@ -47,5 +47,15 @@ CpuFlushTlb (
   VOID
   );
 
+/**
+  Flushes one cache line.
+
+  Flushes one cache line. The size of cache line can be got by CPU register.
+**/
+VOID
+EFIAPI
+CacheLineFlush (
+  IN UINTN  Address
+  );
 
 #endif
-- 
2.7.4.windows.1

[PATCH 2/4] MdePkg/BaseCpuLib: Add CacheLineFlush function.

[Jiewen Yao]

This function will be used by IntelVTd driver.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Star Zeng <star.zeng@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.0
Signed-off-by: Jiewen Yao <jiewen.yao@intel.com>
---
 MdePkg/Library/BaseCpuLib/BaseCpuLib.inf           |  5 ++-
 MdePkg/Library/BaseCpuLib/Ebc/CpuSleepFlushTlb.c   | 16 ++++++++-
 MdePkg/Library/BaseCpuLib/Ia32/CacheLineFlush.nasm | 37 ++++++++++++++++++++
 MdePkg/Library/BaseCpuLib/X64/CacheLineFlush.nasm  | 37 ++++++++++++++++++++
 4 files changed, 93 insertions(+), 2 deletions(-)

diff --git a/MdePkg/Library/BaseCpuLib/BaseCpuLib.inf b/MdePkg/Library/BaseCpuLib/BaseCpuLib.inf
index 637a3c5..b43b393 100644
--- a/MdePkg/Library/BaseCpuLib/BaseCpuLib.inf
+++ b/MdePkg/Library/BaseCpuLib/BaseCpuLib.inf
@@ -4,7 +4,7 @@
 #  CPU Library implemented using ASM functions for IA-32 and X64,
 #  PAL CALLs for IPF, and empty functions for EBC.
 #
-#  Copyright (c) 2007 - 2014, Intel Corporation. All rights reserved.<BR>
+#  Copyright (c) 2007 - 2017, Intel Corporation. All rights reserved.<BR>
 #  Portions copyright (c) 2008 - 2009, Apple Inc. All rights reserved.<BR>
 #  Portions copyright (c) 2011 - 2013, ARM Ltd. All rights reserved.<BR>
 #
@@ -33,6 +33,8 @@
 #
 
 [Sources.IA32]
+  Ia32/CacheLineFlush.nasm
+
   Ia32/CpuSleep.c | MSFT 
   Ia32/CpuFlushTlb.c | MSFT 
 
@@ -45,6 +47,7 @@
   Ia32/CpuFlushTlbGcc.c | GCC 
 
 [Sources.X64]
+  X64/CacheLineFlush.nasm
   X64/CpuFlushTlb.nasm
   X64/CpuFlushTlb.asm 
   X64/CpuSleep.nasm
diff --git a/MdePkg/Library/BaseCpuLib/Ebc/CpuSleepFlushTlb.c b/MdePkg/Library/BaseCpuLib/Ebc/CpuSleepFlushTlb.c
index de63d63..0c554c2 100644
--- a/MdePkg/Library/BaseCpuLib/Ebc/CpuSleepFlushTlb.c
+++ b/MdePkg/Library/BaseCpuLib/Ebc/CpuSleepFlushTlb.c
@@ -1,7 +1,7 @@
 /** @file
   Base Library CPU Functions for EBC
 
-  Copyright (c) 2006 - 2008, Intel Corporation. All rights reserved.<BR>
+  Copyright (c) 2006 - 2017, Intel Corporation. All rights reserved.<BR>
   This program and the accompanying materials
   are licensed and made available under the terms and conditions of the BSD License
   which accompanies this distribution.  The full text of the license may be found at
@@ -15,6 +15,20 @@
 #include <Library/DebugLib.h>
 
 /**
+  Flushes one cache line.
+
+  Flushes one cache line. The size of cache line can be got by CPU register.
+**/
+VOID
+EFIAPI
+CacheLineFlush (
+  IN UINTN  Address
+  )
+{
+  ASSERT (FALSE);
+}
+
+/**
   Flushes all the Translation Lookaside Buffers(TLB) entries in a CPU.
 
   Flushes all the Translation Lookaside Buffers(TLB) entries in a CPU.
diff --git a/MdePkg/Library/BaseCpuLib/Ia32/CacheLineFlush.nasm b/MdePkg/Library/BaseCpuLib/Ia32/CacheLineFlush.nasm
new file mode 100644
index 0000000..a2557ab
--- /dev/null
+++ b/MdePkg/Library/BaseCpuLib/Ia32/CacheLineFlush.nasm
@@ -0,0 +1,37 @@
+;------------------------------------------------------------------------------ ;
+; Copyright (c) 2017, Intel Corporation. All rights reserved.<BR>
+; This program and the accompanying materials
+; are licensed and made available under the terms and conditions of the BSD License
+; which accompanies this distribution.  The full text of the license may be found at
+; http://opensource.org/licenses/bsd-license.php.
+;
+; THE PROGRAM IS DISTRIBUTED UNDER THE BSD LICENSE ON AN "AS IS" BASIS,
+; WITHOUT WARRANTIES OR REPRESENTATIONS OF ANY KIND, EITHER EXPRESS OR IMPLIED.
+;
+; Module Name:
+;
+;   CacheLineFlush.Asm
+;
+; Abstract:
+;
+;   CacheLineFlush function
+;
+; Notes:
+;
+;------------------------------------------------------------------------------
+
+    SECTION .text
+
+;------------------------------------------------------------------------------
+; VOID
+; EFIAPI
+; CacheLineFlush (
+;   IN UINTN Address
+;   );
+;------------------------------------------------------------------------------
+global ASM_PFX(CacheLineFlush)
+ASM_PFX(CacheLineFlush):
+    mov     eax, [esp + 4]
+    clflush [eax]
+    ret
+
diff --git a/MdePkg/Library/BaseCpuLib/X64/CacheLineFlush.nasm b/MdePkg/Library/BaseCpuLib/X64/CacheLineFlush.nasm
new file mode 100644
index 0000000..7cf736e
--- /dev/null
+++ b/MdePkg/Library/BaseCpuLib/X64/CacheLineFlush.nasm
@@ -0,0 +1,37 @@
+;------------------------------------------------------------------------------ ;
+; Copyright (c) 2017, Intel Corporation. All rights reserved.<BR>
+; This program and the accompanying materials
+; are licensed and made available under the terms and conditions of the BSD License
+; which accompanies this distribution.  The full text of the license may be found at
+; http://opensource.org/licenses/bsd-license.php.
+;
+; THE PROGRAM IS DISTRIBUTED UNDER THE BSD LICENSE ON AN "AS IS" BASIS,
+; WITHOUT WARRANTIES OR REPRESENTATIONS OF ANY KIND, EITHER EXPRESS OR IMPLIED.
+;
+; Module Name:
+;
+;   CacheLineFlush.Asm
+;
+; Abstract:
+;
+;   CacheLineFlush function
+;
+; Notes:
+;
+;------------------------------------------------------------------------------
+
+    DEFAULT REL
+    SECTION .text
+
+;------------------------------------------------------------------------------
+; VOID
+; EFIAPI
+; CacheLineFlush (
+;   IN UINTN Address
+;   );
+;------------------------------------------------------------------------------
+global ASM_PFX(CacheLineFlush)
+ASM_PFX(CacheLineFlush):
+    clflush [rcx]
+    ret
+
-- 
2.7.4.windows.1

[PATCH 3/4] IntelSiliconPkg/dsc: Add CpuLib.

[Jiewen Yao]

It will be used by IntelVTdDxe.

Cc: Star Zeng <star.zeng@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.0
Signed-off-by: Jiewen Yao <jiewen.yao@intel.com>
---
 IntelSiliconPkg/IntelSiliconPkg.dsc | 1 +
 1 file changed, 1 insertion(+)

diff --git a/IntelSiliconPkg/IntelSiliconPkg.dsc b/IntelSiliconPkg/IntelSiliconPkg.dsc
index d837d84..7d7981e 100644
--- a/IntelSiliconPkg/IntelSiliconPkg.dsc
+++ b/IntelSiliconPkg/IntelSiliconPkg.dsc
@@ -38,6 +38,7 @@
   UefiDecompressLib|MdePkg/Library/BaseUefiDecompressLib/BaseUefiDecompressLib.inf
   PerformanceLib|MdePkg/Library/BasePerformanceLibNull/BasePerformanceLibNull.inf
   SerialPortLib|MdePkg/Library/BaseSerialPortLibNull/BaseSerialPortLibNull.inf
+  CpuLib|MdePkg/Library/BaseCpuLib/BaseCpuLib.inf
 
 [LibraryClasses.common.DXE_DRIVER]
   UefiDriverEntryPoint|MdePkg/Library/UefiDriverEntryPoint/UefiDriverEntryPoint.inf
-- 
2.7.4.windows.1

[PATCH 4/4] IntelSiliconPkg/IntelVTdDxe: Improve performance.

[Jiewen Yao]

This patch is to improve IOMMU performance.
All WBINVD is removed due to performance issue.
CLFLUSH is used to to only flush the context table or
second level page table if they are changed.

This patch also removed some unused functions.

Cc: Star Zeng <star.zeng@intel.com>
Contributed-under: TianoCore Contribution Agreement 1.0
Signed-off-by: Jiewen Yao <jiewen.yao@intel.com>
---
 IntelSiliconPkg/IntelVTdDxe/DmaProtection.c      |   6 +
 IntelSiliconPkg/IntelVTdDxe/DmaProtection.h      |  61 ++---
 IntelSiliconPkg/IntelVTdDxe/IntelVTdDxe.c        |  10 +
 IntelSiliconPkg/IntelVTdDxe/IntelVTdDxe.inf      |   4 +
 IntelSiliconPkg/IntelVTdDxe/PciInfo.c            |  12 +
 IntelSiliconPkg/IntelVTdDxe/TranslationTable.c   |  58 +++-
 IntelSiliconPkg/IntelVTdDxe/TranslationTableEx.c |   2 +
 IntelSiliconPkg/IntelVTdDxe/VtdReg.c             | 283 +++++++-------------
 8 files changed, 213 insertions(+), 223 deletions(-)

diff --git a/IntelSiliconPkg/IntelVTdDxe/DmaProtection.c b/IntelSiliconPkg/IntelVTdDxe/DmaProtection.c
index f0628b5..91f34d7 100644
--- a/IntelSiliconPkg/IntelVTdDxe/DmaProtection.c
+++ b/IntelSiliconPkg/IntelVTdDxe/DmaProtection.c
@@ -17,6 +17,8 @@ EFI_ACPI_SDT_PROTOCOL                   *mAcpiSdt;
 UINT64                                  mBelow4GMemoryLimit;
 UINT64                                  mAbove4GMemoryLimit;
 
+UINTN                                   mCacheLineSize;
+
 EDKII_PLATFORM_VTD_POLICY_PROTOCOL      *mPlatformVTdPolicy;
 
 /**
@@ -196,6 +198,7 @@ SetupVtd (
   UINTN           Index;
   UINT64          Below4GMemoryLimit;
   UINT64          Above4GMemoryLimit;
+  CPUID_VERSION_INFO_EBX  Ebx;
 
   //
   // PCI Enumeration must be done
@@ -207,6 +210,9 @@ SetupVtd (
                   );
   ASSERT_EFI_ERROR (Status);
 
+  AsmCpuid (CPUID_VERSION_INFO, NULL, &Ebx.Uint32, NULL, NULL);
+  mCacheLineSize = Ebx.Bits.CacheLineSize;
+
   ReturnUefiMemoryMap (&Below4GMemoryLimit, &Above4GMemoryLimit);
   Below4GMemoryLimit = ALIGN_VALUE_UP(Below4GMemoryLimit, SIZE_256MB);
   DEBUG ((DEBUG_INFO, " Adjusted Below4GMemoryLimit: 0x%016lx\n", Below4GMemoryLimit));
diff --git a/IntelSiliconPkg/IntelVTdDxe/DmaProtection.h b/IntelSiliconPkg/IntelVTdDxe/DmaProtection.h
index 8cfa69c..f6ce365 100644
--- a/IntelSiliconPkg/IntelVTdDxe/DmaProtection.h
+++ b/IntelSiliconPkg/IntelVTdDxe/DmaProtection.h
@@ -25,6 +25,9 @@
 #include <Library/PciSegmentLib.h>
 #include <Library/DebugLib.h>
 #include <Library/UefiLib.h>
+#include <Library/CpuLib.h>
+#include <Library/PerformanceLib.h>
+#include <Library/PrintLib.h>
 
 #include <Guid/EventGroup.h>
 #include <Guid/Acpi.h>
@@ -41,6 +44,8 @@
 #include <IndustryStandard/DmaRemappingReportingTable.h>
 #include <IndustryStandard/Vtd.h>
 
+#include <Register/Cpuid.h>
+
 #define VTD_64BITS_ADDRESS(Lo, Hi) (LShiftU64 (Lo, 12) | LShiftU64 (Hi, 32))
 
 #define ALIGN_VALUE_UP(Value, Alignment)  (((Value) + (Alignment) - 1) & (~((Alignment) - 1)))
@@ -58,6 +63,8 @@ typedef struct {
   UINTN                  PciDescriptorMaxNumber;
   BOOLEAN                *IsRealPciDevice;
   VTD_SOURCE_ID          *PciDescriptors;
+  // for statistic analysis
+  UINTN                  *AccessCount;
 } PCI_DEVICE_INFORMATION;
 
 typedef struct {
@@ -68,6 +75,7 @@ typedef struct {
   VTD_ROOT_ENTRY                   *RootEntryTable;
   VTD_EXT_ROOT_ENTRY               *ExtRootEntryTable;
   VTD_SECOND_LEVEL_PAGING_ENTRY    *FixedSecondLevelPagingEntry;
+  BOOLEAN                          HasDirtyContext;
   BOOLEAN                          HasDirtyPages;
   PCI_DEVICE_INFORMATION           PciDeviceInfo;
 } VTD_UNIT_INFORMATION;
@@ -81,6 +89,8 @@ extern VTD_UNIT_INFORMATION             *mVtdUnitInformation;
 extern UINT64                           mBelow4GMemoryLimit;
 extern UINT64                           mAbove4GMemoryLimit;
 
+extern UINTN                            mCacheLineSize;
+
 extern EDKII_PLATFORM_VTD_POLICY_PROTOCOL   *mPlatformVTdPolicy;
 
 /**
@@ -125,40 +135,6 @@ DisableDmar (
   );
 
 /**
-  Invalid VTd IOTLB page.
-
-  @param[in]  VtdIndex              The index of VTd engine.
-  @param[in]  Address               The address of IOTLB page.
-  @param[in]  AddressMode           The address mode of IOTLB page.
-  @param[in]  DomainIdentifier      The domain ID of the source.
-
-  @retval EFI_SUCCESS           VTd IOTLB page is invalidated.
-  @retval EFI_DEVICE_ERROR      VTd IOTLB page is not invalidated.
-**/
-EFI_STATUS
-InvalidateVtdIOTLBPage (
-  IN UINTN  VtdIndex,
-  IN UINT64 Address,
-  IN UINT8  AddressMode,
-  IN UINT16 DomainIdentifier
-  );
-
-/**
-  Invalid VTd IOTLB domain.
-
-  @param[in]  VtdIndex              The index of VTd engine.
-  @param[in]  DomainIdentifier      The domain ID of the source.
-
-  @retval EFI_SUCCESS           VTd IOTLB domain is invalidated.
-  @retval EFI_DEVICE_ERROR      VTd IOTLB domain is not invalidated.
-**/
-EFI_STATUS
-InvalidateVtdIOTLBDomain (
-  IN UINTN  VtdIndex,
-  IN UINT16 DomainIdentifier
-  );
-
-/**
   Invalid VTd global IOTLB.
 
   @param[in]  VtdIndex              The index of VTd engine.
@@ -362,6 +338,7 @@ DumpSecondLevelPagingEntry (
 EFI_STATUS
 SetPageAttribute (
   IN UINTN                         VtdIndex,
+  IN UINT16                        DomainIdentifier,
   IN VTD_SECOND_LEVEL_PAGING_ENTRY *SecondLevelPagingEntry,
   IN UINT64                        BaseAddress,
   IN UINT64                        Length,
@@ -500,4 +477,20 @@ AllocateZeroPages (
   IN UINTN  Pages
   );
 
+/**
+  Flush VTD page table and context table memory.
+
+  This action is to make sure the IOMMU engine can get final data in memory.
+
+  @param[in]  VtdIndex          The index used to identify a VTd engine.
+  @param[in]  Base              The base address of memory to be flushed.
+  @param[in]  Size              The size of memory in bytes to be flushed.
+**/
+VOID
+FlushPageTableMemory (
+  IN UINTN  VtdIndex,
+  IN UINTN  Base,
+  IN UINTN  Size
+  );
+
 #endif
diff --git a/IntelSiliconPkg/IntelVTdDxe/IntelVTdDxe.c b/IntelSiliconPkg/IntelVTdDxe/IntelVTdDxe.c
index d22222d..7feaaf5 100644
--- a/IntelSiliconPkg/IntelVTdDxe/IntelVTdDxe.c
+++ b/IntelSiliconPkg/IntelVTdDxe/IntelVTdDxe.c
@@ -227,6 +227,7 @@ VTdSetAttribute (
   EFI_STATUS           Status;
   UINT16               Segment;
   VTD_SOURCE_ID        SourceId;
+  CHAR8                PerfToken[sizeof("VTD(S0000.B00.D00.F00)")];
 
   DumpVtdIfError ();
 
@@ -239,8 +240,17 @@ VTdSetAttribute (
   DEBUG ((DEBUG_VERBOSE, "PCI(S%x.B%x.D%x.F%x) ", Segment, SourceId.Bits.Bus, SourceId.Bits.Device, SourceId.Bits.Function));
   DEBUG ((DEBUG_VERBOSE, "(0x%lx~0x%lx) - %lx\n", DeviceAddress, Length, IoMmuAccess));
 
+  PERF_CODE_BEGIN();
+    AsciiSPrint (PerfToken, sizeof(PerfToken), "VTD(S%04x.B%02x.D%02x.F%02x)", Segment, SourceId.Bits.Bus, SourceId.Bits.Device, SourceId.Bits.Function);
+    PERF_START (gImageHandle, PerfToken, "IntelVTD", 0);
+  PERF_CODE_END();
+
   Status = SetAccessAttribute (Segment, SourceId, DeviceAddress, Length, IoMmuAccess);
 
+  PERF_CODE_BEGIN();
+    PERF_END (gImageHandle, PerfToken, "IntelVTD", 0);
+  PERF_CODE_END();
+
   return Status;
 }
 
diff --git a/IntelSiliconPkg/IntelVTdDxe/IntelVTdDxe.inf b/IntelSiliconPkg/IntelVTdDxe/IntelVTdDxe.inf
index 6a61c13..b8ac3cf 100644
--- a/IntelSiliconPkg/IntelVTdDxe/IntelVTdDxe.inf
+++ b/IntelSiliconPkg/IntelVTdDxe/IntelVTdDxe.inf
@@ -45,6 +45,7 @@
 [Packages]
   MdePkg/MdePkg.dec
   MdeModulePkg/MdeModulePkg.dec
+  UefiCpuPkg/UefiCpuPkg.dec
   IntelSiliconPkg/IntelSiliconPkg.dec
 
 [LibraryClasses]
@@ -57,6 +58,9 @@
   BaseMemoryLib
   MemoryAllocationLib
   UefiLib
+  CpuLib
+  PerformanceLib
+  PrintLib
 
 [Guids]
   gEfiEventExitBootServicesGuid   ## CONSUMES ## Event
diff --git a/IntelSiliconPkg/IntelVTdDxe/PciInfo.c b/IntelSiliconPkg/IntelVTdDxe/PciInfo.c
index d5f096f..27e253d 100644
--- a/IntelSiliconPkg/IntelVTdDxe/PciInfo.c
+++ b/IntelSiliconPkg/IntelVTdDxe/PciInfo.c
@@ -77,6 +77,7 @@ RegisterPciDevice (
   UINTN                   Index;
   BOOLEAN                 *NewIsRealPciDevice;
   VTD_SOURCE_ID           *NewPciDescriptors;
+  UINTN                   *NewAccessCount;
 
   PciDeviceInfo = &mVtdUnitInformation[VtdIndex].PciDeviceInfo;
 
@@ -112,6 +113,12 @@ RegisterPciDevice (
         FreePool (NewIsRealPciDevice);
         return EFI_OUT_OF_RESOURCES;
       }
+      NewAccessCount = AllocateZeroPool (sizeof(*NewAccessCount) * (PciDeviceInfo->PciDescriptorMaxNumber + MAX_PCI_DESCRIPTORS));
+      if (NewAccessCount == NULL) {
+        FreePool (NewIsRealPciDevice);
+        FreePool (NewPciDescriptors);
+        return EFI_OUT_OF_RESOURCES;
+      }
       PciDeviceInfo->PciDescriptorMaxNumber += MAX_PCI_DESCRIPTORS;
       if (PciDeviceInfo->IsRealPciDevice != NULL) {
         CopyMem (NewIsRealPciDevice, PciDeviceInfo->IsRealPciDevice, sizeof(*NewIsRealPciDevice) * PciDeviceInfo->PciDescriptorNumber);
@@ -123,6 +130,11 @@ RegisterPciDevice (
         FreePool (PciDeviceInfo->PciDescriptors);
       }
       PciDeviceInfo->PciDescriptors = NewPciDescriptors;
+      if (PciDeviceInfo->AccessCount != NULL) {
+        CopyMem (NewAccessCount, PciDeviceInfo->AccessCount, sizeof(*NewAccessCount) * PciDeviceInfo->PciDescriptorNumber);
+        FreePool (PciDeviceInfo->AccessCount);
+      }
+      PciDeviceInfo->AccessCount = NewAccessCount;
     }
 
     ASSERT (PciDeviceInfo->PciDescriptorNumber < PciDeviceInfo->PciDescriptorMaxNumber);
diff --git a/IntelSiliconPkg/IntelVTdDxe/TranslationTable.c b/IntelSiliconPkg/IntelVTdDxe/TranslationTable.c
index 961d7ca..80fc823 100644
--- a/IntelSiliconPkg/IntelVTdDxe/TranslationTable.c
+++ b/IntelSiliconPkg/IntelVTdDxe/TranslationTable.c
@@ -124,6 +124,7 @@ CreateContextEntry (
       RootEntry->Bits.ContextTablePointerHi  = (UINT32) RShiftU64 ((UINT64)(UINTN)Buffer, 32);
       RootEntry->Bits.Present = 1;
       Buffer = (UINT8 *)Buffer + EFI_PAGES_TO_SIZE (ContextPages);
+      FlushPageTableMemory (VtdIndex, (UINTN)RootEntry, sizeof(*RootEntry));
     }
 
     ContextEntryTable = (VTD_CONTEXT_ENTRY *)(UINTN)VTD_64BITS_ADDRESS(RootEntry->Bits.ContextTablePointerLo, RootEntry->Bits.ContextTablePointerHi) ;
@@ -142,6 +143,7 @@ CreateContextEntry (
       ContextEntry->Bits.AddressWidth = 0x2;
       break;
     }
+    FlushPageTableMemory (VtdIndex, (UINTN)ContextEntry, sizeof(*ContextEntry));
   }
 
   return EFI_SUCCESS;
@@ -250,8 +252,11 @@ CreateSecondLevelPagingEntryTable (
           goto Done;
         }
       }
+      FlushPageTableMemory (VtdIndex, (UINTN)Lvl2PtEntry, SIZE_4KB);
     }
+    FlushPageTableMemory (VtdIndex, (UINTN)&Lvl3PtEntry[Lvl3Start], (UINTN)&Lvl3PtEntry[Lvl3End + 1] - (UINTN)&Lvl3PtEntry[Lvl3Start]);
   }
+  FlushPageTableMemory (VtdIndex, (UINTN)&Lvl4PtEntry[Lvl4Start], (UINTN)&Lvl4PtEntry[Lvl4End + 1] - (UINTN)&Lvl4PtEntry[Lvl4Start]);
 
 Done:
   return SecondLevelPagingEntry;
@@ -429,9 +434,10 @@ InvalidatePageEntry (
   IN UINTN                 VtdIndex
   )
 {
-  if (mVtdUnitInformation[VtdIndex].HasDirtyPages) {
+  if (mVtdUnitInformation[VtdIndex].HasDirtyContext || mVtdUnitInformation[VtdIndex].HasDirtyPages) {
     InvalidateVtdIOTLBGlobal (VtdIndex);
   }
+  mVtdUnitInformation[VtdIndex].HasDirtyContext = FALSE;
   mVtdUnitInformation[VtdIndex].HasDirtyPages = FALSE;
 }
 
@@ -498,6 +504,7 @@ PageAttributeToLength (
 /**
   Return page table entry to match the address.
 
+  @param[in]   VtdIndex                 The index used to identify a VTd engine.
   @param[in]   SecondLevelPagingEntry   The second level paging entry in VTd table for the device.
   @param[in]   Address                  The address to be checked.
   @param[out]  PageAttributes           The page attribute of the page entry.
@@ -506,6 +513,7 @@ PageAttributeToLength (
 **/
 VOID *
 GetSecondLevelPageTableEntry (
+  IN  UINTN                         VtdIndex,
   IN  VTD_SECOND_LEVEL_PAGING_ENTRY *SecondLevelPagingEntry,
   IN  PHYSICAL_ADDRESS              Address,
   OUT PAGE_ATTRIBUTE                *PageAttribute
@@ -535,6 +543,7 @@ GetSecondLevelPageTableEntry (
       return NULL;
     }
     SetSecondLevelPagingEntryAttribute ((VTD_SECOND_LEVEL_PAGING_ENTRY *)&L4PageTable[Index4], EDKII_IOMMU_ACCESS_READ | EDKII_IOMMU_ACCESS_WRITE);
+    FlushPageTableMemory (VtdIndex, (UINTN)&L4PageTable[Index4], sizeof(L4PageTable[Index4]));
   }
 
   L3PageTable = (UINT64 *)(UINTN)(L4PageTable[Index4] & PAGING_4K_ADDRESS_MASK_64);
@@ -547,6 +556,7 @@ GetSecondLevelPageTableEntry (
       return NULL;
     }
     SetSecondLevelPagingEntryAttribute ((VTD_SECOND_LEVEL_PAGING_ENTRY *)&L3PageTable[Index3], EDKII_IOMMU_ACCESS_READ | EDKII_IOMMU_ACCESS_WRITE);
+    FlushPageTableMemory (VtdIndex, (UINTN)&L3PageTable[Index3], sizeof(L3PageTable[Index3]));
   }
   if ((L3PageTable[Index3] & VTD_PG_PS) != 0) {
     // 1G
@@ -559,6 +569,7 @@ GetSecondLevelPageTableEntry (
     L2PageTable[Index2] = Address & PAGING_2M_ADDRESS_MASK_64;
     SetSecondLevelPagingEntryAttribute ((VTD_SECOND_LEVEL_PAGING_ENTRY *)&L2PageTable[Index2], 0);
     L2PageTable[Index2] |= VTD_PG_PS;
+    FlushPageTableMemory (VtdIndex, (UINTN)&L2PageTable[Index2], sizeof(L2PageTable[Index2]));
   }
   if ((L2PageTable[Index2] & VTD_PG_PS) != 0) {
     // 2M
@@ -579,12 +590,14 @@ GetSecondLevelPageTableEntry (
 /**
   Modify memory attributes of page entry.
 
+  @param[in]   VtdIndex         The index used to identify a VTd engine.
   @param[in]   PageEntry        The page entry.
   @param[in]   IoMmuAccess      The IOMMU access.
   @param[out]  IsModified       TRUE means page table modified. FALSE means page table not modified.
 **/
 VOID
 ConvertSecondLevelPageEntryAttribute (
+  IN  UINTN                             VtdIndex,
   IN  VTD_SECOND_LEVEL_PAGING_ENTRY     *PageEntry,
   IN  UINT64                            IoMmuAccess,
   OUT BOOLEAN                           *IsModified
@@ -595,6 +608,7 @@ ConvertSecondLevelPageEntryAttribute (
 
   CurrentPageEntry = PageEntry->Uint64;
   SetSecondLevelPagingEntryAttribute (PageEntry, IoMmuAccess);
+  FlushPageTableMemory (VtdIndex, (UINTN)PageEntry, sizeof(*PageEntry));
   NewPageEntry = PageEntry->Uint64;
   if (CurrentPageEntry != NewPageEntry) {
     *IsModified = TRUE;
@@ -639,6 +653,7 @@ NeedSplitPage (
 /**
   This function splits one page entry to small page entries.
 
+  @param[in]  VtdIndex         The index used to identify a VTd engine.
   @param[in]  PageEntry        The page entry to be splitted.
   @param[in]  PageAttribute    The page attribute of the page entry.
   @param[in]  SplitAttribute   How to split the page entry.
@@ -649,6 +664,7 @@ NeedSplitPage (
 **/
 RETURN_STATUS
 SplitSecondLevelPage (
+  IN  UINTN                             VtdIndex,
   IN  VTD_SECOND_LEVEL_PAGING_ENTRY     *PageEntry,
   IN  PAGE_ATTRIBUTE                    PageAttribute,
   IN  PAGE_ATTRIBUTE                    SplitAttribute
@@ -675,8 +691,11 @@ SplitSecondLevelPage (
       for (Index = 0; Index < SIZE_4KB / sizeof(UINT64); Index++) {
         NewPageEntry[Index] = (BaseAddress + SIZE_4KB * Index) | (PageEntry->Uint64 & PAGE_PROGATE_BITS);
       }
+      FlushPageTableMemory (VtdIndex, (UINTN)NewPageEntry, SIZE_4KB);
+
       PageEntry->Uint64 = (UINT64)(UINTN)NewPageEntry;
       SetSecondLevelPagingEntryAttribute (PageEntry, EDKII_IOMMU_ACCESS_READ | EDKII_IOMMU_ACCESS_WRITE);
+      FlushPageTableMemory (VtdIndex, (UINTN)PageEntry, sizeof(*PageEntry));
       return RETURN_SUCCESS;
     } else {
       return RETURN_UNSUPPORTED;
@@ -697,8 +716,11 @@ SplitSecondLevelPage (
       for (Index = 0; Index < SIZE_4KB / sizeof(UINT64); Index++) {
         NewPageEntry[Index] = (BaseAddress + SIZE_2MB * Index) | VTD_PG_PS | (PageEntry->Uint64 & PAGE_PROGATE_BITS);
       }
+      FlushPageTableMemory (VtdIndex, (UINTN)NewPageEntry, SIZE_4KB);
+
       PageEntry->Uint64 = (UINT64)(UINTN)NewPageEntry;
       SetSecondLevelPagingEntryAttribute (PageEntry, EDKII_IOMMU_ACCESS_READ | EDKII_IOMMU_ACCESS_WRITE);
+      FlushPageTableMemory (VtdIndex, (UINTN)PageEntry, sizeof(*PageEntry));
       return RETURN_SUCCESS;
     } else {
       return RETURN_UNSUPPORTED;
@@ -730,6 +752,7 @@ SplitSecondLevelPage (
 EFI_STATUS
 SetSecondLevelPagingAttribute (
   IN UINTN                         VtdIndex,
+  IN UINT16                        DomainIdentifier,
   IN VTD_SECOND_LEVEL_PAGING_ENTRY *SecondLevelPagingEntry,
   IN UINT64                        BaseAddress,
   IN UINT64                        Length,
@@ -756,7 +779,7 @@ SetSecondLevelPagingAttribute (
   }
 
   while (Length != 0) {
-    PageEntry = GetSecondLevelPageTableEntry (SecondLevelPagingEntry, BaseAddress, &PageAttribute);
+    PageEntry = GetSecondLevelPageTableEntry (VtdIndex, SecondLevelPagingEntry, BaseAddress, &PageAttribute);
     if (PageEntry == NULL) {
       DEBUG ((DEBUG_ERROR, "PageEntry - NULL\n"));
       return RETURN_UNSUPPORTED;
@@ -764,7 +787,7 @@ SetSecondLevelPagingAttribute (
     PageEntryLength = PageAttributeToLength (PageAttribute);
     SplitAttribute = NeedSplitPage (BaseAddress, Length, PageAttribute);
     if (SplitAttribute == PageNone) {
-      ConvertSecondLevelPageEntryAttribute (PageEntry, IoMmuAccess, &IsEntryModified);
+      ConvertSecondLevelPageEntryAttribute (VtdIndex, PageEntry, IoMmuAccess, &IsEntryModified);
       if (IsEntryModified) {
         mVtdUnitInformation[VtdIndex].HasDirtyPages = TRUE;
       }
@@ -774,7 +797,7 @@ SetSecondLevelPagingAttribute (
       BaseAddress += PageEntryLength;
       Length -= PageEntryLength;
     } else {
-      Status = SplitSecondLevelPage (PageEntry, PageAttribute, SplitAttribute);
+      Status = SplitSecondLevelPage (VtdIndex, PageEntry, PageAttribute, SplitAttribute);
       if (RETURN_ERROR (Status)) {
         DEBUG ((DEBUG_ERROR, "SplitSecondLevelPage - %r\n", Status));
         return RETURN_UNSUPPORTED;
@@ -787,8 +810,6 @@ SetSecondLevelPagingAttribute (
     }
   }
 
-  InvalidatePageEntry (VtdIndex);
-
   return EFI_SUCCESS;
 }
 
@@ -814,6 +835,7 @@ SetSecondLevelPagingAttribute (
 EFI_STATUS
 SetPageAttribute (
   IN UINTN                         VtdIndex,
+  IN UINT16                        DomainIdentifier,
   IN VTD_SECOND_LEVEL_PAGING_ENTRY *SecondLevelPagingEntry,
   IN UINT64                        BaseAddress,
   IN UINT64                        Length,
@@ -823,7 +845,7 @@ SetPageAttribute (
   EFI_STATUS Status;
   Status = EFI_NOT_FOUND;
   if (SecondLevelPagingEntry != NULL) {
-    Status = SetSecondLevelPagingAttribute (VtdIndex, SecondLevelPagingEntry, BaseAddress, Length, IoMmuAccess);
+    Status = SetSecondLevelPagingAttribute (VtdIndex, DomainIdentifier, SecondLevelPagingEntry, BaseAddress, Length, IoMmuAccess);
   }
   return Status;
 }
@@ -862,6 +884,8 @@ SetAccessAttribute (
   VTD_CONTEXT_ENTRY             *ContextEntry;
   VTD_SECOND_LEVEL_PAGING_ENTRY *SecondLevelPagingEntry;
   UINT64                        Pt;
+  UINTN                         PciDescriptorIndex;
+  UINT16                        DomainIdentifier;
 
   SecondLevelPagingEntry = NULL;
 
@@ -873,6 +897,13 @@ SetAccessAttribute (
     return EFI_DEVICE_ERROR;
   }
 
+  PciDescriptorIndex = GetPciDescriptor (VtdIndex, Segment, SourceId);
+  mVtdUnitInformation[VtdIndex].PciDeviceInfo.AccessCount[PciDescriptorIndex]++;
+  //
+  // DomainId should not be 0.
+  //
+  DomainIdentifier = (UINT16)(PciDescriptorIndex + 1);
+
   if (ExtContextEntry != NULL) {
     if (ExtContextEntry->Bits.Present == 0) {
       SecondLevelPagingEntry = CreateSecondLevelPagingEntry (VtdIndex, 0);
@@ -881,9 +912,11 @@ SetAccessAttribute (
 
       ExtContextEntry->Bits.SecondLevelPageTranslationPointerLo = (UINT32) Pt;
       ExtContextEntry->Bits.SecondLevelPageTranslationPointerHi = (UINT32) RShiftU64(Pt, 20);
-      ExtContextEntry->Bits.DomainIdentifier = (UINT16) GetPciDescriptor (VtdIndex, Segment, SourceId);
+      ExtContextEntry->Bits.DomainIdentifier = DomainIdentifier;
       ExtContextEntry->Bits.Present = 1;
+      FlushPageTableMemory (VtdIndex, (UINTN)ExtContextEntry, sizeof(*ExtContextEntry));
       DumpDmarExtContextEntryTable (mVtdUnitInformation[VtdIndex].ExtRootEntryTable);
+      mVtdUnitInformation[VtdIndex].HasDirtyContext = TRUE;
     } else {
       SecondLevelPagingEntry = (VOID *)(UINTN)VTD_64BITS_ADDRESS(ExtContextEntry->Bits.SecondLevelPageTranslationPointerLo, ExtContextEntry->Bits.SecondLevelPageTranslationPointerHi);
       DEBUG ((DEBUG_VERBOSE,"SecondLevelPagingEntry - 0x%x (S%04x B%02x D%02x F%02x)\n", SecondLevelPagingEntry, Segment, SourceId.Bits.Bus, SourceId.Bits.Device, SourceId.Bits.Function));
@@ -896,9 +929,11 @@ SetAccessAttribute (
 
       ContextEntry->Bits.SecondLevelPageTranslationPointerLo = (UINT32) Pt;
       ContextEntry->Bits.SecondLevelPageTranslationPointerHi = (UINT32) RShiftU64(Pt, 20);
-      ContextEntry->Bits.DomainIdentifier = (UINT16) GetPciDescriptor (VtdIndex, Segment, SourceId);
+      ContextEntry->Bits.DomainIdentifier = DomainIdentifier;
       ContextEntry->Bits.Present = 1;
+      FlushPageTableMemory (VtdIndex, (UINTN)ContextEntry, sizeof(*ContextEntry));
       DumpDmarContextEntryTable (mVtdUnitInformation[VtdIndex].RootEntryTable);
+      mVtdUnitInformation[VtdIndex].HasDirtyContext = TRUE;
     } else {
       SecondLevelPagingEntry = (VOID *)(UINTN)VTD_64BITS_ADDRESS(ContextEntry->Bits.SecondLevelPageTranslationPointerLo, ContextEntry->Bits.SecondLevelPageTranslationPointerHi);
       DEBUG ((DEBUG_VERBOSE,"SecondLevelPagingEntry - 0x%x (S%04x B%02x D%02x F%02x)\n", SecondLevelPagingEntry, Segment, SourceId.Bits.Bus, SourceId.Bits.Device, SourceId.Bits.Function));
@@ -911,6 +946,7 @@ SetAccessAttribute (
   if (SecondLevelPagingEntry != mVtdUnitInformation[VtdIndex].FixedSecondLevelPagingEntry) {
     Status = SetPageAttribute (
                VtdIndex,
+               DomainIdentifier,
                SecondLevelPagingEntry,
                BaseAddress,
                Length,
@@ -922,6 +958,8 @@ SetAccessAttribute (
     }
   }
 
+  InvalidatePageEntry (VtdIndex);
+
   return EFI_SUCCESS;
 }
 
@@ -965,11 +1003,13 @@ AlwaysEnablePageAttribute (
     ExtContextEntry->Bits.SecondLevelPageTranslationPointerHi = (UINT32) RShiftU64(Pt, 20);
     ExtContextEntry->Bits.DomainIdentifier = ((1 << (UINT8)((UINTN)mVtdUnitInformation[VtdIndex].CapReg.Bits.ND * 2 + 4)) - 1);
     ExtContextEntry->Bits.Present = 1;
+    FlushPageTableMemory (VtdIndex, (UINTN)ExtContextEntry, sizeof(*ExtContextEntry));
   } else if (ContextEntry != NULL) {
     ContextEntry->Bits.SecondLevelPageTranslationPointerLo = (UINT32) Pt;
     ContextEntry->Bits.SecondLevelPageTranslationPointerHi = (UINT32) RShiftU64(Pt, 20);
     ContextEntry->Bits.DomainIdentifier = ((1 << (UINT8)((UINTN)mVtdUnitInformation[VtdIndex].CapReg.Bits.ND * 2 + 4)) - 1);
     ContextEntry->Bits.Present = 1;
+    FlushPageTableMemory (VtdIndex, (UINTN)ContextEntry, sizeof(*ContextEntry));
   }
 
   return EFI_SUCCESS;
diff --git a/IntelSiliconPkg/IntelVTdDxe/TranslationTableEx.c b/IntelSiliconPkg/IntelVTdDxe/TranslationTableEx.c
index 65ed16e..9d4e6ea 100644
--- a/IntelSiliconPkg/IntelVTdDxe/TranslationTableEx.c
+++ b/IntelSiliconPkg/IntelVTdDxe/TranslationTableEx.c
@@ -73,6 +73,7 @@ CreateExtContextEntry (
       ExtRootEntry->Bits.UpperContextTablePointerLo  = (UINT32) RShiftU64 ((UINT64)(UINTN)Buffer, 12) + 1;
       ExtRootEntry->Bits.UpperContextTablePointerHi  = (UINT32) RShiftU64 (RShiftU64 ((UINT64)(UINTN)Buffer, 12) + 1, 20);
       ExtRootEntry->Bits.UpperPresent = 1;
+      FlushPageTableMemory (VtdIndex, (UINTN)ExtRootEntry, sizeof(*ExtRootEntry));
       Buffer = (UINT8 *)Buffer + EFI_PAGES_TO_SIZE (ContextPages);
     }
 
@@ -92,6 +93,7 @@ CreateExtContextEntry (
       ExtContextEntry->Bits.AddressWidth = 0x2;
       break;
     }
+    FlushPageTableMemory (VtdIndex, (UINTN)ExtContextEntry, sizeof(*ExtContextEntry));
   }
 
   return EFI_SUCCESS;
diff --git a/IntelSiliconPkg/IntelVTdDxe/VtdReg.c b/IntelSiliconPkg/IntelVTdDxe/VtdReg.c
index f36e3de..d95a8a7 100644
--- a/IntelSiliconPkg/IntelVTdDxe/VtdReg.c
+++ b/IntelSiliconPkg/IntelVTdDxe/VtdReg.c
@@ -20,145 +20,105 @@ VTD_UNIT_INFORMATION             *mVtdUnitInformation;
 BOOLEAN  mVtdEnabled;
 
 /**
-  Invalid VTd global IOTLB.
+  Flush VTD page table and context table memory.
 
-  @param[in]  VtdIndex              The index of VTd engine.
+  This action is to make sure the IOMMU engine can get final data in memory.
 
-  @retval EFI_SUCCESS           VTd global IOTLB is invalidated.
-  @retval EFI_DEVICE_ERROR      VTd global IOTLB is not invalidated.
+  @param[in]  VtdIndex          The index used to identify a VTd engine.
+  @param[in]  Base              The base address of memory to be flushed.
+  @param[in]  Size              The size of memory in bytes to be flushed.
 **/
-EFI_STATUS
-InvalidateVtdIOTLBGlobal (
-  IN UINTN  VtdIndex
+VOID
+FlushPageTableMemory (
+  IN UINTN  VtdIndex,
+  IN UINTN  Base,
+  IN UINTN  Size
   )
 {
-  UINT64  Reg64;
-  UINT32  Reg32;
+  UINTN  Address;
+  UINTN  AddressEnd;
 
-  if (!mVtdEnabled) {
-    return EFI_SUCCESS;
+  if (mVtdUnitInformation[VtdIndex].ECapReg.Bits.C == 0) {
+    Address = Base & ~(mCacheLineSize - 1);
+    AddressEnd = Base + Size;
+
+    for (; Address <= AddressEnd; Address += mCacheLineSize) {
+      CacheLineFlush (Address);
+    }
   }
+}
 
-  DEBUG((DEBUG_VERBOSE, "InvalidateVtdIOTLBGlobal(%d)\n", VtdIndex));
+/**
+  Flush VTd engine write buffer.
 
-  AsmWbinvd();
+  @param[in]  VtdIndex          The index used to identify a VTd engine.
+**/
+VOID
+FlushWriteBuffer (
+  IN UINTN  VtdIndex
+  )
+{
+  UINT32  Reg32;
 
-  //
-  // Write Buffer Flush before invalidation
-  //
-  Reg32 = MmioRead32 (mVtdUnitInformation[VtdIndex].VtdUnitBaseAddress + R_CAP_REG);
-  if ((Reg32 & B_CAP_REG_RWBF) != 0) {
-    MmioWrite32 (mVtdUnitInformation[VtdIndex].VtdUnitBaseAddress + R_GCMD_REG, B_GMCD_REG_WBF);
+  if (mVtdUnitInformation[VtdIndex].CapReg.Bits.RWBF != 0) {
+    Reg32 = MmioRead32 (mVtdUnitInformation[VtdIndex].VtdUnitBaseAddress + R_GSTS_REG);
+    MmioWrite32 (mVtdUnitInformation[VtdIndex].VtdUnitBaseAddress + R_GCMD_REG, Reg32 | B_GMCD_REG_WBF);
+    do {
+      Reg32 = MmioRead32 (mVtdUnitInformation[VtdIndex].VtdUnitBaseAddress + R_GSTS_REG);
+    } while ((Reg32 & B_GSTS_REG_WBF) != 0);
   }
+}
 
-  //
-  // Invalidate the context cache
-  //
-  Reg64 = MmioRead64 (mVtdUnitInformation[VtdIndex].VtdUnitBaseAddress + R_CCMD_REG);
-  if ((Reg64 & B_CCMD_REG_ICC) != 0) {
-    DEBUG ((DEBUG_ERROR,"ERROR: InvalidateVtdIOTLBGlobal: B_CCMD_REG_ICC is set for VTD(%d)\n",VtdIndex));
-    return EFI_DEVICE_ERROR;
-  }
+/**
+  Invalidate VTd context cache.
 
-  Reg64 &= ((~B_CCMD_REG_ICC) & (~B_CCMD_REG_CIRG_MASK));
-  Reg64 |= (B_CCMD_REG_ICC | V_CCMD_REG_CIRG_GLOBAL);
-  MmioWrite64 (mVtdUnitInformation[VtdIndex].VtdUnitBaseAddress + R_CCMD_REG, Reg64);
+  @param[in]  VtdIndex          The index used to identify a VTd engine.
+**/
+EFI_STATUS
+InvalidateContextCache (
+  IN UINTN  VtdIndex
+  )
+{
+  UINT64  Reg64;
 
-  do {
+  if (mVtdUnitInformation[VtdIndex].HasDirtyContext) {
     Reg64 = MmioRead64 (mVtdUnitInformation[VtdIndex].VtdUnitBaseAddress + R_CCMD_REG);
-  } while ((Reg64 & B_CCMD_REG_ICC) != 0);
+    if ((Reg64 & B_CCMD_REG_ICC) != 0) {
+      DEBUG ((DEBUG_ERROR,"ERROR: InvalidateContextCache: B_CCMD_REG_ICC is set for VTD(%d)\n",VtdIndex));
+      return EFI_DEVICE_ERROR;
+    }
 
-  //
-  // Invalidate the IOTLB cache
-  //
+    Reg64 &= ((~B_CCMD_REG_ICC) & (~B_CCMD_REG_CIRG_MASK));
+    Reg64 |= (B_CCMD_REG_ICC | V_CCMD_REG_CIRG_GLOBAL);
+    MmioWrite64 (mVtdUnitInformation[VtdIndex].VtdUnitBaseAddress + R_CCMD_REG, Reg64);
 
-  Reg64 = MmioRead64 (mVtdUnitInformation[VtdIndex].VtdUnitBaseAddress + (mVtdUnitInformation[VtdIndex].ECapReg.Bits.IRO * 16) + R_IOTLB_REG);
-  if ((Reg64 & B_IOTLB_REG_IVT) != 0) {
-    DEBUG ((DEBUG_ERROR,"ERROR: InvalidateVtdIOTLBGlobal: B_IOTLB_REG_IVT is set for VTD(%d)\n", VtdIndex));
-    return EFI_DEVICE_ERROR;
+    do {
+      Reg64 = MmioRead64 (mVtdUnitInformation[VtdIndex].VtdUnitBaseAddress + R_CCMD_REG);
+    } while ((Reg64 & B_CCMD_REG_ICC) != 0);
   }
-
-  Reg64 &= ((~B_IOTLB_REG_IVT) & (~B_IOTLB_REG_IIRG_MASK));
-  Reg64 |= (B_IOTLB_REG_IVT | V_IOTLB_REG_IIRG_GLOBAL);
-  MmioWrite64 (mVtdUnitInformation[VtdIndex].VtdUnitBaseAddress + (mVtdUnitInformation[VtdIndex].ECapReg.Bits.IRO * 16) + R_IOTLB_REG, Reg64);
-
-  do {
-    Reg64 = MmioRead64 (mVtdUnitInformation[VtdIndex].VtdUnitBaseAddress + (mVtdUnitInformation[VtdIndex].ECapReg.Bits.IRO * 16) + R_IOTLB_REG);
-  } while ((Reg64 & B_IOTLB_REG_IVT) != 0);
-
-  //
-  // Disable VTd
-  //
-  MmioWrite32 (mVtdUnitInformation[VtdIndex].VtdUnitBaseAddress + R_GCMD_REG, B_GMCD_REG_SRTP);
-  do {
-    Reg32 = MmioRead32 (mVtdUnitInformation[VtdIndex].VtdUnitBaseAddress + R_GSTS_REG);
-  } while((Reg32 & B_GSTS_REG_RTPS) == 0);
-
-  //
-  // Enable VTd
-  //
-  MmioWrite32 (mVtdUnitInformation[VtdIndex].VtdUnitBaseAddress + R_GCMD_REG, B_GMCD_REG_TE);
-  do {
-    Reg32 = MmioRead32 (mVtdUnitInformation[VtdIndex].VtdUnitBaseAddress + R_GSTS_REG);
-  } while ((Reg32 & B_GSTS_REG_TE) == 0);
-
   return EFI_SUCCESS;
 }
 
 /**
-  Invalid VTd IOTLB domain.
+  Invalidate VTd IOTLB.
 
-  @param[in]  VtdIndex              The index of VTd engine.
-  @param[in]  DomainIdentifier      The domain ID of the source.
-
-  @retval EFI_SUCCESS           VTd IOTLB domain is invalidated.
-  @retval EFI_DEVICE_ERROR      VTd IOTLB domain is not invalidated.
+  @param[in]  VtdIndex          The index used to identify a VTd engine.
 **/
 EFI_STATUS
-InvalidateVtdIOTLBDomain (
-  IN UINTN  VtdIndex,
-  IN UINT16 DomainIdentifier
+InvalidateIOTLB (
+  IN UINTN  VtdIndex
   )
 {
   UINT64  Reg64;
 
-  if (!mVtdEnabled) {
-    return EFI_SUCCESS;
-  }
-
-  DEBUG((DEBUG_VERBOSE, "InvalidateVtdIOTLBDomain(%d): 0x%016lx (0x%04x)\n", VtdIndex, DomainIdentifier));
-
-  //
-  // Invalidate the context cache
-  //
-  Reg64 = MmioRead64 (mVtdUnitInformation[VtdIndex].VtdUnitBaseAddress + R_CCMD_REG);
-  if ((Reg64 & B_CCMD_REG_ICC) != 0) {
-    DEBUG ((DEBUG_ERROR,"ERROR: InvalidateVtdIOTLBDomain: B_CCMD_REG_ICC is set for VTD(%d)\n",VtdIndex));
-    return EFI_DEVICE_ERROR;
-  }
-
-  Reg64 &= ((~B_CCMD_REG_ICC) & (~B_CCMD_REG_CIRG_MASK));
-  Reg64 |= (B_CCMD_REG_ICC | V_CCMD_REG_CIRG_DOMAIN);
-  Reg64 |= (B_CCMD_REG_ICC | V_CCMD_REG_CIRG_DOMAIN);
-  MmioWrite64 (mVtdUnitInformation[VtdIndex].VtdUnitBaseAddress + R_CCMD_REG, Reg64);
-
-  do {
-    Reg64 = MmioRead64 (mVtdUnitInformation[VtdIndex].VtdUnitBaseAddress + R_CCMD_REG);
-  } while ((Reg64 & B_CCMD_REG_ICC) != 0);
-
-  //
-  // Invalidate the IOTLB cache
-  //
-
   Reg64 = MmioRead64 (mVtdUnitInformation[VtdIndex].VtdUnitBaseAddress + (mVtdUnitInformation[VtdIndex].ECapReg.Bits.IRO * 16) + R_IOTLB_REG);
   if ((Reg64 & B_IOTLB_REG_IVT) != 0) {
-    DEBUG ((DEBUG_ERROR,"ERROR: InvalidateVtdIOTLBDomain: B_IOTLB_REG_IVT is set for VTD(%d)\n", VtdIndex));
+    DEBUG ((DEBUG_ERROR,"ERROR: InvalidateIOTLB: B_IOTLB_REG_IVT is set for VTD(%d)\n", VtdIndex));
     return EFI_DEVICE_ERROR;
   }
 
   Reg64 &= ((~B_IOTLB_REG_IVT) & (~B_IOTLB_REG_IIRG_MASK));
-  Reg64 |= (B_IOTLB_REG_IVT | V_IOTLB_REG_IIRG_DOMAIN);
-  Reg64 |= LShiftU64 (DomainIdentifier, 32);
+  Reg64 |= (B_IOTLB_REG_IVT | V_IOTLB_REG_IIRG_GLOBAL);
   MmioWrite64 (mVtdUnitInformation[VtdIndex].VtdUnitBaseAddress + (mVtdUnitInformation[VtdIndex].ECapReg.Bits.IRO * 16) + R_IOTLB_REG, Reg64);
 
   do {
@@ -169,53 +129,41 @@ InvalidateVtdIOTLBDomain (
 }
 
 /**
-  Invalid VTd IOTLB page.
+  Invalid VTd global IOTLB.
 
   @param[in]  VtdIndex              The index of VTd engine.
-  @param[in]  Address               The address of IOTLB page.
-  @param[in]  AddressMode           The address mode of IOTLB page.
-  @param[in]  DomainIdentifier      The domain ID of the source.
 
-  @retval EFI_SUCCESS           VTd IOTLB page is invalidated.
-  @retval EFI_DEVICE_ERROR      VTd IOTLB page is not invalidated.
+  @retval EFI_SUCCESS           VTd global IOTLB is invalidated.
+  @retval EFI_DEVICE_ERROR      VTd global IOTLB is not invalidated.
 **/
 EFI_STATUS
-InvalidateVtdIOTLBPage (
-  IN UINTN  VtdIndex,
-  IN UINT64 Address,
-  IN UINT8  AddressMode,
-  IN UINT16 DomainIdentifier
+InvalidateVtdIOTLBGlobal (
+  IN UINTN  VtdIndex
   )
 {
-  UINT64  Reg64;
-  UINT64  Data64;
-
   if (!mVtdEnabled) {
     return EFI_SUCCESS;
   }
 
-  DEBUG((DEBUG_VERBOSE, "InvalidateVtdIOTLBPage(%d): 0x%016lx (0x%02x)\n", VtdIndex, Address, AddressMode));
-
-  if (mVtdUnitInformation[VtdIndex].CapReg.Bits.PSI != 0) {
-    Reg64 = MmioRead64 (mVtdUnitInformation[VtdIndex].VtdUnitBaseAddress + (mVtdUnitInformation[VtdIndex].ECapReg.Bits.IRO * 16) + R_IOTLB_REG);
-    if ((Reg64 & B_IOTLB_REG_IVT) != 0) {
-      DEBUG ((DEBUG_ERROR,"ERROR: InvalidateVtdIOTLBPage: B_IOTLB_REG_IVT is set for VTD(%d)\n", VtdIndex));
-      return EFI_DEVICE_ERROR;
-    }
+  DEBUG((DEBUG_VERBOSE, "InvalidateVtdIOTLBGlobal(%d)\n", VtdIndex));
 
-    Data64 = Address | AddressMode;
-    MmioWrite64 (mVtdUnitInformation[VtdIndex].VtdUnitBaseAddress + (mVtdUnitInformation[VtdIndex].ECapReg.Bits.IRO * 16) + R_IVA_REG, Data64);
+  //
+  // Write Buffer Flush before invalidation
+  //
+  FlushWriteBuffer (VtdIndex);
 
-    Reg64 &= ((~B_IOTLB_REG_IVT) & (~B_IOTLB_REG_IIRG_MASK));
-    Reg64 |= (B_IOTLB_REG_IVT | V_IOTLB_REG_IIRG_PAGE);
-    Reg64 |= LShiftU64 (DomainIdentifier, 32);
-    MmioWrite64 (mVtdUnitInformation[VtdIndex].VtdUnitBaseAddress + (mVtdUnitInformation[VtdIndex].ECapReg.Bits.IRO * 16) + R_IOTLB_REG, Reg64);
+  //
+  // Invalidate the context cache
+  //
+  if (mVtdUnitInformation[VtdIndex].HasDirtyContext) {
+    InvalidateContextCache (VtdIndex);
+  }
 
-    do {
-      Reg64 = MmioRead64 (mVtdUnitInformation[VtdIndex].VtdUnitBaseAddress + (mVtdUnitInformation[VtdIndex].ECapReg.Bits.IRO * 16) + R_IOTLB_REG);
-    } while ((Reg64 & B_IOTLB_REG_IVT) != 0);
-  } else {
-    InvalidateVtdIOTLBGlobal (VtdIndex);
+  //
+  // Invalidate the IOTLB cache
+  //
+  if (mVtdUnitInformation[VtdIndex].HasDirtyContext || mVtdUnitInformation[VtdIndex].HasDirtyPages) {
+    InvalidateIOTLB (VtdIndex);
   }
 
   return EFI_SUCCESS;
@@ -268,11 +216,8 @@ EnableDmar (
   )
 {
   UINTN     Index;
-  UINT64    Reg64;
   UINT32    Reg32;
 
-  AsmWbinvd();
-
   for (Index = 0; Index < mVtdUnitNumber; Index++) {
     DEBUG((DEBUG_INFO, ">>>>>>EnableDmar() for engine [%d] \n", Index));
 
@@ -299,48 +244,17 @@ EnableDmar (
     //
     // Write Buffer Flush before invalidation
     //
-    Reg32 = MmioRead32 (mVtdUnitInformation[Index].VtdUnitBaseAddress + R_CAP_REG);
-    if ((Reg32 & B_CAP_REG_RWBF) != 0) {
-      MmioWrite32 (mVtdUnitInformation[Index].VtdUnitBaseAddress + R_GCMD_REG, B_GMCD_REG_WBF);
-    }
+    FlushWriteBuffer (Index);
 
     //
     // Invalidate the context cache
     //
-    Reg64 = MmioRead64 (mVtdUnitInformation[Index].VtdUnitBaseAddress + R_CCMD_REG);
-    if ((Reg64 & B_CCMD_REG_ICC) != 0) {
-      DEBUG ((DEBUG_INFO,"ERROR: EnableDmar: B_CCMD_REG_ICC is set for VTD(%d)\n",Index));
-      return EFI_DEVICE_ERROR;
-    }
-
-    Reg64 &= ((~B_CCMD_REG_ICC) & (~B_CCMD_REG_CIRG_MASK));
-    Reg64 |= (B_CCMD_REG_ICC | V_CCMD_REG_CIRG_GLOBAL);
-    MmioWrite64 (mVtdUnitInformation[Index].VtdUnitBaseAddress + R_CCMD_REG, Reg64);
-
-    DEBUG((DEBUG_INFO, "EnableDmar: Waiting B_CCMD_REG_ICC ...\n"));
-    do {
-      Reg64 = MmioRead64 (mVtdUnitInformation[Index].VtdUnitBaseAddress + R_CCMD_REG);
-    } while ((Reg64 & B_CCMD_REG_ICC) != 0);
+    InvalidateContextCache (Index);
 
     //
     // Invalidate the IOTLB cache
     //
-    DEBUG((DEBUG_INFO, "EnableDmar: IRO 0x%x\n", mVtdUnitInformation[Index].ECapReg.Bits.IRO));
-
-    Reg64 = MmioRead64 (mVtdUnitInformation[Index].VtdUnitBaseAddress + (mVtdUnitInformation[Index].ECapReg.Bits.IRO * 16) + R_IOTLB_REG);
-    if ((Reg64 & B_IOTLB_REG_IVT) != 0) {
-      DEBUG ((DEBUG_INFO,"ERROR: EnableDmar: B_IOTLB_REG_IVT is set for VTD(%d)\n", Index));
-      return EFI_DEVICE_ERROR;
-    }
-
-    Reg64 &= ((~B_IOTLB_REG_IVT) & (~B_IOTLB_REG_IIRG_MASK));
-    Reg64 |= (B_IOTLB_REG_IVT | V_IOTLB_REG_IIRG_GLOBAL);
-    MmioWrite64 (mVtdUnitInformation[Index].VtdUnitBaseAddress + (mVtdUnitInformation[Index].ECapReg.Bits.IRO * 16) + R_IOTLB_REG, Reg64);
-
-    DEBUG((DEBUG_INFO, "EnableDmar: Waiting B_IOTLB_REG_IVT ...\n"));
-    do {
-      Reg64 = MmioRead64 (mVtdUnitInformation[Index].VtdUnitBaseAddress + (mVtdUnitInformation[Index].ECapReg.Bits.IRO * 16) + R_IOTLB_REG);
-    } while ((Reg64 & B_IOTLB_REG_IVT) != 0);
+    InvalidateIOTLB (Index);
 
     //
     // Enable VTd
@@ -371,20 +285,16 @@ DisableDmar (
   )
 {
   UINTN     Index;
+  UINTN     SubIndex;
   UINT32    Reg32;
 
-  AsmWbinvd();
-
   for (Index = 0; Index < mVtdUnitNumber; Index++) {
     DEBUG((DEBUG_INFO, ">>>>>>DisableDmar() for engine [%d] \n", Index));
 
     //
     // Write Buffer Flush before invalidation
     //
-    Reg32 = MmioRead32 (mVtdUnitInformation[Index].VtdUnitBaseAddress + R_CAP_REG);
-    if ((Reg32 & B_CAP_REG_RWBF) != 0) {
-      MmioWrite32 (mVtdUnitInformation[Index].VtdUnitBaseAddress + R_GCMD_REG, B_GMCD_REG_WBF);
-    }
+    FlushWriteBuffer (Index);
 
     //
     // Disable VTd
@@ -402,6 +312,19 @@ DisableDmar (
 
   mVtdEnabled = FALSE;
 
+  for (Index = 0; Index < mVtdUnitNumber; Index++) {
+    DEBUG((DEBUG_INFO, "engine [%d] access\n", Index));
+    for (SubIndex = 0; SubIndex < mVtdUnitInformation[Index].PciDeviceInfo.PciDescriptorNumber; SubIndex++) {
+      DEBUG ((DEBUG_INFO, "  PCI S%04X B%02x D%02x F%02x - %d\n",
+        mVtdUnitInformation[Index].Segment,
+        mVtdUnitInformation[Index].PciDeviceInfo.PciDescriptors[SubIndex].Bits.Bus,
+        mVtdUnitInformation[Index].PciDeviceInfo.PciDescriptors[SubIndex].Bits.Device,
+        mVtdUnitInformation[Index].PciDeviceInfo.PciDescriptors[SubIndex].Bits.Function,
+        mVtdUnitInformation[Index].PciDeviceInfo.AccessCount[SubIndex]
+        ));
+    }
+  }
+
   return EFI_SUCCESS;
 }
 
-- 
2.7.4.windows.1

Re: [PATCH 0/4] Improve IntelVTd performance.

[Yao, Jiewen]

Sorry for the confusing.

I just sent V2 patch series. Please ignore this one.

Thank you
Yao Jiewen

> -----Original Message-----
> From: edk2-devel [mailto:edk2-devel-bounces@lists.01.org] On Behalf Of Jiewen
> Yao
> Sent: Friday, August 11, 2017 10:10 PM
> To: edk2-devel@lists.01.org
> Subject: [edk2] [PATCH 0/4] Improve IntelVTd performance.
> 
> This series patch replace WBINVD with CLFlush
> in IntelVTdDxe driver, which improves boot performance.
> 
> This patch is validated on Intel Kabylake generation.
> 
> Jiewen Yao (4):
>   MdePkg/CpuLib.h: Add CacheLineFlush function.
>   MdePkg/BaseCpuLib: Add CacheLineFlush function.
>   IntelSiliconPkg/dsc: Add CpuLib.
>   IntelSiliconPkg/IntelVTdDxe: Improve performance.
> 
>  IntelSiliconPkg/IntelSiliconPkg.dsc                |   1 +
>  IntelSiliconPkg/IntelVTdDxe/DmaProtection.c        |   6 +
>  IntelSiliconPkg/IntelVTdDxe/DmaProtection.h        |  61 ++---
>  IntelSiliconPkg/IntelVTdDxe/IntelVTdDxe.c          |  10 +
>  IntelSiliconPkg/IntelVTdDxe/IntelVTdDxe.inf        |   4 +
>  IntelSiliconPkg/IntelVTdDxe/PciInfo.c              |  12 +
>  IntelSiliconPkg/IntelVTdDxe/TranslationTable.c     |  58 +++-
>  IntelSiliconPkg/IntelVTdDxe/TranslationTableEx.c   |   2 +
>  IntelSiliconPkg/IntelVTdDxe/VtdReg.c               | 283
> +++++++-------------
>  MdePkg/Include/Library/CpuLib.h                    |  12 +-
>  MdePkg/Library/BaseCpuLib/BaseCpuLib.inf           |   5 +-
>  MdePkg/Library/BaseCpuLib/Ebc/CpuSleepFlushTlb.c   |  16 +-
>  MdePkg/Library/BaseCpuLib/Ia32/CacheLineFlush.nasm |  37 +++
>  MdePkg/Library/BaseCpuLib/X64/CacheLineFlush.nasm  |  37 +++
>  14 files changed, 318 insertions(+), 226 deletions(-)
>  create mode 100644 MdePkg/Library/BaseCpuLib/Ia32/CacheLineFlush.nasm
>  create mode 100644 MdePkg/Library/BaseCpuLib/X64/CacheLineFlush.nasm
> 
> --
> 2.7.4.windows.1
> 
> _______________________________________________
> edk2-devel mailing list
> edk2-devel@lists.01.org
> https://lists.01.org/mailman/listinfo/edk2-devel