[PATCH V6 37/42] OvmfPkg: Add TdxDxe driver

["Min Xu" <min.m.xu@intel.com>]

RFC: https://bugzilla.tianocore.org/show_bug.cgi?id=3429

TdxDxe driver is dispatched early in DXE, due to being list in APRIORI.
This module is responsible for below features:
 - Sets max logical cpus based on TDINFO
 - Sets PCI PCDs based on resource hobs
 - Set shared bit in MMIO region
 - Relocate Td mailbox and set its address in MADT table.

1. Set shared bit in MMIO region

Qemu allows a ROM device to set to ROMD mode (default) or MMIO mode.
When it is in ROMD mode, the device is mapped to guest memory and
satisfies read access directly.

In EDK2 Option ROM is treated as MMIO region. So Tdx guest access
Option ROM via TDVMCALL(MMIO). But as explained above, since Qemu set
the Option ROM to ROMD mode, the call of TDVMCALL(MMIO) always return
INVALID_OPERAND. Tdvf then falls back to direct access. This requires
to set the shared bit to corresponding PageTable entry. Otherwise it
triggers GP fault.

TdxDxe's entry point is the right place to set the shared bit in MMIO
region because Option ROM has not been discoverd yet.

2. Relocate Td mailbox and set the new address in MADT Mutiprocessor
Wakeup Table.

In TDX the guest firmware is designed to publish a multiprocessor-wakeup
structure to let the guest-bootstrap processor wake up guest-application
processors with a mailbox. The mailbox is memory that the guest firmware
can reserve so each guest virtual processor can have the guest OS send
a message to them. The address of the mailbox is recorded in the MADT
table. See [ACPI].

TdxDxe registers for protocol notification
(gQemuAcpiTableNotifyProtocolGuid) to call the AlterAcpiTable(), in
which MADT table is altered by the above Mailbox address. The protocol
will be installed in AcpiPlatformDxe when the MADT table provided by
Qemu is ready. This is to maintain the simplicity of the AcpiPlatformDxe.

AlterAcpiTable is the registered function which traverses the ACPI
table list to find the original MADT from Qemu. After the new MADT is
configured and installed, the original one will be uninstalled.

[ACPI] https://uefi.org/specs/ACPI/6.4/05_ACPI_Software_Programming_Model
/ACPI_Software_Programming_Model.html#multiprocessor-wakeup-structure

Cc: Ard Biesheuvel <ardb+tianocore@kernel.org>
Cc: Jordan Justen <jordan.l.justen@intel.com>
Cc: Brijesh Singh <brijesh.singh@amd.com>
Cc: Erdem Aktas <erdemaktas@google.com>
Cc: James Bottomley <jejb@linux.ibm.com>
Cc: Jiewen Yao <jiewen.yao@intel.com>
Cc: Tom Lendacky <thomas.lendacky@amd.com>
Cc: Gerd Hoffmann <kraxel@redhat.com>
Acked-by: Gerd Hoffmann <kraxel@redhat.com>
Signed-off-by: Min Xu <min.m.xu@intel.com>
---
 OvmfPkg/OvmfPkg.dec               |   1 +
 OvmfPkg/OvmfPkgX64.dsc            |   2 +
 OvmfPkg/OvmfPkgX64.fdf            |   3 +
 OvmfPkg/TdxDxe/TdxAcpiTable.c     | 213 +++++++++++++++++++++++++
 OvmfPkg/TdxDxe/TdxAcpiTable.h     |  60 +++++++
 OvmfPkg/TdxDxe/TdxDxe.c           | 251 ++++++++++++++++++++++++++++++
 OvmfPkg/TdxDxe/TdxDxe.inf         |  64 ++++++++
 OvmfPkg/TdxDxe/X64/ApRunLoop.nasm |  90 +++++++++++
 8 files changed, 684 insertions(+)
 create mode 100644 OvmfPkg/TdxDxe/TdxAcpiTable.c
 create mode 100644 OvmfPkg/TdxDxe/TdxAcpiTable.h
 create mode 100644 OvmfPkg/TdxDxe/TdxDxe.c
 create mode 100644 OvmfPkg/TdxDxe/TdxDxe.inf
 create mode 100644 OvmfPkg/TdxDxe/X64/ApRunLoop.nasm

diff --git a/OvmfPkg/OvmfPkg.dec b/OvmfPkg/OvmfPkg.dec
index e461e334ea84..86ba5481597e 100644
--- a/OvmfPkg/OvmfPkg.dec
+++ b/OvmfPkg/OvmfPkg.dec
@@ -162,6 +162,7 @@
   gEfiLegacyInterruptProtocolGuid       = {0x31ce593d, 0x108a, 0x485d, {0xad, 0xb2, 0x78, 0xf2, 0x1f, 0x29, 0x66, 0xbe}}
   gEfiVgaMiniPortProtocolGuid           = {0xc7735a2f, 0x88f5, 0x4882, {0xae, 0x63, 0xfa, 0xac, 0x8c, 0x8b, 0x86, 0xb3}}
   gOvmfLoadedX86LinuxKernelProtocolGuid = {0xa3edc05d, 0xb618, 0x4ff6, {0x95, 0x52, 0x76, 0xd7, 0x88, 0x63, 0x43, 0xc8}}
+  gQemuAcpiTableNotifyProtocolGuid      = {0x928939b2, 0x4235, 0x462f, {0x95, 0x80, 0xf6, 0xa2, 0xb2, 0xc2, 0x1a, 0x4f}}
 
 [PcdsFixedAtBuild]
   gUefiOvmfPkgTokenSpaceGuid.PcdOvmfPeiMemFvBase|0x0|UINT32|0
diff --git a/OvmfPkg/OvmfPkgX64.dsc b/OvmfPkg/OvmfPkgX64.dsc
index 21cfeab3b5f9..767c324e5eec 100644
--- a/OvmfPkg/OvmfPkgX64.dsc
+++ b/OvmfPkg/OvmfPkgX64.dsc
@@ -946,6 +946,8 @@
   }
   OvmfPkg/IoMmuDxe/IoMmuDxe.inf
 
+  OvmfPkg/TdxDxe/TdxDxe.inf
+
 !if $(SMM_REQUIRE) == TRUE
   OvmfPkg/SmmAccess/SmmAccess2Dxe.inf
   OvmfPkg/SmmControl2Dxe/SmmControl2Dxe.inf
diff --git a/OvmfPkg/OvmfPkgX64.fdf b/OvmfPkg/OvmfPkgX64.fdf
index ba7f26d01dc4..d7c18bbc6ec1 100644
--- a/OvmfPkg/OvmfPkgX64.fdf
+++ b/OvmfPkg/OvmfPkgX64.fdf
@@ -214,6 +214,7 @@ READ_LOCK_STATUS   = TRUE
 APRIORI DXE {
   INF  MdeModulePkg/Universal/DevicePathDxe/DevicePathDxe.inf
   INF  MdeModulePkg/Universal/PCD/Dxe/Pcd.inf
+  INF  OvmfPkg/TdxDxe/TdxDxe.inf
   INF  OvmfPkg/AmdSevDxe/AmdSevDxe.inf
 !if $(SMM_REQUIRE) == FALSE
   INF  OvmfPkg/QemuFlashFvbServicesRuntimeDxe/FvbServicesRuntimeDxe.inf
@@ -318,6 +319,8 @@ INF  ShellPkg/Application/Shell/Shell.inf
 
 INF MdeModulePkg/Logo/LogoDxe.inf
 
+INF OvmfPkg/TdxDxe/TdxDxe.inf
+
 #
 # Network modules
 #
diff --git a/OvmfPkg/TdxDxe/TdxAcpiTable.c b/OvmfPkg/TdxDxe/TdxAcpiTable.c
new file mode 100644
index 000000000000..8a1abe8b1d89
--- /dev/null
+++ b/OvmfPkg/TdxDxe/TdxAcpiTable.c
@@ -0,0 +1,213 @@
+/** @file
+  OVMF ACPI QEMU support
+
+  Copyright (c) 2008 - 2014, Intel Corporation. All rights reserved.<BR>
+
+  Copyright (C) 2012-2014, Red Hat, Inc.
+
+  SPDX-License-Identifier: BSD-2-Clause-Patent
+
+**/
+
+#include <Library/DebugLib.h>
+#include <Library/BaseLib.h>
+#include <Library/BaseMemoryLib.h>
+#include <Library/MemoryAllocationLib.h>
+#include <Library/QemuFwCfgLib.h>
+#include <Library/DxeServicesTableLib.h>
+#include <Library/PcdLib.h>
+#include <Library/OrderedCollectionLib.h>
+#include <Library/TdxLib.h>
+#include <IndustryStandard/Acpi.h>
+#include <Protocol/AcpiSystemDescriptionTable.h>
+#include <Protocol/AcpiTable.h>
+#include <Library/UefiBootServicesTableLib.h>
+#include <Library/UefiLib.h>
+#include <Library/TdxMailboxLib.h>
+#include <Protocol/Cpu.h>
+#include <Uefi.h>
+#include <TdxAcpiTable.h>
+
+/**
+  At the beginning of system boot, a 4K-aligned, 4K-size memory (Td mailbox) is
+  pre-allocated by host VMM. BSP & APs do the page accept together in that memory
+  region.
+
+  After that TDVF is designed to relocate the mailbox to a 4K-aligned, 4K-size
+  memory block which is allocated in the ACPI Nvs memory. APs are waken up and
+  spin around the relocated mailbox for further command.
+
+  @return   EFI_PHYSICAL_ADDRESS    Address of the relocated mailbox
+**/
+EFI_PHYSICAL_ADDRESS
+EFIAPI
+RelocateMailbox (
+  VOID
+  )
+{
+  EFI_PHYSICAL_ADDRESS  Address;
+  VOID                  *ApLoopFunc;
+  UINT32                RelocationPages;
+  MP_RELOCATION_MAP     RelocationMap;
+  MP_WAKEUP_MAILBOX     *RelocatedMailBox;
+  EFI_STATUS            Status;
+
+  Address    = 0;
+  ApLoopFunc = NULL;
+  ZeroMem (&RelocationMap, sizeof (RelocationMap));
+
+  //
+  // Get information needed to setup aps running in their
+  // run loop in allocated acpi reserved memory
+  // Add another page for mailbox
+  //
+  AsmGetRelocationMap (&RelocationMap);
+  if ((RelocationMap.RelocateApLoopFuncAddress == 0) || (RelocationMap.RelocateApLoopFuncSize == 0)) {
+    DEBUG ((DEBUG_ERROR, "Failed to get the RelocationMap.\n"));
+    return 0;
+  }
+
+  RelocationPages = EFI_SIZE_TO_PAGES ((UINT32)RelocationMap.RelocateApLoopFuncSize) + 1;
+
+  Status = gBS->AllocatePages (AllocateAnyPages, EfiACPIMemoryNVS, RelocationPages, &Address);
+  if (EFI_ERROR (Status)) {
+    DEBUG ((DEBUG_ERROR, "Failed to allocate pages for MailboxRelocation. %r\n", Status));
+    return 0;
+  }
+
+  ZeroMem ((VOID *)Address, EFI_PAGES_TO_SIZE (RelocationPages));
+
+  ApLoopFunc = (VOID *)((UINTN)Address + EFI_PAGE_SIZE);
+
+  CopyMem (
+    ApLoopFunc,
+    RelocationMap.RelocateApLoopFuncAddress,
+    RelocationMap.RelocateApLoopFuncSize
+    );
+
+  DEBUG ((
+    DEBUG_INFO,
+    "Ap Relocation: mailbox %llx, loop %p\n",
+    Address,
+    ApLoopFunc
+    ));
+
+  //
+  // Initialize mailbox
+  //
+  RelocatedMailBox               = (MP_WAKEUP_MAILBOX *)Address;
+  RelocatedMailBox->Command      = MpProtectedModeWakeupCommandNoop;
+  RelocatedMailBox->ApicId       = MP_CPU_PROTECTED_MODE_MAILBOX_APICID_INVALID;
+  RelocatedMailBox->WakeUpVector = 0;
+
+  //
+  // Wakup APs and have been move to the finalized run loop
+  // They will spin until guest OS wakes them
+  //
+  MpSerializeStart ();
+
+  MpSendWakeupCommand (
+    MpProtectedModeWakeupCommandWakeup,
+    (UINT64)ApLoopFunc,
+    (UINT64)RelocatedMailBox,
+    0,
+    0,
+    0
+    );
+
+  return Address;
+}
+
+/**
+  Alter the MADT when ACPI Table from QEMU is available.
+
+  @param[in]  Event     Event whose notification function is being invoked
+  @param[in]  Context   Pointer to the notification function's context
+**/
+VOID
+EFIAPI
+AlterAcpiTable (
+  IN EFI_EVENT  Event,
+  IN VOID       *Context
+  )
+{
+  EFI_ACPI_SDT_PROTOCOL                                *AcpiSdtProtocol;
+  EFI_ACPI_TABLE_PROTOCOL                              *AcpiTableProtocol;
+  EFI_STATUS                                           Status;
+  UINTN                                                Index;
+  EFI_ACPI_SDT_HEADER                                  *Table;
+  EFI_ACPI_TABLE_VERSION                               Version;
+  UINTN                                                OriginalTableKey;
+  UINTN                                                NewTableKey;
+  UINT8                                                *NewMadtTable;
+  UINTN                                                NewMadtTableLength;
+  EFI_PHYSICAL_ADDRESS                                 RelocateMailboxAddress;
+  EFI_ACPI_6_4_MULTIPROCESSOR_WAKEUP_STRUCTURE         *MadtMpWk;
+  EFI_ACPI_1_0_MULTIPLE_APIC_DESCRIPTION_TABLE_HEADER  *MadtHeader;
+
+  Index        = 0;
+  NewMadtTable = NULL;
+  MadtHeader   = NULL;
+
+  Status = gBS->LocateProtocol (&gEfiAcpiSdtProtocolGuid, NULL, (void **)&AcpiSdtProtocol);
+  if (EFI_ERROR (Status)) {
+    DEBUG ((DEBUG_ERROR, "Unable to locate ACPI SDT protocol.\n"));
+    return;
+  }
+
+  RelocateMailboxAddress = RelocateMailbox ();
+  if (RelocateMailboxAddress == 0) {
+    ASSERT (FALSE);
+    DEBUG ((DEBUG_ERROR, "Failed to relocate Td mailbox\n"));
+    return;
+  }
+
+  do {
+    Status = AcpiSdtProtocol->GetAcpiTable (Index, &Table, &Version, &OriginalTableKey);
+
+    if (!EFI_ERROR (Status) && (Table->Signature == EFI_ACPI_1_0_APIC_SIGNATURE)) {
+      Status = gBS->LocateProtocol (&gEfiAcpiTableProtocolGuid, NULL, (void **)&AcpiTableProtocol);
+      if (EFI_ERROR (Status)) {
+        DEBUG ((DEBUG_ERROR, "Unable to locate ACPI Table protocol.\n"));
+        break;
+      }
+
+      NewMadtTableLength = Table->Length + sizeof (EFI_ACPI_6_4_MULTIPROCESSOR_WAKEUP_STRUCTURE);
+      NewMadtTable       = AllocatePool (NewMadtTableLength);
+      if (NewMadtTable == NULL) {
+        DEBUG ((DEBUG_ERROR, "%a: OUT_OF_SOURCES error.\n", __FUNCTION__));
+        break;
+      }
+
+      CopyMem (NewMadtTable, (UINT8 *)Table, Table->Length);
+      MadtHeader                = (EFI_ACPI_1_0_MULTIPLE_APIC_DESCRIPTION_TABLE_HEADER *)NewMadtTable;
+      MadtHeader->Header.Length = (UINT32)NewMadtTableLength;
+
+      MadtMpWk                 = (EFI_ACPI_6_4_MULTIPROCESSOR_WAKEUP_STRUCTURE *)(NewMadtTable + Table->Length);
+      MadtMpWk->Type           = EFI_ACPI_6_4_MULTIPROCESSOR_WAKEUP;
+      MadtMpWk->Length         = sizeof (EFI_ACPI_6_4_MULTIPROCESSOR_WAKEUP_STRUCTURE);
+      MadtMpWk->MailBoxVersion = 1;
+      MadtMpWk->Reserved       = 0;
+      MadtMpWk->MailBoxAddress = RelocateMailboxAddress;
+
+      Status = AcpiTableProtocol->InstallAcpiTable (AcpiTableProtocol, NewMadtTable, NewMadtTableLength, &NewTableKey);
+      if (EFI_ERROR (Status)) {
+        DEBUG ((DEBUG_ERROR, "Failed to install new MADT table. %r\n", Status));
+        break;
+      }
+
+      Status = AcpiTableProtocol->UninstallAcpiTable (AcpiTableProtocol, OriginalTableKey);
+      if (EFI_ERROR (Status)) {
+        DEBUG ((DEBUG_ERROR, "Uninstall old MADT table error.\n"));
+      }
+
+      break;
+    }
+
+    Index++;
+  } while (!EFI_ERROR (Status));
+
+  if (NewMadtTable != NULL) {
+    FreePool (NewMadtTable);
+  }
+}
diff --git a/OvmfPkg/TdxDxe/TdxAcpiTable.h b/OvmfPkg/TdxDxe/TdxAcpiTable.h
new file mode 100644
index 000000000000..6b7615dc3687
--- /dev/null
+++ b/OvmfPkg/TdxDxe/TdxAcpiTable.h
@@ -0,0 +1,60 @@
+/** @file
+
+  Copyright (c) 2008 - 2012, Intel Corporation. All rights reserved.<BR>
+  SPDX-License-Identifier: BSD-2-Clause-Patent
+
+**/
+
+#ifndef TDX_ACPI_TABLE_H_
+#define TDX_ACPI_TABLE_H_
+
+#include <PiDxe.h>
+
+#include <Protocol/AcpiTable.h>
+#include <Protocol/FirmwareVolume2.h>
+#include <Protocol/PciIo.h>
+
+#include <Library/BaseLib.h>
+#include <Library/UefiBootServicesTableLib.h>
+#include <Library/DebugLib.h>
+#include <Library/PcdLib.h>
+#include <IndustryStandard/IntelTdx.h>
+#include <IndustryStandard/Acpi.h>
+
+VOID
+EFIAPI
+AsmGetRelocationMap (
+  OUT MP_RELOCATION_MAP  *AddressMap
+  );
+
+/**
+  At the beginning of system boot, a 4K-aligned, 4K-size memory (Td mailbox) is
+  pre-allocated by host VMM. BSP & APs do the page accept together in that memory
+  region.
+
+  After that TDVF is designed to relocate the mailbox to a 4K-aligned, 4K-size
+  memory block which is allocated in the ACPI Nvs memory. APs are waken up and
+  spin around the relocated mailbox for further command.
+
+  @return   EFI_PHYSICAL_ADDRESS    Address of the relocated mailbox
+**/
+EFI_PHYSICAL_ADDRESS
+EFIAPI
+RelocateMailbox (
+  VOID
+  );
+
+/**
+  Alter the MADT when ACPI Table from QEMU is available.
+
+  @param[in]  Event     Event whose notification function is being invoked
+  @param[in]  Context   Pointer to the notification function's context
+**/
+VOID
+EFIAPI
+AlterAcpiTable (
+  IN EFI_EVENT  Event,
+  IN VOID       *Context
+  );
+
+#endif
diff --git a/OvmfPkg/TdxDxe/TdxDxe.c b/OvmfPkg/TdxDxe/TdxDxe.c
new file mode 100644
index 000000000000..8f484a36fda9
--- /dev/null
+++ b/OvmfPkg/TdxDxe/TdxDxe.c
@@ -0,0 +1,251 @@
+/** @file
+
+  TDX Dxe driver. This driver is dispatched early in DXE, due to being list
+  in APRIORI.
+
+  This module is responsible for:
+    - Sets max logical cpus based on TDINFO
+    - Sets PCI PCDs based on resource hobs
+    - Alter MATD table to record address of Mailbox
+
+  Copyright (c) 2020 - 2021, Intel Corporation. All rights reserved.<BR>
+
+  SPDX-License-Identifier: BSD-2-Clause-Patent
+
+**/
+
+#include <Library/BaseLib.h>
+#include <Library/BaseMemoryLib.h>
+#include <Library/DebugLib.h>
+#include <Library/DxeServicesTableLib.h>
+#include <Library/MemoryAllocationLib.h>
+#include <Library/PcdLib.h>
+#include <Library/UefiLib.h>
+#include <Library/HobLib.h>
+#include <Protocol/Cpu.h>
+#include <Library/UefiBootServicesTableLib.h>
+#include <IndustryStandard/Tdx.h>
+#include <Library/PlatformInitLib.h>
+#include <Library/TdxLib.h>
+#include <TdxAcpiTable.h>
+#include <Library/MemEncryptTdxLib.h>
+
+/**
+  Location of resource hob matching type and starting address
+
+  @param[in]  Type             The type of resource hob to locate.
+
+  @param[in]  Start            The resource hob must at least begin at address.
+
+  @retval pointer to resource  Return pointer to a resource hob that matches or NULL.
+**/
+STATIC
+EFI_HOB_RESOURCE_DESCRIPTOR *
+GetResourceDescriptor (
+  EFI_RESOURCE_TYPE     Type,
+  EFI_PHYSICAL_ADDRESS  Start,
+  EFI_PHYSICAL_ADDRESS  End
+  )
+{
+  EFI_PEI_HOB_POINTERS         Hob;
+  EFI_HOB_RESOURCE_DESCRIPTOR  *ResourceDescriptor = NULL;
+
+  Hob.Raw = GetFirstHob (EFI_HOB_TYPE_RESOURCE_DESCRIPTOR);
+  while (Hob.Raw != NULL) {
+    DEBUG ((
+      DEBUG_INFO,
+      "%a:%d: resource type 0x%x %llx %llx\n",
+      __func__,
+      __LINE__,
+      Hob.ResourceDescriptor->ResourceType,
+      Hob.ResourceDescriptor->PhysicalStart,
+      Hob.ResourceDescriptor->ResourceLength
+      ));
+
+    if ((Hob.ResourceDescriptor->ResourceType == Type) &&
+        (Hob.ResourceDescriptor->PhysicalStart >= Start) &&
+        ((Hob.ResourceDescriptor->PhysicalStart + Hob.ResourceDescriptor->ResourceLength) < End))
+    {
+      ResourceDescriptor = Hob.ResourceDescriptor;
+      break;
+    }
+
+    Hob.Raw = GET_NEXT_HOB (Hob);
+    Hob.Raw = GetNextHob (EFI_HOB_TYPE_RESOURCE_DESCRIPTOR, Hob.Raw);
+  }
+
+  return ResourceDescriptor;
+}
+
+/**
+  Location of resource hob matching type and highest address below end
+
+  @param[in]  Type             The type of resource hob to locate.
+
+  @param[in]  End              The resource hob return is the closest to the End address
+
+  @retval pointer to resource  Return pointer to a resource hob that matches or NULL.
+**/
+STATIC
+EFI_HOB_RESOURCE_DESCRIPTOR *
+GetHighestResourceDescriptor (
+  EFI_RESOURCE_TYPE     Type,
+  EFI_PHYSICAL_ADDRESS  End
+  )
+{
+  EFI_PEI_HOB_POINTERS         Hob;
+  EFI_HOB_RESOURCE_DESCRIPTOR  *ResourceDescriptor = NULL;
+
+  Hob.Raw = GetFirstHob (EFI_HOB_TYPE_RESOURCE_DESCRIPTOR);
+  while (Hob.Raw != NULL) {
+    if ((Hob.ResourceDescriptor->ResourceType == Type) &&
+        (Hob.ResourceDescriptor->PhysicalStart < End))
+    {
+      if (!ResourceDescriptor ||
+          (ResourceDescriptor->PhysicalStart < Hob.ResourceDescriptor->PhysicalStart))
+      {
+        ResourceDescriptor = Hob.ResourceDescriptor;
+      }
+    }
+
+    Hob.Raw = GET_NEXT_HOB (Hob);
+    Hob.Raw = GetNextHob (EFI_HOB_TYPE_RESOURCE_DESCRIPTOR, Hob.Raw);
+  }
+
+  return ResourceDescriptor;
+}
+
+/**
+  Set the shared bit for mmio region in Tdx guest.
+
+  In Tdx guest there are 2 ways to access mmio, TdVmcall or direct access.
+  For direct access, the shared bit of the PageTableEntry should be set.
+  The mmio region information is retrieved from hob list.
+
+  @retval EFI_SUCCESS                 The shared bit is set successfully.
+  @retval EFI_UNSUPPORTED             Setting the shared bit of memory region
+                                      is not supported
+**/
+EFI_STATUS
+SetMmioSharedBit (
+  VOID
+  )
+{
+  EFI_PEI_HOB_POINTERS  Hob;
+
+  Hob.Raw = (UINT8 *)GetHobList ();
+
+  //
+  // Parse the HOB list until end of list or matching type is found.
+  //
+  while (!END_OF_HOB_LIST (Hob)) {
+    if (  (Hob.Header->HobType == EFI_HOB_TYPE_RESOURCE_DESCRIPTOR)
+       && (Hob.ResourceDescriptor->ResourceType == EFI_RESOURCE_MEMORY_MAPPED_IO))
+    {
+      MemEncryptTdxSetPageSharedBit (
+        0,
+        Hob.ResourceDescriptor->PhysicalStart,
+        EFI_SIZE_TO_PAGES (Hob.ResourceDescriptor->ResourceLength)
+        );
+    }
+
+    Hob.Raw = GET_NEXT_HOB (Hob);
+  }
+
+  return EFI_SUCCESS;
+}
+
+EFI_STATUS
+EFIAPI
+TdxDxeEntryPoint (
+  IN EFI_HANDLE        ImageHandle,
+  IN EFI_SYSTEM_TABLE  *SystemTable
+  )
+{
+  EFI_STATUS                   Status;
+  RETURN_STATUS                PcdStatus;
+  EFI_HOB_RESOURCE_DESCRIPTOR  *Res          = NULL;
+  EFI_HOB_RESOURCE_DESCRIPTOR  *MemRes       = NULL;
+  EFI_HOB_PLATFORM_INFO        *PlatformInfo = NULL;
+  EFI_HOB_GUID_TYPE            *GuidHob;
+  UINT32                       CpuMaxLogicalProcessorNumber;
+  TD_RETURN_DATA               TdReturnData;
+  EFI_EVENT                    QemuAcpiTableEvent;
+  void                         *Registration;
+
+  GuidHob = GetFirstGuidHob (&gUefiOvmfPkgTdxPlatformGuid);
+
+  if (GuidHob == NULL) {
+    return EFI_UNSUPPORTED;
+  }
+
+  SetMmioSharedBit ();
+
+  PlatformInfo = (EFI_HOB_PLATFORM_INFO *)GET_GUID_HOB_DATA (GuidHob);
+
+  //
+  // Call TDINFO to get actual number of cpus in domain
+  //
+  Status = TdCall (TDCALL_TDINFO, 0, 0, 0, &TdReturnData);
+  ASSERT (Status == EFI_SUCCESS);
+
+  CpuMaxLogicalProcessorNumber = PcdGet32 (PcdCpuMaxLogicalProcessorNumber);
+
+  //
+  // Adjust PcdCpuMaxLogicalProcessorNumber, if needed. If firmware is configured for
+  // more than number of reported cpus, update.
+  //
+  if (CpuMaxLogicalProcessorNumber > TdReturnData.TdInfo.NumVcpus) {
+    PcdStatus = PcdSet32S (PcdCpuMaxLogicalProcessorNumber, TdReturnData.TdInfo.NumVcpus);
+    ASSERT_RETURN_ERROR (PcdStatus);
+  }
+
+  //
+  // Register for protocol notifications to call the AlterAcpiTable(),
+  // the protocol will be installed in AcpiPlatformDxe when the ACPI
+  // table provided by Qemu is ready.
+  //
+  Status = gBS->CreateEvent (
+                  EVT_NOTIFY_SIGNAL,
+                  TPL_CALLBACK,
+                  AlterAcpiTable,
+                  NULL,
+                  &QemuAcpiTableEvent
+                  );
+
+  Status = gBS->RegisterProtocolNotify (
+                  &gQemuAcpiTableNotifyProtocolGuid,
+                  QemuAcpiTableEvent,
+                  &Registration
+                  );
+
+  #define INIT_PCDSET(NAME, RES)  do {\
+  PcdStatus = PcdSet64S (NAME##Base, (RES)->PhysicalStart); \
+  ASSERT_RETURN_ERROR (PcdStatus); \
+  PcdStatus = PcdSet64S (NAME##Size, (RES)->ResourceLength); \
+  ASSERT_RETURN_ERROR (PcdStatus); \
+} while(0)
+
+  if (PlatformInfo) {
+    PcdSet16S (PcdOvmfHostBridgePciDevId, PlatformInfo->HostBridgePciDevId);
+
+    if ((Res = GetResourceDescriptor (EFI_RESOURCE_MEMORY_MAPPED_IO, (EFI_PHYSICAL_ADDRESS)0x100000000, (EFI_PHYSICAL_ADDRESS)-1)) != NULL) {
+      INIT_PCDSET (PcdPciMmio64, Res);
+    }
+
+    if ((Res = GetResourceDescriptor (EFI_RESOURCE_IO, 0, 0x10001)) != NULL) {
+      INIT_PCDSET (PcdPciIo, Res);
+    }
+
+    //
+    // To find low mmio, first find top of low memory, and then search for io space.
+    //
+    if ((MemRes = GetHighestResourceDescriptor (EFI_RESOURCE_SYSTEM_MEMORY, 0xffc00000)) != NULL) {
+      if ((Res = GetResourceDescriptor (EFI_RESOURCE_MEMORY_MAPPED_IO, MemRes->PhysicalStart, 0x100000000)) != NULL) {
+        INIT_PCDSET (PcdPciMmio32, Res);
+      }
+    }
+  }
+
+  return EFI_SUCCESS;
+}
diff --git a/OvmfPkg/TdxDxe/TdxDxe.inf b/OvmfPkg/TdxDxe/TdxDxe.inf
new file mode 100644
index 000000000000..077769bcf70c
--- /dev/null
+++ b/OvmfPkg/TdxDxe/TdxDxe.inf
@@ -0,0 +1,64 @@
+#/** @file
+#
+#  Driver clears the encryption attribute from MMIO regions when TDX is enabled
+#
+#  Copyright (c) 2017, AMD Inc. All rights reserved.<BR>
+#
+#  SPDX-License-Identifier: BSD-2-Clause-Patent
+#
+#**/
+
+[Defines]
+  INF_VERSION                    = 1.25
+  BASE_NAME                      = TdxDxe
+  FILE_GUID                      = E750224E-7BCE-40AF-B5BB-47E3611EB5C2
+  MODULE_TYPE                    = DXE_DRIVER
+  VERSION_STRING                 = 1.0
+  ENTRY_POINT                    = TdxDxeEntryPoint
+
+[Sources]
+  TdxDxe.c
+  TdxAcpiTable.c
+  X64/ApRunLoop.nasm
+
+[Packages]
+  MdeModulePkg/MdeModulePkg.dec
+  MdePkg/MdePkg.dec
+  UefiCpuPkg/UefiCpuPkg.dec
+  OvmfPkg/OvmfPkg.dec
+
+[LibraryClasses]
+  BaseLib
+  BaseMemoryLib
+  DebugLib
+  DxeServicesTableLib
+  MemoryAllocationLib
+  PcdLib
+  UefiDriverEntryPoint
+  TdxLib
+  HobLib
+  TdxMailboxLib
+  MemEncryptTdxLib
+
+[Depex]
+  TRUE
+
+[Guids]
+  gUefiOvmfPkgTdxPlatformGuid                      ## CONSUMES
+
+[Protocols]
+  gQemuAcpiTableNotifyProtocolGuid                 ## CONSUMES
+  gEfiAcpiSdtProtocolGuid                          ## CONSUMES
+  gEfiAcpiTableProtocolGuid                        ## CONSUMES
+
+[Pcd]
+  gUefiOvmfPkgTokenSpaceGuid.PcdPciIoBase
+  gUefiOvmfPkgTokenSpaceGuid.PcdPciIoSize
+  gUefiOvmfPkgTokenSpaceGuid.PcdPciMmio32Base
+  gUefiOvmfPkgTokenSpaceGuid.PcdPciMmio32Size
+  gUefiOvmfPkgTokenSpaceGuid.PcdPciMmio64Base
+  gUefiOvmfPkgTokenSpaceGuid.PcdPciMmio64Size
+  gUefiOvmfPkgTokenSpaceGuid.PcdOvmfHostBridgePciDevId
+  gUefiCpuPkgTokenSpaceGuid.PcdCpuMaxLogicalProcessorNumber
+  gUefiCpuPkgTokenSpaceGuid.PcdCpuLocalApicBaseAddress
+  gUefiOvmfPkgTokenSpaceGuid.PcdOvmfFdBaseAddress
diff --git a/OvmfPkg/TdxDxe/X64/ApRunLoop.nasm b/OvmfPkg/TdxDxe/X64/ApRunLoop.nasm
new file mode 100644
index 000000000000..49bd04415cfd
--- /dev/null
+++ b/OvmfPkg/TdxDxe/X64/ApRunLoop.nasm
@@ -0,0 +1,90 @@
+;------------------------------------------------------------------------------ ;
+; Copyright (c) 2021, Intel Corporation. All rights reserved.<BR>
+; SPDX-License-Identifier: BSD-2-Clause-Patent
+;
+; Module Name:
+;
+;   ApRunLoop.nasm
+;
+; Abstract:
+;
+;   This is the assembly code for run loop for APs in the guest TD
+;
+;-------------------------------------------------------------------------------
+
+%include "TdxCommondefs.inc"
+
+DEFAULT REL
+
+SECTION .text
+
+BITS 64
+
+%define TDVMCALL_EXPOSE_REGS_MASK       0xffec
+%define TDVMCALL                        0x0
+%define EXIT_REASON_CPUID               0xa
+
+%macro  tdcall  0
+  db  0x66, 0x0f, 0x01, 0xcc
+%endmacro
+
+;
+; Relocated Ap Mailbox loop
+;
+; @param[in]  RBX:  Relocated mailbox address
+; @param[in]  RBP:  vCpuId
+;
+; @return     None  This routine does not return
+;
+global ASM_PFX(AsmRelocateApMailBoxLoop)
+ASM_PFX(AsmRelocateApMailBoxLoop):
+AsmRelocateApMailBoxLoopStart:
+
+    mov         rax, TDVMCALL
+    mov         rcx, TDVMCALL_EXPOSE_REGS_MASK
+    mov         r11, EXIT_REASON_CPUID
+    mov         r12, 0xb
+    tdcall
+    test        rax, rax
+    jnz         Panic
+    mov         r8, r15
+
+MailBoxLoop:
+    ; Spin until command set
+    cmp        dword [rbx + CommandOffset], MpProtectedModeWakeupCommandNoop
+    je         MailBoxLoop
+    ; Determine if this is a broadcast or directly for my apic-id, if not, ignore
+    cmp        dword [rbx + ApicidOffset], MailboxApicidBroadcast
+    je         MailBoxProcessCommand
+    cmp        dword [rbx + ApicidOffset], r8d
+    jne        MailBoxLoop
+MailBoxProcessCommand:
+    cmp        dword [rbx + CommandOffset], MpProtectedModeWakeupCommandWakeup
+    je         MailBoxWakeUp
+    cmp        dword [rbx + CommandOffset], MpProtectedModeWakeupCommandSleep
+    je         MailBoxSleep
+    ; Don't support this command, so ignore
+    jmp        MailBoxLoop
+MailBoxWakeUp:
+    mov        rax, [rbx + WakeupVectorOffset]
+    ; OS sends a wakeup command for a given APIC ID, firmware is supposed to reset
+    ; the command field back to zero as acknowledgement.
+    mov        qword [rbx + WakeupVectorOffset], 0
+    jmp        rax
+MailBoxSleep:
+    jmp       $
+Panic:
+    ud2
+BITS 64
+AsmRelocateApMailBoxLoopEnd:
+
+;-------------------------------------------------------------------------------------
+;  AsmGetRelocationMap (&RelocationMap);
+;-------------------------------------------------------------------------------------
+global ASM_PFX(AsmGetRelocationMap)
+ASM_PFX(AsmGetRelocationMap):
+    lea        rax, [ASM_PFX(AsmRelocateApMailBoxLoopStart)]
+    mov        qword [rcx], rax
+    mov        qword [rcx +  8h], AsmRelocateApMailBoxLoopEnd - AsmRelocateApMailBoxLoopStart
+    ret
+
-- 
2.29.2.windows.2