[RFC PATCH 0/9] Firmware Support for Fast Live Migration for AMD SEV

[RFC PATCH 0/9] Firmware Support for Fast Live Migration for AMD SEV

[Tobin Feldman-Fitzthum]

With AMD SEV the hypervisor cannot decrypt or move guest memory pages. 
This makes migration tricky. While the AMD Secure Processor can
export/import pages wrapped with a transport key, the bandwidth is
limited. We look to provide similar support via firmware. In particular,
we implement a Migration Handler (MH) in OVMF. The MH runs in a separate
mirror VM that shares the memory of the guest. To migrate a guest, the
HV asks the MH on the source to export wrapped pages, which the MH
on the target will import. To start the MH on the source or the target
the HV boots the mirror VM to a custom entry vector implemented in these
patches.

This RFC does not include encryption support. The pages are passed
to/from the HV in plaintext. This RFC depends on mirror VM support
already upstreamed in KVM, AMD page encryption status tracking
(Ashish Kalra's v6 OVMF live migration patches), mirror VM support in
QEMU (Ashish's v1 QEMU RFC), page encryption status tracking support in
QEMU (Ashish QEMU guest live migration support v4), and MH support in
QEMU (coming soon). This RFC is aimed at SEV only. The general design
carries over to SEV-ES and SEV-SNP, but extra support is required.

Dov Murik (1):
  OvmfPkg/AmdSev: Build page table for migration handler

Tobin Feldman-Fitzthum (8):
  OvmfPkg/AmdSev: Base for Confidential Migration Handler
  OvmfPkg/PlatfomPei: Set Confidential Migration PCD
  OvmfPkg/AmdSev: Setup Migration Handler Mailbox
  OvmfPkg/AmdSev: MH support for mailbox protocol
  OvmfPkg/AmdSev: Don't overwrite mailbox or pagetables
  OvmfPkg/AmdSev: Don't overwrite MH stack
  OvmfPkg/AmdSev: Add Migration Handler entry point
  OvmfPkg/ResetVector: Expose Migration Handler Entry Addresses

 OvmfPkg/OvmfPkg.dec                           |  13 +
 OvmfPkg/AmdSev/AmdSevX64.dsc                  |   2 +
 OvmfPkg/AmdSev/AmdSevX64.fdf                  |  16 +-
 .../ConfidentialMigrationDxe.inf              |  38 +++
 .../ConfidentialMigrationPei.inf              |  37 +++
 OvmfPkg/PlatformPei/PlatformPei.inf           |   2 +
 OvmfPkg/ResetVector/ResetVector.inf           |   1 +
 .../ConfidentialMigration/VirtualMemory.h     | 177 ++++++++++++
 .../ConfidentialMigrationDxe.c                | 272 ++++++++++++++++++
 .../ConfidentialMigrationPei.c                |  31 ++
 OvmfPkg/PlatformPei/Platform.c                |  10 +
 .../MigrationEntryPoint.nasm                  |  51 ++++
 OvmfPkg/ResetVector/Ia16/ResetVectorVtf0.asm  |  13 +
 OvmfPkg/ResetVector/ResetVector.nasmb         |   1 +
 14 files changed, 660 insertions(+), 4 deletions(-)
 create mode 100644 OvmfPkg/AmdSev/ConfidentialMigration/ConfidentialMigrationDxe.inf
 create mode 100644 OvmfPkg/AmdSev/ConfidentialMigration/ConfidentialMigrationPei.inf
 create mode 100644 OvmfPkg/AmdSev/ConfidentialMigration/VirtualMemory.h
 create mode 100644 OvmfPkg/AmdSev/ConfidentialMigration/ConfidentialMigrationDxe.c
 create mode 100644 OvmfPkg/AmdSev/ConfidentialMigration/ConfidentialMigrationPei.c
 create mode 100644 OvmfPkg/AmdSev/ConfidentialMigration/MigrationEntryPoint.nasm

-- 
2.20.1

[RFC PATCH 1/9] OvmfPkg/AmdSev: Base for Confidential Migration Handler

[Tobin Feldman-Fitzthum]

Base enablement of DXE driver that supports confidential migration.

Signed-off-by: Tobin Feldman-Fitzthum <tobin@linux.ibm.com>
---
 OvmfPkg/OvmfPkg.dec                           |  5 ++
 OvmfPkg/AmdSev/AmdSevX64.dsc                  |  1 +
 OvmfPkg/AmdSev/AmdSevX64.fdf                  |  1 +
 .../ConfidentialMigrationDxe.inf              | 34 ++++++++++++
 .../ConfidentialMigrationDxe.c                | 53 +++++++++++++++++++
 5 files changed, 94 insertions(+)
 create mode 100644 OvmfPkg/AmdSev/ConfidentialMigration/ConfidentialMigrationDxe.inf
 create mode 100644 OvmfPkg/AmdSev/ConfidentialMigration/ConfidentialMigrationDxe.c

diff --git a/OvmfPkg/OvmfPkg.dec b/OvmfPkg/OvmfPkg.dec
index 3978852557..cfc645619d 100644
--- a/OvmfPkg/OvmfPkg.dec
+++ b/OvmfPkg/OvmfPkg.dec
@@ -370,6 +370,11 @@
   #  instance in PiSmmCpuDxeSmm, and CpuHotplugSmm.
   gUefiOvmfPkgTokenSpaceGuid.PcdCpuHotEjectDataAddress|0|UINT64|0x46
 
+  ## Set via FW_CFG to enable confidential migration as source or target.
+  #
+  gUefiOvmfPkgTokenSpaceGuid.PcdIsConfidentialMigrationTarget|FALSE|BOOLEAN|0x49
+  gUefiOvmfPkgTokenSpaceGuid.PcdStartConfidentialMigrationHandler|FALSE|BOOLEAN|0x4a
+
 [PcdsFeatureFlag]
   gUefiOvmfPkgTokenSpaceGuid.PcdQemuBootOrderPciTranslation|TRUE|BOOLEAN|0x1c
   gUefiOvmfPkgTokenSpaceGuid.PcdQemuBootOrderMmioTranslation|FALSE|BOOLEAN|0x1d
diff --git a/OvmfPkg/AmdSev/AmdSevX64.dsc b/OvmfPkg/AmdSev/AmdSevX64.dsc
index e6cd10b759..982ecaf70e 100644
--- a/OvmfPkg/AmdSev/AmdSevX64.dsc
+++ b/OvmfPkg/AmdSev/AmdSevX64.dsc
@@ -790,6 +790,7 @@
 !endif
   OvmfPkg/AmdSev/SecretDxe/SecretDxe.inf
   OvmfPkg/AmdSev/Grub/Grub.inf
+  OvmfPkg/AmdSev/ConfidentialMigration/ConfidentialMigrationDxe.inf
 !if $(BUILD_SHELL) == TRUE
   ShellPkg/Application/Shell/Shell.inf {
     <LibraryClasses>
diff --git a/OvmfPkg/AmdSev/AmdSevX64.fdf b/OvmfPkg/AmdSev/AmdSevX64.fdf
index 0a89749700..9bf17b8d51 100644
--- a/OvmfPkg/AmdSev/AmdSevX64.fdf
+++ b/OvmfPkg/AmdSev/AmdSevX64.fdf
@@ -274,6 +274,7 @@ INF  OvmfPkg/LinuxInitrdDynamicShellCommand/LinuxInitrdDynamicShellCommand.inf
 !endif
 INF OvmfPkg/AmdSev/SecretDxe/SecretDxe.inf
 INF  OvmfPkg/AmdSev/Grub/Grub.inf
+INF OvmfPkg/AmdSev/ConfidentialMigration/ConfidentialMigrationDxe.inf
 !if $(BUILD_SHELL) == TRUE
 INF  ShellPkg/Application/Shell/Shell.inf
 !endif
diff --git a/OvmfPkg/AmdSev/ConfidentialMigration/ConfidentialMigrationDxe.inf b/OvmfPkg/AmdSev/ConfidentialMigration/ConfidentialMigrationDxe.inf
new file mode 100644
index 0000000000..6e3fa7e51c
--- /dev/null
+++ b/OvmfPkg/AmdSev/ConfidentialMigration/ConfidentialMigrationDxe.inf
@@ -0,0 +1,34 @@
+## @file
+#
+#  Copyright (C) 2021 IBM Corporation.
+#  SPDX-License-Identifier: BSD-2-Clause-Patent
+#
+##
+
+[Defines]
+  INF_VERSION                    = 0x00010005
+  BASE_NAME                      = ConfidentialMigration
+  FILE_GUID                      = 5c2978f4-f175-434b-9e6c-9b03bd7e346f
+  MODULE_TYPE                    = DXE_RUNTIME_DRIVER
+  VERSION_STRING                 = 1.0
+  ENTRY_POINT                    = SetupMigrationHandler
+
+[Sources]
+  ConfidentialMigrationDxe.c
+
+[Packages]
+  MdePkg/MdePkg.dec
+  OvmfPkg/OvmfPkg.dec
+
+[LibraryClasses]
+  DebugLib
+  MemoryAllocationLib
+  UefiDriverEntryPoint
+  UefiLib
+
+[Pcd]
+  gUefiOvmfPkgTokenSpaceGuid.PcdIsConfidentialMigrationTarget
+  gUefiOvmfPkgTokenSpaceGuid.PcdStartConfidentialMigrationHandler
+
+[Depex]
+  TRUE
diff --git a/OvmfPkg/AmdSev/ConfidentialMigration/ConfidentialMigrationDxe.c b/OvmfPkg/AmdSev/ConfidentialMigration/ConfidentialMigrationDxe.c
new file mode 100644
index 0000000000..f0dfbd279e
--- /dev/null
+++ b/OvmfPkg/AmdSev/ConfidentialMigration/ConfidentialMigrationDxe.c
@@ -0,0 +1,53 @@
+/** @file
+  In-guest support for confidential migration
+
+  Copyright (C) 2021 IBM Coporation.
+  SPDX-License-Identifier: BSD-2-Clause-Patent
+**/
+
+#include <Library/DebugLib.h>
+#include <Library/UefiLib.h>
+
+VOID
+EFIAPI
+MigrationHandlerMain ()
+{
+  DebugPrint (DEBUG_INFO,"Migration Handler Started\n");
+
+}
+
+/**
+SetupMigrationHandler runs in the firmware of the main VM to setup
+regions of memory that the Migration Handler can use when executing
+in the mirror VM.
+
+**/
+EFI_STATUS
+EFIAPI
+SetupMigrationHandler (
+  IN EFI_HANDLE           ImageHandle,
+  IN EFI_SYSTEM_TABLE     *SystemTable
+  )
+{
+
+  if (!PcdGetBool(PcdStartConfidentialMigrationHandler)) {
+    return 0;
+  }
+
+  //
+  // If VM is migration target, wait until hypervisor modifies CPU state
+  // and restarts execution.
+  //
+  if (PcdGetBool(PcdIsConfidentialMigrationTarget)) {
+    DebugPrint (DEBUG_INFO,"Waiting for incoming confidential migration.\n");
+
+    while (1) {
+      CpuPause ();
+    }
+  }
+
+  //
+  // If VM is migration source, continue with boot.
+  //
+  return 0;
+}
-- 
2.20.1

[RFC PATCH 2/9] OvmfPkg/PlatfomPei: Set Confidential Migration PCD

[Tobin Feldman-Fitzthum]

Confidential Migration relies on two boolean PCDs set from FW_CFG

Signed-off-by: Tobin Feldman-Fitzthum <tobin@linux.ibm.com>
---
 OvmfPkg/PlatformPei/PlatformPei.inf |  2 ++
 OvmfPkg/PlatformPei/Platform.c      | 10 ++++++++++
 2 files changed, 12 insertions(+)

diff --git a/OvmfPkg/PlatformPei/PlatformPei.inf b/OvmfPkg/PlatformPei/PlatformPei.inf
index 89d1f76368..2d92184c19 100644
--- a/OvmfPkg/PlatformPei/PlatformPei.inf
+++ b/OvmfPkg/PlatformPei/PlatformPei.inf
@@ -89,6 +89,8 @@
   gUefiOvmfPkgTokenSpaceGuid.PcdOvmfDecompressionScratchEnd
   gUefiOvmfPkgTokenSpaceGuid.PcdQ35TsegMbytes
   gUefiOvmfPkgTokenSpaceGuid.PcdQ35SmramAtDefaultSmbase
+  gUefiOvmfPkgTokenSpaceGuid.PcdStartConfidentialMigrationHandler
+  gUefiOvmfPkgTokenSpaceGuid.PcdIsConfidentialMigrationTarget
   gEfiMdePkgTokenSpaceGuid.PcdGuidedExtractHandlerTableAddress
   gEfiMdeModulePkgTokenSpaceGuid.PcdFlashNvStorageFtwSpareSize
   gEfiMdeModulePkgTokenSpaceGuid.PcdFlashNvStorageVariableSize
diff --git a/OvmfPkg/PlatformPei/Platform.c b/OvmfPkg/PlatformPei/Platform.c
index d3a20122a2..f0963aaba9 100644
--- a/OvmfPkg/PlatformPei/Platform.c
+++ b/OvmfPkg/PlatformPei/Platform.c
@@ -272,6 +272,15 @@ NoexecDxeInitialization (
   UPDATE_BOOLEAN_PCD_FROM_FW_CFG (PcdSetNxForStack);
 }
 
+VOID
+ConfidentialMigrationInitialization (
+  VOID
+  )
+{
+  UPDATE_BOOLEAN_PCD_FROM_FW_CFG (PcdStartConfidentialMigrationHandler);
+  UPDATE_BOOLEAN_PCD_FROM_FW_CFG (PcdIsConfidentialMigrationTarget);
+}
+
 VOID
 PciExBarInitialization (
   VOID
@@ -742,6 +751,7 @@ InitializePlatform (
 
   InstallClearCacheCallback ();
   AmdSevInitialize ();
+  ConfidentialMigrationInitialization ();
   MiscInitialization ();
   InstallFeatureControlCallback ();
 
-- 
2.20.1

[RFC PATCH 3/9] OvmfPkg/AmdSev: Setup Migration Handler Mailbox

[Tobin Feldman-Fitzthum]

The migration handler communicates with the hypervisor using a
special mailbox, a page of shared memory where pending commands
can be written. Another shared page is used to pass the incoming
or outgoing guest memory pages. These pages are set aside in MEMFD,
which this patch expands, and reserved as runtime memory in
ConfidentialMigrationPei, which this patch introduces.

Signed-off-by: Tobin Feldman-Fitzthum <tobin@linux.ibm.com>
---
 OvmfPkg/OvmfPkg.dec                           |  5 +++
 OvmfPkg/AmdSev/AmdSevX64.dsc                  |  1 +
 OvmfPkg/AmdSev/AmdSevX64.fdf                  | 12 ++++---
 .../ConfidentialMigrationPei.inf              | 35 +++++++++++++++++++
 .../ConfidentialMigrationPei.c                | 25 +++++++++++++
 5 files changed, 74 insertions(+), 4 deletions(-)
 create mode 100644 OvmfPkg/AmdSev/ConfidentialMigration/ConfidentialMigrationPei.inf
 create mode 100644 OvmfPkg/AmdSev/ConfidentialMigration/ConfidentialMigrationPei.c

diff --git a/OvmfPkg/OvmfPkg.dec b/OvmfPkg/OvmfPkg.dec
index cfc645619d..1252582c99 100644
--- a/OvmfPkg/OvmfPkg.dec
+++ b/OvmfPkg/OvmfPkg.dec
@@ -331,6 +331,11 @@
   gUefiOvmfPkgTokenSpaceGuid.PcdQemuHashTableBase|0x0|UINT32|0x47
   gUefiOvmfPkgTokenSpaceGuid.PcdQemuHashTableSize|0x0|UINT32|0x48
 
+  ## Area used by the confidential migration handler to communicate with
+  # the hypervisor.
+  gUefiOvmfPkgTokenSpaceGuid.PcdConfidentialMigrationMailboxBase|0x0|UINT32|0x4b
+  gUefiOvmfPkgTokenSpaceGuid.PcdConfidentialMigrationMailboxSize|0x0|UINT32|0x4c
+
 [PcdsDynamic, PcdsDynamicEx]
   gUefiOvmfPkgTokenSpaceGuid.PcdEmuVariableEvent|0|UINT64|2
   gUefiOvmfPkgTokenSpaceGuid.PcdOvmfFlashVariablesEnable|FALSE|BOOLEAN|0x10
diff --git a/OvmfPkg/AmdSev/AmdSevX64.dsc b/OvmfPkg/AmdSev/AmdSevX64.dsc
index 982ecaf70e..cd6189f330 100644
--- a/OvmfPkg/AmdSev/AmdSevX64.dsc
+++ b/OvmfPkg/AmdSev/AmdSevX64.dsc
@@ -623,6 +623,7 @@
   UefiCpuPkg/Universal/Acpi/S3Resume2Pei/S3Resume2Pei.inf
   UefiCpuPkg/CpuMpPei/CpuMpPei.inf
   OvmfPkg/AmdSev/SecretPei/SecretPei.inf
+  OvmfPkg/AmdSev/ConfidentialMigration/ConfidentialMigrationPei.inf
 
 !if $(TPM_ENABLE) == TRUE
   OvmfPkg/Tcg/TpmMmioSevDecryptPei/TpmMmioSevDecryptPei.inf
diff --git a/OvmfPkg/AmdSev/AmdSevX64.fdf b/OvmfPkg/AmdSev/AmdSevX64.fdf
index 9bf17b8d51..a8e296e641 100644
--- a/OvmfPkg/AmdSev/AmdSevX64.fdf
+++ b/OvmfPkg/AmdSev/AmdSevX64.fdf
@@ -36,10 +36,10 @@ FV = SECFV
 
 [FD.MEMFD]
 BaseAddress   = $(MEMFD_BASE_ADDRESS)
-Size          = 0xD00000
+Size          = 0xE00000
 ErasePolarity = 1
 BlockSize     = 0x10000
-NumBlocks     = 0xD0
+NumBlocks     = 0xE0
 
 0x000000|0x006000
 gUefiOvmfPkgTokenSpaceGuid.PcdOvmfSecPageTablesBase|gUefiOvmfPkgTokenSpaceGuid.PcdOvmfSecPageTablesSize
@@ -71,11 +71,14 @@ gUefiOvmfPkgTokenSpaceGuid.PcdOvmfSecGhcbBackupBase|gUefiOvmfPkgTokenSpaceGuid.P
 0x010000|0x010000
 gUefiOvmfPkgTokenSpaceGuid.PcdOvmfSecPeiTempRamBase|gUefiOvmfPkgTokenSpaceGuid.PcdOvmfSecPeiTempRamSize
 
-0x020000|0x0E0000
+0x020000|0x003000
+gUefiOvmfPkgTokenSpaceGuid.PcdConfidentialMigrationMailboxBase|gUefiOvmfPkgTokenSpaceGuid.PcdConfidentialMigrationMailboxSize
+
+0x120000|0x0E0000
 gUefiOvmfPkgTokenSpaceGuid.PcdOvmfPeiMemFvBase|gUefiOvmfPkgTokenSpaceGuid.PcdOvmfPeiMemFvSize
 FV = PEIFV
 
-0x100000|0xC00000
+0x200000|0xC00000
 gUefiOvmfPkgTokenSpaceGuid.PcdOvmfDxeMemFvBase|gUefiOvmfPkgTokenSpaceGuid.PcdOvmfDxeMemFvSize
 FV = DXEFV
 
@@ -148,6 +151,7 @@ INF  MdeModulePkg/Core/DxeIplPeim/DxeIpl.inf
 INF  UefiCpuPkg/Universal/Acpi/S3Resume2Pei/S3Resume2Pei.inf
 INF  UefiCpuPkg/CpuMpPei/CpuMpPei.inf
 INF  OvmfPkg/AmdSev/SecretPei/SecretPei.inf
+INF  OvmfPkg/AmdSev/ConfidentialMigration/ConfidentialMigrationPei.inf
 
 !if $(TPM_ENABLE) == TRUE
 INF  OvmfPkg/Tcg/TpmMmioSevDecryptPei/TpmMmioSevDecryptPei.inf
diff --git a/OvmfPkg/AmdSev/ConfidentialMigration/ConfidentialMigrationPei.inf b/OvmfPkg/AmdSev/ConfidentialMigration/ConfidentialMigrationPei.inf
new file mode 100644
index 0000000000..918cf22abd
--- /dev/null
+++ b/OvmfPkg/AmdSev/ConfidentialMigration/ConfidentialMigrationPei.inf
@@ -0,0 +1,35 @@
+## @file
+#  PEI support for confidential migration.
+#
+#  Copyright (C) 2021 IBM Corporation.
+#
+#  SPDX-License-Identifier: BSD-2-Clause-Patent
+#
+##
+
+[Defines]
+  INF_VERSION                    = 0x00010005
+  BASE_NAME                      = ConfidentialMigration
+  FILE_GUID                      = a747792e-71a1-4c24-84a9-a76a0a279878
+  MODULE_TYPE                    = PEIM
+  VERSION_STRING                 = 1.0
+  ENTRY_POINT                    = InitializeConfidentialMigrationPei
+
+[Sources]
+  ConfidentialMigrationPei.c
+
+[Packages]
+  OvmfPkg/OvmfPkg.dec
+  MdePkg/MdePkg.dec
+
+[LibraryClasses]
+  HobLib
+  PeimEntryPoint
+  PcdLib
+
+[FixedPcd]
+  gUefiOvmfPkgTokenSpaceGuid.PcdConfidentialMigrationMailboxBase
+  gUefiOvmfPkgTokenSpaceGuid.PcdConfidentialMigrationMailboxSize
+
+[Depex]
+  TRUE
diff --git a/OvmfPkg/AmdSev/ConfidentialMigration/ConfidentialMigrationPei.c b/OvmfPkg/AmdSev/ConfidentialMigration/ConfidentialMigrationPei.c
new file mode 100644
index 0000000000..ce304bc07b
--- /dev/null
+++ b/OvmfPkg/AmdSev/ConfidentialMigration/ConfidentialMigrationPei.c
@@ -0,0 +1,25 @@
+/** @file
+  Reserve memory for confidential migration handler.
+
+  Copyright (C) 2020 IBM Corporation.
+  SPDX-License-Identifier: BSD-2-Clause-Patent
+**/
+#include <PiPei.h>
+#include <Library/HobLib.h>
+#include <Library/PcdLib.h>
+
+EFI_STATUS
+EFIAPI
+InitializeConfidentialMigrationPei (
+  IN       EFI_PEI_FILE_HANDLE  FileHandle,
+  IN CONST EFI_PEI_SERVICES     **PeiServices
+  )
+{
+  BuildMemoryAllocationHob (
+    PcdGet32 (PcdConfidentialMigrationMailboxBase),
+    PcdGet32 (PcdConfidentialMigrationMailboxSize),
+    EfiRuntimeServicesData
+    );
+
+  return EFI_SUCCESS;
+}
-- 
2.20.1

[RFC PATCH 4/9] OvmfPkg/AmdSev: MH support for mailbox protocol

[Tobin Feldman-Fitzthum]

The migration handler communicates with the hypervisor
via a shared mailbox page. The MH can perform four functions
at the behest of the HV: init, save page, restore page, and
reset.

Signed-off-by: Tobin Feldman-Fitzthum <tobin@linux.ibm.com>
---
 .../ConfidentialMigrationDxe.inf              |  1 +
 .../ConfidentialMigrationDxe.c                | 74 +++++++++++++++++++
 2 files changed, 75 insertions(+)

diff --git a/OvmfPkg/AmdSev/ConfidentialMigration/ConfidentialMigrationDxe.inf b/OvmfPkg/AmdSev/ConfidentialMigration/ConfidentialMigrationDxe.inf
index 6e3fa7e51c..cb5609271c 100644
--- a/OvmfPkg/AmdSev/ConfidentialMigration/ConfidentialMigrationDxe.inf
+++ b/OvmfPkg/AmdSev/ConfidentialMigration/ConfidentialMigrationDxe.inf
@@ -29,6 +29,7 @@
 [Pcd]
   gUefiOvmfPkgTokenSpaceGuid.PcdIsConfidentialMigrationTarget
   gUefiOvmfPkgTokenSpaceGuid.PcdStartConfidentialMigrationHandler
+  gUefiOvmfPkgTokenSpaceGuid.PcdConfidentialMigrationMailboxBase
 
 [Depex]
   TRUE
diff --git a/OvmfPkg/AmdSev/ConfidentialMigration/ConfidentialMigrationDxe.c b/OvmfPkg/AmdSev/ConfidentialMigration/ConfidentialMigrationDxe.c
index f0dfbd279e..a981aaeac7 100644
--- a/OvmfPkg/AmdSev/ConfidentialMigration/ConfidentialMigrationDxe.c
+++ b/OvmfPkg/AmdSev/ConfidentialMigration/ConfidentialMigrationDxe.c
@@ -6,14 +6,88 @@
 **/
 
 #include <Library/DebugLib.h>
+#include <Library/BaseMemoryLib.h>
 #include <Library/UefiLib.h>
 
+//
+// Functions implemented by the migration handler
+//
+#define MH_FUNC_INIT          0
+#define MH_FUNC_SAVE_PAGE     1
+#define MH_FUNC_RESTORE_PAGE  2
+#define MH_FUNC_RESET         3
+
+//
+// Return codes for MH functions
+//
+#define MH_SUCCESS            0
+#define MH_INVALID_FUNC     (-1)
+#define MH_AUTH_ERR         (-2)
+
+//
+// Mailbox for communication with hypervisor
+//
+typedef volatile struct {
+  UINT64       Nr;
+  UINT64       Gpa;
+  UINT32       DoPrefetch;
+  UINT32       Ret;
+  UINT32       Go;
+  UINT32       Done;
+} MH_COMMAND_PARAMETERS;
+
+
 VOID
 EFIAPI
 MigrationHandlerMain ()
 {
+  UINT64                       MailboxStart;
+  MH_COMMAND_PARAMETERS        *Params;
+  VOID                         *PageVa;
+
   DebugPrint (DEBUG_INFO,"Migration Handler Started\n");
 
+  MailboxStart = PcdGet32 (PcdConfidentialMigrationMailboxBase);
+  Params = (VOID *)MailboxStart;
+  PageVa = (VOID *)(MailboxStart + 0x1000);
+
+  DisableInterrupts ();
+  Params->Go = 0;
+
+  while (1) {
+    while (!Params->Go) {
+      CpuPause ();
+    }
+    Params->Done = 0;
+
+    switch (Params->Nr) {
+    case MH_FUNC_INIT:
+      Params->Ret = MH_SUCCESS;
+      break;
+
+    case MH_FUNC_SAVE_PAGE:
+      CopyMem (PageVa, (VOID *)Params->Gpa, 4096);
+      Params->Ret = MH_SUCCESS;
+      break;
+
+    case MH_FUNC_RESTORE_PAGE:
+      CopyMem ((VOID *)Params->Gpa, PageVa, 4096);
+      Params->Ret = MH_SUCCESS;
+      break;
+
+    case MH_FUNC_RESET:
+      Params->Ret = MH_SUCCESS;
+      break;
+
+    default:
+      Params->Ret = MH_INVALID_FUNC;
+      break;
+    }
+
+    Params->Go = 0;
+    Params->Done = 1;
+
+  }
 }
 
 /**
-- 
2.20.1

[RFC PATCH 5/9] OvmfPkg/AmdSev: Build page table for migration handler

[Tobin Feldman-Fitzthum]

From: Dov Murik <dovmurik@linux.vnet.ibm.com>

The migration handler builds its own page tables and switches
to them. The MH pagetables are reserved as runtime memory.

When the hypervisor asks the MH to import/export a page, the HV
writes the guest physical address of the page in question to the
mailbox. The MH uses an identity mapping so that it can read/write
whatever GPA is requested by the HV. The hypervisor only asks the
MH to import/export encrypted pages. Thus, the C-Bit can be set
for every page in the identity map.

The MH also needs to read shared pages, such as the mailbox.
These are mapped at an offset. The offset must be added to
the physical address before it can be resolved.

Signed-off-by: Tobin Feldman-Fitzthum <tobin@linux.ibm.com>
Signed-off-by: Dov Murik <dovmurik@linux.vnet.ibm.com>
---
 .../ConfidentialMigrationDxe.inf              |   1 +
 .../ConfidentialMigration/VirtualMemory.h     | 177 ++++++++++++++++++
 .../ConfidentialMigrationDxe.c                |  73 +++++++-
 3 files changed, 249 insertions(+), 2 deletions(-)
 create mode 100644 OvmfPkg/AmdSev/ConfidentialMigration/VirtualMemory.h

diff --git a/OvmfPkg/AmdSev/ConfidentialMigration/ConfidentialMigrationDxe.inf b/OvmfPkg/AmdSev/ConfidentialMigration/ConfidentialMigrationDxe.inf
index cb5609271c..42875095fc 100644
--- a/OvmfPkg/AmdSev/ConfidentialMigration/ConfidentialMigrationDxe.inf
+++ b/OvmfPkg/AmdSev/ConfidentialMigration/ConfidentialMigrationDxe.inf
@@ -15,6 +15,7 @@
 
 [Sources]
   ConfidentialMigrationDxe.c
+  VirtualMemory.h
 
 [Packages]
   MdePkg/MdePkg.dec
diff --git a/OvmfPkg/AmdSev/ConfidentialMigration/VirtualMemory.h b/OvmfPkg/AmdSev/ConfidentialMigration/VirtualMemory.h
new file mode 100644
index 0000000000..c50cb64c63
--- /dev/null
+++ b/OvmfPkg/AmdSev/ConfidentialMigration/VirtualMemory.h
@@ -0,0 +1,177 @@
+/** @file
+  Virtual Memory Management Services to set or clear the memory encryption bit
+  Copyright (c) 2006 - 2016, Intel Corporation. All rights reserved.<BR>
+  Copyright (c) 2017, AMD Incorporated. All rights reserved.<BR>
+  SPDX-License-Identifier: BSD-2-Clause-Patent
+  Code is derived from OvmfPkg/Library/BaseMemEncryptSevLib/X64/VirtualMemory.h
+
+**/
+
+#ifndef __VIRTUAL_MEMORY__
+#define __VIRTUAL_MEMORY__
+
+#include <Library/BaseLib.h>
+#include <Library/BaseMemoryLib.h>
+#include <Library/CacheMaintenanceLib.h>
+#include <Library/DebugLib.h>
+#include <Library/MemoryAllocationLib.h>
+#include <Uefi.h>
+
+#define SYS_CODE64_SEL 0x38
+
+#pragma pack(1)
+
+//
+// Page-Map Level-4 Offset (PML4) and
+// Page-Directory-Pointer Offset (PDPE) entries 4K & 2MB
+//
+
+typedef union {
+  struct {
+    UINT64  Present:1;                // 0 = Not present in memory,
+                                      //   1 = Present in memory
+    UINT64  ReadWrite:1;              // 0 = Read-Only, 1= Read/Write
+    UINT64  UserSupervisor:1;         // 0 = Supervisor, 1=User
+    UINT64  WriteThrough:1;           // 0 = Write-Back caching,
+                                      //   1 = Write-Through caching
+    UINT64  CacheDisabled:1;          // 0 = Cached, 1=Non-Cached
+    UINT64  Accessed:1;               // 0 = Not accessed,
+                                      //   1 = Accessed (set by CPU)
+    UINT64  Reserved:1;               // Reserved
+    UINT64  MustBeZero:2;             // Must Be Zero
+    UINT64  Available:3;              // Available for use by system software
+    UINT64  PageTableBaseAddress:40;  // Page Table Base Address
+    UINT64  AvabilableHigh:11;        // Available for use by system software
+    UINT64  Nx:1;                     // No Execute bit
+  } Bits;
+  UINT64    Uint64;
+} PAGE_MAP_AND_DIRECTORY_POINTER;
+
+//
+// Page Table Entry 4KB
+//
+typedef union {
+  struct {
+    UINT64  Present:1;                // 0 = Not present in memory,
+                                      //   1 = Present in memory
+    UINT64  ReadWrite:1;              // 0 = Read-Only, 1= Read/Write
+    UINT64  UserSupervisor:1;         // 0 = Supervisor, 1=User
+    UINT64  WriteThrough:1;           // 0 = Write-Back caching,
+                                      //   1 = Write-Through caching
+    UINT64  CacheDisabled:1;          // 0 = Cached, 1=Non-Cached
+    UINT64  Accessed:1;               // 0 = Not accessed,
+                                      //   1 = Accessed (set by CPU)
+    UINT64  Dirty:1;                  // 0 = Not Dirty, 1 = written by
+                                      //   processor on access to page
+    UINT64  PAT:1;                    //
+    UINT64  Global:1;                 // 0 = Not global page, 1 = global page
+                                      //   TLB not cleared on CR3 write
+    UINT64  Available:3;              // Available for use by system software
+    UINT64  PageTableBaseAddress:40;  // Page Table Base Address
+    UINT64  AvabilableHigh:11;        // Available for use by system software
+    UINT64  Nx:1;                     // 0 = Execute Code,
+                                      //   1 = No Code Execution
+  } Bits;
+  UINT64    Uint64;
+} PAGE_TABLE_4K_ENTRY;
+
+//
+// Page Table Entry 2MB
+//
+typedef union {
+  struct {
+    UINT64  Present:1;                // 0 = Not present in memory,
+                                      //   1 = Present in memory
+    UINT64  ReadWrite:1;              // 0 = Read-Only, 1= Read/Write
+    UINT64  UserSupervisor:1;         // 0 = Supervisor, 1=User
+    UINT64  WriteThrough:1;           // 0 = Write-Back caching,
+                                      //   1=Write-Through caching
+    UINT64  CacheDisabled:1;          // 0 = Cached, 1=Non-Cached
+    UINT64  Accessed:1;               // 0 = Not accessed,
+                                      //   1 = Accessed (set by CPU)
+    UINT64  Dirty:1;                  // 0 = Not Dirty, 1 = written by
+                                      //   processor on access to page
+    UINT64  MustBe1:1;                // Must be 1
+    UINT64  Global:1;                 // 0 = Not global page, 1 = global page
+                                      //   TLB not cleared on CR3 write
+    UINT64  Available:3;              // Available for use by system software
+    UINT64  PAT:1;                    //
+    UINT64  MustBeZero:8;             // Must be zero;
+    UINT64  PageTableBaseAddress:31;  // Page Table Base Address
+    UINT64  AvabilableHigh:11;        // Available for use by system software
+    UINT64  Nx:1;                     // 0 = Execute Code,
+                                      //   1 = No Code Execution
+  } Bits;
+  UINT64    Uint64;
+} PAGE_TABLE_ENTRY;
+
+//
+// Page Table Entry 1GB
+//
+typedef union {
+  struct {
+    UINT64  Present:1;                // 0 = Not present in memory,
+                                      //   1 = Present in memory
+    UINT64  ReadWrite:1;              // 0 = Read-Only, 1= Read/Write
+    UINT64  UserSupervisor:1;         // 0 = Supervisor, 1=User
+    UINT64  WriteThrough:1;           // 0 = Write-Back caching,
+                                      //   1 = Write-Through caching
+    UINT64  CacheDisabled:1;          // 0 = Cached, 1=Non-Cached
+    UINT64  Accessed:1;               // 0 = Not accessed,
+                                      //   1 = Accessed (set by CPU)
+    UINT64  Dirty:1;                  // 0 = Not Dirty, 1 = written by
+                                      //   processor on access to page
+    UINT64  MustBe1:1;                // Must be 1
+    UINT64  Global:1;                 // 0 = Not global page, 1 = global page
+                                      //   TLB not cleared on CR3 write
+    UINT64  Available:3;              // Available for use by system software
+    UINT64  PAT:1;                    //
+    UINT64  MustBeZero:17;            // Must be zero;
+    UINT64  PageTableBaseAddress:22;  // Page Table Base Address
+    UINT64  AvabilableHigh:11;        // Available for use by system software
+    UINT64  Nx:1;                     // 0 = Execute Code,
+                                      //   1 = No Code Execution
+  } Bits;
+  UINT64    Uint64;
+} PAGE_TABLE_1G_ENTRY;
+
+#pragma pack()
+
+#define IA32_PG_P                   BIT0
+#define IA32_PG_RW                  BIT1
+#define IA32_PG_PS                  BIT7
+
+#define PAGING_PAE_INDEX_MASK       0x1FF
+
+#define PAGING_4K_ADDRESS_MASK_64   0x000FFFFFFFFFF000ull
+#define PAGING_2M_ADDRESS_MASK_64   0x000FFFFFFFE00000ull
+#define PAGING_1G_ADDRESS_MASK_64   0x000FFFFFC0000000ull
+
+#define PAGING_L1_ADDRESS_SHIFT     12
+#define PAGING_L2_ADDRESS_SHIFT     21
+#define PAGING_L3_ADDRESS_SHIFT     30
+#define PAGING_L4_ADDRESS_SHIFT     39
+
+#define PAGING_PML4E_NUMBER         4
+
+#define PAGETABLE_ENTRY_MASK        ((1UL << 9) - 1)
+#define PML4_OFFSET(x)              ( (x >> 39) & PAGETABLE_ENTRY_MASK)
+#define PDP_OFFSET(x)               ( (x >> 30) & PAGETABLE_ENTRY_MASK)
+#define PDE_OFFSET(x)               ( (x >> 21) & PAGETABLE_ENTRY_MASK)
+#define PTE_OFFSET(x)               ( (x >> 12) & PAGETABLE_ENTRY_MASK)
+#define PAGING_1G_ADDRESS_MASK_64   0x000FFFFFC0000000ull
+
+#define PAGE_TABLE_POOL_ALIGNMENT   BASE_2MB
+#define PAGE_TABLE_POOL_UNIT_SIZE   SIZE_2MB
+#define PAGE_TABLE_POOL_UNIT_PAGES  \
+  EFI_SIZE_TO_PAGES (PAGE_TABLE_POOL_UNIT_SIZE)
+#define PAGE_TABLE_POOL_ALIGN_MASK  \
+  (~(EFI_PHYSICAL_ADDRESS)(PAGE_TABLE_POOL_ALIGNMENT - 1))
+
+typedef struct {
+  VOID            *NextPool;
+  UINTN           Offset;
+  UINTN           FreePages;
+} PAGE_TABLE_POOL;
+
+#endif
diff --git a/OvmfPkg/AmdSev/ConfidentialMigration/ConfidentialMigrationDxe.c b/OvmfPkg/AmdSev/ConfidentialMigration/ConfidentialMigrationDxe.c
index a981aaeac7..34d449fe10 100644
--- a/OvmfPkg/AmdSev/ConfidentialMigration/ConfidentialMigrationDxe.c
+++ b/OvmfPkg/AmdSev/ConfidentialMigration/ConfidentialMigrationDxe.c
@@ -9,6 +9,8 @@
 #include <Library/BaseMemoryLib.h>
 #include <Library/UefiLib.h>
 
+#include "VirtualMemory.h"
+
 //
 // Functions implemented by the migration handler
 //
@@ -36,6 +38,71 @@ typedef volatile struct {
   UINT32       Done;
 } MH_COMMAND_PARAMETERS;
 
+//
+// Offset for non-cbit mapping.
+//
+#define UNENC_VIRT_ADDR_BASE    0xffffff8000000000ULL
+
+STATIC PAGE_TABLE_POOL   *mPageTablePool = NULL;
+PHYSICAL_ADDRESS  mMigrationHandlerPageTables = 0;
+
+/**
+  Allocates and fills in custom page tables for Migration Handler.
+  The MH must be able to write to any encrypted page. Thus, it
+  uses an identity map where the C-bit is set for every page. The
+  HV should never ask the MH to import/export a shared page. The
+  MH must also be able to read some shared pages. The first 1GB
+  of memory is mapped at offset UNENC_VIRT_ADDR_BASE.
+**/
+VOID
+PrepareMigrationHandlerPageTables (
+  VOID
+  )
+{
+  UINTN                            PoolPages;
+  VOID                             *Buffer;
+  VOID                             *Start;
+  PAGE_MAP_AND_DIRECTORY_POINTER   *PageMapLevel4Entry;
+  PAGE_TABLE_1G_ENTRY              *PageDirectory1GEntry;
+  PAGE_TABLE_1G_ENTRY              *Unenc1GEntry;
+  UINT64                           AddressEncMask;
+
+  PoolPages = 1 + 10;
+  Buffer = AllocateAlignedRuntimePages (PoolPages, PAGE_TABLE_POOL_ALIGNMENT);
+  mPageTablePool = Buffer;
+  mPageTablePool->NextPool = mPageTablePool;
+  mPageTablePool->FreePages  = PoolPages - 1;
+  mPageTablePool->Offset = EFI_PAGES_TO_SIZE (1);
+
+  Start = (UINT8 *)mPageTablePool + mPageTablePool->Offset;
+  ZeroMem(Start, mPageTablePool->FreePages * EFI_PAGE_SIZE);
+
+  AddressEncMask = 1ULL << 47;
+
+  PageMapLevel4Entry = Start;
+  PageDirectory1GEntry = (PAGE_TABLE_1G_ENTRY*)((UINT8*)Start + EFI_PAGE_SIZE);
+  Unenc1GEntry = (PAGE_TABLE_1G_ENTRY*)((UINT8*)Start + 2 * EFI_PAGE_SIZE);
+
+  PageMapLevel4Entry = Start;
+  PageMapLevel4Entry += PML4_OFFSET(0x0ULL);
+  PageMapLevel4Entry->Uint64 = (UINT64)PageDirectory1GEntry | AddressEncMask | 0x23;
+
+  PageMapLevel4Entry = Start;
+  PageMapLevel4Entry += PML4_OFFSET(UNENC_VIRT_ADDR_BASE); // should be 511
+  PageMapLevel4Entry->Uint64 = (UINT64)Unenc1GEntry | AddressEncMask | 0x23;
+
+  UINT64 PageAddr = 0;
+  for (int i = 0; i < 512; i++, PageAddr += SIZE_1GB) {
+    PAGE_TABLE_1G_ENTRY *e = PageDirectory1GEntry + i;
+    e->Uint64 = PageAddr | AddressEncMask | 0xe3; // 1GB page
+  }
+
+  UINT64 UnencPageAddr = 0;
+  Unenc1GEntry->Uint64 = UnencPageAddr | 0xe3; // 1GB page unencrypted
+
+  mMigrationHandlerPageTables = (UINT64)Start | AddressEncMask;
+}
+
 
 VOID
 EFIAPI
@@ -48,8 +115,8 @@ MigrationHandlerMain ()
   DebugPrint (DEBUG_INFO,"Migration Handler Started\n");
 
   MailboxStart = PcdGet32 (PcdConfidentialMigrationMailboxBase);
-  Params = (VOID *)MailboxStart;
-  PageVa = (VOID *)(MailboxStart + 0x1000);
+  Params = (VOID *)(MailboxStart + UNENC_VIRT_ADDR_BASE);
+  PageVa = (VOID *)(MailboxStart + UNENC_VIRT_ADDR_BASE + 0x1000);
 
   DisableInterrupts ();
   Params->Go = 0;
@@ -108,6 +175,8 @@ SetupMigrationHandler (
     return 0;
   }
 
+	PrepareMigrationHandlerPageTables ();
+
   //
   // If VM is migration target, wait until hypervisor modifies CPU state
   // and restarts execution.
-- 
2.20.1

[RFC PATCH 6/9] OvmfPkg/AmdSev: Don't overwrite mailbox or pagetables

[Tobin Feldman-Fitzthum]

While restoring pages, the MH should avoid overwriting its
pagetables or the mailbox it uses to communicate with the HV.

Signed-off-by: Tobin Feldman-Fitzthum <tobin@linux.ibm.com>
---
 .../ConfidentialMigrationDxe.c                  | 17 ++++++++++++++++-
 1 file changed, 16 insertions(+), 1 deletion(-)

diff --git a/OvmfPkg/AmdSev/ConfidentialMigration/ConfidentialMigrationDxe.c b/OvmfPkg/AmdSev/ConfidentialMigration/ConfidentialMigrationDxe.c
index 34d449fe10..ee1466eb00 100644
--- a/OvmfPkg/AmdSev/ConfidentialMigration/ConfidentialMigrationDxe.c
+++ b/OvmfPkg/AmdSev/ConfidentialMigration/ConfidentialMigrationDxe.c
@@ -109,6 +109,9 @@ EFIAPI
 MigrationHandlerMain ()
 {
   UINT64                       MailboxStart;
+  UINT64                       MailboxEnd;
+  UINT64                       PagetableStart;
+  UINT64                       PagetableEnd;
   MH_COMMAND_PARAMETERS        *Params;
   VOID                         *PageVa;
 
@@ -118,6 +121,11 @@ MigrationHandlerMain ()
   Params = (VOID *)(MailboxStart + UNENC_VIRT_ADDR_BASE);
   PageVa = (VOID *)(MailboxStart + UNENC_VIRT_ADDR_BASE + 0x1000);
 
+  MailboxEnd = MailboxStart + 2 * EFI_PAGE_SIZE;
+
+  PagetableStart = mMigrationHandlerPageTables;
+  PagetableEnd = PagetableStart + 11 * EFI_PAGE_SIZE;
+
   DisableInterrupts ();
   Params->Go = 0;
 
@@ -138,7 +146,14 @@ MigrationHandlerMain ()
       break;
 
     case MH_FUNC_RESTORE_PAGE:
-      CopyMem ((VOID *)Params->Gpa, PageVa, 4096);
+      //
+      // Don't import a page that covers the mailbox or pagetables.
+      //
+      if (!((Params->Gpa >= MailboxStart && Params->Gpa < MailboxEnd) ||
+          (Params->Gpa >= PagetableStart && Params->Gpa < PagetableEnd))) {
+
+        CopyMem ((VOID *)Params->Gpa, PageVa, 4096);
+      }
       Params->Ret = MH_SUCCESS;
       break;
 
-- 
2.20.1

[RFC PATCH 7/9] OvmfPkg/AmdSev: Don't overwrite MH stack

[Tobin Feldman-Fitzthum]

The Migration Handler uses its own stack and should avoid
overwriting the stack when importing pages.

Signed-off-by: Tobin Feldman-Fitzthum <tobin@linux.ibm.com>
---
 .../ConfidentialMigrationDxe.c                  | 17 +++++++++++++++--
 1 file changed, 15 insertions(+), 2 deletions(-)

diff --git a/OvmfPkg/AmdSev/ConfidentialMigration/ConfidentialMigrationDxe.c b/OvmfPkg/AmdSev/ConfidentialMigration/ConfidentialMigrationDxe.c
index ee1466eb00..2de35a7bb1 100644
--- a/OvmfPkg/AmdSev/ConfidentialMigration/ConfidentialMigrationDxe.c
+++ b/OvmfPkg/AmdSev/ConfidentialMigration/ConfidentialMigrationDxe.c
@@ -45,6 +45,8 @@ typedef volatile struct {
 
 STATIC PAGE_TABLE_POOL   *mPageTablePool = NULL;
 PHYSICAL_ADDRESS  mMigrationHandlerPageTables = 0;
+PHYSICAL_ADDRESS  mMigrationHandlerStackBase = 0;
+UINT32            mMigrationHandlerStackSize = 4;
 
 /**
   Allocates and fills in custom page tables for Migration Handler.
@@ -112,6 +114,8 @@ MigrationHandlerMain ()
   UINT64                       MailboxEnd;
   UINT64                       PagetableStart;
   UINT64                       PagetableEnd;
+  UINT64                       StackStart;
+  UINT64                       StackEnd;
   MH_COMMAND_PARAMETERS        *Params;
   VOID                         *PageVa;
 
@@ -126,6 +130,9 @@ MigrationHandlerMain ()
   PagetableStart = mMigrationHandlerPageTables;
   PagetableEnd = PagetableStart + 11 * EFI_PAGE_SIZE;
 
+  StackStart = mMigrationHandlerStackBase;
+  StackEnd = StackStart + mMigrationHandlerStackSize;
+
   DisableInterrupts ();
   Params->Go = 0;
 
@@ -147,10 +154,11 @@ MigrationHandlerMain ()
 
     case MH_FUNC_RESTORE_PAGE:
       //
-      // Don't import a page that covers the mailbox or pagetables.
+      // Don't import a page that covers the mailbox, pagetables, or stack.
       //
       if (!((Params->Gpa >= MailboxStart && Params->Gpa < MailboxEnd) ||
-          (Params->Gpa >= PagetableStart && Params->Gpa < PagetableEnd))) {
+          (Params->Gpa >= PagetableStart && Params->Gpa < PagetableEnd) ||
+          (Params->Gpa >= StackStart && Params->Gpa < StackEnd))) {
 
         CopyMem ((VOID *)Params->Gpa, PageVa, 4096);
       }
@@ -190,6 +198,11 @@ SetupMigrationHandler (
     return 0;
   }
 
+  //
+  // Setup stack and pagetables for Migration Handler
+  //
+  mMigrationHandlerStackBase = (UINTN)AllocateAlignedRuntimePages (mMigrationHandlerStackSize, PAGE_TABLE_POOL_ALIGNMENT);
+
 	PrepareMigrationHandlerPageTables ();
 
   //
-- 
2.20.1

[RFC PATCH 8/9] OvmfPkg/AmdSev: Add Migration Handler entry point

[Tobin Feldman-Fitzthum]

The Migration Handler runs in the mirror VM. The MH is started
directly by the hypervisor. SetupMigrationHandler runs in the main
VM and sets up the migration entry point. The HV starts execution
of the mirror vCPU at the entry point, which trampolines to
MigrationHandlerMain

Signed-off-by: Tobin Feldman-Fitzthum <tobin@linux.ibm.com>
---
 OvmfPkg/OvmfPkg.dec                           |  3 ++
 OvmfPkg/AmdSev/AmdSevX64.fdf                  |  3 ++
 .../ConfidentialMigrationDxe.inf              |  2 +
 .../ConfidentialMigrationPei.inf              |  2 +
 .../ConfidentialMigrationDxe.c                | 48 +++++++++++++++++
 .../ConfidentialMigrationPei.c                |  6 +++
 .../MigrationEntryPoint.nasm                  | 51 +++++++++++++++++++
 7 files changed, 115 insertions(+)
 create mode 100644 OvmfPkg/AmdSev/ConfidentialMigration/MigrationEntryPoint.nasm

diff --git a/OvmfPkg/OvmfPkg.dec b/OvmfPkg/OvmfPkg.dec
index 1252582c99..c6e07accf6 100644
--- a/OvmfPkg/OvmfPkg.dec
+++ b/OvmfPkg/OvmfPkg.dec
@@ -336,6 +336,9 @@
   gUefiOvmfPkgTokenSpaceGuid.PcdConfidentialMigrationMailboxBase|0x0|UINT32|0x4b
   gUefiOvmfPkgTokenSpaceGuid.PcdConfidentialMigrationMailboxSize|0x0|UINT32|0x4c
 
+  gUefiOvmfPkgTokenSpaceGuid.PcdConfidentialMigrationEntryBase|0x0|UINT32|0x4d
+  gUefiOvmfPkgTokenSpaceGuid.PcdConfidentialMigrationEntrySize|0x0|UINT32|0x4e
+
 [PcdsDynamic, PcdsDynamicEx]
   gUefiOvmfPkgTokenSpaceGuid.PcdEmuVariableEvent|0|UINT64|2
   gUefiOvmfPkgTokenSpaceGuid.PcdOvmfFlashVariablesEnable|FALSE|BOOLEAN|0x10
diff --git a/OvmfPkg/AmdSev/AmdSevX64.fdf b/OvmfPkg/AmdSev/AmdSevX64.fdf
index a8e296e641..8687fadfcc 100644
--- a/OvmfPkg/AmdSev/AmdSevX64.fdf
+++ b/OvmfPkg/AmdSev/AmdSevX64.fdf
@@ -74,6 +74,9 @@ gUefiOvmfPkgTokenSpaceGuid.PcdOvmfSecPeiTempRamBase|gUefiOvmfPkgTokenSpaceGuid.P
 0x020000|0x003000
 gUefiOvmfPkgTokenSpaceGuid.PcdConfidentialMigrationMailboxBase|gUefiOvmfPkgTokenSpaceGuid.PcdConfidentialMigrationMailboxSize
 
+0x023000|0x001000
+gUefiOvmfPkgTokenSpaceGuid.PcdConfidentialMigrationEntryBase|gUefiOvmfPkgTokenSpaceGuid.PcdConfidentialMigrationEntrySize
+
 0x120000|0x0E0000
 gUefiOvmfPkgTokenSpaceGuid.PcdOvmfPeiMemFvBase|gUefiOvmfPkgTokenSpaceGuid.PcdOvmfPeiMemFvSize
 FV = PEIFV
diff --git a/OvmfPkg/AmdSev/ConfidentialMigration/ConfidentialMigrationDxe.inf b/OvmfPkg/AmdSev/ConfidentialMigration/ConfidentialMigrationDxe.inf
index 42875095fc..b879037586 100644
--- a/OvmfPkg/AmdSev/ConfidentialMigration/ConfidentialMigrationDxe.inf
+++ b/OvmfPkg/AmdSev/ConfidentialMigration/ConfidentialMigrationDxe.inf
@@ -16,6 +16,7 @@
 [Sources]
   ConfidentialMigrationDxe.c
   VirtualMemory.h
+  MigrationEntryPoint.nasm
 
 [Packages]
   MdePkg/MdePkg.dec
@@ -31,6 +32,7 @@
   gUefiOvmfPkgTokenSpaceGuid.PcdIsConfidentialMigrationTarget
   gUefiOvmfPkgTokenSpaceGuid.PcdStartConfidentialMigrationHandler
   gUefiOvmfPkgTokenSpaceGuid.PcdConfidentialMigrationMailboxBase
+  gUefiOvmfPkgTokenSpaceGuid.PcdConfidentialMigrationEntryBase
 
 [Depex]
   TRUE
diff --git a/OvmfPkg/AmdSev/ConfidentialMigration/ConfidentialMigrationPei.inf b/OvmfPkg/AmdSev/ConfidentialMigration/ConfidentialMigrationPei.inf
index 918cf22abd..6233b82cc2 100644
--- a/OvmfPkg/AmdSev/ConfidentialMigration/ConfidentialMigrationPei.inf
+++ b/OvmfPkg/AmdSev/ConfidentialMigration/ConfidentialMigrationPei.inf
@@ -30,6 +30,8 @@
 [FixedPcd]
   gUefiOvmfPkgTokenSpaceGuid.PcdConfidentialMigrationMailboxBase
   gUefiOvmfPkgTokenSpaceGuid.PcdConfidentialMigrationMailboxSize
+  gUefiOvmfPkgTokenSpaceGuid.PcdConfidentialMigrationEntryBase
+  gUefiOvmfPkgTokenSpaceGuid.PcdConfidentialMigrationEntrySize
 
 [Depex]
   TRUE
diff --git a/OvmfPkg/AmdSev/ConfidentialMigration/ConfidentialMigrationDxe.c b/OvmfPkg/AmdSev/ConfidentialMigration/ConfidentialMigrationDxe.c
index 2de35a7bb1..5e96206d17 100644
--- a/OvmfPkg/AmdSev/ConfidentialMigration/ConfidentialMigrationDxe.c
+++ b/OvmfPkg/AmdSev/ConfidentialMigration/ConfidentialMigrationDxe.c
@@ -38,11 +38,23 @@ typedef volatile struct {
   UINT32       Done;
 } MH_COMMAND_PARAMETERS;
 
+//
+// Addresses to be used in the entry point
+//
+typedef struct {
+  UINT32 Cr3;
+  UINT64 StackBase;
+  UINT64 MhBase;
+} ENTRY_ADDRS;
+
 //
 // Offset for non-cbit mapping.
 //
 #define UNENC_VIRT_ADDR_BASE    0xffffff8000000000ULL
 
+void MigrationHandlerEntryPoint(void);
+void MigrationHandlerEntryPoint64(void);
+
 STATIC PAGE_TABLE_POOL   *mPageTablePool = NULL;
 PHYSICAL_ADDRESS  mMigrationHandlerPageTables = 0;
 PHYSICAL_ADDRESS  mMigrationHandlerStackBase = 0;
@@ -193,6 +205,16 @@ SetupMigrationHandler (
   IN EFI_SYSTEM_TABLE     *SystemTable
   )
 {
+  UINT32            LongModeOffset;
+  UINT32            EntryAddrsOffset;
+  UINT32            GdtOffset;
+  IA32_DESCRIPTOR   GdtPtr;
+  UINT64            EntryPoint;
+  ENTRY_ADDRS       *EntryData;
+
+  LongModeOffset = 0x200;
+  EntryAddrsOffset = 0x400;
+  GdtOffset = 0x600;
 
   if (!PcdGetBool(PcdStartConfidentialMigrationHandler)) {
     return 0;
@@ -205,6 +227,32 @@ SetupMigrationHandler (
 
 	PrepareMigrationHandlerPageTables ();
 
+  //
+  // Copy Migration Handler entry point to a known location.
+  //
+  EntryPoint = PcdGet32 (PcdConfidentialMigrationEntryBase);
+  CopyMem ((void *)EntryPoint, MigrationHandlerEntryPoint, 0x50);
+
+  CopyMem ((void *)(EntryPoint + LongModeOffset),
+      MigrationHandlerEntryPoint64, 0x50);
+
+  //
+  // Copy Migration Handler GDT to a known location.
+  //
+  AsmReadGdtr (&GdtPtr);
+  CopyMem ((void *)(EntryPoint + GdtOffset), (void *)GdtPtr.Base,
+      GdtPtr.Limit);
+
+  //
+  // Populate entry point with address of page tables, stack,
+  // and MigrationHandlerMain
+  //
+  EntryData = (void *)(EntryPoint + EntryAddrsOffset);
+
+  EntryData->Cr3 = mMigrationHandlerPageTables;
+  EntryData->StackBase = mMigrationHandlerStackBase;
+  EntryData->MhBase = (UINT64)MigrationHandlerMain;
+
   //
   // If VM is migration target, wait until hypervisor modifies CPU state
   // and restarts execution.
diff --git a/OvmfPkg/AmdSev/ConfidentialMigration/ConfidentialMigrationPei.c b/OvmfPkg/AmdSev/ConfidentialMigration/ConfidentialMigrationPei.c
index ce304bc07b..5371ef23a9 100644
--- a/OvmfPkg/AmdSev/ConfidentialMigration/ConfidentialMigrationPei.c
+++ b/OvmfPkg/AmdSev/ConfidentialMigration/ConfidentialMigrationPei.c
@@ -21,5 +21,11 @@ InitializeConfidentialMigrationPei (
     EfiRuntimeServicesData
     );
 
+  BuildMemoryAllocationHob (
+    PcdGet32 (PcdConfidentialMigrationEntryBase),
+    PcdGet32 (PcdConfidentialMigrationEntrySize),
+    EfiRuntimeServicesData
+    );
+
   return EFI_SUCCESS;
 }
diff --git a/OvmfPkg/AmdSev/ConfidentialMigration/MigrationEntryPoint.nasm b/OvmfPkg/AmdSev/ConfidentialMigration/MigrationEntryPoint.nasm
new file mode 100644
index 0000000000..9375771b88
--- /dev/null
+++ b/OvmfPkg/AmdSev/ConfidentialMigration/MigrationEntryPoint.nasm
@@ -0,0 +1,51 @@
+; Entrypoint for Migration Handler
+
+  DEFAULT REL
+  SECTION .text
+
+%define ENABLE_DEBUG   1
+%define X86_CR0_PG     BIT31
+%define X86_EFER_LME   BIT8
+%define X86_CR4_PAE    BIT5
+
+%define ENTRY_BASE FixedPcdGet32 (PcdConfidentialMigrationEntryBase)
+
+%define LONG_MODE_OFFSET 0x200;
+%define ENTRY_ADDRS_OFFSET 0x400
+%define GDT_OFFSET 0x600
+
+%define LONG_MODE_ADDR ENTRY_BASE + LONG_MODE_OFFSET
+%define LINEAR_CODE64_SEL 0x38
+
+BITS 32
+
+global ASM_PFX(MigrationHandlerEntryPoint)
+ASM_PFX(MigrationHandlerEntryPoint):
+
+  ; CR3
+  mov edi, [ENTRY_BASE + ENTRY_ADDRS_OFFSET]
+  mov cr3, edi
+
+  ; EFER.LME
+  mov     ecx, 0xc0000080
+  rdmsr
+  bts      eax, 8
+  wrmsr
+
+  ; CR0.PG
+  mov   eax, cr0
+  bts   eax, 31
+  mov   cr0, eax
+
+  ; Far jump to enter long mode
+  jmp LINEAR_CODE64_SEL:LONG_MODE_ADDR
+
+BITS 64
+global ASM_PFX(MigrationHandlerEntryPoint64)
+ASM_PFX(MigrationHandlerEntryPoint64):
+
+  ; RSP
+  mov    rsp, [ENTRY_BASE + ENTRY_ADDRS_OFFSET + 0x8]
+
+  ; Jump to MH
+  jmp [ENTRY_BASE + ENTRY_ADDRS_OFFSET + 0x10]
-- 
2.20.1

[RFC PATCH 9/9] OvmfPkg/ResetVector: Expose Migration Handler Entry Addresses

[Tobin Feldman-Fitzthum]

Exposes the address of the Migration Handler entry point via a
GUIDed struct. To support migration, the HV should find this
struct and start one vCPU at the entry point address.

Signed-off-by: Tobin Feldman-Fitzthum <tobin@linux.ibm.com>
---
 OvmfPkg/ResetVector/ResetVector.inf          |  1 +
 OvmfPkg/ResetVector/Ia16/ResetVectorVtf0.asm | 13 +++++++++++++
 OvmfPkg/ResetVector/ResetVector.nasmb        |  1 +
 3 files changed, 15 insertions(+)

diff --git a/OvmfPkg/ResetVector/ResetVector.inf b/OvmfPkg/ResetVector/ResetVector.inf
index d028c92d8c..642757796c 100644
--- a/OvmfPkg/ResetVector/ResetVector.inf
+++ b/OvmfPkg/ResetVector/ResetVector.inf
@@ -49,3 +49,4 @@
   gUefiOvmfPkgTokenSpaceGuid.PcdSevLaunchSecretSize
   gUefiOvmfPkgTokenSpaceGuid.PcdQemuHashTableBase
   gUefiOvmfPkgTokenSpaceGuid.PcdQemuHashTableSize
+  gUefiOvmfPkgTokenSpaceGuid.PcdConfidentialMigrationEntryBase
diff --git a/OvmfPkg/ResetVector/Ia16/ResetVectorVtf0.asm b/OvmfPkg/ResetVector/Ia16/ResetVectorVtf0.asm
index 7ec3c6e980..fc38bd927d 100644
--- a/OvmfPkg/ResetVector/Ia16/ResetVectorVtf0.asm
+++ b/OvmfPkg/ResetVector/Ia16/ResetVectorVtf0.asm
@@ -86,6 +86,19 @@ sevSecretBlockStart:
     DB      0x80, 0x81, 0x12, 0x7C, 0x90, 0xD3, 0xD2, 0x94
 sevSecretBlockEnd:
 
+;
+; The IP of the migration handler. The hypervisor should start
+; the mirror with this address.
+;
+; GUID = 5c7db037-ab87-4282-b33c-7894f01471ec
+;
+sevMigrationBlockStart:
+    DD      SEV_MIGRATION_ENTRY_IP
+    DW      sevMigrationBlockStart - sevMigrationBlockEnd
+    DB      0x5C, 0x7D, 0xB0, 0x37, 0x87, 0xAB, 0x82, 0x42
+    DB      0xB3, 0x3C, 0x78, 0x94, 0xF0, 0x14, 0x71, 0xEC
+sevMigrationBlockEnd:
+
 ;
 ; SEV-ES Processor Reset support
 ;
diff --git a/OvmfPkg/ResetVector/ResetVector.nasmb b/OvmfPkg/ResetVector/ResetVector.nasmb
index acec46a324..344713bfbb 100644
--- a/OvmfPkg/ResetVector/ResetVector.nasmb
+++ b/OvmfPkg/ResetVector/ResetVector.nasmb
@@ -91,5 +91,6 @@
   %define SEV_LAUNCH_SECRET_SIZE  FixedPcdGet32 (PcdSevLaunchSecretSize)
   %define SEV_FW_HASH_BLOCK_BASE  FixedPcdGet32 (PcdQemuHashTableBase)
   %define SEV_FW_HASH_BLOCK_SIZE  FixedPcdGet32 (PcdQemuHashTableSize)
+  %define SEV_MIGRATION_ENTRY_IP  FixedPcdGet32 (PcdConfidentialMigrationEntryBase)
 %include "Ia16/ResetVectorVtf0.asm"
 
-- 
2.20.1