Re: [edk2-devel] [RFC PATCH 00/14] Firmware Support for Fast Live Migration for AMD SEV

["Singh, Brijesh" <brijesh.singh@amd.com>]

[AMD Official Use Only - Internal Distribution Only]

Hi Yao,

In the current proposal the accelerated migration does not involve the PSP. I will let Tobin and Dov comment on how things works in current prototype.

If PSP was involved in the migration, then flow would be like this:

- During the guest creation time two things will happen (both source and destination VMs go through this step)
  a) create a random VM encryption key (VEK) -- the key is used for encrypting the guest pages.
  b) guest owner supplies a session blob to the PSP. The session blob contains transport encryption key (TEK).  The TEK is used to encrypt all the confidential information exchanged between the PSP and the external entities such as a guest owner or another PSP.

During the migration
i) source VMM asks PSP to get a page that can be migrated.
ii) source PSP  encrypt the guest pages using the TEK
iii) source VMM write the encrypted pages on the wire
iv) destination VMM  will call PSP to put the received encrypted page in the guest memory.
v) destination PSP will decrypt the received pages using TEK, then encrypt it using the VEK before copying it to the guest memory.

As you see in the flow, the PSP's never share the keys. The TEK is wrapped in the session blob provided to the PSP on launch.

You are correct that the SEV/SEV-ES does not support querying the attestation report after the guest boot. All the attestation need to be done during the guest creation time.

With SEV-SNP, a guest OS/BIOS can call PSP to get the attestation report. The SEV-SNP, provides a method in which the guest owner can provide an IMI (Initial migration agent) through the launch process. The IMI will be measured separately and stored in IMD (Initial Migration Digest). When source VMM is ready to migrate it will use a PSP command (VM_EXPORT) to export the data from source to destination. The export will contains information about IMD etc. The destination VMM will use the PSP command (ABSORB) to import the incoming data. During the absorb process the destination PSP will check the IMD to ensure that same IMI is used at the source end. I have cut short few details in the email; See the SEV-SNP spec (section migration 4.11) for more.

Thanks
Brijesh

-----Original Message-----
From: Yao, Jiewen <jiewen.yao@intel.com>
Sent: Friday, March 12, 2021 8:32 PM
To: devel@edk2.groups.io; Yao, Jiewen <jiewen.yao@intel.com>; tobin@linux.ibm.com
Cc: Dov Murik <dovmurik@linux.vnet.ibm.com>; Tobin Feldman-Fitzthum <tobin@ibm.com>; James Bottomley <jejb@linux.ibm.com>; Hubertus Franke <frankeh@us.ibm.com>; Singh, Brijesh <brijesh.singh@amd.com>; Kalra, Ashish <Ashish.Kalra@amd.com>; Grimm, Jon <Jon.Grimm@amd.com>; Lendacky, Thomas <Thomas.Lendacky@amd.com>
Subject: RE: [edk2-devel] [RFC PATCH 00/14] Firmware Support for Fast Live Migration for AMD SEV

Hi
We discuss the patch internally. We do see PROs and CONs with this approach.
The advantage is that it is very simple. In-VM migration can save lots of effort on security context restore.
On the other hand, we feel not so comfortable to reserve a dedicate CPU to achieve that. Similar to the feedback in the community.

Using Hot-Plug is not a solution for Intel TDX as well. It is unsupported now.

I like the idea to diverge the migration boot mode v.s. normal boot mode in SEC phase.
We must be very carefully handle this migration boot mode, to avoid any touching on system memory.
Intel TDX Virtual Firmware skips the PEI phase directly. If we choose this approach, SEC-based migration is our preference.

Besides this patch, we would like to understand a full picture.
1) How the key is passed from source VM to destination?
I saw you mentions: "Key sharing is out of scope for this part of the RFC."
"This will probably be implemented via inject-launch-secret in the future"

Does that mean two PSP will sync with each other and negotiate the key, after the Migration Agent (MA) checks the policy?

2) How the attestation is supported?
I read the whitepaper https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.amd.com%2Fsystem%2Ffiles%2FTechDocs%2FSEV-SNP-strengthening-vm-isolation-with-integrity-protection-and-more.pdf&amp;data=04%7C01%7Cbrijesh.singh%40amd.com%7Cb19ccecd6ca946abd0eb08d8e5c84177%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637511995981376795%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=h67VntbdjigZFvhRfP6%2FGYTE9eqrFDqJRojWqG0C25c%3D&amp;reserved=0.
It seems SEV and SEV-ES only support attestation during launch, I don't believe this migration feature will impact the attestation report. Am I right?
SEV-SNP supports more flexible attestation, does it include any information about the new migrated content?


> -----Original Message-----
> From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Yao,
> Jiewen
> Sent: Thursday, March 4, 2021 9:49 AM
> To: devel@edk2.groups.io; tobin@linux.ibm.com
> Cc: Dov Murik <dovmurik@linux.vnet.ibm.com>; Tobin Feldman-Fitzthum
> <tobin@ibm.com>; James Bottomley <jejb@linux.ibm.com>; Hubertus Franke
> <frankeh@us.ibm.com>; Brijesh Singh <brijesh.singh@amd.com>; Ashish
> Kalra <ashish.kalra@amd.com>; Jon Grimm <jon.grimm@amd.com>; Tom
> Lendacky <thomas.lendacky@amd.com>; Yao, Jiewen <jiewen.yao@intel.com>
> Subject: Re: [edk2-devel] [RFC PATCH 00/14] Firmware Support for Fast
> Live Migration for AMD SEV
>
> Hi Tobin
> Thanks for your patch.
> You may that Intel is working on TDX for the same live migration feature.
>
> Please give me some time (about 1 work week) to digest and evaluate
> the patch and impact.
> Then I will provide feedback.
>
> Thank you
> Yao Jiewen
>
> > -----Original Message-----
> > From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Tobin
> > Feldman-Fitzthum
> > Sent: Wednesday, March 3, 2021 4:48 AM
> > To: devel@edk2.groups.io
> > Cc: Dov Murik <dovmurik@linux.vnet.ibm.com>; Tobin Feldman-Fitzthum
> > <tobin@ibm.com>; Tobin Feldman-Fitzthum <tobin@linux.ibm.com>; James
> > Bottomley <jejb@linux.ibm.com>; Hubertus Franke
> > <frankeh@us.ibm.com>; Brijesh Singh <brijesh.singh@amd.com>; Ashish
> > Kalra
> <ashish.kalra@amd.com>;
> > Jon Grimm <jon.grimm@amd.com>; Tom Lendacky
> > <thomas.lendacky@amd.com>
> > Subject: [edk2-devel] [RFC PATCH 00/14] Firmware Support for Fast
> > Live Migration for AMD SEV
> >
> > This is a demonstration of fast migration for encrypted virtual
> > machines using a Migration Handler that lives in OVMF. This demo
> > uses AMD SEV, but the ideas may generalize to other confidential computing platforms.
> > With AMD SEV, guest memory is encrypted and the hypervisor cannot
> > access or move it. This makes migration tricky. In this demo, we
> > show how the HV can ask a Migration Handler (MH) in the firmware for
> > an encrypted page. The MH encrypts the page with a transport key
> > prior to releasing it to the HV. The target machine also runs an MH
> > that decrypts the page once it is passed in by the target HV. These
> > patches are not ready for production, but the are a full end-to-end
> > solution that facilitates a fast live migration between two SEV VMs.
> >
> > Corresponding patches for QEMU have been posted my colleague Dov
> > Murik on qemu-devel. Our approach needs little kernel support,
> > requiring only one hypercall that the guest can use to mark a page
> > as encrypted or shared. This series includes updated patches from
> > Ashish Kalra and Brijesh Singh that allow OVMF to use this hypercall.
> >
> > The MH runs continuously in the guest, waiting for communication
> > from the HV. The HV starts an additional vCPU for the MH but does
> > not expose it to the guest OS via ACPI. We use the MpService to
> > start the MH. The MpService is only available at runtime and
> > processes that are started by it are usually cleaned up on
> > ExitBootServices. Since we need the MH to run continuously, we had
> > to make some modifications. Ideally a feature could be added to the
> > MpService to allow for the starting of long-running processes.
> > Besides migration, this could support other background processes
> > that need to operate within the encryption boundary. For now, we
> > have included a handful of patches that modify the MpService to
> > allow the MH to keep running after ExitBootServices. These are temporary.
> >
> > Ashish Kalra (2):
> >   OvmfPkg/PlatformPei: Mark SEC GHCB page in the page encrpytion bitmap.
> >   OvmfPkg/PlatformDxe: Add support for SEV live migration.
> >
> > Brijesh Singh (1):
> >   OvmfPkg/BaseMemEncryptLib: Support to issue unencrypted hypercall
> >
> > Dov Murik (1):
> >   OvmfPkg/AmdSev: Build page table for migration handler
> >
> > Tobin Feldman-Fitzthum (10):
> >   OvmfPkg/AmdSev: Base for Confidential Migration Handler
> >   OvmfPkg/PlatfomPei: Set Confidential Migration PCD
> >   OvmfPkg/AmdSev: Setup Migration Handler Mailbox
> >   OvmfPkg/AmdSev: MH support for mailbox protocol
> >   UefiCpuPkg/MpInitLib: temp removal of MpLib cleanup
> >   UefiCpuPkg/MpInitLib: Allocate MP buffer as runtime memory
> >   UefiCpuPkg/CpuExceptionHandlerLib: Exception handling as runtime
> >     memory
> >   OvmfPkg/AmdSev: Don't overwrite mailbox or pagetables
> >   OvmfPkg/AmdSev: Don't overwrite MH stack
> >   OvmfPkg/AmdSev: MH page encryption POC
> >
> >  OvmfPkg/OvmfPkg.dec                           |  11 +
> >  OvmfPkg/AmdSev/AmdSevX64.dsc                  |   2 +
> >  OvmfPkg/AmdSev/AmdSevX64.fdf                  |  13 +-
> >  .../ConfidentialMigrationDxe.inf              |  45 +++
> >  .../ConfidentialMigrationPei.inf              |  35 ++
> >  .../DxeMemEncryptSevLib.inf                   |   1 +
> >  .../PeiMemEncryptSevLib.inf                   |   1 +
> >  OvmfPkg/PlatformDxe/Platform.inf              |   2 +
> >  OvmfPkg/PlatformPei/PlatformPei.inf           |   2 +
> >  UefiCpuPkg/Library/MpInitLib/DxeMpInitLib.inf |   2 +
> >  UefiCpuPkg/Library/MpInitLib/PeiMpInitLib.inf |   2 +
> >  OvmfPkg/AmdSev/ConfidentialMigration/MpLib.h  | 235 +++++++++++++
> >  .../ConfidentialMigration/VirtualMemory.h     | 177 ++++++++++
> >  OvmfPkg/Include/Guid/MemEncryptLib.h          |  16 +
> >  OvmfPkg/PlatformDxe/PlatformConfig.h          |   5 +
> >  .../ConfidentialMigrationDxe.c                | 325 ++++++++++++++++++
> >  .../ConfidentialMigrationPei.c                |  25 ++
> >  .../X64/PeiDxeVirtualMemory.c                 |  18 +
> >  OvmfPkg/PlatformDxe/AmdSev.c                  |  99 ++++++
> >  OvmfPkg/PlatformDxe/Platform.c                |   6 +
> >  OvmfPkg/PlatformPei/AmdSev.c                  |  10 +
> >  OvmfPkg/PlatformPei/Platform.c                |  10 +
> >  .../CpuExceptionHandlerLib/DxeException.c     |   8 +-
> >  UefiCpuPkg/Library/MpInitLib/DxeMpLib.c       |  21 +-
> >  UefiCpuPkg/Library/MpInitLib/MpLib.c          |   7 +-
> >  25 files changed, 1061 insertions(+), 17 deletions(-)  create mode
> > 100644
> > OvmfPkg/AmdSev/ConfidentialMigration/ConfidentialMigrationDxe.inf
> >  create mode 100644
> > OvmfPkg/AmdSev/ConfidentialMigration/ConfidentialMigrationPei.inf
> >  create mode 100644 OvmfPkg/AmdSev/ConfidentialMigration/MpLib.h
> >  create mode 100644
> > OvmfPkg/AmdSev/ConfidentialMigration/VirtualMemory.h
> >  create mode 100644 OvmfPkg/Include/Guid/MemEncryptLib.h
> >  create mode 100644
> > OvmfPkg/AmdSev/ConfidentialMigration/ConfidentialMigrationDxe.c
> >  create mode 100644
> > OvmfPkg/AmdSev/ConfidentialMigration/ConfidentialMigrationPei.c
> >  create mode 100644 OvmfPkg/PlatformDxe/AmdSev.c
> >
> > --
> > 2.20.1
> >
> >
> >
> >
> >
>
>
>
> 
>