Re: [edk2-devel] [RFC PATCH 00/14] Firmware Support for Fast Live Migration for AMD SEV

["Tobin Feldman-Fitzthum" <tobin@linux.ibm.com>]

On 3/12/21 9:32 PM, Yao, Jiewen wrote:

> Hi
> We discuss the patch internally. We do see PROs and CONs with this approach.
> The advantage is that it is very simple. In-VM migration can save lots of effort on security context restore.
> On the other hand, we feel not so comfortable to reserve a dedicate CPU to achieve that. Similar to the feedback in the community.
>
> Using Hot-Plug is not a solution for Intel TDX as well. It is unsupported now.
>
> I like the idea to diverge the migration boot mode v.s. normal boot mode in SEC phase.
> We must be very carefully handle this migration boot mode, to avoid any touching on system memory.
> Intel TDX Virtual Firmware skips the PEI phase directly. If we choose this approach, SEC-based migration is our preference.
>
> Besides this patch, we would like to understand a full picture.
> 1) How the key is passed from source VM to destination?
> I saw you mentions: "Key sharing is out of scope for this part of the RFC."
> "This will probably be implemented via inject-launch-secret in the future"
>
> Does that mean two PSP will sync with each other and negotiate the key, after the Migration Agent (MA) checks the policy?

The source and destination migration handlers will need to share a key. 
If we only relied on the PSP for migration, we could use the existing 
secure channel between the PSP and the guest owner to transfer the 
pages. Unfortunately the throughput of this approach is far too low. 
Thus, we have some migration handler running on a guest vCPU with a 
transport key shared between the source and the target.

The main mechanism for getting a key to the migration handler is 
inject-launch-secret. Here the guest owner can provide a secret to the 
PSP via a secure channel and the PSP will inject it at some guest 
physical address. You use inject-launch-secret after the launch 
measurement of the guest has been generated to inject the secret 
conditionally. One approach would be to inject the transport key 
directly in the source and the target. This is pretty simple, but might 
have a few drawbacks. The injection has to happen at boot, meaning that 
the source machine would have to be provisioned with a transport key 
before a migration happens and that all migrations from that machine 
would have to use the same transport key. One way around this would be 
to inject asymmetric keys and use them to derive the transport key.

Another approach entirely is to use the PSP to migrate just a few pages, 
which might include a secret set by the source MH that the target MH 
could use to decrypt incoming pages. Using the PSP to migrate pages 
requires some extra kernel support.

For the RFC, we just assume that there is some shared key. We have 
talked some about the various options internally.

> 2) How the attestation is supported?
> I read the whitepaper https://www.amd.com/system/files/TechDocs/SEV-SNP-strengthening-vm-isolation-with-integrity-protection-and-more.pdf.
> It seems SEV and SEV-ES only support attestation during launch, I don't believe this migration feature will impact the attestation report. Am I right?
> SEV-SNP supports more flexible attestation, does it include any information about the new migrated content?

Brijesh already addressed most of this. In our approach the MH is baked 
into the firmware, which can be attested prior to injecting the key. In 
other words there aren't any additional steps to attest the MH and it 
does not change the functionality of any existing attestation mechanisms.

-Tobin

>
>> -----Original Message-----
>> From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Yao, Jiewen
>> Sent: Thursday, March 4, 2021 9:49 AM
>> To: devel@edk2.groups.io; tobin@linux.ibm.com
>> Cc: Dov Murik <dovmurik@linux.vnet.ibm.com>; Tobin Feldman-Fitzthum
>> <tobin@ibm.com>; James Bottomley <jejb@linux.ibm.com>; Hubertus Franke
>> <frankeh@us.ibm.com>; Brijesh Singh <brijesh.singh@amd.com>; Ashish Kalra
>> <ashish.kalra@amd.com>; Jon Grimm <jon.grimm@amd.com>; Tom Lendacky
>> <thomas.lendacky@amd.com>; Yao, Jiewen <jiewen.yao@intel.com>
>> Subject: Re: [edk2-devel] [RFC PATCH 00/14] Firmware Support for Fast Live
>> Migration for AMD SEV
>>
>> Hi Tobin
>> Thanks for your patch.
>> You may that Intel is working on TDX for the same live migration feature.
>>
>> Please give me some time (about 1 work week) to digest and evaluate the patch
>> and impact.
>> Then I will provide feedback.
>>
>> Thank you
>> Yao Jiewen
>>
>>> -----Original Message-----
>>> From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Tobin
>>> Feldman-Fitzthum
>>> Sent: Wednesday, March 3, 2021 4:48 AM
>>> To: devel@edk2.groups.io
>>> Cc: Dov Murik <dovmurik@linux.vnet.ibm.com>; Tobin Feldman-Fitzthum
>>> <tobin@ibm.com>; Tobin Feldman-Fitzthum <tobin@linux.ibm.com>; James
>>> Bottomley <jejb@linux.ibm.com>; Hubertus Franke <frankeh@us.ibm.com>;
>>> Brijesh Singh <brijesh.singh@amd.com>; Ashish Kalra
>> <ashish.kalra@amd.com>;
>>> Jon Grimm <jon.grimm@amd.com>; Tom Lendacky
>>> <thomas.lendacky@amd.com>
>>> Subject: [edk2-devel] [RFC PATCH 00/14] Firmware Support for Fast Live
>>> Migration for AMD SEV
>>>
>>> This is a demonstration of fast migration for encrypted virtual machines
>>> using a Migration Handler that lives in OVMF. This demo uses AMD SEV,
>>> but the ideas may generalize to other confidential computing platforms.
>>> With AMD SEV, guest memory is encrypted and the hypervisor cannot access
>>> or move it. This makes migration tricky. In this demo, we show how the
>>> HV can ask a Migration Handler (MH) in the firmware for an encrypted
>>> page. The MH encrypts the page with a transport key prior to releasing
>>> it to the HV. The target machine also runs an MH that decrypts the page
>>> once it is passed in by the target HV. These patches are not ready for
>>> production, but the are a full end-to-end solution that facilitates a
>>> fast live migration between two SEV VMs.
>>>
>>> Corresponding patches for QEMU have been posted my colleague Dov Murik
>>> on qemu-devel. Our approach needs little kernel support, requiring only
>>> one hypercall that the guest can use to mark a page as encrypted or
>>> shared. This series includes updated patches from Ashish Kalra and
>>> Brijesh Singh that allow OVMF to use this hypercall.
>>>
>>> The MH runs continuously in the guest, waiting for communication from
>>> the HV. The HV starts an additional vCPU for the MH but does not expose
>>> it to the guest OS via ACPI. We use the MpService to start the MH. The
>>> MpService is only available at runtime and processes that are started by
>>> it are usually cleaned up on ExitBootServices. Since we need the MH to
>>> run continuously, we had to make some modifications. Ideally a feature
>>> could be added to the MpService to allow for the starting of
>>> long-running processes. Besides migration, this could support other
>>> background processes that need to operate within the encryption
>>> boundary. For now, we have included a handful of patches that modify the
>>> MpService to allow the MH to keep running after ExitBootServices. These
>>> are temporary.
>>>
>>> Ashish Kalra (2):
>>>    OvmfPkg/PlatformPei: Mark SEC GHCB page in the page encrpytion bitmap.
>>>    OvmfPkg/PlatformDxe: Add support for SEV live migration.
>>>
>>> Brijesh Singh (1):
>>>    OvmfPkg/BaseMemEncryptLib: Support to issue unencrypted hypercall
>>>
>>> Dov Murik (1):
>>>    OvmfPkg/AmdSev: Build page table for migration handler
>>>
>>> Tobin Feldman-Fitzthum (10):
>>>    OvmfPkg/AmdSev: Base for Confidential Migration Handler
>>>    OvmfPkg/PlatfomPei: Set Confidential Migration PCD
>>>    OvmfPkg/AmdSev: Setup Migration Handler Mailbox
>>>    OvmfPkg/AmdSev: MH support for mailbox protocol
>>>    UefiCpuPkg/MpInitLib: temp removal of MpLib cleanup
>>>    UefiCpuPkg/MpInitLib: Allocate MP buffer as runtime memory
>>>    UefiCpuPkg/CpuExceptionHandlerLib: Exception handling as runtime
>>>      memory
>>>    OvmfPkg/AmdSev: Don't overwrite mailbox or pagetables
>>>    OvmfPkg/AmdSev: Don't overwrite MH stack
>>>    OvmfPkg/AmdSev: MH page encryption POC
>>>
>>>   OvmfPkg/OvmfPkg.dec                           |  11 +
>>>   OvmfPkg/AmdSev/AmdSevX64.dsc                  |   2 +
>>>   OvmfPkg/AmdSev/AmdSevX64.fdf                  |  13 +-
>>>   .../ConfidentialMigrationDxe.inf              |  45 +++
>>>   .../ConfidentialMigrationPei.inf              |  35 ++
>>>   .../DxeMemEncryptSevLib.inf                   |   1 +
>>>   .../PeiMemEncryptSevLib.inf                   |   1 +
>>>   OvmfPkg/PlatformDxe/Platform.inf              |   2 +
>>>   OvmfPkg/PlatformPei/PlatformPei.inf           |   2 +
>>>   UefiCpuPkg/Library/MpInitLib/DxeMpInitLib.inf |   2 +
>>>   UefiCpuPkg/Library/MpInitLib/PeiMpInitLib.inf |   2 +
>>>   OvmfPkg/AmdSev/ConfidentialMigration/MpLib.h  | 235 +++++++++++++
>>>   .../ConfidentialMigration/VirtualMemory.h     | 177 ++++++++++
>>>   OvmfPkg/Include/Guid/MemEncryptLib.h          |  16 +
>>>   OvmfPkg/PlatformDxe/PlatformConfig.h          |   5 +
>>>   .../ConfidentialMigrationDxe.c                | 325 ++++++++++++++++++
>>>   .../ConfidentialMigrationPei.c                |  25 ++
>>>   .../X64/PeiDxeVirtualMemory.c                 |  18 +
>>>   OvmfPkg/PlatformDxe/AmdSev.c                  |  99 ++++++
>>>   OvmfPkg/PlatformDxe/Platform.c                |   6 +
>>>   OvmfPkg/PlatformPei/AmdSev.c                  |  10 +
>>>   OvmfPkg/PlatformPei/Platform.c                |  10 +
>>>   .../CpuExceptionHandlerLib/DxeException.c     |   8 +-
>>>   UefiCpuPkg/Library/MpInitLib/DxeMpLib.c       |  21 +-
>>>   UefiCpuPkg/Library/MpInitLib/MpLib.c          |   7 +-
>>>   25 files changed, 1061 insertions(+), 17 deletions(-)
>>>   create mode 100644
>>> OvmfPkg/AmdSev/ConfidentialMigration/ConfidentialMigrationDxe.inf
>>>   create mode 100644
>>> OvmfPkg/AmdSev/ConfidentialMigration/ConfidentialMigrationPei.inf
>>>   create mode 100644 OvmfPkg/AmdSev/ConfidentialMigration/MpLib.h
>>>   create mode 100644
>>> OvmfPkg/AmdSev/ConfidentialMigration/VirtualMemory.h
>>>   create mode 100644 OvmfPkg/Include/Guid/MemEncryptLib.h
>>>   create mode 100644
>>> OvmfPkg/AmdSev/ConfidentialMigration/ConfidentialMigrationDxe.c
>>>   create mode 100644
>>> OvmfPkg/AmdSev/ConfidentialMigration/ConfidentialMigrationPei.c
>>>   create mode 100644 OvmfPkg/PlatformDxe/AmdSev.c
>>>
>>> --
>>> 2.20.1
>>>
>>>
>>>
>>>
>>>
>>
>>
>> 
>>