[PATCH v6 0/6] SEV Live Migration support for OVMF.

[PATCH v6 0/6] SEV Live Migration support for OVMF.

[Ashish Kalra]

From: Ashish Kalra <ashish.kalra@amd.com>

BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=3467

By default all the SEV guest memory regions are considered encrypted,
if a guest changes the encryption attribute of the page (e.g mark a
page as decrypted) then notify hypervisor. Hypervisor will need to
track the unencrypted pages. The information will be used during
guest live migration, guest page migration and guest debugging.

The patch-set detects if it is running under KVM hypervisor and then
checks for SEV live migration feature support via KVM_FEATURE_CPUID,
if detected setup a new UEFI enviroment variable to indicate OVMF
support for SEV live migration.

A branch containing these patches is available here:
https://github.com/ashkalra/edk2-1/tree/sev_live_migration_v5_10

Changes since v5:
 - Split first patch into three components, one patch for the
   MemEncryptSevLiveMigrationIsEnabled() API, one patch for the
   SetMemoryEncDecHypercall3() API, one patch to make use of the
   SetMemoryEncDecHypercall3() API. 
 - Fix patch subject, in code and patch comments and 
   additionally add relevant comments.
 - Replace SetMemoryEncDecHypercall3() API's Status argument
   with a boolean IsEncrypted argument and corresponding fixes
   to users of this API call.
 - Fix AsciiStrCmp() usage in KVM hypervisor detection code.

Changes since v4:
 - Remove MemEncryptHypercallLib Library and add support to issue
   hypercall in the BaseMemEncryptSevLib library itself.
 - For SEV-ES, make the VC handler hypercall aware by comparing
   the hypercall number and add the additional register values
   in the GHCB.
 - Fix comments in the hypercall API interface.
 - The encryption bit is set/clear on the smallest page size, hence
   use the 4k page size in MAP_GPA_RANGE hypercall.
 - Make the hypercall expect the guest physical address to be
   page-aligned.
 - Add KVM live migration feature flag check in BaseMemEncryptSevLib
   library similar to how BaseMemEncryptSevLib does for the
   MemEncryptSevIsEnabled() and check it before invoking HC. Also
   export the MemEncryptSevLiveMigrationIsEnabled() function as 
   part of the library.
 - Add error handling on hypercall return, on failure, return error
   code to caller which potentially will cause an assert() and
   terminate the boot.
 
Changes since v3:
 - Fix all DSC files under OvmfPkg except X64 to add support for 
   BaseMemEncryptLib and add NULL instance of BaseMemEncryptLib
   for 32 bit platforms.
 - Add the MemEncryptHypercallLib-related files to Maintainers.txt,
   in section "OvmfPkg: Confidential Computing".
 - Add support for the new KVM_HC_MAP_GPA_RANGE hypercall interface.
 - Add patch for SEV live migration support.

Changes since v2:
 - GHCB_BASE setup during reset-vector as decrypted is marked explicitly
   in the hypervisor page encryption bitmap after setting the 
   PcdSevEsIsEnabled PCD.

Changes since v1:
 - Mark GHCB_BASE setup during reset-vector as decrypted explicitly in
   the hypervisor page encryption bitmap.
 - Resending the series with correct shallow threading.

Ashish Kalra (6):
  OvmfPkg/BaseMemEncryptLib: Detect SEV live migration feature.
  OvmfPkg/BaseMemEncryptLib: Hypercall API for page encryption state
    change
  OvmfPkg/BaseMemEncryptLib: Invoke page encryption state change
    hypercall
  OvmfPkg/VmgExitLib: Encryption state change hypercall support in VC
    handler
  OvmfPkg/PlatformPei: Mark SEC GHCB page as unencrypted via hypercall
  OvmfPkg/AmdSevDxe: Add support for SEV live migration.

 OvmfPkg/AmdSevDxe/AmdSevDxe.c                 | 64 +++++++++++++++++
 OvmfPkg/AmdSevDxe/AmdSevDxe.inf               |  4 ++
 OvmfPkg/Include/Guid/AmdSevMemEncryptLib.h    | 20 ++++++
 OvmfPkg/Include/Library/MemEncryptSevLib.h    | 70 +++++++++++++++++++
 .../DxeMemEncryptSevLib.inf                   |  1 +
 .../DxeMemEncryptSevLibInternal.c             | 39 +++++++++++
 .../Ia32/MemEncryptSevLib.c                   | 27 +++++++
 .../PeiDxeMemEncryptSevLibInternal.c          | 52 ++++++++++++++
 .../PeiMemEncryptSevLib.inf                   |  1 +
 .../PeiMemEncryptSevLibInternal.c             | 39 +++++++++++
 .../SecMemEncryptSevLibInternal.c             | 38 ++++++++++
 .../X64/AsmHelperStub.nasm                    | 33 +++++++++
 .../X64/MemEncryptSevLib.c                    | 62 ++++++++++++++++
 .../X64/PeiDxeVirtualMemory.c                 | 20 ++++++
 OvmfPkg/Library/VmgExitLib/VmgExitVcHandler.c | 13 ++++
 OvmfPkg/OvmfPkg.dec                           |  1 +
 OvmfPkg/PlatformPei/AmdSev.c                  | 11 +++
 17 files changed, 495 insertions(+)
 create mode 100644 OvmfPkg/Include/Guid/AmdSevMemEncryptLib.h
 create mode 100644 OvmfPkg/Library/BaseMemEncryptSevLib/X64/AsmHelperStub.nasm

-- 
2.17.1

Re: [edk2-devel] [PATCH v6 0/6] SEV Live Migration support for OVMF.

[Yao, Jiewen]

Hi
I have some questions:

1) May I know what is the usage of this UEFI variable - SevLiveMigrationEnabled? 
I only see it is created, but I do not see how it is consumed.

2) Is this a full live migration patch, or is this just a startup and there will be more on the way?

Thank you
Yao Jiewen


> -----Original Message-----
> From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Ashish Kalra
> via groups.io
> Sent: Monday, August 2, 2021 8:31 PM
> To: devel@edk2.groups.io
> Cc: dovmurik@linux.vnet.ibm.com; brijesh.singh@amd.com; tobin@ibm.com;
> Thomas.Lendacky@amd.com; jejb@linux.ibm.com; Justen, Jordan L
> <jordan.l.justen@intel.com>; ard.biesheuvel@arm.com;
> erdemaktas@google.com; Yao, Jiewen <jiewen.yao@intel.com>; Xu, Min M
> <min.m.xu@intel.com>
> Subject: [edk2-devel] [PATCH v6 0/6] SEV Live Migration support for OVMF.
> 
> From: Ashish Kalra <ashish.kalra@amd.com>
> 
> BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=3467
> 
> By default all the SEV guest memory regions are considered encrypted,
> if a guest changes the encryption attribute of the page (e.g mark a
> page as decrypted) then notify hypervisor. Hypervisor will need to
> track the unencrypted pages. The information will be used during
> guest live migration, guest page migration and guest debugging.
> 
> The patch-set detects if it is running under KVM hypervisor and then
> checks for SEV live migration feature support via KVM_FEATURE_CPUID,
> if detected setup a new UEFI enviroment variable to indicate OVMF
> support for SEV live migration.
> 
> A branch containing these patches is available here:
> https://github.com/ashkalra/edk2-1/tree/sev_live_migration_v5_10
> 
> Changes since v5:
>  - Split first patch into three components, one patch for the
>    MemEncryptSevLiveMigrationIsEnabled() API, one patch for the
>    SetMemoryEncDecHypercall3() API, one patch to make use of the
>    SetMemoryEncDecHypercall3() API.
>  - Fix patch subject, in code and patch comments and
>    additionally add relevant comments.
>  - Replace SetMemoryEncDecHypercall3() API's Status argument
>    with a boolean IsEncrypted argument and corresponding fixes
>    to users of this API call.
>  - Fix AsciiStrCmp() usage in KVM hypervisor detection code.
> 
> Changes since v4:
>  - Remove MemEncryptHypercallLib Library and add support to issue
>    hypercall in the BaseMemEncryptSevLib library itself.
>  - For SEV-ES, make the VC handler hypercall aware by comparing
>    the hypercall number and add the additional register values
>    in the GHCB.
>  - Fix comments in the hypercall API interface.
>  - The encryption bit is set/clear on the smallest page size, hence
>    use the 4k page size in MAP_GPA_RANGE hypercall.
>  - Make the hypercall expect the guest physical address to be
>    page-aligned.
>  - Add KVM live migration feature flag check in BaseMemEncryptSevLib
>    library similar to how BaseMemEncryptSevLib does for the
>    MemEncryptSevIsEnabled() and check it before invoking HC. Also
>    export the MemEncryptSevLiveMigrationIsEnabled() function as
>    part of the library.
>  - Add error handling on hypercall return, on failure, return error
>    code to caller which potentially will cause an assert() and
>    terminate the boot.
> 
> Changes since v3:
>  - Fix all DSC files under OvmfPkg except X64 to add support for
>    BaseMemEncryptLib and add NULL instance of BaseMemEncryptLib
>    for 32 bit platforms.
>  - Add the MemEncryptHypercallLib-related files to Maintainers.txt,
>    in section "OvmfPkg: Confidential Computing".
>  - Add support for the new KVM_HC_MAP_GPA_RANGE hypercall interface.
>  - Add patch for SEV live migration support.
> 
> Changes since v2:
>  - GHCB_BASE setup during reset-vector as decrypted is marked explicitly
>    in the hypervisor page encryption bitmap after setting the
>    PcdSevEsIsEnabled PCD.
> 
> Changes since v1:
>  - Mark GHCB_BASE setup during reset-vector as decrypted explicitly in
>    the hypervisor page encryption bitmap.
>  - Resending the series with correct shallow threading.
> 
> Ashish Kalra (6):
>   OvmfPkg/BaseMemEncryptLib: Detect SEV live migration feature.
>   OvmfPkg/BaseMemEncryptLib: Hypercall API for page encryption state
>     change
>   OvmfPkg/BaseMemEncryptLib: Invoke page encryption state change
>     hypercall
>   OvmfPkg/VmgExitLib: Encryption state change hypercall support in VC
>     handler
>   OvmfPkg/PlatformPei: Mark SEC GHCB page as unencrypted via hypercall
>   OvmfPkg/AmdSevDxe: Add support for SEV live migration.
> 
>  OvmfPkg/AmdSevDxe/AmdSevDxe.c                 | 64 +++++++++++++++++
>  OvmfPkg/AmdSevDxe/AmdSevDxe.inf               |  4 ++
>  OvmfPkg/Include/Guid/AmdSevMemEncryptLib.h    | 20 ++++++
>  OvmfPkg/Include/Library/MemEncryptSevLib.h    | 70 +++++++++++++++++++
>  .../DxeMemEncryptSevLib.inf                   |  1 +
>  .../DxeMemEncryptSevLibInternal.c             | 39 +++++++++++
>  .../Ia32/MemEncryptSevLib.c                   | 27 +++++++
>  .../PeiDxeMemEncryptSevLibInternal.c          | 52 ++++++++++++++
>  .../PeiMemEncryptSevLib.inf                   |  1 +
>  .../PeiMemEncryptSevLibInternal.c             | 39 +++++++++++
>  .../SecMemEncryptSevLibInternal.c             | 38 ++++++++++
>  .../X64/AsmHelperStub.nasm                    | 33 +++++++++
>  .../X64/MemEncryptSevLib.c                    | 62 ++++++++++++++++
>  .../X64/PeiDxeVirtualMemory.c                 | 20 ++++++
>  OvmfPkg/Library/VmgExitLib/VmgExitVcHandler.c | 13 ++++
>  OvmfPkg/OvmfPkg.dec                           |  1 +
>  OvmfPkg/PlatformPei/AmdSev.c                  | 11 +++
>  17 files changed, 495 insertions(+)
>  create mode 100644 OvmfPkg/Include/Guid/AmdSevMemEncryptLib.h
>  create mode 100644
> OvmfPkg/Library/BaseMemEncryptSevLib/X64/AsmHelperStub.nasm
> 
> --
> 2.17.1
> 
> 
> 
> 
> 

Re: [edk2-devel] [PATCH v6 0/6] SEV Live Migration support for OVMF.

[Ashish Kalra]

On Thu, Aug 05, 2021 at 05:17:15AM +0000, Yao, Jiewen wrote:
> Hi
> I have some questions:
> 
> 1) May I know what is the usage of this UEFI variable - SevLiveMigrationEnabled? 
> I only see it is created, but I do not see how it is consumed.

Yes, it is created by UEFI but consumed by the (guest) linux kernel.

This is actually part of a 3-way negotiation of the live migration
feature between hypervisor, guest OVMF and guest kernel. Host indicates
support for live migration, which is detected by OVMF and
correspondingly OVMF sets this SetLiveMigrationEnabled UEFI variable,
which is read by the guest kernel and it indicates to the guest kernel
that both host and OVMF support and have enabled the live migration
feature.

The guest flow for detecting live migration feature is captured below
(from the corresponding kernel patch): 

The guest support for detecting and enabling SEV Live migration
feature uses the following logic :

 - kvm_init_plaform() checks if its booted under the EFI

   - If not EFI,

     i) if kvm_para_has_feature(KVM_FEATURE_MIGRATION_CONTROL), issue a wrmsrl()
         to enable the SEV live migration support

   - If EFI,

     i) If kvm_para_has_feature(KVM_FEATURE_MIGRATION_CONTROL), read
        the UEFI variable which indicates OVMF support for live migration
	(SetLiveMigrationEnabled)

     ii) the variable indicates live migration is supported, issue a wrmsrl() to
          enable the SEV live migration support

> 
> 2) Is this a full live migration patch, or is this just a startup and there will be more on the way?
> 

Yes. Actually this is hypervisor based live migration, so it relies on
host KVM and qemu support and it mainly requires OVMF support to make
hypercalls for any decrypted memory regions it sets up, for example,
MMIO and non-existant memory regions, so that hypervisor can keep track
of guest firmware's page encryption status.

So this is the full patch-set for UEFI for PSP based live migration.

Thanks,
Ashish

> Thank you
> Yao Jiewen
> 
> 
> > -----Original Message-----
> > From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Ashish Kalra
> > via groups.io
> > Sent: Monday, August 2, 2021 8:31 PM
> > To: devel@edk2.groups.io
> > Cc: dovmurik@linux.vnet.ibm.com; brijesh.singh@amd.com; tobin@ibm.com;
> > Thomas.Lendacky@amd.com; jejb@linux.ibm.com; Justen, Jordan L
> > <jordan.l.justen@intel.com>; ard.biesheuvel@arm.com;
> > erdemaktas@google.com; Yao, Jiewen <jiewen.yao@intel.com>; Xu, Min M
> > <min.m.xu@intel.com>
> > Subject: [edk2-devel] [PATCH v6 0/6] SEV Live Migration support for OVMF.
> > 
> > From: Ashish Kalra <ashish.kalra@amd.com>
> > 
> > BZ: https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugzilla.tianocore.org%2Fshow_bug.cgi%3Fid%3D3467&amp;data=04%7C01%7Cashish.kalra%40amd.com%7Cd4bf7bc0c17840d8a25208d957d04dcd%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637637374441109760%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=238zqkur1zpuKDorzZLrzcbTNhlO6gM8aFoi7Zr%2BUao%3D&amp;reserved=0
> > 
> > By default all the SEV guest memory regions are considered encrypted,
> > if a guest changes the encryption attribute of the page (e.g mark a
> > page as decrypted) then notify hypervisor. Hypervisor will need to
> > track the unencrypted pages. The information will be used during
> > guest live migration, guest page migration and guest debugging.
> > 
> > The patch-set detects if it is running under KVM hypervisor and then
> > checks for SEV live migration feature support via KVM_FEATURE_CPUID,
> > if detected setup a new UEFI enviroment variable to indicate OVMF
> > support for SEV live migration.
> > 
> > A branch containing these patches is available here:
> > https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fashkalra%2Fedk2-1%2Ftree%2Fsev_live_migration_v5_10&amp;data=04%7C01%7Cashish.kalra%40amd.com%7Cd4bf7bc0c17840d8a25208d957d04dcd%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637637374441109760%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=PyChejF9mzIVLCvQiCW5olQ8LOFWM3%2FGGO938qL80f8%3D&amp;reserved=0
> > 
> > Changes since v5:
> >  - Split first patch into three components, one patch for the
> >    MemEncryptSevLiveMigrationIsEnabled() API, one patch for the
> >    SetMemoryEncDecHypercall3() API, one patch to make use of the
> >    SetMemoryEncDecHypercall3() API.
> >  - Fix patch subject, in code and patch comments and
> >    additionally add relevant comments.
> >  - Replace SetMemoryEncDecHypercall3() API's Status argument
> >    with a boolean IsEncrypted argument and corresponding fixes
> >    to users of this API call.
> >  - Fix AsciiStrCmp() usage in KVM hypervisor detection code.
> > 
> > Changes since v4:
> >  - Remove MemEncryptHypercallLib Library and add support to issue
> >    hypercall in the BaseMemEncryptSevLib library itself.
> >  - For SEV-ES, make the VC handler hypercall aware by comparing
> >    the hypercall number and add the additional register values
> >    in the GHCB.
> >  - Fix comments in the hypercall API interface.
> >  - The encryption bit is set/clear on the smallest page size, hence
> >    use the 4k page size in MAP_GPA_RANGE hypercall.
> >  - Make the hypercall expect the guest physical address to be
> >    page-aligned.
> >  - Add KVM live migration feature flag check in BaseMemEncryptSevLib
> >    library similar to how BaseMemEncryptSevLib does for the
> >    MemEncryptSevIsEnabled() and check it before invoking HC. Also
> >    export the MemEncryptSevLiveMigrationIsEnabled() function as
> >    part of the library.
> >  - Add error handling on hypercall return, on failure, return error
> >    code to caller which potentially will cause an assert() and
> >    terminate the boot.
> > 
> > Changes since v3:
> >  - Fix all DSC files under OvmfPkg except X64 to add support for
> >    BaseMemEncryptLib and add NULL instance of BaseMemEncryptLib
> >    for 32 bit platforms.
> >  - Add the MemEncryptHypercallLib-related files to Maintainers.txt,
> >    in section "OvmfPkg: Confidential Computing".
> >  - Add support for the new KVM_HC_MAP_GPA_RANGE hypercall interface.
> >  - Add patch for SEV live migration support.
> > 
> > Changes since v2:
> >  - GHCB_BASE setup during reset-vector as decrypted is marked explicitly
> >    in the hypervisor page encryption bitmap after setting the
> >    PcdSevEsIsEnabled PCD.
> > 
> > Changes since v1:
> >  - Mark GHCB_BASE setup during reset-vector as decrypted explicitly in
> >    the hypervisor page encryption bitmap.
> >  - Resending the series with correct shallow threading.
> > 
> > Ashish Kalra (6):
> >   OvmfPkg/BaseMemEncryptLib: Detect SEV live migration feature.
> >   OvmfPkg/BaseMemEncryptLib: Hypercall API for page encryption state
> >     change
> >   OvmfPkg/BaseMemEncryptLib: Invoke page encryption state change
> >     hypercall
> >   OvmfPkg/VmgExitLib: Encryption state change hypercall support in VC
> >     handler
> >   OvmfPkg/PlatformPei: Mark SEC GHCB page as unencrypted via hypercall
> >   OvmfPkg/AmdSevDxe: Add support for SEV live migration.
> > 
> >  OvmfPkg/AmdSevDxe/AmdSevDxe.c                 | 64 +++++++++++++++++
> >  OvmfPkg/AmdSevDxe/AmdSevDxe.inf               |  4 ++
> >  OvmfPkg/Include/Guid/AmdSevMemEncryptLib.h    | 20 ++++++
> >  OvmfPkg/Include/Library/MemEncryptSevLib.h    | 70 +++++++++++++++++++
> >  .../DxeMemEncryptSevLib.inf                   |  1 +
> >  .../DxeMemEncryptSevLibInternal.c             | 39 +++++++++++
> >  .../Ia32/MemEncryptSevLib.c                   | 27 +++++++
> >  .../PeiDxeMemEncryptSevLibInternal.c          | 52 ++++++++++++++
> >  .../PeiMemEncryptSevLib.inf                   |  1 +
> >  .../PeiMemEncryptSevLibInternal.c             | 39 +++++++++++
> >  .../SecMemEncryptSevLibInternal.c             | 38 ++++++++++
> >  .../X64/AsmHelperStub.nasm                    | 33 +++++++++
> >  .../X64/MemEncryptSevLib.c                    | 62 ++++++++++++++++
> >  .../X64/PeiDxeVirtualMemory.c                 | 20 ++++++
> >  OvmfPkg/Library/VmgExitLib/VmgExitVcHandler.c | 13 ++++
> >  OvmfPkg/OvmfPkg.dec                           |  1 +
> >  OvmfPkg/PlatformPei/AmdSev.c                  | 11 +++
> >  17 files changed, 495 insertions(+)
> >  create mode 100644 OvmfPkg/Include/Guid/AmdSevMemEncryptLib.h
> >  create mode 100644
> > OvmfPkg/Library/BaseMemEncryptSevLib/X64/AsmHelperStub.nasm
> > 
> > --
> > 2.17.1
> > 
> > 
> > 
> > 
> > 
>