Re: [edk2-devel] [RFC PATCH 00/14] Firmware Support for Fast Live Migration for AMD SEV

["Ashish Kalra" <ashish.kalra@amd.com>]

On Fri, Mar 05, 2021 at 10:44:23AM +0000, Ashish Kalra wrote:
> On Wed, Mar 03, 2021 at 01:25:40PM -0500, Tobin Feldman-Fitzthum wrote:
> > 
> > > Hi Tobin,
> > > 
> > > On 03/02/21 21:48, Tobin Feldman-Fitzthum wrote:
> > > > This is a demonstration of fast migration for encrypted virtual machines
> > > > using a Migration Handler that lives in OVMF. This demo uses AMD SEV,
> > > > but the ideas may generalize to other confidential computing platforms.
> > > > With AMD SEV, guest memory is encrypted and the hypervisor cannot access
> > > > or move it. This makes migration tricky. In this demo, we show how the
> > > > HV can ask a Migration Handler (MH) in the firmware for an encrypted
> > > > page. The MH encrypts the page with a transport key prior to releasing
> > > > it to the HV. The target machine also runs an MH that decrypts the page
> > > > once it is passed in by the target HV. These patches are not ready for
> > > > production, but the are a full end-to-end solution that facilitates a
> > > > fast live migration between two SEV VMs.
> > > > 
> > > > Corresponding patches for QEMU have been posted my colleague Dov Murik
> > > > on qemu-devel. Our approach needs little kernel support, requiring only
> > > > one hypercall that the guest can use to mark a page as encrypted or
> > > > shared. This series includes updated patches from Ashish Kalra and
> > > > Brijesh Singh that allow OVMF to use this hypercall.
> > > > 
> > > > The MH runs continuously in the guest, waiting for communication from
> > > > the HV. The HV starts an additional vCPU for the MH but does not expose
> > > > it to the guest OS via ACPI. We use the MpService to start the MH. The
> > > > MpService is only available at runtime and processes that are started by
> > > > it are usually cleaned up on ExitBootServices. Since we need the MH to
> > > > run continuously, we had to make some modifications. Ideally a feature
> > > > could be added to the MpService to allow for the starting of
> > > > long-running processes. Besides migration, this could support other
> > > > background processes that need to operate within the encryption
> > > > boundary. For now, we have included a handful of patches that modify the
> > > > MpService to allow the MH to keep running after ExitBootServices. These
> > > > are temporary.
> > > I plan to do a lightweight review for this series. (My understanding is
> > > that it's an RFC and not actually being proposed for merging.)
> > > 
> > > Regarding the MH's availability at runtime -- does that necessarily
> > > require the isolation of an AP? Because in the current approach,
> > > allowing the MP Services to survive into OS runtime (in some form or
> > > another) seems critical, and I don't think it's going to fly.
> > > 
> > > I agree that the UefiCpuPkg patches have been well separated from the
> > > rest of the series, but I'm somewhat doubtful the "firmware-initiated
> > > background process" idea will be accepted. Have you investigated
> > > exposing a new "runtime service" (a function pointer) via the UEFI
> > > Configuration table, and calling that (perhaps periodically?) from the
> > > guest kernel? It would be a form of polling I guess. Or maybe, poll the
> > > mailbox directly in the kernel, and call the new firmware runtime
> > > service when there's an actual command to process.
> > Continuous runtime availability for the MH is almost certainly the most
> > controversial part of this proposal, which is why I put it in the cover
> > letter and why it's good to discuss.
> > > (You do spell out "little kernel support", and I'm not sure if that's a
> > > technical benefit, or a political / community benefit.)
> > 
> > As you allude to, minimal kernel support is really one of the main things
> > that shapes our approach. This is partly a political and practical benefit,
> > but there are also technical benefits. Having the MH in firmware likely
> > leads to higher availability. It can be accessed when the OS is unreachable,
> > perhaps during boot or when the OS is hung. There are also potential
> > portability advantages although we do currently require support for one
> > hypercall. The cost of implementing this hypercall is low.
> > 
> > Generally speaking, our task is to find a home for functionality that was
> > traditionally provided by the hypervisor, but that needs to be inside the
> > trust domain, but that isn't really part of a guest. A meta-goal of this
> > project is to figure out the best way to do this.
> > 
> > > 
> > > I'm quite uncomfortable with an attempt to hide a CPU from the OS via
> > > ACPI. The OS has other ways to learn (for example, a boot loader could
> > > use the MP services itself, stash the information, and hand it to the OS
> > > kernel -- this would minimally allow for detecting an inconsistency in
> > > the OS). What about "all-but-self" IPIs too -- the kernel might think
> > > all the processors it's poking like that were under its control.
> > 
> > This might be the second most controversial piece. Here's a question: if we
> > could successfully hide the MH vCPU from the OS, would it still make you
> > uncomfortable? In other words, is the worry that there might be some
> > inconsistency or more generally that there is something hidden from the OS?
> > One thing to think about is that the guest owner should generally be aware
> > that there is a migration handler running. The way I see it, a guest owner
> > of an SEV VM would need to opt-in to migration and should then expect that
> > there is an MH running even if they aren't able to see it. Of course we need
> > to be certain that the MH isn't going to break the OS.
> > 
> > > Also, as far as I can tell from patch #7, the AP seems to be
> > > busy-looping (with a CpuPause() added in), for the entire lifetime of
> > > the OS. Do I understand right? If so -- is it a temporary trait as well?
> > 
> > In our approach the MH continuously checks for commands from the hypervisor.
> > There are potentially ways to optimize this, such as having the hypervisor
> > de-schedule the MH vCPU while not migrating. You could potentially shut down
> > down the MH on the target after receiving the MH_RESET command (when the
> > migration finishes), but what if you want to migrate that VM somewhere else?
> > 
> 
> I think another approach can be considered here, why not implement MH
> vCPU(s) as hot-plugged vCPU(s), basically hot-plug a new vCPU when migration
> is started and hot unplug the vCPU when migration is completed, then we
> won't need a vCPU running (and potentially consuming cycles) forever and
> busy-looping with CpuPause(). 
> 

After internal discussions, realized that this approach will not work as
vCPU hotplug will not work for SEV-ES, SNP. As the VMSA has to be
encrypted as part of the LAUNCH command, therefore we can't create/add a
new vCPU after LAUNCH has completed.

Thanks,
Ashish