From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from NAM10-DM6-obe.outbound.protection.outlook.com (NAM10-DM6-obe.outbound.protection.outlook.com [40.107.93.79]) by mx.groups.io with SMTP id smtpd.web10.14161.1614960661865855358 for ; Fri, 05 Mar 2021 08:11:02 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@amd.com header.s=selector1 header.b=E1c7w+V5; spf=permerror, err=parse error for token &{10 18 %{i}._ip.%{h}._ehlo.%{d}._spf.vali.email}: invalid domain name (domain: amd.com, ip: 40.107.93.79, mailfrom: ashish.kalra@amd.com) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=LWAFogANvgte0dyv2GzSvdUhU/BIRtuYG7o2UcyVGwlLqYUBdb8Kg+Frooc99H+dG0MbTQxLD6cFQUI4QZq5vXRcRhD2/SGaW+ykkG2+hVA4VNNlamm5RSZ34sYbp4sPXulOlBWK5COdXQ9zE/7VR3RHT+nE+/P/PdLNrEcAZzM+evTBr6lM/YKU3vmkMuKON0EaVdbeozxxCpMmpIwkkRMuR/ofszMNk9rn1WFJTgugKslJc2sBPEUbhHEKaDisf3kXdbV1kAtWwNd0hLRHMS5M9GZ4xLFNMHUGMQUfTLK3ovrPhvXv4anKivtA1QS4MMBYaoZP95gYtkb7MYTxYQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Gd35C11cAG4U8jxUg1E+6B9nANPgHFDEufkcKP9ejSU=; b=T+h6Qsrl4PL5apJxy/6BuPMOFiT1W18tYteWrx49MBlbDcP7TqzcW5/o/aK0/GQyTgSl0qKANsYSJWncxN46ihjROIUoe9Ka/qD+ZBv6WM4e/r67QwNg/b7GzdKJz1d+H5d9t7Dk/WUmnzSAo7dLvQheLlUYorRJU+X+8mqMfhwjYlwn1PBRY/EP+axB56T5EJ/FV40CjDJvvGinSXVl/aRA2K88l7SnZV5QJbb9CegFDypvN+LoDzso/C41J0z+PX60+ueC0FAN6m0rmTCgvCUx6jWKUEbDMZemMuB6y86uMyR4H/WCntTOh4G9xVswd7xfvu+2pnms7vbskld/hg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=amd.com; dmarc=pass action=none header.from=amd.com; dkim=pass header.d=amd.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Gd35C11cAG4U8jxUg1E+6B9nANPgHFDEufkcKP9ejSU=; b=E1c7w+V5mK++0S7Tl3QRree+fxwEBINRLSq/WfTbX5ZrevVeUs9CD1CthsUtNFSK5njhInHOoVrrd+H31eDG8LaDdLwAuIntbpiD96IjbtfoE7nDRo267Ps5fGOGQSGY455mFj9UBzHiWFF+auYaraW+mYa3DXs6tzQ+ar6GA/g= Authentication-Results: linux.ibm.com; dkim=none (message not signed) header.d=none;linux.ibm.com; dmarc=none action=none header.from=amd.com; Received: from SN6PR12MB2767.namprd12.prod.outlook.com (2603:10b6:805:75::23) by SN1PR12MB2414.namprd12.prod.outlook.com (2603:10b6:802:2e::31) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3890.19; Fri, 5 Mar 2021 16:11:00 +0000 Received: from SN6PR12MB2767.namprd12.prod.outlook.com ([fe80::24bb:3e53:c95e:cb8e]) by SN6PR12MB2767.namprd12.prod.outlook.com ([fe80::24bb:3e53:c95e:cb8e%7]) with mapi id 15.20.3912.018; Fri, 5 Mar 2021 16:11:00 +0000 Date: Fri, 5 Mar 2021 16:10:53 +0000 From: "Ashish Kalra" To: Tobin Feldman-Fitzthum Cc: Laszlo Ersek , devel@edk2.groups.io, Dov Murik , Tobin Feldman-Fitzthum , James Bottomley , Hubertus Franke , Brijesh Singh , Jon Grimm , Tom Lendacky Subject: Re: [edk2-devel] [RFC PATCH 00/14] Firmware Support for Fast Live Migration for AMD SEV Message-ID: <20210305161053.GA2148@ashkalra_ubuntu_server> References: <20210302204839.82042-1-tobin@linux.ibm.com> <9d7de545-7902-4d38-ba49-f084a750ee2a@redhat.com> <4869b086-f329-b79b-39ee-e21bfc5f95b5@linux.ibm.com> <20210305104422.GA1984@ashkalra_ubuntu_server> In-Reply-To: <20210305104422.GA1984@ashkalra_ubuntu_server> User-Agent: Mutt/1.9.4 (2018-02-28) X-Originating-IP: [165.204.77.1] X-ClientProxiedBy: SN4PR0201CA0045.namprd02.prod.outlook.com (2603:10b6:803:2e::31) To SN6PR12MB2767.namprd12.prod.outlook.com (2603:10b6:805:75::23) Return-Path: ashish.kalra@amd.com MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from ashkalra_ubuntu_server (165.204.77.1) by SN4PR0201CA0045.namprd02.prod.outlook.com (2603:10b6:803:2e::31) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3912.17 via Frontend Transport; Fri, 5 Mar 2021 16:10:59 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-HT: Tenant X-MS-Office365-Filtering-Correlation-Id: 0b8a3495-59d8-4729-d532-08d8dff14429 X-MS-TrafficTypeDiagnostic: SN1PR12MB2414: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:4125; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: uspQa+sfnhvTQux6FUVgrdWCdGs9tAWY5qHt5IHtfStMPMzdhB5uB0Jdzat/FPtPK+F9KING4kZ99ZjZuAtU114G3EiJMAPdaam3Cr8ZxEhXHZHHRYgiD2ZqEspemq9Z27cYpgTP/sLn1SDpgHeiFdK+0sbiWju8b4TQ6kg0clztpsGT/Kvj6J0c/6NvOskjgaYuAAG8BkZxsRP56bM8E/O3LXfJE+MJej9DMsgNUr0r42YbEC0eINun76S+MoUcHCNZHVNB1lGyuSXYaitO/YhGdKLoF3suatqyk187vLiVkIYxrUxYYDXs1oCwBzAq0paD6HL9wuHCRhIGUgfhq8cLLLOhOGYfLHikwa75+d/MmE1yQWE5wZi27rGWAQwc/bOk+W6cxDXJDKaMm/PCnJjKuCLfeW3yJCIMYqsttQEffL/w9zYTo4OsswsXJ/CK/P+ZfpYr7xKGZI5wUcUlp95uW/NP2XLC7u9IHBLflW0/NoHXbHVivuKjWDH4Gx2wQ7XqVzObyQPuJAO4vxh9OA== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:SN6PR12MB2767.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(4636009)(366004)(376002)(396003)(39860400002)(136003)(346002)(33716001)(9686003)(956004)(83380400001)(26005)(54906003)(55016002)(6916009)(2906002)(6666004)(16526019)(478600001)(86362001)(6496006)(66476007)(66556008)(33656002)(8676002)(66946007)(44832011)(5660300002)(52116002)(1076003)(53546011)(4326008)(8936002)(316002)(186003);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: =?us-ascii?Q?mX/Os7IA46DUUWbIAGqfkwfj7oXdgm/W0P/58Q71o6PT4DZ0rPeDBQe7mtIu?= =?us-ascii?Q?MsHwz1XAoNogLalfNNLNZlPYnZojRxlCCij0pM/I00Ww1Y/FSeCrstwh5YiO?= =?us-ascii?Q?MfhJV1Cq1/WLJ0Gwl8IxJFnRoNv6b/jdo+yYf7pWpSjXA1xfL8X2njccHZAf?= =?us-ascii?Q?FqtABEnx2CTCQuu/cjhw0bs868gejLEA1b/nucaWFrm+1liC1Jzk82WOyxU5?= =?us-ascii?Q?chgOwGMW3AI7E4U3AYGCeIn+IKmFPAULwDrdK5MSA8CdhiBVvuN8qZSYXXlU?= =?us-ascii?Q?OVUzykenE2Kxez+5mSo2hfkuqpvx3BHluEpxBlL+DryXcRWzQqrSiJ5Q48MR?= =?us-ascii?Q?Q5PQQ09zTMGKYPPbBXzlvYPuVpO/8K8KTe4svIQTFtWF1nbeur/mhQ7fEsio?= =?us-ascii?Q?GarhXk6AtkNmK6JT6ZaXZH7YjCO0OCIql0MOtaGdk51Xy41MSUjubiwqINQs?= =?us-ascii?Q?PGlchhYap8zuFhvOy4wJsUTTzPBEBeNBGCCkROiZaNn3XZhTYBjcY53zECW8?= =?us-ascii?Q?0WTFMMBbg9foEDpP4X23qX7S8FuAqNmMtZ9Ip/dbKCayblnkNiODmTEHJxgJ?= =?us-ascii?Q?8edi2iBKDq4cP+gYsl+RDt9ymIy3B1OrzMBWcXFyliiqvNZbQ3wlVGe6Y1UR?= =?us-ascii?Q?PmOI2p1Jp7CYCRWkfPEgjaeJmkg/80eQRTSX3VU5cPxPfuMx3EzDjsGr4zNp?= =?us-ascii?Q?L/IOg2Wbf2pbbWvT4MgeAr608bj8pHRcNQO2FACggvRWhVZSn54cs4L8/VKU?= =?us-ascii?Q?EhMDfARa8hk+lN52O5FPKK9biNJKz7E6sZaLnidO9f/wH1/Yz4Q/zh8TXoK5?= =?us-ascii?Q?BGFadVw9v97GD/E69HrIh7k293xgS/9pfcHQ6R0VFCsyUH+TNuDfg0x4dvvV?= =?us-ascii?Q?WLtiDg3fVpQnESa2O+Ta1GWuNJ3gHmvZlY3D1zYU5deuPukDAfd4AiqZG55w?= =?us-ascii?Q?5tTlfffAY6AWNIbUUT46Aq1u2XWdAZHrtMl5DUIrIWXBgTvZiacfN18AqavJ?= =?us-ascii?Q?kmFicU/Vla5AEjvbb0yCteUx+zgcjCDfPuseWzSKVEuiuZBXAgGVx8dr1E6U?= =?us-ascii?Q?zknx17JJtR9ch0EH0Ejfde+jwjC2aMSagBk51fDocRgCa1TTrWKMwBCwCLka?= =?us-ascii?Q?FQs0vc4aqiWHweyVlIEC2poSKedbpqpy+ReGqkj0gNyYKVwfQWgbAKOiFn5U?= =?us-ascii?Q?jaMQ1Q3FpJXOmWEpP2UuVhM0vnG7HwcnwiB9wm6+6jLViFrzwB438CQ7FEfV?= =?us-ascii?Q?oUQ/OGpllSN3lolFGIyiPvz353qaELxUhUTEJsdTaNLHjLYme923HWAuIqQu?= =?us-ascii?Q?K8nBlPnzjlLLYDksedNTo5Ii?= X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-Network-Message-Id: 0b8a3495-59d8-4729-d532-08d8dff14429 X-MS-Exchange-CrossTenant-AuthSource: SN6PR12MB2767.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Mar 2021 16:11:00.0807 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 6GLuCUhVSZnLoirK10+GhBmw4QcKCSVaiGyvf85rap+gTNrBLKpxd+UyJLULWCYyciM09XeeUbh/GO5KYV+tGw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN1PR12MB2414 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline On Fri, Mar 05, 2021 at 10:44:23AM +0000, Ashish Kalra wrote: > On Wed, Mar 03, 2021 at 01:25:40PM -0500, Tobin Feldman-Fitzthum wrote: > > > > > Hi Tobin, > > > > > > On 03/02/21 21:48, Tobin Feldman-Fitzthum wrote: > > > > This is a demonstration of fast migration for encrypted virtual machines > > > > using a Migration Handler that lives in OVMF. This demo uses AMD SEV, > > > > but the ideas may generalize to other confidential computing platforms. > > > > With AMD SEV, guest memory is encrypted and the hypervisor cannot access > > > > or move it. This makes migration tricky. In this demo, we show how the > > > > HV can ask a Migration Handler (MH) in the firmware for an encrypted > > > > page. The MH encrypts the page with a transport key prior to releasing > > > > it to the HV. The target machine also runs an MH that decrypts the page > > > > once it is passed in by the target HV. These patches are not ready for > > > > production, but the are a full end-to-end solution that facilitates a > > > > fast live migration between two SEV VMs. > > > > > > > > Corresponding patches for QEMU have been posted my colleague Dov Murik > > > > on qemu-devel. Our approach needs little kernel support, requiring only > > > > one hypercall that the guest can use to mark a page as encrypted or > > > > shared. This series includes updated patches from Ashish Kalra and > > > > Brijesh Singh that allow OVMF to use this hypercall. > > > > > > > > The MH runs continuously in the guest, waiting for communication from > > > > the HV. The HV starts an additional vCPU for the MH but does not expose > > > > it to the guest OS via ACPI. We use the MpService to start the MH. The > > > > MpService is only available at runtime and processes that are started by > > > > it are usually cleaned up on ExitBootServices. Since we need the MH to > > > > run continuously, we had to make some modifications. Ideally a feature > > > > could be added to the MpService to allow for the starting of > > > > long-running processes. Besides migration, this could support other > > > > background processes that need to operate within the encryption > > > > boundary. For now, we have included a handful of patches that modify the > > > > MpService to allow the MH to keep running after ExitBootServices. These > > > > are temporary. > > > I plan to do a lightweight review for this series. (My understanding is > > > that it's an RFC and not actually being proposed for merging.) > > > > > > Regarding the MH's availability at runtime -- does that necessarily > > > require the isolation of an AP? Because in the current approach, > > > allowing the MP Services to survive into OS runtime (in some form or > > > another) seems critical, and I don't think it's going to fly. > > > > > > I agree that the UefiCpuPkg patches have been well separated from the > > > rest of the series, but I'm somewhat doubtful the "firmware-initiated > > > background process" idea will be accepted. Have you investigated > > > exposing a new "runtime service" (a function pointer) via the UEFI > > > Configuration table, and calling that (perhaps periodically?) from the > > > guest kernel? It would be a form of polling I guess. Or maybe, poll the > > > mailbox directly in the kernel, and call the new firmware runtime > > > service when there's an actual command to process. > > Continuous runtime availability for the MH is almost certainly the most > > controversial part of this proposal, which is why I put it in the cover > > letter and why it's good to discuss. > > > (You do spell out "little kernel support", and I'm not sure if that's a > > > technical benefit, or a political / community benefit.) > > > > As you allude to, minimal kernel support is really one of the main things > > that shapes our approach. This is partly a political and practical benefit, > > but there are also technical benefits. Having the MH in firmware likely > > leads to higher availability. It can be accessed when the OS is unreachable, > > perhaps during boot or when the OS is hung. There are also potential > > portability advantages although we do currently require support for one > > hypercall. The cost of implementing this hypercall is low. > > > > Generally speaking, our task is to find a home for functionality that was > > traditionally provided by the hypervisor, but that needs to be inside the > > trust domain, but that isn't really part of a guest. A meta-goal of this > > project is to figure out the best way to do this. > > > > > > > > I'm quite uncomfortable with an attempt to hide a CPU from the OS via > > > ACPI. The OS has other ways to learn (for example, a boot loader could > > > use the MP services itself, stash the information, and hand it to the OS > > > kernel -- this would minimally allow for detecting an inconsistency in > > > the OS). What about "all-but-self" IPIs too -- the kernel might think > > > all the processors it's poking like that were under its control. > > > > This might be the second most controversial piece. Here's a question: if we > > could successfully hide the MH vCPU from the OS, would it still make you > > uncomfortable? In other words, is the worry that there might be some > > inconsistency or more generally that there is something hidden from the OS? > > One thing to think about is that the guest owner should generally be aware > > that there is a migration handler running. The way I see it, a guest owner > > of an SEV VM would need to opt-in to migration and should then expect that > > there is an MH running even if they aren't able to see it. Of course we need > > to be certain that the MH isn't going to break the OS. > > > > > Also, as far as I can tell from patch #7, the AP seems to be > > > busy-looping (with a CpuPause() added in), for the entire lifetime of > > > the OS. Do I understand right? If so -- is it a temporary trait as well? > > > > In our approach the MH continuously checks for commands from the hypervisor. > > There are potentially ways to optimize this, such as having the hypervisor > > de-schedule the MH vCPU while not migrating. You could potentially shut down > > down the MH on the target after receiving the MH_RESET command (when the > > migration finishes), but what if you want to migrate that VM somewhere else? > > > > I think another approach can be considered here, why not implement MH > vCPU(s) as hot-plugged vCPU(s), basically hot-plug a new vCPU when migration > is started and hot unplug the vCPU when migration is completed, then we > won't need a vCPU running (and potentially consuming cycles) forever and > busy-looping with CpuPause(). > After internal discussions, realized that this approach will not work as vCPU hotplug will not work for SEV-ES, SNP. As the VMSA has to be encrypted as part of the LAUNCH command, therefore we can't create/add a new vCPU after LAUNCH has completed. Thanks, Ashish