From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from NAM04-BN3-obe.outbound.protection.outlook.com (NAM04-BN3-obe.outbound.protection.outlook.com [40.107.68.47]) by mx.groups.io with SMTP id smtpd.web12.10178.1614941072956730726 for ; Fri, 05 Mar 2021 02:44:33 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@amd.com header.s=selector1 header.b=Ht22nJLA; spf=permerror, err=parse error for token &{10 18 %{i}._ip.%{h}._ehlo.%{d}._spf.vali.email}: invalid domain name (domain: amd.com, ip: 40.107.68.47, mailfrom: ashish.kalra@amd.com) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=lH7n+z1thFi7Y6bnfWvDrBpCLM1Ji8CYeWWX8gOfuk7zWAg0OTvG5mgS9kOgad0WhSGiACjPRgbyMibDfEK807TXTcSZRhcAvQtNGZbPevfk8pz7kWy+OmKNGpxhgaBUvHMTvPWfXY3I8YUVPuhzzqGcZF1jS3O/vAkqx1RvCbrNgfWocpxi5NLOSzcnMFFa36mADomAvjjsasvsPppn7OkOVtONDQP9qkCSoCzdhNM/DnjALI/yxNr88HGBNAaeE4bDBWj9fBTLeaSJ77q0hQaNtW8pIvvlbYPa6yMal0J7vjNVU/H+RqnMgrNn73kk1A6B5HeKDAdAlHw0Ebazrw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=qcxpVS5sZzlltpdtO2+K0FbZTPXHssO7J2bnBCYiEmQ=; b=aonx8jVOWdDpFnbt+b1fVhHrnnfTJU4FELIU3il54waven7rYadHCGInVcKpn/RGIEvlIbpZ94A9tc6PxKPNTRdLfwIAv22Dl20ygBWDRgk5N2ISqyUKJluGjfZxBfsosdrTiXNbxb/h5VgXx47HIaRADoNghdWq4HRzBGRbMj19ZTgcSnzGXdU0YUmlItIW9VH2QOcLVfSrEv7nlCWqIQAjWNke/a7jr7rkfGn7huMEtbgVPG1z8945RmOvqxvp2tWqmoP2aI+Mcgvtj8sxbfafmJ/HXxyMPB009H8OF2qALWxBRwTnxT59FM/AwJEp/ncR7SsJlWHoRzQCov3oJg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=amd.com; dmarc=pass action=none header.from=amd.com; dkim=pass header.d=amd.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=qcxpVS5sZzlltpdtO2+K0FbZTPXHssO7J2bnBCYiEmQ=; b=Ht22nJLAujBD5FfXBxnaOBDOSaXkPPsqfoIwAQiEydr1jwF0vj4aika9KdOJBHCphxaXTNqqUjtnrsxGwWr1E/igS41MtegxwZQiGXJkzXvGS4pmwROsLCIhS1waC6TYYRU2mtHXLAIEF0/7GDKLPRmeNAMQP6gRDxfVutCpjtY= Authentication-Results: linux.ibm.com; dkim=none (message not signed) header.d=none;linux.ibm.com; dmarc=none action=none header.from=amd.com; Received: from SN6PR12MB2767.namprd12.prod.outlook.com (2603:10b6:805:75::23) by SN1PR12MB2416.namprd12.prod.outlook.com (2603:10b6:802:2f::30) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3912.17; Fri, 5 Mar 2021 10:44:29 +0000 Received: from SN6PR12MB2767.namprd12.prod.outlook.com ([fe80::24bb:3e53:c95e:cb8e]) by SN6PR12MB2767.namprd12.prod.outlook.com ([fe80::24bb:3e53:c95e:cb8e%7]) with mapi id 15.20.3912.018; Fri, 5 Mar 2021 10:44:29 +0000 Date: Fri, 5 Mar 2021 10:44:23 +0000 From: "Ashish Kalra" To: Tobin Feldman-Fitzthum Cc: Laszlo Ersek , devel@edk2.groups.io, Dov Murik , Tobin Feldman-Fitzthum , James Bottomley , Hubertus Franke , Brijesh Singh , Jon Grimm , Tom Lendacky Subject: Re: [edk2-devel] [RFC PATCH 00/14] Firmware Support for Fast Live Migration for AMD SEV Message-ID: <20210305104422.GA1984@ashkalra_ubuntu_server> References: <20210302204839.82042-1-tobin@linux.ibm.com> <9d7de545-7902-4d38-ba49-f084a750ee2a@redhat.com> <4869b086-f329-b79b-39ee-e21bfc5f95b5@linux.ibm.com> In-Reply-To: <4869b086-f329-b79b-39ee-e21bfc5f95b5@linux.ibm.com> User-Agent: Mutt/1.9.4 (2018-02-28) X-Originating-IP: [165.204.77.1] X-ClientProxiedBy: SN6PR2101CA0013.namprd21.prod.outlook.com (2603:10b6:805:106::23) To SN6PR12MB2767.namprd12.prod.outlook.com (2603:10b6:805:75::23) Return-Path: ashish.kalra@amd.com MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from ashkalra_ubuntu_server (165.204.77.1) by SN6PR2101CA0013.namprd21.prod.outlook.com (2603:10b6:805:106::23) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3933.7 via Frontend Transport; Fri, 5 Mar 2021 10:44:29 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-HT: Tenant X-MS-Office365-Filtering-Correlation-Id: be6bb1cb-6835-4e67-8b57-08d8dfc3a768 X-MS-TrafficTypeDiagnostic: SN1PR12MB2416: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:7219; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: Ag47gmWNcw1oPMJKQNKcTDAR4EUAYDAk5Or5gzbZ4roPkgQHQPq+hgyavozFp7JYZzOHtysDMjuI5tPW8DewS7EhL4MdG20oPlfwhhMZIMrfNV9RCqOLxhuthhM8yPyMRGkc+1XoN/YGZbFD4960Co7ug3ekzbqDrb9+yqr9hKBd0WjKEERkgLbwedxcFnPTd+I7SWbJvCZP1tbqKLvLBmI28fc1PGiOAVkfHFuyYYTompgDQeJgH5BD5u6fg8JrgzT7hVylo39wlqWKkV1E42QjsG9R6dYZywyKXjbj+KcqmDEpdXFTWN0HutPLhjjHjNWApr3f9VmhRbLSh406GRhYZpINq5eON5/ew7r98GMMMZFWXeUBSbCv4LcJZlO/s6EpXqKRvnAPRDOwZydIDdammSSLHSGqzlBLtbwNmXiyOQ4D+lIQkvA2otrhWo3szhbT9RA77KaY9Nkg/uQA/PeAYMp3zKjX7xnTcbltBkJVC58zyWv+fgWAecNIRZdn4ywRKeYW2Yr6AJcwisGQAA== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:SN6PR12MB2767.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(4636009)(376002)(346002)(366004)(136003)(39860400002)(396003)(4326008)(6666004)(52116002)(956004)(6916009)(26005)(478600001)(66556008)(86362001)(316002)(8676002)(186003)(66946007)(8936002)(6496006)(16526019)(5660300002)(55016002)(66476007)(83380400001)(54906003)(1076003)(33656002)(9686003)(33716001)(44832011)(53546011)(2906002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: =?us-ascii?Q?n6JulNFjENKkM5FMusOu4inYauPupBNq3ZH9s5QC7ANpaoHLZoE7i7Xwxu1R?= =?us-ascii?Q?HKJnePKvgUT5taMl9iULb9TKJEHqf08RZGjzuYkuQ66xPYco9OBSxKS0MRTx?= =?us-ascii?Q?1SshJVgJ5ObHtqDOwfu8grWVGr4jcWFxLobTW+3p4gXf+loaQ4R6T1184Vjm?= =?us-ascii?Q?291eJIlxFMg9CANKxMXRCS6GCj3m28dD/azzo7FTDnraPWPlMZxN94ug6+/b?= =?us-ascii?Q?7asD4Bc00LsBrdOcuUwv+kJIIwa7YHMmXuG1CfGDEr/qyV/Oi2VwXm0yRVpZ?= =?us-ascii?Q?MeunxS9UNvjWYwCclcTz0iOsToS7VfNgMdOqCTgwWnIiRPN7My9fSF4ywyRs?= =?us-ascii?Q?+V8I1/53EAtTBOL5+rk22VmyGzgHr5t5ZQJup5C6aOR2Wj82O4n0ROS6nyki?= =?us-ascii?Q?2UP8yPZi5bhzYuLCDn1DAcrd7e/o49iNIFudyLu5X7RJQLyOH/r7qr8cR+iW?= =?us-ascii?Q?OPIHmzH1Q6WWVv89C5IAcEqH3gysr3NDrnzBAXZIcaFkrjjoqIwSHdZ7w2RA?= =?us-ascii?Q?7goBEWCL2z3G5Ob44ucDY5T6b3qcS64fcUZdgQn8/xHcE3JOmq/1nje9gWYr?= =?us-ascii?Q?f2yK1YCy0wdu5DRTos4uCCVA/kQTnrS2g72yRZdNrL+IpM47BZcMMcfgcx/p?= =?us-ascii?Q?KPEI36XtzlVMEr0ZSpqE+5ETrzdDH+WOXDjFCa8V3KIzyAjMqiww0rFW8jn1?= =?us-ascii?Q?2Z+Z4VxynuelcgIUJOTNuRAbUSz5xyoyY3GqLsQAltqKaKMOwfFL2G9z/E70?= =?us-ascii?Q?dvgI9YvZNzZLvYrIYNdEQ/1olHL9q1BqBTl7LDkFC8I1E58kl0F1HqQ9khKb?= =?us-ascii?Q?8QbWQ+X4jKb+UBI7bQf2ZvoHHmTTYDMBwgqNcGEPBR8acHaPm2cIP8622Lia?= =?us-ascii?Q?P7iI9ezGIguaIteRp7Vtb6k9pVLqGbKGtNFH7hsN9j1SIVA4Aw3vjce4L85s?= =?us-ascii?Q?XlCt0z29DB1nS6+EV0JPxk1fOTFV+7Kw8SIZTE96INbkIANZQig6NsLMVLOl?= =?us-ascii?Q?9nSwyFwNkolg9OIIRBlElzo+LvlWxY0P7EoL+1VBuaN/7cdWFMB9qw4WD8VG?= =?us-ascii?Q?eoO4A/Z/w3HcmeqYjRnUufO/vHt+qCLexxvJWtbNZni45OGNM+052RmiOQ4E?= =?us-ascii?Q?xUb8D1oC7Wbg/xdXVye48SROyPm/Ni4511IQCW4jbQR5L3gwVEEOT73ByIaC?= =?us-ascii?Q?bHxIYuGz3mtIFQ7tS0fWgwWjsNpk1ioh7rO09hPhBTslUOywoJZ+EwixzvGk?= =?us-ascii?Q?rlc4Mor0MvUbCHqiGuC/7E1lkEQpi/DdRwKqqGD+9uCJDoA9aKgPVJRtnWAW?= =?us-ascii?Q?wX0PZv7rgJIacXdftjUv0LHY?= X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-Network-Message-Id: be6bb1cb-6835-4e67-8b57-08d8dfc3a768 X-MS-Exchange-CrossTenant-AuthSource: SN6PR12MB2767.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Mar 2021 10:44:29.6876 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 2/sZY1qBBNNlOrSQz+pgeOaA+/30k3EuQDaLAvARBKbfyF+kHVf0Cj+gVoN13z6EsQYZXdp/HvcVrdBTx8CcqA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN1PR12MB2416 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline On Wed, Mar 03, 2021 at 01:25:40PM -0500, Tobin Feldman-Fitzthum wrote: > > > Hi Tobin, > > > > On 03/02/21 21:48, Tobin Feldman-Fitzthum wrote: > > > This is a demonstration of fast migration for encrypted virtual machines > > > using a Migration Handler that lives in OVMF. This demo uses AMD SEV, > > > but the ideas may generalize to other confidential computing platforms. > > > With AMD SEV, guest memory is encrypted and the hypervisor cannot access > > > or move it. This makes migration tricky. In this demo, we show how the > > > HV can ask a Migration Handler (MH) in the firmware for an encrypted > > > page. The MH encrypts the page with a transport key prior to releasing > > > it to the HV. The target machine also runs an MH that decrypts the page > > > once it is passed in by the target HV. These patches are not ready for > > > production, but the are a full end-to-end solution that facilitates a > > > fast live migration between two SEV VMs. > > > > > > Corresponding patches for QEMU have been posted my colleague Dov Murik > > > on qemu-devel. Our approach needs little kernel support, requiring only > > > one hypercall that the guest can use to mark a page as encrypted or > > > shared. This series includes updated patches from Ashish Kalra and > > > Brijesh Singh that allow OVMF to use this hypercall. > > > > > > The MH runs continuously in the guest, waiting for communication from > > > the HV. The HV starts an additional vCPU for the MH but does not expose > > > it to the guest OS via ACPI. We use the MpService to start the MH. The > > > MpService is only available at runtime and processes that are started by > > > it are usually cleaned up on ExitBootServices. Since we need the MH to > > > run continuously, we had to make some modifications. Ideally a feature > > > could be added to the MpService to allow for the starting of > > > long-running processes. Besides migration, this could support other > > > background processes that need to operate within the encryption > > > boundary. For now, we have included a handful of patches that modify the > > > MpService to allow the MH to keep running after ExitBootServices. These > > > are temporary. > > I plan to do a lightweight review for this series. (My understanding is > > that it's an RFC and not actually being proposed for merging.) > > > > Regarding the MH's availability at runtime -- does that necessarily > > require the isolation of an AP? Because in the current approach, > > allowing the MP Services to survive into OS runtime (in some form or > > another) seems critical, and I don't think it's going to fly. > > > > I agree that the UefiCpuPkg patches have been well separated from the > > rest of the series, but I'm somewhat doubtful the "firmware-initiated > > background process" idea will be accepted. Have you investigated > > exposing a new "runtime service" (a function pointer) via the UEFI > > Configuration table, and calling that (perhaps periodically?) from the > > guest kernel? It would be a form of polling I guess. Or maybe, poll the > > mailbox directly in the kernel, and call the new firmware runtime > > service when there's an actual command to process. > Continuous runtime availability for the MH is almost certainly the most > controversial part of this proposal, which is why I put it in the cover > letter and why it's good to discuss. > > (You do spell out "little kernel support", and I'm not sure if that's a > > technical benefit, or a political / community benefit.) > > As you allude to, minimal kernel support is really one of the main things > that shapes our approach. This is partly a political and practical benefit, > but there are also technical benefits. Having the MH in firmware likely > leads to higher availability. It can be accessed when the OS is unreachable, > perhaps during boot or when the OS is hung. There are also potential > portability advantages although we do currently require support for one > hypercall. The cost of implementing this hypercall is low. > > Generally speaking, our task is to find a home for functionality that was > traditionally provided by the hypervisor, but that needs to be inside the > trust domain, but that isn't really part of a guest. A meta-goal of this > project is to figure out the best way to do this. > > > > > I'm quite uncomfortable with an attempt to hide a CPU from the OS via > > ACPI. The OS has other ways to learn (for example, a boot loader could > > use the MP services itself, stash the information, and hand it to the OS > > kernel -- this would minimally allow for detecting an inconsistency in > > the OS). What about "all-but-self" IPIs too -- the kernel might think > > all the processors it's poking like that were under its control. > > This might be the second most controversial piece. Here's a question: if we > could successfully hide the MH vCPU from the OS, would it still make you > uncomfortable? In other words, is the worry that there might be some > inconsistency or more generally that there is something hidden from the OS? > One thing to think about is that the guest owner should generally be aware > that there is a migration handler running. The way I see it, a guest owner > of an SEV VM would need to opt-in to migration and should then expect that > there is an MH running even if they aren't able to see it. Of course we need > to be certain that the MH isn't going to break the OS. > > > Also, as far as I can tell from patch #7, the AP seems to be > > busy-looping (with a CpuPause() added in), for the entire lifetime of > > the OS. Do I understand right? If so -- is it a temporary trait as well? > > In our approach the MH continuously checks for commands from the hypervisor. > There are potentially ways to optimize this, such as having the hypervisor > de-schedule the MH vCPU while not migrating. You could potentially shut down > down the MH on the target after receiving the MH_RESET command (when the > migration finishes), but what if you want to migrate that VM somewhere else? > I think another approach can be considered here, why not implement MH vCPU(s) as hot-plugged vCPU(s), basically hot-plug a new vCPU when migration is started and hot unplug the vCPU when migration is completed, then we won't need a vCPU running (and potentially consuming cycles) forever and busy-looping with CpuPause(). Thanks, Ashish