From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from aserp2120.oracle.com (aserp2120.oracle.com [141.146.126.78]) by mx.groups.io with SMTP id smtpd.web09.3202.1611882071984432895 for ; Thu, 28 Jan 2021 17:01:12 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@oracle.com header.s=corp-2020-01-29 header.b=r7JcBYj/; spf=pass (domain: oracle.com, ip: 141.146.126.78, mailfrom: ankur.a.arora@oracle.com) Received: from pps.filterd (aserp2120.oracle.com [127.0.0.1]) by aserp2120.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 10T0eplt064441; Fri, 29 Jan 2021 01:01:09 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2020-01-29; bh=JCwKS/cvEanE1H3hBiZcnWxV/RiXa5isy4L/1PprdC0=; b=r7JcBYj/odN7FFcBRXAVaQ2IOtX/Hvo0cB6CUBwCFE+SZnjJ/Pkft9+XLmjiHNI1IGS6 9ffLIkQ99BE4tjxGaBeqlXEMT5mjwSBlmaQi/hS/shz1SZLxhb1vN5eJM/Ayx96fKDZ7 umGXE50vEsTkVXy+cibP5L7xLasEg4dGZgF2DCttZjcGI4Q57ZRSmp0EiU3ExnYnwl4S kTGWPPssb/PLz7EGTjU6zXEbfyj5s3vCOsVBM5tGjml4hg5LeIuSgWshEExOci4K64Y7 AHkjh/ktw9cxmXrS4Wjsly6P8d+YzcI6i/5YrqkiQiRapZDEOeLIdlSaH3/5YsF01iQz Tg== Received: from userp3020.oracle.com (userp3020.oracle.com [156.151.31.79]) by aserp2120.oracle.com with ESMTP id 368brkxs9h-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 29 Jan 2021 01:01:09 +0000 Received: from pps.filterd (userp3020.oracle.com [127.0.0.1]) by userp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 10T0dtF1084478; Fri, 29 Jan 2021 01:01:08 GMT Received: from nam11-co1-obe.outbound.protection.outlook.com (mail-co1nam11lp2172.outbound.protection.outlook.com [104.47.56.172]) by userp3020.oracle.com with ESMTP id 368wjutpvx-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 29 Jan 2021 01:01:08 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=cm68AqGZGw//plVfF/Y0m9jMLrEsabv+Jf7msq6JoapvuAvAJ1MjDRr/81Np0gbINPlYB6Qz8y+ugW6w/oQn+ObYMKkCQMB6vJLYV/DUpKX82gqRcwVGnlERZwemCRwmLGu0vPwTomTWCbywZE7BpYHw0gBuZcVAApR8fOsDb8mdeFx+FE1TzhqFwh8rq1ztO4MTCaKtdDu8LWRc+koeTNFXUIFgEILOByyWtZ1U0l64Z2WCm4HfVxQL+ADjWSZmAT7+4tylc3y7g/xFXZcVr/ScEJgJ4f/CSli9Kr/dhRQe+AfYsr5lFNvakjlGOsWusP1/3s3VyDw1/q+hvil9dQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=JCwKS/cvEanE1H3hBiZcnWxV/RiXa5isy4L/1PprdC0=; b=GU7AXLY1dW4C3JY2jjJ+lw3jiB9Z0GBP28Tk+9QVIcs0PG6cP+DpjG7pxAknD8aXl7Q5vtDLp7Z8rQUg0+JRlDc+BkBgB/QHNevDPoClrbkVV42ITurri2QKpwrNkwQt9kLI7fSth92Rzr64jGTN8gCoxaXNvyJL6TOxkCzpeAzff03dTfAJYiI/x2TR3mPADtqwlS+ocT5ki5F4Xj+Mq6zQtXA2FnR/C6GbYmrMYfcFotZBCJ3lN0Hm0dVEBZ0mMMH1FBw2gQ7pt+4lUbF19IHwQtSXFivMvRDshbb7YKybz5cHo+HEQEVl9IYjDAJsCDDWRTV8+xQf9vXX10QFbA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=JCwKS/cvEanE1H3hBiZcnWxV/RiXa5isy4L/1PprdC0=; b=DeOZug+/KrIgnPD0bcfJuasneHJC7e8HfHcdNP3zoRF2jOPRNsvCc7CGamx7Rz6rks7/x3yNAkqWkOWvGRbnAIVeoQl7SVIsD7aEkVFaCTFWoxMUp2tSwh1lbZJ/vaH3Uh/bYO6ShEzzuhfWlVJMbnoCAB5ewhB965eJoUfPFM0= Authentication-Results: edk2.groups.io; dkim=none (message not signed) header.d=none;edk2.groups.io; dmarc=none action=none header.from=oracle.com; Received: from CY4PR10MB1718.namprd10.prod.outlook.com (2603:10b6:910:9::17) by CY4PR1001MB2246.namprd10.prod.outlook.com (2603:10b6:910:41::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3784.16; Fri, 29 Jan 2021 01:01:04 +0000 Received: from CY4PR10MB1718.namprd10.prod.outlook.com ([fe80::39dc:6ba7:9397:cb6]) by CY4PR10MB1718.namprd10.prod.outlook.com ([fe80::39dc:6ba7:9397:cb6%6]) with mapi id 15.20.3805.019; Fri, 29 Jan 2021 01:01:04 +0000 From: "Ankur Arora" To: devel@edk2.groups.io Cc: lersek@redhat.com, imammedo@redhat.com, boris.ostrovsky@oracle.com, Ankur Arora , Jordan Justen , Ard Biesheuvel , Aaron Young Subject: [PATCH v6 8/9] OvmfPkg/CpuHotplugSmm: add worker to do CPU ejection Date: Thu, 28 Jan 2021 16:59:49 -0800 Message-Id: <20210129005950.467638-9-ankur.a.arora@oracle.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210129005950.467638-1-ankur.a.arora@oracle.com> References: <20210129005950.467638-1-ankur.a.arora@oracle.com> X-Originating-IP: [148.87.23.8] X-ClientProxiedBy: MWHPR1401CA0022.namprd14.prod.outlook.com (2603:10b6:301:4b::32) To CY4PR10MB1718.namprd10.prod.outlook.com (2603:10b6:910:9::17) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from prion.us.oracle.com (148.87.23.8) by MWHPR1401CA0022.namprd14.prod.outlook.com (2603:10b6:301:4b::32) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3805.17 via Frontend Transport; Fri, 29 Jan 2021 01:00:07 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: c12de2ec-cc4e-4d07-f023-08d8c3f138df X-MS-TrafficTypeDiagnostic: CY4PR1001MB2246: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:10000; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: HlklQwGDx+sOxIISa3ler/tnNd7RFDlP5pYVYIl0AERY7sQzYpFe2vDsjd/agkcJyofBwSQpVdcvke0BFe5kdjNRjsnLKjbtifRVdrsl3KL+cp8oMpwrL8s5AUd6iArEmzdedbfxiHnuUoInegQaIhlTabVq/ddbLjbESpamHFu91d6WLHXpzQqHDhWZeS3MCMsqmsdXL59ZT0zJ54POvld4wuNzIOBaUfblBQkCtR3RbYin47FXUsxbT7IQJLe3Z52WE43bB1ZhfpNdOjeGiCEnT8qwLytPY7eWOG17YG5SU4P2hvbUEbH3xAK/T4GkS0R82iizfokf0Qa9THbYjuY6a0dVZi9gZHcxS/rr5V5m7f6iSdee0e7zgl4Fc2TxxHqw8YwPn//vI8A29DSURk9i5bsWW5Rz7fMmF1XwxMOapVoZ9ZbL+MPkXyPCnu+HJzU2hYBhUjM7a9mox4AD68JYV5ZHnM1ceS2NrgZBrVfaYiqGcTl2VB6lmRluFSpQQMH7oCcjcovT6jzSqLRIqwHiP7OixU9K/6TIbi2jn4k7QrtqA7+OWPfQe967beuGqFuQ+OYB3tS6JnyL3N/o3D4WVUGTi4EZpDSMTLkzLIk= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CY4PR10MB1718.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(366004)(39860400002)(346002)(376002)(396003)(136003)(8936002)(107886003)(2906002)(66946007)(66476007)(66556008)(36756003)(1076003)(103116003)(16526019)(956004)(966005)(7696005)(478600001)(52116002)(54906003)(6486002)(86362001)(83380400001)(6666004)(8676002)(186003)(5660300002)(4326008)(6916009)(26005)(316002)(2616005);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: =?us-ascii?Q?qn0ehnigrCcA7VLK6uGGuW7cWw1kxy3l4SMcS18Uaz1GjQdo87Yz3a8oEV76?= =?us-ascii?Q?H+sR0CbT3OShOEco09LuNJV8XmMemZXBFxuZY1uxM/cxX5im7O6raUr6UcND?= =?us-ascii?Q?XyjLLMtwCU0MHymBsqLKCNP+IQGz+gzwXhD9FQ7fYCLppNxx9+1hSNQyOo8Q?= =?us-ascii?Q?sV2kmvTJlIvzOsUtqRO+cISJPR6Yo4Aq5SLHjoXBQlrvTugKzgNWjxsPPthn?= =?us-ascii?Q?BQXk8Icgb1aeKYiCRRrR4qPkvm5GcPSqp+7HWmIq1twpoGB7bJbptnhg+fAK?= =?us-ascii?Q?dQGRR9as+reVhOzAn+hGDxCKaETLwLJK7TKa8Q6Wtl6wrw61df6moN5EOEHg?= =?us-ascii?Q?EDwIHa2gyvx2FlXgfs1gQCGHlpEKmK0ZD9mqv3DTSB3T1bNJjTInjUcmHCPO?= =?us-ascii?Q?w/cNAESW2yIsxquQYfPgoBcRyUFhYW2b3TAcOK/Cjw5D0qe9k2HpZmTTwLNM?= =?us-ascii?Q?X6Y8zdRTPL3FOViSg3wiW/KBzOhZ4+f559W1f31+AAEe96nAIcxHUOu7YsHP?= =?us-ascii?Q?mwB4Nud/oiMGCjpqkY+4HP0QufTF5brKugDn7m4SHpwvDPzX7V7fyw1av8yl?= =?us-ascii?Q?+kADqQ96p8VB2SktnrOgoEBX9ASU5WKP7HWsM5Q5h2Ensh1tIj+lKeMx03XP?= =?us-ascii?Q?9IaISIjFTDYHmQRFYF9TL8UoNDi6f+c508v1QnNT7AoQDympYZTwSDgJ46MU?= =?us-ascii?Q?pFowWWC8DpRcoa5yNZlG9umx8PthlxxbBbDmmlap0aV0j5bX5vrIcVYLgOxz?= =?us-ascii?Q?i8v4GkULx08DpXIJWGPGTXsqGovpIOlsJzBuJHbKtkfpxoXmrHjxMX1iu1Mv?= =?us-ascii?Q?vBl62IalMHBSPk42bwQivodhsuVIPGNrFmGnHxdPZqjoCCCzlh4L3Jhn4lI1?= =?us-ascii?Q?Y8KLTfX0mHp/6XJY3dMOmsmpXL2iy83Tz5Y5va4NyklSbGHxmUKF0xwz4X4Y?= =?us-ascii?Q?+S1bFQGkqpXyBGKALZ3moMdd4E56cAfz90PDe8n3EZi30tpKYd2oDRyjJgjU?= =?us-ascii?Q?CjcTs+u2yplnqJJYR6LjQE51u9SsgXhNdwrrJq+7fYT0SBrv6yGXGC/GXSTO?= =?us-ascii?Q?qfJCmo+I?= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: c12de2ec-cc4e-4d07-f023-08d8c3f138df X-MS-Exchange-CrossTenant-AuthSource: CY4PR10MB1718.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 29 Jan 2021 01:00:08.5218 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: elx558NLx7BawA20Ume7sl9zyaUeyTP772QEn0+5hu+lFwqoNrPisVAu9SEhkqJpbKiayDBWUCfKCyBftpgd6VUMtc7ZTAxnWnZzDNjsQEw= X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY4PR1001MB2246 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9878 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 suspectscore=0 adultscore=0 mlxscore=0 malwarescore=0 spamscore=0 mlxlogscore=999 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2101290001 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9878 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 impostorscore=0 phishscore=0 bulkscore=0 priorityscore=1501 mlxlogscore=999 lowpriorityscore=0 spamscore=0 mlxscore=0 suspectscore=0 malwarescore=0 clxscore=1015 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2101290001 Content-Transfer-Encoding: 8bit Content-Type: text/plain Designate a worker CPU (we use the one executing the root MMI handler), which will do the actual ejection via QEMU in CpuEject(). CpuEject(), on the worker CPU, ejects each marked CPU by first selecting its APIC ID and then sending the QEMU "eject" command. QEMU in-turn signals the remote VCPU thread which context-switches it out of the SMI. CpuEject(), on the CPU being ejected, spins around in its holding area until this final context-switch. This does mean that there is some CPU state that would ordinarily be restored (in SmiRendezvous() and in SmiEntry.nasm::CommonHandler), but will not be anymore. This unrestored state includes FPU state, CET enable, stuffing of RSB and the final RSM. Since the CPU state is destroyed by QEMU, this should be okay. Cc: Laszlo Ersek Cc: Jordan Justen Cc: Ard Biesheuvel Cc: Igor Mammedov Cc: Boris Ostrovsky Cc: Aaron Young Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3132 Signed-off-by: Ankur Arora --- OvmfPkg/CpuHotplugSmm/CpuHotplug.c | 73 ++++++++++++++++++++++++++++++++++---- 1 file changed, 67 insertions(+), 6 deletions(-) diff --git a/OvmfPkg/CpuHotplugSmm/CpuHotplug.c b/OvmfPkg/CpuHotplugSmm/CpuHotplug.c index 526f51faf070..bf91344eef9c 100644 --- a/OvmfPkg/CpuHotplugSmm/CpuHotplug.c +++ b/OvmfPkg/CpuHotplugSmm/CpuHotplug.c @@ -193,9 +193,12 @@ RevokeNewSlot: CPU Hot-eject handler, called from SmmCpuFeaturesRendezvousExit(), on each CPU at exit from SMM. - If, the executing CPU is not being ejected, nothing to be done. + If, the executing CPU is neither a worker, nor being ejected, nothing + to be done. If, the executing CPU is being ejected, wait in a CpuDeadLoop() until ejected. + If, the executing CPU is a worker CPU, set QEMU CPU status to eject + for CPUs being ejected. @param[in] ProcessorNum Index of executing CPU. @@ -217,6 +220,56 @@ CpuEject ( return; } + if (ApicId == CPU_EJECT_WORKER) { + UINT32 CpuIndex; + + for (CpuIndex = 0; CpuIndex < mCpuHotEjectData->ArrayLength; CpuIndex++) { + UINT64 RemoveApicId; + + RemoveApicId = mCpuHotEjectData->ApicIdMap[CpuIndex]; + + if ((RemoveApicId != CPU_EJECT_INVALID && + RemoveApicId != CPU_EJECT_WORKER)) { + // + // This to-be-ejected-CPU has already received the BSP's SMI exit + // signal and, will execute SmmCpuFeaturesSmiRendezvousExit() + // followed by this callback or is already waiting in the + // CpuDeadLoop() below. + // + // Tell QEMU to context-switch it out. + // + QemuCpuhpWriteCpuSelector (mMmCpuIo, (APIC_ID) RemoveApicId); + QemuCpuhpWriteCpuStatus (mMmCpuIo, QEMU_CPUHP_STAT_EJECTED); + + // + // Compiler barrier to ensure the next store isn't reordered + // + MemoryFence (); + + // + // Clear the eject status for CpuIndex to ensure that an invalid + // SMI later does not end up trying to eject it or a newly + // hotplugged CpuIndex does not go into the dead loop. + // + mCpuHotEjectData->ApicIdMap[CpuIndex] = CPU_EJECT_INVALID; + + DEBUG ((DEBUG_INFO, "%a: Unplugged CPU %u -> " FMT_APIC_ID "\n", + __FUNCTION__, CpuIndex, RemoveApicId)); + } + } + + // + // Clear our own worker status. + // + mCpuHotEjectData->ApicIdMap[ProcessorNum] = CPU_EJECT_INVALID; + + // + // We are done until the next hot-unplug; clear the handler. + // + mCpuHotEjectData->Handler = NULL; + return; + } + // // CPU(s) being unplugged get here from SmmCpuFeaturesSmiRendezvousExit() // after having been cleared to exit the SMI by the monarch and thus have @@ -327,6 +380,19 @@ UnplugCpus ( } if (EjectCount != 0) { + UINTN Worker; + + Status = mMmCpuService->WhoAmI (mMmCpuService, &Worker); + ASSERT_EFI_ERROR (Status); + // + // UnplugCpus() is called via the root MMI handler and thus we are + // executing in the BSP context. + // + // Mark ourselves as the worker CPU. + // + ASSERT (mCpuHotEjectData->ApicIdMap[Worker] == CPU_EJECT_INVALID); + mCpuHotEjectData->ApicIdMap[Worker] = CPU_EJECT_WORKER; + // // We have processors to be ejected; install the handler. // @@ -451,11 +517,6 @@ CpuHotplugMmi ( if (EFI_ERROR (Status)) { goto Fatal; } - if (ToUnplugCount > 0) { - DEBUG ((DEBUG_ERROR, "%a: hot-unplug is not supported yet\n", - __FUNCTION__)); - goto Fatal; - } if (PluggedCount > 0) { Status = ProcessHotAddedCpus (mPluggedApicIds, PluggedCount); -- 2.9.3