From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from userp2120.oracle.com (userp2120.oracle.com [156.151.31.85]) by mx.groups.io with SMTP id smtpd.web10.31722.1613978394805808714 for ; Sun, 21 Feb 2021 23:19:54 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@oracle.com header.s=corp-2020-01-29 header.b=Dml0x3sB; spf=pass (domain: oracle.com, ip: 156.151.31.85, mailfrom: ankur.a.arora@oracle.com) Received: from pps.filterd (userp2120.oracle.com [127.0.0.1]) by userp2120.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 11M7EsEs062555; Mon, 22 Feb 2021 07:19:52 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2020-01-29; bh=MgpR46G4gEZ1i5F/quRXYVLNsxZpGa9WkQv6hNoH3I8=; b=Dml0x3sBrZPH9qI+J1EMB0hkrI0CdBEBNQlTOhl6WwCGdkL3WpIZrBQn31HDVywAPGrA pFyZhK35DTuom2XQ1RyaqXgaH0qaywu0PfathwpvuM/Q+q90OEDEFpp77gLQiE1zk+PK Kmcg63xB+bbRv0yk+47cnYlp+RCdVSMchdDzsvihxxDgEBLX0+4Kzy2DWyp3xlDtA67w 15TC6DsbSS0HI4UW2CN6b47NxCtTYcuCeYewTPsJxNMvHuWGwJTx4BhLYXOu+xJ7O8nJ YwGnDTIpTSE8toJHqWi5C4sGOY7KwVARPnaUaoPKmhRDmkCjaAjV+FPeDM5rae1Qi1xT 8Q== Received: from aserp3020.oracle.com (aserp3020.oracle.com [141.146.126.70]) by userp2120.oracle.com with ESMTP id 36ugq39fhj-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 22 Feb 2021 07:19:52 +0000 Received: from pps.filterd (aserp3020.oracle.com [127.0.0.1]) by aserp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 11M7GSLB067574; Mon, 22 Feb 2021 07:19:51 GMT Received: from nam11-dm6-obe.outbound.protection.outlook.com (mail-dm6nam11lp2168.outbound.protection.outlook.com [104.47.57.168]) by aserp3020.oracle.com with ESMTP id 36ucawnxq0-4 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 22 Feb 2021 07:19:51 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=RyuYmAgZfZRTpHHcdlcDMW1jdc+exYTGfnwwCxTO33iYJbIqoVtwhHsH8lW+7uvMMMeFktz1hja0RkFYc+0zxE7n2pCZ6sVSXE5lKI1oAybhYt2JJsFgBlJXQul63IEjVdzpuzDIURYxSlGwKdUMGLhuyysjh9NnwMikfQciiIA5poSY1chYKEPKMH8uirzMmv//Gs9X+O9c1GZLweGjET8l8iIA4y6rSe6rBPTMQa+zzo3u2vymzoK9JKY/CdtXj2zp+KqnixlaR87dKpFYAHrgMKRIqMYSduiMH2ZUwP7bw02JnIMk+bSQG/hvK90GXz99VgsoN9gXnajpFEJW6g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=MgpR46G4gEZ1i5F/quRXYVLNsxZpGa9WkQv6hNoH3I8=; b=cWg4Qm/xHtlgI0exzoOrrNYl8ZJqh3aq2N2IeYDGsyD6iSR/47hINlU22wnvUIUjmhtLpmriJETR2ZClMkGm3X+AW0rekqZCivXf3zDLwb991QKvteSnoFsRlpCqAgU4Rr2ABypclc7Kq1lHBE6IuyJ27OTQZFGLky6eIP0R8wdoUJoL+rgXvutDO8DnPPoG3AXh9daEvkDa9rcr378yNCKb7JaeY+U+igGN2+gjQNZofQHLzt5MvbMU0QBlfMb4ubt6X983IbpyvR3KErSfJqWTbqyr0khn/SgQ1Mv9JzkQ2wWrcd2xqs0k6/JDNkkHwNPyZjv0Racq2IAhQT0Q/g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=MgpR46G4gEZ1i5F/quRXYVLNsxZpGa9WkQv6hNoH3I8=; b=ciVwyGRqo1TAMJC/n8Uw3VSQ4qdX35Nbx0DB06D/49FGhZEqtlhbGKl9z2r8IN+xp9wV/FO2HeOHiD0ICT35PlyLdhs/itfpChHDjJUaZ8hr//wscbxocgItUETI91GeR0Hx3gH5YOXF0pamLIQQ/UoLmmZbdW9PNO7kVhlb/s8= Authentication-Results: edk2.groups.io; dkim=none (message not signed) header.d=none;edk2.groups.io; dmarc=none action=none header.from=oracle.com; Received: from SJ0PR10MB4605.namprd10.prod.outlook.com (2603:10b6:a03:2d9::24) by BY5PR10MB3922.namprd10.prod.outlook.com (2603:10b6:a03:1fd::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3868.32; Mon, 22 Feb 2021 07:19:49 +0000 Received: from SJ0PR10MB4605.namprd10.prod.outlook.com ([fe80::a021:790:7ce6:6f16]) by SJ0PR10MB4605.namprd10.prod.outlook.com ([fe80::a021:790:7ce6:6f16%6]) with mapi id 15.20.3868.033; Mon, 22 Feb 2021 07:19:48 +0000 From: "Ankur Arora" To: devel@edk2.groups.io Cc: lersek@redhat.com, imammedo@redhat.com, boris.ostrovsky@oracle.com, Ankur Arora , Jordan Justen , Ard Biesheuvel , Aaron Young Subject: [PATCH v8 09/10] OvmfPkg/CpuHotplugSmm: do actual CPU hot-eject Date: Sun, 21 Feb 2021 23:19:27 -0800 Message-Id: <20210222071928.1401820-10-ankur.a.arora@oracle.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210222071928.1401820-1-ankur.a.arora@oracle.com> References: <20210222071928.1401820-1-ankur.a.arora@oracle.com> X-Originating-IP: [148.87.23.11] X-ClientProxiedBy: MWHPR17CA0065.namprd17.prod.outlook.com (2603:10b6:300:93::27) To SJ0PR10MB4605.namprd10.prod.outlook.com (2603:10b6:a03:2d9::24) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from prion.us.oracle.com (148.87.23.11) by MWHPR17CA0065.namprd17.prod.outlook.com (2603:10b6:300:93::27) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3868.27 via Frontend Transport; Mon, 22 Feb 2021 07:19:47 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 4a67388a-8ca5-430f-09e0-08d8d7023cd7 X-MS-TrafficTypeDiagnostic: BY5PR10MB3922: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:5236; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: Q4anziotrNaQ/9CJVL22sTb570wuwZk2AMHFRgjKSnlf/w3B5jxknaa17SEZT320XPOeEU6jwnoxKjZYiyf/uSXCEAXwe7xLe04gY2LulRE1H7XqXMybr6haSUdFVeUkHmXGikaN6bBm1uiuAVP60tKvYLRcjbXPTbmhpjKkkA2R6CYkImyjFRTvGj8AYzXi9Va2g2iZZdBG9wzPuiRJ0CTSdOtWdAdUiRYrLWzISdxbbwd+vl9gceINpDZWiqlqynYlro94Bs4VKpiEd/Obaw/8kevGVz2CKF0zqurnmKPUa9FiXxOBEs5xuy2nQ54UfhxWgZga7yA2BuUvDlGwMxaNLqjalx7Js0H0BF6xrSWHMkE1rPPHYmKU0osjY0gSsSBnrHWKTb3tVQBwpArx42STApc+RZJetd+KTqBoX2I+uwQsSoFW4EbSEfuP4Fl0SgROL9oavqLolWYNyQv2QntG5zht1Xo93Hg/vP374t5luZ1yby+Zqv1pEVUx7485lJH+pmaxLy6Q8ELwTbudW6mMbqSksIz3iTdRMqybiR4Na6PKsmOR9GOWEsBnpEYTU5SC7D/o27Omxdyni7F4Ic/5arFw0bHgdQwXc7EgcQ4= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:SJ0PR10MB4605.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(396003)(376002)(39860400002)(346002)(136003)(366004)(6666004)(6486002)(6916009)(2906002)(86362001)(52116002)(1076003)(7696005)(316002)(36756003)(107886003)(4326008)(26005)(478600001)(2616005)(8676002)(8936002)(966005)(16526019)(5660300002)(186003)(54906003)(103116003)(66476007)(66556008)(66946007)(83380400001)(30864003)(956004)(19627235002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: =?us-ascii?Q?27gA4IZ31iXKvE0scfCypzbp+VibftkKVqmWdscH1CSyDgGyKpkAVYWhvt7Z?= =?us-ascii?Q?6dqbLpZepHYMz3dBQW+jJeHQAzT5mgWRK0QipuufMvOLeNI49HYZSDsw9QOR?= =?us-ascii?Q?TM3KjX+MVKp36C5GWlNSNQq7WN/suPZftCJKXEYF2RcFqK/qSfgJkiKy38iK?= =?us-ascii?Q?EQMtkcYFc5Gd3iX0CPGXsw/xPY5i//CO+7YT8zEBhHTtc2e6+kN6H2b7k2+M?= =?us-ascii?Q?rMBqekhwtGL1VwOv9uqvDPwbpp4OOF1bCx+oK2bjSZ+cLkPR3r4J/dGIxLzo?= =?us-ascii?Q?hbEmWK3FFiLZ8OsUnVnEkCXzIiyT45I5G+VxWfuKw520HQShrI5IYStYXpjI?= =?us-ascii?Q?dJc6m3XVwjmZzRI/a8mG+ilDrVKhJpThF/OF95SNSP6rW6bZsoaKHkvkJs+X?= =?us-ascii?Q?Yvm/UKMlGhJrPTNn8BCD5bCe8nzGo/GWl2EIIb9/a7Jj7fvX0FHjr3Ny5mWK?= =?us-ascii?Q?dG/d/WaUawAkKqMkBOwhM8SEZEm6BkwY1ERUmRxlisntNUSheAZcbwLxKVqp?= =?us-ascii?Q?plM2TeTMbBE6cBZtRGkX1VRF0MS4kYE/4f6qldW52on3es4GjDUTiy9/Sv1Y?= =?us-ascii?Q?BU1SgCxXnndMb9rEfRTEKa+WyZh+J2iyQ/VXDY+sw3/CN8tTIu2xDYivdhml?= =?us-ascii?Q?CNfeZKxFkgPFhDFjoVPX3pNG7dpZr/0ey5lKuYnRivQmFp/q+17oXMvE/z3u?= =?us-ascii?Q?vT0KQ9VrFlxS5ZJmvzW03/OlvAmYom7f1ImNJPI3B++h/TMtzHN096OezoYd?= =?us-ascii?Q?TXNcoGYS4sbBBkKPJ4VvEkhHaV1d2BrVglZNXsPG49lhKA5486E7EiMiaUGI?= =?us-ascii?Q?ahqgZ/J+WSIZqZqZuw7AwdWw+PkcqChjf3/nElS120I/fR+rfdoLW3Um6sDz?= =?us-ascii?Q?pvLDiyIVqC9yTLRvgMOZYxMZ7K7n8LAKeDqrO2KOOFH782Ir69ufUC9hYxzl?= =?us-ascii?Q?cUJTZZxZBwAJ20c9mF2cvdiael2zoFNTGYzJa+tg1mgRnR4mtNppwPNw9zcE?= =?us-ascii?Q?NuuAWwT29fCYj5qGF6SrqoSQgmyxclPEs63B/ctJOD1Tj2h8bxs/h9jc4gvU?= =?us-ascii?Q?8Jw8Aj4TaEkJqjn33K9l/ijV8TU4iGSXVlRHO0hFGmUEGjcIIvtcobmoo9Tz?= =?us-ascii?Q?tZvlO9GO07hXW/S9Zt3CtqToS/zWdmCS88MvZSP5+meQF3QaJbbdDBMOc4Mr?= =?us-ascii?Q?WYaMNNCIOsyENI22506fakmWfvi6H1XdmeJsSrMuguBVT6LeZUbH4ji+r8ET?= =?us-ascii?Q?XM7jP64MG7CDM9uAr9Ozkq+MD+ZwnVzVMlAuDLewoABHSezNIgKHgMJSSyIN?= =?us-ascii?Q?xy+x2fZk9sQgZUvPx4L4cjWb?= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 4a67388a-8ca5-430f-09e0-08d8d7023cd7 X-MS-Exchange-CrossTenant-AuthSource: SJ0PR10MB4605.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Feb 2021 07:19:48.4811 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: nhAzVFo94FUIEE+vxwTGNNavbysHH4iLRf79uC/j0OrTQQDc0z4JFEoP0Cjz20Hj4r+O0mmVdyUbpIs+ZvXKjmokp5BRAhTLTfInoJmq5u0= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY5PR10MB3922 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9902 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 suspectscore=0 malwarescore=0 mlxlogscore=999 adultscore=0 bulkscore=0 mlxscore=0 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2102220064 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9902 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 adultscore=0 phishscore=0 malwarescore=0 spamscore=0 mlxscore=0 suspectscore=0 priorityscore=1501 clxscore=1015 impostorscore=0 lowpriorityscore=0 mlxlogscore=999 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2102220064 Content-Transfer-Encoding: 8bit Content-Type: text/plain Add logic in EjectCpu() to do the actual the CPU ejection. On the BSP, ejection happens by first selecting the CPU via its QemuSelector and then sending the QEMU "eject" command. QEMU in-turn signals the remote VCPU thread which context-switches the CPU out of the SMI handler. Meanwhile the CPU being ejected, waits around in its holding area until it is context-switched out. Note that it is possible that a slow CPU gets ejected before it reaches the wait loop. However, this would never happen before it has executed the "AllCpusInSync" loop in SmiRendezvous(). It can mean that an ejected CPU does not execute code after that point but given that the CPU state will be destroyed by QEMU, the missed cleanup is no great loss. Cc: Laszlo Ersek Cc: Jordan Justen Cc: Ard Biesheuvel Cc: Igor Mammedov Cc: Boris Ostrovsky Cc: Aaron Young Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3132 Signed-off-by: Ankur Arora --- Notes: Addresses the following reviewing comments from v6: (1) s/CpuEject/EjectCpu/g (2,2a,2c) Get rid of eject-worker and related. (2b,2d) Use the PlatformSmmBspElection() logic to find out IsBSP. (3,3b) Use CPU_HOT_EJECT_DATA->QemuSelector instead of ApicIdMap to do the actual ejection. (4,5a,5b) Fix the format etc in the final unplugged log message () Also as discussed elsewhere document the ordering requirements for mCpuHotEjectData->QemuSelector[] and mCpuHotEjectData->Handler. () [from patch 2] Move definition of QEMU_CPUHP_STAT_EJECTED to this patch. () s/QEMU_CPUHP_STAT_EJECTED/QEMU_CPUHP_STAT_EJECT/ OvmfPkg/Include/IndustryStandard/QemuCpuHotplug.h | 1 + OvmfPkg/CpuHotplugSmm/CpuHotplug.c | 127 +++++++++++++++++++-- .../Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.c | 31 +++++ 3 files changed, 152 insertions(+), 7 deletions(-) diff --git a/OvmfPkg/Include/IndustryStandard/QemuCpuHotplug.h b/OvmfPkg/Include/IndustryStandard/QemuCpuHotplug.h index 2ec7a107a64d..d0e83102c13f 100644 --- a/OvmfPkg/Include/IndustryStandard/QemuCpuHotplug.h +++ b/OvmfPkg/Include/IndustryStandard/QemuCpuHotplug.h @@ -34,6 +34,7 @@ #define QEMU_CPUHP_STAT_ENABLED BIT0 #define QEMU_CPUHP_STAT_INSERT BIT1 #define QEMU_CPUHP_STAT_REMOVE BIT2 +#define QEMU_CPUHP_STAT_EJECT BIT3 #define QEMU_CPUHP_STAT_FW_REMOVE BIT4 #define QEMU_CPUHP_RW_CMD_DATA 0x8 diff --git a/OvmfPkg/CpuHotplugSmm/CpuHotplug.c b/OvmfPkg/CpuHotplugSmm/CpuHotplug.c index 851e2b28aad9..0484be8fe43c 100644 --- a/OvmfPkg/CpuHotplugSmm/CpuHotplug.c +++ b/OvmfPkg/CpuHotplugSmm/CpuHotplug.c @@ -18,6 +18,7 @@ #include // CPU_HOT_EJECT_DATA #include // EFI_MM_CPU_IO_PROTOCOL #include // EFI_SMM_CPU_SERVICE_PROTOCOL +#include // MSR_IA32_APIC_BASE_REGISTER #include // EFI_STATUS #include "ApicId.h" // APIC_ID @@ -191,12 +192,39 @@ RevokeNewSlot: } /** + EjectCpu needs to know the BSP at SMI exit at a point when + some of the EFI_SMM_CPU_SERVICE_PROTOCOL state has been torn + down. + Reuse the logic from OvmfPkg::PlatformSmmBspElection() to + do that. + + @param[in] ProcessorNum ProcessorNum denotes the processor handle number + in EFI_SMM_CPU_SERVICE_PROTOCOL. +**/ +STATIC +BOOLEAN +CheckIfBsp ( + IN UINTN ProcessorNum + ) +{ + MSR_IA32_APIC_BASE_REGISTER ApicBaseMsr; + BOOLEAN IsBsp; + + ApicBaseMsr.Uint64 = AsmReadMsr64 (MSR_IA32_APIC_BASE); + IsBsp = (BOOLEAN)(ApicBaseMsr.Bits.BSP == 1); + return IsBsp; +} + +/** CPU Hot-eject handler, called from SmmCpuFeaturesRendezvousExit() on each CPU at exit from SMM. - If, the executing CPU is not being ejected, nothing to be done. + If, the executing CPU is neither the BSP, nor being ejected, nothing + to be done. If, the executing CPU is being ejected, wait in a halted loop until ejected. + If, the executing CPU is the BSP, set QEMU CPU status to eject + for CPUs being ejected. @param[in] ProcessorNum ProcessorNum denotes the CPU exiting SMM, and will be used as an index into @@ -211,9 +239,99 @@ EjectCpu ( ) { UINT64 QemuSelector; + BOOLEAN IsBsp; + IsBsp = CheckIfBsp (ProcessorNum); + + // + // mCpuHotEjectData->QemuSelectorMap[ProcessorNum] is updated + // on the BSP in the ongoing SMI iteration at two places: + // + // - UnplugCpus() where the BSP determines if a CPU is under ejection + // or not. As the comment where mCpuHotEjectData->Handler is set-up + // describes any such updates are guaranteed to be ordered-before the + // dereference below. + // + // - EjectCpu() on the BSP updates QemuSelectorMap[ProcessorNum] for + // CPUs after they have been hot-ejected. + // + // The CPU under ejection: might be executing anywhere between the + // "AllCpusInSync" exit loop in SmiRendezvous() to about to + // dereference QemuSelectorMap[ProcessorNum]. + // Given that the BSP ensures that this store only happens after the + // CPU has been ejected, this CPU would never see the after value. + // (Note that any CPU that is already executing the CpuSleep() loop + // below never raced any updates and always saw the before value.) + // + // CPUs not-under ejection: never see any changes so they are fine. + // + // Lastly, note that we are also guaranteed that any dereferencing + // CPU only sees the before or after value and not an intermediate + // value. This is because QemuSelectorMap[ProcessorNum] is aligned at + // a natural boundary. + // QemuSelector = mCpuHotEjectData->QemuSelectorMap[ProcessorNum]; - if (QemuSelector == CPU_EJECT_QEMU_SELECTOR_INVALID) { + if (QemuSelector == CPU_EJECT_QEMU_SELECTOR_INVALID && !IsBsp) { + return; + } + + if (IsBsp) { + UINT32 Idx; + + for (Idx = 0; Idx < mCpuHotEjectData->ArrayLength; Idx++) { + UINT64 QemuSelector; + + QemuSelector = mCpuHotEjectData->QemuSelectorMap[Idx]; + + if (QemuSelector != CPU_EJECT_QEMU_SELECTOR_INVALID) { + // + // This to-be-ejected-CPU has already received the BSP's SMI exit + // signal and, will execute SmmCpuFeaturesRendezvousExit() + // followed by this callback or is already waiting in the + // CpuSleep() loop below. + // + // Tell QEMU to context-switch it out. + // + QemuCpuhpWriteCpuSelector (mMmCpuIo, (UINT32) QemuSelector); + QemuCpuhpWriteCpuStatus (mMmCpuIo, QEMU_CPUHP_STAT_EJECT); + + // + // We need a compiler barrier here to ensure that the compiler + // does not reorder the CpuStatus and QemuSelectorMap[Idx] stores. + // + // A store fence is not strictly necessary on x86 which has + // TSO; however, both of these stores are in different address spaces + // so also add a Store Fence here. + // + MemoryFence (); + + // + // Clear the eject status for this CPU Idx to ensure that an invalid + // SMI later does not end up trying to eject it or a newly + // hotplugged CPU Idx does not go into the dead loop. + // + mCpuHotEjectData->QemuSelectorMap[Idx] = + CPU_EJECT_QEMU_SELECTOR_INVALID; + + DEBUG ((DEBUG_INFO, "%a: Unplugged ProcessorNum %u, " + "QemuSelector 0x%Lx\n", __FUNCTION__, Idx, QemuSelector)); + } + } + + // + // We are done until the next hot-unplug; clear the handler. + // + // By virtue of the MemoryFence() in the ejection loop above, the + // following store is ordered-after all the ejections are done. + // (We know that there is at least one CPU hot-eject handler if this + // handler was installed.) + // + // As described in OvmfPkg::SmmCpuFeaturesRendezvousExit() this + // means that the only CPUs which might dereference + // mCpuHotEjectData->Handler are not under ejection, so we can + // safely reset. + // + mCpuHotEjectData->Handler = NULL; return; } @@ -496,11 +614,6 @@ CpuHotplugMmi ( if (EFI_ERROR (Status)) { goto Fatal; } - if (ToUnplugCount > 0) { - DEBUG ((DEBUG_ERROR, "%a: hot-unplug is not supported yet\n", - __FUNCTION__)); - goto Fatal; - } if (PluggedCount > 0) { Status = ProcessHotAddedCpus (mPluggedApicIds, PluggedCount); diff --git a/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.c b/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.c index 99988285b6a2..ddfef05ee6cf 100644 --- a/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.c +++ b/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.c @@ -472,6 +472,37 @@ SmmCpuFeaturesRendezvousExit ( // (PcdCpuMaxLogicalProcessorNumber > 1), and hot-eject is needed // in this SMI exit (otherwise mCpuHotEjectData->Handler is not armed.) // + // mCpuHotEjectData itself is stable once setup so it can be + // dereferenced without needing any synchronization, + // but, mCpuHotEjectData->Handler is updated on the BSP in the + // ongoing SMI iteration at two places: + // + // - UnplugCpus() where the BSP determines if a CPU is under ejection + // or not. As the comment where mCpuHotEjectData->Handler is set-up + // describes any such updates are guaranteed to be ordered-before the + // dereference below. + // + // - EjectCpu() (which is called via the Handler below), on the BSP + // updates mCpuHotEjectData->Handler once it is done with all ejections. + // + // The CPU under ejection: might be executing anywhere between the + // "AllCpusInSync" exit loop in SmiRendezvous() to about to + // dereference the Handler field. + // Given that the BSP ensures that this store only happens after all + // CPUs under ejection have been ejected, this CPU would never see + // the after value. + // (Note that any CPU that is already executing the CpuSleep() loop + // below never raced any updates and always saw the before value.) + // + // CPUs not-under ejection: might see either value of the Handler + // which is fine, because the Handler is a NOP for CPUs not-under + // ejection. + // + // Lastly, note that we are also guaranteed that any dereferencing + // CPU only sees the before or after value and not an intermediate + // value. This is because mCpuHotEjectData->Handler is aligned at a + // natural boundary. + // if (mCpuHotEjectData != NULL) { CPU_HOT_EJECT_HANDLER Handler; -- 2.9.3