From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from userp2130.oracle.com (userp2130.oracle.com [156.151.31.86]) by mx.groups.io with SMTP id smtpd.web09.3201.1611882069925198477 for ; Thu, 28 Jan 2021 17:01:10 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@oracle.com header.s=corp-2020-01-29 header.b=MC8mX/vV; spf=pass (domain: oracle.com, ip: 156.151.31.86, mailfrom: ankur.a.arora@oracle.com) Received: from pps.filterd (userp2130.oracle.com [127.0.0.1]) by userp2130.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 10T0dwYl041203; Fri, 29 Jan 2021 01:01:07 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-transfer-encoding : content-type : mime-version; s=corp-2020-01-29; bh=Cas9E3Jz2zOrncuZdVvfzFahgTeS8mCUq401jOp9ndU=; b=MC8mX/vVduUUFmjVj5zimdPu4y7SUn+Y3fMnMW0SUO7wF8hqmIzrYubckqlzTflOn+M2 wQYigFKxgVvQBVtpBVgX3IwCHCC7WY1PftLon2j7odoGcandRFfJcOdwG31QKeK3Kox4 ss0XX7pI6ghfacAP3gInjS+XvspN/RG6i1SOXExofY16bk6tmxHRX6xubLyN8b2P9mQd mn4jqqzxLmwvVeZThJbEDcZYkkiwEIgEMy8h0ouQC8Pkg87i1PtSj0mMa31ddPbYoxF3 gWtmpg7zB9mN04Tu9sYCSyUfvxEAysoqNjl8glwJGz8VKd7wXtZBxAIDJMJOrPGAi+15 Tw== Received: from userp3030.oracle.com (userp3030.oracle.com [156.151.31.80]) by userp2130.oracle.com with ESMTP id 368b7r6v3j-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 29 Jan 2021 01:01:06 +0000 Received: from pps.filterd (userp3030.oracle.com [127.0.0.1]) by userp3030.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 10T0eFsm081629; Fri, 29 Jan 2021 01:01:06 GMT Received: from nam11-co1-obe.outbound.protection.outlook.com (mail-co1nam11lp2173.outbound.protection.outlook.com [104.47.56.173]) by userp3030.oracle.com with ESMTP id 368wr13gd2-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 29 Jan 2021 01:01:06 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=AF3/6VG7E43wTCRLT5SV8aAndBifGBIlTfBAjdgaxCgN0YvtcaqsElKel81y7GENRPtUXiTKUEdOSIp4Kd+HRjTLbw0/7SW3o4Y7UFmS9unUoyWa0hlXbFnCCh7xux+6osmllLIi2VwvOrvUJzm2/W0A/amNyFmcpvpcsb1Pb1Pv+qjBMKQ7KVLFeNXBxtPa6dEOWsnf2rjAy/bz4rawT9sqnX2Ury/k4zhNU2omPWNNioH4iJzEGBj1LhCEv0JtJ+HW3xvQcSVUrIx+mOvIVL2ff9CTO4OpfCT7Qya+GVpAtjAf8g6C6k3b+7wGYfDN/r/PqhU+ahwppzUZvhQ/Ng== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Cas9E3Jz2zOrncuZdVvfzFahgTeS8mCUq401jOp9ndU=; b=mrHqEscHOPKBLMeEE8qMenoUbVbH9sr0y/eCPUt3WJ6B8YffZlsCaUxf3W1x4wnGJUT+Fgbt8OuJepiKjFzoNqQIct+78hDNQwCnp5VqHsPz7/ODyCmbe4FPt4n6pVy/qBq59rboPg2t4iiUMuzVuuPx4X4hvXFky5WA4iMJrMTD9kRVBfgZ1o2yJLr46KGrYSXyLr0If1vJUZ4rhlWkatSrTYszUZuIRDbYFrr0pbPKmF2bYeiYYDm99nDoxzk48nvV+hCAUlMI8Qm4ee5SdphHnQtVAY2e0AuIT3nV7CdSSP7T2NBfMiw/OC4IOWmEp6wE1+sADjdhBnTHUmVVHA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Cas9E3Jz2zOrncuZdVvfzFahgTeS8mCUq401jOp9ndU=; b=HZty8AkjMdN41x8Fw7iof0TqHQsEX21TBaeKk6qU3WPiLTIyR8Nxkwer1sHzZ/LqIsOvgtbN4zT3cRyuMPqeRK14Ci0EjcJ2+nAFRsxwtoihWh04QJhQlrsb+L/cKUAOEn7ZzCX9KyZAzij20W/5N3NV0NrB0316ltBEdhfby84= Authentication-Results: edk2.groups.io; dkim=none (message not signed) header.d=none;edk2.groups.io; dmarc=none action=none header.from=oracle.com; Received: from CY4PR10MB1718.namprd10.prod.outlook.com (2603:10b6:910:9::17) by CY4PR1001MB2246.namprd10.prod.outlook.com (2603:10b6:910:41::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3784.16; Fri, 29 Jan 2021 01:00:55 +0000 Received: from CY4PR10MB1718.namprd10.prod.outlook.com ([fe80::39dc:6ba7:9397:cb6]) by CY4PR10MB1718.namprd10.prod.outlook.com ([fe80::39dc:6ba7:9397:cb6%6]) with mapi id 15.20.3805.019; Fri, 29 Jan 2021 01:00:55 +0000 From: "Ankur Arora" To: devel@edk2.groups.io Cc: lersek@redhat.com, imammedo@redhat.com, boris.ostrovsky@oracle.com, Ankur Arora , Jordan Justen , Ard Biesheuvel , Aaron Young Subject: [PATCH v6 7/9] OvmfPkg/CpuHotplugSmm: add CpuEject() Date: Thu, 28 Jan 2021 16:59:48 -0800 Message-Id: <20210129005950.467638-8-ankur.a.arora@oracle.com> X-Mailer: git-send-email 2.29.2 In-Reply-To: <20210129005950.467638-1-ankur.a.arora@oracle.com> References: <20210129005950.467638-1-ankur.a.arora@oracle.com> X-Originating-IP: [148.87.23.8] X-ClientProxiedBy: MWHPR1401CA0022.namprd14.prod.outlook.com (2603:10b6:301:4b::32) To CY4PR10MB1718.namprd10.prod.outlook.com (2603:10b6:910:9::17) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from prion.us.oracle.com (148.87.23.8) by MWHPR1401CA0022.namprd14.prod.outlook.com (2603:10b6:301:4b::32) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3805.17 via Frontend Transport; Fri, 29 Jan 2021 01:00:05 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: fbc69340-ae6a-4d3d-5758-08d8c3f13809 X-MS-TrafficTypeDiagnostic: CY4PR1001MB2246: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:9508; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: rq5PwNupKrFG4myNxQzrfYwWPEgpeeuYVbOqkzQ5FK60xwTH71H/swzk/qak8MQgk83gaTr6ri8aNn+UojlrPPoNp6ik/8aFFvzxpieq3Q2tsOIgdwHbc9or2bWW+uT01/doO1s2iYleHiYAlib3oLjMN/ihkKsO6uvOLWIDI9agTeUzTi3xV1it+223ES+1gyceg4nNg3NMhpuO6HptPvW0Gn4LTVrOWvfNwgkIqf9RuSYAMzMEkNi3ak5zKPnP6YU6q2UZ29LIGU1iu3HU+hvB2pk2W/QdYP8ufLgLgWbqb3pRhKT7y5WbKJNwQGmhUfBN0ODAd5zI7Mv6es0SwYA2sZASutmgffIlFh6lHrglMOX5yR9VsI4Zf4vnKvLZPBqP3FdkMF6pOvoFHY0mrblJRIletOOIagYBTZ6mYtVbdbikZW8kqg/yROmJld0/vDGFXZVKcRNHW6iQR+WxgPb05KfPRxAmqDNgrlVGFtkHqq+5BkPStbJYPg/MpCncWFrRXJpPo5BMfoBMYuSg8jiFCpoS90v6fE7KM+82hgGCKynM71VRJYI8IqbNUTMv/Fojm6VAxZn0E1iyHOtTodxoEYyml3ni3uLuD9MAS8k= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CY4PR10MB1718.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(366004)(39860400002)(346002)(376002)(396003)(136003)(8936002)(107886003)(2906002)(66946007)(66476007)(66556008)(36756003)(1076003)(103116003)(16526019)(956004)(966005)(7696005)(478600001)(52116002)(54906003)(6486002)(86362001)(83380400001)(6666004)(8676002)(186003)(5660300002)(4326008)(6916009)(26005)(316002)(2616005);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: =?us-ascii?Q?wKIPZjpGV5qEnvfsJdlyzX/j7uCoxp3d36lKA4v/otgNuBwQDAi9x4yb9tbx?= =?us-ascii?Q?pKXyZfZ2qra8gHqxevgJwn7y9x41BILi6bZoEEo1OnRhsABF95wHsyWtE6Un?= =?us-ascii?Q?0YIlQuQPKYnol6va8GXPKsYRwoy3gVUUFT6aMwNC6kEw9Y1KphDPkdhSw3JK?= =?us-ascii?Q?y8FLVSEOdqD/Y5Y+A7LxYUKV5u0FsLLeo4NoPkerboKhK6/mqI8YYoJcdxwy?= =?us-ascii?Q?e6wvJaFVeVzEiYwk8uebGitpB1GUJ+bxze9tQs0XSP8zm4tPCwCVcFXDkjR2?= =?us-ascii?Q?eT7VhZxnN1gyvOLlrz6IoXGTqXwm236GgtMyr0l6JlmOhigbmMqK6ycTs8lb?= =?us-ascii?Q?UUlPuuopw+ECJGyazfNre9hxsnylOZMq2zn8h6E6+vfECiwunyVtIeP9eZ2Z?= =?us-ascii?Q?wZApIA/OPr8jGWHWDt7T7e7581AoU+kjLCPH3ON0o2wwWf4mNoY/LyFcRVBw?= =?us-ascii?Q?85Fme60SK2aKVv3dA0V0QIKQwtgUd/wiigzkG1fPsSSRv8eGeMILrpWmkU5i?= =?us-ascii?Q?bqXEr0AbpDYi1xtWo+LbAZ2sAJ2uP2gL1KsL7WZpblRi5CCEjDjjG40Egtzc?= =?us-ascii?Q?FcbiTJjCwVyg7gw1hjAZU5IHSYOuVN02tfKFQWPzCKoJDMuBJZqU2uThKqYv?= =?us-ascii?Q?2Jc52MGcWqRDX0Yjpt/NtcweP+2qujqO7rrfl3Nn2bHgnUkfCtHB4mX64GbM?= =?us-ascii?Q?Avvm5xybL8X2EBbPt4O8/zDLyf0Vg+Db7QDuJZVhq9vVz7fMsPAOlrIw5Rq+?= =?us-ascii?Q?ZUZzF0UMuNj58R/p3zA8VVy9/hA1oun49raCAjYcRH/Xvll5Zs6hHKkx29gv?= =?us-ascii?Q?AZAX+A+P2VOWg08rBcHZ9EemVMqMdjHhaaD5ZbzLm8EUofC6nbZPOe5tm7QX?= =?us-ascii?Q?JwHVysrgiyc8N1MLyfuW5aUpdaZCoB2TuXmuV6s+XuXKNlQHNKG49gBrDuG2?= =?us-ascii?Q?D67ab5a49iVFXN9T0ZXs/xLr1gOVZQ2WfzmBd33Qi4M/27Yxqjx2FT9QMixU?= =?us-ascii?Q?1OOzjyoQlHw3dQNojiS91I95Y9AT7wIY7e/UFHz47Tf333+oO8XyIO0IeI1I?= =?us-ascii?Q?Vw8P24jI?= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: fbc69340-ae6a-4d3d-5758-08d8c3f13809 X-MS-Exchange-CrossTenant-AuthSource: CY4PR10MB1718.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 29 Jan 2021 01:00:06.9995 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: fE8T43r/GGwJErxkZdmqc5J5Vb7wW9zkpLUcYW2R81E6d7b4lwaUXZlKzr1N8L+XAtSJszx/xTkO17kvw0F4tTywzB0vqvAJAi5dgPtvO68= X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY4PR1001MB2246 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9878 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxscore=0 spamscore=0 phishscore=0 adultscore=0 mlxlogscore=999 malwarescore=0 suspectscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2101290001 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9878 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 spamscore=0 phishscore=0 adultscore=0 impostorscore=0 malwarescore=0 lowpriorityscore=0 bulkscore=0 priorityscore=1501 mlxscore=0 clxscore=1015 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2101290001 Content-Transfer-Encoding: 8bit Content-Type: text/plain Add CpuEject(), which handles the CPU ejection, and provides a holding area for said CPUs. It is called via SmmCpuFeaturesRendezvousExit(), at the tail end of the SMI handling. Also UnplugCpus() now stashes APIC IDs of CPUs which need to be ejected in CPU_HOT_EJECT_DATA.ApicIdMap. These are used by CpuEject() to identify such CPUs. Cc: Laszlo Ersek Cc: Jordan Justen Cc: Ard Biesheuvel Cc: Igor Mammedov Cc: Boris Ostrovsky Cc: Aaron Young Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3132 Signed-off-by: Ankur Arora --- OvmfPkg/CpuHotplugSmm/CpuHotplug.c | 109 +++++++++++++++++++++++++++++++++++-- 1 file changed, 105 insertions(+), 4 deletions(-) diff --git a/OvmfPkg/CpuHotplugSmm/CpuHotplug.c b/OvmfPkg/CpuHotplugSmm/CpuHotplug.c index 70d69f6ed65b..526f51faf070 100644 --- a/OvmfPkg/CpuHotplugSmm/CpuHotplug.c +++ b/OvmfPkg/CpuHotplugSmm/CpuHotplug.c @@ -14,6 +14,7 @@ #include // gMmst #include // PcdGetBool() #include // SafeUintnSub() +#include // CPU_HOT_EJECT_DATA #include // EFI_MM_CPU_IO_PROTOCOL #include // EFI_SMM_CPU_SERVICE_PROTOCOL #include // EFI_STATUS @@ -32,11 +33,12 @@ STATIC EFI_MM_CPU_IO_PROTOCOL *mMmCpuIo; // STATIC EFI_SMM_CPU_SERVICE_PROTOCOL *mMmCpuService; // -// This structure is a communication side-channel between the +// These structures serve as communication side-channels between the // EFI_SMM_CPU_SERVICE_PROTOCOL consumer (i.e., this driver) and provider // (i.e., PiSmmCpuDxeSmm). // STATIC CPU_HOT_PLUG_DATA *mCpuHotPlugData; +STATIC CPU_HOT_EJECT_DATA *mCpuHotEjectData; // // SMRAM arrays for fetching the APIC IDs of processors with pending events (of // known event types), for the time of just one MMI. @@ -188,11 +190,53 @@ RevokeNewSlot: } /** + CPU Hot-eject handler, called from SmmCpuFeaturesRendezvousExit(), + on each CPU at exit from SMM. + + If, the executing CPU is not being ejected, nothing to be done. + If, the executing CPU is being ejected, wait in a CpuDeadLoop() + until ejected. + + @param[in] ProcessorNum Index of executing CPU. + +**/ +VOID +EFIAPI +CpuEject ( + IN UINTN ProcessorNum + ) +{ + // + // APIC ID is UINT32, but mCpuHotEjectData->ApicIdMap[] is UINT64 + // so use UINT64 throughout. + // + UINT64 ApicId; + + ApicId = mCpuHotEjectData->ApicIdMap[ProcessorNum]; + if (ApicId == CPU_EJECT_INVALID) { + return; + } + + // + // CPU(s) being unplugged get here from SmmCpuFeaturesSmiRendezvousExit() + // after having been cleared to exit the SMI by the monarch and thus have + // no SMM processing remaining. + // + // Given that we cannot allow them to escape to the guest, we pen them + // here until the SMM monarch tells the HW to unplug them. + // + CpuDeadLoop (); +} + +/** Process to be hot-unplugged CPUs, per QemuCpuhpCollectApicIds(). For each such CPU, report the CPU to PiSmmCpuDxeSmm via - EFI_SMM_CPU_SERVICE_PROTOCOL. If the to be hot-unplugged CPU is - unknown, skip it silently. + EFI_SMM_CPU_SERVICE_PROTOCOL and stash the APIC ID for later ejection. + If the to be hot-unplugged CPU is unknown, skip it silently. + + Additonally, if we do stash any APIC IDs, also install a CPU eject handler + which would handle the ejection. @param[in] ToUnplugApicIds The APIC IDs of the CPUs that are about to be hot-unplugged. @@ -216,9 +260,11 @@ UnplugCpus ( { EFI_STATUS Status; UINT32 ToUnplugIdx; + UINT32 EjectCount; UINTN ProcessorNum; ToUnplugIdx = 0; + EjectCount = 0; while (ToUnplugIdx < ToUnplugCount) { APIC_ID RemoveApicId; @@ -255,13 +301,41 @@ UnplugCpus ( DEBUG ((DEBUG_ERROR, "%a: RemoveProcessor(" FMT_APIC_ID "): %r\n", __FUNCTION__, RemoveApicId, Status)); goto Fatal; + } else { + // + // Stash the APIC IDs so we can do the actual ejection later. + // + if (mCpuHotEjectData->ApicIdMap[ProcessorNum] != CPU_EJECT_INVALID) { + // + // Since ProcessorNum and APIC-ID map 1-1, so a valid + // mCpuHotEjectData->ApicIdMap[ProcessorNum] means something + // is horribly wrong. + // + DEBUG ((DEBUG_ERROR, "%a: ProcessorNum %u maps to %llx, cannot " + "map to " FMT_APIC_ID "\n", __FUNCTION__, ProcessorNum, + mCpuHotEjectData->ApicIdMap[ProcessorNum], RemoveApicId)); + + Status = EFI_INVALID_PARAMETER; + goto Fatal; + } + + mCpuHotEjectData->ApicIdMap[ProcessorNum] = (UINT64)RemoveApicId; + EjectCount++; } ToUnplugIdx++; } + if (EjectCount != 0) { + // + // We have processors to be ejected; install the handler. + // + mCpuHotEjectData->Handler = CpuEject; + } + // - // We've removed this set of APIC IDs from SMM data structures. + // We've removed this set of APIC IDs from SMM data structures and + // have installed an ejection handler if needed. // return EFI_SUCCESS; @@ -458,7 +532,13 @@ CpuHotplugEntry ( // Our DEPEX on EFI_SMM_CPU_SERVICE_PROTOCOL guarantees that PiSmmCpuDxeSmm // has pointed PcdCpuHotPlugDataAddress to CPU_HOT_PLUG_DATA in SMRAM. // + // Additionally, CPU Hot-unplug is available only if CPU Hotplug is, so + // the same DEPEX also guarantees that PcdCpuHotEjectDataAddress points + // to CPU_HOT_EJECT_DATA in SMRAM. + // mCpuHotPlugData = (VOID *)(UINTN)PcdGet64 (PcdCpuHotPlugDataAddress); + mCpuHotEjectData = (VOID *)(UINTN)PcdGet64 (PcdCpuHotEjectDataAddress); + if (mCpuHotPlugData == NULL) { Status = EFI_NOT_FOUND; DEBUG ((DEBUG_ERROR, "%a: CPU_HOT_PLUG_DATA: %r\n", __FUNCTION__, Status)); @@ -470,6 +550,9 @@ CpuHotplugEntry ( if (mCpuHotPlugData->ArrayLength == 1) { return EFI_UNSUPPORTED; } + ASSERT (mCpuHotEjectData && + (mCpuHotPlugData->ArrayLength == mCpuHotEjectData->ArrayLength)); + // // Allocate the data structures that depend on the possible CPU count. // @@ -552,6 +635,24 @@ CpuHotplugEntry ( // SmbaseInstallFirstSmiHandler (); + if (mCpuHotEjectData) { + UINT32 Idx; + // + // For CPU ejection we need to map ProcessorNum -> APIC_ID. By the time + // we need the mapping, however, the Processor's APIC ID has already been + // removed from SMM data structures. So we will maintain a local map + // in mCpuHotEjectData->ApicIdMap. + // + for (Idx = 0; Idx < mCpuHotEjectData->ArrayLength; Idx++) { + mCpuHotEjectData->ApicIdMap[Idx] = CPU_EJECT_INVALID; + } + + // + // Wait to init the handler until an ejection is warranted + // + mCpuHotEjectData->Handler = NULL; + } + return EFI_SUCCESS; ReleasePostSmmPen: -- 2.9.3