From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from aserp2130.oracle.com (aserp2130.oracle.com [141.146.126.79]) by mx.groups.io with SMTP id smtpd.web09.2312.1612407470267386157 for ; Wed, 03 Feb 2021 18:57:50 -0800 Authentication-Results: mx.groups.io; dkim=fail reason="body hash did not verify" header.i=@oracle.com header.s=corp-2020-01-29 header.b=u31zDQ+S; spf=pass (domain: oracle.com, ip: 141.146.126.79, mailfrom: ankur.a.arora@oracle.com) Received: from pps.filterd (aserp2130.oracle.com [127.0.0.1]) by aserp2130.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 1142sYnr056149; Thu, 4 Feb 2021 02:57:46 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=subject : to : cc : references : from : message-id : date : in-reply-to : content-type : content-transfer-encoding : mime-version; s=corp-2020-01-29; bh=2+KM+LZ4oITAvlqlk0gNwyGPWndUe40qpMtjc9vIMm8=; b=u31zDQ+SAUtL2twU87E1e0OqQVUywraR7z16wncafFErvtRq5fRBRDBgtM/b3drGCGn6 lTFaUrjPVTiSNtBT9c9i+8lUvQR4ZFHbJBZ+jjYZxTouja8poGx8Mnh8U1xBunIgQNsB FsZvu+Z/7IkGJ9gaz/poqzIxl7w8RDLesrJbrEQ2bbGXh41H+4gOV91kwT4SP4fXZ9Ak nNwbo73wQXr8tNXrIjddJX9dtreC2MjFtvN99J5aClwWZo0U2Tj/L8hhCwrq/a0iyd68 1Ht6TqenO2aSVa6h+Zas3oZ+/NbVTLxm5FClL25aDPn5REcF1HAhcb6ecDfJ2hVDO/A7 Kw== Received: from userp3030.oracle.com (userp3030.oracle.com [156.151.31.80]) by aserp2130.oracle.com with ESMTP id 36cvyb3be7-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 04 Feb 2021 02:57:46 +0000 Received: from pps.filterd (userp3030.oracle.com [127.0.0.1]) by userp3030.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 1142sdfF158993; Thu, 4 Feb 2021 02:57:45 GMT Received: from nam11-co1-obe.outbound.protection.outlook.com (mail-co1nam11lp2177.outbound.protection.outlook.com [104.47.56.177]) by userp3030.oracle.com with ESMTP id 36dhd0pvr5-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Thu, 04 Feb 2021 02:57:45 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=m9ufCEd4Hb/gWiaTmoQvt2cTjS2XTc/UK6CaG92O9EokOKbo6X+JBFyHJsYBAzos5TNX9eH0YkXo6wGId6kmabD4uwiCO3H+hnBQayED3nj7n+nQYnqtz7IXw/SZEO3txYvB48f/JOBvi8gM+ngoeSsZt5Oa/xomdoSu7XVNhClL//28qsfqH9kRxHaOYn5uBN2aF0XRtwCYuZhtkiAhXLc+6CZgzxIuV8CFi7xzQxMEBEriWplVT5Rg7aDG7yjjesJ8lS4/YEm1H6WCGIsgk38XgB640h5MiGc07rG5LyVYllRWA7A9TMPLsuCqNCkt7HWUmy9E05BTGPXTEWFXrA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=2+KM+LZ4oITAvlqlk0gNwyGPWndUe40qpMtjc9vIMm8=; b=SK9uiHhEXXuXBgGEl8N6I5WyzTtlznQunkzhLQoi5roWKxq4zXXl34JqqjTcIo49UxwPArzNprCGrfWixcvlvo9dVb0f7opN+sKUZ79C3vddSpZuQhj0kLcVu2H3wIp1GFXU4J3joXj7XABVZ4hrTaKMgixdKwUGl6HuXrMV5S0GWeQQiOIT7+M9oYSJtSO3pnsdZ2glge20g0foCManObreihmP1ivnaq5TnEO4InWovVD26ggVTxcB/f8Ep07PGb9QdMA+D5Z7uHWxmlZHacVb3dGNTPMSvfLb/q4kURUsbD2t6qYCaa3jZdkwdK3W8QMybGw8y+FiEPtkXS/q2Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=2+KM+LZ4oITAvlqlk0gNwyGPWndUe40qpMtjc9vIMm8=; b=D7/VkkNWknumG4anwCxpaGnB9izf3DhxAEotUElDFaBjJes+tHmsHk6Ipx6aNJ1ghIu7pIePwFTuF5XEhn4Qo3Pyv3B6nRosp0P3PSrrSSDngl9975Ub9lHZr4FF4NKBcGYTYuO8bVisRzmEI0KYkjDBr+U9c4MN08nMhbMd1uY= Received: from SJ0PR10MB4605.namprd10.prod.outlook.com (2603:10b6:a03:2d9::24) by BYAPR10MB3607.namprd10.prod.outlook.com (2603:10b6:a03:121::24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3805.24; Thu, 4 Feb 2021 02:57:43 +0000 Received: from SJ0PR10MB4605.namprd10.prod.outlook.com ([fe80::b1e9:811d:aa8e:f62a]) by SJ0PR10MB4605.namprd10.prod.outlook.com ([fe80::b1e9:811d:aa8e:f62a%6]) with mapi id 15.20.3805.028; Thu, 4 Feb 2021 02:57:43 +0000 Subject: Re: [PATCH v6 7/9] OvmfPkg/CpuHotplugSmm: add CpuEject() To: Laszlo Ersek , devel@edk2.groups.io Cc: imammedo@redhat.com, boris.ostrovsky@oracle.com, Jordan Justen , Ard Biesheuvel , Aaron Young References: <20210129005950.467638-1-ankur.a.arora@oracle.com> <20210129005950.467638-8-ankur.a.arora@oracle.com> <14068eb6-dc43-c244-5985-709d685fc750@oracle.com> <3fc5ff97-6ea9-e943-523f-9a7462072c59@oracle.com> <21c215e8-b21c-7642-4bbc-b3538f94e513@redhat.com> From: "Ankur Arora" Message-ID: <5d5b0e1a-bb68-4130-db53-b5510845838e@oracle.com> Date: Wed, 3 Feb 2021 18:57:40 -0800 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.6.0 In-Reply-To: <21c215e8-b21c-7642-4bbc-b3538f94e513@redhat.com> X-Originating-IP: [148.87.23.5] X-ClientProxiedBy: MW4PR04CA0427.namprd04.prod.outlook.com (2603:10b6:303:8b::12) To SJ0PR10MB4605.namprd10.prod.outlook.com (2603:10b6:a03:2d9::24) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from [192.168.0.108] (148.87.23.5) by MW4PR04CA0427.namprd04.prod.outlook.com (2603:10b6:303:8b::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3825.19 via Frontend Transport; Thu, 4 Feb 2021 02:57:42 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: afa5759f-3c97-4fb6-f563-08d8c8b8a46e X-MS-TrafficTypeDiagnostic: BYAPR10MB3607: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:8882; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: BcaevD7KZf8OUfACXsfW8Uqi2wKjYrvKTZ3WNed6f2J3m9U0e+RxZTL3VkmBPpmJi60wBqOy08NRxcvh1jPJRzbdnDGxd3HyOQrMNLaoAfpXOlTQIC72VQUNOsFCZ1aVKQbin/JSzFaMtnbSdUK2VKRsyoPuD1c3GNbbHeBk1oa81pdsrpdUvjSSoZRm4c+8BQIy5S2BUWeO+8nE2JBssnIaiskdDB4GpVnkUG6K8W3xrihGlfEplO5xQThU/25oGHvIp9kyC91b6iYH0tOxI+KydSo6BAYWeF5d293YGhvsqWdZfl609s9e/BMTFFPXIyT971xaZFjzgCH3tZkXh5Nn2iMTGqvoUzsgR2JedbtwDm11EoC1PHS8MhjOvW2GpNSu5I7A0/tMsUHM5ynYy1+Q5Zq05OWfjeCuX+PgEUhh3q/lOG46D0QMBS60WOUs6bPseJ/u0Tp/G/ROVUKKMGi97Q9L/jBiUBc3dA5xprWjyGuHROypkBNLgqYUGUj2B1o/4mYzlmYCEmsEkVUZdBV3LrVVgQgoCX76JvW4GoLtqJnfuezNfbNlX8jS5hdJh8uQnp61m/tN/fBcSJqq+NEKGSkysi8vofsnqdDQ8j4= X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:SJ0PR10MB4605.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(366004)(39860400002)(136003)(346002)(376002)(396003)(5660300002)(86362001)(53546011)(8676002)(26005)(186003)(16526019)(478600001)(83380400001)(36756003)(4326008)(956004)(31696002)(6486002)(2616005)(107886003)(316002)(8936002)(66946007)(66556008)(66476007)(54906003)(2906002)(31686004)(16576012)(45980500001)(43740500002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: =?utf-8?B?bnczN0RuaU1jdVk1TjNPZTFHQnFMM2xaWmRDczBEa3ROcW9mak9iWE8zdmNn?= =?utf-8?B?Y3p5WTh2ZEVxRVdOM1FMY1JRN0pSb2c5RmZubGZNTllNR25SKy9hUkJ3YkYr?= =?utf-8?B?amZGNXMwTXdncCtvcE1JakIyTmhDNzhvRVhPQXFzaFhvdUF3VmIvcXk5WmF6?= =?utf-8?B?VDZvVUFPV21QY2xXZWdzMmcwWFlna1EyU1IwRmNvSVpOck11cFNoYW5CM1Bu?= =?utf-8?B?dTA5QkFpL0lDQWNIb0I4MC8zZ2tnaFVjZXh5T3hhZ0RqV3M0L2lJODR3T0or?= =?utf-8?B?UjhkSnFRR1FMKzdpeGk1ZWlEcHREM0JId3BBdVpXdnl6RjkvYmUvYUliYXYz?= =?utf-8?B?eklOOHZuVkpWZHRVVUJ1OSt1NWMvSVVtNmhPOUxlVE9mVWNNZE1rSVdFdG5F?= =?utf-8?B?eWJJdTJzYlBVL2xwWXhjb1VFZXJjY2Q2UHBHQnljQWYxaG5TNVRrL1U0YjNE?= =?utf-8?B?VlV2Rzl2UmZGM3IvVzEvUEdJb05yeHllWTNJM0YzRmRPUWFZVmFNWE1vUVBo?= =?utf-8?B?T0dHWVk4d3RleGhBZUpTT2hDQWNPWFhVYWZCck5MaFl3RmVzUU9mYVRHK0pT?= =?utf-8?B?U085cFBmVElRK0V1M2dkODhiV1hZa3JKZmdhS3JmTFJWMGppK1YzckZaWDlC?= =?utf-8?B?emFxL21JN2gyM3d4YVc2VFkwUG92ZU90aXdvTE84UEI4SGRObEg0SCszT1Ra?= =?utf-8?B?cHA3cXFBWnpYTVRKSzdvSEtLWCtGMXlaQmk0dlAzcWgvYzh3ejR3VjRka1lo?= =?utf-8?B?QmptdnpHb2ZpVGdPWnZiSzBaRHpnV3FRMzhWZkUyVSs2RkxqT3BhRlAzdVg4?= =?utf-8?B?WGpSUDY3MjJXdzJwazNKMUZ5TDlSaUlJN0hzUWFuQW5EaXhJenBRRDVLZ2JS?= =?utf-8?B?bW53Y0dEV1ZUMlhXQXlYUUlXQWNtWkpEbWZEM3dkOUlpcE9YekU3dmR5QXBU?= =?utf-8?B?TWVUWVU2TzZQeE8zejl0bXdqWUREYlBkTEpvMkErcThTdUZrYUZFVzBkRkFv?= =?utf-8?B?aFBYcXEzendiY0pDZkluSjVlUjA0amtkUDNJcTBrM0pYMDNxVlpGbTh1NzNN?= =?utf-8?B?Z3FEN2g5N0R3Zk9TV2hFQnJvNTRaN1JlU1REVDEvN2srZ1d1N3NxUlEyTllz?= =?utf-8?B?aW1tMzNlRTdsaCsvcmg3T095RTByR0dZMnhRVFFDby9QeDFoVU9vZFB5NGl1?= =?utf-8?B?MzZDdmZiVDROQlJMenFOTEZuMXRaamhsV0JhZDc3eU4rbEQzMWRiQzdVUGxL?= =?utf-8?B?MzVNd1pBUnZjVGhjeTF0SWFOdWdmb0FtNDhLZW9tdXJ3bUJRK0JjLzdOYnBE?= =?utf-8?B?VC9wWWlqUHU5KzRnUld5VDRya1Z4bThVeENla2s1YUk4djQyV1krUjNvRXRJ?= =?utf-8?B?VUhLOXpVcTZPd2lFMi9VMllzR2Q3Z3Y1RUU3SzBQMFhRYkEyQU5OdCtrdm5x?= =?utf-8?Q?Y0cf7MUV?= X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: afa5759f-3c97-4fb6-f563-08d8c8b8a46e X-MS-Exchange-CrossTenant-AuthSource: SJ0PR10MB4605.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 04 Feb 2021 02:57:43.2522 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: YKfzyVRecTixr6goClyUet4ZQ8jVH8uT2rgRxAslAFnZop4QmCdCSgVqF2v/xuXZphRcHKGNI6gLm4FFgvzJKVKGiE51Tk20viOkVZEUCgc= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR10MB3607 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9884 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 mlxscore=0 mlxlogscore=999 phishscore=0 spamscore=0 suspectscore=0 malwarescore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2102040018 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9884 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 clxscore=1015 impostorscore=0 mlxscore=0 spamscore=0 bulkscore=0 priorityscore=1501 adultscore=0 lowpriorityscore=0 malwarescore=0 phishscore=0 mlxlogscore=999 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2102040018 X-MIME-Autoconverted: from 8bit to quoted-printable by aserp2130.oracle.com id 1142sYnr056149 Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: quoted-printable On 2021-02-03 12:55 p.m., Laszlo Ersek wrote: > On 02/03/21 07:13, Ankur Arora wrote: >> On 2021-02-02 6:00 a.m., Laszlo Ersek wrote: >>> On 02/01/21 21:12, Ankur Arora wrote: >>>> On 2021-02-01 11:08 a.m., Laszlo Ersek wrote: >=20 >>>>> (16) This function uses a data structure for communication between = BSP >>>>> and APs -- mCpuHotEjectData->ApicIdMap is modified in UnplugCpus() = on >>>>> the BSP, and checked above by the APs (too). >>>>> >>>>> What guarantees the visibility of mCpuHotEjectData->ApicIdMap? >>>> >>>> I was banking on SmiRendezvous() explicitly signalling that all >>>> processing on the BSP was done before any AP will look at >>>> mCpuHotEjectData in SmmCpuFeaturesRendezvousExit(). >>>> >>>> 1716=C2=A0=C2=A0=C2=A0=C2=A0 // >>>> 1717=C2=A0=C2=A0=C2=A0=C2=A0 // Wait for BSP's signal to exit SMI >>>> 1718=C2=A0=C2=A0=C2=A0=C2=A0 // >>>> 1719=C2=A0=C2=A0=C2=A0=C2=A0 while (*mSmmMpSyncData->AllCpusInSync) = { >>>> 1720=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 CpuPause (); >>>> 1721=C2=A0=C2=A0=C2=A0=C2=A0 } >>>> 1722=C2=A0=C2=A0 } >>>> 1723 >>>> 1724 Exit: >>>> 1725=C2=A0=C2=A0 SmmCpuFeaturesRendezvousExit (CpuIndex); >>> >>> Right; it's a general pattern in edk2: volatile UINT8 (aka BOOLEAN) >>> objects are considered atomic. (See >>> SMM_DISPATCHER_MP_SYNC_DATA.AllCpusInSync -- it's a pointer to a >>> volatile BOOLEAN.) >>> >>> But our UINT64 values are neither volatile nor UINT8, and I got sudde= nly >>> doubtful about "AllCpusInSync" working as a multiprocessor barrier. >>> >>> (I could be unjustifiedly worried, as a bunch of other fields in >>> SMM_DISPATCHER_MP_SYNC_DATA are volatile, wider than UINT8, and *not* >>> accessed with InterlockedCompareExchageXx().) >> >> Thanks for pointing me to this code. There's a curious comment in >> about making this structure uncache-able in the declaration here >> (though I couldn't figure out how that is done): >> >> 418 typedef struct { >> 419=C2=A0=C2=A0 // >> 420=C2=A0=C2=A0 // Pointer to an array. The array should be located im= mediately >> after this structure >> 421=C2=A0=C2=A0 // so that UC cache-ability can be set together. >> 422=C2=A0=C2=A0 // >=20 > This is probably through SMRR manipulation. >=20 > The "UefiCpuPkg/Library/SmmCpuFeaturesLib" instance contains SMRR suppo= rt. >=20 > The "OvmfPkg/Library/SmmCpuFeaturesLib" instance contains no SMRR > support. (Just search both source files for "SMRR".) >=20 Oh, now I see what SMRR does. Thanks that helps make sense of what's going on here. Ankur >=20 >> 423=C2=A0=C2=A0 SMM_CPU_DATA_BLOCK=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 *CpuData; >> 424=C2=A0=C2=A0 volatile UINT32=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 *Counter; >> 425=C2=A0=C2=A0 volatile UINT32=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 BspIndex; >> 426=C2=A0=C2=A0 volatile BOOLEAN=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 *InsideSmm; >> 427=C2=A0=C2=A0 volatile BOOLEAN=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 *AllCpusInSync; >> 428=C2=A0=C2=A0 volatile SMM_CPU_SYNC_MODE=C2=A0=C2=A0=C2=A0 Effective= SyncMode; >> 429=C2=A0=C2=A0 volatile BOOLEAN=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 SwitchBsp; >> 430=C2=A0=C2=A0 volatile BOOLEAN=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 *CandidateBsp; >> 431=C2=A0=C2=A0 EFI_AP_PROCEDURE=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 StartupProcedure; >> 432=C2=A0=C2=A0 VOID=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0 *StartupProcArgs; >> 433 } SMM_DISPATCHER_MP_SYNC_DATA; >> >> Also, is there an expectation that these fields (at least some of >> them) switch over when a new leader is chosen? >=20 > Yes, see for example the "Elect BSP" section in SmiRendezvous(). >=20 >=20 >> Otherwise I'm not sure why for instance, AllCpusInSync would be >> a pointer. >=20 > TBH I can't explain that; I'm not too familiar with those parts... >=20 >=20 >>>> CpuEject(): >>>> 218=C2=A0=C2=A0 ApicId =3D mCpuHotEjectData->ApicIdMap[ProcessorNum]= ; >>>> >>>> For the to-be-ejected-AP, this value can only move from >>>> =C2=A0=C2=A0=C2=A0 valid-APIC-ID (=3D> wait in CpuDeadLoop()) -> CP= U_EJECT_INVALID. >>>> >>>> Given that, by the time the worker does the write on line 254, this >>>> AP is guaranteed to be dead already, I don't think there's any >>>> scenario where the to-be-ejected-AP can see anything other than >>>> a valid-APIC-ID. >>> >>> The scenario I had in mind was different: what guarantees that the >>> effect of >>> >>> =C2=A0=C2=A0=C2=A0 375=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 mCp= uHotEjectData->ApicIdMap[ProcessorNum] =3D >>> (UINT64)RemoveApicId; >>> >>> which is performed by the BSP in UnplugCpus(), is visible by the AP o= n >>> line 218 (see your quote above)? >>> >>> What if the AP gets to line 218 before the BSP's write on line 375 >>> *propagates* sufficiently? >> >> I understand. That does make sense. And, as you said elsewhere, a real >> memory fence would come in useful here. >> >> We could use AsmCpuid() as a poor man's mfence, but that seems overkil= l >> given that x86 at least guarantees store-order. >=20 > Right -- I don't recall any examples of AsmCpuid() being used like that > in edk2. >=20 > Thanks! > Laszlo >=20