Re: [PATCH v6 7/9] OvmfPkg/CpuHotplugSmm: add CpuEject()

["Laszlo Ersek" <lersek@redhat.com>]

On 02/03/21 07:13, Ankur Arora wrote:
> On 2021-02-02 6:00 a.m., Laszlo Ersek wrote:
>> On 02/01/21 21:12, Ankur Arora wrote:
>>> On 2021-02-01 11:08 a.m., Laszlo Ersek wrote:

>>>> (16) This function uses a data structure for communication between BSP
>>>> and APs -- mCpuHotEjectData->ApicIdMap is modified in UnplugCpus() on
>>>> the BSP, and checked above by the APs (too).
>>>>
>>>> What guarantees the visibility of mCpuHotEjectData->ApicIdMap?
>>>
>>> I was banking on SmiRendezvous() explicitly signalling that all
>>> processing on the BSP was done before any AP will look at
>>> mCpuHotEjectData in SmmCpuFeaturesRendezvousExit().
>>>
>>> 1716     //
>>> 1717     // Wait for BSP's signal to exit SMI
>>> 1718     //
>>> 1719     while (*mSmmMpSyncData->AllCpusInSync) {
>>> 1720       CpuPause ();
>>> 1721     }
>>> 1722   }
>>> 1723
>>> 1724 Exit:
>>> 1725   SmmCpuFeaturesRendezvousExit (CpuIndex);
>>
>> Right; it's a general pattern in edk2: volatile UINT8 (aka BOOLEAN)
>> objects are considered atomic. (See
>> SMM_DISPATCHER_MP_SYNC_DATA.AllCpusInSync -- it's a pointer to a
>> volatile BOOLEAN.)
>>
>> But our UINT64 values are neither volatile nor UINT8, and I got suddenly
>> doubtful about "AllCpusInSync" working as a multiprocessor barrier.
>>
>> (I could be unjustifiedly worried, as a bunch of other fields in
>> SMM_DISPATCHER_MP_SYNC_DATA are volatile, wider than UINT8, and *not*
>> accessed with InterlockedCompareExchageXx().)
> 
> Thanks for pointing me to this code. There's a curious comment in
> about making this structure uncache-able in the declaration here
> (though I couldn't figure out how that is done):
> 
> 418 typedef struct {
> 419   //
> 420   // Pointer to an array. The array should be located immediately
> after this structure
> 421   // so that UC cache-ability can be set together.
> 422   //

This is probably through SMRR manipulation.

The "UefiCpuPkg/Library/SmmCpuFeaturesLib" instance contains SMRR support.

The "OvmfPkg/Library/SmmCpuFeaturesLib" instance contains no SMRR
support. (Just search both source files for "SMRR".)


> 423   SMM_CPU_DATA_BLOCK            *CpuData;
> 424   volatile UINT32               *Counter;
> 425   volatile UINT32               BspIndex;
> 426   volatile BOOLEAN              *InsideSmm;
> 427   volatile BOOLEAN              *AllCpusInSync;
> 428   volatile SMM_CPU_SYNC_MODE    EffectiveSyncMode;
> 429   volatile BOOLEAN              SwitchBsp;
> 430   volatile BOOLEAN              *CandidateBsp;
> 431   EFI_AP_PROCEDURE              StartupProcedure;
> 432   VOID                          *StartupProcArgs;
> 433 } SMM_DISPATCHER_MP_SYNC_DATA;
> 
> Also, is there an expectation that these fields (at least some of
> them) switch over when a new leader is chosen?

Yes, see for example the "Elect BSP" section in SmiRendezvous().


> Otherwise I'm not sure why for instance, AllCpusInSync would be
> a pointer.

TBH I can't explain that; I'm not too familiar with those parts...


>>> CpuEject():
>>> 218   ApicId = mCpuHotEjectData->ApicIdMap[ProcessorNum];
>>>
>>> For the to-be-ejected-AP, this value can only move from
>>>     valid-APIC-ID (=> wait in CpuDeadLoop()) -> CPU_EJECT_INVALID.
>>>
>>> Given that, by the time the worker does the write on line 254, this
>>> AP is guaranteed to be dead already, I don't think there's any
>>> scenario where the to-be-ejected-AP can see anything other than
>>> a valid-APIC-ID.
>>
>> The scenario I had in mind was different: what guarantees that the
>> effect of
>>
>>     375        mCpuHotEjectData->ApicIdMap[ProcessorNum] =
>> (UINT64)RemoveApicId;
>>
>> which is performed by the BSP in UnplugCpus(), is visible by the AP on
>> line 218 (see your quote above)?
>>
>> What if the AP gets to line 218 before the BSP's write on line 375
>> *propagates* sufficiently?
> 
> I understand. That does make sense. And, as you said elsewhere, a real
> memory fence would come in useful here.
> 
> We could use AsmCpuid() as a poor man's mfence, but that seems overkill
> given that x86 at least guarantees store-order.

Right -- I don't recall any examples of AsmCpuid() being used like that
in edk2.

Thanks!
Laszlo