Re: [edk2-devel] [PATCH] UefiCpuPkg/PiSmmCpuDxeSmm: pause in WaitForSemaphore() before re-fetch

["Laszlo Ersek" <lersek@redhat.com>]

On 07/31/20 03:10, Dong, Eric wrote:
> Reviewed-by: Eric Dong <eric.dong@intel.com>

Thank you, merged as commit 9001b750df64, via
<https://github.com/tianocore/edk2/pull/843>.

Laszlo

>> -----Original Message-----
>> From: Laszlo Ersek <lersek@redhat.com>
>> Sent: Thursday, July 30, 2020 2:52 AM
>> To: edk2-devel-groups-io <devel@edk2.groups.io>
>> Cc: Dong, Eric <eric.dong@intel.com>; Philippe Mathieu-Daudé
>> <philmd@redhat.com>; Kumar, Rahul1 <rahul1.kumar@intel.com>; Ni, Ray
>> <ray.ni@intel.com>
>> Subject: [PATCH] UefiCpuPkg/PiSmmCpuDxeSmm: pause in
>> WaitForSemaphore() before re-fetch
>>
>> Most busy waits (spinlocks) in
>> "UefiCpuPkg/PiSmmCpuDxeSmm/MpService.c"
>> already call CpuPause() in their loop bodies; see SmmWaitForApArrival(),
>> APHandler(), and SmiRendezvous(). However, the "main wait" within
>> APHandler():
>>
>>>     //
>>>     // Wait for something to happen
>>>     //
>>>     WaitForSemaphore (mSmmMpSyncData->CpuData[CpuIndex].Run);
>>
>> doesn't do so, as WaitForSemaphore() keeps trying to acquire the
>> semaphore without pausing.
>>
>> The performance impact is especially notable in QEMU/KVM + OVMF
>> virtualization with CPU overcommit (that is, when the guest has significantly
>> more VCPUs than the host has physical CPUs). The guest BSP is working
>> heavily in:
>>
>>   BSPHandler()                  [MpService.c]
>>     PerformRemainingTasks()     [PiSmmCpuDxeSmm.c]
>>       SetUefiMemMapAttributes() [SmmCpuMemoryManagement.c]
>>
>> while the many guest APs are spinning in the "Wait for something to happen"
>> semaphore acquisition, in APHandler(). The guest APs are generating useless
>> memory traffic and saturating host CPUs, hindering the guest BSP's progress
>> in SetUefiMemMapAttributes().
>>
>> Rework the loop in WaitForSemaphore(): call CpuPause() in every iteration
>> after the first check fails. Due to Pause Loop Exiting (known as Pause Filter on
>> AMD), the host scheduler can favor the guest BSP over the guest APs.
>>
>> Running a 16 GB RAM + 512 VCPU guest on a 448 PCPU host, this patch
>> reduces OVMF boot time (counted until reaching grub) from 20-30 minutes
>> to less than 4 minutes.
>>
>> The patch should benefit physical machines as well -- according to the Intel
>> SDM, PAUSE "Improves the performance of spin-wait loops". Adding PAUSE
>> to the generic WaitForSemaphore() function is considered a general
>> improvement.
>>
>> Cc: Eric Dong <eric.dong@intel.com>
>> Cc: Philippe Mathieu-Daudé <philmd@redhat.com>
>> Cc: Rahul Kumar <rahul1.kumar@intel.com>
>> Cc: Ray Ni <ray.ni@intel.com>
>> Ref: https://bugzilla.redhat.com/show_bug.cgi?id=1861718
>> Signed-off-by: Laszlo Ersek <lersek@redhat.com>
>> ---
>>
>> Notes:
>>     Repo:   https://pagure.io/lersek/edk2.git
>>     Branch: sem_wait_pause_rhbz1861718
>>
>>  UefiCpuPkg/PiSmmCpuDxeSmm/MpService.c | 18 +++++++++++-------
>>  1 file changed, 11 insertions(+), 7 deletions(-)
>>
>> diff --git a/UefiCpuPkg/PiSmmCpuDxeSmm/MpService.c
>> b/UefiCpuPkg/PiSmmCpuDxeSmm/MpService.c
>> index 57e788c01b1f..4bcd217917d7 100644
>> --- a/UefiCpuPkg/PiSmmCpuDxeSmm/MpService.c
>> +++ b/UefiCpuPkg/PiSmmCpuDxeSmm/MpService.c
>> @@ -40,14 +40,18 @@ WaitForSemaphore (
>>  {
>>    UINT32                            Value;
>>
>> -  do {
>> +  for (;;) {
>>      Value = *Sem;
>> -  } while (Value == 0 ||
>> -           InterlockedCompareExchange32 (
>> -             (UINT32*)Sem,
>> -             Value,
>> -             Value - 1
>> -             ) != Value);
>> +    if (Value != 0 &&
>> +        InterlockedCompareExchange32 (
>> +          (UINT32*)Sem,
>> +          Value,
>> +          Value - 1
>> +          ) == Value) {
>> +      break;
>> +    }
>> +    CpuPause ();
>> +  }
>>    return Value - 1;
>>  }
>>
>> --
>> 2.19.1.3.g30247aa5d201
> 
> 
>