[PATCH v2 0/2] Fix the issue that OS complains memory < 1MB changes across S3

[PATCH v2 0/2] Fix the issue that OS complains memory < 1MB changes across S3

[Ni, Ray]

v2:
  Answer Laszlo's comments:
    1. Add 1/2 patch to change IA32 waking up vector assembly code to increase
       NumApsExecuting only for ApInitConfig.
    2. Change 2/2 patch to remove the unnecessary debug messages.

Ray Ni (2):
  UefiCpuPkg/MpInitLib: increase NumApsExecuting only for ApInitConfig
  UefiCpuPkg/MpInitLib: Decrease NumApsExecuting only for ApInitConfig

 UefiCpuPkg/Library/MpInitLib/Ia32/MpFuncs.nasm | 14 +++++++-------
 UefiCpuPkg/Library/MpInitLib/MpLib.c           |  5 +++--
 2 files changed, 10 insertions(+), 9 deletions(-)

-- 
2.21.0.windows.1

[PATCH v2 1/2] UefiCpuPkg/MpInitLib: increase NumApsExecuting only for ApInitConfig

[Ni, Ray]

NumApsExecuting is only used when InitFlag == ApInitConfig for
counting the processor count.

The patch changes Ia32 version of waking up vector assembly code
to align to x64 version of waking up vector assembly code.
After the change both versions of waking up vector increase
NumApsExecuting when InitFlag == ApInitConfig.

Signed-off-by: Ray Ni <ray.ni@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Cc: Eric Dong <eric.dong@intel.com>
---
 UefiCpuPkg/Library/MpInitLib/Ia32/MpFuncs.nasm | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)

diff --git a/UefiCpuPkg/Library/MpInitLib/Ia32/MpFuncs.nasm b/UefiCpuPkg/Library/MpInitLib/Ia32/MpFuncs.nasm
index 34b8705adb..b74046b76a 100644
--- a/UefiCpuPkg/Library/MpInitLib/Ia32/MpFuncs.nasm
+++ b/UefiCpuPkg/Library/MpInitLib/Ia32/MpFuncs.nasm
@@ -1,5 +1,5 @@
 ;------------------------------------------------------------------------------ ;
-; Copyright (c) 2015 - 2018, Intel Corporation. All rights reserved.<BR>
+; Copyright (c) 2015 - 2019, Intel Corporation. All rights reserved.<BR>
 ; SPDX-License-Identifier: BSD-2-Clause-Patent
 ;
 ; Module Name:
@@ -81,12 +81,6 @@ Flat32Start:                                   ; protected mode entry point
 
     mov        esi, ebx
 
-    ; Increment the number of APs executing here as early as possible
-    ; This is decremented in C code when AP is finished executing
-    mov        edi, esi
-    add        edi, NumApsExecutingLocation
-    lock inc   dword [edi]
-
     mov         edi, esi
     add         edi, EnableExecuteDisableLocation
     cmp         byte [edi], 0
@@ -120,6 +114,12 @@ SkipEnableExecuteDisable:
     cmp        dword [edi], 1       ; 1 == ApInitConfig
     jnz        GetApicId
 
+    ; Increment the number of APs executing here as early as possible
+    ; This is decremented in C code when AP is finished executing
+    mov        edi, esi
+    add        edi, NumApsExecutingLocation
+    lock inc   dword [edi]
+
     ; AP init
     mov        edi, esi
     add        edi, LockLocation
-- 
2.21.0.windows.1

Re: [PATCH v2 1/2] UefiCpuPkg/MpInitLib: increase NumApsExecuting only for ApInitConfig

[Dong, Eric]

Reviewed-by: Eric Dong <eric.dong@intel.com>

> -----Original Message-----
> From: Ni, Ray
> Sent: Wednesday, June 5, 2019 1:49 PM
> To: devel@edk2.groups.io
> Cc: Laszlo Ersek <lersek@redhat.com>; Dong, Eric <eric.dong@intel.com>
> Subject: [PATCH v2 1/2] UefiCpuPkg/MpInitLib: increase NumApsExecuting
> only for ApInitConfig
> 
> NumApsExecuting is only used when InitFlag == ApInitConfig for counting the
> processor count.
> 
> The patch changes Ia32 version of waking up vector assembly code to align to
> x64 version of waking up vector assembly code.
> After the change both versions of waking up vector increase
> NumApsExecuting when InitFlag == ApInitConfig.
> 
> Signed-off-by: Ray Ni <ray.ni@intel.com>
> Cc: Laszlo Ersek <lersek@redhat.com>
> Cc: Eric Dong <eric.dong@intel.com>
> ---
>  UefiCpuPkg/Library/MpInitLib/Ia32/MpFuncs.nasm | 14 +++++++-------
>  1 file changed, 7 insertions(+), 7 deletions(-)
> 
> diff --git a/UefiCpuPkg/Library/MpInitLib/Ia32/MpFuncs.nasm
> b/UefiCpuPkg/Library/MpInitLib/Ia32/MpFuncs.nasm
> index 34b8705adb..b74046b76a 100644
> --- a/UefiCpuPkg/Library/MpInitLib/Ia32/MpFuncs.nasm
> +++ b/UefiCpuPkg/Library/MpInitLib/Ia32/MpFuncs.nasm
> @@ -1,5 +1,5 @@
>  ;------------------------------------------------------------------------------ ; -;
> Copyright (c) 2015 - 2018, Intel Corporation. All rights reserved.<BR>
> +; Copyright (c) 2015 - 2019, Intel Corporation. All rights
> +reserved.<BR>
>  ; SPDX-License-Identifier: BSD-2-Clause-Patent  ;  ; Module Name:
> @@ -81,12 +81,6 @@ Flat32Start:                                   ; protected mode entry
> point
> 
>      mov        esi, ebx
> 
> -    ; Increment the number of APs executing here as early as possible
> -    ; This is decremented in C code when AP is finished executing
> -    mov        edi, esi
> -    add        edi, NumApsExecutingLocation
> -    lock inc   dword [edi]
> -
>      mov         edi, esi
>      add         edi, EnableExecuteDisableLocation
>      cmp         byte [edi], 0
> @@ -120,6 +114,12 @@ SkipEnableExecuteDisable:
>      cmp        dword [edi], 1       ; 1 == ApInitConfig
>      jnz        GetApicId
> 
> +    ; Increment the number of APs executing here as early as possible
> +    ; This is decremented in C code when AP is finished executing
> +    mov        edi, esi
> +    add        edi, NumApsExecutingLocation
> +    lock inc   dword [edi]
> +
>      ; AP init
>      mov        edi, esi
>      add        edi, LockLocation
> --
> 2.21.0.windows.1

[PATCH v2 2/2] UefiCpuPkg/MpInitLib: Decrease NumApsExecuting only for ApInitConfig

[Ni, Ray]

The patch fixes the bug that the memory under 1MB is modified by
firmware in S3 boot.

Root cause is a racing condition in MpInitLib:
1. BSP: WakeUpByInitSipiSipi is set by NotifyOnS3SmmInitDonePpi()
2. BSP: WakeUpAP() wakes all APs to run certain procedure.
  2.1. AllocateResetVector() uses <1MB memory for wake up vector.
  2.1. FillExchangeInfoData() resets NumApsExecuting to 0.
  2.2. WaitApWakeup() waits AP to clear WAKEUP_AP_SIGNAL.
3. AP: ApWakeupFunction() clears WAKEUP_AP_SIGNAL to inform BSP.
5. BSP: FreeResetVector() restores the <1MB memory
4. AP: ApWakeupFunction() calls the certain procedure.
  4.1. NumApsExecuting is decreased.

#4.1 happens after the 1MB memory is restored so the result is
memory below 1MB is changed by #4.1
It happens only when the AP executes procedure a bit longer.
AP returns back to ApWakeupFunction() from procedure after
BSP restores the <1MB memory.

Since NumApsExecuting is only used when InitFlag == ApInitConfig
for counting the processor count.
The patch moves the NumApsExecuting decrease to the path when
InitFlag == ApInitConfig.

Signed-off-by: Ray Ni <ray.ni@intel.com>
Cc: Eric Dong <eric.dong@intel.com>
Cc: Nandagopal Sathyanarayanan <nandagopal.sathyanarayanan@intel.com>
---
 UefiCpuPkg/Library/MpInitLib/MpLib.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/UefiCpuPkg/Library/MpInitLib/MpLib.c b/UefiCpuPkg/Library/MpInitLib/MpLib.c
index 3337488892..6f51bc4ebf 100644
--- a/UefiCpuPkg/Library/MpInitLib/MpLib.c
+++ b/UefiCpuPkg/Library/MpInitLib/MpLib.c
@@ -1,7 +1,7 @@
 /** @file
   CPU MP Initialize Library common functions.
 
-  Copyright (c) 2016 - 2018, Intel Corporation. All rights reserved.<BR>
+  Copyright (c) 2016 - 2019, Intel Corporation. All rights reserved.<BR>
   SPDX-License-Identifier: BSD-2-Clause-Patent
 
 **/
@@ -619,6 +619,8 @@ ApWakeupFunction (
       RestoreVolatileRegisters (&CpuMpData->CpuData[0].VolatileRegisters, FALSE);
       InitializeApData (CpuMpData, ProcessorNumber, BistData, ApTopOfStack);
       ApStartupSignalBuffer = CpuMpData->CpuData[ProcessorNumber].StartupApSignal;
+
+      InterlockedDecrement ((UINT32 *) &CpuMpData->MpCpuExchangeInfo->NumApsExecuting);
     } else {
       //
       // Execute AP function if AP is ready
@@ -698,7 +700,6 @@ ApWakeupFunction (
     // AP finished executing C code
     //
     InterlockedIncrement ((UINT32 *) &CpuMpData->FinishedCount);
-    InterlockedDecrement ((UINT32 *) &CpuMpData->MpCpuExchangeInfo->NumApsExecuting);
 
     //
     // Place AP is specified loop mode
-- 
2.21.0.windows.1

Re: [PATCH v2 2/2] UefiCpuPkg/MpInitLib: Decrease NumApsExecuting only for ApInitConfig

[Dong, Eric]

Reviewed-by: Eric Dong <eric.dong@intel.com>

> -----Original Message-----
> From: Ni, Ray
> Sent: Wednesday, June 5, 2019 1:49 PM
> To: devel@edk2.groups.io
> Cc: Dong, Eric <eric.dong@intel.com>; Sathyanarayanan, Nandagopal
> <nandagopal.sathyanarayanan@intel.com>
> Subject: [PATCH v2 2/2] UefiCpuPkg/MpInitLib: Decrease NumApsExecuting
> only for ApInitConfig
> 
> The patch fixes the bug that the memory under 1MB is modified by firmware
> in S3 boot.
> 
> Root cause is a racing condition in MpInitLib:
> 1. BSP: WakeUpByInitSipiSipi is set by NotifyOnS3SmmInitDonePpi() 2. BSP:
> WakeUpAP() wakes all APs to run certain procedure.
>   2.1. AllocateResetVector() uses <1MB memory for wake up vector.
>   2.1. FillExchangeInfoData() resets NumApsExecuting to 0.
>   2.2. WaitApWakeup() waits AP to clear WAKEUP_AP_SIGNAL.
> 3. AP: ApWakeupFunction() clears WAKEUP_AP_SIGNAL to inform BSP.
> 5. BSP: FreeResetVector() restores the <1MB memory 4. AP:
> ApWakeupFunction() calls the certain procedure.
>   4.1. NumApsExecuting is decreased.
> 
> #4.1 happens after the 1MB memory is restored so the result is memory
> below 1MB is changed by #4.1 It happens only when the AP executes
> procedure a bit longer.
> AP returns back to ApWakeupFunction() from procedure after BSP restores
> the <1MB memory.
> 
> Since NumApsExecuting is only used when InitFlag == ApInitConfig for
> counting the processor count.
> The patch moves the NumApsExecuting decrease to the path when InitFlag
> == ApInitConfig.
> 
> Signed-off-by: Ray Ni <ray.ni@intel.com>
> Cc: Eric Dong <eric.dong@intel.com>
> Cc: Nandagopal Sathyanarayanan <nandagopal.sathyanarayanan@intel.com>
> ---
>  UefiCpuPkg/Library/MpInitLib/MpLib.c | 5 +++--
>  1 file changed, 3 insertions(+), 2 deletions(-)
> 
> diff --git a/UefiCpuPkg/Library/MpInitLib/MpLib.c
> b/UefiCpuPkg/Library/MpInitLib/MpLib.c
> index 3337488892..6f51bc4ebf 100644
> --- a/UefiCpuPkg/Library/MpInitLib/MpLib.c
> +++ b/UefiCpuPkg/Library/MpInitLib/MpLib.c
> @@ -1,7 +1,7 @@
>  /** @file
>    CPU MP Initialize Library common functions.
> 
> -  Copyright (c) 2016 - 2018, Intel Corporation. All rights reserved.<BR>
> +  Copyright (c) 2016 - 2019, Intel Corporation. All rights
> + reserved.<BR>
>    SPDX-License-Identifier: BSD-2-Clause-Patent
> 
>  **/
> @@ -619,6 +619,8 @@ ApWakeupFunction (
>        RestoreVolatileRegisters (&CpuMpData->CpuData[0].VolatileRegisters,
> FALSE);
>        InitializeApData (CpuMpData, ProcessorNumber, BistData,
> ApTopOfStack);
>        ApStartupSignalBuffer = CpuMpData-
> >CpuData[ProcessorNumber].StartupApSignal;
> +
> +      InterlockedDecrement ((UINT32 *)
> + &CpuMpData->MpCpuExchangeInfo->NumApsExecuting);
>      } else {
>        //
>        // Execute AP function if AP is ready @@ -698,7 +700,6 @@
> ApWakeupFunction (
>      // AP finished executing C code
>      //
>      InterlockedIncrement ((UINT32 *) &CpuMpData->FinishedCount);
> -    InterlockedDecrement ((UINT32 *) &CpuMpData->MpCpuExchangeInfo-
> >NumApsExecuting);
> 
>      //
>      // Place AP is specified loop mode
> --
> 2.21.0.windows.1

Re: [edk2-devel] [PATCH v2 0/2] Fix the issue that OS complains memory < 1MB changes across S3

[Laszlo Ersek]

On 06/05/19 07:49, Ni, Ray wrote:
> v2:
>   Answer Laszlo's comments:
>     1. Add 1/2 patch to change IA32 waking up vector assembly code to increase
>        NumApsExecuting only for ApInitConfig.
>     2. Change 2/2 patch to remove the unnecessary debug messages.
> 
> Ray Ni (2):
>   UefiCpuPkg/MpInitLib: increase NumApsExecuting only for ApInitConfig
>   UefiCpuPkg/MpInitLib: Decrease NumApsExecuting only for ApInitConfig
> 
>  UefiCpuPkg/Library/MpInitLib/Ia32/MpFuncs.nasm | 14 +++++++-------
>  UefiCpuPkg/Library/MpInitLib/MpLib.c           |  5 +++--
>  2 files changed, 10 insertions(+), 9 deletions(-)
> 

For the series:

Reviewed-by: Laszlo Ersek <lersek@redhat.com>
Regression-tested-by: Laszlo Ersek <lersek@redhat.com>

Thanks,
Laszlo