Why need to put AP in HLT/MWAIT in ExitBootServices callback

Why need to put AP in HLT/MWAIT in ExitBootServices callback

[Ni, Ray]

All,
When I am reviewing today's MpInitLib code, I don't understand why below complex logic is there to put AP in HLT or MWAIT on a ExitBootServices event callback.
https://github.com/tianocore/edk2/blob/ff36b2550f94dc5fac838cf298ae5a23cfddf204/UefiCpuPkg/Library/MpInitLib/DxeMpLib.c#L423
https://github.com/tianocore/edk2/blob/ff36b2550f94dc5fac838cf298ae5a23cfddf204/UefiCpuPkg/Library/MpInitLib/X64/MpFuncs.nasm#L424

The above code is to put AP in a state that when it's woken up by interrupts or SMIs, it's executing code and using stack in reserved memory.
(not in BootServices memory which might be overwritten by OS.)

But according to the SDM content below, we can simply send a INIT IPI to all APs to let AP enter a wait-for-SIPI state.
Then a simple SendInitIpiAllExcludingSelf() from BSP can replace all the complicated code above.

* The MP protocol will be executed only after a power-up or RESET. If the MP protocol has been completed and a
* BSP has been chosen, subsequent INITs (either to a specific processor or system wide) do not cause the MP
* protocol to be repeated. Instead, each processor examines its BSP flag (in the APIC_BASE MSR) to determine
* whether it should execute the BIOS boot-strap code (if it is the BSP) or enter a wait-for-SIPI state (if it is an
* AP).

Did I miss something here?

Thanks,
Ray

Re: Why need to put AP in HLT/MWAIT in ExitBootServices callback

[Yao, Jiewen]

I remember a wait-for-SIPI CPU cannot response to SMI.

We do want to let AP response SMI to perform some action, because some SMM code will rely on all APs in SMM.

Thank you
Yao Jiewen


> -----Original Message-----
> From: Ni, Ray <ray.ni@intel.com>
> Sent: Thursday, June 9, 2022 4:46 PM
> To: devel@edk2.groups.io
> Cc: Ni, Ray <ray.ni@intel.com>; Kinney, Michael D
> <michael.d.kinney@intel.com>; Tom Lendacky <thomas.lendacky@amd.com>;
> Yao, Jiewen <jiewen.yao@intel.com>
> Subject: Why need to put AP in HLT/MWAIT in ExitBootServices callback
> 
> All,
> When I am reviewing today's MpInitLib code, I don't understand why below
> complex logic is there to put AP in HLT or MWAIT on a ExitBootServices event
> callback.
> https://github.com/tianocore/edk2/blob/ff36b2550f94dc5fac838cf298ae5a23c
> fddf204/UefiCpuPkg/Library/MpInitLib/DxeMpLib.c#L423
> https://github.com/tianocore/edk2/blob/ff36b2550f94dc5fac838cf298ae5a23c
> fddf204/UefiCpuPkg/Library/MpInitLib/X64/MpFuncs.nasm#L424
> 
> The above code is to put AP in a state that when it's woken up by interrupts or
> SMIs, it's executing code and using stack in reserved memory.
> (not in BootServices memory which might be overwritten by OS.)
> 
> But according to the SDM content below, we can simply send a INIT IPI to all APs
> to let AP enter a wait-for-SIPI state.
> Then a simple SendInitIpiAllExcludingSelf() from BSP can replace all the
> complicated code above.
> 
> * The MP protocol will be executed only after a power-up or RESET. If the MP
> protocol has been completed and a
> * BSP has been chosen, subsequent INITs (either to a specific processor or
> system wide) do not cause the MP
> * protocol to be repeated. Instead, each processor examines its BSP flag (in the
> APIC_BASE MSR) to determine
> * whether it should execute the BIOS boot-strap code (if it is the BSP) or enter a
> wait-for-SIPI state (if it is an
> * AP).
> 
> Did I miss something here?
> 
> Thanks,
> Ray

Re: Why need to put AP in HLT/MWAIT in ExitBootServices callback

[Ni, Ray]

Jiewen,
You are right!:)

* In the Pentium 4, Intel Xeon, and P6 family processors, when a processor that is designated as an
* application processor during an MP initialization sequence is waiting for a startup IPI (SIPI), it is in
* a mode where SMIs are masked. However if a SMI is received while an application processor is in
* the wait for SIPI mode, the SMI will be pended. The processor then responds on receipt of a SIPI by
* immediately servicing the pended SMI and going into SMM before handling the SIPI.

> -----Original Message-----
> From: Yao, Jiewen <jiewen.yao@intel.com>
> Sent: Thursday, June 9, 2022 4:53 PM
> To: Ni, Ray <ray.ni@intel.com>; devel@edk2.groups.io
> Cc: Kinney, Michael D <michael.d.kinney@intel.com>; Tom Lendacky <thomas.lendacky@amd.com>
> Subject: RE: Why need to put AP in HLT/MWAIT in ExitBootServices callback
> 
> I remember a wait-for-SIPI CPU cannot response to SMI.
> 
> We do want to let AP response SMI to perform some action, because some SMM code will rely on all APs in SMM.
> 
> Thank you
> Yao Jiewen
> 
> 
> > -----Original Message-----
> > From: Ni, Ray <ray.ni@intel.com>
> > Sent: Thursday, June 9, 2022 4:46 PM
> > To: devel@edk2.groups.io
> > Cc: Ni, Ray <ray.ni@intel.com>; Kinney, Michael D
> > <michael.d.kinney@intel.com>; Tom Lendacky <thomas.lendacky@amd.com>;
> > Yao, Jiewen <jiewen.yao@intel.com>
> > Subject: Why need to put AP in HLT/MWAIT in ExitBootServices callback
> >
> > All,
> > When I am reviewing today's MpInitLib code, I don't understand why below
> > complex logic is there to put AP in HLT or MWAIT on a ExitBootServices event
> > callback.
> > https://github.com/tianocore/edk2/blob/ff36b2550f94dc5fac838cf298ae5a23c
> > fddf204/UefiCpuPkg/Library/MpInitLib/DxeMpLib.c#L423
> > https://github.com/tianocore/edk2/blob/ff36b2550f94dc5fac838cf298ae5a23c
> > fddf204/UefiCpuPkg/Library/MpInitLib/X64/MpFuncs.nasm#L424
> >
> > The above code is to put AP in a state that when it's woken up by interrupts or
> > SMIs, it's executing code and using stack in reserved memory.
> > (not in BootServices memory which might be overwritten by OS.)
> >
> > But according to the SDM content below, we can simply send a INIT IPI to all APs
> > to let AP enter a wait-for-SIPI state.
> > Then a simple SendInitIpiAllExcludingSelf() from BSP can replace all the
> > complicated code above.
> >
> > * The MP protocol will be executed only after a power-up or RESET. If the MP
> > protocol has been completed and a
> > * BSP has been chosen, subsequent INITs (either to a specific processor or
> > system wide) do not cause the MP
> > * protocol to be repeated. Instead, each processor examines its BSP flag (in the
> > APIC_BASE MSR) to determine
> > * whether it should execute the BIOS boot-strap code (if it is the BSP) or enter a
> > wait-for-SIPI state (if it is an
> > * AP).
> >
> > Did I miss something here?
> >
> > Thanks,
> > Ray