[Patch v3 0/2] Enable new MM MP protocol

[Patch v3 0/2] Enable new MM MP protocol

[Dong, Eric]

v3 changes:
1. Fix Token clean up too early caused CheckProcedure return error.

https://bugzilla.tianocore.org/show_bug.cgi?id=1937

PI spec added a new protocol named MM MP protocol. This protocol 
allows for better remote queuing of execution of procedures on an AP.
This extends the existing procedures to allow:
1. A function to be called in blocking and non-blocking manner explicitly 
2. Allow broadcasts.
3. Allow execution of a procedure when a processor powers up.

This patch serial enable this new protocol.

Cc: Ray Ni <ray.ni@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>

Eric Dong (2):
  MdePkg: Add new MM MP Protocol definition.
  UefiCpuPkg/PiSmmCpuDxeSmm: Enable MM MP Protocol.

 MdePkg/Include/Pi/PiMultiPhase.h             |  16 +
 MdePkg/Include/Protocol/MmMp.h               | 333 +++++++++++
 MdePkg/MdePkg.dec                            |   3 +
 UefiCpuPkg/PiSmmCpuDxeSmm/MpService.c        | 555 ++++++++++++++++++-
 UefiCpuPkg/PiSmmCpuDxeSmm/PiSmmCpuDxeSmm.c   |  11 +
 UefiCpuPkg/PiSmmCpuDxeSmm/PiSmmCpuDxeSmm.h   | 160 +++++-
 UefiCpuPkg/PiSmmCpuDxeSmm/PiSmmCpuDxeSmm.inf |   3 +
 UefiCpuPkg/PiSmmCpuDxeSmm/SmmMp.c            | 372 +++++++++++++
 UefiCpuPkg/PiSmmCpuDxeSmm/SmmMp.h            | 286 ++++++++++
 9 files changed, 1722 insertions(+), 17 deletions(-)
 create mode 100644 MdePkg/Include/Protocol/MmMp.h
 create mode 100644 UefiCpuPkg/PiSmmCpuDxeSmm/SmmMp.c
 create mode 100644 UefiCpuPkg/PiSmmCpuDxeSmm/SmmMp.h

-- 
2.21.0.windows.1

[Patch v3 1/2] MdePkg: Add new MM MP Protocol definition.

[Dong, Eric]

V2 change:
1. Remove SMM prefix definition which not required by spec.

V1 changes:
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=1937

EFI MM MP Protocol is defined in the PI 1.5 specification.

The MM MP protocol provides a set of functions to allow execution of
procedures on processors that have entered MM. This protocol has the
following properties:
1. The caller can invoke execution of a procedure on a processor, other
than the caller, that has also entered MM. Supports blocking and
non-blocking modes of operation.
2. The caller can invoke a procedure on multiple processors. Supports
blocking and non-blocking modes of operation.

Cc: Ray Ni <ray.ni@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Signed-off-by: Eric Dong <eric.dong@intel.com>
---
 MdePkg/Include/Pi/PiMultiPhase.h |  16 ++
 MdePkg/Include/Protocol/MmMp.h   | 333 +++++++++++++++++++++++++++++++
 MdePkg/MdePkg.dec                |   3 +
 3 files changed, 352 insertions(+)
 create mode 100644 MdePkg/Include/Protocol/MmMp.h

diff --git a/MdePkg/Include/Pi/PiMultiPhase.h b/MdePkg/Include/Pi/PiMultiPhase.h
index eb12527767..a5056799e1 100644
--- a/MdePkg/Include/Pi/PiMultiPhase.h
+++ b/MdePkg/Include/Pi/PiMultiPhase.h
@@ -176,4 +176,20 @@ VOID
   IN OUT VOID  *Buffer
   );
 
+/**
+  The function prototype for invoking a function on an Application Processor.
+
+  This definition is used by the UEFI MM MP Serices Protocol.
+
+  @param[in] ProcedureArgument    The pointer to private data buffer.
+
+  @retval EFI_SUCCESS             Excutive the procedure successfully
+
+**/
+typedef
+EFI_STATUS
+(EFIAPI *EFI_AP_PROCEDURE2)(
+  IN VOID  *ProcedureArgument
+);
+
 #endif
diff --git a/MdePkg/Include/Protocol/MmMp.h b/MdePkg/Include/Protocol/MmMp.h
new file mode 100644
index 0000000000..beace1386c
--- /dev/null
+++ b/MdePkg/Include/Protocol/MmMp.h
@@ -0,0 +1,333 @@
+/** @file
+  EFI MM MP Protocol is defined in the PI 1.5 specification.
+
+  The MM MP protocol provides a set of functions to allow execution of procedures on processors that
+  have entered MM. This protocol has the following properties:
+  1. The caller can only invoke execution of a procedure on a processor, other than the caller, that
+     has also entered MM.
+  2. It is possible to invoke a procedure on multiple processors. Supports blocking and non-blocking
+     modes of operation.
+
+  Copyright (c) 2019, Intel Corporation. All rights reserved.<BR>
+  SPDX-License-Identifier: BSD-2-Clause-Patent
+
+**/
+
+#ifndef _MM_MP_H_
+#define _MM_MP_H_
+
+#include <Pi/PiMmCis.h>
+
+#define EFI_MM_MP_PROTOCOL_GUID \
+  { \
+    0x5d5450d7, 0x990c, 0x4180, {0xa8, 0x3, 0x8e, 0x63, 0xf0, 0x60, 0x83, 0x7  }  \
+  }
+
+//
+// Revision definition.
+//
+#define EFI_MM_MP_PROTOCOL_REVISION    0x00
+
+//
+// Attribute flags
+//
+#define EFI_MM_MP_TIMEOUT_SUPPORTED    0x01
+
+//
+// Completion token
+//
+typedef VOID* MM_COMPLETION;
+
+typedef struct {
+  MM_COMPLETION  Completion;
+  EFI_STATUS     Status;
+} MM_DISPATCH_COMPLETION_TOKEN;
+
+typedef struct _EFI_MM_MP_PROTOCOL  EFI_MM_MP_PROTOCOL;
+
+/**
+  Service to retrieves the number of logical processor in the platform.
+
+  @param[in]  This                The EFI_MM_MP_PROTOCOL instance.
+  @param[out] NumberOfProcessors  Pointer to the total number of logical processors in the system,
+                                  including the BSP and all APs.
+
+  @retval EFI_SUCCESS             The number of processors was retrieved successfully
+  @retval EFI_INVALID_PARAMETER   NumberOfProcessors is NULL
+**/
+typedef
+EFI_STATUS
+(EFIAPI *EFI_MM_GET_NUMBER_OF_PROCESSORS) (
+  IN CONST EFI_MM_MP_PROTOCOL  *This,
+  OUT      UINTN               *NumberOfProcessors
+);
+
+
+/**
+  This service allows the caller to invoke a procedure one of the application processors (AP). This
+  function uses an optional token parameter to support blocking and non-blocking modes. If the token
+  is passed into the call, the function will operate in a non-blocking fashion and the caller can
+  check for completion with CheckOnProcedure or WaitForProcedure.
+
+  @param[in]     This                   The EFI_MM_MP_PROTOCOL instance.
+  @param[in]     Procedure              A pointer to the procedure to be run on the designated target
+                                        AP of the system. Type EFI_AP_PROCEDURE2 is defined below in
+                                        related definitions.
+  @param[in]     CpuNumber              The zero-based index of the processor number of the target
+                                        AP, on which the code stream is supposed to run. If the number
+                                        points to the calling processor then it will not run the
+                                        supplied code.
+  @param[in]     TimeoutInMicroseconds  Indicates the time limit in microseconds for this AP to
+                                        finish execution of Procedure, either for blocking or
+                                        non-blocking mode. Zero means infinity. If the timeout
+                                        expires before this AP returns from Procedure, then Procedure
+                                        on the AP is terminated. If the timeout expires in blocking
+                                        mode, the call returns EFI_TIMEOUT. If the timeout expires
+                                        in non-blocking mode, the timeout determined can be through
+                                        CheckOnProcedure or WaitForProcedure.
+                                        Note that timeout support is optional. Whether an
+                                        implementation supports this feature, can be determined via
+                                        the Attributes data member.
+  @param[in,out] ProcedureArguments     Allows the caller to pass a list of parameters to the code
+                                        that is run by the AP. It is an optional common mailbox
+                                        between APs and the caller to share information.
+  @param[in,out] Token                  This is parameter is broken into two components:
+                                        1.Token->Completion is an optional parameter that allows the
+                                        caller to execute the procedure in a blocking or non-blocking
+                                        fashion. If it is NULL the call is blocking, and the call will
+                                        not return until the AP has completed the procedure. If the
+                                        token is not NULL, the call will return immediately. The caller
+                                        can check whether the procedure has completed with
+                                        CheckOnProcedure or WaitForProcedure.
+                                        2.Token->Status The implementation updates the address pointed
+                                        at by this variable with the status code returned by Procedure
+                                        when it completes execution on the target AP, or with EFI_TIMEOUT
+                                        if the Procedure fails to complete within the optional timeout.
+                                        The implementation will update this variable with EFI_NOT_READY
+                                        prior to starting Procedure on the target AP.
+  @param[in,out] CPUStatus              This optional pointer may be used to get the status code returned
+                                        by Procedure when it completes execution on the target AP, or with
+                                        EFI_TIMEOUT if the Procedure fails to complete within the optional
+                                        timeout. The implementation will update this variable with
+                                        EFI_NOT_READY prior to starting Procedure on the target AP.
+
+  @retval EFI_SUCCESS                   In the blocking case, this indicates that Procedure has completed
+                                        execution on the target AP.
+                                        In the non-blocking case this indicates that the procedure has
+                                        been successfully scheduled for execution on the target AP.
+  @retval EFI_INVALID_PARAMETER         The input arguments are out of range. Either the target AP is the
+                                        caller of the function, or the Procedure or Token is NULL
+  @retval EFI_NOT_READY                 If the target AP is busy executing another procedure
+  @retval EFI_ALREADY_STARTED           Token is already in use for another procedure
+  @retval EFI_TIMEOUT                   In blocking mode, the timeout expired before the specified AP
+                                        has finished
+**/
+typedef
+EFI_STATUS
+(EFIAPI *EFI_MM_DISPATCH_PROCEDURE) (
+  IN CONST EFI_MM_MP_PROTOCOL            *This,
+  IN       EFI_AP_PROCEDURE2             Procedure,
+  IN       UINTN                         CpuNumber,
+  IN       UINTN                         TimeoutInMicroseconds,
+  IN OUT   VOID                          *ProcedureArguments OPTIONAL,
+  IN OUT   MM_COMPLETION                 *Token,
+  IN OUT   EFI_STATUS                    *CPUStatus
+);
+
+/**
+  This service allows the caller to invoke a procedure on all running application processors (AP)
+  except the caller. This function uses an optional token parameter to support blocking and
+  nonblocking modes. If the token is passed into the call, the function will operate in a non-blocking
+  fashion and the caller can check for completion with CheckOnProcedure or WaitForProcedure.
+
+  It is not necessary for the implementation to run the procedure on every processor on the platform.
+  Processors that are powered down in such a way that they cannot respond to interrupts, may be
+  excluded from the broadcast.
+
+
+  @param[in]     This                   The EFI_MM_MP_PROTOCOL instance.
+  @param[in]     Procedure              A pointer to the code stream to be run on the APs that have
+                                        entered MM. Type EFI_AP_PROCEDURE is defined below in related
+                                        definitions.
+  @param[in]     TimeoutInMicroseconds  Indicates the time limit in microseconds for the APs to finish
+                                        execution of Procedure, either for blocking or non-blocking mode.
+                                        Zero means infinity. If the timeout expires before all APs return
+                                        from Procedure, then Procedure on the failed APs is terminated. If
+                                        the timeout expires in blocking mode, the call returns EFI_TIMEOUT.
+                                        If the timeout expires in non-blocking mode, the timeout determined
+                                        can be through CheckOnProcedure or WaitForProcedure.
+                                        Note that timeout support is optional. Whether an implementation
+                                        supports this feature can be determined via the Attributes data
+                                        member.
+  @param[in,out] ProcedureArguments     Allows the caller to pass a list of parameters to the code
+                                        that is run by the AP. It is an optional common mailbox
+                                        between APs and the caller to share information.
+  @param[in,out] Token                  This is parameter is broken into two components:
+                                        1.Token->Completion is an optional parameter that allows the
+                                        caller to execute the procedure in a blocking or non-blocking
+                                        fashion. If it is NULL the call is blocking, and the call will
+                                        not return until the AP has completed the procedure. If the
+                                        token is not NULL, the call will return immediately. The caller
+                                        can check whether the procedure has completed with
+                                        CheckOnProcedure or WaitForProcedure.
+                                        2.Token->Status The implementation updates the address pointed
+                                        at by this variable with the status code returned by Procedure
+                                        when it completes execution on the target AP, or with EFI_TIMEOUT
+                                        if the Procedure fails to complete within the optional timeout.
+                                        The implementation will update this variable with EFI_NOT_READY
+                                        prior to starting Procedure on the target AP
+  @param[in,out] CPUStatus              This optional pointer may be used to get the individual status
+                                        returned by every AP that participated in the broadcast. This
+                                        parameter if used provides the base address of an array to hold
+                                        the EFI_STATUS value of each AP in the system. The size of the
+                                        array can be ascertained by the GetNumberOfProcessors function.
+                                        As mentioned above, the broadcast may not include every processor
+                                        in the system. Some implementations may exclude processors that
+                                        have been powered down in such a way that they are not responsive
+                                        to interrupts. Additionally the broadcast excludes the processor
+                                        which is making the BroadcastProcedure call. For every excluded
+                                        processor, the array entry must contain a value of EFI_NOT_STARTED
+
+  @retval EFI_SUCCESS                   In the blocking case, this indicates that Procedure has completed
+                                        execution on the APs. In the non-blocking case this indicates that
+                                        the procedure has been successfully scheduled for execution on the
+                                        APs.
+  @retval EFI_INVALID_PARAMETER         Procedure or Token is NULL.
+  @retval EFI_NOT_READY                 If a target AP is busy executing another procedure.
+  @retval EFI_TIMEOUT                   In blocking mode, the timeout expired before all enabled APs have
+                                        finished.
+  @retval EFI_ALREADY_STARTED           Before the AP procedure associated with the Token is finished, the
+                                        same Token cannot be used to dispatch or broadcast another procedure.
+
+**/
+typedef
+EFI_STATUS
+(EFIAPI *EFI_MM_BROADCAST_PROCEDURE) (
+  IN CONST EFI_MM_MP_PROTOCOL            *This,
+  IN       EFI_AP_PROCEDURE2             Procedure,
+  IN       UINTN                         TimeoutInMicroseconds,
+  IN OUT   VOID                          *ProcedureArguments OPTIONAL,
+  IN OUT   MM_COMPLETION                 *Token,
+  IN OUT   EFI_STATUS                    *CPUStatus
+);
+
+
+/**
+  This service allows the caller to set a startup procedure that will be executed when an AP powers
+  up from a state where core configuration and context is lost. The procedure is execution has the
+  following properties:
+  1. The procedure executes before the processor is handed over to the operating system.
+  2. All processors execute the same startup procedure.
+  3. The procedure may run in parallel with other procedures invoked through the functions in this
+  protocol, or with processors that are executing an MM handler or running in the operating system.
+
+
+  @param[in]      This                 The EFI_MM_MP_PROTOCOL instance.
+  @param[in]      Procedure            A pointer to the code stream to be run on the designated target AP
+                                       of the system. Type EFI_AP_PROCEDURE is defined below in Volume 2
+                                       with the related definitions of
+                                       EFI_MP_SERVICES_PROTOCOL.StartupAllAPs.
+                                       If caller may pass a value of NULL to deregister any existing
+                                       startup procedure.
+  @param[in,out]  ProcedureArguments   Allows the caller to pass a list of parameters to the code that is
+                                       run by the AP. It is an optional common mailbox between APs and
+                                       the caller to share information
+
+  @retval EFI_SUCCESS                  The Procedure has been set successfully.
+  @retval EFI_INVALID_PARAMETER        The Procedure is NULL.
+**/
+typedef
+EFI_STATUS
+(EFIAPI *EFI_MM_SET_STARTUP_PROCEDURE) (
+  IN CONST EFI_MM_MP_PROTOCOL *This,
+  IN       EFI_AP_PROCEDURE   Procedure,
+  IN OUT   VOID               *ProcedureArguments OPTIONAL
+);
+
+/**
+  When non-blocking execution of a procedure on an AP is invoked with DispatchProcedure,
+  via the use of a token, this function can be used to check for completion of the procedure on the AP.
+  The function takes the token that was passed into the DispatchProcedure call. If the procedure
+  is complete, and therefore it is now possible to run another procedure on the same AP, this function
+  returns EFI_SUCESS. In this case the status returned by the procedure that executed on the AP is
+  returned in the token's Status field. If the procedure has not yet completed, then this function
+  returns EFI_NOT_READY.
+
+  When a non-blocking execution of a procedure is invoked with BroadcastProcedure, via the
+  use of a token, this function can be used to check for completion of the procedure on all the
+  broadcast APs. The function takes the token that was passed into the BroadcastProcedure
+  call. If the procedure is complete on all broadcast APs this function returns EFI_SUCESS. In this
+  case the Status field in the token passed into the function reflects the overall result of the
+  invocation, which may be EFI_SUCCESS, if all executions succeeded, or the first observed failure.
+  If the procedure has not yet completed on the broadcast APs, the function returns
+  EFI_NOT_READY.
+
+  @param[in]      This                 The EFI_MM_MP_PROTOCOL instance.
+  @param[in]      Token                This parameter describes the token that was passed into
+                                       DispatchProcedure or BroadcastProcedure.
+
+  @retval EFI_SUCCESS                  Procedure has completed.
+  @retval EFI_NOT_READY                The Procedure has not completed.
+  @retval EFI_INVALID_PARAMETER        Token or Token->Completion is NULL
+  @retval EFI_NOT_FOUND                Token is not currently in use for a non-blocking call
+
+**/
+typedef
+EFI_STATUS
+(EFIAPI *EFI_CHECK_FOR_PROCEDURE) (
+  IN CONST EFI_MM_MP_PROTOCOL            *This,
+  IN       MM_COMPLETION                 Token
+);
+
+/**
+  When a non-blocking execution of a procedure on an AP is invoked via DispatchProcedure,
+  this function will block the caller until the remote procedure has completed on the designated AP.
+  The non-blocking procedure invocation is identified by the Token parameter, which must match the
+  token that used when DispatchProcedure was called. Upon completion the status returned by
+  the procedure that executed on the AP is used to update the token's Status field.
+
+  When a non-blocking execution of a procedure on an AP is invoked via BroadcastProcedure
+  this function will block the caller until the remote procedure has completed on all of the APs that
+  entered MM. The non-blocking procedure invocation is identified by the Token parameter, which
+  must match the token that used when BroadcastProcedure was called. Upon completion the
+  overall status returned by the procedures that executed on the broadcast AP is used to update the
+  token's Status field. The overall status may be EFI_SUCCESS, if all executions succeeded, or the
+  first observed failure.
+
+
+  @param[in]      This                 The EFI_MM_MP_PROTOCOL instance.
+  @param[in]      Token                This parameter describes the token that was passed into
+                                       DispatchProcedure or BroadcastProcedure.
+
+  @retval EFI_SUCCESS                  Procedure has completed.
+  @retval EFI_INVALID_PARAMETER        Token or Token->Completion is NULL
+  @retval EFI_NOT_FOUND                Token is not currently in use for a non-blocking call
+
+**/
+typedef
+EFI_STATUS
+(EFIAPI *EFI_WAIT_FOR_PROCEDURE) (
+  IN CONST EFI_MM_MP_PROTOCOL            *This,
+  IN       MM_COMPLETION                 Token
+);
+
+
+
+///
+/// The MM MP protocol provides a set of functions to allow execution of procedures on processors that
+/// have entered MM.
+///
+struct _EFI_MM_MP_PROTOCOL {
+  UINT32                            Revision;
+  UINT32                            Attributes;
+  EFI_MM_GET_NUMBER_OF_PROCESSORS   GetNumberOfProcessors;
+  EFI_MM_DISPATCH_PROCEDURE         DispatchProcedure;
+  EFI_MM_BROADCAST_PROCEDURE        BroadcastProcedure;
+  EFI_MM_SET_STARTUP_PROCEDURE      SetStartupProcedure;
+  EFI_CHECK_FOR_PROCEDURE           CheckForProcedure;
+  EFI_WAIT_FOR_PROCEDURE            WaitForProcedure;
+};
+
+extern EFI_GUID gEfiMmMpProtocolGuid;
+
+#endif
diff --git a/MdePkg/MdePkg.dec b/MdePkg/MdePkg.dec
index 6c563375ee..b382efd578 100644
--- a/MdePkg/MdePkg.dec
+++ b/MdePkg/MdePkg.dec
@@ -1167,6 +1167,9 @@
   # Protocols defined in PI 1.5.
   #
 
+  ## Include/Protocol/MmMp.h
+  gEfiMmMpProtocolGuid = { 0x5d5450d7, 0x990c, 0x4180, { 0xa8, 0x3, 0x8e, 0x63, 0xf0, 0x60, 0x83, 0x7 }}
+
   ## Include/Protocol/MmEndOfDxe.h
   gEfiMmEndOfDxeProtocolGuid = { 0x24e70042, 0xd5c5, 0x4260, { 0x8c, 0x39, 0xa, 0xd3, 0xaa, 0x32, 0xe9, 0x3d }}
 
-- 
2.21.0.windows.1

Re: [edk2-devel] [Patch v3 1/2] MdePkg: Add new MM MP Protocol definition.

[Ni, Ray]

Reviewed-by: Ray Ni <ray.ni@intel.com>

> -----Original Message-----
> From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Dong,
> Eric
> Sent: Wednesday, July 3, 2019 10:43 AM
> To: devel@edk2.groups.io
> Cc: Ni, Ray <ray.ni@intel.com>; Laszlo Ersek <lersek@redhat.com>
> Subject: [edk2-devel] [Patch v3 1/2] MdePkg: Add new MM MP Protocol
> definition.
> 
> V2 change:
> 1. Remove SMM prefix definition which not required by spec.
> 
> V1 changes:
> REF: https://bugzilla.tianocore.org/show_bug.cgi?id=1937
> 
> EFI MM MP Protocol is defined in the PI 1.5 specification.
> 
> The MM MP protocol provides a set of functions to allow execution of
> procedures on processors that have entered MM. This protocol has the
> following properties:
> 1. The caller can invoke execution of a procedure on a processor, other than
> the caller, that has also entered MM. Supports blocking and non-blocking
> modes of operation.
> 2. The caller can invoke a procedure on multiple processors. Supports
> blocking and non-blocking modes of operation.
> 
> Cc: Ray Ni <ray.ni@intel.com>
> Cc: Laszlo Ersek <lersek@redhat.com>
> Signed-off-by: Eric Dong <eric.dong@intel.com>
> ---
>  MdePkg/Include/Pi/PiMultiPhase.h |  16 ++
>  MdePkg/Include/Protocol/MmMp.h   | 333
> +++++++++++++++++++++++++++++++
>  MdePkg/MdePkg.dec                |   3 +
>  3 files changed, 352 insertions(+)
>  create mode 100644 MdePkg/Include/Protocol/MmMp.h
> 
> diff --git a/MdePkg/Include/Pi/PiMultiPhase.h
> b/MdePkg/Include/Pi/PiMultiPhase.h
> index eb12527767..a5056799e1 100644
> --- a/MdePkg/Include/Pi/PiMultiPhase.h
> +++ b/MdePkg/Include/Pi/PiMultiPhase.h
> @@ -176,4 +176,20 @@ VOID
>    IN OUT VOID  *Buffer
>    );
> 
> +/**
> +  The function prototype for invoking a function on an Application Processor.
> +
> +  This definition is used by the UEFI MM MP Serices Protocol.
> +
> +  @param[in] ProcedureArgument    The pointer to private data buffer.
> +
> +  @retval EFI_SUCCESS             Excutive the procedure successfully
> +
> +**/
> +typedef
> +EFI_STATUS
> +(EFIAPI *EFI_AP_PROCEDURE2)(
> +  IN VOID  *ProcedureArgument
> +);
> +
>  #endif
> diff --git a/MdePkg/Include/Protocol/MmMp.h
> b/MdePkg/Include/Protocol/MmMp.h new file mode 100644 index
> 0000000000..beace1386c
> --- /dev/null
> +++ b/MdePkg/Include/Protocol/MmMp.h
> @@ -0,0 +1,333 @@
> +/** @file
> +  EFI MM MP Protocol is defined in the PI 1.5 specification.
> +
> +  The MM MP protocol provides a set of functions to allow execution of
> + procedures on processors that  have entered MM. This protocol has the
> following properties:
> +  1. The caller can only invoke execution of a procedure on a processor,
> other than the caller, that
> +     has also entered MM.
> +  2. It is possible to invoke a procedure on multiple processors. Supports
> blocking and non-blocking
> +     modes of operation.
> +
> +  Copyright (c) 2019, Intel Corporation. All rights reserved.<BR>
> +  SPDX-License-Identifier: BSD-2-Clause-Patent
> +
> +**/
> +
> +#ifndef _MM_MP_H_
> +#define _MM_MP_H_
> +
> +#include <Pi/PiMmCis.h>
> +
> +#define EFI_MM_MP_PROTOCOL_GUID \
> +  { \
> +    0x5d5450d7, 0x990c, 0x4180, {0xa8, 0x3, 0x8e, 0x63, 0xf0, 0x60,
> +0x83, 0x7  }  \
> +  }
> +
> +//
> +// Revision definition.
> +//
> +#define EFI_MM_MP_PROTOCOL_REVISION    0x00
> +
> +//
> +// Attribute flags
> +//
> +#define EFI_MM_MP_TIMEOUT_SUPPORTED    0x01
> +
> +//
> +// Completion token
> +//
> +typedef VOID* MM_COMPLETION;
> +
> +typedef struct {
> +  MM_COMPLETION  Completion;
> +  EFI_STATUS     Status;
> +} MM_DISPATCH_COMPLETION_TOKEN;
> +
> +typedef struct _EFI_MM_MP_PROTOCOL  EFI_MM_MP_PROTOCOL;
> +
> +/**
> +  Service to retrieves the number of logical processor in the platform.
> +
> +  @param[in]  This                The EFI_MM_MP_PROTOCOL instance.
> +  @param[out] NumberOfProcessors  Pointer to the total number of logical
> processors in the system,
> +                                  including the BSP and all APs.
> +
> +  @retval EFI_SUCCESS             The number of processors was retrieved
> successfully
> +  @retval EFI_INVALID_PARAMETER   NumberOfProcessors is NULL
> +**/
> +typedef
> +EFI_STATUS
> +(EFIAPI *EFI_MM_GET_NUMBER_OF_PROCESSORS) (
> +  IN CONST EFI_MM_MP_PROTOCOL  *This,
> +  OUT      UINTN               *NumberOfProcessors
> +);
> +
> +
> +/**
> +  This service allows the caller to invoke a procedure one of the
> +application processors (AP). This
> +  function uses an optional token parameter to support blocking and
> +non-blocking modes. If the token
> +  is passed into the call, the function will operate in a non-blocking
> +fashion and the caller can
> +  check for completion with CheckOnProcedure or WaitForProcedure.
> +
> +  @param[in]     This                   The EFI_MM_MP_PROTOCOL instance.
> +  @param[in]     Procedure              A pointer to the procedure to be run on
> the designated target
> +                                        AP of the system. Type EFI_AP_PROCEDURE2 is defined
> below in
> +                                        related definitions.
> +  @param[in]     CpuNumber              The zero-based index of the processor
> number of the target
> +                                        AP, on which the code stream is supposed to run. If the
> number
> +                                        points to the calling processor then it will not run the
> +                                        supplied code.
> +  @param[in]     TimeoutInMicroseconds  Indicates the time limit in
> microseconds for this AP to
> +                                        finish execution of Procedure, either for blocking or
> +                                        non-blocking mode. Zero means infinity. If the timeout
> +                                        expires before this AP returns from Procedure, then
> Procedure
> +                                        on the AP is terminated. If the timeout expires in
> blocking
> +                                        mode, the call returns EFI_TIMEOUT. If the timeout
> expires
> +                                        in non-blocking mode, the timeout determined can be
> through
> +                                        CheckOnProcedure or WaitForProcedure.
> +                                        Note that timeout support is optional. Whether an
> +                                        implementation supports this feature, can be
> determined via
> +                                        the Attributes data member.
> +  @param[in,out] ProcedureArguments     Allows the caller to pass a list of
> parameters to the code
> +                                        that is run by the AP. It is an optional common mailbox
> +                                        between APs and the caller to share information.
> +  @param[in,out] Token                  This is parameter is broken into two
> components:
> +                                        1.Token->Completion is an optional parameter that
> allows the
> +                                        caller to execute the procedure in a blocking or non-
> blocking
> +                                        fashion. If it is NULL the call is blocking, and the call will
> +                                        not return until the AP has completed the procedure. If
> the
> +                                        token is not NULL, the call will return immediately. The
> caller
> +                                        can check whether the procedure has completed with
> +                                        CheckOnProcedure or WaitForProcedure.
> +                                        2.Token->Status The implementation updates the
> address pointed
> +                                        at by this variable with the status code returned by
> Procedure
> +                                        when it completes execution on the target AP, or with
> EFI_TIMEOUT
> +                                        if the Procedure fails to complete within the optional
> timeout.
> +                                        The implementation will update this variable with
> EFI_NOT_READY
> +                                        prior to starting Procedure on the target AP.
> +  @param[in,out] CPUStatus              This optional pointer may be used to get
> the status code returned
> +                                        by Procedure when it completes execution on the
> target AP, or with
> +                                        EFI_TIMEOUT if the Procedure fails to complete within
> the optional
> +                                        timeout. The implementation will update this variable
> with
> +                                        EFI_NOT_READY prior to starting Procedure on the
> target AP.
> +
> +  @retval EFI_SUCCESS                   In the blocking case, this indicates that
> Procedure has completed
> +                                        execution on the target AP.
> +                                        In the non-blocking case this indicates that the
> procedure has
> +                                        been successfully scheduled for execution on the target
> AP.
> +  @retval EFI_INVALID_PARAMETER         The input arguments are out of
> range. Either the target AP is the
> +                                        caller of the function, or the Procedure or Token is NULL
> +  @retval EFI_NOT_READY                 If the target AP is busy executing another
> procedure
> +  @retval EFI_ALREADY_STARTED           Token is already in use for another
> procedure
> +  @retval EFI_TIMEOUT                   In blocking mode, the timeout expired
> before the specified AP
> +                                        has finished **/ typedef
> +EFI_STATUS (EFIAPI *EFI_MM_DISPATCH_PROCEDURE) (
> +  IN CONST EFI_MM_MP_PROTOCOL            *This,
> +  IN       EFI_AP_PROCEDURE2             Procedure,
> +  IN       UINTN                         CpuNumber,
> +  IN       UINTN                         TimeoutInMicroseconds,
> +  IN OUT   VOID                          *ProcedureArguments OPTIONAL,
> +  IN OUT   MM_COMPLETION                 *Token,
> +  IN OUT   EFI_STATUS                    *CPUStatus
> +);
> +
> +/**
> +  This service allows the caller to invoke a procedure on all running
> +application processors (AP)
> +  except the caller. This function uses an optional token parameter to
> +support blocking and
> +  nonblocking modes. If the token is passed into the call, the function
> +will operate in a non-blocking
> +  fashion and the caller can check for completion with CheckOnProcedure or
> WaitForProcedure.
> +
> +  It is not necessary for the implementation to run the procedure on every
> processor on the platform.
> +  Processors that are powered down in such a way that they cannot
> + respond to interrupts, may be  excluded from the broadcast.
> +
> +
> +  @param[in]     This                   The EFI_MM_MP_PROTOCOL instance.
> +  @param[in]     Procedure              A pointer to the code stream to be run on
> the APs that have
> +                                        entered MM. Type EFI_AP_PROCEDURE is defined
> below in related
> +                                        definitions.
> +  @param[in]     TimeoutInMicroseconds  Indicates the time limit in
> microseconds for the APs to finish
> +                                        execution of Procedure, either for blocking or non-
> blocking mode.
> +                                        Zero means infinity. If the timeout expires before all
> APs return
> +                                        from Procedure, then Procedure on the failed APs is
> terminated. If
> +                                        the timeout expires in blocking mode, the call returns
> EFI_TIMEOUT.
> +                                        If the timeout expires in non-blocking mode, the
> timeout determined
> +                                        can be through CheckOnProcedure or
> WaitForProcedure.
> +                                        Note that timeout support is optional. Whether an
> implementation
> +                                        supports this feature can be determined via the
> Attributes data
> +                                        member.
> +  @param[in,out] ProcedureArguments     Allows the caller to pass a list of
> parameters to the code
> +                                        that is run by the AP. It is an optional common mailbox
> +                                        between APs and the caller to share information.
> +  @param[in,out] Token                  This is parameter is broken into two
> components:
> +                                        1.Token->Completion is an optional parameter that
> allows the
> +                                        caller to execute the procedure in a blocking or non-
> blocking
> +                                        fashion. If it is NULL the call is blocking, and the call will
> +                                        not return until the AP has completed the procedure. If
> the
> +                                        token is not NULL, the call will return immediately. The
> caller
> +                                        can check whether the procedure has completed with
> +                                        CheckOnProcedure or WaitForProcedure.
> +                                        2.Token->Status The implementation updates the
> address pointed
> +                                        at by this variable with the status code returned by
> Procedure
> +                                        when it completes execution on the target AP, or with
> EFI_TIMEOUT
> +                                        if the Procedure fails to complete within the optional
> timeout.
> +                                        The implementation will update this variable with
> EFI_NOT_READY
> +                                        prior to starting Procedure on the target AP
> +  @param[in,out] CPUStatus              This optional pointer may be used to get
> the individual status
> +                                        returned by every AP that participated in the broadcast.
> This
> +                                        parameter if used provides the base address of an array
> to hold
> +                                        the EFI_STATUS value of each AP in the system. The size
> of the
> +                                        array can be ascertained by the
> GetNumberOfProcessors function.
> +                                        As mentioned above, the broadcast may not include
> every processor
> +                                        in the system. Some implementations may exclude
> processors that
> +                                        have been powered down in such a way that they are
> not responsive
> +                                        to interrupts. Additionally the broadcast excludes the
> processor
> +                                        which is making the BroadcastProcedure call. For every
> excluded
> +                                        processor, the array entry must
> + contain a value of EFI_NOT_STARTED
> +
> +  @retval EFI_SUCCESS                   In the blocking case, this indicates that
> Procedure has completed
> +                                        execution on the APs. In the non-blocking case this
> indicates that
> +                                        the procedure has been successfully scheduled for
> execution on the
> +                                        APs.
> +  @retval EFI_INVALID_PARAMETER         Procedure or Token is NULL.
> +  @retval EFI_NOT_READY                 If a target AP is busy executing another
> procedure.
> +  @retval EFI_TIMEOUT                   In blocking mode, the timeout expired
> before all enabled APs have
> +                                        finished.
> +  @retval EFI_ALREADY_STARTED           Before the AP procedure associated
> with the Token is finished, the
> +                                        same Token cannot be used to dispatch or broadcast
> another procedure.
> +
> +**/
> +typedef
> +EFI_STATUS
> +(EFIAPI *EFI_MM_BROADCAST_PROCEDURE) (
> +  IN CONST EFI_MM_MP_PROTOCOL            *This,
> +  IN       EFI_AP_PROCEDURE2             Procedure,
> +  IN       UINTN                         TimeoutInMicroseconds,
> +  IN OUT   VOID                          *ProcedureArguments OPTIONAL,
> +  IN OUT   MM_COMPLETION                 *Token,
> +  IN OUT   EFI_STATUS                    *CPUStatus
> +);
> +
> +
> +/**
> +  This service allows the caller to set a startup procedure that will
> +be executed when an AP powers
> +  up from a state where core configuration and context is lost. The
> +procedure is execution has the
> +  following properties:
> +  1. The procedure executes before the processor is handed over to the
> operating system.
> +  2. All processors execute the same startup procedure.
> +  3. The procedure may run in parallel with other procedures invoked
> +through the functions in this
> +  protocol, or with processors that are executing an MM handler or running
> in the operating system.
> +
> +
> +  @param[in]      This                 The EFI_MM_MP_PROTOCOL instance.
> +  @param[in]      Procedure            A pointer to the code stream to be run on
> the designated target AP
> +                                       of the system. Type EFI_AP_PROCEDURE is defined
> below in Volume 2
> +                                       with the related definitions of
> +                                       EFI_MP_SERVICES_PROTOCOL.StartupAllAPs.
> +                                       If caller may pass a value of NULL to deregister any
> existing
> +                                       startup procedure.
> +  @param[in,out]  ProcedureArguments   Allows the caller to pass a list of
> parameters to the code that is
> +                                       run by the AP. It is an optional common mailbox
> between APs and
> +                                       the caller to share information
> +
> +  @retval EFI_SUCCESS                  The Procedure has been set successfully.
> +  @retval EFI_INVALID_PARAMETER        The Procedure is NULL.
> +**/
> +typedef
> +EFI_STATUS
> +(EFIAPI *EFI_MM_SET_STARTUP_PROCEDURE) (
> +  IN CONST EFI_MM_MP_PROTOCOL *This,
> +  IN       EFI_AP_PROCEDURE   Procedure,
> +  IN OUT   VOID               *ProcedureArguments OPTIONAL
> +);
> +
> +/**
> +  When non-blocking execution of a procedure on an AP is invoked with
> +DispatchProcedure,
> +  via the use of a token, this function can be used to check for completion of
> the procedure on the AP.
> +  The function takes the token that was passed into the
> +DispatchProcedure call. If the procedure
> +  is complete, and therefore it is now possible to run another
> +procedure on the same AP, this function
> +  returns EFI_SUCESS. In this case the status returned by the procedure
> +that executed on the AP is
> +  returned in the token's Status field. If the procedure has not yet
> +completed, then this function
> +  returns EFI_NOT_READY.
> +
> +  When a non-blocking execution of a procedure is invoked with
> + BroadcastProcedure, via the  use of a token, this function can be used
> + to check for completion of the procedure on all the  broadcast APs.
> + The function takes the token that was passed into the
> + BroadcastProcedure  call. If the procedure is complete on all
> + broadcast APs this function returns EFI_SUCESS. In this  case the Status
> field in the token passed into the function reflects the overall result of the
> invocation, which may be EFI_SUCCESS, if all executions succeeded, or the
> first observed failure.
> +  If the procedure has not yet completed on the broadcast APs, the
> + function returns  EFI_NOT_READY.
> +
> +  @param[in]      This                 The EFI_MM_MP_PROTOCOL instance.
> +  @param[in]      Token                This parameter describes the token that was
> passed into
> +                                       DispatchProcedure or BroadcastProcedure.
> +
> +  @retval EFI_SUCCESS                  Procedure has completed.
> +  @retval EFI_NOT_READY                The Procedure has not completed.
> +  @retval EFI_INVALID_PARAMETER        Token or Token->Completion is
> NULL
> +  @retval EFI_NOT_FOUND                Token is not currently in use for a non-
> blocking call
> +
> +**/
> +typedef
> +EFI_STATUS
> +(EFIAPI *EFI_CHECK_FOR_PROCEDURE) (
> +  IN CONST EFI_MM_MP_PROTOCOL            *This,
> +  IN       MM_COMPLETION                 Token
> +);
> +
> +/**
> +  When a non-blocking execution of a procedure on an AP is invoked via
> +DispatchProcedure,
> +  this function will block the caller until the remote procedure has completed
> on the designated AP.
> +  The non-blocking procedure invocation is identified by the Token
> +parameter, which must match the
> +  token that used when DispatchProcedure was called. Upon completion
> +the status returned by
> +  the procedure that executed on the AP is used to update the token's
> Status field.
> +
> +  When a non-blocking execution of a procedure on an AP is invoked via
> + BroadcastProcedure  this function will block the caller until the
> + remote procedure has completed on all of the APs that  entered MM. The
> + non-blocking procedure invocation is identified by the Token
> + parameter, which  must match the token that used when
> + BroadcastProcedure was called. Upon completion the  overall status
> + returned by the procedures that executed on the broadcast AP is used to
> update the  token's Status field. The overall status may be EFI_SUCCESS, if all
> executions succeeded, or the  first observed failure.
> +
> +
> +  @param[in]      This                 The EFI_MM_MP_PROTOCOL instance.
> +  @param[in]      Token                This parameter describes the token that was
> passed into
> +                                       DispatchProcedure or BroadcastProcedure.
> +
> +  @retval EFI_SUCCESS                  Procedure has completed.
> +  @retval EFI_INVALID_PARAMETER        Token or Token->Completion is
> NULL
> +  @retval EFI_NOT_FOUND                Token is not currently in use for a non-
> blocking call
> +
> +**/
> +typedef
> +EFI_STATUS
> +(EFIAPI *EFI_WAIT_FOR_PROCEDURE) (
> +  IN CONST EFI_MM_MP_PROTOCOL            *This,
> +  IN       MM_COMPLETION                 Token
> +);
> +
> +
> +
> +///
> +/// The MM MP protocol provides a set of functions to allow execution
> +of procedures on processors that /// have entered MM.
> +///
> +struct _EFI_MM_MP_PROTOCOL {
> +  UINT32                            Revision;
> +  UINT32                            Attributes;
> +  EFI_MM_GET_NUMBER_OF_PROCESSORS   GetNumberOfProcessors;
> +  EFI_MM_DISPATCH_PROCEDURE         DispatchProcedure;
> +  EFI_MM_BROADCAST_PROCEDURE        BroadcastProcedure;
> +  EFI_MM_SET_STARTUP_PROCEDURE      SetStartupProcedure;
> +  EFI_CHECK_FOR_PROCEDURE           CheckForProcedure;
> +  EFI_WAIT_FOR_PROCEDURE            WaitForProcedure;
> +};
> +
> +extern EFI_GUID gEfiMmMpProtocolGuid;
> +
> +#endif
> diff --git a/MdePkg/MdePkg.dec b/MdePkg/MdePkg.dec index
> 6c563375ee..b382efd578 100644
> --- a/MdePkg/MdePkg.dec
> +++ b/MdePkg/MdePkg.dec
> @@ -1167,6 +1167,9 @@
>    # Protocols defined in PI 1.5.
>    #
> 
> +  ## Include/Protocol/MmMp.h
> +  gEfiMmMpProtocolGuid = { 0x5d5450d7, 0x990c, 0x4180, { 0xa8, 0x3,
> + 0x8e, 0x63, 0xf0, 0x60, 0x83, 0x7 }}
> +
>    ## Include/Protocol/MmEndOfDxe.h
>    gEfiMmEndOfDxeProtocolGuid = { 0x24e70042, 0xd5c5, 0x4260, { 0x8c, 0x39,
> 0xa, 0xd3, 0xaa, 0x32, 0xe9, 0x3d }}
> 
> --
> 2.21.0.windows.1
> 
> 
> 

[Patch v3 2/2] UefiCpuPkg/PiSmmCpuDxeSmm: Enable MM MP Protocol.

[Dong, Eric]

v3 changes:
1. Fix Token clean up too early caused CheckProcedure return error.

v2 changes:
1. Remove some duplicated global variables.
2. Enhance token design to support multiple task trig for different APs at the same time.

V1 changes:
REF: https://bugzilla.tianocore.org/show_bug.cgi?id=1937

Add MM Mp Protocol in PiSmmCpuDxeSmm driver.

Cc: Ray Ni <ray.ni@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Signed-off-by: Eric Dong <eric.dong@intel.com>
---
 UefiCpuPkg/PiSmmCpuDxeSmm/MpService.c        | 536 ++++++++++++++++++-
 UefiCpuPkg/PiSmmCpuDxeSmm/PiSmmCpuDxeSmm.c   |  11 +
 UefiCpuPkg/PiSmmCpuDxeSmm/PiSmmCpuDxeSmm.h   | 160 +++++-
 UefiCpuPkg/PiSmmCpuDxeSmm/PiSmmCpuDxeSmm.inf |   3 +
 UefiCpuPkg/PiSmmCpuDxeSmm/SmmMp.c            | 372 +++++++++++++
 UefiCpuPkg/PiSmmCpuDxeSmm/SmmMp.h            | 286 ++++++++++
 6 files changed, 1351 insertions(+), 17 deletions(-)
 create mode 100644 UefiCpuPkg/PiSmmCpuDxeSmm/SmmMp.c
 create mode 100644 UefiCpuPkg/PiSmmCpuDxeSmm/SmmMp.h

diff --git a/UefiCpuPkg/PiSmmCpuDxeSmm/MpService.c b/UefiCpuPkg/PiSmmCpuDxeSmm/MpService.c
index 64fb4d6344..13858f5d2d 100644
--- a/UefiCpuPkg/PiSmmCpuDxeSmm/MpService.c
+++ b/UefiCpuPkg/PiSmmCpuDxeSmm/MpService.c
@@ -22,6 +22,7 @@ UINTN                                       mSemaphoreSize;
 SPIN_LOCK                                   *mPFLock = NULL;
 SMM_CPU_SYNC_MODE                           mCpuSmmSyncMode;
 BOOLEAN                                     mMachineCheckSupported = FALSE;
+SPIN_LOCK                                   **mApTokenLock;
 
 /**
   Performs an atomic compare exchange operation to get semaphore.
@@ -146,6 +147,45 @@ ReleaseAllAPs (
   }
 }
 
+/**
+  Wheck whether task has been finished by all APs.
+
+  @param       BlockMode   Whether did it in block mode or non-block mode.
+
+  @retval      TRUE        Task has been finished by all APs.
+  @retval      FALSE       Task not has been finished by all APs.
+
+**/
+BOOLEAN
+IsTaskFinishInAllAPs (
+  IN BOOLEAN                        BlockMode
+  )
+{
+  UINTN                             Index;
+
+  for (Index = mMaxNumberOfCpus; Index-- > 0;) {
+    //
+    // Ignore BSP and APs which not call in SMM.
+    //
+    if ((Index == gSmmCpuPrivate->SmmCoreEntryContext.CurrentlyExecutingCpu) || (!*(mSmmMpSyncData->CpuData[Index].Present))) {
+      continue;
+    }
+
+    if (BlockMode) {
+      AcquireSpinLock(mSmmMpSyncData->CpuData[Index].Busy);
+      ReleaseSpinLock(mSmmMpSyncData->CpuData[Index].Busy);
+    } else {
+      if (AcquireSpinLockOrFail (mSmmMpSyncData->CpuData[Index].Busy)) {
+        ReleaseSpinLock(mSmmMpSyncData->CpuData[Index].Busy);
+      } else {
+        return FALSE;
+      }
+    }
+  }
+
+  return TRUE;
+}
+
 /**
   Checks if all CPUs (with certain exceptions) have checked in for this SMI run
 
@@ -347,6 +387,92 @@ ReplaceOSMtrrs (
   MtrrSetAllMtrrs (&gSmiMtrrs);
 }
 
+/**
+  Check whether execute in single AP or all APs.
+
+  Compare two Tokens used by different APs to know whether in StartAllAps call.
+
+  Whether is an valid AP base on AP's Present flag.
+
+  @retval  TRUE      IN StartAllAps call.
+  @retval  FALSE     Not in StartAllAps call.
+
+**/
+BOOLEAN
+InStartAllApsCall (
+  VOID
+  )
+{
+  UINTN      ApIndex;
+  UINTN      ApIndex2;
+
+  for (ApIndex = mMaxNumberOfCpus; ApIndex-- > 0;) {
+    if ((ApIndex != gSmmCpuPrivate->SmmCoreEntryContext.CurrentlyExecutingCpu) && 
+        *(mSmmMpSyncData->CpuData[ApIndex].Present) &&
+        (mSmmMpSyncData->CpuData[ApIndex].Token != NULL)) {
+      for (ApIndex2 = ApIndex; ApIndex2-- > 0;) {
+        if ((ApIndex2 != gSmmCpuPrivate->SmmCoreEntryContext.CurrentlyExecutingCpu) && 
+            *(mSmmMpSyncData->CpuData[ApIndex2].Present) &&
+            (mSmmMpSyncData->CpuData[ApIndex2].Token != NULL)) {
+          return mSmmMpSyncData->CpuData[ApIndex2].Token == mSmmMpSyncData->CpuData[ApIndex].Token;
+        }
+      }
+    }
+  }
+
+  return FALSE;
+}
+
+/**
+  Clean up the status flags used during executing the procedure.
+
+  @param   CpuIndex      The AP index which calls this function.
+
+**/
+VOID
+CleanUpStatusFlags (
+  IN UINTN                  CpuIndex
+  )
+{
+  UINTN                             Index;
+
+  if (InStartAllApsCall ()) {
+    //
+    // In Start All APs mode, make sure all APs have finished task.
+    //
+    if (IsTaskFinishInAllAPs (FALSE)) {
+      //
+      // Clean the flags update in the function call.
+      //
+      for (Index = mMaxNumberOfCpus; Index-- > 0;) {
+        //
+        // Only In SMM APs need to be clean up.
+        //
+        if (mSmmMpSyncData->CpuData[Index].Present) {
+          if (mSmmMpSyncData->CpuData[Index].Token != NULL) {
+            ReleaseSpinLock (mSmmMpSyncData->CpuData[Index].Token);
+            break;
+          }
+        }
+      }
+    }
+  } else {
+    //
+    // In single AP mode.
+    //
+    if (mSmmMpSyncData->CpuData[CpuIndex].Token != NULL) {
+      //
+      // Free the wrapper buffer in non-block mode used by SmmMpDispatchProcedure function.
+      //
+      if (mApTokenLock[CpuIndex] == mSmmMpSyncData->CpuData[CpuIndex].Token) {
+        FreePool ((VOID *)mSmmMpSyncData->CpuData[CpuIndex].Parameter);
+      }
+
+      ReleaseSpinLock (mSmmMpSyncData->CpuData[CpuIndex].Token);
+    }
+  }
+}
+
 /**
   SMI handler for BSP.
 
@@ -604,6 +730,7 @@ APHandler (
   UINT64                            Timer;
   UINTN                             BspIndex;
   MTRR_SETTINGS                     Mtrrs;
+  EFI_STATUS                        ProcedureStatus;
 
   //
   // Timeout BSP
@@ -730,14 +857,19 @@ APHandler (
     //
     // Invoke the scheduled procedure
     //
-    (*mSmmMpSyncData->CpuData[CpuIndex].Procedure) (
-      (VOID*)mSmmMpSyncData->CpuData[CpuIndex].Parameter
-      );
+    ProcedureStatus = (*mSmmMpSyncData->CpuData[CpuIndex].Procedure) (
+                          (VOID*)mSmmMpSyncData->CpuData[CpuIndex].Parameter
+                          );
+    if (mSmmMpSyncData->CpuData[CpuIndex].Status != NULL) {
+      *mSmmMpSyncData->CpuData[CpuIndex].Status = ProcedureStatus;
+    }
 
     //
     // Release BUSY
     //
     ReleaseSpinLock (mSmmMpSyncData->CpuData[CpuIndex].Busy);
+
+    CleanUpStatusFlags (CpuIndex);
   }
 
   if (SmmCpuFeaturesNeedConfigureMtrrs()) {
@@ -906,13 +1038,95 @@ Gen4GPageTable (
   return (UINT32)(UINTN)PageTable;
 }
 
+/**
+  Checks whether the input token is the current used token.
+
+  @param[in]  CpuIndex   Cpu Index.
+  @param[in]  Token      This parameter describes the token that was passed into DispatchProcedure or
+                         BroadcastProcedure.
+
+  @retval TRUE           The input token is the current used token.
+  @retval FALSE          The input token is not the current used token.
+**/
+BOOLEAN
+IsCurrentToken (
+  IN UINTN               CpuIndex,
+  IN SPIN_LOCK           *Token
+  )
+{
+  UINTN         Index;
+
+  if (Token == NULL) {
+    return FALSE;
+  }
+
+  if (CpuIndex == (UINTN) -1) {
+    for (Index = 0; Index < gSmmCpuPrivate->SmmCoreEntryContext.NumberOfCpus; Index ++) {
+      if (mSmmMpSyncData->CpuData[Index].Present && 
+          (mSmmMpSyncData->CpuData[Index].Token != NULL) &&
+          (mSmmMpSyncData->CpuData[Index].Token == Token)) {
+        return TRUE;
+      }
+    }
+
+    return FALSE;
+  }
+
+  return mSmmMpSyncData->CpuData[CpuIndex].Token == Token;
+}
+
+/**
+  Checks status of specified AP.
+
+  This function checks whether the specified AP has finished the task assigned
+  by StartupThisAP(), and whether timeout expires.
+
+  @param[in]  Token             This parameter describes the token that was passed into DispatchProcedure or
+                                BroadcastProcedure.
+
+  @retval EFI_SUCCESS           Specified AP has finished task assigned by StartupThisAPs().
+  @retval EFI_NOT_READY         Specified AP has not finished task and timeout has not expired.
+**/
+EFI_STATUS
+IsApReady (
+  IN SPIN_LOCK          *Token
+  )
+{
+  if (AcquireSpinLockOrFail (Token)) {
+    ReleaseSpinLock (Token);
+    return EFI_SUCCESS;
+  }
+
+  return EFI_NOT_READY;
+}
+
 /**
   Schedule a procedure to run on the specified CPU.
 
   @param[in]       Procedure                The address of the procedure to run
   @param[in]       CpuIndex                 Target CPU Index
-  @param[in, out]  ProcArguments            The parameter to pass to the procedure
-  @param[in]       BlockingMode             Startup AP in blocking mode or not
+  @param[in,out]   ProcArguments            The parameter to pass to the procedure
+  @param[in]       Token                    This is an optional parameter that allows the caller to execute the
+                                            procedure in a blocking or non-blocking fashion. If it is NULL the
+                                            call is blocking, and the call will not return until the AP has
+                                            completed the procedure. If the token is not NULL, the call will
+                                            return immediately. The caller can check whether the procedure has
+                                            completed with CheckOnProcedure or WaitForProcedure.
+  @param[in]       TimeoutInMicroseconds    Indicates the time limit in microseconds for the APs to finish
+                                            execution of Procedure, either for blocking or non-blocking mode.
+                                            Zero means infinity. If the timeout expires before all APs return
+                                            from Procedure, then Procedure on the failed APs is terminated. If
+                                            the timeout expires in blocking mode, the call returns EFI_TIMEOUT.
+                                            If the timeout expires in non-blocking mode, the timeout determined
+                                            can be through CheckOnProcedure or WaitForProcedure.
+                                            Note that timeout support is optional. Whether an implementation
+                                            supports this feature can be determined via the Attributes data
+                                            member.
+  @param[in,out]   CpuStatus                This optional pointer may be used to get the status code returned
+                                            by Procedure when it completes execution on the target AP, or with
+                                            EFI_TIMEOUT if the Procedure fails to complete within the optional
+                                            timeout. The implementation will update this variable with
+                                            EFI_NOT_READY prior to starting Procedure on the target AP.
 
   @retval EFI_INVALID_PARAMETER    CpuNumber not valid
   @retval EFI_INVALID_PARAMETER    CpuNumber specifying BSP
@@ -923,10 +1137,12 @@ Gen4GPageTable (
 **/
 EFI_STATUS
 InternalSmmStartupThisAp (
-  IN      EFI_AP_PROCEDURE          Procedure,
-  IN      UINTN                     CpuIndex,
-  IN OUT  VOID                      *ProcArguments OPTIONAL,
-  IN      BOOLEAN                   BlockingMode
+  IN      EFI_AP_PROCEDURE2              Procedure,
+  IN      UINTN                          CpuIndex,
+  IN OUT  VOID                           *ProcArguments OPTIONAL,
+  IN      SPIN_LOCK                      *Token,
+  IN      UINTN                          TimeoutInMicroseconds,
+  IN OUT  EFI_STATUS                     *CpuStatus
   )
 {
   if (CpuIndex >= gSmmCpuPrivate->SmmCoreEntryContext.NumberOfCpus) {
@@ -952,24 +1168,208 @@ InternalSmmStartupThisAp (
     }
     return EFI_INVALID_PARAMETER;
   }
+  if ((TimeoutInMicroseconds != 0) && ((mSmmMp.Attributes & EFI_MM_MP_TIMEOUT_SUPPORTED) == 0)) {
+    return EFI_INVALID_PARAMETER;
+  }
+  if (Procedure == NULL) {
+    return EFI_INVALID_PARAMETER;
+  }
+  if (IsCurrentToken (CpuIndex, Token)) {
+    return EFI_ALREADY_STARTED;
+  }
 
-  if (BlockingMode) {
+  if (Token == NULL) {
     AcquireSpinLock (mSmmMpSyncData->CpuData[CpuIndex].Busy);
   } else {
-    if (!AcquireSpinLockOrFail (mSmmMpSyncData->CpuData[CpuIndex].Busy)) {
-      DEBUG((DEBUG_ERROR, "mSmmMpSyncData->CpuData[%d].Busy\n", CpuIndex));
+    if (!AcquireSpinLockOrFail (Token)) {
+      DEBUG((DEBUG_ERROR, "Token can't acquire\n"));
       return EFI_INVALID_PARAMETER;
     }
+    if (!AcquireSpinLockOrFail (mSmmMpSyncData->CpuData[CpuIndex].Busy)) {
+      DEBUG((DEBUG_ERROR, "Can't acquire mSmmMpSyncData->CpuData[%d].Busy\n", CpuIndex));
+      ReleaseSpinLock (Token);
+      return EFI_NOT_READY;
+    }
   }
 
   mSmmMpSyncData->CpuData[CpuIndex].Procedure = Procedure;
   mSmmMpSyncData->CpuData[CpuIndex].Parameter = ProcArguments;
+  mSmmMpSyncData->CpuData[CpuIndex].Token     = Token;
+  mSmmMpSyncData->CpuData[CpuIndex].Status    = CpuStatus;
+  if (mSmmMpSyncData->CpuData[CpuIndex].Status != NULL) {
+    *mSmmMpSyncData->CpuData[CpuIndex].Status = EFI_NOT_READY;
+  }
+
   ReleaseSemaphore (mSmmMpSyncData->CpuData[CpuIndex].Run);
 
-  if (BlockingMode) {
+  if (Token == NULL) {
     AcquireSpinLock (mSmmMpSyncData->CpuData[CpuIndex].Busy);
     ReleaseSpinLock (mSmmMpSyncData->CpuData[CpuIndex].Busy);
   }
+
+  return EFI_SUCCESS;
+}
+
+/**
+  Worker function to execute a caller provided function on all enabled APs.
+
+  @param[in]     Procedure               A pointer to the function to be run on
+                                         enabled APs of the system.
+  @param[in]     TimeoutInMicroseconds   Indicates the time limit in microseconds for
+                                         APs to return from Procedure, either for
+                                         blocking or non-blocking mode.
+  @param[in,out] ProcedureArguments      The parameter passed into Procedure for
+                                         all APs.
+  @param[in,out] Token                   This is an optional parameter that allows the caller to execute the
+                                         procedure in a blocking or non-blocking fashion. If it is NULL the
+                                         call is blocking, and the call will not return until the AP has
+                                         completed the procedure. If the token is not NULL, the call will
+                                         return immediately. The caller can check whether the procedure has
+                                         completed with CheckOnProcedure or WaitForProcedure.
+  @param[in,out] CPUStatus               This optional pointer may be used to get the status code returned
+                                         by Procedure when it completes execution on the target AP, or with
+                                         EFI_TIMEOUT if the Procedure fails to complete within the optional
+                                         timeout. The implementation will update this variable with
+                                         EFI_NOT_READY prior to starting Procedure on the target AP.
+
+
+  @retval EFI_SUCCESS             In blocking mode, all APs have finished before
+                                  the timeout expired.
+  @retval EFI_SUCCESS             In non-blocking mode, function has been dispatched
+                                  to all enabled APs.
+  @retval others                  Failed to Startup all APs.
+
+**/
+EFI_STATUS
+InternalSmmStartupAllAPs (
+  IN       EFI_AP_PROCEDURE2             Procedure,
+  IN       UINTN                         TimeoutInMicroseconds,
+  IN OUT   VOID                          *ProcedureArguments OPTIONAL,
+  IN OUT   SPIN_LOCK                     *Token,
+  IN OUT   EFI_STATUS                    *CPUStatus
+  )
+{
+  UINTN               Index;
+  UINTN               CpuCount;
+
+  if ((TimeoutInMicroseconds != 0) && ((mSmmMp.Attributes & EFI_MM_MP_TIMEOUT_SUPPORTED) == 0)) {
+    return EFI_INVALID_PARAMETER;
+  }
+  if (Procedure == NULL) {
+    return EFI_INVALID_PARAMETER;
+  }
+  if (IsCurrentToken ((UINTN)-1, Token)) {
+    return EFI_ALREADY_STARTED;
+  }
+
+  CpuCount = 0;
+  for (Index = mMaxNumberOfCpus; Index-- > 0;) {
+    if (*mSmmMpSyncData->CpuData[Index].Present && (Index != gSmmCpuPrivate->SmmCoreEntryContext.CurrentlyExecutingCpu)) {
+      CpuCount ++;
+
+      if (gSmmCpuPrivate->Operation[Index] == SmmCpuRemove) {
+        return EFI_INVALID_PARAMETER;
+      }
+
+      if (!AcquireSpinLockOrFail(mSmmMpSyncData->CpuData[Index].Busy)) {
+        return EFI_NOT_READY;
+      }
+      ReleaseSpinLock (mSmmMpSyncData->CpuData[Index].Busy);
+    }
+  }
+  if (CpuCount == 0) {
+    return EFI_NOT_STARTED;
+  }
+
+  if (Token == NULL) {
+    //
+    // Make sure all BUSY should be acquired.
+    //
+    for (Index = mMaxNumberOfCpus; Index-- > 0;) {
+      if (Index != gSmmCpuPrivate->SmmCoreEntryContext.CurrentlyExecutingCpu && *(mSmmMpSyncData->CpuData[Index].Present)) {
+        AcquireSpinLock (mSmmMpSyncData->CpuData[Index].Busy);
+      }
+    }
+  } else {
+    if (!AcquireSpinLockOrFail (Token)) {
+      return EFI_INVALID_PARAMETER;
+    }
+
+    //
+    // Make sure all BUSY should be acquired.
+    //
+    for (Index = mMaxNumberOfCpus; Index-- > 0;) {
+      if (Index != gSmmCpuPrivate->SmmCoreEntryContext.CurrentlyExecutingCpu && *(mSmmMpSyncData->CpuData[Index].Present)) {
+        if (!AcquireSpinLockOrFail (mSmmMpSyncData->CpuData[Index].Busy)) {
+          DEBUG((DEBUG_ERROR, "Can't acquire mSmmMpSyncData->CpuData[%d].Busy\n", Index));
+
+          //
+          // Release BUSY accquired before.
+          //
+          for (CpuCount = mMaxNumberOfCpus; CpuCount -- > Index;) {
+            if (CpuCount != gSmmCpuPrivate->SmmCoreEntryContext.CurrentlyExecutingCpu && *(mSmmMpSyncData->CpuData[CpuCount].Present)) {
+              ReleaseSpinLock (mSmmMpSyncData->CpuData[CpuCount].Busy);
+            }
+          }
+
+          ReleaseSpinLock (Token);
+          return EFI_INVALID_PARAMETER;
+        }
+      }
+    }
+  }
+
+  for (Index = mMaxNumberOfCpus; Index-- > 0;) {
+    if (Index != gSmmCpuPrivate->SmmCoreEntryContext.CurrentlyExecutingCpu && *(mSmmMpSyncData->CpuData[Index].Present)) {
+      mSmmMpSyncData->CpuData[Index].Procedure = (EFI_AP_PROCEDURE2) Procedure;
+      mSmmMpSyncData->CpuData[Index].Parameter = ProcedureArguments;
+      mSmmMpSyncData->CpuData[Index].Token     = Token;
+      if (CPUStatus != NULL) {
+        mSmmMpSyncData->CpuData[Index].Status    = CPUStatus + Index;
+        if (mSmmMpSyncData->CpuData[Index].Status != NULL) {
+          *mSmmMpSyncData->CpuData[Index].Status = EFI_NOT_READY;
+        }
+      }
+    } else {
+      //
+      // PI spec requirement:
+      // For every excluded processor, the array entry must contain a value of EFI_NOT_STARTED.
+      //
+      if (CPUStatus != NULL) {
+        *(CPUStatus + Index) = EFI_NOT_STARTED;
+      }
+    }
+  }
+
+  ReleaseAllAPs ();
+
+  if (Token == NULL) {
+    //
+    // Make sure all APs have completed their tasks.
+    //
+    IsTaskFinishInAllAPs (TRUE);
+  }
+
+  return EFI_SUCCESS;
+}
+
+/**
+  Wrapper for Procedures.
+
+  @param[in]  Buffer              Pointer to PROCEDURE_WRAPPER buffer.
+
+**/
+EFI_STATUS
+EFIAPI
+ProcedureWrapper (
+  IN OUT VOID *Buffer
+  )
+{
+  PROCEDURE_WRAPPER *Wrapper;
+
+  Wrapper = Buffer;
+  Wrapper->Procedure (Wrapper->ProcedureArgument);
+
   return EFI_SUCCESS;
 }
 
@@ -995,7 +1395,24 @@ SmmBlockingStartupThisAp (
   IN OUT  VOID                      *ProcArguments OPTIONAL
   )
 {
-  return InternalSmmStartupThisAp(Procedure, CpuIndex, ProcArguments, TRUE);
+  PROCEDURE_WRAPPER *Wrapper;
+  EFI_STATUS        Status;
+
+  Wrapper = AllocatePool (sizeof(PROCEDURE_WRAPPER));
+  if (Wrapper == NULL) {
+    return EFI_OUT_OF_RESOURCES;
+  }
+  Wrapper->Procedure = Procedure;
+  Wrapper->ProcedureArgument = ProcArguments;
+
+  //
+  // Use wrapper function to convert EFI_AP_PROCEDURE to EFI_AP_PROCEDURE2.
+  //
+  Status = InternalSmmStartupThisAp(ProcedureWrapper, CpuIndex, Wrapper, NULL, 0, NULL);
+
+  FreePool (Wrapper);
+
+  return Status;
 }
 
 /**
@@ -1020,7 +1437,37 @@ SmmStartupThisAp (
   IN OUT  VOID                      *ProcArguments OPTIONAL
   )
 {
-  return InternalSmmStartupThisAp(Procedure, CpuIndex, ProcArguments, FeaturePcdGet (PcdCpuSmmBlockStartupThisAp));
+  SPIN_LOCK               *CpuToken;
+  PROCEDURE_WRAPPER       *Wrapper;
+  EFI_STATUS              Status;
+
+  Wrapper = AllocatePool (sizeof(PROCEDURE_WRAPPER));
+  if (Wrapper == NULL) {
+    return EFI_OUT_OF_RESOURCES;
+  }
+  Wrapper->Procedure = Procedure;
+  Wrapper->ProcedureArgument = ProcArguments;
+
+  if (FeaturePcdGet (PcdCpuSmmBlockStartupThisAp)) {
+    CpuToken = NULL;
+  } else {
+    CpuToken = mApTokenLock[CpuIndex];
+  }
+
+  //
+  // Use wrapper function to convert EFI_AP_PROCEDURE to EFI_AP_PROCEDURE2.
+  //
+  Status = InternalSmmStartupThisAp(ProcedureWrapper, CpuIndex, Wrapper, CpuToken, 0, NULL);
+
+  //
+  // Free wrapper buffer for block mode.
+  // Non-block mode frees buffer in ApHandler function.
+  //
+  if (CpuToken == NULL) {
+    FreePool (Wrapper);
+  }
+
+  return Status;
 }
 
 /**
@@ -1112,6 +1559,13 @@ SmiRendezvous (
   Cr2 = 0;
   SaveCr2 (&Cr2);
 
+  //
+  // Call the user register Startup function first.
+  //
+  if (mSmmMpSyncData->StartupProcedure != NULL) {
+    mSmmMpSyncData->StartupProcedure (mSmmMpSyncData->StartupProcArgs);
+  }
+
   //
   // Perform CPU specific entry hooks
   //
@@ -1273,13 +1727,17 @@ InitializeSmmCpuSemaphores (
   UINTN                      Pages;
   UINTN                      *SemaphoreBlock;
   UINTN                      SemaphoreAddr;
+  UINTN                      ApTokenSize;
+  UINTN                      CpuIndex; 
 
   SemaphoreSize   = GetSpinLockProperties ();
   ProcessorCount = gSmmCpuPrivate->SmmCoreEntryContext.NumberOfCpus;
   GlobalSemaphoresSize = (sizeof (SMM_CPU_SEMAPHORE_GLOBAL) / sizeof (VOID *)) * SemaphoreSize;
   CpuSemaphoresSize    = (sizeof (SMM_CPU_SEMAPHORE_CPU) / sizeof (VOID *)) * ProcessorCount * SemaphoreSize;
-  TotalSize = GlobalSemaphoresSize + CpuSemaphoresSize;
+  ApTokenSize = SemaphoreSize * ProcessorCount;
+  TotalSize = GlobalSemaphoresSize + CpuSemaphoresSize + ApTokenSize;
   DEBUG((EFI_D_INFO, "One Semaphore Size    = 0x%x\n", SemaphoreSize));
+  DEBUG((EFI_D_INFO, "Token Spin Lock Size = 0x%x\n", ApTokenSize));
   DEBUG((EFI_D_INFO, "Total Semaphores Size = 0x%x\n", TotalSize));
   Pages = EFI_SIZE_TO_PAGES (TotalSize);
   SemaphoreBlock = AllocatePages (Pages);
@@ -1305,10 +1763,19 @@ InitializeSmmCpuSemaphores (
   mSmmCpuSemaphores.SemaphoreCpu.Run     = (UINT32 *)SemaphoreAddr;
   SemaphoreAddr += ProcessorCount * SemaphoreSize;
   mSmmCpuSemaphores.SemaphoreCpu.Present = (BOOLEAN *)SemaphoreAddr;
+  SemaphoreAddr += ProcessorCount * SemaphoreSize;
+  mSmmCpuSemaphores.SemaphoreCpu.Token = (SPIN_LOCK *)SemaphoreAddr;
 
   mPFLock                       = mSmmCpuSemaphores.SemaphoreGlobal.PFLock;
   mConfigSmmCodeAccessCheckLock = mSmmCpuSemaphores.SemaphoreGlobal.CodeAccessCheckLock;
 
+  mApTokenLock = AllocatePool (sizeof (SPIN_LOCK *) * gSmmCpuPrivate->SmmCoreEntryContext.NumberOfCpus);
+  ASSERT (mApTokenLock != NULL);
+  for (CpuIndex = 0; CpuIndex < gSmmCpuPrivate->SmmCoreEntryContext.NumberOfCpus; CpuIndex ++) {
+    mApTokenLock[CpuIndex] = (SPIN_LOCK *)((UINTN)mSmmCpuSemaphores.SemaphoreCpu.Token + SemaphoreSize * CpuIndex);
+    InitializeSpinLock (mApTokenLock[CpuIndex]);
+  }
+
   mSemaphoreSize = SemaphoreSize;
 }
 
@@ -1469,3 +1936,40 @@ RegisterSmmEntry (
   gSmmCpuPrivate->SmmCoreEntry = SmmEntryPoint;
   return EFI_SUCCESS;
 }
+
+/**
+
+  Register the SMM Foundation entry point.
+
+  @param[in]      Procedure            A pointer to the code stream to be run on the designated target AP
+                                       of the system. Type EFI_AP_PROCEDURE is defined below in Volume 2
+                                       with the related definitions of
+                                       EFI_MP_SERVICES_PROTOCOL.StartupAllAPs.
+                                       If caller may pass a value of NULL to deregister any existing
+                                       startup procedure.
+  @param[in]      ProcedureArguments   Allows the caller to pass a list of parameters to the code that is
+                                       run by the AP. It is an optional common mailbox between APs and
+                                       the caller to share information
+
+  @retval EFI_SUCCESS                  The Procedure has been set successfully.
+  @retval EFI_INVALID_PARAMETER        The Procedure is NULL but ProcedureArguments not NULL.
+
+**/
+EFI_STATUS
+RegisterStartupProcedure (
+  IN EFI_AP_PROCEDURE    Procedure,
+  IN VOID                *ProcedureArguments OPTIONAL
+  )
+{
+  if (Procedure == NULL && ProcedureArguments != NULL) {
+    return EFI_INVALID_PARAMETER;
+  }
+  if (mSmmMpSyncData == NULL) {
+    return EFI_NOT_READY;
+  }
+
+  mSmmMpSyncData->StartupProcedure = Procedure;
+  mSmmMpSyncData->StartupProcArgs  = ProcedureArguments;
+
+  return EFI_SUCCESS;
+}
diff --git a/UefiCpuPkg/PiSmmCpuDxeSmm/PiSmmCpuDxeSmm.c b/UefiCpuPkg/PiSmmCpuDxeSmm/PiSmmCpuDxeSmm.c
index 2f7d777ee7..dd1b3be0f5 100644
--- a/UefiCpuPkg/PiSmmCpuDxeSmm/PiSmmCpuDxeSmm.c
+++ b/UefiCpuPkg/PiSmmCpuDxeSmm/PiSmmCpuDxeSmm.c
@@ -996,6 +996,17 @@ PiCpuSmmEntry (
                     );
   ASSERT_EFI_ERROR (Status);
 
+  //
+  // Install the SMM Mp Protocol into SMM protocol database
+  //
+  Status = gSmst->SmmInstallProtocolInterface (
+                    &mSmmCpuHandle,
+                    &gEfiMmMpProtocolGuid,
+                    EFI_NATIVE_INTERFACE,
+                    &mSmmMp
+                    );
+  ASSERT_EFI_ERROR (Status);
+
   //
   // Expose address of CPU Hot Plug Data structure if CPU hot plug is supported.
   //
diff --git a/UefiCpuPkg/PiSmmCpuDxeSmm/PiSmmCpuDxeSmm.h b/UefiCpuPkg/PiSmmCpuDxeSmm/PiSmmCpuDxeSmm.h
index 2bb35a424d..5df09687e1 100644
--- a/UefiCpuPkg/PiSmmCpuDxeSmm/PiSmmCpuDxeSmm.h
+++ b/UefiCpuPkg/PiSmmCpuDxeSmm/PiSmmCpuDxeSmm.h
@@ -20,6 +20,7 @@ SPDX-License-Identifier: BSD-2-Clause-Patent
 #include <Protocol/SmmReadyToLock.h>
 #include <Protocol/SmmCpuService.h>
 #include <Protocol/SmmMemoryAttribute.h>
+#include <Protocol/MmMp.h>
 
 #include <Guid/AcpiS3Context.h>
 #include <Guid/MemoryAttributesTable.h>
@@ -221,11 +222,20 @@ typedef struct {
   EFI_SMM_CONFIGURATION_PROTOCOL  SmmConfiguration;
 } SMM_CPU_PRIVATE_DATA;
 
+//
+// Wrapper used to convert EFI_AP_PROCEDURE2 and EFI_AP_PROCEDURE.
+//
+typedef struct {
+  EFI_AP_PROCEDURE  Procedure;
+  VOID              *ProcedureArgument;
+} PROCEDURE_WRAPPER;
+
 extern SMM_CPU_PRIVATE_DATA  *gSmmCpuPrivate;
 extern CPU_HOT_PLUG_DATA      mCpuHotPlugData;
 extern UINTN                  mMaxNumberOfCpus;
 extern UINTN                  mNumberOfCpus;
 extern EFI_SMM_CPU_PROTOCOL   mSmmCpu;
+extern EFI_MM_MP_PROTOCOL     mSmmMp;
 
 ///
 /// The mode of the CPU at the time an SMI occurs
@@ -363,10 +373,12 @@ SmmRelocationSemaphoreComplete (
 ///
 typedef struct {
   SPIN_LOCK                         *Busy;
-  volatile EFI_AP_PROCEDURE         Procedure;
+  volatile EFI_AP_PROCEDURE2        Procedure;
   volatile VOID                     *Parameter;
   volatile UINT32                   *Run;
   volatile BOOLEAN                  *Present;
+  SPIN_LOCK                         *Token;
+  EFI_STATUS                        *Status;
 } SMM_CPU_DATA_BLOCK;
 
 typedef enum {
@@ -388,6 +400,8 @@ typedef struct {
   volatile SMM_CPU_SYNC_MODE    EffectiveSyncMode;
   volatile BOOLEAN              SwitchBsp;
   volatile BOOLEAN              *CandidateBsp;
+  EFI_AP_PROCEDURE              StartupProcedure;
+  VOID                          *StartupProcArgs;
 } SMM_DISPATCHER_MP_SYNC_DATA;
 
 #define SMM_PSD_OFFSET              0xfb00
@@ -410,6 +424,7 @@ typedef struct {
   SPIN_LOCK                         *Busy;
   volatile UINT32                   *Run;
   volatile BOOLEAN                  *Present;
+  SPIN_LOCK                         *Token;
 } SMM_CPU_SEMAPHORE_CPU;
 
 ///
@@ -439,6 +454,7 @@ extern SPIN_LOCK                           *mConfigSmmCodeAccessCheckLock;
 extern EFI_SMRAM_DESCRIPTOR                *mSmmCpuSmramRanges;
 extern UINTN                               mSmmCpuSmramRangeCount;
 extern UINT8                               mPhysicalAddressBits;
+extern SPIN_LOCK                           **mApTokenLock;
 
 //
 // Copy of the PcdPteMemoryEncryptionAddressOrMask
@@ -1259,4 +1275,146 @@ RestoreCr2 (
   IN UINTN  Cr2
   );
 
+/**
+  Schedule a procedure to run on the specified CPU.
+
+  @param[in]       Procedure                The address of the procedure to run
+  @param[in]       CpuIndex                 Target CPU Index
+  @param[in,out]   ProcArguments            The parameter to pass to the procedure
+  @param[in,out]   Token                    This is an optional parameter that allows the caller to execute the
+                                            procedure in a blocking or non-blocking fashion. If it is NULL the
+                                            call is blocking, and the call will not return until the AP has
+                                            completed the procedure. If the token is not NULL, the call will
+                                            return immediately. The caller can check whether the procedure has
+                                            completed with CheckOnProcedure or WaitForProcedure.
+  @param[in]       TimeoutInMicroseconds    Indicates the time limit in microseconds for the APs to finish
+                                            execution of Procedure, either for blocking or non-blocking mode.
+                                            Zero means infinity. If the timeout expires before all APs return
+                                            from Procedure, then Procedure on the failed APs is terminated. If
+                                            the timeout expires in blocking mode, the call returns EFI_TIMEOUT.
+                                            If the timeout expires in non-blocking mode, the timeout determined
+                                            can be through CheckOnProcedure or WaitForProcedure.
+                                            Note that timeout support is optional. Whether an implementation
+                                            supports this feature can be determined via the Attributes data
+                                            member.
+  @param[in,out]   CPUStatus                This optional pointer may be used to get the status code returned
+                                            by Procedure when it completes execution on the target AP, or with
+                                            EFI_TIMEOUT if the Procedure fails to complete within the optional
+                                            timeout. The implementation will update this variable with
+                                            EFI_NOT_READY prior to starting Procedure on the target AP.
+
+
+  @retval EFI_INVALID_PARAMETER    CpuNumber not valid
+  @retval EFI_INVALID_PARAMETER    CpuNumber specifying BSP
+  @retval EFI_INVALID_PARAMETER    The AP specified by CpuNumber did not enter SMM
+  @retval EFI_INVALID_PARAMETER    The AP specified by CpuNumber is busy
+  @retval EFI_SUCCESS              The procedure has been successfully scheduled
+
+**/
+EFI_STATUS
+InternalSmmStartupThisAp (
+  IN      EFI_AP_PROCEDURE2              Procedure,
+  IN      UINTN                          CpuIndex,
+  IN OUT  VOID                           *ProcArguments OPTIONAL,
+  IN      SPIN_LOCK                      *Token,
+  IN      UINTN                          TimeoutInMicroseconds,
+  IN OUT  EFI_STATUS                     *CpuStatus
+  );
+
+/**
+  Checks whether the input token is the current used token.
+
+  @param[in]  CpuIndex   Cpu Index.
+  @param[in]  Token      This parameter describes the token that was passed into DispatchProcedure or
+                         BroadcastProcedure.
+
+  @retval TRUE           The input token is the current used token.
+  @retval FALSE          The input token is not the current used token.
+**/
+BOOLEAN
+IsCurrentToken (
+  IN UINTN               CpuIndex,
+  IN SPIN_LOCK           *Token
+  );
+
+/**
+  Checks status of specified AP.
+
+  This function checks whether the specified AP has finished the task assigned
+  by StartupThisAP(), and whether timeout expires.
+
+  @param[in]  Token             This parameter describes the token that was passed into DispatchProcedure or
+                                BroadcastProcedure.
+
+  @retval EFI_SUCCESS           Specified AP has finished task assigned by StartupThisAPs().
+  @retval EFI_NOT_READY         Specified AP has not finished task and timeout has not expired.
+**/
+EFI_STATUS
+IsApReady (
+  IN SPIN_LOCK  *Token
+  );
+
+/**
+  Worker function to execute a caller provided function on all enabled APs.
+
+  @param[in]     Procedure               A pointer to the function to be run on
+                                         enabled APs of the system.
+  @param[in]     TimeoutInMicroseconds   Indicates the time limit in microseconds for
+                                         APs to return from Procedure, either for
+                                         blocking or non-blocking mode.
+  @param[in,out] ProcedureArgument       The parameter passed into Procedure for
+                                         all APs.
+  @param[in,out] Token                   This is an optional parameter that allows the caller to execute the
+                                         procedure in a blocking or non-blocking fashion. If it is NULL the
+                                         call is blocking, and the call will not return until the AP has
+                                         completed the procedure. If the token is not NULL, the call will
+                                         return immediately. The caller can check whether the procedure has
+                                         completed with CheckOnProcedure or WaitForProcedure.
+  @param[in,out] CPUStatus               This optional pointer may be used to get the status code returned
+                                         by Procedure when it completes execution on the target AP, or with
+                                         EFI_TIMEOUT if the Procedure fails to complete within the optional
+                                         timeout. The implementation will update this variable with
+                                         EFI_NOT_READY prior to starting Procedure on the target AP.
+
+
+  @retval EFI_SUCCESS             In blocking mode, all APs have finished before
+                                  the timeout expired.
+  @retval EFI_SUCCESS             In non-blocking mode, function has been dispatched
+                                  to all enabled APs.
+  @retval others                  Failed to Startup all APs.
+
+**/
+EFI_STATUS
+InternalSmmStartupAllAPs (
+  IN       EFI_AP_PROCEDURE2             Procedure,
+  IN       UINTN                         TimeoutInMicroseconds,
+  IN OUT   VOID                          *ProcedureArguments OPTIONAL,
+  IN OUT   SPIN_LOCK                     *Token,
+  IN OUT   EFI_STATUS                    *CPUStatus
+  );
+
+/**
+
+  Register the SMM Foundation entry point.
+
+  @param[in]      Procedure            A pointer to the code stream to be run on the designated target AP
+                                       of the system. Type EFI_AP_PROCEDURE is defined below in Volume 2
+                                       with the related definitions of
+                                       EFI_MP_SERVICES_PROTOCOL.StartupAllAPs.
+                                       If caller may pass a value of NULL to deregister any existing
+                                       startup procedure.
+  @param[in,out]  ProcedureArguments   Allows the caller to pass a list of parameters to the code that is
+                                       run by the AP. It is an optional common mailbox between APs and
+                                       the caller to share information
+
+  @retval EFI_SUCCESS                  The Procedure has been set successfully.
+  @retval EFI_INVALID_PARAMETER        The Procedure is NULL but ProcedureArguments not NULL.
+
+**/
+EFI_STATUS
+RegisterStartupProcedure (
+  IN EFI_AP_PROCEDURE    Procedure,
+  IN VOID                *ProcedureArguments OPTIONAL
+  );
+
 #endif
diff --git a/UefiCpuPkg/PiSmmCpuDxeSmm/PiSmmCpuDxeSmm.inf b/UefiCpuPkg/PiSmmCpuDxeSmm/PiSmmCpuDxeSmm.inf
index 466c568d49..da0308c47f 100644
--- a/UefiCpuPkg/PiSmmCpuDxeSmm/PiSmmCpuDxeSmm.inf
+++ b/UefiCpuPkg/PiSmmCpuDxeSmm/PiSmmCpuDxeSmm.inf
@@ -40,6 +40,8 @@
   SmmProfileInternal.h
   SmramSaveState.c
   SmmCpuMemoryManagement.c
+  SmmMp.h
+  SmmMp.c
 
 [Sources.Ia32]
   Ia32/Semaphore.c
@@ -105,6 +107,7 @@
   gEfiSmmReadyToLockProtocolGuid           ## NOTIFY
   gEfiSmmCpuServiceProtocolGuid            ## PRODUCES
   gEdkiiSmmMemoryAttributeProtocolGuid     ## PRODUCES
+  gEfiMmMpProtocolGuid                    ## PRODUCES
 
 [Guids]
   gEfiAcpiVariableGuid                     ## SOMETIMES_CONSUMES ## HOB # it is used for S3 boot.
diff --git a/UefiCpuPkg/PiSmmCpuDxeSmm/SmmMp.c b/UefiCpuPkg/PiSmmCpuDxeSmm/SmmMp.c
new file mode 100644
index 0000000000..bfbc78691f
--- /dev/null
+++ b/UefiCpuPkg/PiSmmCpuDxeSmm/SmmMp.c
@@ -0,0 +1,372 @@
+/** @file
+SMM MP protocol implementation
+
+Copyright (c) 2019, Intel Corporation. All rights reserved.<BR>
+
+SPDX-License-Identifier: BSD-2-Clause-Patent
+
+**/
+
+#include "PiSmmCpuDxeSmm.h"
+#include "SmmMp.h"
+
+///
+/// SMM MP Protocol instance
+///
+EFI_MM_MP_PROTOCOL  mSmmMp  = {
+  EFI_MM_MP_PROTOCOL_REVISION,
+  0,
+  SmmMpGetNumberOfProcessors,
+  SmmMpDispatchProcedure,
+  SmmMpBroadcastProcedure,
+  SmmMpSetStartupProcedure,
+  SmmMpCheckForProcedure,
+  SmmMpWaitForProcedure
+};
+
+/**
+  Service to retrieves the number of logical processor in the platform.
+
+  @param[in]  This                The EFI_MM_MP_PROTOCOL instance.
+  @param[out] NumberOfProcessors  Pointer to the total number of logical processors in the system,
+                                  including the BSP and all APs.
+
+  @retval EFI_SUCCESS             The number of processors was retrieved successfully
+  @retval EFI_INVALID_PARAMETER   NumberOfProcessors is NULL
+**/
+EFI_STATUS
+EFIAPI
+SmmMpGetNumberOfProcessors (
+  IN CONST EFI_MM_MP_PROTOCOL   *This,
+  OUT      UINTN                *NumberOfProcessors
+  )
+{
+  if (NumberOfProcessors == NULL) {
+    return EFI_INVALID_PARAMETER;
+  }
+
+  *NumberOfProcessors = gSmmCpuPrivate->SmmCoreEntryContext.NumberOfCpus;
+
+  return EFI_SUCCESS;
+}
+
+
+/**
+  This service allows the caller to invoke a procedure one of the application processors (AP). This
+  function uses an optional token parameter to support blocking and non-blocking modes. If the token
+  is passed into the call, the function will operate in a non-blocking fashion and the caller can
+  check for completion with CheckOnProcedure or WaitForProcedure.
+
+  @param[in]     This                   The EFI_MM_MP_PROTOCOL instance.
+  @param[in]     Procedure              A pointer to the procedure to be run on the designated target
+                                        AP of the system. Type EFI_AP_PROCEDURE2 is defined below in
+                                        related definitions.
+  @param[in]     CpuNumber              The zero-based index of the processor number of the target
+                                        AP, on which the code stream is supposed to run. If the number
+                                        points to the calling processor then it will not run the
+                                        supplied code.
+  @param[in]     TimeoutInMicroseconds  Indicates the time limit in microseconds for this AP to
+                                        finish execution of Procedure, either for blocking or
+                                        non-blocking mode. Zero means infinity. If the timeout
+                                        expires before this AP returns from Procedure, then Procedure
+                                        on the AP is terminated. If the timeout expires in blocking
+                                        mode, the call returns EFI_TIMEOUT. If the timeout expires
+                                        in non-blocking mode, the timeout determined can be through
+                                        CheckOnProcedure or WaitForProcedure.
+                                        Note that timeout support is optional. Whether an
+                                        implementation supports this feature, can be determined via
+                                        the Attributes data member.
+  @param[in,out] ProcedureArguments     Allows the caller to pass a list of parameters to the code
+                                        that is run by the AP. It is an optional common mailbox
+                                        between APs and the caller to share information.
+  @param[in,out] Token                  This is parameter is broken into two components:
+                                        1.Token->Completion is an optional parameter that allows the
+                                        caller to execute the procedure in a blocking or non-blocking
+                                        fashion. If it is NULL the call is blocking, and the call will
+                                        not return until the AP has completed the procedure. If the
+                                        token is not NULL, the call will return immediately. The caller
+                                        can check whether the procedure has completed with
+                                        CheckOnProcedure or WaitForProcedure.
+                                        2.Token->Status The implementation updates the address pointed
+                                        at by this variable with the status code returned by Procedure
+                                        when it completes execution on the target AP, or with EFI_TIMEOUT
+                                        if the Procedure fails to complete within the optional timeout.
+                                        The implementation will update this variable with EFI_NOT_READY
+                                        prior to starting Procedure on the target AP
+  @param[in,out] CPUStatus              This optional pointer may be used to get the status code returned
+                                        by Procedure when it completes execution on the target AP, or with
+                                        EFI_TIMEOUT if the Procedure fails to complete within the optional
+                                        timeout. The implementation will update this variable with
+                                        EFI_NOT_READY prior to starting Procedure on the target AP.
+
+  @retval EFI_SUCCESS                   In the blocking case, this indicates that Procedure has completed
+                                        execution on the target AP.
+                                        In the non-blocking case this indicates that the procedure has
+                                        been successfully scheduled for execution on the target AP.
+  @retval EFI_INVALID_PARAMETER         The input arguments are out of range. Either the target AP is the
+                                        caller of the function, or the Procedure or Token is NULL
+  @retval EFI_NOT_READY                 If the target AP is busy executing another procedure
+  @retval EFI_ALREADY_STARTED           Token is already in use for another procedure
+  @retval EFI_TIMEOUT                   In blocking mode, the timeout expired before the specified AP
+                                        has finished
+  @retval EFI_OUT_OF_RESOURCES          Could not allocate a required resource.
+
+**/
+EFI_STATUS
+EFIAPI
+SmmMpDispatchProcedure (
+  IN CONST EFI_MM_MP_PROTOCOL            *This,
+  IN       EFI_AP_PROCEDURE2             Procedure,
+  IN       UINTN                         CpuNumber,
+  IN       UINTN                         TimeoutInMicroseconds,
+  IN OUT   VOID                          *ProcedureArguments OPTIONAL,
+  IN OUT   MM_COMPLETION                 *Token,
+  IN OUT   EFI_STATUS                    *CPUStatus
+  )
+{
+  SPIN_LOCK      *CpuToken;
+
+  if (Token != NULL) {
+    CpuToken = AllocatePool (sizeof (SPIN_LOCK));
+    ASSERT (CpuToken != NULL);
+    if (CpuToken == NULL) {
+      return EFI_OUT_OF_RESOURCES;
+    }
+    InitializeSpinLock ((SPIN_LOCK *)(CpuToken));
+
+    *Token = (MM_COMPLETION)CpuToken;
+  }
+
+  return InternalSmmStartupThisAp (
+    Procedure,
+    CpuNumber,
+    ProcedureArguments,
+    Token != NULL ? CpuToken : NULL,
+    TimeoutInMicroseconds,
+    CPUStatus
+    );
+}
+
+/**
+  This service allows the caller to invoke a procedure on all running application processors (AP)
+  except the caller. This function uses an optional token parameter to support blocking and
+  nonblocking modes. If the token is passed into the call, the function will operate in a non-blocking
+  fashion and the caller can check for completion with CheckOnProcedure or WaitForProcedure.
+
+  It is not necessary for the implementation to run the procedure on every processor on the platform.
+  Processors that are powered down in such a way that they cannot respond to interrupts, may be
+  excluded from the broadcast.
+
+
+  @param[in]     This                   The EFI_MM_MP_PROTOCOL instance.
+  @param[in]     Procedure              A pointer to the code stream to be run on the APs that have
+                                        entered MM. Type EFI_AP_PROCEDURE is defined below in related
+                                        definitions.
+  @param[in]     TimeoutInMicroseconds  Indicates the time limit in microseconds for the APs to finish
+                                        execution of Procedure, either for blocking or non-blocking mode.
+                                        Zero means infinity. If the timeout expires before all APs return
+                                        from Procedure, then Procedure on the failed APs is terminated. If
+                                        the timeout expires in blocking mode, the call returns EFI_TIMEOUT.
+                                        If the timeout expires in non-blocking mode, the timeout determined
+                                        can be through CheckOnProcedure or WaitForProcedure.
+                                        Note that timeout support is optional. Whether an implementation
+                                        supports this feature can be determined via the Attributes data
+                                        member.
+  @param[in,out] ProcedureArguments     Allows the caller to pass a list of parameters to the code
+                                        that is run by the AP. It is an optional common mailbox
+                                        between APs and the caller to share information.
+  @param[in,out] Token                  This is parameter is broken into two components:
+                                        1.Token->Completion is an optional parameter that allows the
+                                        caller to execute the procedure in a blocking or non-blocking
+                                        fashion. If it is NULL the call is blocking, and the call will
+                                        not return until the AP has completed the procedure. If the
+                                        token is not NULL, the call will return immediately. The caller
+                                        can check whether the procedure has completed with
+                                        CheckOnProcedure or WaitForProcedure.
+                                        2.Token->Status The implementation updates the address pointed
+                                        at by this variable with the status code returned by Procedure
+                                        when it completes execution on the target AP, or with EFI_TIMEOUT
+                                        if the Procedure fails to complete within the optional timeout.
+                                        The implementation will update this variable with EFI_NOT_READY
+                                        prior to starting Procedure on the target AP
+  @param[in,out] CPUStatus              This optional pointer may be used to get the individual status
+                                        returned by every AP that participated in the broadcast. This
+                                        parameter if used provides the base address of an array to hold
+                                        the EFI_STATUS value of each AP in the system. The size of the
+                                        array can be ascertained by the GetNumberOfProcessors function.
+                                        As mentioned above, the broadcast may not include every processor
+                                        in the system. Some implementations may exclude processors that
+                                        have been powered down in such a way that they are not responsive
+                                        to interrupts. Additionally the broadcast excludes the processor
+                                        which is making the BroadcastProcedure call. For every excluded
+                                        processor, the array entry must contain a value of EFI_NOT_STARTED
+
+  @retval EFI_SUCCESS                   In the blocking case, this indicates that Procedure has completed
+                                        execution on the APs.
+                                        In the non-blocking case this indicates that the procedure has
+                                        been successfully scheduled for execution on the APs.
+  @retval EFI_INVALID_PARAMETER         The Procedure or Token is NULL
+  @retval EFI_NOT_READY                 If the target AP is busy executing another procedure
+  @retval EFI_ALREADY_STARTED           Token is already in use for another procedure
+  @retval EFI_TIMEOUT                   In blocking mode, the timeout expired before the specified AP
+                                        has finished.
+  @retval EFI_OUT_OF_RESOURCES          Could not allocate a required resource.
+
+**/
+EFI_STATUS
+EFIAPI
+SmmMpBroadcastProcedure (
+  IN CONST EFI_MM_MP_PROTOCOL            *This,
+  IN       EFI_AP_PROCEDURE2             Procedure,
+  IN       UINTN                         TimeoutInMicroseconds,
+  IN OUT   VOID                          *ProcedureArguments OPTIONAL,
+  IN OUT   MM_COMPLETION                 *Token,
+  IN OUT   EFI_STATUS                    *CPUStatus
+  )
+{
+  SPIN_LOCK        *CpuToken;
+
+  if (Token != NULL) {
+    CpuToken = AllocatePool (sizeof (SPIN_LOCK));
+    ASSERT (CpuToken != NULL);
+    if (CpuToken == NULL) {
+      return EFI_OUT_OF_RESOURCES;
+    }
+    InitializeSpinLock ((SPIN_LOCK *)(CpuToken));
+
+    *Token = (MM_COMPLETION)CpuToken;
+  }
+
+  return InternalSmmStartupAllAPs(
+    Procedure,
+    TimeoutInMicroseconds,
+    ProcedureArguments,
+    Token != NULL ? CpuToken : NULL,
+    CPUStatus
+    );
+}
+
+
+/**
+  This service allows the caller to set a startup procedure that will be executed when an AP powers
+  up from a state where core configuration and context is lost. The procedure is execution has the
+  following properties:
+  1. The procedure executes before the processor is handed over to the operating system.
+  2. All processors execute the same startup procedure.
+  3. The procedure may run in parallel with other procedures invoked through the functions in this
+  protocol, or with processors that are executing an MM handler or running in the operating system.
+
+
+  @param[in]      This                 The EFI_MM_MP_PROTOCOL instance.
+  @param[in]      Procedure            A pointer to the code stream to be run on the designated target AP
+                                       of the system. Type EFI_AP_PROCEDURE is defined below in Volume 2
+                                       with the related definitions of
+                                       EFI_MP_SERVICES_PROTOCOL.StartupAllAPs.
+                                       If caller may pass a value of NULL to deregister any existing
+                                       startup procedure.
+  @param[in,out]  ProcedureArguments   Allows the caller to pass a list of parameters to the code that is
+                                       run by the AP. It is an optional common mailbox between APs and
+                                       the caller to share information
+
+  @retval EFI_SUCCESS                  The Procedure has been set successfully.
+  @retval EFI_INVALID_PARAMETER        The Procedure is NULL but ProcedureArguments not NULL.
+
+**/
+EFI_STATUS
+EFIAPI
+SmmMpSetStartupProcedure (
+  IN CONST EFI_MM_MP_PROTOCOL  *This,
+  IN       EFI_AP_PROCEDURE    Procedure,
+  IN OUT   VOID                *ProcedureArguments OPTIONAL
+  )
+{
+  return RegisterStartupProcedure (Procedure, ProcedureArguments);
+}
+
+/**
+  When non-blocking execution of a procedure on an AP is invoked with DispatchProcedure,
+  via the use of a token, this function can be used to check for completion of the procedure on the AP.
+  The function takes the token that was passed into the DispatchProcedure call. If the procedure
+  is complete, and therefore it is now possible to run another procedure on the same AP, this function
+  returns EFI_SUCESS. In this case the status returned by the procedure that executed on the AP is
+  returned in the token's Status field. If the procedure has not yet completed, then this function
+  returns EFI_NOT_READY.
+
+  When a non-blocking execution of a procedure is invoked with BroadcastProcedure, via the
+  use of a token, this function can be used to check for completion of the procedure on all the
+  broadcast APs. The function takes the token that was passed into the BroadcastProcedure
+  call. If the procedure is complete on all broadcast APs this function returns EFI_SUCESS. In this
+  case the Status field in the token passed into the function reflects the overall result of the
+  invocation, which may be EFI_SUCCESS, if all executions succeeded, or the first observed failure.
+  If the procedure has not yet completed on the broadcast APs, the function returns
+  EFI_NOT_READY.
+
+  @param[in]      This                 The EFI_MM_MP_PROTOCOL instance.
+  @param[in]      Token                This parameter describes the token that was passed into
+                                       DispatchProcedure or BroadcastProcedure.
+
+  @retval EFI_SUCCESS                  Procedure has completed.
+  @retval EFI_NOT_READY                The Procedure has not completed.
+  @retval EFI_INVALID_PARAMETER        Token or Token->Completion is NULL
+  @retval EFI_NOT_FOUND                Token is not currently in use for a non-blocking call
+
+**/
+EFI_STATUS
+EFIAPI
+SmmMpCheckForProcedure (
+  IN CONST EFI_MM_MP_PROTOCOL            *This,
+  IN       MM_COMPLETION                 Token
+  )
+{
+  if (Token == NULL) {
+    return EFI_INVALID_PARAMETER;
+  }
+
+  if (!IsCurrentToken ((UINTN)-1, (SPIN_LOCK *)Token)) {
+    return EFI_NOT_FOUND;
+  }
+
+  return IsApReady ((SPIN_LOCK *)Token);
+}
+
+/**
+  When a non-blocking execution of a procedure on an AP is invoked via DispatchProcedure,
+  this function will block the caller until the remote procedure has completed on the designated AP.
+  The non-blocking procedure invocation is identified by the Token parameter, which must match the
+  token that used when DispatchProcedure was called. Upon completion the status returned by
+  the procedure that executed on the AP is used to update the token's Status field.
+
+  When a non-blocking execution of a procedure on an AP is invoked via BroadcastProcedure
+  this function will block the caller until the remote procedure has completed on all of the APs that
+  entered MM. The non-blocking procedure invocation is identified by the Token parameter, which
+  must match the token that used when BroadcastProcedure was called. Upon completion the
+  overall status returned by the procedures that executed on the broadcast AP is used to update the
+  token's Status field. The overall status may be EFI_SUCCESS, if all executions succeeded, or the
+  first observed failure.
+
+
+  @param[in]      This                 The EFI_MM_MP_PROTOCOL instance.
+  @param[in]      Token                This parameter describes the token that was passed into
+                                       DispatchProcedure or BroadcastProcedure.
+
+  @retval EFI_SUCCESS                  Procedure has completed.
+  @retval EFI_INVALID_PARAMETER        Token or Token->Completion is NULL
+  @retval EFI_NOT_FOUND                Token is not currently in use for a non-blocking call
+
+**/
+EFI_STATUS
+EFIAPI
+SmmMpWaitForProcedure (
+  IN CONST EFI_MM_MP_PROTOCOL            *This,
+  IN       MM_COMPLETION                 Token
+  )
+{
+  EFI_STATUS    Status;
+
+  do {
+    Status = SmmMpCheckForProcedure (This, Token);
+  } while (Status == EFI_NOT_READY);
+
+  return Status;
+}
+
diff --git a/UefiCpuPkg/PiSmmCpuDxeSmm/SmmMp.h b/UefiCpuPkg/PiSmmCpuDxeSmm/SmmMp.h
new file mode 100644
index 0000000000..e0d823a4b1
--- /dev/null
+++ b/UefiCpuPkg/PiSmmCpuDxeSmm/SmmMp.h
@@ -0,0 +1,286 @@
+/** @file
+Include file for SMM MP protocol implementation.
+
+Copyright (c) 2019, Intel Corporation. All rights reserved.<BR>
+
+SPDX-License-Identifier: BSD-2-Clause-Patent
+
+**/
+
+#ifndef _SMM_MP_PROTOCOL_H_
+#define _SMM_MP_PROTOCOL_H_
+
+//
+// SMM MP Protocol function prototypes.
+//
+
+/**
+  Service to retrieves the number of logical processor in the platform.
+
+  @param[in]  This                The EFI_MM_MP_PROTOCOL instance.
+  @param[out] NumberOfProcessors  Pointer to the total number of logical processors in the system,
+                                  including the BSP and all APs.
+
+  @retval EFI_SUCCESS             The number of processors was retrieved successfully
+  @retval EFI_INVALID_PARAMETER   NumberOfProcessors is NULL
+**/
+
+EFI_STATUS
+EFIAPI
+SmmMpGetNumberOfProcessors (
+  IN CONST EFI_MM_MP_PROTOCOL   *This,
+  OUT      UINTN                *NumberOfProcessors
+  );
+
+
+/**
+  This service allows the caller to invoke a procedure one of the application processors (AP). This
+  function uses an optional token parameter to support blocking and non-blocking modes. If the token
+  is passed into the call, the function will operate in a non-blocking fashion and the caller can
+  check for completion with CheckOnProcedure or WaitForProcedure.
+
+  @param[in]     This                   The EFI_MM_MP_PROTOCOL instance.
+  @param[in]     Procedure              A pointer to the procedure to be run on the designated target
+                                        AP of the system. Type EFI_AP_PROCEDURE2 is defined below in
+                                        related definitions.
+  @param[in]     CpuNumber              The zero-based index of the processor number of the target
+                                        AP, on which the code stream is supposed to run. If the number
+                                        points to the calling processor then it will not run the
+                                        supplied code.
+  @param[in]     TimeoutInMicroseconds  Indicates the time limit in microseconds for this AP to
+                                        finish execution of Procedure, either for blocking or
+                                        non-blocking mode. Zero means infinity. If the timeout
+                                        expires before this AP returns from Procedure, then Procedure
+                                        on the AP is terminated. If the timeout expires in blocking
+                                        mode, the call returns EFI_TIMEOUT. If the timeout expires
+                                        in non-blocking mode, the timeout determined can be through
+                                        CheckOnProcedure or WaitForProcedure.
+                                        Note that timeout support is optional. Whether an
+                                        implementation supports this feature, can be determined via
+                                        the Attributes data member.
+  @param[in,out] ProcedureArguments     Allows the caller to pass a list of parameters to the code
+                                        that is run by the AP. It is an optional common mailbox
+                                        between APs and the caller to share information.
+  @param[in,out] Token                  This is parameter is broken into two components:
+                                        1.Token->Completion is an optional parameter that allows the
+                                        caller to execute the procedure in a blocking or non-blocking
+                                        fashion. If it is NULL the call is blocking, and the call will
+                                        not return until the AP has completed the procedure. If the
+                                        token is not NULL, the call will return immediately. The caller
+                                        can check whether the procedure has completed with
+                                        CheckOnProcedure or WaitForProcedure.
+                                        2.Token->Status The implementation updates the address pointed
+                                        at by this variable with the status code returned by Procedure
+                                        when it completes execution on the target AP, or with EFI_TIMEOUT
+                                        if the Procedure fails to complete within the optional timeout.
+                                        The implementation will update this variable with EFI_NOT_READY
+                                        prior to starting Procedure on the target AP
+  @param[in,out] CPUStatus              This optional pointer may be used to get the status code returned
+                                        by Procedure when it completes execution on the target AP, or with
+                                        EFI_TIMEOUT if the Procedure fails to complete within the optional
+                                        timeout. The implementation will update this variable with
+                                        EFI_NOT_READY prior to starting Procedure on the target AP.
+
+  @retval EFI_SUCCESS                   In the blocking case, this indicates that Procedure has completed
+                                        execution on the target AP.
+                                        In the non-blocking case this indicates that the procedure has
+                                        been successfully scheduled for execution on the target AP.
+  @retval EFI_INVALID_PARAMETER         The input arguments are out of range. Either the target AP is the
+                                        caller of the function, or the Procedure or Token is NULL
+  @retval EFI_NOT_READY                 If the target AP is busy executing another procedure
+  @retval EFI_ALREADY_STARTED           Token is already in use for another procedure
+  @retval EFI_TIMEOUT                   In blocking mode, the timeout expired before the specified AP
+                                        has finished
+  @retval EFI_OUT_OF_RESOURCES          Could not allocate a required resource.
+
+**/
+EFI_STATUS
+EFIAPI
+SmmMpDispatchProcedure (
+  IN CONST EFI_MM_MP_PROTOCOL             *This,
+  IN       EFI_AP_PROCEDURE2              Procedure,
+  IN       UINTN                          CpuNumber,
+  IN       UINTN                          TimeoutInMicroseconds,
+  IN OUT   VOID                           *ProcedureArguments OPTIONAL,
+  IN OUT   MM_COMPLETION                  *Token,
+  IN OUT   EFI_STATUS                     *CPUStatus
+  );
+
+/**
+  This service allows the caller to invoke a procedure on all running application processors (AP)
+  except the caller. This function uses an optional token parameter to support blocking and
+  nonblocking modes. If the token is passed into the call, the function will operate in a non-blocking
+  fashion and the caller can check for completion with CheckOnProcedure or WaitForProcedure.
+
+  It is not necessary for the implementation to run the procedure on every processor on the platform.
+  Processors that are powered down in such a way that they cannot respond to interrupts, may be
+  excluded from the broadcast.
+
+
+  @param[in]     This                   The EFI_MM_MP_PROTOCOL instance.
+  @param[in]     Procedure              A pointer to the code stream to be run on the APs that have
+                                        entered MM. Type EFI_AP_PROCEDURE is defined below in related
+                                        definitions.
+  @param[in]     TimeoutInMicroseconds  Indicates the time limit in microseconds for the APs to finish
+                                        execution of Procedure, either for blocking or non-blocking mode.
+                                        Zero means infinity. If the timeout expires before all APs return
+                                        from Procedure, then Procedure on the failed APs is terminated. If
+                                        the timeout expires in blocking mode, the call returns EFI_TIMEOUT.
+                                        If the timeout expires in non-blocking mode, the timeout determined
+                                        can be through CheckOnProcedure or WaitForProcedure.
+                                        Note that timeout support is optional. Whether an implementation
+                                        supports this feature can be determined via the Attributes data
+                                        member.
+  @param[in,out] ProcedureArguments     Allows the caller to pass a list of parameters to the code
+                                        that is run by the AP. It is an optional common mailbox
+                                        between APs and the caller to share information.
+  @param[in,out] Token                  This is parameter is broken into two components:
+                                        1.Token->Completion is an optional parameter that allows the
+                                        caller to execute the procedure in a blocking or non-blocking
+                                        fashion. If it is NULL the call is blocking, and the call will
+                                        not return until the AP has completed the procedure. If the
+                                        token is not NULL, the call will return immediately. The caller
+                                        can check whether the procedure has completed with
+                                        CheckOnProcedure or WaitForProcedure.
+                                        2.Token->Status The implementation updates the address pointed
+                                        at by this variable with the status code returned by Procedure
+                                        when it completes execution on the target AP, or with EFI_TIMEOUT
+                                        if the Procedure fails to complete within the optional timeout.
+                                        The implementation will update this variable with EFI_NOT_READY
+                                        prior to starting Procedure on the target AP
+  @param[in,out] CPUStatus              This optional pointer may be used to get the individual status
+                                        returned by every AP that participated in the broadcast. This
+                                        parameter if used provides the base address of an array to hold
+                                        the EFI_STATUS value of each AP in the system. The size of the
+                                        array can be ascertained by the GetNumberOfProcessors function.
+                                        As mentioned above, the broadcast may not include every processor
+                                        in the system. Some implementations may exclude processors that
+                                        have been powered down in such a way that they are not responsive
+                                        to interrupts. Additionally the broadcast excludes the processor
+                                        which is making the BroadcastProcedure call. For every excluded
+                                        processor, the array entry must contain a value of EFI_NOT_STARTED
+
+  @retval EFI_SUCCESS                   In the blocking case, this indicates that Procedure has completed
+                                        execution on the APs.
+                                        In the non-blocking case this indicates that the procedure has
+                                        been successfully scheduled for execution on the APs.
+  @retval EFI_INVALID_PARAMETER         The Procedure or Token is NULL
+  @retval EFI_NOT_READY                 If the target AP is busy executing another procedure
+  @retval EFI_ALREADY_STARTED           Token is already in use for another procedure
+  @retval EFI_TIMEOUT                   In blocking mode, the timeout expired before the specified AP
+                                        has finished
+  @retval EFI_OUT_OF_RESOURCES          Could not allocate a required resource.
+
+**/
+EFI_STATUS
+EFIAPI
+SmmMpBroadcastProcedure (
+  IN CONST EFI_MM_MP_PROTOCOL             *This,
+  IN       EFI_AP_PROCEDURE2              Procedure,
+  IN       UINTN                          TimeoutInMicroseconds,
+  IN OUT   VOID                           *ProcedureArguments OPTIONAL,
+  IN OUT   MM_COMPLETION                  *Token,
+  IN OUT   EFI_STATUS                     *CPUStatus
+  );
+
+
+/**
+  This service allows the caller to set a startup procedure that will be executed when an AP powers
+  up from a state where core configuration and context is lost. The procedure is execution has the
+  following properties:
+  1. The procedure executes before the processor is handed over to the operating system.
+  2. All processors execute the same startup procedure.
+  3. The procedure may run in parallel with other procedures invoked through the functions in this
+  protocol, or with processors that are executing an MM handler or running in the operating system.
+
+
+  @param[in]      This                 The EFI_MM_MP_PROTOCOL instance.
+  @param[in]      Procedure            A pointer to the code stream to be run on the designated target AP
+                                       of the system. Type EFI_AP_PROCEDURE is defined below in Volume 2
+                                       with the related definitions of
+                                       EFI_MP_SERVICES_PROTOCOL.StartupAllAPs.
+                                       If caller may pass a value of NULL to deregister any existing
+                                       startup procedure.
+  @param[in,out]  ProcedureArguments   Allows the caller to pass a list of parameters to the code that is
+                                       run by the AP. It is an optional common mailbox between APs and
+                                       the caller to share information
+
+  @retval EFI_SUCCESS                  The Procedure has been set successfully.
+  @retval EFI_INVALID_PARAMETER        The Procedure is NULL but ProcedureArguments not NULL.
+**/
+EFI_STATUS
+EFIAPI
+SmmMpSetStartupProcedure (
+  IN CONST EFI_MM_MP_PROTOCOL  *This,
+  IN       EFI_AP_PROCEDURE    Procedure,
+  IN OUT   VOID                *ProcedureArguments OPTIONAL
+  );
+
+/**
+  When non-blocking execution of a procedure on an AP is invoked with DispatchProcedure,
+  via the use of a token, this function can be used to check for completion of the procedure on the AP.
+  The function takes the token that was passed into the DispatchProcedure call. If the procedure
+  is complete, and therefore it is now possible to run another procedure on the same AP, this function
+  returns EFI_SUCESS. In this case the status returned by the procedure that executed on the AP is
+  returned in the token's Status field. If the procedure has not yet completed, then this function
+  returns EFI_NOT_READY.
+
+  When a non-blocking execution of a procedure is invoked with BroadcastProcedure, via the
+  use of a token, this function can be used to check for completion of the procedure on all the
+  broadcast APs. The function takes the token that was passed into the BroadcastProcedure
+  call. If the procedure is complete on all broadcast APs this function returns EFI_SUCESS. In this
+  case the Status field in the token passed into the function reflects the overall result of the
+  invocation, which may be EFI_SUCCESS, if all executions succeeded, or the first observed failure.
+  If the procedure has not yet completed on the broadcast APs, the function returns
+  EFI_NOT_READY.
+
+  @param[in]      This                 The EFI_MM_MP_PROTOCOL instance.
+  @param[in]      Token                This parameter describes the token that was passed into
+                                       DispatchProcedure or BroadcastProcedure.
+
+  @retval EFI_SUCCESS                  Procedure has completed.
+  @retval EFI_NOT_READY                The Procedure has not completed.
+  @retval EFI_INVALID_PARAMETER        Token or Token->Completion is NULL
+  @retval EFI_NOT_FOUND                Token is not currently in use for a non-blocking call
+
+**/
+EFI_STATUS
+EFIAPI
+SmmMpCheckForProcedure (
+  IN CONST EFI_MM_MP_PROTOCOL             *This,
+  IN       MM_COMPLETION                  Token
+  );
+
+/**
+  When a non-blocking execution of a procedure on an AP is invoked via DispatchProcedure,
+  this function will block the caller until the remote procedure has completed on the designated AP.
+  The non-blocking procedure invocation is identified by the Token parameter, which must match the
+  token that used when DispatchProcedure was called. Upon completion the status returned by
+  the procedure that executed on the AP is used to update the token's Status field.
+
+  When a non-blocking execution of a procedure on an AP is invoked via BroadcastProcedure
+  this function will block the caller until the remote procedure has completed on all of the APs that
+  entered MM. The non-blocking procedure invocation is identified by the Token parameter, which
+  must match the token that used when BroadcastProcedure was called. Upon completion the
+  overall status returned by the procedures that executed on the broadcast AP is used to update the
+  token's Status field. The overall status may be EFI_SUCCESS, if all executions succeeded, or the
+  first observed failure.
+
+
+  @param[in]      This                 The EFI_MM_MP_PROTOCOL instance.
+  @param[in]      Token                This parameter describes the token that was passed into
+                                       DispatchProcedure or BroadcastProcedure.
+
+  @retval EFI_SUCCESS                  Procedure has completed.
+  @retval EFI_INVALID_PARAMETER        Token or Token->Completion is NULL
+  @retval EFI_NOT_FOUND                Token is not currently in use for a non-blocking call
+
+**/
+EFI_STATUS
+EFIAPI
+SmmMpWaitForProcedure (
+  IN CONST EFI_MM_MP_PROTOCOL            *This,
+  IN       MM_COMPLETION                 Token
+  );
+
+#endif
-- 
2.21.0.windows.1

Re: [edk2-devel] [Patch v3 2/2] UefiCpuPkg/PiSmmCpuDxeSmm: Enable MM MP Protocol.

[Laszlo Ersek]

On 07/03/19 04:42, Dong, Eric wrote:
> v3 changes:
> 1. Fix Token clean up too early caused CheckProcedure return error.
>
> v2 changes:
> 1. Remove some duplicated global variables.
> 2. Enhance token design to support multiple task trig for different APs at the same time.
>
> V1 changes:
> REF: https://bugzilla.tianocore.org/show_bug.cgi?id=1937
>
> Add MM Mp Protocol in PiSmmCpuDxeSmm driver.
>
> Cc: Ray Ni <ray.ni@intel.com>
> Cc: Laszlo Ersek <lersek@redhat.com>
> Signed-off-by: Eric Dong <eric.dong@intel.com>
> ---
>  UefiCpuPkg/PiSmmCpuDxeSmm/MpService.c        | 536 ++++++++++++++++++-
>  UefiCpuPkg/PiSmmCpuDxeSmm/PiSmmCpuDxeSmm.c   |  11 +
>  UefiCpuPkg/PiSmmCpuDxeSmm/PiSmmCpuDxeSmm.h   | 160 +++++-
>  UefiCpuPkg/PiSmmCpuDxeSmm/PiSmmCpuDxeSmm.inf |   3 +
>  UefiCpuPkg/PiSmmCpuDxeSmm/SmmMp.c            | 372 +++++++++++++
>  UefiCpuPkg/PiSmmCpuDxeSmm/SmmMp.h            | 286 ++++++++++
>  6 files changed, 1351 insertions(+), 17 deletions(-)
>  create mode 100644 UefiCpuPkg/PiSmmCpuDxeSmm/SmmMp.c
>  create mode 100644 UefiCpuPkg/PiSmmCpuDxeSmm/SmmMp.h

The patch applies successfully now; it just adds a few whitespace
errors:

> Applying: UefiCpuPkg/PiSmmCpuDxeSmm: Enable MM MP Protocol.
> .git/rebase-apply/patch:94: trailing whitespace.
>     if ((ApIndex != gSmmCpuPrivate->SmmCoreEntryContext.CurrentlyExecutingCpu) &&
> .git/rebase-apply/patch:98: trailing whitespace.
>         if ((ApIndex2 != gSmmCpuPrivate->SmmCoreEntryContext.CurrentlyExecutingCpu) &&
> .git/rebase-apply/patch:222: trailing whitespace.
>       if (mSmmMpSyncData->CpuData[Index].Present &&
> .git/rebase-apply/patch:606: trailing whitespace.
>   UINTN                      CpuIndex;
> .git/rebase-apply/patch:1320: new blank line at EOF.
> +
> warning: 5 lines add whitespace errors.

No need to resubmit the series just because of those; please clean them
up on push.

I'll follow up with regression test results.

Thanks
Laszlo

Re: [edk2-devel] [Patch v3 2/2] UefiCpuPkg/PiSmmCpuDxeSmm: Enable MM MP Protocol.

[Ni, Ray]

1. mMaxNumberOfCpus and gSmmCpuPrivate->SmmCoreEntryContext.NumberOfCpus are both used. Can you please refine the code to use only one?
2. I do think that MM_MP code needs to maintain a internal linked list containing all created tokens. Because it's possible that caller is written in below way:
    DispatchProcedure (AP#1, &TokenA);
    DispatchProcedure (AP#1, &TokenB);
    DispatchProcedure (AP#1, &TokenC);
    CheckOnProcedure (&TokenA);
    Without having a list to contain all created tokens, CheckOnProcedure (&TokenA) may return NotFound.
    The linked list can be freed upon exiting SMM
3. IsCurrentToken() can be changed to look up the token in the linked list and the name can be changed to IsTokenInUse().
    The function name reflects the spec wording "EFI_ALREADY_STARTED - Token is already in use for another procedure."
4. In InternalSmmStartup[AllAPs|ThisAp], Token can be acquired after Busy is acquired. Token is associated with the procedure.
     Busy is associated with the individual processor.
5. *(CPUStatus + Index) can be changed to CPUStatus[Index]. Use [] instead of address operation.
6. Below error handling in InternalSmmStartupAllAPs may not be necessary. Just AcquireSpinLock (...Busy) should be ok.
    //
    // Make sure all BUSY should be acquired.
    //
    for (Index = mMaxNumberOfCpus; Index-- > 0;) {
      if (Index != gSmmCpuPrivate->SmmCoreEntryContext.CurrentlyExecutingCpu && *(mSmmMpSyncData->CpuData[Index].Present)) {
        if (!AcquireSpinLockOrFail (mSmmMpSyncData->CpuData[Index].Busy)) {
          DEBUG((DEBUG_ERROR, "Can't acquire mSmmMpSyncData->CpuData[%d].Busy\n", Index));

          //
          // Release BUSY accquired before.
          //
          for (CpuCount = mMaxNumberOfCpus; CpuCount -- > Index;) {
            if (CpuCount != gSmmCpuPrivate->SmmCoreEntryContext.CurrentlyExecutingCpu && *(mSmmMpSyncData->CpuData[CpuCount].Present)) {
              ReleaseSpinLock (mSmmMpSyncData->CpuData[CpuCount].Busy);
            }
          }

          ReleaseSpinLock (Token);
          return EFI_INVALID_PARAMETER;
        }
      }
    }
7. IsTaskFinishInAllAPs() can be used by BSPHandler() as well. The function name can be WaitForAllAPsNotBusy() or some other name because the function
    has side affect, not just check.
8. Maybe you don't need to have a mApTokenLock[]. For old style async SMM MP call, the token is not returned to caller. But it's still possible to be freed
     as long as it's inserted to the linked list.

Thanks,
Ray

> -----Original Message-----
> From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Dong,
> Eric
> Sent: Wednesday, July 3, 2019 10:43 AM
> To: devel@edk2.groups.io
> Cc: Ni, Ray <ray.ni@intel.com>; Laszlo Ersek <lersek@redhat.com>
> Subject: [edk2-devel] [Patch v3 2/2] UefiCpuPkg/PiSmmCpuDxeSmm: Enable
> MM MP Protocol.
> 
> v3 changes:
> 1. Fix Token clean up too early caused CheckProcedure return error.
> 
> v2 changes:
> 1. Remove some duplicated global variables.
> 2. Enhance token design to support multiple task trig for different APs at the
> same time.
> 
> V1 changes:
> REF: https://bugzilla.tianocore.org/show_bug.cgi?id=1937
> 
> Add MM Mp Protocol in PiSmmCpuDxeSmm driver.
> 
> Cc: Ray Ni <ray.ni@intel.com>
> Cc: Laszlo Ersek <lersek@redhat.com>
> Signed-off-by: Eric Dong <eric.dong@intel.com>
> ---
>  UefiCpuPkg/PiSmmCpuDxeSmm/MpService.c        | 536
> ++++++++++++++++++-
>  UefiCpuPkg/PiSmmCpuDxeSmm/PiSmmCpuDxeSmm.c   |  11 +
>  UefiCpuPkg/PiSmmCpuDxeSmm/PiSmmCpuDxeSmm.h   | 160 +++++-
>  UefiCpuPkg/PiSmmCpuDxeSmm/PiSmmCpuDxeSmm.inf |   3 +
>  UefiCpuPkg/PiSmmCpuDxeSmm/SmmMp.c            | 372 +++++++++++++
>  UefiCpuPkg/PiSmmCpuDxeSmm/SmmMp.h            | 286 ++++++++++
>  6 files changed, 1351 insertions(+), 17 deletions(-)
>  create mode 100644 UefiCpuPkg/PiSmmCpuDxeSmm/SmmMp.c
>  create mode 100644 UefiCpuPkg/PiSmmCpuDxeSmm/SmmMp.h
> 
> diff --git a/UefiCpuPkg/PiSmmCpuDxeSmm/MpService.c
> b/UefiCpuPkg/PiSmmCpuDxeSmm/MpService.c
> index 64fb4d6344..13858f5d2d 100644
> --- a/UefiCpuPkg/PiSmmCpuDxeSmm/MpService.c
> +++ b/UefiCpuPkg/PiSmmCpuDxeSmm/MpService.c
> @@ -22,6 +22,7 @@ UINTN                                       mSemaphoreSize;
>  SPIN_LOCK                                   *mPFLock = NULL;
>  SMM_CPU_SYNC_MODE                           mCpuSmmSyncMode;
>  BOOLEAN                                     mMachineCheckSupported = FALSE;
> +SPIN_LOCK                                   **mApTokenLock;
> 
>  /**
>    Performs an atomic compare exchange operation to get semaphore.
> @@ -146,6 +147,45 @@ ReleaseAllAPs (
>    }
>  }
> 
> +/**
> +  Wheck whether task has been finished by all APs.
> +
> +  @param       BlockMode   Whether did it in block mode or non-block mode.
> +
> +  @retval      TRUE        Task has been finished by all APs.
> +  @retval      FALSE       Task not has been finished by all APs.
> +
> +**/
> +BOOLEAN
> +IsTaskFinishInAllAPs (
> +  IN BOOLEAN                        BlockMode
> +  )
> +{
> +  UINTN                             Index;
> +
> +  for (Index = mMaxNumberOfCpus; Index-- > 0;) {
> +    //
> +    // Ignore BSP and APs which not call in SMM.
> +    //
> +    if ((Index == gSmmCpuPrivate-
> >SmmCoreEntryContext.CurrentlyExecutingCpu) || (!*(mSmmMpSyncData-
> >CpuData[Index].Present))) {
> +      continue;
> +    }
> +
> +    if (BlockMode) {
> +      AcquireSpinLock(mSmmMpSyncData->CpuData[Index].Busy);
> +      ReleaseSpinLock(mSmmMpSyncData->CpuData[Index].Busy);
> +    } else {
> +      if (AcquireSpinLockOrFail (mSmmMpSyncData->CpuData[Index].Busy)) {
> +        ReleaseSpinLock(mSmmMpSyncData->CpuData[Index].Busy);
> +      } else {
> +        return FALSE;
> +      }
> +    }
> +  }
> +
> +  return TRUE;
> +}
> +
>  /**
>    Checks if all CPUs (with certain exceptions) have checked in for this SMI run
> 
> @@ -347,6 +387,92 @@ ReplaceOSMtrrs (
>    MtrrSetAllMtrrs (&gSmiMtrrs);
>  }
> 
> +/**
> +  Check whether execute in single AP or all APs.
> +
> +  Compare two Tokens used by different APs to know whether in
> StartAllAps call.
> +
> +  Whether is an valid AP base on AP's Present flag.
> +
> +  @retval  TRUE      IN StartAllAps call.
> +  @retval  FALSE     Not in StartAllAps call.
> +
> +**/
> +BOOLEAN
> +InStartAllApsCall (
> +  VOID
> +  )
> +{
> +  UINTN      ApIndex;
> +  UINTN      ApIndex2;
> +
> +  for (ApIndex = mMaxNumberOfCpus; ApIndex-- > 0;) {
> +    if ((ApIndex != gSmmCpuPrivate-
> >SmmCoreEntryContext.CurrentlyExecutingCpu) &&
> +        *(mSmmMpSyncData->CpuData[ApIndex].Present) &&
> +        (mSmmMpSyncData->CpuData[ApIndex].Token != NULL)) {
> +      for (ApIndex2 = ApIndex; ApIndex2-- > 0;) {
> +        if ((ApIndex2 != gSmmCpuPrivate-
> >SmmCoreEntryContext.CurrentlyExecutingCpu) &&
> +            *(mSmmMpSyncData->CpuData[ApIndex2].Present) &&
> +            (mSmmMpSyncData->CpuData[ApIndex2].Token != NULL)) {
> +          return mSmmMpSyncData->CpuData[ApIndex2].Token ==
> mSmmMpSyncData->CpuData[ApIndex].Token;
> +        }
> +      }
> +    }
> +  }
> +
> +  return FALSE;
> +}
> +
> +/**
> +  Clean up the status flags used during executing the procedure.
> +
> +  @param   CpuIndex      The AP index which calls this function.
> +
> +**/
> +VOID
> +CleanUpStatusFlags (
> +  IN UINTN                  CpuIndex
> +  )
> +{
> +  UINTN                             Index;
> +
> +  if (InStartAllApsCall ()) {
> +    //
> +    // In Start All APs mode, make sure all APs have finished task.
> +    //
> +    if (IsTaskFinishInAllAPs (FALSE)) {
> +      //
> +      // Clean the flags update in the function call.
> +      //
> +      for (Index = mMaxNumberOfCpus; Index-- > 0;) {
> +        //
> +        // Only In SMM APs need to be clean up.
> +        //
> +        if (mSmmMpSyncData->CpuData[Index].Present) {
> +          if (mSmmMpSyncData->CpuData[Index].Token != NULL) {
> +            ReleaseSpinLock (mSmmMpSyncData->CpuData[Index].Token);
> +            break;
> +          }
> +        }
> +      }
> +    }
> +  } else {
> +    //
> +    // In single AP mode.
> +    //
> +    if (mSmmMpSyncData->CpuData[CpuIndex].Token != NULL) {
> +      //
> +      // Free the wrapper buffer in non-block mode used by
> SmmMpDispatchProcedure function.
> +      //
> +      if (mApTokenLock[CpuIndex] == mSmmMpSyncData-
> >CpuData[CpuIndex].Token) {
> +        FreePool ((VOID *)mSmmMpSyncData-
> >CpuData[CpuIndex].Parameter);
> +      }
> +
> +      ReleaseSpinLock (mSmmMpSyncData->CpuData[CpuIndex].Token);
> +    }
> +  }
> +}
> +
>  /**
>    SMI handler for BSP.
> 
> @@ -604,6 +730,7 @@ APHandler (
>    UINT64                            Timer;
>    UINTN                             BspIndex;
>    MTRR_SETTINGS                     Mtrrs;
> +  EFI_STATUS                        ProcedureStatus;
> 
>    //
>    // Timeout BSP
> @@ -730,14 +857,19 @@ APHandler (
>      //
>      // Invoke the scheduled procedure
>      //
> -    (*mSmmMpSyncData->CpuData[CpuIndex].Procedure) (
> -      (VOID*)mSmmMpSyncData->CpuData[CpuIndex].Parameter
> -      );
> +    ProcedureStatus = (*mSmmMpSyncData-
> >CpuData[CpuIndex].Procedure) (
> +                          (VOID*)mSmmMpSyncData->CpuData[CpuIndex].Parameter
> +                          );
> +    if (mSmmMpSyncData->CpuData[CpuIndex].Status != NULL) {
> +      *mSmmMpSyncData->CpuData[CpuIndex].Status = ProcedureStatus;
> +    }
> 
>      //
>      // Release BUSY
>      //
>      ReleaseSpinLock (mSmmMpSyncData->CpuData[CpuIndex].Busy);
> +
> +    CleanUpStatusFlags (CpuIndex);
>    }
> 
>    if (SmmCpuFeaturesNeedConfigureMtrrs()) {
> @@ -906,13 +1038,95 @@ Gen4GPageTable (
>    return (UINT32)(UINTN)PageTable;
>  }
> 
> +/**
> +  Checks whether the input token is the current used token.
> +
> +  @param[in]  CpuIndex   Cpu Index.
> +  @param[in]  Token      This parameter describes the token that was passed
> into DispatchProcedure or
> +                         BroadcastProcedure.
> +
> +  @retval TRUE           The input token is the current used token.
> +  @retval FALSE          The input token is not the current used token.
> +**/
> +BOOLEAN
> +IsCurrentToken (
> +  IN UINTN               CpuIndex,
> +  IN SPIN_LOCK           *Token
> +  )
> +{
> +  UINTN         Index;
> +
> +  if (Token == NULL) {
> +    return FALSE;
> +  }
> +
> +  if (CpuIndex == (UINTN) -1) {
> +    for (Index = 0; Index < gSmmCpuPrivate-
> >SmmCoreEntryContext.NumberOfCpus; Index ++) {
> +      if (mSmmMpSyncData->CpuData[Index].Present &&
> +          (mSmmMpSyncData->CpuData[Index].Token != NULL) &&
> +          (mSmmMpSyncData->CpuData[Index].Token == Token)) {
> +        return TRUE;
> +      }
> +    }
> +
> +    return FALSE;
> +  }
> +
> +  return mSmmMpSyncData->CpuData[CpuIndex].Token == Token;
> +}
> +
> +/**
> +  Checks status of specified AP.
> +
> +  This function checks whether the specified AP has finished the task
> assigned
> +  by StartupThisAP(), and whether timeout expires.
> +
> +  @param[in]  Token             This parameter describes the token that was
> passed into DispatchProcedure or
> +                                BroadcastProcedure.
> +
> +  @retval EFI_SUCCESS           Specified AP has finished task assigned by
> StartupThisAPs().
> +  @retval EFI_NOT_READY         Specified AP has not finished task and
> timeout has not expired.
> +**/
> +EFI_STATUS
> +IsApReady (
> +  IN SPIN_LOCK          *Token
> +  )
> +{
> +  if (AcquireSpinLockOrFail (Token)) {
> +    ReleaseSpinLock (Token);
> +    return EFI_SUCCESS;
> +  }
> +
> +  return EFI_NOT_READY;
> +}
> +
>  /**
>    Schedule a procedure to run on the specified CPU.
> 
>    @param[in]       Procedure                The address of the procedure to run
>    @param[in]       CpuIndex                 Target CPU Index
> -  @param[in, out]  ProcArguments            The parameter to pass to the
> procedure
> -  @param[in]       BlockingMode             Startup AP in blocking mode or not
> +  @param[in,out]   ProcArguments            The parameter to pass to the
> procedure
> +  @param[in]       Token                    This is an optional parameter that allows
> the caller to execute the
> +                                            procedure in a blocking or non-blocking fashion. If it is
> NULL the
> +                                            call is blocking, and the call will not return until the AP
> has
> +                                            completed the procedure. If the token is not NULL,
> the call will
> +                                            return immediately. The caller can check whether the
> procedure has
> +                                            completed with CheckOnProcedure or
> WaitForProcedure.
> +  @param[in]       TimeoutInMicroseconds    Indicates the time limit in
> microseconds for the APs to finish
> +                                            execution of Procedure, either for blocking or non-
> blocking mode.
> +                                            Zero means infinity. If the timeout expires before all
> APs return
> +                                            from Procedure, then Procedure on the failed APs is
> terminated. If
> +                                            the timeout expires in blocking mode, the call returns
> EFI_TIMEOUT.
> +                                            If the timeout expires in non-blocking mode, the
> timeout determined
> +                                            can be through CheckOnProcedure or
> WaitForProcedure.
> +                                            Note that timeout support is optional. Whether an
> implementation
> +                                            supports this feature can be determined via the
> Attributes data
> +                                            member.
> +  @param[in,out]   CpuStatus                This optional pointer may be used to
> get the status code returned
> +                                            by Procedure when it completes execution on the
> target AP, or with
> +                                            EFI_TIMEOUT if the Procedure fails to complete
> within the optional
> +                                            timeout. The implementation will update this variable
> with
> +                                            EFI_NOT_READY prior to starting Procedure on the
> target AP.
> 
>    @retval EFI_INVALID_PARAMETER    CpuNumber not valid
>    @retval EFI_INVALID_PARAMETER    CpuNumber specifying BSP
> @@ -923,10 +1137,12 @@ Gen4GPageTable (
>  **/
>  EFI_STATUS
>  InternalSmmStartupThisAp (
> -  IN      EFI_AP_PROCEDURE          Procedure,
> -  IN      UINTN                     CpuIndex,
> -  IN OUT  VOID                      *ProcArguments OPTIONAL,
> -  IN      BOOLEAN                   BlockingMode
> +  IN      EFI_AP_PROCEDURE2              Procedure,
> +  IN      UINTN                          CpuIndex,
> +  IN OUT  VOID                           *ProcArguments OPTIONAL,
> +  IN      SPIN_LOCK                      *Token,
> +  IN      UINTN                          TimeoutInMicroseconds,
> +  IN OUT  EFI_STATUS                     *CpuStatus
>    )
>  {
>    if (CpuIndex >= gSmmCpuPrivate->SmmCoreEntryContext.NumberOfCpus)
> {
> @@ -952,24 +1168,208 @@ InternalSmmStartupThisAp (
>      }
>      return EFI_INVALID_PARAMETER;
>    }
> +  if ((TimeoutInMicroseconds != 0) && ((mSmmMp.Attributes &
> EFI_MM_MP_TIMEOUT_SUPPORTED) == 0)) {
> +    return EFI_INVALID_PARAMETER;
> +  }
> +  if (Procedure == NULL) {
> +    return EFI_INVALID_PARAMETER;
> +  }
> +  if (IsCurrentToken (CpuIndex, Token)) {
> +    return EFI_ALREADY_STARTED;
> +  }
> 
> -  if (BlockingMode) {
> +  if (Token == NULL) {
>      AcquireSpinLock (mSmmMpSyncData->CpuData[CpuIndex].Busy);
>    } else {
> -    if (!AcquireSpinLockOrFail (mSmmMpSyncData-
> >CpuData[CpuIndex].Busy)) {
> -      DEBUG((DEBUG_ERROR, "mSmmMpSyncData->CpuData[%d].Busy\n",
> CpuIndex));
> +    if (!AcquireSpinLockOrFail (Token)) {
> +      DEBUG((DEBUG_ERROR, "Token can't acquire\n"));
>        return EFI_INVALID_PARAMETER;
>      }
> +    if (!AcquireSpinLockOrFail (mSmmMpSyncData-
> >CpuData[CpuIndex].Busy)) {
> +      DEBUG((DEBUG_ERROR, "Can't acquire mSmmMpSyncData-
> >CpuData[%d].Busy\n", CpuIndex));
> +      ReleaseSpinLock (Token);
> +      return EFI_NOT_READY;
> +    }
>    }
> 
>    mSmmMpSyncData->CpuData[CpuIndex].Procedure = Procedure;
>    mSmmMpSyncData->CpuData[CpuIndex].Parameter = ProcArguments;
> +  mSmmMpSyncData->CpuData[CpuIndex].Token     = Token;
> +  mSmmMpSyncData->CpuData[CpuIndex].Status    = CpuStatus;
> +  if (mSmmMpSyncData->CpuData[CpuIndex].Status != NULL) {
> +    *mSmmMpSyncData->CpuData[CpuIndex].Status = EFI_NOT_READY;
> +  }
> +
>    ReleaseSemaphore (mSmmMpSyncData->CpuData[CpuIndex].Run);
> 
> -  if (BlockingMode) {
> +  if (Token == NULL) {
>      AcquireSpinLock (mSmmMpSyncData->CpuData[CpuIndex].Busy);
>      ReleaseSpinLock (mSmmMpSyncData->CpuData[CpuIndex].Busy);
>    }
> +
> +  return EFI_SUCCESS;
> +}
> +
> +/**
> +  Worker function to execute a caller provided function on all enabled APs.
> +
> +  @param[in]     Procedure               A pointer to the function to be run on
> +                                         enabled APs of the system.
> +  @param[in]     TimeoutInMicroseconds   Indicates the time limit in
> microseconds for
> +                                         APs to return from Procedure, either for
> +                                         blocking or non-blocking mode.
> +  @param[in,out] ProcedureArguments      The parameter passed into
> Procedure for
> +                                         all APs.
> +  @param[in,out] Token                   This is an optional parameter that allows
> the caller to execute the
> +                                         procedure in a blocking or non-blocking fashion. If it is
> NULL the
> +                                         call is blocking, and the call will not return until the AP
> has
> +                                         completed the procedure. If the token is not NULL, the
> call will
> +                                         return immediately. The caller can check whether the
> procedure has
> +                                         completed with CheckOnProcedure or
> WaitForProcedure.
> +  @param[in,out] CPUStatus               This optional pointer may be used to get
> the status code returned
> +                                         by Procedure when it completes execution on the
> target AP, or with
> +                                         EFI_TIMEOUT if the Procedure fails to complete within
> the optional
> +                                         timeout. The implementation will update this variable
> with
> +                                         EFI_NOT_READY prior to starting Procedure on the
> target AP.
> +
> +
> +  @retval EFI_SUCCESS             In blocking mode, all APs have finished before
> +                                  the timeout expired.
> +  @retval EFI_SUCCESS             In non-blocking mode, function has been
> dispatched
> +                                  to all enabled APs.
> +  @retval others                  Failed to Startup all APs.
> +
> +**/
> +EFI_STATUS
> +InternalSmmStartupAllAPs (
> +  IN       EFI_AP_PROCEDURE2             Procedure,
> +  IN       UINTN                         TimeoutInMicroseconds,
> +  IN OUT   VOID                          *ProcedureArguments OPTIONAL,
> +  IN OUT   SPIN_LOCK                     *Token,
> +  IN OUT   EFI_STATUS                    *CPUStatus
> +  )
> +{
> +  UINTN               Index;
> +  UINTN               CpuCount;
> +
> +  if ((TimeoutInMicroseconds != 0) && ((mSmmMp.Attributes &
> EFI_MM_MP_TIMEOUT_SUPPORTED) == 0)) {
> +    return EFI_INVALID_PARAMETER;
> +  }
> +  if (Procedure == NULL) {
> +    return EFI_INVALID_PARAMETER;
> +  }
> +  if (IsCurrentToken ((UINTN)-1, Token)) {
> +    return EFI_ALREADY_STARTED;
> +  }
> +
> +  CpuCount = 0;
> +  for (Index = mMaxNumberOfCpus; Index-- > 0;) {
> +    if (*mSmmMpSyncData->CpuData[Index].Present && (Index !=
> gSmmCpuPrivate->SmmCoreEntryContext.CurrentlyExecutingCpu)) {
> +      CpuCount ++;
> +
> +      if (gSmmCpuPrivate->Operation[Index] == SmmCpuRemove) {
> +        return EFI_INVALID_PARAMETER;
> +      }
> +
> +      if (!AcquireSpinLockOrFail(mSmmMpSyncData->CpuData[Index].Busy)) {
> +        return EFI_NOT_READY;
> +      }
> +      ReleaseSpinLock (mSmmMpSyncData->CpuData[Index].Busy);
> +    }
> +  }
> +  if (CpuCount == 0) {
> +    return EFI_NOT_STARTED;
> +  }
> +
> +  if (Token == NULL) {
> +    //
> +    // Make sure all BUSY should be acquired.
> +    //
> +    for (Index = mMaxNumberOfCpus; Index-- > 0;) {
> +      if (Index != gSmmCpuPrivate-
> >SmmCoreEntryContext.CurrentlyExecutingCpu && *(mSmmMpSyncData-
> >CpuData[Index].Present)) {
> +        AcquireSpinLock (mSmmMpSyncData->CpuData[Index].Busy);
> +      }
> +    }
> +  } else {
> +    if (!AcquireSpinLockOrFail (Token)) {
> +      return EFI_INVALID_PARAMETER;
> +    }
> +
> +    //
> +    // Make sure all BUSY should be acquired.
> +    //
> +    for (Index = mMaxNumberOfCpus; Index-- > 0;) {
> +      if (Index != gSmmCpuPrivate-
> >SmmCoreEntryContext.CurrentlyExecutingCpu && *(mSmmMpSyncData-
> >CpuData[Index].Present)) {
> +        if (!AcquireSpinLockOrFail (mSmmMpSyncData->CpuData[Index].Busy))
> {
> +          DEBUG((DEBUG_ERROR, "Can't acquire mSmmMpSyncData-
> >CpuData[%d].Busy\n", Index));
> +
> +          //
> +          // Release BUSY accquired before.
> +          //
> +          for (CpuCount = mMaxNumberOfCpus; CpuCount -- > Index;) {
> +            if (CpuCount != gSmmCpuPrivate-
> >SmmCoreEntryContext.CurrentlyExecutingCpu && *(mSmmMpSyncData-
> >CpuData[CpuCount].Present)) {
> +              ReleaseSpinLock (mSmmMpSyncData->CpuData[CpuCount].Busy);
> +            }
> +          }
> +
> +          ReleaseSpinLock (Token);
> +          return EFI_INVALID_PARAMETER;
> +        }
> +      }
> +    }
> +  }
> +
> +  for (Index = mMaxNumberOfCpus; Index-- > 0;) {
> +    if (Index != gSmmCpuPrivate-
> >SmmCoreEntryContext.CurrentlyExecutingCpu && *(mSmmMpSyncData-
> >CpuData[Index].Present)) {
> +      mSmmMpSyncData->CpuData[Index].Procedure =
> (EFI_AP_PROCEDURE2) Procedure;
> +      mSmmMpSyncData->CpuData[Index].Parameter =
> ProcedureArguments;
> +      mSmmMpSyncData->CpuData[Index].Token     = Token;
> +      if (CPUStatus != NULL) {
> +        mSmmMpSyncData->CpuData[Index].Status    = CPUStatus + Index;
> +        if (mSmmMpSyncData->CpuData[Index].Status != NULL) {
> +          *mSmmMpSyncData->CpuData[Index].Status = EFI_NOT_READY;
> +        }
> +      }
> +    } else {
> +      //
> +      // PI spec requirement:
> +      // For every excluded processor, the array entry must contain a value of
> EFI_NOT_STARTED.
> +      //
> +      if (CPUStatus != NULL) {
> +        *(CPUStatus + Index) = EFI_NOT_STARTED;
> +      }
> +    }
> +  }
> +
> +  ReleaseAllAPs ();
> +
> +  if (Token == NULL) {
> +    //
> +    // Make sure all APs have completed their tasks.
> +    //
> +    IsTaskFinishInAllAPs (TRUE);
> +  }
> +
> +  return EFI_SUCCESS;
> +}
> +
> +/**
> +  Wrapper for Procedures.
> +
> +  @param[in]  Buffer              Pointer to PROCEDURE_WRAPPER buffer.
> +
> +**/
> +EFI_STATUS
> +EFIAPI
> +ProcedureWrapper (
> +  IN OUT VOID *Buffer
> +  )
> +{
> +  PROCEDURE_WRAPPER *Wrapper;
> +
> +  Wrapper = Buffer;
> +  Wrapper->Procedure (Wrapper->ProcedureArgument);
> +
>    return EFI_SUCCESS;
>  }
> 
> @@ -995,7 +1395,24 @@ SmmBlockingStartupThisAp (
>    IN OUT  VOID                      *ProcArguments OPTIONAL
>    )
>  {
> -  return InternalSmmStartupThisAp(Procedure, CpuIndex, ProcArguments,
> TRUE);
> +  PROCEDURE_WRAPPER *Wrapper;
> +  EFI_STATUS        Status;
> +
> +  Wrapper = AllocatePool (sizeof(PROCEDURE_WRAPPER));
> +  if (Wrapper == NULL) {
> +    return EFI_OUT_OF_RESOURCES;
> +  }
> +  Wrapper->Procedure = Procedure;
> +  Wrapper->ProcedureArgument = ProcArguments;
> +
> +  //
> +  // Use wrapper function to convert EFI_AP_PROCEDURE to
> EFI_AP_PROCEDURE2.
> +  //
> +  Status = InternalSmmStartupThisAp(ProcedureWrapper, CpuIndex,
> Wrapper, NULL, 0, NULL);
> +
> +  FreePool (Wrapper);
> +
> +  return Status;
>  }
> 
>  /**
> @@ -1020,7 +1437,37 @@ SmmStartupThisAp (
>    IN OUT  VOID                      *ProcArguments OPTIONAL
>    )
>  {
> -  return InternalSmmStartupThisAp(Procedure, CpuIndex, ProcArguments,
> FeaturePcdGet (PcdCpuSmmBlockStartupThisAp));
> +  SPIN_LOCK               *CpuToken;
> +  PROCEDURE_WRAPPER       *Wrapper;
> +  EFI_STATUS              Status;
> +
> +  Wrapper = AllocatePool (sizeof(PROCEDURE_WRAPPER));
> +  if (Wrapper == NULL) {
> +    return EFI_OUT_OF_RESOURCES;
> +  }
> +  Wrapper->Procedure = Procedure;
> +  Wrapper->ProcedureArgument = ProcArguments;
> +
> +  if (FeaturePcdGet (PcdCpuSmmBlockStartupThisAp)) {
> +    CpuToken = NULL;
> +  } else {
> +    CpuToken = mApTokenLock[CpuIndex];
> +  }
> +
> +  //
> +  // Use wrapper function to convert EFI_AP_PROCEDURE to
> EFI_AP_PROCEDURE2.
> +  //
> +  Status = InternalSmmStartupThisAp(ProcedureWrapper, CpuIndex,
> Wrapper, CpuToken, 0, NULL);
> +
> +  //
> +  // Free wrapper buffer for block mode.
> +  // Non-block mode frees buffer in ApHandler function.
> +  //
> +  if (CpuToken == NULL) {
> +    FreePool (Wrapper);
> +  }
> +
> +  return Status;
>  }
> 
>  /**
> @@ -1112,6 +1559,13 @@ SmiRendezvous (
>    Cr2 = 0;
>    SaveCr2 (&Cr2);
> 
> +  //
> +  // Call the user register Startup function first.
> +  //
> +  if (mSmmMpSyncData->StartupProcedure != NULL) {
> +    mSmmMpSyncData->StartupProcedure (mSmmMpSyncData-
> >StartupProcArgs);
> +  }
> +
>    //
>    // Perform CPU specific entry hooks
>    //
> @@ -1273,13 +1727,17 @@ InitializeSmmCpuSemaphores (
>    UINTN                      Pages;
>    UINTN                      *SemaphoreBlock;
>    UINTN                      SemaphoreAddr;
> +  UINTN                      ApTokenSize;
> +  UINTN                      CpuIndex;
> 
>    SemaphoreSize   = GetSpinLockProperties ();
>    ProcessorCount = gSmmCpuPrivate-
> >SmmCoreEntryContext.NumberOfCpus;
>    GlobalSemaphoresSize = (sizeof (SMM_CPU_SEMAPHORE_GLOBAL) /
> sizeof (VOID *)) * SemaphoreSize;
>    CpuSemaphoresSize    = (sizeof (SMM_CPU_SEMAPHORE_CPU) / sizeof
> (VOID *)) * ProcessorCount * SemaphoreSize;
> -  TotalSize = GlobalSemaphoresSize + CpuSemaphoresSize;
> +  ApTokenSize = SemaphoreSize * ProcessorCount;
> +  TotalSize = GlobalSemaphoresSize + CpuSemaphoresSize + ApTokenSize;
>    DEBUG((EFI_D_INFO, "One Semaphore Size    = 0x%x\n", SemaphoreSize));
> +  DEBUG((EFI_D_INFO, "Token Spin Lock Size = 0x%x\n", ApTokenSize));
>    DEBUG((EFI_D_INFO, "Total Semaphores Size = 0x%x\n", TotalSize));
>    Pages = EFI_SIZE_TO_PAGES (TotalSize);
>    SemaphoreBlock = AllocatePages (Pages);
> @@ -1305,10 +1763,19 @@ InitializeSmmCpuSemaphores (
>    mSmmCpuSemaphores.SemaphoreCpu.Run     = (UINT32
> *)SemaphoreAddr;
>    SemaphoreAddr += ProcessorCount * SemaphoreSize;
>    mSmmCpuSemaphores.SemaphoreCpu.Present = (BOOLEAN
> *)SemaphoreAddr;
> +  SemaphoreAddr += ProcessorCount * SemaphoreSize;
> +  mSmmCpuSemaphores.SemaphoreCpu.Token = (SPIN_LOCK
> *)SemaphoreAddr;
> 
>    mPFLock                       = mSmmCpuSemaphores.SemaphoreGlobal.PFLock;
>    mConfigSmmCodeAccessCheckLock =
> mSmmCpuSemaphores.SemaphoreGlobal.CodeAccessCheckLock;
> 
> +  mApTokenLock = AllocatePool (sizeof (SPIN_LOCK *) * gSmmCpuPrivate-
> >SmmCoreEntryContext.NumberOfCpus);
> +  ASSERT (mApTokenLock != NULL);
> +  for (CpuIndex = 0; CpuIndex < gSmmCpuPrivate-
> >SmmCoreEntryContext.NumberOfCpus; CpuIndex ++) {
> +    mApTokenLock[CpuIndex] = (SPIN_LOCK
> *)((UINTN)mSmmCpuSemaphores.SemaphoreCpu.Token + SemaphoreSize
> * CpuIndex);
> +    InitializeSpinLock (mApTokenLock[CpuIndex]);
> +  }
> +
>    mSemaphoreSize = SemaphoreSize;
>  }
> 
> @@ -1469,3 +1936,40 @@ RegisterSmmEntry (
>    gSmmCpuPrivate->SmmCoreEntry = SmmEntryPoint;
>    return EFI_SUCCESS;
>  }
> +
> +/**
> +
> +  Register the SMM Foundation entry point.
> +
> +  @param[in]      Procedure            A pointer to the code stream to be run on
> the designated target AP
> +                                       of the system. Type EFI_AP_PROCEDURE is defined
> below in Volume 2
> +                                       with the related definitions of
> +                                       EFI_MP_SERVICES_PROTOCOL.StartupAllAPs.
> +                                       If caller may pass a value of NULL to deregister any
> existing
> +                                       startup procedure.
> +  @param[in]      ProcedureArguments   Allows the caller to pass a list of
> parameters to the code that is
> +                                       run by the AP. It is an optional common mailbox
> between APs and
> +                                       the caller to share information
> +
> +  @retval EFI_SUCCESS                  The Procedure has been set successfully.
> +  @retval EFI_INVALID_PARAMETER        The Procedure is NULL but
> ProcedureArguments not NULL.
> +
> +**/
> +EFI_STATUS
> +RegisterStartupProcedure (
> +  IN EFI_AP_PROCEDURE    Procedure,
> +  IN VOID                *ProcedureArguments OPTIONAL
> +  )
> +{
> +  if (Procedure == NULL && ProcedureArguments != NULL) {
> +    return EFI_INVALID_PARAMETER;
> +  }
> +  if (mSmmMpSyncData == NULL) {
> +    return EFI_NOT_READY;
> +  }
> +
> +  mSmmMpSyncData->StartupProcedure = Procedure;
> +  mSmmMpSyncData->StartupProcArgs  = ProcedureArguments;
> +
> +  return EFI_SUCCESS;
> +}
> diff --git a/UefiCpuPkg/PiSmmCpuDxeSmm/PiSmmCpuDxeSmm.c
> b/UefiCpuPkg/PiSmmCpuDxeSmm/PiSmmCpuDxeSmm.c
> index 2f7d777ee7..dd1b3be0f5 100644
> --- a/UefiCpuPkg/PiSmmCpuDxeSmm/PiSmmCpuDxeSmm.c
> +++ b/UefiCpuPkg/PiSmmCpuDxeSmm/PiSmmCpuDxeSmm.c
> @@ -996,6 +996,17 @@ PiCpuSmmEntry (
>                      );
>    ASSERT_EFI_ERROR (Status);
> 
> +  //
> +  // Install the SMM Mp Protocol into SMM protocol database
> +  //
> +  Status = gSmst->SmmInstallProtocolInterface (
> +                    &mSmmCpuHandle,
> +                    &gEfiMmMpProtocolGuid,
> +                    EFI_NATIVE_INTERFACE,
> +                    &mSmmMp
> +                    );
> +  ASSERT_EFI_ERROR (Status);
> +
>    //
>    // Expose address of CPU Hot Plug Data structure if CPU hot plug is
> supported.
>    //
> diff --git a/UefiCpuPkg/PiSmmCpuDxeSmm/PiSmmCpuDxeSmm.h
> b/UefiCpuPkg/PiSmmCpuDxeSmm/PiSmmCpuDxeSmm.h
> index 2bb35a424d..5df09687e1 100644
> --- a/UefiCpuPkg/PiSmmCpuDxeSmm/PiSmmCpuDxeSmm.h
> +++ b/UefiCpuPkg/PiSmmCpuDxeSmm/PiSmmCpuDxeSmm.h
> @@ -20,6 +20,7 @@ SPDX-License-Identifier: BSD-2-Clause-Patent
>  #include <Protocol/SmmReadyToLock.h>
>  #include <Protocol/SmmCpuService.h>
>  #include <Protocol/SmmMemoryAttribute.h>
> +#include <Protocol/MmMp.h>
> 
>  #include <Guid/AcpiS3Context.h>
>  #include <Guid/MemoryAttributesTable.h>
> @@ -221,11 +222,20 @@ typedef struct {
>    EFI_SMM_CONFIGURATION_PROTOCOL  SmmConfiguration;
>  } SMM_CPU_PRIVATE_DATA;
> 
> +//
> +// Wrapper used to convert EFI_AP_PROCEDURE2 and EFI_AP_PROCEDURE.
> +//
> +typedef struct {
> +  EFI_AP_PROCEDURE  Procedure;
> +  VOID              *ProcedureArgument;
> +} PROCEDURE_WRAPPER;
> +
>  extern SMM_CPU_PRIVATE_DATA  *gSmmCpuPrivate;
>  extern CPU_HOT_PLUG_DATA      mCpuHotPlugData;
>  extern UINTN                  mMaxNumberOfCpus;
>  extern UINTN                  mNumberOfCpus;
>  extern EFI_SMM_CPU_PROTOCOL   mSmmCpu;
> +extern EFI_MM_MP_PROTOCOL     mSmmMp;
> 
>  ///
>  /// The mode of the CPU at the time an SMI occurs
> @@ -363,10 +373,12 @@ SmmRelocationSemaphoreComplete (
>  ///
>  typedef struct {
>    SPIN_LOCK                         *Busy;
> -  volatile EFI_AP_PROCEDURE         Procedure;
> +  volatile EFI_AP_PROCEDURE2        Procedure;
>    volatile VOID                     *Parameter;
>    volatile UINT32                   *Run;
>    volatile BOOLEAN                  *Present;
> +  SPIN_LOCK                         *Token;
> +  EFI_STATUS                        *Status;
>  } SMM_CPU_DATA_BLOCK;
> 
>  typedef enum {
> @@ -388,6 +400,8 @@ typedef struct {
>    volatile SMM_CPU_SYNC_MODE    EffectiveSyncMode;
>    volatile BOOLEAN              SwitchBsp;
>    volatile BOOLEAN              *CandidateBsp;
> +  EFI_AP_PROCEDURE              StartupProcedure;
> +  VOID                          *StartupProcArgs;
>  } SMM_DISPATCHER_MP_SYNC_DATA;
> 
>  #define SMM_PSD_OFFSET              0xfb00
> @@ -410,6 +424,7 @@ typedef struct {
>    SPIN_LOCK                         *Busy;
>    volatile UINT32                   *Run;
>    volatile BOOLEAN                  *Present;
> +  SPIN_LOCK                         *Token;
>  } SMM_CPU_SEMAPHORE_CPU;
> 
>  ///
> @@ -439,6 +454,7 @@ extern SPIN_LOCK
> *mConfigSmmCodeAccessCheckLock;
>  extern EFI_SMRAM_DESCRIPTOR                *mSmmCpuSmramRanges;
>  extern UINTN                               mSmmCpuSmramRangeCount;
>  extern UINT8                               mPhysicalAddressBits;
> +extern SPIN_LOCK                           **mApTokenLock;
> 
>  //
>  // Copy of the PcdPteMemoryEncryptionAddressOrMask
> @@ -1259,4 +1275,146 @@ RestoreCr2 (
>    IN UINTN  Cr2
>    );
> 
> +/**
> +  Schedule a procedure to run on the specified CPU.
> +
> +  @param[in]       Procedure                The address of the procedure to run
> +  @param[in]       CpuIndex                 Target CPU Index
> +  @param[in,out]   ProcArguments            The parameter to pass to the
> procedure
> +  @param[in,out]   Token                    This is an optional parameter that allows
> the caller to execute the
> +                                            procedure in a blocking or non-blocking fashion. If it is
> NULL the
> +                                            call is blocking, and the call will not return until the AP
> has
> +                                            completed the procedure. If the token is not NULL,
> the call will
> +                                            return immediately. The caller can check whether the
> procedure has
> +                                            completed with CheckOnProcedure or
> WaitForProcedure.
> +  @param[in]       TimeoutInMicroseconds    Indicates the time limit in
> microseconds for the APs to finish
> +                                            execution of Procedure, either for blocking or non-
> blocking mode.
> +                                            Zero means infinity. If the timeout expires before all
> APs return
> +                                            from Procedure, then Procedure on the failed APs is
> terminated. If
> +                                            the timeout expires in blocking mode, the call returns
> EFI_TIMEOUT.
> +                                            If the timeout expires in non-blocking mode, the
> timeout determined
> +                                            can be through CheckOnProcedure or
> WaitForProcedure.
> +                                            Note that timeout support is optional. Whether an
> implementation
> +                                            supports this feature can be determined via the
> Attributes data
> +                                            member.
> +  @param[in,out]   CPUStatus                This optional pointer may be used to
> get the status code returned
> +                                            by Procedure when it completes execution on the
> target AP, or with
> +                                            EFI_TIMEOUT if the Procedure fails to complete
> within the optional
> +                                            timeout. The implementation will update this variable
> with
> +                                            EFI_NOT_READY prior to starting Procedure on the
> target AP.
> +
> +
> +  @retval EFI_INVALID_PARAMETER    CpuNumber not valid
> +  @retval EFI_INVALID_PARAMETER    CpuNumber specifying BSP
> +  @retval EFI_INVALID_PARAMETER    The AP specified by CpuNumber did
> not enter SMM
> +  @retval EFI_INVALID_PARAMETER    The AP specified by CpuNumber is
> busy
> +  @retval EFI_SUCCESS              The procedure has been successfully
> scheduled
> +
> +**/
> +EFI_STATUS
> +InternalSmmStartupThisAp (
> +  IN      EFI_AP_PROCEDURE2              Procedure,
> +  IN      UINTN                          CpuIndex,
> +  IN OUT  VOID                           *ProcArguments OPTIONAL,
> +  IN      SPIN_LOCK                      *Token,
> +  IN      UINTN                          TimeoutInMicroseconds,
> +  IN OUT  EFI_STATUS                     *CpuStatus
> +  );
> +
> +/**
> +  Checks whether the input token is the current used token.
> +
> +  @param[in]  CpuIndex   Cpu Index.
> +  @param[in]  Token      This parameter describes the token that was passed
> into DispatchProcedure or
> +                         BroadcastProcedure.
> +
> +  @retval TRUE           The input token is the current used token.
> +  @retval FALSE          The input token is not the current used token.
> +**/
> +BOOLEAN
> +IsCurrentToken (
> +  IN UINTN               CpuIndex,
> +  IN SPIN_LOCK           *Token
> +  );
> +
> +/**
> +  Checks status of specified AP.
> +
> +  This function checks whether the specified AP has finished the task
> assigned
> +  by StartupThisAP(), and whether timeout expires.
> +
> +  @param[in]  Token             This parameter describes the token that was
> passed into DispatchProcedure or
> +                                BroadcastProcedure.
> +
> +  @retval EFI_SUCCESS           Specified AP has finished task assigned by
> StartupThisAPs().
> +  @retval EFI_NOT_READY         Specified AP has not finished task and
> timeout has not expired.
> +**/
> +EFI_STATUS
> +IsApReady (
> +  IN SPIN_LOCK  *Token
> +  );
> +
> +/**
> +  Worker function to execute a caller provided function on all enabled APs.
> +
> +  @param[in]     Procedure               A pointer to the function to be run on
> +                                         enabled APs of the system.
> +  @param[in]     TimeoutInMicroseconds   Indicates the time limit in
> microseconds for
> +                                         APs to return from Procedure, either for
> +                                         blocking or non-blocking mode.
> +  @param[in,out] ProcedureArgument       The parameter passed into
> Procedure for
> +                                         all APs.
> +  @param[in,out] Token                   This is an optional parameter that allows
> the caller to execute the
> +                                         procedure in a blocking or non-blocking fashion. If it is
> NULL the
> +                                         call is blocking, and the call will not return until the AP
> has
> +                                         completed the procedure. If the token is not NULL, the
> call will
> +                                         return immediately. The caller can check whether the
> procedure has
> +                                         completed with CheckOnProcedure or
> WaitForProcedure.
> +  @param[in,out] CPUStatus               This optional pointer may be used to get
> the status code returned
> +                                         by Procedure when it completes execution on the
> target AP, or with
> +                                         EFI_TIMEOUT if the Procedure fails to complete within
> the optional
> +                                         timeout. The implementation will update this variable
> with
> +                                         EFI_NOT_READY prior to starting Procedure on the
> target AP.
> +
> +
> +  @retval EFI_SUCCESS             In blocking mode, all APs have finished before
> +                                  the timeout expired.
> +  @retval EFI_SUCCESS             In non-blocking mode, function has been
> dispatched
> +                                  to all enabled APs.
> +  @retval others                  Failed to Startup all APs.
> +
> +**/
> +EFI_STATUS
> +InternalSmmStartupAllAPs (
> +  IN       EFI_AP_PROCEDURE2             Procedure,
> +  IN       UINTN                         TimeoutInMicroseconds,
> +  IN OUT   VOID                          *ProcedureArguments OPTIONAL,
> +  IN OUT   SPIN_LOCK                     *Token,
> +  IN OUT   EFI_STATUS                    *CPUStatus
> +  );
> +
> +/**
> +
> +  Register the SMM Foundation entry point.
> +
> +  @param[in]      Procedure            A pointer to the code stream to be run on
> the designated target AP
> +                                       of the system. Type EFI_AP_PROCEDURE is defined
> below in Volume 2
> +                                       with the related definitions of
> +                                       EFI_MP_SERVICES_PROTOCOL.StartupAllAPs.
> +                                       If caller may pass a value of NULL to deregister any
> existing
> +                                       startup procedure.
> +  @param[in,out]  ProcedureArguments   Allows the caller to pass a list of
> parameters to the code that is
> +                                       run by the AP. It is an optional common mailbox
> between APs and
> +                                       the caller to share information
> +
> +  @retval EFI_SUCCESS                  The Procedure has been set successfully.
> +  @retval EFI_INVALID_PARAMETER        The Procedure is NULL but
> ProcedureArguments not NULL.
> +
> +**/
> +EFI_STATUS
> +RegisterStartupProcedure (
> +  IN EFI_AP_PROCEDURE    Procedure,
> +  IN VOID                *ProcedureArguments OPTIONAL
> +  );
> +
>  #endif
> diff --git a/UefiCpuPkg/PiSmmCpuDxeSmm/PiSmmCpuDxeSmm.inf
> b/UefiCpuPkg/PiSmmCpuDxeSmm/PiSmmCpuDxeSmm.inf
> index 466c568d49..da0308c47f 100644
> --- a/UefiCpuPkg/PiSmmCpuDxeSmm/PiSmmCpuDxeSmm.inf
> +++ b/UefiCpuPkg/PiSmmCpuDxeSmm/PiSmmCpuDxeSmm.inf
> @@ -40,6 +40,8 @@
>    SmmProfileInternal.h
>    SmramSaveState.c
>    SmmCpuMemoryManagement.c
> +  SmmMp.h
> +  SmmMp.c
> 
>  [Sources.Ia32]
>    Ia32/Semaphore.c
> @@ -105,6 +107,7 @@
>    gEfiSmmReadyToLockProtocolGuid           ## NOTIFY
>    gEfiSmmCpuServiceProtocolGuid            ## PRODUCES
>    gEdkiiSmmMemoryAttributeProtocolGuid     ## PRODUCES
> +  gEfiMmMpProtocolGuid                    ## PRODUCES
> 
>  [Guids]
>    gEfiAcpiVariableGuid                     ## SOMETIMES_CONSUMES ## HOB # it is
> used for S3 boot.
> diff --git a/UefiCpuPkg/PiSmmCpuDxeSmm/SmmMp.c
> b/UefiCpuPkg/PiSmmCpuDxeSmm/SmmMp.c
> new file mode 100644
> index 0000000000..bfbc78691f
> --- /dev/null
> +++ b/UefiCpuPkg/PiSmmCpuDxeSmm/SmmMp.c
> @@ -0,0 +1,372 @@
> +/** @file
> +SMM MP protocol implementation
> +
> +Copyright (c) 2019, Intel Corporation. All rights reserved.<BR>
> +
> +SPDX-License-Identifier: BSD-2-Clause-Patent
> +
> +**/
> +
> +#include "PiSmmCpuDxeSmm.h"
> +#include "SmmMp.h"
> +
> +///
> +/// SMM MP Protocol instance
> +///
> +EFI_MM_MP_PROTOCOL  mSmmMp  = {
> +  EFI_MM_MP_PROTOCOL_REVISION,
> +  0,
> +  SmmMpGetNumberOfProcessors,
> +  SmmMpDispatchProcedure,
> +  SmmMpBroadcastProcedure,
> +  SmmMpSetStartupProcedure,
> +  SmmMpCheckForProcedure,
> +  SmmMpWaitForProcedure
> +};
> +
> +/**
> +  Service to retrieves the number of logical processor in the platform.
> +
> +  @param[in]  This                The EFI_MM_MP_PROTOCOL instance.
> +  @param[out] NumberOfProcessors  Pointer to the total number of logical
> processors in the system,
> +                                  including the BSP and all APs.
> +
> +  @retval EFI_SUCCESS             The number of processors was retrieved
> successfully
> +  @retval EFI_INVALID_PARAMETER   NumberOfProcessors is NULL
> +**/
> +EFI_STATUS
> +EFIAPI
> +SmmMpGetNumberOfProcessors (
> +  IN CONST EFI_MM_MP_PROTOCOL   *This,
> +  OUT      UINTN                *NumberOfProcessors
> +  )
> +{
> +  if (NumberOfProcessors == NULL) {
> +    return EFI_INVALID_PARAMETER;
> +  }
> +
> +  *NumberOfProcessors = gSmmCpuPrivate-
> >SmmCoreEntryContext.NumberOfCpus;
> +
> +  return EFI_SUCCESS;
> +}
> +
> +
> +/**
> +  This service allows the caller to invoke a procedure one of the application
> processors (AP). This
> +  function uses an optional token parameter to support blocking and non-
> blocking modes. If the token
> +  is passed into the call, the function will operate in a non-blocking fashion
> and the caller can
> +  check for completion with CheckOnProcedure or WaitForProcedure.
> +
> +  @param[in]     This                   The EFI_MM_MP_PROTOCOL instance.
> +  @param[in]     Procedure              A pointer to the procedure to be run on
> the designated target
> +                                        AP of the system. Type EFI_AP_PROCEDURE2 is defined
> below in
> +                                        related definitions.
> +  @param[in]     CpuNumber              The zero-based index of the processor
> number of the target
> +                                        AP, on which the code stream is supposed to run. If the
> number
> +                                        points to the calling processor then it will not run the
> +                                        supplied code.
> +  @param[in]     TimeoutInMicroseconds  Indicates the time limit in
> microseconds for this AP to
> +                                        finish execution of Procedure, either for blocking or
> +                                        non-blocking mode. Zero means infinity. If the timeout
> +                                        expires before this AP returns from Procedure, then
> Procedure
> +                                        on the AP is terminated. If the timeout expires in
> blocking
> +                                        mode, the call returns EFI_TIMEOUT. If the timeout
> expires
> +                                        in non-blocking mode, the timeout determined can be
> through
> +                                        CheckOnProcedure or WaitForProcedure.
> +                                        Note that timeout support is optional. Whether an
> +                                        implementation supports this feature, can be
> determined via
> +                                        the Attributes data member.
> +  @param[in,out] ProcedureArguments     Allows the caller to pass a list of
> parameters to the code
> +                                        that is run by the AP. It is an optional common mailbox
> +                                        between APs and the caller to share information.
> +  @param[in,out] Token                  This is parameter is broken into two
> components:
> +                                        1.Token->Completion is an optional parameter that
> allows the
> +                                        caller to execute the procedure in a blocking or non-
> blocking
> +                                        fashion. If it is NULL the call is blocking, and the call will
> +                                        not return until the AP has completed the procedure. If
> the
> +                                        token is not NULL, the call will return immediately. The
> caller
> +                                        can check whether the procedure has completed with
> +                                        CheckOnProcedure or WaitForProcedure.
> +                                        2.Token->Status The implementation updates the
> address pointed
> +                                        at by this variable with the status code returned by
> Procedure
> +                                        when it completes execution on the target AP, or with
> EFI_TIMEOUT
> +                                        if the Procedure fails to complete within the optional
> timeout.
> +                                        The implementation will update this variable with
> EFI_NOT_READY
> +                                        prior to starting Procedure on the target AP
> +  @param[in,out] CPUStatus              This optional pointer may be used to get
> the status code returned
> +                                        by Procedure when it completes execution on the
> target AP, or with
> +                                        EFI_TIMEOUT if the Procedure fails to complete within
> the optional
> +                                        timeout. The implementation will update this variable
> with
> +                                        EFI_NOT_READY prior to starting Procedure on the
> target AP.
> +
> +  @retval EFI_SUCCESS                   In the blocking case, this indicates that
> Procedure has completed
> +                                        execution on the target AP.
> +                                        In the non-blocking case this indicates that the
> procedure has
> +                                        been successfully scheduled for execution on the target
> AP.
> +  @retval EFI_INVALID_PARAMETER         The input arguments are out of
> range. Either the target AP is the
> +                                        caller of the function, or the Procedure or Token is NULL
> +  @retval EFI_NOT_READY                 If the target AP is busy executing another
> procedure
> +  @retval EFI_ALREADY_STARTED           Token is already in use for another
> procedure
> +  @retval EFI_TIMEOUT                   In blocking mode, the timeout expired
> before the specified AP
> +                                        has finished
> +  @retval EFI_OUT_OF_RESOURCES          Could not allocate a required
> resource.
> +
> +**/
> +EFI_STATUS
> +EFIAPI
> +SmmMpDispatchProcedure (
> +  IN CONST EFI_MM_MP_PROTOCOL            *This,
> +  IN       EFI_AP_PROCEDURE2             Procedure,
> +  IN       UINTN                         CpuNumber,
> +  IN       UINTN                         TimeoutInMicroseconds,
> +  IN OUT   VOID                          *ProcedureArguments OPTIONAL,
> +  IN OUT   MM_COMPLETION                 *Token,
> +  IN OUT   EFI_STATUS                    *CPUStatus
> +  )
> +{
> +  SPIN_LOCK      *CpuToken;
> +
> +  if (Token != NULL) {
> +    CpuToken = AllocatePool (sizeof (SPIN_LOCK));
> +    ASSERT (CpuToken != NULL);
> +    if (CpuToken == NULL) {
> +      return EFI_OUT_OF_RESOURCES;
> +    }
> +    InitializeSpinLock ((SPIN_LOCK *)(CpuToken));
> +
> +    *Token = (MM_COMPLETION)CpuToken;
> +  }
> +
> +  return InternalSmmStartupThisAp (
> +    Procedure,
> +    CpuNumber,
> +    ProcedureArguments,
> +    Token != NULL ? CpuToken : NULL,
> +    TimeoutInMicroseconds,
> +    CPUStatus
> +    );
> +}
> +
> +/**
> +  This service allows the caller to invoke a procedure on all running
> application processors (AP)
> +  except the caller. This function uses an optional token parameter to
> support blocking and
> +  nonblocking modes. If the token is passed into the call, the function will
> operate in a non-blocking
> +  fashion and the caller can check for completion with CheckOnProcedure or
> WaitForProcedure.
> +
> +  It is not necessary for the implementation to run the procedure on every
> processor on the platform.
> +  Processors that are powered down in such a way that they cannot respond
> to interrupts, may be
> +  excluded from the broadcast.
> +
> +
> +  @param[in]     This                   The EFI_MM_MP_PROTOCOL instance.
> +  @param[in]     Procedure              A pointer to the code stream to be run on
> the APs that have
> +                                        entered MM. Type EFI_AP_PROCEDURE is defined
> below in related
> +                                        definitions.
> +  @param[in]     TimeoutInMicroseconds  Indicates the time limit in
> microseconds for the APs to finish
> +                                        execution of Procedure, either for blocking or non-
> blocking mode.
> +                                        Zero means infinity. If the timeout expires before all
> APs return
> +                                        from Procedure, then Procedure on the failed APs is
> terminated. If
> +                                        the timeout expires in blocking mode, the call returns
> EFI_TIMEOUT.
> +                                        If the timeout expires in non-blocking mode, the
> timeout determined
> +                                        can be through CheckOnProcedure or
> WaitForProcedure.
> +                                        Note that timeout support is optional. Whether an
> implementation
> +                                        supports this feature can be determined via the
> Attributes data
> +                                        member.
> +  @param[in,out] ProcedureArguments     Allows the caller to pass a list of
> parameters to the code
> +                                        that is run by the AP. It is an optional common mailbox
> +                                        between APs and the caller to share information.
> +  @param[in,out] Token                  This is parameter is broken into two
> components:
> +                                        1.Token->Completion is an optional parameter that
> allows the
> +                                        caller to execute the procedure in a blocking or non-
> blocking
> +                                        fashion. If it is NULL the call is blocking, and the call will
> +                                        not return until the AP has completed the procedure. If
> the
> +                                        token is not NULL, the call will return immediately. The
> caller
> +                                        can check whether the procedure has completed with
> +                                        CheckOnProcedure or WaitForProcedure.
> +                                        2.Token->Status The implementation updates the
> address pointed
> +                                        at by this variable with the status code returned by
> Procedure
> +                                        when it completes execution on the target AP, or with
> EFI_TIMEOUT
> +                                        if the Procedure fails to complete within the optional
> timeout.
> +                                        The implementation will update this variable with
> EFI_NOT_READY
> +                                        prior to starting Procedure on the target AP
> +  @param[in,out] CPUStatus              This optional pointer may be used to get
> the individual status
> +                                        returned by every AP that participated in the broadcast.
> This
> +                                        parameter if used provides the base address of an array
> to hold
> +                                        the EFI_STATUS value of each AP in the system. The size
> of the
> +                                        array can be ascertained by the
> GetNumberOfProcessors function.
> +                                        As mentioned above, the broadcast may not include
> every processor
> +                                        in the system. Some implementations may exclude
> processors that
> +                                        have been powered down in such a way that they are
> not responsive
> +                                        to interrupts. Additionally the broadcast excludes the
> processor
> +                                        which is making the BroadcastProcedure call. For every
> excluded
> +                                        processor, the array entry must contain a value of
> EFI_NOT_STARTED
> +
> +  @retval EFI_SUCCESS                   In the blocking case, this indicates that
> Procedure has completed
> +                                        execution on the APs.
> +                                        In the non-blocking case this indicates that the
> procedure has
> +                                        been successfully scheduled for execution on the APs.
> +  @retval EFI_INVALID_PARAMETER         The Procedure or Token is NULL
> +  @retval EFI_NOT_READY                 If the target AP is busy executing another
> procedure
> +  @retval EFI_ALREADY_STARTED           Token is already in use for another
> procedure
> +  @retval EFI_TIMEOUT                   In blocking mode, the timeout expired
> before the specified AP
> +                                        has finished.
> +  @retval EFI_OUT_OF_RESOURCES          Could not allocate a required
> resource.
> +
> +**/
> +EFI_STATUS
> +EFIAPI
> +SmmMpBroadcastProcedure (
> +  IN CONST EFI_MM_MP_PROTOCOL            *This,
> +  IN       EFI_AP_PROCEDURE2             Procedure,
> +  IN       UINTN                         TimeoutInMicroseconds,
> +  IN OUT   VOID                          *ProcedureArguments OPTIONAL,
> +  IN OUT   MM_COMPLETION                 *Token,
> +  IN OUT   EFI_STATUS                    *CPUStatus
> +  )
> +{
> +  SPIN_LOCK        *CpuToken;
> +
> +  if (Token != NULL) {
> +    CpuToken = AllocatePool (sizeof (SPIN_LOCK));
> +    ASSERT (CpuToken != NULL);
> +    if (CpuToken == NULL) {
> +      return EFI_OUT_OF_RESOURCES;
> +    }
> +    InitializeSpinLock ((SPIN_LOCK *)(CpuToken));
> +
> +    *Token = (MM_COMPLETION)CpuToken;
> +  }
> +
> +  return InternalSmmStartupAllAPs(
> +    Procedure,
> +    TimeoutInMicroseconds,
> +    ProcedureArguments,
> +    Token != NULL ? CpuToken : NULL,
> +    CPUStatus
> +    );
> +}
> +
> +
> +/**
> +  This service allows the caller to set a startup procedure that will be
> executed when an AP powers
> +  up from a state where core configuration and context is lost. The
> procedure is execution has the
> +  following properties:
> +  1. The procedure executes before the processor is handed over to the
> operating system.
> +  2. All processors execute the same startup procedure.
> +  3. The procedure may run in parallel with other procedures invoked
> through the functions in this
> +  protocol, or with processors that are executing an MM handler or running
> in the operating system.
> +
> +
> +  @param[in]      This                 The EFI_MM_MP_PROTOCOL instance.
> +  @param[in]      Procedure            A pointer to the code stream to be run on
> the designated target AP
> +                                       of the system. Type EFI_AP_PROCEDURE is defined
> below in Volume 2
> +                                       with the related definitions of
> +                                       EFI_MP_SERVICES_PROTOCOL.StartupAllAPs.
> +                                       If caller may pass a value of NULL to deregister any
> existing
> +                                       startup procedure.
> +  @param[in,out]  ProcedureArguments   Allows the caller to pass a list of
> parameters to the code that is
> +                                       run by the AP. It is an optional common mailbox
> between APs and
> +                                       the caller to share information
> +
> +  @retval EFI_SUCCESS                  The Procedure has been set successfully.
> +  @retval EFI_INVALID_PARAMETER        The Procedure is NULL but
> ProcedureArguments not NULL.
> +
> +**/
> +EFI_STATUS
> +EFIAPI
> +SmmMpSetStartupProcedure (
> +  IN CONST EFI_MM_MP_PROTOCOL  *This,
> +  IN       EFI_AP_PROCEDURE    Procedure,
> +  IN OUT   VOID                *ProcedureArguments OPTIONAL
> +  )
> +{
> +  return RegisterStartupProcedure (Procedure, ProcedureArguments);
> +}
> +
> +/**
> +  When non-blocking execution of a procedure on an AP is invoked with
> DispatchProcedure,
> +  via the use of a token, this function can be used to check for completion of
> the procedure on the AP.
> +  The function takes the token that was passed into the DispatchProcedure
> call. If the procedure
> +  is complete, and therefore it is now possible to run another procedure on
> the same AP, this function
> +  returns EFI_SUCESS. In this case the status returned by the procedure that
> executed on the AP is
> +  returned in the token's Status field. If the procedure has not yet
> completed, then this function
> +  returns EFI_NOT_READY.
> +
> +  When a non-blocking execution of a procedure is invoked with
> BroadcastProcedure, via the
> +  use of a token, this function can be used to check for completion of the
> procedure on all the
> +  broadcast APs. The function takes the token that was passed into the
> BroadcastProcedure
> +  call. If the procedure is complete on all broadcast APs this function returns
> EFI_SUCESS. In this
> +  case the Status field in the token passed into the function reflects the
> overall result of the
> +  invocation, which may be EFI_SUCCESS, if all executions succeeded, or the
> first observed failure.
> +  If the procedure has not yet completed on the broadcast APs, the function
> returns
> +  EFI_NOT_READY.
> +
> +  @param[in]      This                 The EFI_MM_MP_PROTOCOL instance.
> +  @param[in]      Token                This parameter describes the token that was
> passed into
> +                                       DispatchProcedure or BroadcastProcedure.
> +
> +  @retval EFI_SUCCESS                  Procedure has completed.
> +  @retval EFI_NOT_READY                The Procedure has not completed.
> +  @retval EFI_INVALID_PARAMETER        Token or Token->Completion is
> NULL
> +  @retval EFI_NOT_FOUND                Token is not currently in use for a non-
> blocking call
> +
> +**/
> +EFI_STATUS
> +EFIAPI
> +SmmMpCheckForProcedure (
> +  IN CONST EFI_MM_MP_PROTOCOL            *This,
> +  IN       MM_COMPLETION                 Token
> +  )
> +{
> +  if (Token == NULL) {
> +    return EFI_INVALID_PARAMETER;
> +  }
> +
> +  if (!IsCurrentToken ((UINTN)-1, (SPIN_LOCK *)Token)) {
> +    return EFI_NOT_FOUND;
> +  }
> +
> +  return IsApReady ((SPIN_LOCK *)Token);
> +}
> +
> +/**
> +  When a non-blocking execution of a procedure on an AP is invoked via
> DispatchProcedure,
> +  this function will block the caller until the remote procedure has completed
> on the designated AP.
> +  The non-blocking procedure invocation is identified by the Token
> parameter, which must match the
> +  token that used when DispatchProcedure was called. Upon completion the
> status returned by
> +  the procedure that executed on the AP is used to update the token's
> Status field.
> +
> +  When a non-blocking execution of a procedure on an AP is invoked via
> BroadcastProcedure
> +  this function will block the caller until the remote procedure has completed
> on all of the APs that
> +  entered MM. The non-blocking procedure invocation is identified by the
> Token parameter, which
> +  must match the token that used when BroadcastProcedure was called.
> Upon completion the
> +  overall status returned by the procedures that executed on the broadcast
> AP is used to update the
> +  token's Status field. The overall status may be EFI_SUCCESS, if all
> executions succeeded, or the
> +  first observed failure.
> +
> +
> +  @param[in]      This                 The EFI_MM_MP_PROTOCOL instance.
> +  @param[in]      Token                This parameter describes the token that was
> passed into
> +                                       DispatchProcedure or BroadcastProcedure.
> +
> +  @retval EFI_SUCCESS                  Procedure has completed.
> +  @retval EFI_INVALID_PARAMETER        Token or Token->Completion is
> NULL
> +  @retval EFI_NOT_FOUND                Token is not currently in use for a non-
> blocking call
> +
> +**/
> +EFI_STATUS
> +EFIAPI
> +SmmMpWaitForProcedure (
> +  IN CONST EFI_MM_MP_PROTOCOL            *This,
> +  IN       MM_COMPLETION                 Token
> +  )
> +{
> +  EFI_STATUS    Status;
> +
> +  do {
> +    Status = SmmMpCheckForProcedure (This, Token);
> +  } while (Status == EFI_NOT_READY);
> +
> +  return Status;
> +}
> +
> diff --git a/UefiCpuPkg/PiSmmCpuDxeSmm/SmmMp.h
> b/UefiCpuPkg/PiSmmCpuDxeSmm/SmmMp.h
> new file mode 100644
> index 0000000000..e0d823a4b1
> --- /dev/null
> +++ b/UefiCpuPkg/PiSmmCpuDxeSmm/SmmMp.h
> @@ -0,0 +1,286 @@
> +/** @file
> +Include file for SMM MP protocol implementation.
> +
> +Copyright (c) 2019, Intel Corporation. All rights reserved.<BR>
> +
> +SPDX-License-Identifier: BSD-2-Clause-Patent
> +
> +**/
> +
> +#ifndef _SMM_MP_PROTOCOL_H_
> +#define _SMM_MP_PROTOCOL_H_
> +
> +//
> +// SMM MP Protocol function prototypes.
> +//
> +
> +/**
> +  Service to retrieves the number of logical processor in the platform.
> +
> +  @param[in]  This                The EFI_MM_MP_PROTOCOL instance.
> +  @param[out] NumberOfProcessors  Pointer to the total number of logical
> processors in the system,
> +                                  including the BSP and all APs.
> +
> +  @retval EFI_SUCCESS             The number of processors was retrieved
> successfully
> +  @retval EFI_INVALID_PARAMETER   NumberOfProcessors is NULL
> +**/
> +
> +EFI_STATUS
> +EFIAPI
> +SmmMpGetNumberOfProcessors (
> +  IN CONST EFI_MM_MP_PROTOCOL   *This,
> +  OUT      UINTN                *NumberOfProcessors
> +  );
> +
> +
> +/**
> +  This service allows the caller to invoke a procedure one of the application
> processors (AP). This
> +  function uses an optional token parameter to support blocking and non-
> blocking modes. If the token
> +  is passed into the call, the function will operate in a non-blocking fashion
> and the caller can
> +  check for completion with CheckOnProcedure or WaitForProcedure.
> +
> +  @param[in]     This                   The EFI_MM_MP_PROTOCOL instance.
> +  @param[in]     Procedure              A pointer to the procedure to be run on
> the designated target
> +                                        AP of the system. Type EFI_AP_PROCEDURE2 is defined
> below in
> +                                        related definitions.
> +  @param[in]     CpuNumber              The zero-based index of the processor
> number of the target
> +                                        AP, on which the code stream is supposed to run. If the
> number
> +                                        points to the calling processor then it will not run the
> +                                        supplied code.
> +  @param[in]     TimeoutInMicroseconds  Indicates the time limit in
> microseconds for this AP to
> +                                        finish execution of Procedure, either for blocking or
> +                                        non-blocking mode. Zero means infinity. If the timeout
> +                                        expires before this AP returns from Procedure, then
> Procedure
> +                                        on the AP is terminated. If the timeout expires in
> blocking
> +                                        mode, the call returns EFI_TIMEOUT. If the timeout
> expires
> +                                        in non-blocking mode, the timeout determined can be
> through
> +                                        CheckOnProcedure or WaitForProcedure.
> +                                        Note that timeout support is optional. Whether an
> +                                        implementation supports this feature, can be
> determined via
> +                                        the Attributes data member.
> +  @param[in,out] ProcedureArguments     Allows the caller to pass a list of
> parameters to the code
> +                                        that is run by the AP. It is an optional common mailbox
> +                                        between APs and the caller to share information.
> +  @param[in,out] Token                  This is parameter is broken into two
> components:
> +                                        1.Token->Completion is an optional parameter that
> allows the
> +                                        caller to execute the procedure in a blocking or non-
> blocking
> +                                        fashion. If it is NULL the call is blocking, and the call will
> +                                        not return until the AP has completed the procedure. If
> the
> +                                        token is not NULL, the call will return immediately. The
> caller
> +                                        can check whether the procedure has completed with
> +                                        CheckOnProcedure or WaitForProcedure.
> +                                        2.Token->Status The implementation updates the
> address pointed
> +                                        at by this variable with the status code returned by
> Procedure
> +                                        when it completes execution on the target AP, or with
> EFI_TIMEOUT
> +                                        if the Procedure fails to complete within the optional
> timeout.
> +                                        The implementation will update this variable with
> EFI_NOT_READY
> +                                        prior to starting Procedure on the target AP
> +  @param[in,out] CPUStatus              This optional pointer may be used to get
> the status code returned
> +                                        by Procedure when it completes execution on the
> target AP, or with
> +                                        EFI_TIMEOUT if the Procedure fails to complete within
> the optional
> +                                        timeout. The implementation will update this variable
> with
> +                                        EFI_NOT_READY prior to starting Procedure on the
> target AP.
> +
> +  @retval EFI_SUCCESS                   In the blocking case, this indicates that
> Procedure has completed
> +                                        execution on the target AP.
> +                                        In the non-blocking case this indicates that the
> procedure has
> +                                        been successfully scheduled for execution on the target
> AP.
> +  @retval EFI_INVALID_PARAMETER         The input arguments are out of
> range. Either the target AP is the
> +                                        caller of the function, or the Procedure or Token is NULL
> +  @retval EFI_NOT_READY                 If the target AP is busy executing another
> procedure
> +  @retval EFI_ALREADY_STARTED           Token is already in use for another
> procedure
> +  @retval EFI_TIMEOUT                   In blocking mode, the timeout expired
> before the specified AP
> +                                        has finished
> +  @retval EFI_OUT_OF_RESOURCES          Could not allocate a required
> resource.
> +
> +**/
> +EFI_STATUS
> +EFIAPI
> +SmmMpDispatchProcedure (
> +  IN CONST EFI_MM_MP_PROTOCOL             *This,
> +  IN       EFI_AP_PROCEDURE2              Procedure,
> +  IN       UINTN                          CpuNumber,
> +  IN       UINTN                          TimeoutInMicroseconds,
> +  IN OUT   VOID                           *ProcedureArguments OPTIONAL,
> +  IN OUT   MM_COMPLETION                  *Token,
> +  IN OUT   EFI_STATUS                     *CPUStatus
> +  );
> +
> +/**
> +  This service allows the caller to invoke a procedure on all running
> application processors (AP)
> +  except the caller. This function uses an optional token parameter to
> support blocking and
> +  nonblocking modes. If the token is passed into the call, the function will
> operate in a non-blocking
> +  fashion and the caller can check for completion with CheckOnProcedure or
> WaitForProcedure.
> +
> +  It is not necessary for the implementation to run the procedure on every
> processor on the platform.
> +  Processors that are powered down in such a way that they cannot respond
> to interrupts, may be
> +  excluded from the broadcast.
> +
> +
> +  @param[in]     This                   The EFI_MM_MP_PROTOCOL instance.
> +  @param[in]     Procedure              A pointer to the code stream to be run on
> the APs that have
> +                                        entered MM. Type EFI_AP_PROCEDURE is defined
> below in related
> +                                        definitions.
> +  @param[in]     TimeoutInMicroseconds  Indicates the time limit in
> microseconds for the APs to finish
> +                                        execution of Procedure, either for blocking or non-
> blocking mode.
> +                                        Zero means infinity. If the timeout expires before all
> APs return
> +                                        from Procedure, then Procedure on the failed APs is
> terminated. If
> +                                        the timeout expires in blocking mode, the call returns
> EFI_TIMEOUT.
> +                                        If the timeout expires in non-blocking mode, the
> timeout determined
> +                                        can be through CheckOnProcedure or
> WaitForProcedure.
> +                                        Note that timeout support is optional. Whether an
> implementation
> +                                        supports this feature can be determined via the
> Attributes data
> +                                        member.
> +  @param[in,out] ProcedureArguments     Allows the caller to pass a list of
> parameters to the code
> +                                        that is run by the AP. It is an optional common mailbox
> +                                        between APs and the caller to share information.
> +  @param[in,out] Token                  This is parameter is broken into two
> components:
> +                                        1.Token->Completion is an optional parameter that
> allows the
> +                                        caller to execute the procedure in a blocking or non-
> blocking
> +                                        fashion. If it is NULL the call is blocking, and the call will
> +                                        not return until the AP has completed the procedure. If
> the
> +                                        token is not NULL, the call will return immediately. The
> caller
> +                                        can check whether the procedure has completed with
> +                                        CheckOnProcedure or WaitForProcedure.
> +                                        2.Token->Status The implementation updates the
> address pointed
> +                                        at by this variable with the status code returned by
> Procedure
> +                                        when it completes execution on the target AP, or with
> EFI_TIMEOUT
> +                                        if the Procedure fails to complete within the optional
> timeout.
> +                                        The implementation will update this variable with
> EFI_NOT_READY
> +                                        prior to starting Procedure on the target AP
> +  @param[in,out] CPUStatus              This optional pointer may be used to get
> the individual status
> +                                        returned by every AP that participated in the broadcast.
> This
> +                                        parameter if used provides the base address of an array
> to hold
> +                                        the EFI_STATUS value of each AP in the system. The size
> of the
> +                                        array can be ascertained by the
> GetNumberOfProcessors function.
> +                                        As mentioned above, the broadcast may not include
> every processor
> +                                        in the system. Some implementations may exclude
> processors that
> +                                        have been powered down in such a way that they are
> not responsive
> +                                        to interrupts. Additionally the broadcast excludes the
> processor
> +                                        which is making the BroadcastProcedure call. For every
> excluded
> +                                        processor, the array entry must contain a value of
> EFI_NOT_STARTED
> +
> +  @retval EFI_SUCCESS                   In the blocking case, this indicates that
> Procedure has completed
> +                                        execution on the APs.
> +                                        In the non-blocking case this indicates that the
> procedure has
> +                                        been successfully scheduled for execution on the APs.
> +  @retval EFI_INVALID_PARAMETER         The Procedure or Token is NULL
> +  @retval EFI_NOT_READY                 If the target AP is busy executing another
> procedure
> +  @retval EFI_ALREADY_STARTED           Token is already in use for another
> procedure
> +  @retval EFI_TIMEOUT                   In blocking mode, the timeout expired
> before the specified AP
> +                                        has finished
> +  @retval EFI_OUT_OF_RESOURCES          Could not allocate a required
> resource.
> +
> +**/
> +EFI_STATUS
> +EFIAPI
> +SmmMpBroadcastProcedure (
> +  IN CONST EFI_MM_MP_PROTOCOL             *This,
> +  IN       EFI_AP_PROCEDURE2              Procedure,
> +  IN       UINTN                          TimeoutInMicroseconds,
> +  IN OUT   VOID                           *ProcedureArguments OPTIONAL,
> +  IN OUT   MM_COMPLETION                  *Token,
> +  IN OUT   EFI_STATUS                     *CPUStatus
> +  );
> +
> +
> +/**
> +  This service allows the caller to set a startup procedure that will be
> executed when an AP powers
> +  up from a state where core configuration and context is lost. The
> procedure is execution has the
> +  following properties:
> +  1. The procedure executes before the processor is handed over to the
> operating system.
> +  2. All processors execute the same startup procedure.
> +  3. The procedure may run in parallel with other procedures invoked
> through the functions in this
> +  protocol, or with processors that are executing an MM handler or running
> in the operating system.
> +
> +
> +  @param[in]      This                 The EFI_MM_MP_PROTOCOL instance.
> +  @param[in]      Procedure            A pointer to the code stream to be run on
> the designated target AP
> +                                       of the system. Type EFI_AP_PROCEDURE is defined
> below in Volume 2
> +                                       with the related definitions of
> +                                       EFI_MP_SERVICES_PROTOCOL.StartupAllAPs.
> +                                       If caller may pass a value of NULL to deregister any
> existing
> +                                       startup procedure.
> +  @param[in,out]  ProcedureArguments   Allows the caller to pass a list of
> parameters to the code that is
> +                                       run by the AP. It is an optional common mailbox
> between APs and
> +                                       the caller to share information
> +
> +  @retval EFI_SUCCESS                  The Procedure has been set successfully.
> +  @retval EFI_INVALID_PARAMETER        The Procedure is NULL but
> ProcedureArguments not NULL.
> +**/
> +EFI_STATUS
> +EFIAPI
> +SmmMpSetStartupProcedure (
> +  IN CONST EFI_MM_MP_PROTOCOL  *This,
> +  IN       EFI_AP_PROCEDURE    Procedure,
> +  IN OUT   VOID                *ProcedureArguments OPTIONAL
> +  );
> +
> +/**
> +  When non-blocking execution of a procedure on an AP is invoked with
> DispatchProcedure,
> +  via the use of a token, this function can be used to check for completion of
> the procedure on the AP.
> +  The function takes the token that was passed into the DispatchProcedure
> call. If the procedure
> +  is complete, and therefore it is now possible to run another procedure on
> the same AP, this function
> +  returns EFI_SUCESS. In this case the status returned by the procedure that
> executed on the AP is
> +  returned in the token's Status field. If the procedure has not yet
> completed, then this function
> +  returns EFI_NOT_READY.
> +
> +  When a non-blocking execution of a procedure is invoked with
> BroadcastProcedure, via the
> +  use of a token, this function can be used to check for completion of the
> procedure on all the
> +  broadcast APs. The function takes the token that was passed into the
> BroadcastProcedure
> +  call. If the procedure is complete on all broadcast APs this function returns
> EFI_SUCESS. In this
> +  case the Status field in the token passed into the function reflects the
> overall result of the
> +  invocation, which may be EFI_SUCCESS, if all executions succeeded, or the
> first observed failure.
> +  If the procedure has not yet completed on the broadcast APs, the function
> returns
> +  EFI_NOT_READY.
> +
> +  @param[in]      This                 The EFI_MM_MP_PROTOCOL instance.
> +  @param[in]      Token                This parameter describes the token that was
> passed into
> +                                       DispatchProcedure or BroadcastProcedure.
> +
> +  @retval EFI_SUCCESS                  Procedure has completed.
> +  @retval EFI_NOT_READY                The Procedure has not completed.
> +  @retval EFI_INVALID_PARAMETER        Token or Token->Completion is
> NULL
> +  @retval EFI_NOT_FOUND                Token is not currently in use for a non-
> blocking call
> +
> +**/
> +EFI_STATUS
> +EFIAPI
> +SmmMpCheckForProcedure (
> +  IN CONST EFI_MM_MP_PROTOCOL             *This,
> +  IN       MM_COMPLETION                  Token
> +  );
> +
> +/**
> +  When a non-blocking execution of a procedure on an AP is invoked via
> DispatchProcedure,
> +  this function will block the caller until the remote procedure has completed
> on the designated AP.
> +  The non-blocking procedure invocation is identified by the Token
> parameter, which must match the
> +  token that used when DispatchProcedure was called. Upon completion the
> status returned by
> +  the procedure that executed on the AP is used to update the token's
> Status field.
> +
> +  When a non-blocking execution of a procedure on an AP is invoked via
> BroadcastProcedure
> +  this function will block the caller until the remote procedure has completed
> on all of the APs that
> +  entered MM. The non-blocking procedure invocation is identified by the
> Token parameter, which
> +  must match the token that used when BroadcastProcedure was called.
> Upon completion the
> +  overall status returned by the procedures that executed on the broadcast
> AP is used to update the
> +  token's Status field. The overall status may be EFI_SUCCESS, if all
> executions succeeded, or the
> +  first observed failure.
> +
> +
> +  @param[in]      This                 The EFI_MM_MP_PROTOCOL instance.
> +  @param[in]      Token                This parameter describes the token that was
> passed into
> +                                       DispatchProcedure or BroadcastProcedure.
> +
> +  @retval EFI_SUCCESS                  Procedure has completed.
> +  @retval EFI_INVALID_PARAMETER        Token or Token->Completion is
> NULL
> +  @retval EFI_NOT_FOUND                Token is not currently in use for a non-
> blocking call
> +
> +**/
> +EFI_STATUS
> +EFIAPI
> +SmmMpWaitForProcedure (
> +  IN CONST EFI_MM_MP_PROTOCOL            *This,
> +  IN       MM_COMPLETION                 Token
> +  );
> +
> +#endif
> --
> 2.21.0.windows.1
> 
> 
> 

Re: [edk2-devel] [Patch v3 2/2] UefiCpuPkg/PiSmmCpuDxeSmm: Enable MM MP Protocol.

[Dong, Eric]

Hi Ray,

> -----Original Message-----
> From: Ni, Ray
> Sent: Thursday, July 4, 2019 2:18 PM
> To: devel@edk2.groups.io; Dong, Eric <eric.dong@intel.com>
> Cc: Laszlo Ersek <lersek@redhat.com>
> Subject: RE: [edk2-devel] [Patch v3 2/2] UefiCpuPkg/PiSmmCpuDxeSmm:
> Enable MM MP Protocol.
> 
> 1. mMaxNumberOfCpus and gSmmCpuPrivate-
> >SmmCoreEntryContext.NumberOfCpus are both used. Can you please
> refine the code to use only one?
[Eric] will do it in other patch.
 
> 2. I do think that MM_MP code needs to maintain a internal linked list
> containing all created tokens. Because it's possible that caller is written in
> below way:
>     DispatchProcedure (AP#1, &TokenA);
>     DispatchProcedure (AP#1, &TokenB);
>     DispatchProcedure (AP#1, &TokenC);
>     CheckOnProcedure (&TokenA);
>     Without having a list to contain all created tokens, CheckOnProcedure
> (&TokenA) may return NotFound.
>     The linked list can be freed upon exiting SMM
[Eric] will enable it in my next version change. 
 
> 3. IsCurrentToken() can be changed to look up the token in the linked list and
> the name can be changed to IsTokenInUse().
>     The function name reflects the spec wording "EFI_ALREADY_STARTED -
> Token is already in use for another procedure."	
[Eric]  will enable it in my next version change.

> 4. In InternalSmmStartup[AllAPs|ThisAp], Token can be acquired after Busy is
> acquired. Token is associated with the procedure.
>      Busy is associated with the individual processor.
[Eric]  will enable it in my next version change.

> 5. *(CPUStatus + Index) can be changed to CPUStatus[Index]. Use [] instead
> of address operation.
[Eric]  will enable it in my next version change.

> 6. Below error handling in InternalSmmStartupAllAPs may not be necessary.
> Just AcquireSpinLock (...Busy) should be ok.
>     //
>     // Make sure all BUSY should be acquired.
>     //
>     for (Index = mMaxNumberOfCpus; Index-- > 0;) {
>       if (Index != gSmmCpuPrivate-
> >SmmCoreEntryContext.CurrentlyExecutingCpu && *(mSmmMpSyncData-
> >CpuData[Index].Present)) {
>         if (!AcquireSpinLockOrFail (mSmmMpSyncData->CpuData[Index].Busy)) {
>           DEBUG((DEBUG_ERROR, "Can't acquire mSmmMpSyncData-
> >CpuData[%d].Busy\n", Index));
> 
>           //
>           // Release BUSY accquired before.
>           //
>           for (CpuCount = mMaxNumberOfCpus; CpuCount -- > Index;) {
>             if (CpuCount != gSmmCpuPrivate-
> >SmmCoreEntryContext.CurrentlyExecutingCpu && *(mSmmMpSyncData-
> >CpuData[CpuCount].Present)) {
>               ReleaseSpinLock (mSmmMpSyncData->CpuData[CpuCount].Busy);
>             }
>           }
> 
>           ReleaseSpinLock (Token);
>           return EFI_INVALID_PARAMETER;
>         }
>       }
>     }
[Eric]  will enable it in my next version change.

> 7. IsTaskFinishInAllAPs() can be used by BSPHandler() as well. The function
> name can be WaitForAllAPsNotBusy() or some other name because the
> function
>     has side affect, not just check.
[Eric]  will enable it in my next version change.

> 8. Maybe you don't need to have a mApTokenLock[]. For old style async
> SMM MP call, the token is not returned to caller. But it's still possible to be
> freed
>      as long as it's inserted to the linked list.
[Eric]  will enable it in my next version change.

Thanks,
Eric
> 
> Thanks,
> Ray
> 
> > -----Original Message-----
> > From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Dong,
> > Eric
> > Sent: Wednesday, July 3, 2019 10:43 AM
> > To: devel@edk2.groups.io
> > Cc: Ni, Ray <ray.ni@intel.com>; Laszlo Ersek <lersek@redhat.com>
> > Subject: [edk2-devel] [Patch v3 2/2] UefiCpuPkg/PiSmmCpuDxeSmm:
> Enable
> > MM MP Protocol.
> >
> > v3 changes:
> > 1. Fix Token clean up too early caused CheckProcedure return error.
> >
> > v2 changes:
> > 1. Remove some duplicated global variables.
> > 2. Enhance token design to support multiple task trig for different APs at
> the
> > same time.
> >
> > V1 changes:
> > REF: https://bugzilla.tianocore.org/show_bug.cgi?id=1937
> >
> > Add MM Mp Protocol in PiSmmCpuDxeSmm driver.
> >
> > Cc: Ray Ni <ray.ni@intel.com>
> > Cc: Laszlo Ersek <lersek@redhat.com>
> > Signed-off-by: Eric Dong <eric.dong@intel.com>
> > ---
> >  UefiCpuPkg/PiSmmCpuDxeSmm/MpService.c        | 536
> > ++++++++++++++++++-
> >  UefiCpuPkg/PiSmmCpuDxeSmm/PiSmmCpuDxeSmm.c   |  11 +
> >  UefiCpuPkg/PiSmmCpuDxeSmm/PiSmmCpuDxeSmm.h   | 160 +++++-
> >  UefiCpuPkg/PiSmmCpuDxeSmm/PiSmmCpuDxeSmm.inf |   3 +
> >  UefiCpuPkg/PiSmmCpuDxeSmm/SmmMp.c            | 372 +++++++++++++
> >  UefiCpuPkg/PiSmmCpuDxeSmm/SmmMp.h            | 286 ++++++++++
> >  6 files changed, 1351 insertions(+), 17 deletions(-)
> >  create mode 100644 UefiCpuPkg/PiSmmCpuDxeSmm/SmmMp.c
> >  create mode 100644 UefiCpuPkg/PiSmmCpuDxeSmm/SmmMp.h
> >
> > diff --git a/UefiCpuPkg/PiSmmCpuDxeSmm/MpService.c
> > b/UefiCpuPkg/PiSmmCpuDxeSmm/MpService.c
> > index 64fb4d6344..13858f5d2d 100644
> > --- a/UefiCpuPkg/PiSmmCpuDxeSmm/MpService.c
> > +++ b/UefiCpuPkg/PiSmmCpuDxeSmm/MpService.c
> > @@ -22,6 +22,7 @@ UINTN                                       mSemaphoreSize;
> >  SPIN_LOCK                                   *mPFLock = NULL;
> >  SMM_CPU_SYNC_MODE                           mCpuSmmSyncMode;
> >  BOOLEAN                                     mMachineCheckSupported = FALSE;
> > +SPIN_LOCK                                   **mApTokenLock;
> >
> >  /**
> >    Performs an atomic compare exchange operation to get semaphore.
> > @@ -146,6 +147,45 @@ ReleaseAllAPs (
> >    }
> >  }
> >
> > +/**
> > +  Wheck whether task has been finished by all APs.
> > +
> > +  @param       BlockMode   Whether did it in block mode or non-block
> mode.
> > +
> > +  @retval      TRUE        Task has been finished by all APs.
> > +  @retval      FALSE       Task not has been finished by all APs.
> > +
> > +**/
> > +BOOLEAN
> > +IsTaskFinishInAllAPs (
> > +  IN BOOLEAN                        BlockMode
> > +  )
> > +{
> > +  UINTN                             Index;
> > +
> > +  for (Index = mMaxNumberOfCpus; Index-- > 0;) {
> > +    //
> > +    // Ignore BSP and APs which not call in SMM.
> > +    //
> > +    if ((Index == gSmmCpuPrivate-
> > >SmmCoreEntryContext.CurrentlyExecutingCpu) ||
> (!*(mSmmMpSyncData-
> > >CpuData[Index].Present))) {
> > +      continue;
> > +    }
> > +
> > +    if (BlockMode) {
> > +      AcquireSpinLock(mSmmMpSyncData->CpuData[Index].Busy);
> > +      ReleaseSpinLock(mSmmMpSyncData->CpuData[Index].Busy);
> > +    } else {
> > +      if (AcquireSpinLockOrFail (mSmmMpSyncData->CpuData[Index].Busy))
> {
> > +        ReleaseSpinLock(mSmmMpSyncData->CpuData[Index].Busy);
> > +      } else {
> > +        return FALSE;
> > +      }
> > +    }
> > +  }
> > +
> > +  return TRUE;
> > +}
> > +
> >  /**
> >    Checks if all CPUs (with certain exceptions) have checked in for this SMI
> run
> >
> > @@ -347,6 +387,92 @@ ReplaceOSMtrrs (
> >    MtrrSetAllMtrrs (&gSmiMtrrs);
> >  }
> >
> > +/**
> > +  Check whether execute in single AP or all APs.
> > +
> > +  Compare two Tokens used by different APs to know whether in
> > StartAllAps call.
> > +
> > +  Whether is an valid AP base on AP's Present flag.
> > +
> > +  @retval  TRUE      IN StartAllAps call.
> > +  @retval  FALSE     Not in StartAllAps call.
> > +
> > +**/
> > +BOOLEAN
> > +InStartAllApsCall (
> > +  VOID
> > +  )
> > +{
> > +  UINTN      ApIndex;
> > +  UINTN      ApIndex2;
> > +
> > +  for (ApIndex = mMaxNumberOfCpus; ApIndex-- > 0;) {
> > +    if ((ApIndex != gSmmCpuPrivate-
> > >SmmCoreEntryContext.CurrentlyExecutingCpu) &&
> > +        *(mSmmMpSyncData->CpuData[ApIndex].Present) &&
> > +        (mSmmMpSyncData->CpuData[ApIndex].Token != NULL)) {
> > +      for (ApIndex2 = ApIndex; ApIndex2-- > 0;) {
> > +        if ((ApIndex2 != gSmmCpuPrivate-
> > >SmmCoreEntryContext.CurrentlyExecutingCpu) &&
> > +            *(mSmmMpSyncData->CpuData[ApIndex2].Present) &&
> > +            (mSmmMpSyncData->CpuData[ApIndex2].Token != NULL)) {
> > +          return mSmmMpSyncData->CpuData[ApIndex2].Token ==
> > mSmmMpSyncData->CpuData[ApIndex].Token;
> > +        }
> > +      }
> > +    }
> > +  }
> > +
> > +  return FALSE;
> > +}
> > +
> > +/**
> > +  Clean up the status flags used during executing the procedure.
> > +
> > +  @param   CpuIndex      The AP index which calls this function.
> > +
> > +**/
> > +VOID
> > +CleanUpStatusFlags (
> > +  IN UINTN                  CpuIndex
> > +  )
> > +{
> > +  UINTN                             Index;
> > +
> > +  if (InStartAllApsCall ()) {
> > +    //
> > +    // In Start All APs mode, make sure all APs have finished task.
> > +    //
> > +    if (IsTaskFinishInAllAPs (FALSE)) {
> > +      //
> > +      // Clean the flags update in the function call.
> > +      //
> > +      for (Index = mMaxNumberOfCpus; Index-- > 0;) {
> > +        //
> > +        // Only In SMM APs need to be clean up.
> > +        //
> > +        if (mSmmMpSyncData->CpuData[Index].Present) {
> > +          if (mSmmMpSyncData->CpuData[Index].Token != NULL) {
> > +            ReleaseSpinLock (mSmmMpSyncData->CpuData[Index].Token);
> > +            break;
> > +          }
> > +        }
> > +      }
> > +    }
> > +  } else {
> > +    //
> > +    // In single AP mode.
> > +    //
> > +    if (mSmmMpSyncData->CpuData[CpuIndex].Token != NULL) {
> > +      //
> > +      // Free the wrapper buffer in non-block mode used by
> > SmmMpDispatchProcedure function.
> > +      //
> > +      if (mApTokenLock[CpuIndex] == mSmmMpSyncData-
> > >CpuData[CpuIndex].Token) {
> > +        FreePool ((VOID *)mSmmMpSyncData-
> > >CpuData[CpuIndex].Parameter);
> > +      }
> > +
> > +      ReleaseSpinLock (mSmmMpSyncData->CpuData[CpuIndex].Token);
> > +    }
> > +  }
> > +}
> > +
> >  /**
> >    SMI handler for BSP.
> >
> > @@ -604,6 +730,7 @@ APHandler (
> >    UINT64                            Timer;
> >    UINTN                             BspIndex;
> >    MTRR_SETTINGS                     Mtrrs;
> > +  EFI_STATUS                        ProcedureStatus;
> >
> >    //
> >    // Timeout BSP
> > @@ -730,14 +857,19 @@ APHandler (
> >      //
> >      // Invoke the scheduled procedure
> >      //
> > -    (*mSmmMpSyncData->CpuData[CpuIndex].Procedure) (
> > -      (VOID*)mSmmMpSyncData->CpuData[CpuIndex].Parameter
> > -      );
> > +    ProcedureStatus = (*mSmmMpSyncData-
> > >CpuData[CpuIndex].Procedure) (
> > +                          (VOID*)mSmmMpSyncData->CpuData[CpuIndex].Parameter
> > +                          );
> > +    if (mSmmMpSyncData->CpuData[CpuIndex].Status != NULL) {
> > +      *mSmmMpSyncData->CpuData[CpuIndex].Status = ProcedureStatus;
> > +    }
> >
> >      //
> >      // Release BUSY
> >      //
> >      ReleaseSpinLock (mSmmMpSyncData->CpuData[CpuIndex].Busy);
> > +
> > +    CleanUpStatusFlags (CpuIndex);
> >    }
> >
> >    if (SmmCpuFeaturesNeedConfigureMtrrs()) {
> > @@ -906,13 +1038,95 @@ Gen4GPageTable (
> >    return (UINT32)(UINTN)PageTable;
> >  }
> >
> > +/**
> > +  Checks whether the input token is the current used token.
> > +
> > +  @param[in]  CpuIndex   Cpu Index.
> > +  @param[in]  Token      This parameter describes the token that was
> passed
> > into DispatchProcedure or
> > +                         BroadcastProcedure.
> > +
> > +  @retval TRUE           The input token is the current used token.
> > +  @retval FALSE          The input token is not the current used token.
> > +**/
> > +BOOLEAN
> > +IsCurrentToken (
> > +  IN UINTN               CpuIndex,
> > +  IN SPIN_LOCK           *Token
> > +  )
> > +{
> > +  UINTN         Index;
> > +
> > +  if (Token == NULL) {
> > +    return FALSE;
> > +  }
> > +
> > +  if (CpuIndex == (UINTN) -1) {
> > +    for (Index = 0; Index < gSmmCpuPrivate-
> > >SmmCoreEntryContext.NumberOfCpus; Index ++) {
> > +      if (mSmmMpSyncData->CpuData[Index].Present &&
> > +          (mSmmMpSyncData->CpuData[Index].Token != NULL) &&
> > +          (mSmmMpSyncData->CpuData[Index].Token == Token)) {
> > +        return TRUE;
> > +      }
> > +    }
> > +
> > +    return FALSE;
> > +  }
> > +
> > +  return mSmmMpSyncData->CpuData[CpuIndex].Token == Token;
> > +}
> > +
> > +/**
> > +  Checks status of specified AP.
> > +
> > +  This function checks whether the specified AP has finished the task
> > assigned
> > +  by StartupThisAP(), and whether timeout expires.
> > +
> > +  @param[in]  Token             This parameter describes the token that was
> > passed into DispatchProcedure or
> > +                                BroadcastProcedure.
> > +
> > +  @retval EFI_SUCCESS           Specified AP has finished task assigned by
> > StartupThisAPs().
> > +  @retval EFI_NOT_READY         Specified AP has not finished task and
> > timeout has not expired.
> > +**/
> > +EFI_STATUS
> > +IsApReady (
> > +  IN SPIN_LOCK          *Token
> > +  )
> > +{
> > +  if (AcquireSpinLockOrFail (Token)) {
> > +    ReleaseSpinLock (Token);
> > +    return EFI_SUCCESS;
> > +  }
> > +
> > +  return EFI_NOT_READY;
> > +}
> > +
> >  /**
> >    Schedule a procedure to run on the specified CPU.
> >
> >    @param[in]       Procedure                The address of the procedure to run
> >    @param[in]       CpuIndex                 Target CPU Index
> > -  @param[in, out]  ProcArguments            The parameter to pass to the
> > procedure
> > -  @param[in]       BlockingMode             Startup AP in blocking mode or not
> > +  @param[in,out]   ProcArguments            The parameter to pass to the
> > procedure
> > +  @param[in]       Token                    This is an optional parameter that allows
> > the caller to execute the
> > +                                            procedure in a blocking or non-blocking fashion. If it
> is
> > NULL the
> > +                                            call is blocking, and the call will not return until the
> AP
> > has
> > +                                            completed the procedure. If the token is not NULL,
> > the call will
> > +                                            return immediately. The caller can check whether
> the
> > procedure has
> > +                                            completed with CheckOnProcedure or
> > WaitForProcedure.
> > +  @param[in]       TimeoutInMicroseconds    Indicates the time limit in
> > microseconds for the APs to finish
> > +                                            execution of Procedure, either for blocking or non-
> > blocking mode.
> > +                                            Zero means infinity. If the timeout expires before all
> > APs return
> > +                                            from Procedure, then Procedure on the failed APs is
> > terminated. If
> > +                                            the timeout expires in blocking mode, the call
> returns
> > EFI_TIMEOUT.
> > +                                            If the timeout expires in non-blocking mode, the
> > timeout determined
> > +                                            can be through CheckOnProcedure or
> > WaitForProcedure.
> > +                                            Note that timeout support is optional. Whether an
> > implementation
> > +                                            supports this feature can be determined via the
> > Attributes data
> > +                                            member.
> > +  @param[in,out]   CpuStatus                This optional pointer may be used to
> > get the status code returned
> > +                                            by Procedure when it completes execution on the
> > target AP, or with
> > +                                            EFI_TIMEOUT if the Procedure fails to complete
> > within the optional
> > +                                            timeout. The implementation will update this
> variable
> > with
> > +                                            EFI_NOT_READY prior to starting Procedure on the
> > target AP.
> >
> >    @retval EFI_INVALID_PARAMETER    CpuNumber not valid
> >    @retval EFI_INVALID_PARAMETER    CpuNumber specifying BSP
> > @@ -923,10 +1137,12 @@ Gen4GPageTable (
> >  **/
> >  EFI_STATUS
> >  InternalSmmStartupThisAp (
> > -  IN      EFI_AP_PROCEDURE          Procedure,
> > -  IN      UINTN                     CpuIndex,
> > -  IN OUT  VOID                      *ProcArguments OPTIONAL,
> > -  IN      BOOLEAN                   BlockingMode
> > +  IN      EFI_AP_PROCEDURE2              Procedure,
> > +  IN      UINTN                          CpuIndex,
> > +  IN OUT  VOID                           *ProcArguments OPTIONAL,
> > +  IN      SPIN_LOCK                      *Token,
> > +  IN      UINTN                          TimeoutInMicroseconds,
> > +  IN OUT  EFI_STATUS                     *CpuStatus
> >    )
> >  {
> >    if (CpuIndex >= gSmmCpuPrivate-
> >SmmCoreEntryContext.NumberOfCpus)
> > {
> > @@ -952,24 +1168,208 @@ InternalSmmStartupThisAp (
> >      }
> >      return EFI_INVALID_PARAMETER;
> >    }
> > +  if ((TimeoutInMicroseconds != 0) && ((mSmmMp.Attributes &
> > EFI_MM_MP_TIMEOUT_SUPPORTED) == 0)) {
> > +    return EFI_INVALID_PARAMETER;
> > +  }
> > +  if (Procedure == NULL) {
> > +    return EFI_INVALID_PARAMETER;
> > +  }
> > +  if (IsCurrentToken (CpuIndex, Token)) {
> > +    return EFI_ALREADY_STARTED;
> > +  }
> >
> > -  if (BlockingMode) {
> > +  if (Token == NULL) {
> >      AcquireSpinLock (mSmmMpSyncData->CpuData[CpuIndex].Busy);
> >    } else {
> > -    if (!AcquireSpinLockOrFail (mSmmMpSyncData-
> > >CpuData[CpuIndex].Busy)) {
> > -      DEBUG((DEBUG_ERROR, "mSmmMpSyncData->CpuData[%d].Busy\n",
> > CpuIndex));
> > +    if (!AcquireSpinLockOrFail (Token)) {
> > +      DEBUG((DEBUG_ERROR, "Token can't acquire\n"));
> >        return EFI_INVALID_PARAMETER;
> >      }
> > +    if (!AcquireSpinLockOrFail (mSmmMpSyncData-
> > >CpuData[CpuIndex].Busy)) {
> > +      DEBUG((DEBUG_ERROR, "Can't acquire mSmmMpSyncData-
> > >CpuData[%d].Busy\n", CpuIndex));
> > +      ReleaseSpinLock (Token);
> > +      return EFI_NOT_READY;
> > +    }
> >    }
> >
> >    mSmmMpSyncData->CpuData[CpuIndex].Procedure = Procedure;
> >    mSmmMpSyncData->CpuData[CpuIndex].Parameter = ProcArguments;
> > +  mSmmMpSyncData->CpuData[CpuIndex].Token     = Token;
> > +  mSmmMpSyncData->CpuData[CpuIndex].Status    = CpuStatus;
> > +  if (mSmmMpSyncData->CpuData[CpuIndex].Status != NULL) {
> > +    *mSmmMpSyncData->CpuData[CpuIndex].Status = EFI_NOT_READY;
> > +  }
> > +
> >    ReleaseSemaphore (mSmmMpSyncData->CpuData[CpuIndex].Run);
> >
> > -  if (BlockingMode) {
> > +  if (Token == NULL) {
> >      AcquireSpinLock (mSmmMpSyncData->CpuData[CpuIndex].Busy);
> >      ReleaseSpinLock (mSmmMpSyncData->CpuData[CpuIndex].Busy);
> >    }
> > +
> > +  return EFI_SUCCESS;
> > +}
> > +
> > +/**
> > +  Worker function to execute a caller provided function on all enabled APs.
> > +
> > +  @param[in]     Procedure               A pointer to the function to be run on
> > +                                         enabled APs of the system.
> > +  @param[in]     TimeoutInMicroseconds   Indicates the time limit in
> > microseconds for
> > +                                         APs to return from Procedure, either for
> > +                                         blocking or non-blocking mode.
> > +  @param[in,out] ProcedureArguments      The parameter passed into
> > Procedure for
> > +                                         all APs.
> > +  @param[in,out] Token                   This is an optional parameter that allows
> > the caller to execute the
> > +                                         procedure in a blocking or non-blocking fashion. If it is
> > NULL the
> > +                                         call is blocking, and the call will not return until the AP
> > has
> > +                                         completed the procedure. If the token is not NULL,
> the
> > call will
> > +                                         return immediately. The caller can check whether the
> > procedure has
> > +                                         completed with CheckOnProcedure or
> > WaitForProcedure.
> > +  @param[in,out] CPUStatus               This optional pointer may be used to
> get
> > the status code returned
> > +                                         by Procedure when it completes execution on the
> > target AP, or with
> > +                                         EFI_TIMEOUT if the Procedure fails to complete
> within
> > the optional
> > +                                         timeout. The implementation will update this variable
> > with
> > +                                         EFI_NOT_READY prior to starting Procedure on the
> > target AP.
> > +
> > +
> > +  @retval EFI_SUCCESS             In blocking mode, all APs have finished
> before
> > +                                  the timeout expired.
> > +  @retval EFI_SUCCESS             In non-blocking mode, function has been
> > dispatched
> > +                                  to all enabled APs.
> > +  @retval others                  Failed to Startup all APs.
> > +
> > +**/
> > +EFI_STATUS
> > +InternalSmmStartupAllAPs (
> > +  IN       EFI_AP_PROCEDURE2             Procedure,
> > +  IN       UINTN                         TimeoutInMicroseconds,
> > +  IN OUT   VOID                          *ProcedureArguments OPTIONAL,
> > +  IN OUT   SPIN_LOCK                     *Token,
> > +  IN OUT   EFI_STATUS                    *CPUStatus
> > +  )
> > +{
> > +  UINTN               Index;
> > +  UINTN               CpuCount;
> > +
> > +  if ((TimeoutInMicroseconds != 0) && ((mSmmMp.Attributes &
> > EFI_MM_MP_TIMEOUT_SUPPORTED) == 0)) {
> > +    return EFI_INVALID_PARAMETER;
> > +  }
> > +  if (Procedure == NULL) {
> > +    return EFI_INVALID_PARAMETER;
> > +  }
> > +  if (IsCurrentToken ((UINTN)-1, Token)) {
> > +    return EFI_ALREADY_STARTED;
> > +  }
> > +
> > +  CpuCount = 0;
> > +  for (Index = mMaxNumberOfCpus; Index-- > 0;) {
> > +    if (*mSmmMpSyncData->CpuData[Index].Present && (Index !=
> > gSmmCpuPrivate->SmmCoreEntryContext.CurrentlyExecutingCpu)) {
> > +      CpuCount ++;
> > +
> > +      if (gSmmCpuPrivate->Operation[Index] == SmmCpuRemove) {
> > +        return EFI_INVALID_PARAMETER;
> > +      }
> > +
> > +      if (!AcquireSpinLockOrFail(mSmmMpSyncData->CpuData[Index].Busy))
> {
> > +        return EFI_NOT_READY;
> > +      }
> > +      ReleaseSpinLock (mSmmMpSyncData->CpuData[Index].Busy);
> > +    }
> > +  }
> > +  if (CpuCount == 0) {
> > +    return EFI_NOT_STARTED;
> > +  }
> > +
> > +  if (Token == NULL) {
> > +    //
> > +    // Make sure all BUSY should be acquired.
> > +    //
> > +    for (Index = mMaxNumberOfCpus; Index-- > 0;) {
> > +      if (Index != gSmmCpuPrivate-
> > >SmmCoreEntryContext.CurrentlyExecutingCpu && *(mSmmMpSyncData-
> > >CpuData[Index].Present)) {
> > +        AcquireSpinLock (mSmmMpSyncData->CpuData[Index].Busy);
> > +      }
> > +    }
> > +  } else {
> > +    if (!AcquireSpinLockOrFail (Token)) {
> > +      return EFI_INVALID_PARAMETER;
> > +    }
> > +
> > +    //
> > +    // Make sure all BUSY should be acquired.
> > +    //
> > +    for (Index = mMaxNumberOfCpus; Index-- > 0;) {
> > +      if (Index != gSmmCpuPrivate-
> > >SmmCoreEntryContext.CurrentlyExecutingCpu && *(mSmmMpSyncData-
> > >CpuData[Index].Present)) {
> > +        if (!AcquireSpinLockOrFail (mSmmMpSyncData-
> >CpuData[Index].Busy))
> > {
> > +          DEBUG((DEBUG_ERROR, "Can't acquire mSmmMpSyncData-
> > >CpuData[%d].Busy\n", Index));
> > +
> > +          //
> > +          // Release BUSY accquired before.
> > +          //
> > +          for (CpuCount = mMaxNumberOfCpus; CpuCount -- > Index;) {
> > +            if (CpuCount != gSmmCpuPrivate-
> > >SmmCoreEntryContext.CurrentlyExecutingCpu && *(mSmmMpSyncData-
> > >CpuData[CpuCount].Present)) {
> > +              ReleaseSpinLock (mSmmMpSyncData->CpuData[CpuCount].Busy);
> > +            }
> > +          }
> > +
> > +          ReleaseSpinLock (Token);
> > +          return EFI_INVALID_PARAMETER;
> > +        }
> > +      }
> > +    }
> > +  }
> > +
> > +  for (Index = mMaxNumberOfCpus; Index-- > 0;) {
> > +    if (Index != gSmmCpuPrivate-
> > >SmmCoreEntryContext.CurrentlyExecutingCpu && *(mSmmMpSyncData-
> > >CpuData[Index].Present)) {
> > +      mSmmMpSyncData->CpuData[Index].Procedure =
> > (EFI_AP_PROCEDURE2) Procedure;
> > +      mSmmMpSyncData->CpuData[Index].Parameter =
> > ProcedureArguments;
> > +      mSmmMpSyncData->CpuData[Index].Token     = Token;
> > +      if (CPUStatus != NULL) {
> > +        mSmmMpSyncData->CpuData[Index].Status    = CPUStatus + Index;
> > +        if (mSmmMpSyncData->CpuData[Index].Status != NULL) {
> > +          *mSmmMpSyncData->CpuData[Index].Status = EFI_NOT_READY;
> > +        }
> > +      }
> > +    } else {
> > +      //
> > +      // PI spec requirement:
> > +      // For every excluded processor, the array entry must contain a value
> of
> > EFI_NOT_STARTED.
> > +      //
> > +      if (CPUStatus != NULL) {
> > +        *(CPUStatus + Index) = EFI_NOT_STARTED;
> > +      }
> > +    }
> > +  }
> > +
> > +  ReleaseAllAPs ();
> > +
> > +  if (Token == NULL) {
> > +    //
> > +    // Make sure all APs have completed their tasks.
> > +    //
> > +    IsTaskFinishInAllAPs (TRUE);
> > +  }
> > +
> > +  return EFI_SUCCESS;
> > +}
> > +
> > +/**
> > +  Wrapper for Procedures.
> > +
> > +  @param[in]  Buffer              Pointer to PROCEDURE_WRAPPER buffer.
> > +
> > +**/
> > +EFI_STATUS
> > +EFIAPI
> > +ProcedureWrapper (
> > +  IN OUT VOID *Buffer
> > +  )
> > +{
> > +  PROCEDURE_WRAPPER *Wrapper;
> > +
> > +  Wrapper = Buffer;
> > +  Wrapper->Procedure (Wrapper->ProcedureArgument);
> > +
> >    return EFI_SUCCESS;
> >  }
> >
> > @@ -995,7 +1395,24 @@ SmmBlockingStartupThisAp (
> >    IN OUT  VOID                      *ProcArguments OPTIONAL
> >    )
> >  {
> > -  return InternalSmmStartupThisAp(Procedure, CpuIndex, ProcArguments,
> > TRUE);
> > +  PROCEDURE_WRAPPER *Wrapper;
> > +  EFI_STATUS        Status;
> > +
> > +  Wrapper = AllocatePool (sizeof(PROCEDURE_WRAPPER));
> > +  if (Wrapper == NULL) {
> > +    return EFI_OUT_OF_RESOURCES;
> > +  }
> > +  Wrapper->Procedure = Procedure;
> > +  Wrapper->ProcedureArgument = ProcArguments;
> > +
> > +  //
> > +  // Use wrapper function to convert EFI_AP_PROCEDURE to
> > EFI_AP_PROCEDURE2.
> > +  //
> > +  Status = InternalSmmStartupThisAp(ProcedureWrapper, CpuIndex,
> > Wrapper, NULL, 0, NULL);
> > +
> > +  FreePool (Wrapper);
> > +
> > +  return Status;
> >  }
> >
> >  /**
> > @@ -1020,7 +1437,37 @@ SmmStartupThisAp (
> >    IN OUT  VOID                      *ProcArguments OPTIONAL
> >    )
> >  {
> > -  return InternalSmmStartupThisAp(Procedure, CpuIndex, ProcArguments,
> > FeaturePcdGet (PcdCpuSmmBlockStartupThisAp));
> > +  SPIN_LOCK               *CpuToken;
> > +  PROCEDURE_WRAPPER       *Wrapper;
> > +  EFI_STATUS              Status;
> > +
> > +  Wrapper = AllocatePool (sizeof(PROCEDURE_WRAPPER));
> > +  if (Wrapper == NULL) {
> > +    return EFI_OUT_OF_RESOURCES;
> > +  }
> > +  Wrapper->Procedure = Procedure;
> > +  Wrapper->ProcedureArgument = ProcArguments;
> > +
> > +  if (FeaturePcdGet (PcdCpuSmmBlockStartupThisAp)) {
> > +    CpuToken = NULL;
> > +  } else {
> > +    CpuToken = mApTokenLock[CpuIndex];
> > +  }
> > +
> > +  //
> > +  // Use wrapper function to convert EFI_AP_PROCEDURE to
> > EFI_AP_PROCEDURE2.
> > +  //
> > +  Status = InternalSmmStartupThisAp(ProcedureWrapper, CpuIndex,
> > Wrapper, CpuToken, 0, NULL);
> > +
> > +  //
> > +  // Free wrapper buffer for block mode.
> > +  // Non-block mode frees buffer in ApHandler function.
> > +  //
> > +  if (CpuToken == NULL) {
> > +    FreePool (Wrapper);
> > +  }
> > +
> > +  return Status;
> >  }
> >
> >  /**
> > @@ -1112,6 +1559,13 @@ SmiRendezvous (
> >    Cr2 = 0;
> >    SaveCr2 (&Cr2);
> >
> > +  //
> > +  // Call the user register Startup function first.
> > +  //
> > +  if (mSmmMpSyncData->StartupProcedure != NULL) {
> > +    mSmmMpSyncData->StartupProcedure (mSmmMpSyncData-
> > >StartupProcArgs);
> > +  }
> > +
> >    //
> >    // Perform CPU specific entry hooks
> >    //
> > @@ -1273,13 +1727,17 @@ InitializeSmmCpuSemaphores (
> >    UINTN                      Pages;
> >    UINTN                      *SemaphoreBlock;
> >    UINTN                      SemaphoreAddr;
> > +  UINTN                      ApTokenSize;
> > +  UINTN                      CpuIndex;
> >
> >    SemaphoreSize   = GetSpinLockProperties ();
> >    ProcessorCount = gSmmCpuPrivate-
> > >SmmCoreEntryContext.NumberOfCpus;
> >    GlobalSemaphoresSize = (sizeof (SMM_CPU_SEMAPHORE_GLOBAL) /
> > sizeof (VOID *)) * SemaphoreSize;
> >    CpuSemaphoresSize    = (sizeof (SMM_CPU_SEMAPHORE_CPU) / sizeof
> > (VOID *)) * ProcessorCount * SemaphoreSize;
> > -  TotalSize = GlobalSemaphoresSize + CpuSemaphoresSize;
> > +  ApTokenSize = SemaphoreSize * ProcessorCount;
> > +  TotalSize = GlobalSemaphoresSize + CpuSemaphoresSize + ApTokenSize;
> >    DEBUG((EFI_D_INFO, "One Semaphore Size    = 0x%x\n",
> SemaphoreSize));
> > +  DEBUG((EFI_D_INFO, "Token Spin Lock Size = 0x%x\n", ApTokenSize));
> >    DEBUG((EFI_D_INFO, "Total Semaphores Size = 0x%x\n", TotalSize));
> >    Pages = EFI_SIZE_TO_PAGES (TotalSize);
> >    SemaphoreBlock = AllocatePages (Pages);
> > @@ -1305,10 +1763,19 @@ InitializeSmmCpuSemaphores (
> >    mSmmCpuSemaphores.SemaphoreCpu.Run     = (UINT32
> > *)SemaphoreAddr;
> >    SemaphoreAddr += ProcessorCount * SemaphoreSize;
> >    mSmmCpuSemaphores.SemaphoreCpu.Present = (BOOLEAN
> > *)SemaphoreAddr;
> > +  SemaphoreAddr += ProcessorCount * SemaphoreSize;
> > +  mSmmCpuSemaphores.SemaphoreCpu.Token = (SPIN_LOCK
> > *)SemaphoreAddr;
> >
> >    mPFLock                       = mSmmCpuSemaphores.SemaphoreGlobal.PFLock;
> >    mConfigSmmCodeAccessCheckLock =
> > mSmmCpuSemaphores.SemaphoreGlobal.CodeAccessCheckLock;
> >
> > +  mApTokenLock = AllocatePool (sizeof (SPIN_LOCK *) * gSmmCpuPrivate-
> > >SmmCoreEntryContext.NumberOfCpus);
> > +  ASSERT (mApTokenLock != NULL);
> > +  for (CpuIndex = 0; CpuIndex < gSmmCpuPrivate-
> > >SmmCoreEntryContext.NumberOfCpus; CpuIndex ++) {
> > +    mApTokenLock[CpuIndex] = (SPIN_LOCK
> > *)((UINTN)mSmmCpuSemaphores.SemaphoreCpu.Token +
> SemaphoreSize
> > * CpuIndex);
> > +    InitializeSpinLock (mApTokenLock[CpuIndex]);
> > +  }
> > +
> >    mSemaphoreSize = SemaphoreSize;
> >  }
> >
> > @@ -1469,3 +1936,40 @@ RegisterSmmEntry (
> >    gSmmCpuPrivate->SmmCoreEntry = SmmEntryPoint;
> >    return EFI_SUCCESS;
> >  }
> > +
> > +/**
> > +
> > +  Register the SMM Foundation entry point.
> > +
> > +  @param[in]      Procedure            A pointer to the code stream to be run on
> > the designated target AP
> > +                                       of the system. Type EFI_AP_PROCEDURE is defined
> > below in Volume 2
> > +                                       with the related definitions of
> > +                                       EFI_MP_SERVICES_PROTOCOL.StartupAllAPs.
> > +                                       If caller may pass a value of NULL to deregister any
> > existing
> > +                                       startup procedure.
> > +  @param[in]      ProcedureArguments   Allows the caller to pass a list of
> > parameters to the code that is
> > +                                       run by the AP. It is an optional common mailbox
> > between APs and
> > +                                       the caller to share information
> > +
> > +  @retval EFI_SUCCESS                  The Procedure has been set successfully.
> > +  @retval EFI_INVALID_PARAMETER        The Procedure is NULL but
> > ProcedureArguments not NULL.
> > +
> > +**/
> > +EFI_STATUS
> > +RegisterStartupProcedure (
> > +  IN EFI_AP_PROCEDURE    Procedure,
> > +  IN VOID                *ProcedureArguments OPTIONAL
> > +  )
> > +{
> > +  if (Procedure == NULL && ProcedureArguments != NULL) {
> > +    return EFI_INVALID_PARAMETER;
> > +  }
> > +  if (mSmmMpSyncData == NULL) {
> > +    return EFI_NOT_READY;
> > +  }
> > +
> > +  mSmmMpSyncData->StartupProcedure = Procedure;
> > +  mSmmMpSyncData->StartupProcArgs  = ProcedureArguments;
> > +
> > +  return EFI_SUCCESS;
> > +}
> > diff --git a/UefiCpuPkg/PiSmmCpuDxeSmm/PiSmmCpuDxeSmm.c
> > b/UefiCpuPkg/PiSmmCpuDxeSmm/PiSmmCpuDxeSmm.c
> > index 2f7d777ee7..dd1b3be0f5 100644
> > --- a/UefiCpuPkg/PiSmmCpuDxeSmm/PiSmmCpuDxeSmm.c
> > +++ b/UefiCpuPkg/PiSmmCpuDxeSmm/PiSmmCpuDxeSmm.c
> > @@ -996,6 +996,17 @@ PiCpuSmmEntry (
> >                      );
> >    ASSERT_EFI_ERROR (Status);
> >
> > +  //
> > +  // Install the SMM Mp Protocol into SMM protocol database
> > +  //
> > +  Status = gSmst->SmmInstallProtocolInterface (
> > +                    &mSmmCpuHandle,
> > +                    &gEfiMmMpProtocolGuid,
> > +                    EFI_NATIVE_INTERFACE,
> > +                    &mSmmMp
> > +                    );
> > +  ASSERT_EFI_ERROR (Status);
> > +
> >    //
> >    // Expose address of CPU Hot Plug Data structure if CPU hot plug is
> > supported.
> >    //
> > diff --git a/UefiCpuPkg/PiSmmCpuDxeSmm/PiSmmCpuDxeSmm.h
> > b/UefiCpuPkg/PiSmmCpuDxeSmm/PiSmmCpuDxeSmm.h
> > index 2bb35a424d..5df09687e1 100644
> > --- a/UefiCpuPkg/PiSmmCpuDxeSmm/PiSmmCpuDxeSmm.h
> > +++ b/UefiCpuPkg/PiSmmCpuDxeSmm/PiSmmCpuDxeSmm.h
> > @@ -20,6 +20,7 @@ SPDX-License-Identifier: BSD-2-Clause-Patent
> >  #include <Protocol/SmmReadyToLock.h>
> >  #include <Protocol/SmmCpuService.h>
> >  #include <Protocol/SmmMemoryAttribute.h>
> > +#include <Protocol/MmMp.h>
> >
> >  #include <Guid/AcpiS3Context.h>
> >  #include <Guid/MemoryAttributesTable.h>
> > @@ -221,11 +222,20 @@ typedef struct {
> >    EFI_SMM_CONFIGURATION_PROTOCOL  SmmConfiguration;
> >  } SMM_CPU_PRIVATE_DATA;
> >
> > +//
> > +// Wrapper used to convert EFI_AP_PROCEDURE2 and
> EFI_AP_PROCEDURE.
> > +//
> > +typedef struct {
> > +  EFI_AP_PROCEDURE  Procedure;
> > +  VOID              *ProcedureArgument;
> > +} PROCEDURE_WRAPPER;
> > +
> >  extern SMM_CPU_PRIVATE_DATA  *gSmmCpuPrivate;
> >  extern CPU_HOT_PLUG_DATA      mCpuHotPlugData;
> >  extern UINTN                  mMaxNumberOfCpus;
> >  extern UINTN                  mNumberOfCpus;
> >  extern EFI_SMM_CPU_PROTOCOL   mSmmCpu;
> > +extern EFI_MM_MP_PROTOCOL     mSmmMp;
> >
> >  ///
> >  /// The mode of the CPU at the time an SMI occurs
> > @@ -363,10 +373,12 @@ SmmRelocationSemaphoreComplete (
> >  ///
> >  typedef struct {
> >    SPIN_LOCK                         *Busy;
> > -  volatile EFI_AP_PROCEDURE         Procedure;
> > +  volatile EFI_AP_PROCEDURE2        Procedure;
> >    volatile VOID                     *Parameter;
> >    volatile UINT32                   *Run;
> >    volatile BOOLEAN                  *Present;
> > +  SPIN_LOCK                         *Token;
> > +  EFI_STATUS                        *Status;
> >  } SMM_CPU_DATA_BLOCK;
> >
> >  typedef enum {
> > @@ -388,6 +400,8 @@ typedef struct {
> >    volatile SMM_CPU_SYNC_MODE    EffectiveSyncMode;
> >    volatile BOOLEAN              SwitchBsp;
> >    volatile BOOLEAN              *CandidateBsp;
> > +  EFI_AP_PROCEDURE              StartupProcedure;
> > +  VOID                          *StartupProcArgs;
> >  } SMM_DISPATCHER_MP_SYNC_DATA;
> >
> >  #define SMM_PSD_OFFSET              0xfb00
> > @@ -410,6 +424,7 @@ typedef struct {
> >    SPIN_LOCK                         *Busy;
> >    volatile UINT32                   *Run;
> >    volatile BOOLEAN                  *Present;
> > +  SPIN_LOCK                         *Token;
> >  } SMM_CPU_SEMAPHORE_CPU;
> >
> >  ///
> > @@ -439,6 +454,7 @@ extern SPIN_LOCK
> > *mConfigSmmCodeAccessCheckLock;
> >  extern EFI_SMRAM_DESCRIPTOR                *mSmmCpuSmramRanges;
> >  extern UINTN                               mSmmCpuSmramRangeCount;
> >  extern UINT8                               mPhysicalAddressBits;
> > +extern SPIN_LOCK                           **mApTokenLock;
> >
> >  //
> >  // Copy of the PcdPteMemoryEncryptionAddressOrMask
> > @@ -1259,4 +1275,146 @@ RestoreCr2 (
> >    IN UINTN  Cr2
> >    );
> >
> > +/**
> > +  Schedule a procedure to run on the specified CPU.
> > +
> > +  @param[in]       Procedure                The address of the procedure to run
> > +  @param[in]       CpuIndex                 Target CPU Index
> > +  @param[in,out]   ProcArguments            The parameter to pass to the
> > procedure
> > +  @param[in,out]   Token                    This is an optional parameter that
> allows
> > the caller to execute the
> > +                                            procedure in a blocking or non-blocking fashion. If it
> is
> > NULL the
> > +                                            call is blocking, and the call will not return until the
> AP
> > has
> > +                                            completed the procedure. If the token is not NULL,
> > the call will
> > +                                            return immediately. The caller can check whether
> the
> > procedure has
> > +                                            completed with CheckOnProcedure or
> > WaitForProcedure.
> > +  @param[in]       TimeoutInMicroseconds    Indicates the time limit in
> > microseconds for the APs to finish
> > +                                            execution of Procedure, either for blocking or non-
> > blocking mode.
> > +                                            Zero means infinity. If the timeout expires before all
> > APs return
> > +                                            from Procedure, then Procedure on the failed APs is
> > terminated. If
> > +                                            the timeout expires in blocking mode, the call
> returns
> > EFI_TIMEOUT.
> > +                                            If the timeout expires in non-blocking mode, the
> > timeout determined
> > +                                            can be through CheckOnProcedure or
> > WaitForProcedure.
> > +                                            Note that timeout support is optional. Whether an
> > implementation
> > +                                            supports this feature can be determined via the
> > Attributes data
> > +                                            member.
> > +  @param[in,out]   CPUStatus                This optional pointer may be used to
> > get the status code returned
> > +                                            by Procedure when it completes execution on the
> > target AP, or with
> > +                                            EFI_TIMEOUT if the Procedure fails to complete
> > within the optional
> > +                                            timeout. The implementation will update this
> variable
> > with
> > +                                            EFI_NOT_READY prior to starting Procedure on the
> > target AP.
> > +
> > +
> > +  @retval EFI_INVALID_PARAMETER    CpuNumber not valid
> > +  @retval EFI_INVALID_PARAMETER    CpuNumber specifying BSP
> > +  @retval EFI_INVALID_PARAMETER    The AP specified by CpuNumber did
> > not enter SMM
> > +  @retval EFI_INVALID_PARAMETER    The AP specified by CpuNumber is
> > busy
> > +  @retval EFI_SUCCESS              The procedure has been successfully
> > scheduled
> > +
> > +**/
> > +EFI_STATUS
> > +InternalSmmStartupThisAp (
> > +  IN      EFI_AP_PROCEDURE2              Procedure,
> > +  IN      UINTN                          CpuIndex,
> > +  IN OUT  VOID                           *ProcArguments OPTIONAL,
> > +  IN      SPIN_LOCK                      *Token,
> > +  IN      UINTN                          TimeoutInMicroseconds,
> > +  IN OUT  EFI_STATUS                     *CpuStatus
> > +  );
> > +
> > +/**
> > +  Checks whether the input token is the current used token.
> > +
> > +  @param[in]  CpuIndex   Cpu Index.
> > +  @param[in]  Token      This parameter describes the token that was
> passed
> > into DispatchProcedure or
> > +                         BroadcastProcedure.
> > +
> > +  @retval TRUE           The input token is the current used token.
> > +  @retval FALSE          The input token is not the current used token.
> > +**/
> > +BOOLEAN
> > +IsCurrentToken (
> > +  IN UINTN               CpuIndex,
> > +  IN SPIN_LOCK           *Token
> > +  );
> > +
> > +/**
> > +  Checks status of specified AP.
> > +
> > +  This function checks whether the specified AP has finished the task
> > assigned
> > +  by StartupThisAP(), and whether timeout expires.
> > +
> > +  @param[in]  Token             This parameter describes the token that was
> > passed into DispatchProcedure or
> > +                                BroadcastProcedure.
> > +
> > +  @retval EFI_SUCCESS           Specified AP has finished task assigned by
> > StartupThisAPs().
> > +  @retval EFI_NOT_READY         Specified AP has not finished task and
> > timeout has not expired.
> > +**/
> > +EFI_STATUS
> > +IsApReady (
> > +  IN SPIN_LOCK  *Token
> > +  );
> > +
> > +/**
> > +  Worker function to execute a caller provided function on all enabled APs.
> > +
> > +  @param[in]     Procedure               A pointer to the function to be run on
> > +                                         enabled APs of the system.
> > +  @param[in]     TimeoutInMicroseconds   Indicates the time limit in
> > microseconds for
> > +                                         APs to return from Procedure, either for
> > +                                         blocking or non-blocking mode.
> > +  @param[in,out] ProcedureArgument       The parameter passed into
> > Procedure for
> > +                                         all APs.
> > +  @param[in,out] Token                   This is an optional parameter that allows
> > the caller to execute the
> > +                                         procedure in a blocking or non-blocking fashion. If it is
> > NULL the
> > +                                         call is blocking, and the call will not return until the AP
> > has
> > +                                         completed the procedure. If the token is not NULL,
> the
> > call will
> > +                                         return immediately. The caller can check whether the
> > procedure has
> > +                                         completed with CheckOnProcedure or
> > WaitForProcedure.
> > +  @param[in,out] CPUStatus               This optional pointer may be used to
> get
> > the status code returned
> > +                                         by Procedure when it completes execution on the
> > target AP, or with
> > +                                         EFI_TIMEOUT if the Procedure fails to complete
> within
> > the optional
> > +                                         timeout. The implementation will update this variable
> > with
> > +                                         EFI_NOT_READY prior to starting Procedure on the
> > target AP.
> > +
> > +
> > +  @retval EFI_SUCCESS             In blocking mode, all APs have finished
> before
> > +                                  the timeout expired.
> > +  @retval EFI_SUCCESS             In non-blocking mode, function has been
> > dispatched
> > +                                  to all enabled APs.
> > +  @retval others                  Failed to Startup all APs.
> > +
> > +**/
> > +EFI_STATUS
> > +InternalSmmStartupAllAPs (
> > +  IN       EFI_AP_PROCEDURE2             Procedure,
> > +  IN       UINTN                         TimeoutInMicroseconds,
> > +  IN OUT   VOID                          *ProcedureArguments OPTIONAL,
> > +  IN OUT   SPIN_LOCK                     *Token,
> > +  IN OUT   EFI_STATUS                    *CPUStatus
> > +  );
> > +
> > +/**
> > +
> > +  Register the SMM Foundation entry point.
> > +
> > +  @param[in]      Procedure            A pointer to the code stream to be run on
> > the designated target AP
> > +                                       of the system. Type EFI_AP_PROCEDURE is defined
> > below in Volume 2
> > +                                       with the related definitions of
> > +                                       EFI_MP_SERVICES_PROTOCOL.StartupAllAPs.
> > +                                       If caller may pass a value of NULL to deregister any
> > existing
> > +                                       startup procedure.
> > +  @param[in,out]  ProcedureArguments   Allows the caller to pass a list of
> > parameters to the code that is
> > +                                       run by the AP. It is an optional common mailbox
> > between APs and
> > +                                       the caller to share information
> > +
> > +  @retval EFI_SUCCESS                  The Procedure has been set successfully.
> > +  @retval EFI_INVALID_PARAMETER        The Procedure is NULL but
> > ProcedureArguments not NULL.
> > +
> > +**/
> > +EFI_STATUS
> > +RegisterStartupProcedure (
> > +  IN EFI_AP_PROCEDURE    Procedure,
> > +  IN VOID                *ProcedureArguments OPTIONAL
> > +  );
> > +
> >  #endif
> > diff --git a/UefiCpuPkg/PiSmmCpuDxeSmm/PiSmmCpuDxeSmm.inf
> > b/UefiCpuPkg/PiSmmCpuDxeSmm/PiSmmCpuDxeSmm.inf
> > index 466c568d49..da0308c47f 100644
> > --- a/UefiCpuPkg/PiSmmCpuDxeSmm/PiSmmCpuDxeSmm.inf
> > +++ b/UefiCpuPkg/PiSmmCpuDxeSmm/PiSmmCpuDxeSmm.inf
> > @@ -40,6 +40,8 @@
> >    SmmProfileInternal.h
> >    SmramSaveState.c
> >    SmmCpuMemoryManagement.c
> > +  SmmMp.h
> > +  SmmMp.c
> >
> >  [Sources.Ia32]
> >    Ia32/Semaphore.c
> > @@ -105,6 +107,7 @@
> >    gEfiSmmReadyToLockProtocolGuid           ## NOTIFY
> >    gEfiSmmCpuServiceProtocolGuid            ## PRODUCES
> >    gEdkiiSmmMemoryAttributeProtocolGuid     ## PRODUCES
> > +  gEfiMmMpProtocolGuid                    ## PRODUCES
> >
> >  [Guids]
> >    gEfiAcpiVariableGuid                     ## SOMETIMES_CONSUMES ## HOB # it is
> > used for S3 boot.
> > diff --git a/UefiCpuPkg/PiSmmCpuDxeSmm/SmmMp.c
> > b/UefiCpuPkg/PiSmmCpuDxeSmm/SmmMp.c
> > new file mode 100644
> > index 0000000000..bfbc78691f
> > --- /dev/null
> > +++ b/UefiCpuPkg/PiSmmCpuDxeSmm/SmmMp.c
> > @@ -0,0 +1,372 @@
> > +/** @file
> > +SMM MP protocol implementation
> > +
> > +Copyright (c) 2019, Intel Corporation. All rights reserved.<BR>
> > +
> > +SPDX-License-Identifier: BSD-2-Clause-Patent
> > +
> > +**/
> > +
> > +#include "PiSmmCpuDxeSmm.h"
> > +#include "SmmMp.h"
> > +
> > +///
> > +/// SMM MP Protocol instance
> > +///
> > +EFI_MM_MP_PROTOCOL  mSmmMp  = {
> > +  EFI_MM_MP_PROTOCOL_REVISION,
> > +  0,
> > +  SmmMpGetNumberOfProcessors,
> > +  SmmMpDispatchProcedure,
> > +  SmmMpBroadcastProcedure,
> > +  SmmMpSetStartupProcedure,
> > +  SmmMpCheckForProcedure,
> > +  SmmMpWaitForProcedure
> > +};
> > +
> > +/**
> > +  Service to retrieves the number of logical processor in the platform.
> > +
> > +  @param[in]  This                The EFI_MM_MP_PROTOCOL instance.
> > +  @param[out] NumberOfProcessors  Pointer to the total number of
> logical
> > processors in the system,
> > +                                  including the BSP and all APs.
> > +
> > +  @retval EFI_SUCCESS             The number of processors was retrieved
> > successfully
> > +  @retval EFI_INVALID_PARAMETER   NumberOfProcessors is NULL
> > +**/
> > +EFI_STATUS
> > +EFIAPI
> > +SmmMpGetNumberOfProcessors (
> > +  IN CONST EFI_MM_MP_PROTOCOL   *This,
> > +  OUT      UINTN                *NumberOfProcessors
> > +  )
> > +{
> > +  if (NumberOfProcessors == NULL) {
> > +    return EFI_INVALID_PARAMETER;
> > +  }
> > +
> > +  *NumberOfProcessors = gSmmCpuPrivate-
> > >SmmCoreEntryContext.NumberOfCpus;
> > +
> > +  return EFI_SUCCESS;
> > +}
> > +
> > +
> > +/**
> > +  This service allows the caller to invoke a procedure one of the application
> > processors (AP). This
> > +  function uses an optional token parameter to support blocking and non-
> > blocking modes. If the token
> > +  is passed into the call, the function will operate in a non-blocking fashion
> > and the caller can
> > +  check for completion with CheckOnProcedure or WaitForProcedure.
> > +
> > +  @param[in]     This                   The EFI_MM_MP_PROTOCOL instance.
> > +  @param[in]     Procedure              A pointer to the procedure to be run on
> > the designated target
> > +                                        AP of the system. Type EFI_AP_PROCEDURE2 is
> defined
> > below in
> > +                                        related definitions.
> > +  @param[in]     CpuNumber              The zero-based index of the processor
> > number of the target
> > +                                        AP, on which the code stream is supposed to run. If
> the
> > number
> > +                                        points to the calling processor then it will not run the
> > +                                        supplied code.
> > +  @param[in]     TimeoutInMicroseconds  Indicates the time limit in
> > microseconds for this AP to
> > +                                        finish execution of Procedure, either for blocking or
> > +                                        non-blocking mode. Zero means infinity. If the
> timeout
> > +                                        expires before this AP returns from Procedure, then
> > Procedure
> > +                                        on the AP is terminated. If the timeout expires in
> > blocking
> > +                                        mode, the call returns EFI_TIMEOUT. If the timeout
> > expires
> > +                                        in non-blocking mode, the timeout determined can be
> > through
> > +                                        CheckOnProcedure or WaitForProcedure.
> > +                                        Note that timeout support is optional. Whether an
> > +                                        implementation supports this feature, can be
> > determined via
> > +                                        the Attributes data member.
> > +  @param[in,out] ProcedureArguments     Allows the caller to pass a list of
> > parameters to the code
> > +                                        that is run by the AP. It is an optional common mailbox
> > +                                        between APs and the caller to share information.
> > +  @param[in,out] Token                  This is parameter is broken into two
> > components:
> > +                                        1.Token->Completion is an optional parameter that
> > allows the
> > +                                        caller to execute the procedure in a blocking or non-
> > blocking
> > +                                        fashion. If it is NULL the call is blocking, and the call will
> > +                                        not return until the AP has completed the procedure.
> If
> > the
> > +                                        token is not NULL, the call will return immediately. The
> > caller
> > +                                        can check whether the procedure has completed with
> > +                                        CheckOnProcedure or WaitForProcedure.
> > +                                        2.Token->Status The implementation updates the
> > address pointed
> > +                                        at by this variable with the status code returned by
> > Procedure
> > +                                        when it completes execution on the target AP, or with
> > EFI_TIMEOUT
> > +                                        if the Procedure fails to complete within the optional
> > timeout.
> > +                                        The implementation will update this variable with
> > EFI_NOT_READY
> > +                                        prior to starting Procedure on the target AP
> > +  @param[in,out] CPUStatus              This optional pointer may be used to
> get
> > the status code returned
> > +                                        by Procedure when it completes execution on the
> > target AP, or with
> > +                                        EFI_TIMEOUT if the Procedure fails to complete within
> > the optional
> > +                                        timeout. The implementation will update this variable
> > with
> > +                                        EFI_NOT_READY prior to starting Procedure on the
> > target AP.
> > +
> > +  @retval EFI_SUCCESS                   In the blocking case, this indicates that
> > Procedure has completed
> > +                                        execution on the target AP.
> > +                                        In the non-blocking case this indicates that the
> > procedure has
> > +                                        been successfully scheduled for execution on the
> target
> > AP.
> > +  @retval EFI_INVALID_PARAMETER         The input arguments are out of
> > range. Either the target AP is the
> > +                                        caller of the function, or the Procedure or Token is
> NULL
> > +  @retval EFI_NOT_READY                 If the target AP is busy executing
> another
> > procedure
> > +  @retval EFI_ALREADY_STARTED           Token is already in use for another
> > procedure
> > +  @retval EFI_TIMEOUT                   In blocking mode, the timeout expired
> > before the specified AP
> > +                                        has finished
> > +  @retval EFI_OUT_OF_RESOURCES          Could not allocate a required
> > resource.
> > +
> > +**/
> > +EFI_STATUS
> > +EFIAPI
> > +SmmMpDispatchProcedure (
> > +  IN CONST EFI_MM_MP_PROTOCOL            *This,
> > +  IN       EFI_AP_PROCEDURE2             Procedure,
> > +  IN       UINTN                         CpuNumber,
> > +  IN       UINTN                         TimeoutInMicroseconds,
> > +  IN OUT   VOID                          *ProcedureArguments OPTIONAL,
> > +  IN OUT   MM_COMPLETION                 *Token,
> > +  IN OUT   EFI_STATUS                    *CPUStatus
> > +  )
> > +{
> > +  SPIN_LOCK      *CpuToken;
> > +
> > +  if (Token != NULL) {
> > +    CpuToken = AllocatePool (sizeof (SPIN_LOCK));
> > +    ASSERT (CpuToken != NULL);
> > +    if (CpuToken == NULL) {
> > +      return EFI_OUT_OF_RESOURCES;
> > +    }
> > +    InitializeSpinLock ((SPIN_LOCK *)(CpuToken));
> > +
> > +    *Token = (MM_COMPLETION)CpuToken;
> > +  }
> > +
> > +  return InternalSmmStartupThisAp (
> > +    Procedure,
> > +    CpuNumber,
> > +    ProcedureArguments,
> > +    Token != NULL ? CpuToken : NULL,
> > +    TimeoutInMicroseconds,
> > +    CPUStatus
> > +    );
> > +}
> > +
> > +/**
> > +  This service allows the caller to invoke a procedure on all running
> > application processors (AP)
> > +  except the caller. This function uses an optional token parameter to
> > support blocking and
> > +  nonblocking modes. If the token is passed into the call, the function will
> > operate in a non-blocking
> > +  fashion and the caller can check for completion with CheckOnProcedure
> or
> > WaitForProcedure.
> > +
> > +  It is not necessary for the implementation to run the procedure on every
> > processor on the platform.
> > +  Processors that are powered down in such a way that they cannot
> respond
> > to interrupts, may be
> > +  excluded from the broadcast.
> > +
> > +
> > +  @param[in]     This                   The EFI_MM_MP_PROTOCOL instance.
> > +  @param[in]     Procedure              A pointer to the code stream to be run
> on
> > the APs that have
> > +                                        entered MM. Type EFI_AP_PROCEDURE is defined
> > below in related
> > +                                        definitions.
> > +  @param[in]     TimeoutInMicroseconds  Indicates the time limit in
> > microseconds for the APs to finish
> > +                                        execution of Procedure, either for blocking or non-
> > blocking mode.
> > +                                        Zero means infinity. If the timeout expires before all
> > APs return
> > +                                        from Procedure, then Procedure on the failed APs is
> > terminated. If
> > +                                        the timeout expires in blocking mode, the call returns
> > EFI_TIMEOUT.
> > +                                        If the timeout expires in non-blocking mode, the
> > timeout determined
> > +                                        can be through CheckOnProcedure or
> > WaitForProcedure.
> > +                                        Note that timeout support is optional. Whether an
> > implementation
> > +                                        supports this feature can be determined via the
> > Attributes data
> > +                                        member.
> > +  @param[in,out] ProcedureArguments     Allows the caller to pass a list of
> > parameters to the code
> > +                                        that is run by the AP. It is an optional common mailbox
> > +                                        between APs and the caller to share information.
> > +  @param[in,out] Token                  This is parameter is broken into two
> > components:
> > +                                        1.Token->Completion is an optional parameter that
> > allows the
> > +                                        caller to execute the procedure in a blocking or non-
> > blocking
> > +                                        fashion. If it is NULL the call is blocking, and the call will
> > +                                        not return until the AP has completed the procedure.
> If
> > the
> > +                                        token is not NULL, the call will return immediately. The
> > caller
> > +                                        can check whether the procedure has completed with
> > +                                        CheckOnProcedure or WaitForProcedure.
> > +                                        2.Token->Status The implementation updates the
> > address pointed
> > +                                        at by this variable with the status code returned by
> > Procedure
> > +                                        when it completes execution on the target AP, or with
> > EFI_TIMEOUT
> > +                                        if the Procedure fails to complete within the optional
> > timeout.
> > +                                        The implementation will update this variable with
> > EFI_NOT_READY
> > +                                        prior to starting Procedure on the target AP
> > +  @param[in,out] CPUStatus              This optional pointer may be used to
> get
> > the individual status
> > +                                        returned by every AP that participated in the
> broadcast.
> > This
> > +                                        parameter if used provides the base address of an
> array
> > to hold
> > +                                        the EFI_STATUS value of each AP in the system. The
> size
> > of the
> > +                                        array can be ascertained by the
> > GetNumberOfProcessors function.
> > +                                        As mentioned above, the broadcast may not include
> > every processor
> > +                                        in the system. Some implementations may exclude
> > processors that
> > +                                        have been powered down in such a way that they are
> > not responsive
> > +                                        to interrupts. Additionally the broadcast excludes the
> > processor
> > +                                        which is making the BroadcastProcedure call. For
> every
> > excluded
> > +                                        processor, the array entry must contain a value of
> > EFI_NOT_STARTED
> > +
> > +  @retval EFI_SUCCESS                   In the blocking case, this indicates that
> > Procedure has completed
> > +                                        execution on the APs.
> > +                                        In the non-blocking case this indicates that the
> > procedure has
> > +                                        been successfully scheduled for execution on the APs.
> > +  @retval EFI_INVALID_PARAMETER         The Procedure or Token is NULL
> > +  @retval EFI_NOT_READY                 If the target AP is busy executing
> another
> > procedure
> > +  @retval EFI_ALREADY_STARTED           Token is already in use for another
> > procedure
> > +  @retval EFI_TIMEOUT                   In blocking mode, the timeout expired
> > before the specified AP
> > +                                        has finished.
> > +  @retval EFI_OUT_OF_RESOURCES          Could not allocate a required
> > resource.
> > +
> > +**/
> > +EFI_STATUS
> > +EFIAPI
> > +SmmMpBroadcastProcedure (
> > +  IN CONST EFI_MM_MP_PROTOCOL            *This,
> > +  IN       EFI_AP_PROCEDURE2             Procedure,
> > +  IN       UINTN                         TimeoutInMicroseconds,
> > +  IN OUT   VOID                          *ProcedureArguments OPTIONAL,
> > +  IN OUT   MM_COMPLETION                 *Token,
> > +  IN OUT   EFI_STATUS                    *CPUStatus
> > +  )
> > +{
> > +  SPIN_LOCK        *CpuToken;
> > +
> > +  if (Token != NULL) {
> > +    CpuToken = AllocatePool (sizeof (SPIN_LOCK));
> > +    ASSERT (CpuToken != NULL);
> > +    if (CpuToken == NULL) {
> > +      return EFI_OUT_OF_RESOURCES;
> > +    }
> > +    InitializeSpinLock ((SPIN_LOCK *)(CpuToken));
> > +
> > +    *Token = (MM_COMPLETION)CpuToken;
> > +  }
> > +
> > +  return InternalSmmStartupAllAPs(
> > +    Procedure,
> > +    TimeoutInMicroseconds,
> > +    ProcedureArguments,
> > +    Token != NULL ? CpuToken : NULL,
> > +    CPUStatus
> > +    );
> > +}
> > +
> > +
> > +/**
> > +  This service allows the caller to set a startup procedure that will be
> > executed when an AP powers
> > +  up from a state where core configuration and context is lost. The
> > procedure is execution has the
> > +  following properties:
> > +  1. The procedure executes before the processor is handed over to the
> > operating system.
> > +  2. All processors execute the same startup procedure.
> > +  3. The procedure may run in parallel with other procedures invoked
> > through the functions in this
> > +  protocol, or with processors that are executing an MM handler or
> running
> > in the operating system.
> > +
> > +
> > +  @param[in]      This                 The EFI_MM_MP_PROTOCOL instance.
> > +  @param[in]      Procedure            A pointer to the code stream to be run on
> > the designated target AP
> > +                                       of the system. Type EFI_AP_PROCEDURE is defined
> > below in Volume 2
> > +                                       with the related definitions of
> > +                                       EFI_MP_SERVICES_PROTOCOL.StartupAllAPs.
> > +                                       If caller may pass a value of NULL to deregister any
> > existing
> > +                                       startup procedure.
> > +  @param[in,out]  ProcedureArguments   Allows the caller to pass a list of
> > parameters to the code that is
> > +                                       run by the AP. It is an optional common mailbox
> > between APs and
> > +                                       the caller to share information
> > +
> > +  @retval EFI_SUCCESS                  The Procedure has been set successfully.
> > +  @retval EFI_INVALID_PARAMETER        The Procedure is NULL but
> > ProcedureArguments not NULL.
> > +
> > +**/
> > +EFI_STATUS
> > +EFIAPI
> > +SmmMpSetStartupProcedure (
> > +  IN CONST EFI_MM_MP_PROTOCOL  *This,
> > +  IN       EFI_AP_PROCEDURE    Procedure,
> > +  IN OUT   VOID                *ProcedureArguments OPTIONAL
> > +  )
> > +{
> > +  return RegisterStartupProcedure (Procedure, ProcedureArguments);
> > +}
> > +
> > +/**
> > +  When non-blocking execution of a procedure on an AP is invoked with
> > DispatchProcedure,
> > +  via the use of a token, this function can be used to check for completion
> of
> > the procedure on the AP.
> > +  The function takes the token that was passed into the
> DispatchProcedure
> > call. If the procedure
> > +  is complete, and therefore it is now possible to run another procedure
> on
> > the same AP, this function
> > +  returns EFI_SUCESS. In this case the status returned by the procedure
> that
> > executed on the AP is
> > +  returned in the token's Status field. If the procedure has not yet
> > completed, then this function
> > +  returns EFI_NOT_READY.
> > +
> > +  When a non-blocking execution of a procedure is invoked with
> > BroadcastProcedure, via the
> > +  use of a token, this function can be used to check for completion of the
> > procedure on all the
> > +  broadcast APs. The function takes the token that was passed into the
> > BroadcastProcedure
> > +  call. If the procedure is complete on all broadcast APs this function
> returns
> > EFI_SUCESS. In this
> > +  case the Status field in the token passed into the function reflects the
> > overall result of the
> > +  invocation, which may be EFI_SUCCESS, if all executions succeeded, or
> the
> > first observed failure.
> > +  If the procedure has not yet completed on the broadcast APs, the
> function
> > returns
> > +  EFI_NOT_READY.
> > +
> > +  @param[in]      This                 The EFI_MM_MP_PROTOCOL instance.
> > +  @param[in]      Token                This parameter describes the token that
> was
> > passed into
> > +                                       DispatchProcedure or BroadcastProcedure.
> > +
> > +  @retval EFI_SUCCESS                  Procedure has completed.
> > +  @retval EFI_NOT_READY                The Procedure has not completed.
> > +  @retval EFI_INVALID_PARAMETER        Token or Token->Completion is
> > NULL
> > +  @retval EFI_NOT_FOUND                Token is not currently in use for a non-
> > blocking call
> > +
> > +**/
> > +EFI_STATUS
> > +EFIAPI
> > +SmmMpCheckForProcedure (
> > +  IN CONST EFI_MM_MP_PROTOCOL            *This,
> > +  IN       MM_COMPLETION                 Token
> > +  )
> > +{
> > +  if (Token == NULL) {
> > +    return EFI_INVALID_PARAMETER;
> > +  }
> > +
> > +  if (!IsCurrentToken ((UINTN)-1, (SPIN_LOCK *)Token)) {
> > +    return EFI_NOT_FOUND;
> > +  }
> > +
> > +  return IsApReady ((SPIN_LOCK *)Token);
> > +}
> > +
> > +/**
> > +  When a non-blocking execution of a procedure on an AP is invoked via
> > DispatchProcedure,
> > +  this function will block the caller until the remote procedure has
> completed
> > on the designated AP.
> > +  The non-blocking procedure invocation is identified by the Token
> > parameter, which must match the
> > +  token that used when DispatchProcedure was called. Upon completion
> the
> > status returned by
> > +  the procedure that executed on the AP is used to update the token's
> > Status field.
> > +
> > +  When a non-blocking execution of a procedure on an AP is invoked via
> > BroadcastProcedure
> > +  this function will block the caller until the remote procedure has
> completed
> > on all of the APs that
> > +  entered MM. The non-blocking procedure invocation is identified by the
> > Token parameter, which
> > +  must match the token that used when BroadcastProcedure was called.
> > Upon completion the
> > +  overall status returned by the procedures that executed on the
> broadcast
> > AP is used to update the
> > +  token's Status field. The overall status may be EFI_SUCCESS, if all
> > executions succeeded, or the
> > +  first observed failure.
> > +
> > +
> > +  @param[in]      This                 The EFI_MM_MP_PROTOCOL instance.
> > +  @param[in]      Token                This parameter describes the token that
> was
> > passed into
> > +                                       DispatchProcedure or BroadcastProcedure.
> > +
> > +  @retval EFI_SUCCESS                  Procedure has completed.
> > +  @retval EFI_INVALID_PARAMETER        Token or Token->Completion is
> > NULL
> > +  @retval EFI_NOT_FOUND                Token is not currently in use for a non-
> > blocking call
> > +
> > +**/
> > +EFI_STATUS
> > +EFIAPI
> > +SmmMpWaitForProcedure (
> > +  IN CONST EFI_MM_MP_PROTOCOL            *This,
> > +  IN       MM_COMPLETION                 Token
> > +  )
> > +{
> > +  EFI_STATUS    Status;
> > +
> > +  do {
> > +    Status = SmmMpCheckForProcedure (This, Token);
> > +  } while (Status == EFI_NOT_READY);
> > +
> > +  return Status;
> > +}
> > +
> > diff --git a/UefiCpuPkg/PiSmmCpuDxeSmm/SmmMp.h
> > b/UefiCpuPkg/PiSmmCpuDxeSmm/SmmMp.h
> > new file mode 100644
> > index 0000000000..e0d823a4b1
> > --- /dev/null
> > +++ b/UefiCpuPkg/PiSmmCpuDxeSmm/SmmMp.h
> > @@ -0,0 +1,286 @@
> > +/** @file
> > +Include file for SMM MP protocol implementation.
> > +
> > +Copyright (c) 2019, Intel Corporation. All rights reserved.<BR>
> > +
> > +SPDX-License-Identifier: BSD-2-Clause-Patent
> > +
> > +**/
> > +
> > +#ifndef _SMM_MP_PROTOCOL_H_
> > +#define _SMM_MP_PROTOCOL_H_
> > +
> > +//
> > +// SMM MP Protocol function prototypes.
> > +//
> > +
> > +/**
> > +  Service to retrieves the number of logical processor in the platform.
> > +
> > +  @param[in]  This                The EFI_MM_MP_PROTOCOL instance.
> > +  @param[out] NumberOfProcessors  Pointer to the total number of
> logical
> > processors in the system,
> > +                                  including the BSP and all APs.
> > +
> > +  @retval EFI_SUCCESS             The number of processors was retrieved
> > successfully
> > +  @retval EFI_INVALID_PARAMETER   NumberOfProcessors is NULL
> > +**/
> > +
> > +EFI_STATUS
> > +EFIAPI
> > +SmmMpGetNumberOfProcessors (
> > +  IN CONST EFI_MM_MP_PROTOCOL   *This,
> > +  OUT      UINTN                *NumberOfProcessors
> > +  );
> > +
> > +
> > +/**
> > +  This service allows the caller to invoke a procedure one of the application
> > processors (AP). This
> > +  function uses an optional token parameter to support blocking and non-
> > blocking modes. If the token
> > +  is passed into the call, the function will operate in a non-blocking fashion
> > and the caller can
> > +  check for completion with CheckOnProcedure or WaitForProcedure.
> > +
> > +  @param[in]     This                   The EFI_MM_MP_PROTOCOL instance.
> > +  @param[in]     Procedure              A pointer to the procedure to be run on
> > the designated target
> > +                                        AP of the system. Type EFI_AP_PROCEDURE2 is
> defined
> > below in
> > +                                        related definitions.
> > +  @param[in]     CpuNumber              The zero-based index of the processor
> > number of the target
> > +                                        AP, on which the code stream is supposed to run. If
> the
> > number
> > +                                        points to the calling processor then it will not run the
> > +                                        supplied code.
> > +  @param[in]     TimeoutInMicroseconds  Indicates the time limit in
> > microseconds for this AP to
> > +                                        finish execution of Procedure, either for blocking or
> > +                                        non-blocking mode. Zero means infinity. If the
> timeout
> > +                                        expires before this AP returns from Procedure, then
> > Procedure
> > +                                        on the AP is terminated. If the timeout expires in
> > blocking
> > +                                        mode, the call returns EFI_TIMEOUT. If the timeout
> > expires
> > +                                        in non-blocking mode, the timeout determined can be
> > through
> > +                                        CheckOnProcedure or WaitForProcedure.
> > +                                        Note that timeout support is optional. Whether an
> > +                                        implementation supports this feature, can be
> > determined via
> > +                                        the Attributes data member.
> > +  @param[in,out] ProcedureArguments     Allows the caller to pass a list of
> > parameters to the code
> > +                                        that is run by the AP. It is an optional common mailbox
> > +                                        between APs and the caller to share information.
> > +  @param[in,out] Token                  This is parameter is broken into two
> > components:
> > +                                        1.Token->Completion is an optional parameter that
> > allows the
> > +                                        caller to execute the procedure in a blocking or non-
> > blocking
> > +                                        fashion. If it is NULL the call is blocking, and the call will
> > +                                        not return until the AP has completed the procedure.
> If
> > the
> > +                                        token is not NULL, the call will return immediately. The
> > caller
> > +                                        can check whether the procedure has completed with
> > +                                        CheckOnProcedure or WaitForProcedure.
> > +                                        2.Token->Status The implementation updates the
> > address pointed
> > +                                        at by this variable with the status code returned by
> > Procedure
> > +                                        when it completes execution on the target AP, or with
> > EFI_TIMEOUT
> > +                                        if the Procedure fails to complete within the optional
> > timeout.
> > +                                        The implementation will update this variable with
> > EFI_NOT_READY
> > +                                        prior to starting Procedure on the target AP
> > +  @param[in,out] CPUStatus              This optional pointer may be used to
> get
> > the status code returned
> > +                                        by Procedure when it completes execution on the
> > target AP, or with
> > +                                        EFI_TIMEOUT if the Procedure fails to complete within
> > the optional
> > +                                        timeout. The implementation will update this variable
> > with
> > +                                        EFI_NOT_READY prior to starting Procedure on the
> > target AP.
> > +
> > +  @retval EFI_SUCCESS                   In the blocking case, this indicates that
> > Procedure has completed
> > +                                        execution on the target AP.
> > +                                        In the non-blocking case this indicates that the
> > procedure has
> > +                                        been successfully scheduled for execution on the
> target
> > AP.
> > +  @retval EFI_INVALID_PARAMETER         The input arguments are out of
> > range. Either the target AP is the
> > +                                        caller of the function, or the Procedure or Token is
> NULL
> > +  @retval EFI_NOT_READY                 If the target AP is busy executing
> another
> > procedure
> > +  @retval EFI_ALREADY_STARTED           Token is already in use for another
> > procedure
> > +  @retval EFI_TIMEOUT                   In blocking mode, the timeout expired
> > before the specified AP
> > +                                        has finished
> > +  @retval EFI_OUT_OF_RESOURCES          Could not allocate a required
> > resource.
> > +
> > +**/
> > +EFI_STATUS
> > +EFIAPI
> > +SmmMpDispatchProcedure (
> > +  IN CONST EFI_MM_MP_PROTOCOL             *This,
> > +  IN       EFI_AP_PROCEDURE2              Procedure,
> > +  IN       UINTN                          CpuNumber,
> > +  IN       UINTN                          TimeoutInMicroseconds,
> > +  IN OUT   VOID                           *ProcedureArguments OPTIONAL,
> > +  IN OUT   MM_COMPLETION                  *Token,
> > +  IN OUT   EFI_STATUS                     *CPUStatus
> > +  );
> > +
> > +/**
> > +  This service allows the caller to invoke a procedure on all running
> > application processors (AP)
> > +  except the caller. This function uses an optional token parameter to
> > support blocking and
> > +  nonblocking modes. If the token is passed into the call, the function will
> > operate in a non-blocking
> > +  fashion and the caller can check for completion with CheckOnProcedure
> or
> > WaitForProcedure.
> > +
> > +  It is not necessary for the implementation to run the procedure on every
> > processor on the platform.
> > +  Processors that are powered down in such a way that they cannot
> respond
> > to interrupts, may be
> > +  excluded from the broadcast.
> > +
> > +
> > +  @param[in]     This                   The EFI_MM_MP_PROTOCOL instance.
> > +  @param[in]     Procedure              A pointer to the code stream to be run
> on
> > the APs that have
> > +                                        entered MM. Type EFI_AP_PROCEDURE is defined
> > below in related
> > +                                        definitions.
> > +  @param[in]     TimeoutInMicroseconds  Indicates the time limit in
> > microseconds for the APs to finish
> > +                                        execution of Procedure, either for blocking or non-
> > blocking mode.
> > +                                        Zero means infinity. If the timeout expires before all
> > APs return
> > +                                        from Procedure, then Procedure on the failed APs is
> > terminated. If
> > +                                        the timeout expires in blocking mode, the call returns
> > EFI_TIMEOUT.
> > +                                        If the timeout expires in non-blocking mode, the
> > timeout determined
> > +                                        can be through CheckOnProcedure or
> > WaitForProcedure.
> > +                                        Note that timeout support is optional. Whether an
> > implementation
> > +                                        supports this feature can be determined via the
> > Attributes data
> > +                                        member.
> > +  @param[in,out] ProcedureArguments     Allows the caller to pass a list of
> > parameters to the code
> > +                                        that is run by the AP. It is an optional common mailbox
> > +                                        between APs and the caller to share information.
> > +  @param[in,out] Token                  This is parameter is broken into two
> > components:
> > +                                        1.Token->Completion is an optional parameter that
> > allows the
> > +                                        caller to execute the procedure in a blocking or non-
> > blocking
> > +                                        fashion. If it is NULL the call is blocking, and the call will
> > +                                        not return until the AP has completed the procedure.
> If
> > the
> > +                                        token is not NULL, the call will return immediately. The
> > caller
> > +                                        can check whether the procedure has completed with
> > +                                        CheckOnProcedure or WaitForProcedure.
> > +                                        2.Token->Status The implementation updates the
> > address pointed
> > +                                        at by this variable with the status code returned by
> > Procedure
> > +                                        when it completes execution on the target AP, or with
> > EFI_TIMEOUT
> > +                                        if the Procedure fails to complete within the optional
> > timeout.
> > +                                        The implementation will update this variable with
> > EFI_NOT_READY
> > +                                        prior to starting Procedure on the target AP
> > +  @param[in,out] CPUStatus              This optional pointer may be used to
> get
> > the individual status
> > +                                        returned by every AP that participated in the
> broadcast.
> > This
> > +                                        parameter if used provides the base address of an
> array
> > to hold
> > +                                        the EFI_STATUS value of each AP in the system. The
> size
> > of the
> > +                                        array can be ascertained by the
> > GetNumberOfProcessors function.
> > +                                        As mentioned above, the broadcast may not include
> > every processor
> > +                                        in the system. Some implementations may exclude
> > processors that
> > +                                        have been powered down in such a way that they are
> > not responsive
> > +                                        to interrupts. Additionally the broadcast excludes the
> > processor
> > +                                        which is making the BroadcastProcedure call. For
> every
> > excluded
> > +                                        processor, the array entry must contain a value of
> > EFI_NOT_STARTED
> > +
> > +  @retval EFI_SUCCESS                   In the blocking case, this indicates that
> > Procedure has completed
> > +                                        execution on the APs.
> > +                                        In the non-blocking case this indicates that the
> > procedure has
> > +                                        been successfully scheduled for execution on the APs.
> > +  @retval EFI_INVALID_PARAMETER         The Procedure or Token is NULL
> > +  @retval EFI_NOT_READY                 If the target AP is busy executing
> another
> > procedure
> > +  @retval EFI_ALREADY_STARTED           Token is already in use for another
> > procedure
> > +  @retval EFI_TIMEOUT                   In blocking mode, the timeout expired
> > before the specified AP
> > +                                        has finished
> > +  @retval EFI_OUT_OF_RESOURCES          Could not allocate a required
> > resource.
> > +
> > +**/
> > +EFI_STATUS
> > +EFIAPI
> > +SmmMpBroadcastProcedure (
> > +  IN CONST EFI_MM_MP_PROTOCOL             *This,
> > +  IN       EFI_AP_PROCEDURE2              Procedure,
> > +  IN       UINTN                          TimeoutInMicroseconds,
> > +  IN OUT   VOID                           *ProcedureArguments OPTIONAL,
> > +  IN OUT   MM_COMPLETION                  *Token,
> > +  IN OUT   EFI_STATUS                     *CPUStatus
> > +  );
> > +
> > +
> > +/**
> > +  This service allows the caller to set a startup procedure that will be
> > executed when an AP powers
> > +  up from a state where core configuration and context is lost. The
> > procedure is execution has the
> > +  following properties:
> > +  1. The procedure executes before the processor is handed over to the
> > operating system.
> > +  2. All processors execute the same startup procedure.
> > +  3. The procedure may run in parallel with other procedures invoked
> > through the functions in this
> > +  protocol, or with processors that are executing an MM handler or
> running
> > in the operating system.
> > +
> > +
> > +  @param[in]      This                 The EFI_MM_MP_PROTOCOL instance.
> > +  @param[in]      Procedure            A pointer to the code stream to be run on
> > the designated target AP
> > +                                       of the system. Type EFI_AP_PROCEDURE is defined
> > below in Volume 2
> > +                                       with the related definitions of
> > +                                       EFI_MP_SERVICES_PROTOCOL.StartupAllAPs.
> > +                                       If caller may pass a value of NULL to deregister any
> > existing
> > +                                       startup procedure.
> > +  @param[in,out]  ProcedureArguments   Allows the caller to pass a list of
> > parameters to the code that is
> > +                                       run by the AP. It is an optional common mailbox
> > between APs and
> > +                                       the caller to share information
> > +
> > +  @retval EFI_SUCCESS                  The Procedure has been set successfully.
> > +  @retval EFI_INVALID_PARAMETER        The Procedure is NULL but
> > ProcedureArguments not NULL.
> > +**/
> > +EFI_STATUS
> > +EFIAPI
> > +SmmMpSetStartupProcedure (
> > +  IN CONST EFI_MM_MP_PROTOCOL  *This,
> > +  IN       EFI_AP_PROCEDURE    Procedure,
> > +  IN OUT   VOID                *ProcedureArguments OPTIONAL
> > +  );
> > +
> > +/**
> > +  When non-blocking execution of a procedure on an AP is invoked with
> > DispatchProcedure,
> > +  via the use of a token, this function can be used to check for completion
> of
> > the procedure on the AP.
> > +  The function takes the token that was passed into the
> DispatchProcedure
> > call. If the procedure
> > +  is complete, and therefore it is now possible to run another procedure
> on
> > the same AP, this function
> > +  returns EFI_SUCESS. In this case the status returned by the procedure
> that
> > executed on the AP is
> > +  returned in the token's Status field. If the procedure has not yet
> > completed, then this function
> > +  returns EFI_NOT_READY.
> > +
> > +  When a non-blocking execution of a procedure is invoked with
> > BroadcastProcedure, via the
> > +  use of a token, this function can be used to check for completion of the
> > procedure on all the
> > +  broadcast APs. The function takes the token that was passed into the
> > BroadcastProcedure
> > +  call. If the procedure is complete on all broadcast APs this function
> returns
> > EFI_SUCESS. In this
> > +  case the Status field in the token passed into the function reflects the
> > overall result of the
> > +  invocation, which may be EFI_SUCCESS, if all executions succeeded, or
> the
> > first observed failure.
> > +  If the procedure has not yet completed on the broadcast APs, the
> function
> > returns
> > +  EFI_NOT_READY.
> > +
> > +  @param[in]      This                 The EFI_MM_MP_PROTOCOL instance.
> > +  @param[in]      Token                This parameter describes the token that
> was
> > passed into
> > +                                       DispatchProcedure or BroadcastProcedure.
> > +
> > +  @retval EFI_SUCCESS                  Procedure has completed.
> > +  @retval EFI_NOT_READY                The Procedure has not completed.
> > +  @retval EFI_INVALID_PARAMETER        Token or Token->Completion is
> > NULL
> > +  @retval EFI_NOT_FOUND                Token is not currently in use for a non-
> > blocking call
> > +
> > +**/
> > +EFI_STATUS
> > +EFIAPI
> > +SmmMpCheckForProcedure (
> > +  IN CONST EFI_MM_MP_PROTOCOL             *This,
> > +  IN       MM_COMPLETION                  Token
> > +  );
> > +
> > +/**
> > +  When a non-blocking execution of a procedure on an AP is invoked via
> > DispatchProcedure,
> > +  this function will block the caller until the remote procedure has
> completed
> > on the designated AP.
> > +  The non-blocking procedure invocation is identified by the Token
> > parameter, which must match the
> > +  token that used when DispatchProcedure was called. Upon completion
> the
> > status returned by
> > +  the procedure that executed on the AP is used to update the token's
> > Status field.
> > +
> > +  When a non-blocking execution of a procedure on an AP is invoked via
> > BroadcastProcedure
> > +  this function will block the caller until the remote procedure has
> completed
> > on all of the APs that
> > +  entered MM. The non-blocking procedure invocation is identified by the
> > Token parameter, which
> > +  must match the token that used when BroadcastProcedure was called.
> > Upon completion the
> > +  overall status returned by the procedures that executed on the
> broadcast
> > AP is used to update the
> > +  token's Status field. The overall status may be EFI_SUCCESS, if all
> > executions succeeded, or the
> > +  first observed failure.
> > +
> > +
> > +  @param[in]      This                 The EFI_MM_MP_PROTOCOL instance.
> > +  @param[in]      Token                This parameter describes the token that
> was
> > passed into
> > +                                       DispatchProcedure or BroadcastProcedure.
> > +
> > +  @retval EFI_SUCCESS                  Procedure has completed.
> > +  @retval EFI_INVALID_PARAMETER        Token or Token->Completion is
> > NULL
> > +  @retval EFI_NOT_FOUND                Token is not currently in use for a non-
> > blocking call
> > +
> > +**/
> > +EFI_STATUS
> > +EFIAPI
> > +SmmMpWaitForProcedure (
> > +  IN CONST EFI_MM_MP_PROTOCOL            *This,
> > +  IN       MM_COMPLETION                 Token
> > +  );
> > +
> > +#endif
> > --
> > 2.21.0.windows.1
> >
> >
> > 

Re: [edk2-devel] [Patch v3 0/2] Enable new MM MP protocol

[Yao, Jiewen]

HI Eric
Would you please share what unit test you have done for this new feature?


> -----Original Message-----
> From: devel@edk2.groups.io [mailto:devel@edk2.groups.io] On Behalf Of
> Dong, Eric
> Sent: Wednesday, July 3, 2019 10:43 AM
> To: devel@edk2.groups.io
> Cc: Ni, Ray <ray.ni@intel.com>; Laszlo Ersek <lersek@redhat.com>
> Subject: [edk2-devel] [Patch v3 0/2] Enable new MM MP protocol
> 
> v3 changes:
> 1. Fix Token clean up too early caused CheckProcedure return error.
> 
> https://bugzilla.tianocore.org/show_bug.cgi?id=1937
> 
> PI spec added a new protocol named MM MP protocol. This protocol
> allows for better remote queuing of execution of procedures on an AP.
> This extends the existing procedures to allow:
> 1. A function to be called in blocking and non-blocking manner explicitly
> 2. Allow broadcasts.
> 3. Allow execution of a procedure when a processor powers up.
> 
> This patch serial enable this new protocol.
> 
> Cc: Ray Ni <ray.ni@intel.com>
> Cc: Laszlo Ersek <lersek@redhat.com>
> 
> Eric Dong (2):
>   MdePkg: Add new MM MP Protocol definition.
>   UefiCpuPkg/PiSmmCpuDxeSmm: Enable MM MP Protocol.
> 
>  MdePkg/Include/Pi/PiMultiPhase.h             |  16 +
>  MdePkg/Include/Protocol/MmMp.h               | 333 +++++++++++
>  MdePkg/MdePkg.dec                            |   3 +
>  UefiCpuPkg/PiSmmCpuDxeSmm/MpService.c        | 555
> ++++++++++++++++++-
>  UefiCpuPkg/PiSmmCpuDxeSmm/PiSmmCpuDxeSmm.c   |  11 +
>  UefiCpuPkg/PiSmmCpuDxeSmm/PiSmmCpuDxeSmm.h   | 160 +++++-
>  UefiCpuPkg/PiSmmCpuDxeSmm/PiSmmCpuDxeSmm.inf |   3 +
>  UefiCpuPkg/PiSmmCpuDxeSmm/SmmMp.c            | 372
> +++++++++++++
>  UefiCpuPkg/PiSmmCpuDxeSmm/SmmMp.h            | 286
> ++++++++++
>  9 files changed, 1722 insertions(+), 17 deletions(-)
>  create mode 100644 MdePkg/Include/Protocol/MmMp.h
>  create mode 100644 UefiCpuPkg/PiSmmCpuDxeSmm/SmmMp.c
>  create mode 100644 UefiCpuPkg/PiSmmCpuDxeSmm/SmmMp.h
> 
> --
> 2.21.0.windows.1
> 
> 
> 

Re: [edk2-devel] [Patch v3 0/2] Enable new MM MP protocol

[Dong, Eric]

Hi Jiewen,

I did below tests:
1. Use MpGetNumberOfProcessors and MpSetStartupProcedure.
2. Test MpDispatchProcedure
	1.sync mode and async mode, 
	2. with and without CpuStatus.
	3. In async mode, use long delay get the final result with MpWaitForProcedure
	4. In async mode, use short delay to check the final result with MpCheckForProcedure
3. Test  MpBroadcastProcedure
	1.sync mode and async mode, 
	2. with and without CpuStatus.
	3. In async mode, use long delay get the final result with MpWaitForProcedure.
	4. In async mode, use short delay to check the final result with MpCheckForProcedure.

Detail test code can be found in MmMpUnitTest folder at git@github.com:ydong10/UnitTestPkg.git

Thanks,
Eric

> -----Original Message-----
> From: Yao, Jiewen
> Sent: Wednesday, July 3, 2019 10:45 AM
> To: devel@edk2.groups.io; Dong, Eric <eric.dong@intel.com>
> Cc: Ni, Ray <ray.ni@intel.com>; Laszlo Ersek <lersek@redhat.com>
> Subject: RE: [edk2-devel] [Patch v3 0/2] Enable new MM MP protocol
> 
> HI Eric
> Would you please share what unit test you have done for this new feature?
> 
> 
> > -----Original Message-----
> > From: devel@edk2.groups.io [mailto:devel@edk2.groups.io] On Behalf Of
> > Dong, Eric
> > Sent: Wednesday, July 3, 2019 10:43 AM
> > To: devel@edk2.groups.io
> > Cc: Ni, Ray <ray.ni@intel.com>; Laszlo Ersek <lersek@redhat.com>
> > Subject: [edk2-devel] [Patch v3 0/2] Enable new MM MP protocol
> >
> > v3 changes:
> > 1. Fix Token clean up too early caused CheckProcedure return error.
> >
> > https://bugzilla.tianocore.org/show_bug.cgi?id=1937
> >
> > PI spec added a new protocol named MM MP protocol. This protocol
> > allows for better remote queuing of execution of procedures on an AP.
> > This extends the existing procedures to allow:
> > 1. A function to be called in blocking and non-blocking manner
> > explicitly 2. Allow broadcasts.
> > 3. Allow execution of a procedure when a processor powers up.
> >
> > This patch serial enable this new protocol.
> >
> > Cc: Ray Ni <ray.ni@intel.com>
> > Cc: Laszlo Ersek <lersek@redhat.com>
> >
> > Eric Dong (2):
> >   MdePkg: Add new MM MP Protocol definition.
> >   UefiCpuPkg/PiSmmCpuDxeSmm: Enable MM MP Protocol.
> >
> >  MdePkg/Include/Pi/PiMultiPhase.h             |  16 +
> >  MdePkg/Include/Protocol/MmMp.h               | 333 +++++++++++
> >  MdePkg/MdePkg.dec                            |   3 +
> >  UefiCpuPkg/PiSmmCpuDxeSmm/MpService.c        | 555
> > ++++++++++++++++++-
> >  UefiCpuPkg/PiSmmCpuDxeSmm/PiSmmCpuDxeSmm.c   |  11 +
> >  UefiCpuPkg/PiSmmCpuDxeSmm/PiSmmCpuDxeSmm.h   | 160 +++++-
> >  UefiCpuPkg/PiSmmCpuDxeSmm/PiSmmCpuDxeSmm.inf |   3 +
> >  UefiCpuPkg/PiSmmCpuDxeSmm/SmmMp.c            | 372
> > +++++++++++++
> >  UefiCpuPkg/PiSmmCpuDxeSmm/SmmMp.h            | 286
> > ++++++++++
> >  9 files changed, 1722 insertions(+), 17 deletions(-)  create mode
> > 100644 MdePkg/Include/Protocol/MmMp.h  create mode 100644
> > UefiCpuPkg/PiSmmCpuDxeSmm/SmmMp.c  create mode 100644
> > UefiCpuPkg/PiSmmCpuDxeSmm/SmmMp.h
> >
> > --
> > 2.21.0.windows.1
> >
> >
> > 

Re: [edk2-devel] [Patch v3 0/2] Enable new MM MP protocol

[Yao, Jiewen]

It looks great. Thanks!

> -----Original Message-----
> From: Dong, Eric
> Sent: Wednesday, July 3, 2019 10:58 AM
> To: Yao, Jiewen <jiewen.yao@intel.com>; devel@edk2.groups.io
> Cc: Ni, Ray <ray.ni@intel.com>; Laszlo Ersek <lersek@redhat.com>
> Subject: RE: [edk2-devel] [Patch v3 0/2] Enable new MM MP protocol
> 
> Hi Jiewen,
> 
> I did below tests:
> 1. Use MpGetNumberOfProcessors and MpSetStartupProcedure.
> 2. Test MpDispatchProcedure
> 	1.sync mode and async mode,
> 	2. with and without CpuStatus.
> 	3. In async mode, use long delay get the final result with
> MpWaitForProcedure
> 	4. In async mode, use short delay to check the final result with
> MpCheckForProcedure
> 3. Test  MpBroadcastProcedure
> 	1.sync mode and async mode,
> 	2. with and without CpuStatus.
> 	3. In async mode, use long delay get the final result with
> MpWaitForProcedure.
> 	4. In async mode, use short delay to check the final result with
> MpCheckForProcedure.
> 
> Detail test code can be found in MmMpUnitTest folder at
> git@github.com:ydong10/UnitTestPkg.git
> 
> Thanks,
> Eric
> 
> > -----Original Message-----
> > From: Yao, Jiewen
> > Sent: Wednesday, July 3, 2019 10:45 AM
> > To: devel@edk2.groups.io; Dong, Eric <eric.dong@intel.com>
> > Cc: Ni, Ray <ray.ni@intel.com>; Laszlo Ersek <lersek@redhat.com>
> > Subject: RE: [edk2-devel] [Patch v3 0/2] Enable new MM MP protocol
> >
> > HI Eric
> > Would you please share what unit test you have done for this new feature?
> >
> >
> > > -----Original Message-----
> > > From: devel@edk2.groups.io [mailto:devel@edk2.groups.io] On Behalf
> Of
> > > Dong, Eric
> > > Sent: Wednesday, July 3, 2019 10:43 AM
> > > To: devel@edk2.groups.io
> > > Cc: Ni, Ray <ray.ni@intel.com>; Laszlo Ersek <lersek@redhat.com>
> > > Subject: [edk2-devel] [Patch v3 0/2] Enable new MM MP protocol
> > >
> > > v3 changes:
> > > 1. Fix Token clean up too early caused CheckProcedure return error.
> > >
> > > https://bugzilla.tianocore.org/show_bug.cgi?id=1937
> > >
> > > PI spec added a new protocol named MM MP protocol. This protocol
> > > allows for better remote queuing of execution of procedures on an AP.
> > > This extends the existing procedures to allow:
> > > 1. A function to be called in blocking and non-blocking manner
> > > explicitly 2. Allow broadcasts.
> > > 3. Allow execution of a procedure when a processor powers up.
> > >
> > > This patch serial enable this new protocol.
> > >
> > > Cc: Ray Ni <ray.ni@intel.com>
> > > Cc: Laszlo Ersek <lersek@redhat.com>
> > >
> > > Eric Dong (2):
> > >   MdePkg: Add new MM MP Protocol definition.
> > >   UefiCpuPkg/PiSmmCpuDxeSmm: Enable MM MP Protocol.
> > >
> > >  MdePkg/Include/Pi/PiMultiPhase.h             |  16 +
> > >  MdePkg/Include/Protocol/MmMp.h               | 333
> +++++++++++
> > >  MdePkg/MdePkg.dec                            |   3 +
> > >  UefiCpuPkg/PiSmmCpuDxeSmm/MpService.c        | 555
> > > ++++++++++++++++++-
> > >  UefiCpuPkg/PiSmmCpuDxeSmm/PiSmmCpuDxeSmm.c   |  11 +
> > >  UefiCpuPkg/PiSmmCpuDxeSmm/PiSmmCpuDxeSmm.h   | 160 +++++-
> > >  UefiCpuPkg/PiSmmCpuDxeSmm/PiSmmCpuDxeSmm.inf |   3 +
> > >  UefiCpuPkg/PiSmmCpuDxeSmm/SmmMp.c            | 372
> > > +++++++++++++
> > >  UefiCpuPkg/PiSmmCpuDxeSmm/SmmMp.h            | 286
> > > ++++++++++
> > >  9 files changed, 1722 insertions(+), 17 deletions(-)  create mode
> > > 100644 MdePkg/Include/Protocol/MmMp.h  create mode 100644
> > > UefiCpuPkg/PiSmmCpuDxeSmm/SmmMp.c  create mode 100644
> > > UefiCpuPkg/PiSmmCpuDxeSmm/SmmMp.h
> > >
> > > --
> > > 2.21.0.windows.1
> > >
> > >
> > > 

Re: [edk2-devel] [Patch v3 0/2] Enable new MM MP protocol

[Laszlo Ersek]

On 07/03/19 04:42, Dong, Eric wrote:
> v3 changes:
> 1. Fix Token clean up too early caused CheckProcedure return error.
> 
> https://bugzilla.tianocore.org/show_bug.cgi?id=1937
> 
> PI spec added a new protocol named MM MP protocol. This protocol 
> allows for better remote queuing of execution of procedures on an AP.
> This extends the existing procedures to allow:
> 1. A function to be called in blocking and non-blocking manner explicitly 
> 2. Allow broadcasts.
> 3. Allow execution of a procedure when a processor powers up.
> 
> This patch serial enable this new protocol.
> 
> Cc: Ray Ni <ray.ni@intel.com>
> Cc: Laszlo Ersek <lersek@redhat.com>
> 
> Eric Dong (2):
>   MdePkg: Add new MM MP Protocol definition.
>   UefiCpuPkg/PiSmmCpuDxeSmm: Enable MM MP Protocol.
> 
>  MdePkg/Include/Pi/PiMultiPhase.h             |  16 +
>  MdePkg/Include/Protocol/MmMp.h               | 333 +++++++++++
>  MdePkg/MdePkg.dec                            |   3 +
>  UefiCpuPkg/PiSmmCpuDxeSmm/MpService.c        | 555 ++++++++++++++++++-
>  UefiCpuPkg/PiSmmCpuDxeSmm/PiSmmCpuDxeSmm.c   |  11 +
>  UefiCpuPkg/PiSmmCpuDxeSmm/PiSmmCpuDxeSmm.h   | 160 +++++-
>  UefiCpuPkg/PiSmmCpuDxeSmm/PiSmmCpuDxeSmm.inf |   3 +
>  UefiCpuPkg/PiSmmCpuDxeSmm/SmmMp.c            | 372 +++++++++++++
>  UefiCpuPkg/PiSmmCpuDxeSmm/SmmMp.h            | 286 ++++++++++
>  9 files changed, 1722 insertions(+), 17 deletions(-)
>  create mode 100644 MdePkg/Include/Protocol/MmMp.h
>  create mode 100644 UefiCpuPkg/PiSmmCpuDxeSmm/SmmMp.c
>  create mode 100644 UefiCpuPkg/PiSmmCpuDxeSmm/SmmMp.h
> 

Regression-tested-by: Laszlo Ersek <lersek@redhat.com>