[PATCH v4 0/9] support CPU hot-unplug

[PATCH v4 0/9] support CPU hot-unplug

[Ankur Arora]

Hi,

This series adds support for CPU hot-unplug with OVMF.

Please see this in conjunction with the QEMU secureboot hot-unplug v2
series posted here (now upstreamed):
  https://lore.kernel.org/qemu-devel/20201207140739.3829993-1-imammedo@redhat.com/

Patches 1 and 3,
  ("OvmfPkg/CpuHotplugSmm: refactor hotplug logic")
  ("OvmfPkg/CpuHotplugSmm: add Qemu Cpu Status helper")
are either refactors or add support functions.

  OvmfPkg/CpuHotplugSmm: introduce UnplugCpus()
  OvmfPkg/CpuHotplugSmm: add CpuEject()
  OvmfPkg/CpuHotplugSmm: add worker to do CPU ejection

Patch 2 and 9,
  ("OvmfPkg/CpuHotplugSmm: collect hot-unplug events")
  ("OvmfPkg/SmmControl2Dxe: negotiate CPU hot-unplug")
handle the QEMU protocol logic for collection of CPU hot-unplug events
or the protocol negotiation.

Patch 4,
  ("OvmfPkg/CpuHotplugSmm: introduce UnplugCpus()")
adds the MMI logic for CPU hot-unplug handling and informing
the PiSmmCpuDxeSmm of CPU removal.

Patches 5 and 6,
  ("OvmfPkg/CpuHotplugSmm: define CPU_HOT_EJECT_DATA")
  ("OvmfPkg/SmmCpuFeaturesLib: init CPU ejection state")
sets up state for doing the CPU ejection as part of hot-unplug.

Patches 7, and 8,
  ("OvmfPkg/CpuHotplugSmm: introduce UnplugCpus()")
  ("OvmfPkg/CpuHotplugSmm: add worker to do CPU ejection")
add the CPU ejection logic.

Testing (with QEMU 5.2.50):
 - Stable with randomized CPU plug/unplug (guest maxcpus=1,8,128)
 - Synthetic tests with simultaneous multi CPU hot-unplug
 - Negotiation with/without CPU hotplug enabled

Also at:
  github.com/terminus/edk2/ hot-unplug-v4

Changelog:
v4:
  - Gets rid of unnecessary UefiCpuPkg changes

v3:
  - Use a saner PCD based interface to share state between PiSmmCpuDxeSmm
    and OvmfPkg/CpuHotplugSmm
  - Cleaner split of the hot-unplug code
  URL: https://patchew.org/EDK2/20210115074533.277448-1-ankur.a.arora@oracle.com/

v2:
  - Do the ejection via SmmCpuFeaturesRendezvousExit()
  URL: https://patchew.org/EDK2/20210107195515.106158-1-ankur.a.arora@oracle.com/

RFC:
  URL: https://patchew.org/EDK2/20201208053432.2690694-1-ankur.a.arora@oracle.com/


Please review.

Thanks
Ankur

Ankur Arora (9):
  OvmfPkg/CpuHotplugSmm: refactor hotplug logic
  OvmfPkg/CpuHotplugSmm: collect hot-unplug events
  OvmfPkg/CpuHotplugSmm: add Qemu Cpu Status helper
  OvmfPkg/CpuHotplugSmm: introduce UnplugCpus()
  OvmfPkg/CpuHotplugSmm: define CPU_HOT_EJECT_DATA
  OvmfPkg/SmmCpuFeaturesLib: init CPU ejection state
  OvmfPkg/CpuHotplugSmm: add CpuEject()
  OvmfPkg/CpuHotplugSmm: add worker to do CPU ejection
  OvmfPkg/SmmControl2Dxe: negotiate CPU hot-unplug

 OvmfPkg/CpuHotplugSmm/CpuHotplug.c                 | 435 ++++++++++++++++-----
 OvmfPkg/CpuHotplugSmm/CpuHotplugSmm.inf            |   1 +
 OvmfPkg/CpuHotplugSmm/QemuCpuhp.c                  |  58 ++-
 OvmfPkg/CpuHotplugSmm/QemuCpuhp.h                  |   6 +
 OvmfPkg/Include/IndustryStandard/QemuCpuHotplug.h  |   2 +
 OvmfPkg/Include/Library/CpuHotEjectData.h          |  31 ++
 .../Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.c  |  62 +++
 .../SmmCpuFeaturesLib/SmmCpuFeaturesLib.inf        |   3 +
 OvmfPkg/OvmfPkg.dec                                |   6 +
 OvmfPkg/SmmControl2Dxe/SmiFeatures.c               |  25 +-
 10 files changed, 527 insertions(+), 102 deletions(-)
 create mode 100644 OvmfPkg/Include/Library/CpuHotEjectData.h

-- 
2.9.3

[PATCH v4 1/9] OvmfPkg/CpuHotplugSmm: refactor hotplug logic

[Ankur Arora]

Refactor CpuHotplugMmi() to pull out the CPU hotplug logic into
PlugCpus(). This is in preparation for supporting CPU hot-unplug.

Cc: Laszlo Ersek <lersek@redhat.com>
Cc: Jordan Justen <jordan.l.justen@intel.com>
Cc: Ard Biesheuvel <ard.biesheuvel@arm.com>
Cc: Igor Mammedov <imammedo@redhat.com>
Cc: Boris Ostrovsky <boris.ostrovsky@oracle.com>
Cc: Aaron Young <aaron.young@oracle.com>
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3132
Signed-off-by: Ankur Arora <ankur.a.arora@oracle.com>
---
 OvmfPkg/CpuHotplugSmm/CpuHotplug.c | 208 ++++++++++++++++++++++---------------
 1 file changed, 123 insertions(+), 85 deletions(-)

diff --git a/OvmfPkg/CpuHotplugSmm/CpuHotplug.c b/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
index cfe698ed2b5e..38c71bc11864 100644
--- a/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
+++ b/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
@@ -62,6 +62,124 @@ STATIC UINT32 mPostSmmPenAddress;
 //
 STATIC EFI_HANDLE mDispatchHandle;
 
+/**
+  CPU Hotplug handler function.
+
+  @param[in] mPluggedApicIds     List of APIC IDs to be plugged.
+
+  @param[in] PluggedCount        Count of APIC IDs to be plugged.
+
+  @retval EFI_SUCCESS	         Some of the requested APIC IDs were hot-plugged.
+
+  @retval EFI_INTERRUPT_PENDING  Fatal error while hot-plugging.
+
+**/
+STATIC
+EFI_STATUS
+EFIAPI
+PlugCpus(
+  IN APIC_ID                      *mPluggedApicIds,
+  IN UINT32                       PluggedCount
+  )
+{
+  EFI_STATUS Status;
+  UINT32     PluggedIdx;
+  UINT32     NewSlot;
+
+  //
+  // Process hot-added CPUs.
+  //
+  // The Post-SMM Pen need not be reinstalled multiple times within a single
+  // root MMI handling. Even reinstalling once per root MMI is only prudence;
+  // in theory installing the pen in the driver's entry point function should
+  // suffice.
+  //
+  SmbaseReinstallPostSmmPen (mPostSmmPenAddress);
+
+  PluggedIdx = 0;
+  NewSlot = 0;
+  while (PluggedIdx < PluggedCount) {
+    APIC_ID NewApicId;
+    UINT32  CheckSlot;
+    UINTN   NewProcessorNumberByProtocol;
+
+    NewApicId = mPluggedApicIds[PluggedIdx];
+
+    //
+    // Check if the supposedly hot-added CPU is already known to us.
+    //
+    for (CheckSlot = 0;
+         CheckSlot < mCpuHotPlugData->ArrayLength;
+         CheckSlot++) {
+      if (mCpuHotPlugData->ApicId[CheckSlot] == NewApicId) {
+        break;
+      }
+    }
+    if (CheckSlot < mCpuHotPlugData->ArrayLength) {
+      DEBUG ((DEBUG_VERBOSE, "%a: APIC ID " FMT_APIC_ID " was hot-plugged "
+        "before; ignoring it\n", __FUNCTION__, NewApicId));
+      PluggedIdx++;
+      continue;
+    }
+
+    //
+    // Find the first empty slot in CPU_HOT_PLUG_DATA.
+    //
+    while (NewSlot < mCpuHotPlugData->ArrayLength &&
+           mCpuHotPlugData->ApicId[NewSlot] != MAX_UINT64) {
+      NewSlot++;
+    }
+    if (NewSlot == mCpuHotPlugData->ArrayLength) {
+      DEBUG ((DEBUG_ERROR, "%a: no room for APIC ID " FMT_APIC_ID "\n",
+        __FUNCTION__, NewApicId));
+      goto Fatal;
+    }
+
+    //
+    // Store the APIC ID of the new processor to the slot.
+    //
+    mCpuHotPlugData->ApicId[NewSlot] = NewApicId;
+
+    //
+    // Relocate the SMBASE of the new CPU.
+    //
+    Status = SmbaseRelocate (NewApicId, mCpuHotPlugData->SmBase[NewSlot],
+               mPostSmmPenAddress);
+    if (EFI_ERROR (Status)) {
+      goto RevokeNewSlot;
+    }
+
+    //
+    // Add the new CPU with EFI_SMM_CPU_SERVICE_PROTOCOL.
+    //
+    Status = mMmCpuService->AddProcessor (mMmCpuService, NewApicId,
+                              &NewProcessorNumberByProtocol);
+    if (EFI_ERROR (Status)) {
+      DEBUG ((DEBUG_ERROR, "%a: AddProcessor(" FMT_APIC_ID "): %r\n",
+        __FUNCTION__, NewApicId, Status));
+      goto RevokeNewSlot;
+    }
+
+    DEBUG ((DEBUG_INFO, "%a: hot-added APIC ID " FMT_APIC_ID ", SMBASE 0x%Lx, "
+      "EFI_SMM_CPU_SERVICE_PROTOCOL assigned number %Lu\n", __FUNCTION__,
+      NewApicId, (UINT64)mCpuHotPlugData->SmBase[NewSlot],
+      (UINT64)NewProcessorNumberByProtocol));
+
+    NewSlot++;
+    PluggedIdx++;
+  }
+
+  //
+  // We've handled this hotplug.
+  //
+  return EFI_SUCCESS;
+
+RevokeNewSlot:
+  mCpuHotPlugData->ApicId[NewSlot] = MAX_UINT64;
+
+Fatal:
+  return EFI_INTERRUPT_PENDING;
+}
 
 /**
   CPU Hotplug MMI handler function.
@@ -122,8 +240,6 @@ CpuHotplugMmi (
   UINT8      ApmControl;
   UINT32     PluggedCount;
   UINT32     ToUnplugCount;
-  UINT32     PluggedIdx;
-  UINT32     NewSlot;
 
   //
   // Assert that we are entering this function due to our root MMI handler
@@ -179,87 +295,12 @@ CpuHotplugMmi (
     goto Fatal;
   }
 
-  //
-  // Process hot-added CPUs.
-  //
-  // The Post-SMM Pen need not be reinstalled multiple times within a single
-  // root MMI handling. Even reinstalling once per root MMI is only prudence;
-  // in theory installing the pen in the driver's entry point function should
-  // suffice.
-  //
-  SmbaseReinstallPostSmmPen (mPostSmmPenAddress);
+  if (PluggedCount > 0) {
+    Status = PlugCpus(mPluggedApicIds, PluggedCount);
+  }
 
-  PluggedIdx = 0;
-  NewSlot = 0;
-  while (PluggedIdx < PluggedCount) {
-    APIC_ID NewApicId;
-    UINT32  CheckSlot;
-    UINTN   NewProcessorNumberByProtocol;
-
-    NewApicId = mPluggedApicIds[PluggedIdx];
-
-    //
-    // Check if the supposedly hot-added CPU is already known to us.
-    //
-    for (CheckSlot = 0;
-         CheckSlot < mCpuHotPlugData->ArrayLength;
-         CheckSlot++) {
-      if (mCpuHotPlugData->ApicId[CheckSlot] == NewApicId) {
-        break;
-      }
-    }
-    if (CheckSlot < mCpuHotPlugData->ArrayLength) {
-      DEBUG ((DEBUG_VERBOSE, "%a: APIC ID " FMT_APIC_ID " was hot-plugged "
-        "before; ignoring it\n", __FUNCTION__, NewApicId));
-      PluggedIdx++;
-      continue;
-    }
-
-    //
-    // Find the first empty slot in CPU_HOT_PLUG_DATA.
-    //
-    while (NewSlot < mCpuHotPlugData->ArrayLength &&
-           mCpuHotPlugData->ApicId[NewSlot] != MAX_UINT64) {
-      NewSlot++;
-    }
-    if (NewSlot == mCpuHotPlugData->ArrayLength) {
-      DEBUG ((DEBUG_ERROR, "%a: no room for APIC ID " FMT_APIC_ID "\n",
-        __FUNCTION__, NewApicId));
-      goto Fatal;
-    }
-
-    //
-    // Store the APIC ID of the new processor to the slot.
-    //
-    mCpuHotPlugData->ApicId[NewSlot] = NewApicId;
-
-    //
-    // Relocate the SMBASE of the new CPU.
-    //
-    Status = SmbaseRelocate (NewApicId, mCpuHotPlugData->SmBase[NewSlot],
-               mPostSmmPenAddress);
-    if (EFI_ERROR (Status)) {
-      goto RevokeNewSlot;
-    }
-
-    //
-    // Add the new CPU with EFI_SMM_CPU_SERVICE_PROTOCOL.
-    //
-    Status = mMmCpuService->AddProcessor (mMmCpuService, NewApicId,
-                              &NewProcessorNumberByProtocol);
-    if (EFI_ERROR (Status)) {
-      DEBUG ((DEBUG_ERROR, "%a: AddProcessor(" FMT_APIC_ID "): %r\n",
-        __FUNCTION__, NewApicId, Status));
-      goto RevokeNewSlot;
-    }
-
-    DEBUG ((DEBUG_INFO, "%a: hot-added APIC ID " FMT_APIC_ID ", SMBASE 0x%Lx, "
-      "EFI_SMM_CPU_SERVICE_PROTOCOL assigned number %Lu\n", __FUNCTION__,
-      NewApicId, (UINT64)mCpuHotPlugData->SmBase[NewSlot],
-      (UINT64)NewProcessorNumberByProtocol));
-
-    NewSlot++;
-    PluggedIdx++;
+  if (EFI_ERROR(Status)) {
+    goto Fatal;
   }
 
   //
@@ -267,9 +308,6 @@ CpuHotplugMmi (
   //
   return EFI_SUCCESS;
 
-RevokeNewSlot:
-  mCpuHotPlugData->ApicId[NewSlot] = MAX_UINT64;
-
 Fatal:
   ASSERT (FALSE);
   CpuDeadLoop ();
-- 
2.9.3

[PATCH v4 2/9] OvmfPkg/CpuHotplugSmm: collect hot-unplug events

[Ankur Arora]

Process fw_remove events in QemuCpuhpCollectApicIds() and collect
corresponding APIC IDs for CPUs that are being hot-unplugged.

In addition, we now ignore CPUs which only have remove set. These
CPUs haven't been processed by OSPM yet.

This is based on the QEMU hot-unplug protocol documented here:
  https://lore.kernel.org/qemu-devel/20201204170939.1815522-3-imammedo@redhat.com/

Also define QEMU_CPUHP_STAT_EJECTED while we are at it.

Cc: Laszlo Ersek <lersek@redhat.com>
Cc: Jordan Justen <jordan.l.justen@intel.com>
Cc: Ard Biesheuvel <ard.biesheuvel@arm.com>
Cc: Igor Mammedov <imammedo@redhat.com>
Cc: Boris Ostrovsky <boris.ostrovsky@oracle.com>
Cc: Aaron Young <aaron.young@oracle.com>
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3132
Signed-off-by: Ankur Arora <ankur.a.arora@oracle.com>
---
 OvmfPkg/CpuHotplugSmm/QemuCpuhp.c                 | 35 +++++++++++++++++------
 OvmfPkg/Include/IndustryStandard/QemuCpuHotplug.h |  2 ++
 2 files changed, 29 insertions(+), 8 deletions(-)

diff --git a/OvmfPkg/CpuHotplugSmm/QemuCpuhp.c b/OvmfPkg/CpuHotplugSmm/QemuCpuhp.c
index 8d4a6693c8d6..16ecf8cc433e 100644
--- a/OvmfPkg/CpuHotplugSmm/QemuCpuhp.c
+++ b/OvmfPkg/CpuHotplugSmm/QemuCpuhp.c
@@ -205,7 +205,7 @@ QemuCpuhpCollectApicIds (
     UINT8   CpuStatus;
     APIC_ID *ExtendIds;
     UINT32  *ExtendCount;
-    APIC_ID NewApicId;
+    APIC_ID OpApicId;
 
     //
     // Write CurrentSelector (which is valid) to the CPU selector register.
@@ -245,10 +245,10 @@ QemuCpuhpCollectApicIds (
     if ((CpuStatus & QEMU_CPUHP_STAT_INSERT) != 0) {
       //
       // The "insert" event guarantees the "enabled" status; plus it excludes
-      // the "remove" event.
+      // the "fw_remove" event.
       //
       if ((CpuStatus & QEMU_CPUHP_STAT_ENABLED) == 0 ||
-          (CpuStatus & QEMU_CPUHP_STAT_REMOVE) != 0) {
+          (CpuStatus & QEMU_CPUHP_STAT_FW_REMOVE) != 0) {
         DEBUG ((DEBUG_ERROR, "%a: CurrentSelector=%u CpuStatus=0x%x: "
           "inconsistent CPU status\n", __FUNCTION__, CurrentSelector,
           CpuStatus));
@@ -260,12 +260,31 @@ QemuCpuhpCollectApicIds (
 
       ExtendIds   = PluggedApicIds;
       ExtendCount = PluggedCount;
-    } else if ((CpuStatus & QEMU_CPUHP_STAT_REMOVE) != 0) {
-      DEBUG ((DEBUG_VERBOSE, "%a: CurrentSelector=%u: remove\n", __FUNCTION__,
+    } else if ((CpuStatus & QEMU_CPUHP_STAT_FW_REMOVE) != 0) {
+      //
+      // "fw_remove" event guarantees "enabled".
+      //
+      if ((CpuStatus & QEMU_CPUHP_STAT_ENABLED) == 0) {
+        DEBUG ((DEBUG_ERROR, "%a: CurrentSelector=%u CpuStatus=0x%x: "
+          "inconsistent CPU status\n", __FUNCTION__, CurrentSelector,
+          CpuStatus));
+        return EFI_PROTOCOL_ERROR;
+      }
+
+      DEBUG ((DEBUG_VERBOSE, "%a: CurrentSelector=%u: fw_remove\n", __FUNCTION__,
         CurrentSelector));
 
       ExtendIds   = ToUnplugApicIds;
       ExtendCount = ToUnplugCount;
+    } else if ((CpuStatus & QEMU_CPUHP_STAT_REMOVE) != 0) {
+      //
+      // Let the OSPM deal with the "remove" event.
+      //
+      DEBUG ((DEBUG_INFO, "%a: CurrentSelector=%u: remove (ignored)\n", __FUNCTION__,
+        CurrentSelector));
+
+      CurrentSelector++;
+      continue;
     } else {
       DEBUG ((DEBUG_VERBOSE, "%a: CurrentSelector=%u: no event\n",
         __FUNCTION__, CurrentSelector));
@@ -281,10 +300,10 @@ QemuCpuhpCollectApicIds (
       return EFI_BUFFER_TOO_SMALL;
     }
     QemuCpuhpWriteCommand (MmCpuIo, QEMU_CPUHP_CMD_GET_ARCH_ID);
-    NewApicId = QemuCpuhpReadCommandData (MmCpuIo);
+    OpApicId = QemuCpuhpReadCommandData (MmCpuIo);
     DEBUG ((DEBUG_VERBOSE, "%a: ApicId=" FMT_APIC_ID "\n", __FUNCTION__,
-      NewApicId));
-    ExtendIds[(*ExtendCount)++] = NewApicId;
+      OpApicId, ExtendCount));
+    ExtendIds[(*ExtendCount)++] = OpApicId;
 
     //
     // We've processed the CPU with (known) pending events, but we must never
diff --git a/OvmfPkg/Include/IndustryStandard/QemuCpuHotplug.h b/OvmfPkg/Include/IndustryStandard/QemuCpuHotplug.h
index a34a6d3fae61..bcae60f77ba6 100644
--- a/OvmfPkg/Include/IndustryStandard/QemuCpuHotplug.h
+++ b/OvmfPkg/Include/IndustryStandard/QemuCpuHotplug.h
@@ -34,6 +34,8 @@
 #define QEMU_CPUHP_STAT_ENABLED                BIT0
 #define QEMU_CPUHP_STAT_INSERT                 BIT1
 #define QEMU_CPUHP_STAT_REMOVE                 BIT2
+#define QEMU_CPUHP_STAT_EJECTED                BIT3
+#define QEMU_CPUHP_STAT_FW_REMOVE              BIT4
 
 #define QEMU_CPUHP_RW_CMD_DATA               0x8
 
-- 
2.9.3

[PATCH v4 3/9] OvmfPkg/CpuHotplugSmm: add Qemu Cpu Status helper

[Ankur Arora]

Add QemuCpuhpWriteCpuStatus() which will be used to update the QEMU
CPU status register. On error, it hangs in a similar fashion as
other helper functions.

Cc: Laszlo Ersek <lersek@redhat.com>
Cc: Jordan Justen <jordan.l.justen@intel.com>
Cc: Ard Biesheuvel <ard.biesheuvel@arm.com>
Cc: Igor Mammedov <imammedo@redhat.com>
Cc: Boris Ostrovsky <boris.ostrovsky@oracle.com>
Cc: Aaron Young <aaron.young@oracle.com>
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3132
Signed-off-by: Ankur Arora <ankur.a.arora@oracle.com>
---
 OvmfPkg/CpuHotplugSmm/QemuCpuhp.c | 23 +++++++++++++++++++++++
 OvmfPkg/CpuHotplugSmm/QemuCpuhp.h |  6 ++++++
 2 files changed, 29 insertions(+)

diff --git a/OvmfPkg/CpuHotplugSmm/QemuCpuhp.c b/OvmfPkg/CpuHotplugSmm/QemuCpuhp.c
index 16ecf8cc433e..31c06b4d74dd 100644
--- a/OvmfPkg/CpuHotplugSmm/QemuCpuhp.c
+++ b/OvmfPkg/CpuHotplugSmm/QemuCpuhp.c
@@ -67,6 +67,29 @@ QemuCpuhpReadCpuStatus (
   return CpuStatus;
 }
 
+UINT8
+QemuCpuhpWriteCpuStatus (
+  IN CONST EFI_MM_CPU_IO_PROTOCOL *MmCpuIo,
+  IN UINT8                        CpuStatus
+  )
+{
+  EFI_STATUS Status;
+
+  Status = MmCpuIo->Io.Write(
+                         MmCpuIo,
+                         MM_IO_UINT8,
+                         ICH9_CPU_HOTPLUG_BASE + QEMU_CPUHP_R_CPU_STAT,
+                         1,
+                         &CpuStatus
+                         );
+  if (EFI_ERROR (Status)) {
+    DEBUG ((DEBUG_ERROR, "%a: %r\n", __FUNCTION__, Status));
+    ASSERT (FALSE);
+    CpuDeadLoop ();
+  }
+  return Status;
+}
+
 UINT32
 QemuCpuhpReadCommandData (
   IN CONST EFI_MM_CPU_IO_PROTOCOL *MmCpuIo
diff --git a/OvmfPkg/CpuHotplugSmm/QemuCpuhp.h b/OvmfPkg/CpuHotplugSmm/QemuCpuhp.h
index 8adaa0ad91f0..3f31c9e4c0f4 100644
--- a/OvmfPkg/CpuHotplugSmm/QemuCpuhp.h
+++ b/OvmfPkg/CpuHotplugSmm/QemuCpuhp.h
@@ -30,6 +30,12 @@ QemuCpuhpReadCpuStatus (
   IN CONST EFI_MM_CPU_IO_PROTOCOL *MmCpuIo
   );
 
+UINT8
+QemuCpuhpWriteCpuStatus (
+  IN CONST EFI_MM_CPU_IO_PROTOCOL *MmCpuIo,
+  IN UINT8                        CpuStatus
+  );
+
 UINT32
 QemuCpuhpReadCommandData (
   IN CONST EFI_MM_CPU_IO_PROTOCOL *MmCpuIo
-- 
2.9.3

[PATCH v4 4/9] OvmfPkg/CpuHotplugSmm: introduce UnplugCpus()

[Ankur Arora]

Introduce UnplugCpus() which maps each APIC ID being unplugged
onto the hardware ID of the processor and informs PiSmmCpuDxeSmm
of removal by calling EFI_SMM_CPU_SERVICE_PROTOCOL.RemoveProcessor().

With this change we handle the first phase of unplug where we collect
the CPUs that need to be unplugged and mark them for removal in SMM
data structures.

Cc: Laszlo Ersek <lersek@redhat.com>
Cc: Jordan Justen <jordan.l.justen@intel.com>
Cc: Ard Biesheuvel <ard.biesheuvel@arm.com>
Cc: Igor Mammedov <imammedo@redhat.com>
Cc: Boris Ostrovsky <boris.ostrovsky@oracle.com>
Cc: Aaron Young <aaron.young@oracle.com>
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3132
Signed-off-by: Ankur Arora <ankur.a.arora@oracle.com>
---
 OvmfPkg/CpuHotplugSmm/CpuHotplug.c | 77 ++++++++++++++++++++++++++++++++++++++
 1 file changed, 77 insertions(+)

diff --git a/OvmfPkg/CpuHotplugSmm/CpuHotplug.c b/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
index 38c71bc11864..ccd5787b7327 100644
--- a/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
+++ b/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
@@ -182,6 +182,81 @@ Fatal:
 }
 
 /**
+  CPU Hot-unplug MMI handler function.
+
+  @param[in] mUnplugApicIds      List of APIC IDs to be unplugged.
+
+  @param[in] ToUnplugCount       Count of APIC IDs to be unplugged.
+
+  @retval EFI_SUCCESS            Some of the requested APIC IDs will be hot-unplugged.
+
+  @retval EFI_INTERRUPT_PENDING  Fatal error while hot-plugging.
+
+**/
+STATIC
+EFI_STATUS
+EFIAPI
+UnplugCpus(
+  IN APIC_ID                      *mUnplugApicIds,
+  IN UINT32                       ToUnplugCount
+  )
+{
+  EFI_STATUS Status = EFI_SUCCESS;
+  UINT32     ToUnplugIdx;
+  UINTN      ProcessorNum;
+
+  ToUnplugIdx = 0;
+  while (ToUnplugIdx < ToUnplugCount) {
+    APIC_ID    RemoveApicId;
+
+    RemoveApicId = mUnplugApicIds[ToUnplugIdx];
+
+    //
+    // mCpuHotPlugData->ApicId maps ProcessorNum -> ApicId. Use it to find
+    // the ProcessorNum for the APIC ID to be removed.
+    //
+    for (ProcessorNum = 0;
+         ProcessorNum < mCpuHotPlugData->ArrayLength;
+         ProcessorNum++) {
+      if (mCpuHotPlugData->ApicId[ProcessorNum] == RemoveApicId) {
+        break;
+      }
+    }
+
+    //
+    // Ignore the unplug if APIC ID not found
+    //
+    if (ProcessorNum == mCpuHotPlugData->ArrayLength) {
+      DEBUG ((DEBUG_INFO, "%a: did not find APIC ID " FMT_APIC_ID " to unplug\n",
+        __FUNCTION__, RemoveApicId));
+      ToUnplugIdx++;
+      continue;
+    }
+
+    //
+    // Mark ProcessorNum for removal from SMM data structures
+    //
+    Status = mMmCpuService->RemoveProcessor (mMmCpuService, ProcessorNum);
+
+    if (EFI_ERROR(Status)) {
+      DEBUG ((DEBUG_ERROR, "%a: RemoveProcessor(" FMT_APIC_ID "): %r\n",
+        __FUNCTION__, RemoveApicId, Status));
+      goto Fatal;
+    }
+
+    ToUnplugIdx++;
+  }
+
+  //
+  // We've handled this unplug.
+  //
+  return EFI_SUCCESS;
+
+Fatal:
+  return EFI_INTERRUPT_PENDING;
+}
+
+/**
   CPU Hotplug MMI handler function.
 
   This is a root MMI handler.
@@ -297,6 +372,8 @@ CpuHotplugMmi (
 
   if (PluggedCount > 0) {
     Status = PlugCpus(mPluggedApicIds, PluggedCount);
+  } else if (ToUnplugCount > 0) {
+    Status = UnplugCpus(mToUnplugApicIds, ToUnplugCount);
   }
 
   if (EFI_ERROR(Status)) {
-- 
2.9.3

[PATCH v4 5/9] OvmfPkg/CpuHotplugSmm: define CPU_HOT_EJECT_DATA

[Ankur Arora]

Define CPU_HOT_EJECT_DATA and add PCD PcdCpuHotEjectDataAddress, which
will be used to share CPU ejection state between OvmfPkg/CpuHotPlugSmm
and PiSmmCpuDxeSmm.

Cc: Laszlo Ersek <lersek@redhat.com>
Cc: Jordan Justen <jordan.l.justen@intel.com>
Cc: Ard Biesheuvel <ard.biesheuvel@arm.com>
Cc: Igor Mammedov <imammedo@redhat.com>
Cc: Boris Ostrovsky <boris.ostrovsky@oracle.com>
Cc: Aaron Young <aaron.young@oracle.com>
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3132
Signed-off-by: Ankur Arora <ankur.a.arora@oracle.com>
---
 OvmfPkg/CpuHotplugSmm/CpuHotplugSmm.inf   |  1 +
 OvmfPkg/Include/Library/CpuHotEjectData.h | 31 +++++++++++++++++++++++++++++++
 OvmfPkg/OvmfPkg.dec                       |  6 ++++++
 3 files changed, 38 insertions(+)
 create mode 100644 OvmfPkg/Include/Library/CpuHotEjectData.h

diff --git a/OvmfPkg/CpuHotplugSmm/CpuHotplugSmm.inf b/OvmfPkg/CpuHotplugSmm/CpuHotplugSmm.inf
index 04322b0d7855..e08b572ef169 100644
--- a/OvmfPkg/CpuHotplugSmm/CpuHotplugSmm.inf
+++ b/OvmfPkg/CpuHotplugSmm/CpuHotplugSmm.inf
@@ -54,6 +54,7 @@
 
 [Pcd]
   gUefiCpuPkgTokenSpaceGuid.PcdCpuHotPlugDataAddress                ## CONSUMES
+  gUefiOvmfPkgTokenSpaceGuid.PcdCpuHotEjectDataAddress              ## CONSUMES
   gUefiOvmfPkgTokenSpaceGuid.PcdQ35SmramAtDefaultSmbase             ## CONSUMES
 
 [FeaturePcd]
diff --git a/OvmfPkg/Include/Library/CpuHotEjectData.h b/OvmfPkg/Include/Library/CpuHotEjectData.h
new file mode 100644
index 000000000000..9716bbcafcec
--- /dev/null
+++ b/OvmfPkg/Include/Library/CpuHotEjectData.h
@@ -0,0 +1,31 @@
+/** @file
+ * Definition for a structure sharing information for CPU hot plug.
+ *
+ * Copyright (c) 2021, Oracle Corporation.
+ * SPDX-License-Identifier: BSD-2-Clause-Patent
+ **/
+
+#ifndef _CPU_HOT_EJECT_DATA_H
+#define _CPU_HOT_EJECT_DATA_H
+
+typedef
+VOID
+(EFIAPI *CPU_HOT_EJECT_FN)(
+  IN UINTN  ProcessorNum
+  );
+
+#define CPU_EJECT_INVALID               (MAX_UINT64)
+#define CPU_EJECT_WORKER                (MAX_UINT64-1)
+
+#define  CPU_HOT_EJECT_DATA_REVISION_1	0x00000001
+
+typedef struct {
+  UINT32           Revision;          // Used for version identification for this structure
+  UINT32           ArrayLength;       // Number of entries for the ApicIdMap array
+
+  UINT64           *ApicIdMap;        // Pointer to CpuIndex->ApicId map for pending ejects
+  CPU_HOT_EJECT_FN Handler;           // Handler to do the CPU ejection
+  UINT64           Reserved;
+} CPU_HOT_EJECT_DATA;
+
+#endif /* _CPU_HOT_EJECT_DATA_H */
diff --git a/OvmfPkg/OvmfPkg.dec b/OvmfPkg/OvmfPkg.dec
index 54804962ec02..e0f247181fe2 100644
--- a/OvmfPkg/OvmfPkg.dec
+++ b/OvmfPkg/OvmfPkg.dec
@@ -347,6 +347,12 @@
   #
   #  This PCD is only accessed if PcdSmmSmramRequire is TRUE (see below).
   gUefiOvmfPkgTokenSpaceGuid.PcdQ35SmramAtDefaultSmbase|FALSE|BOOLEAN|0x34
+
+  ## This PCD adds a communication channel between PiSmmCpuDxe and
+  #  CpuHotplugSmm.
+  #
+  #  Only accessed if PcdCpuHotPlugSupport is TRUE
+  gUefiOvmfPkgTokenSpaceGuid.PcdCpuHotEjectDataAddress|0|UINT64|0x46
 
 [PcdsFeatureFlag]
   gUefiOvmfPkgTokenSpaceGuid.PcdQemuBootOrderPciTranslation|TRUE|BOOLEAN|0x1c
-- 
2.9.3

[PATCH v4 6/9] OvmfPkg/SmmCpuFeaturesLib: init CPU ejection state

[Ankur Arora]

Init CPU_HOT_EJECT_DATA, which will be used to share CPU ejection state
between SmmCpuFeaturesLib (via PiSmmCpuDxeSmm) and CpuHotPlugSmm.
CpuHotplugSmm also sets up the CPU ejection mechanism via
CPU_HOT_EJECT_DATA->Handler.

Additionally, expose CPU_HOT_EJECT_DATA via PcdCpuHotEjectDataAddress.

Cc: Laszlo Ersek <lersek@redhat.com>
Cc: Jordan Justen <jordan.l.justen@intel.com>
Cc: Ard Biesheuvel <ard.biesheuvel@arm.com>
Cc: Igor Mammedov <imammedo@redhat.com>
Cc: Boris Ostrovsky <boris.ostrovsky@oracle.com>
Cc: Aaron Young <aaron.young@oracle.com>
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3132
Signed-off-by: Ankur Arora <ankur.a.arora@oracle.com>
---
 .../Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.c  | 62 ++++++++++++++++++++++
 .../SmmCpuFeaturesLib/SmmCpuFeaturesLib.inf        |  3 ++
 2 files changed, 65 insertions(+)

diff --git a/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.c b/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.c
index 7ef7ed98342e..70313fb754e2 100644
--- a/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.c
+++ b/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.c
@@ -14,7 +14,9 @@
 #include <Library/PcdLib.h>
 #include <Library/SmmCpuFeaturesLib.h>
 #include <Library/SmmServicesTableLib.h>
+#include <Library/MemoryAllocationLib.h>                 // AllocatePool()
 #include <Library/UefiBootServicesTableLib.h>
+#include <Library/CpuHotEjectData.h>
 #include <PiSmm.h>
 #include <Register/Intel/SmramSaveStateMap.h>
 #include <Register/QemuSmramSaveStateMap.h>
@@ -171,6 +173,54 @@ SmmCpuFeaturesHookReturnFromSmm (
   return OriginalInstructionPointer;
 }
 
+GLOBAL_REMOVE_IF_UNREFERENCED
+CPU_HOT_EJECT_DATA *mCpuHotEjectData = NULL;
+
+STATIC
+VOID
+EFIAPI
+SmmCpuFeaturesSmmInitHotEject(
+  VOID
+  )
+{
+  UINT32 mMaxNumberOfCpus;
+  EFI_STATUS Status;
+
+  if (!FeaturePcdGet (PcdCpuHotPlugSupport)) {
+    return;
+  }
+
+  // PcdCpuHotPlugSupport => PcdCpuMaxLogicalProcessorNumber
+  mMaxNumberOfCpus = PcdGet32 (PcdCpuMaxLogicalProcessorNumber);
+
+  // No spare CPUs to eject
+  if (mMaxNumberOfCpus == 1) {
+    return;
+  }
+
+  mCpuHotEjectData =
+    (CPU_HOT_EJECT_DATA *)AllocatePool (sizeof (*mCpuHotEjectData));
+  ASSERT (mCpuHotEjectData != NULL);
+
+  //
+  // Allocate buffer for pointers to array in CPU_HOT_EJECT_DATA.
+  //
+  mCpuHotEjectData->Revision = CPU_HOT_EJECT_DATA_REVISION_1;   // Revision
+  mCpuHotEjectData->ArrayLength = mMaxNumberOfCpus;             // Array Length of APIC ID
+  mCpuHotEjectData->ApicIdMap =                                 // CpuIndex -> APIC ID map
+    (UINT64 *)AllocatePool (sizeof (UINT64) * mCpuHotEjectData->ArrayLength);
+  mCpuHotEjectData->Handler = NULL;                             // Hot Eject handler
+  mCpuHotEjectData->Handler = 0;                                // Reserved
+
+  ASSERT (mCpuHotEjectData->ApicIdMap != NULL);
+
+  //
+  // Expose address of CPU Hot eject Data structure
+  //
+  Status = PcdSet64S (PcdCpuHotEjectDataAddress, (UINT64)(VOID *)mCpuHotEjectData);
+  ASSERT_EFI_ERROR (Status);
+}
+
 /**
   Hook point in normal execution mode that allows the one CPU that was elected
   as monarch during System Management Mode initialization to perform additional
@@ -188,6 +238,9 @@ SmmCpuFeaturesSmmRelocationComplete (
   UINTN      MapPagesBase;
   UINTN      MapPagesCount;
 
+
+  SmmCpuFeaturesSmmInitHotEject();
+
   if (!MemEncryptSevIsEnabled ()) {
     return;
   }
@@ -375,6 +428,15 @@ SmmCpuFeaturesRendezvousExit (
   IN UINTN  CpuIndex
   )
 {
+  //
+  // CPU Hot eject not enabled.
+  //
+  if (mCpuHotEjectData == NULL ||
+      mCpuHotEjectData->Handler == NULL) {
+    return;
+  }
+
+  mCpuHotEjectData->Handler(CpuIndex);
 }
 
 /**
diff --git a/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.inf b/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.inf
index 97a10afb6e27..c27b0f2c4d03 100644
--- a/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.inf
+++ b/OvmfPkg/Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.inf
@@ -35,4 +35,7 @@
   UefiBootServicesTableLib
 
 [Pcd]
+  gUefiCpuPkgTokenSpaceGuid.PcdCpuHotPlugSupport
+  gUefiCpuPkgTokenSpaceGuid.PcdCpuMaxLogicalProcessorNumber
+  gUefiOvmfPkgTokenSpaceGuid.PcdCpuHotEjectDataAddress
   gUefiOvmfPkgTokenSpaceGuid.PcdQ35SmramAtDefaultSmbase
-- 
2.9.3

[PATCH v4 7/9] OvmfPkg/CpuHotplugSmm: add CpuEject()

[Ankur Arora]

Add CpuEject(), which handles the CPU ejection, and provides
a holding area for said CPUs. It is called via
SmmCpuFeaturesRendezvousExit(), at the tail end of the SMI
handling.

Also UnplugCpus() now stashes APIC IDs of CPUs which need to
be ejected in CPU_HOT_EJECT_DATA.ApicIdMap. These are used by
CpuEject() to identify such CPUs.

Cc: Laszlo Ersek <lersek@redhat.com>
Cc: Jordan Justen <jordan.l.justen@intel.com>
Cc: Ard Biesheuvel <ard.biesheuvel@arm.com>
Cc: Igor Mammedov <imammedo@redhat.com>
Cc: Boris Ostrovsky <boris.ostrovsky@oracle.com>
Cc: Aaron Young <aaron.young@oracle.com>
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3132
Signed-off-by: Ankur Arora <ankur.a.arora@oracle.com>
---
 OvmfPkg/CpuHotplugSmm/CpuHotplug.c | 84 +++++++++++++++++++++++++++++++++++++-
 1 file changed, 83 insertions(+), 1 deletion(-)

diff --git a/OvmfPkg/CpuHotplugSmm/CpuHotplug.c b/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
index ccd5787b7327..e8ba9ae59e69 100644
--- a/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
+++ b/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
@@ -14,6 +14,7 @@
 #include <Library/MmServicesTableLib.h>      // gMmst
 #include <Library/PcdLib.h>                  // PcdGetBool()
 #include <Library/SafeIntLib.h>              // SafeUintnSub()
+#include <Library/CpuHotEjectData.h>         // CPU_HOT_EJECT_DATA
 #include <Protocol/MmCpuIo.h>                // EFI_MM_CPU_IO_PROTOCOL
 #include <Protocol/SmmCpuService.h>          // EFI_SMM_CPU_SERVICE_PROTOCOL
 #include <Uefi/UefiBaseType.h>               // EFI_STATUS
@@ -32,11 +33,12 @@ STATIC EFI_MM_CPU_IO_PROTOCOL *mMmCpuIo;
 //
 STATIC EFI_SMM_CPU_SERVICE_PROTOCOL *mMmCpuService;
 //
-// This structure is a communication side-channel between the
+// These structures serve as communication side-channels between the
 // EFI_SMM_CPU_SERVICE_PROTOCOL consumer (i.e., this driver) and provider
 // (i.e., PiSmmCpuDxeSmm).
 //
 STATIC CPU_HOT_PLUG_DATA *mCpuHotPlugData;
+STATIC CPU_HOT_EJECT_DATA *mCpuHotEjectData;
 //
 // SMRAM arrays for fetching the APIC IDs of processors with pending events (of
 // known event types), for the time of just one MMI.
@@ -181,6 +183,34 @@ Fatal:
   return EFI_INTERRUPT_PENDING;
 }
 
+VOID
+EFIAPI
+CpuEject(
+  IN UINTN ProcessorNum
+  )
+{
+  //
+  // APIC ID is UINT32, but mCpuHotEjectData->ApicIdMap[] is UINT64
+  // so use UINT64 throughout.
+  //
+  UINT64 ApicId;
+
+  ApicId = mCpuHotEjectData->ApicIdMap[ProcessorNum];
+  if (ApicId == CPU_EJECT_INVALID) {
+    return;
+  }
+
+  //
+  // CPU(s) being unplugged get here from SmmCpuFeaturesSmiRendezvousExit()
+  // after having been cleared to exit the SMI by the monarch and thus have
+  // no SMM processing remaining.
+  //
+  // Given that we cannot allow them to escape to the guest, we pen them
+  // here until the SMM monarch tells the HW to unplug them.
+  //
+  CpuDeadLoop ();
+}
+
 /**
   CPU Hot-unplug MMI handler function.
 
@@ -203,6 +233,7 @@ UnplugCpus(
 {
   EFI_STATUS Status = EFI_SUCCESS;
   UINT32     ToUnplugIdx;
+  UINT32     EjectCount = 0;
   UINTN      ProcessorNum;
 
   ToUnplugIdx = 0;
@@ -242,11 +273,35 @@ UnplugCpus(
       DEBUG ((DEBUG_ERROR, "%a: RemoveProcessor(" FMT_APIC_ID "): %r\n",
         __FUNCTION__, RemoveApicId, Status));
       goto Fatal;
+    } else {
+      //
+      // Stash the APIC IDs so we can do the actual unplug later.
+      //
+      if (mCpuHotEjectData->ApicIdMap[ProcessorNum] != CPU_EJECT_INVALID) {
+        //
+        // Since ProcessorNum and APIC-ID map 1-1, so a valid
+        // mCpuHotEjectData->ApicIdMap[ProcessorNum] means something
+        // is horribly wrong.
+        //
+        DEBUG ((DEBUG_ERROR, "%a: ProcessorNum %u maps to %llx, cannot map to " FMT_APIC_ID "\n",
+          __FUNCTION__, ProcessorNum, mCpuHotEjectData->ApicIdMap[ProcessorNum], RemoveApicId));
+        goto Fatal;
+      }
+
+      mCpuHotEjectData->ApicIdMap[ProcessorNum] = (UINT64)RemoveApicId;
+      EjectCount++;
     }
 
     ToUnplugIdx++;
   }
 
+  if (EjectCount) {
+    //
+    // We have processors to be ejected; install the handler.
+    //
+    mCpuHotEjectData->Handler = CpuEject;
+  }
+
   //
   // We've handled this unplug.
   //
@@ -445,7 +500,13 @@ CpuHotplugEntry (
   // Our DEPEX on EFI_SMM_CPU_SERVICE_PROTOCOL guarantees that PiSmmCpuDxeSmm
   // has pointed PcdCpuHotPlugDataAddress to CPU_HOT_PLUG_DATA in SMRAM.
   //
+  // Additionally, CPU Hot-unplug is available only if CPU Hotplug is, so
+  // the same DEPEX also guarantees that PcdCpuHotEjectDataAddress points
+  // to CPU_HOT_EJECT_DATA in SMRAM.
+  //
   mCpuHotPlugData = (VOID *)(UINTN)PcdGet64 (PcdCpuHotPlugDataAddress);
+  mCpuHotEjectData = (VOID *)(UINTN)PcdGet64 (PcdCpuHotEjectDataAddress);
+
   if (mCpuHotPlugData == NULL) {
     Status = EFI_NOT_FOUND;
     DEBUG ((DEBUG_ERROR, "%a: CPU_HOT_PLUG_DATA: %r\n", __FUNCTION__, Status));
@@ -457,6 +518,9 @@ CpuHotplugEntry (
   if (mCpuHotPlugData->ArrayLength == 1) {
     return EFI_UNSUPPORTED;
   }
+  ASSERT (mCpuHotEjectData &&
+          (mCpuHotPlugData->ArrayLength == mCpuHotEjectData->ArrayLength));
+
   //
   // Allocate the data structures that depend on the possible CPU count.
   //
@@ -539,6 +603,24 @@ CpuHotplugEntry (
   //
   SmbaseInstallFirstSmiHandler ();
 
+  if (mCpuHotEjectData) {
+  UINT32     Idx;
+    //
+    // For CPU ejection we need to map ProcessorNum -> APIC_ID. By the time
+    // we do that, however, the Processor's APIC ID has already been removed
+    // from SMM data structures. So we will use mCpuHotEjectData->ApicIdMap
+    // to map from ProcessorNum -> APIC_ID.
+    //
+    for (Idx = 0; Idx < mCpuHotEjectData->ArrayLength; Idx++) {
+      mCpuHotEjectData->ApicIdMap[Idx] = CPU_EJECT_INVALID;
+    }
+
+    //
+    // Wait to init the handler until an ejection is warranted
+    //
+    mCpuHotEjectData->Handler = NULL;
+  }
+
   return EFI_SUCCESS;
 
 ReleasePostSmmPen:
-- 
2.9.3

[PATCH v4 8/9] OvmfPkg/CpuHotplugSmm: add worker to do CPU ejection

[Ankur Arora]

Designate a worker CPU (we use the one executing the root MMI handler),
which will do the actual ejection via QEMU in CpuEject().

CpuEject(), on the worker CPU, ejects each marked CPU by first
selecting its APIC ID and then sending the QEMU "eject" command.
QEMU in-turn signals the remote VCPU thread which context-switches
it out of the SMI.

CpuEject(), on the CPU being ejected, spins around in its holding
area until this final context-switch. This does mean that there is
some CPU state that would ordinarily be restored (in SmiRendezvous()
and in SmiEntry.nasm::CommonHandler), but will not be anymore.
This unrestored state includes FPU state, CET enable, stuffing of
RSB and the final RSM. Since the CPU state is destroyed by QEMU,
this should be okay.

Cc: Laszlo Ersek <lersek@redhat.com>
Cc: Jordan Justen <jordan.l.justen@intel.com>
Cc: Ard Biesheuvel <ard.biesheuvel@arm.com>
Cc: Igor Mammedov <imammedo@redhat.com>
Cc: Boris Ostrovsky <boris.ostrovsky@oracle.com>
Cc: Aaron Young <aaron.young@oracle.com>
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3132
Signed-off-by: Ankur Arora <ankur.a.arora@oracle.com>
---
 OvmfPkg/CpuHotplugSmm/CpuHotplug.c | 66 +++++++++++++++++++++++++++++++++++---
 1 file changed, 61 insertions(+), 5 deletions(-)

diff --git a/OvmfPkg/CpuHotplugSmm/CpuHotplug.c b/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
index e8ba9ae59e69..27fd982d6771 100644
--- a/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
+++ b/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
@@ -200,6 +200,54 @@ CpuEject(
     return;
   }
 
+  if (ApicId == CPU_EJECT_WORKER) {
+    UINT32 CpuIndex;
+
+    for (CpuIndex = 0; CpuIndex < mCpuHotEjectData->ArrayLength; CpuIndex++) {
+      UINT64 RemoveApicId = mCpuHotEjectData->ApicIdMap[CpuIndex];
+
+      if ((RemoveApicId != CPU_EJECT_INVALID &&
+           RemoveApicId != CPU_EJECT_WORKER)) {
+        //
+        // This to-be-ejected-CPU has already received the BSP's SMI exit
+        // signal and, will execute SmmCpuFeaturesSmiRendezvousExit()
+        // followed by this callback or is already waiting in the
+        // CpuDeadLoop() below.
+        //
+        // Tell QEMU to context-switch it out.
+        //
+        QemuCpuhpWriteCpuSelector (mMmCpuIo, RemoveApicId);
+        QemuCpuhpWriteCpuStatus (mMmCpuIo, QEMU_CPUHP_STAT_EJECTED);
+
+        //
+        // Compiler barrier to ensure the next store isn't reordered
+        //
+        MemoryFence();
+
+        //
+        // Clear the eject status for CpuIndex to ensure that an invalid
+        // SMI later does not end up trying to eject it or a newly
+        // hotplugged CpuIndex does not go into the dead loop.
+        //
+        mCpuHotEjectData->ApicIdMap[CpuIndex] = CPU_EJECT_INVALID;
+
+        DEBUG ((DEBUG_INFO, "%a: Unplugged CPU %u -> " FMT_APIC_ID "\n",
+               __FUNCTION__, CpuIndex, RemoveApicId));
+      }
+    }
+
+    //
+    // Clear our own worker status.
+    //
+    mCpuHotEjectData->ApicIdMap[ProcessorNum] = CPU_EJECT_INVALID;
+
+    //
+    // We are done until the next hot-unplug; clear the handler.
+    //
+    mCpuHotEjectData->Handler = NULL;
+    return;
+  }
+
   //
   // CPU(s) being unplugged get here from SmmCpuFeaturesSmiRendezvousExit()
   // after having been cleared to exit the SMI by the monarch and thus have
@@ -296,6 +344,19 @@ UnplugCpus(
   }
 
   if (EjectCount) {
+    UINTN  Worker;
+
+    Status = mMmCpuService->WhoAmI(mMmCpuService, &Worker);
+    ASSERT_EFI_ERROR(Status);
+    //
+    // UnplugCpus() is called via the root MMI handler and thus we are
+    // executing in the BSP context.
+    //
+    // Mark ourselves as the worker CPU.
+    //
+    ASSERT (mCpuHotEjectData->ApicIdMap[Worker] == CPU_EJECT_INVALID);
+    mCpuHotEjectData->ApicIdMap[Worker] = CPU_EJECT_WORKER;
+
     //
     // We have processors to be ejected; install the handler.
     //
@@ -419,11 +480,6 @@ CpuHotplugMmi (
   if (EFI_ERROR (Status)) {
     goto Fatal;
   }
-  if (ToUnplugCount > 0) {
-    DEBUG ((DEBUG_ERROR, "%a: hot-unplug is not supported yet\n",
-      __FUNCTION__));
-    goto Fatal;
-  }
 
   if (PluggedCount > 0) {
     Status = PlugCpus(mPluggedApicIds, PluggedCount);
-- 
2.9.3

[PATCH v4 9/9] OvmfPkg/SmmControl2Dxe: negotiate CPU hot-unplug

[Ankur Arora]

As part of the negotiation treat ICH9_LPC_SMI_F_CPU_HOT_UNPLUG as a
subfeature of feature flag ICH9_LPC_SMI_F_CPU_HOTPLUG, so enable it
only if the other is also being negotiated.

Cc: Laszlo Ersek <lersek@redhat.com>
Cc: Jordan Justen <jordan.l.justen@intel.com>
Cc: Ard Biesheuvel <ard.biesheuvel@arm.com>
Cc: Igor Mammedov <imammedo@redhat.com>
Cc: Boris Ostrovsky <boris.ostrovsky@oracle.com>
Cc: Aaron Young <aaron.young@oracle.com>
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3132
Signed-off-by: Ankur Arora <ankur.a.arora@oracle.com>
---
 OvmfPkg/SmmControl2Dxe/SmiFeatures.c | 25 ++++++++++++++++++++++---
 1 file changed, 22 insertions(+), 3 deletions(-)

diff --git a/OvmfPkg/SmmControl2Dxe/SmiFeatures.c b/OvmfPkg/SmmControl2Dxe/SmiFeatures.c
index c9d875543205..00185253a820 100644
--- a/OvmfPkg/SmmControl2Dxe/SmiFeatures.c
+++ b/OvmfPkg/SmmControl2Dxe/SmiFeatures.c
@@ -29,6 +29,13 @@
 //
 #define ICH9_LPC_SMI_F_CPU_HOTPLUG BIT1
 
+// The following bit value stands for "enable CPU hot unplug, and inject an SMI
+// with control value ICH9_APM_CNT_CPU_HOT_UNPLUG upon hot unplug", in the
+// "etc/smi/supported-features" and "etc/smi/requested-features" fw_cfg files.
+// Can only be negotiated alongside ICH9_LPC_SMI_F_CPU_HOTPLUG.
+//
+#define ICH9_LPC_SMI_F_CPU_HOT_UNPLUG BIT2
+
 //
 // Provides a scratch buffer (allocated in EfiReservedMemoryType type memory)
 // for the S3 boot script fragment to write to and read from.
@@ -112,7 +119,8 @@ NegotiateSmiFeatures (
   QemuFwCfgReadBytes (sizeof mSmiFeatures, &mSmiFeatures);
 
   //
-  // We want broadcast SMI, SMI on CPU hotplug, and nothing else.
+  // We want broadcast SMI, SMI on CPU hotplug, on CPU hot-unplug
+  // and nothing else.
   //
   RequestedFeaturesMask = ICH9_LPC_SMI_F_BROADCAST;
   if (!MemEncryptSevIsEnabled ()) {
@@ -120,8 +128,18 @@ NegotiateSmiFeatures (
     // For now, we only support hotplug with SEV disabled.
     //
     RequestedFeaturesMask |= ICH9_LPC_SMI_F_CPU_HOTPLUG;
+    RequestedFeaturesMask |= ICH9_LPC_SMI_F_CPU_HOT_UNPLUG;
   }
   mSmiFeatures &= RequestedFeaturesMask;
+
+  if (!(mSmiFeatures & ICH9_LPC_SMI_F_CPU_HOTPLUG) &&
+      (mSmiFeatures & ICH9_LPC_SMI_F_CPU_HOT_UNPLUG)) {
+    DEBUG ((DEBUG_WARN, "%a CPU host-features %Lx, requested mask %Lx\n",
+      __FUNCTION__, mSmiFeatures, RequestedFeaturesMask));
+
+    mSmiFeatures &= ~ICH9_LPC_SMI_F_CPU_HOT_UNPLUG;
+  }
+
   QemuFwCfgSelectItem (mRequestedFeaturesItem);
   QemuFwCfgWriteBytes (sizeof mSmiFeatures, &mSmiFeatures);
 
@@ -162,8 +180,9 @@ NegotiateSmiFeatures (
   if ((mSmiFeatures & ICH9_LPC_SMI_F_CPU_HOTPLUG) == 0) {
     DEBUG ((DEBUG_INFO, "%a: CPU hotplug not negotiated\n", __FUNCTION__));
   } else {
-    DEBUG ((DEBUG_INFO, "%a: CPU hotplug with SMI negotiated\n",
-      __FUNCTION__));
+    DEBUG ((DEBUG_INFO, "%a: CPU hotplug%s with SMI negotiated\n",
+      __FUNCTION__,
+      (mSmiFeatures & ICH9_LPC_SMI_F_CPU_HOT_UNPLUG) ? ", unplug" : ""));
   }
 
   //
-- 
2.9.3

Re: [edk2-devel] [PATCH v4 0/9] support CPU hot-unplug

[Laszlo Ersek]

On 01/18/21 07:34, Ankur Arora wrote:
> Hi,
> 
> This series adds support for CPU hot-unplug with OVMF.
> 
> Please see this in conjunction with the QEMU secureboot hot-unplug v2
> series posted here (now upstreamed):
>   https://lore.kernel.org/qemu-devel/20201207140739.3829993-1-imammedo@redhat.com/
> 
> Patches 1 and 3,
>   ("OvmfPkg/CpuHotplugSmm: refactor hotplug logic")
>   ("OvmfPkg/CpuHotplugSmm: add Qemu Cpu Status helper")
> are either refactors or add support functions.
> 
>   OvmfPkg/CpuHotplugSmm: introduce UnplugCpus()
>   OvmfPkg/CpuHotplugSmm: add CpuEject()
>   OvmfPkg/CpuHotplugSmm: add worker to do CPU ejection
> 
> Patch 2 and 9,
>   ("OvmfPkg/CpuHotplugSmm: collect hot-unplug events")
>   ("OvmfPkg/SmmControl2Dxe: negotiate CPU hot-unplug")
> handle the QEMU protocol logic for collection of CPU hot-unplug events
> or the protocol negotiation.
> 
> Patch 4,
>   ("OvmfPkg/CpuHotplugSmm: introduce UnplugCpus()")
> adds the MMI logic for CPU hot-unplug handling and informing
> the PiSmmCpuDxeSmm of CPU removal.
> 
> Patches 5 and 6,
>   ("OvmfPkg/CpuHotplugSmm: define CPU_HOT_EJECT_DATA")
>   ("OvmfPkg/SmmCpuFeaturesLib: init CPU ejection state")
> sets up state for doing the CPU ejection as part of hot-unplug.
> 
> Patches 7, and 8,
>   ("OvmfPkg/CpuHotplugSmm: introduce UnplugCpus()")
>   ("OvmfPkg/CpuHotplugSmm: add worker to do CPU ejection")
> add the CPU ejection logic.
> 
> Testing (with QEMU 5.2.50):
>  - Stable with randomized CPU plug/unplug (guest maxcpus=1,8,128)
>  - Synthetic tests with simultaneous multi CPU hot-unplug
>  - Negotiation with/without CPU hotplug enabled
> 
> Also at:
>   github.com/terminus/edk2/ hot-unplug-v4
> 
> Changelog:
> v4:
>   - Gets rid of unnecessary UefiCpuPkg changes
> 
> v3:
>   - Use a saner PCD based interface to share state between PiSmmCpuDxeSmm
>     and OvmfPkg/CpuHotplugSmm
>   - Cleaner split of the hot-unplug code
>   URL: https://patchew.org/EDK2/20210115074533.277448-1-ankur.a.arora@oracle.com/
> 
> v2:
>   - Do the ejection via SmmCpuFeaturesRendezvousExit()
>   URL: https://patchew.org/EDK2/20210107195515.106158-1-ankur.a.arora@oracle.com/
> 
> RFC:
>   URL: https://patchew.org/EDK2/20201208053432.2690694-1-ankur.a.arora@oracle.com/
> 
> 
> Please review.

I've got this series in my review queue (confirming).

I'd like to finish review on the
<https://bugzilla.tianocore.org/show_bug.cgi?id=3059> series first,
since that's what I've got mostly in my mind at this point.

I hope to start reviewing the unplug series in a few days.

Thanks
Laszlo

> 
> Thanks
> Ankur
> 
> Ankur Arora (9):
>   OvmfPkg/CpuHotplugSmm: refactor hotplug logic
>   OvmfPkg/CpuHotplugSmm: collect hot-unplug events
>   OvmfPkg/CpuHotplugSmm: add Qemu Cpu Status helper
>   OvmfPkg/CpuHotplugSmm: introduce UnplugCpus()
>   OvmfPkg/CpuHotplugSmm: define CPU_HOT_EJECT_DATA
>   OvmfPkg/SmmCpuFeaturesLib: init CPU ejection state
>   OvmfPkg/CpuHotplugSmm: add CpuEject()
>   OvmfPkg/CpuHotplugSmm: add worker to do CPU ejection
>   OvmfPkg/SmmControl2Dxe: negotiate CPU hot-unplug
> 
>  OvmfPkg/CpuHotplugSmm/CpuHotplug.c                 | 435 ++++++++++++++++-----
>  OvmfPkg/CpuHotplugSmm/CpuHotplugSmm.inf            |   1 +
>  OvmfPkg/CpuHotplugSmm/QemuCpuhp.c                  |  58 ++-
>  OvmfPkg/CpuHotplugSmm/QemuCpuhp.h                  |   6 +
>  OvmfPkg/Include/IndustryStandard/QemuCpuHotplug.h  |   2 +
>  OvmfPkg/Include/Library/CpuHotEjectData.h          |  31 ++
>  .../Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.c  |  62 +++
>  .../SmmCpuFeaturesLib/SmmCpuFeaturesLib.inf        |   3 +
>  OvmfPkg/OvmfPkg.dec                                |   6 +
>  OvmfPkg/SmmControl2Dxe/SmiFeatures.c               |  25 +-
>  10 files changed, 527 insertions(+), 102 deletions(-)
>  create mode 100644 OvmfPkg/Include/Library/CpuHotEjectData.h
> 

Re: [edk2-devel] [PATCH v4 0/9] support CPU hot-unplug

[Ankur Arora]

On 2021-01-18 9:09 a.m., Laszlo Ersek wrote:
> On 01/18/21 07:34, Ankur Arora wrote:
>> Hi,
>>
>> This series adds support for CPU hot-unplug with OVMF.
>>
>> Please see this in conjunction with the QEMU secureboot hot-unplug v2
>> series posted here (now upstreamed):
>>    https://lore.kernel.org/qemu-devel/20201207140739.3829993-1-imammedo@redhat.com/
>>
>> Patches 1 and 3,
>>    ("OvmfPkg/CpuHotplugSmm: refactor hotplug logic")
>>    ("OvmfPkg/CpuHotplugSmm: add Qemu Cpu Status helper")
>> are either refactors or add support functions.
>>
>>    OvmfPkg/CpuHotplugSmm: introduce UnplugCpus()
>>    OvmfPkg/CpuHotplugSmm: add CpuEject()
>>    OvmfPkg/CpuHotplugSmm: add worker to do CPU ejection
>>
>> Patch 2 and 9,
>>    ("OvmfPkg/CpuHotplugSmm: collect hot-unplug events")
>>    ("OvmfPkg/SmmControl2Dxe: negotiate CPU hot-unplug")
>> handle the QEMU protocol logic for collection of CPU hot-unplug events
>> or the protocol negotiation.
>>
>> Patch 4,
>>    ("OvmfPkg/CpuHotplugSmm: introduce UnplugCpus()")
>> adds the MMI logic for CPU hot-unplug handling and informing
>> the PiSmmCpuDxeSmm of CPU removal.
>>
>> Patches 5 and 6,
>>    ("OvmfPkg/CpuHotplugSmm: define CPU_HOT_EJECT_DATA")
>>    ("OvmfPkg/SmmCpuFeaturesLib: init CPU ejection state")
>> sets up state for doing the CPU ejection as part of hot-unplug.
>>
>> Patches 7, and 8,
>>    ("OvmfPkg/CpuHotplugSmm: introduce UnplugCpus()")
>>    ("OvmfPkg/CpuHotplugSmm: add worker to do CPU ejection")
>> add the CPU ejection logic.
>>
>> Testing (with QEMU 5.2.50):
>>   - Stable with randomized CPU plug/unplug (guest maxcpus=1,8,128)
>>   - Synthetic tests with simultaneous multi CPU hot-unplug
>>   - Negotiation with/without CPU hotplug enabled
>>
>> Also at:
>>    github.com/terminus/edk2/ hot-unplug-v4
>>
>> Changelog:
>> v4:
>>    - Gets rid of unnecessary UefiCpuPkg changes
>>
>> v3:
>>    - Use a saner PCD based interface to share state between PiSmmCpuDxeSmm
>>      and OvmfPkg/CpuHotplugSmm
>>    - Cleaner split of the hot-unplug code
>>    URL: https://patchew.org/EDK2/20210115074533.277448-1-ankur.a.arora@oracle.com/
>>
>> v2:
>>    - Do the ejection via SmmCpuFeaturesRendezvousExit()
>>    URL: https://patchew.org/EDK2/20210107195515.106158-1-ankur.a.arora@oracle.com/
>>
>> RFC:
>>    URL: https://patchew.org/EDK2/20201208053432.2690694-1-ankur.a.arora@oracle.com/
>>
>>
>> Please review.
> 
> I've got this series in my review queue (confirming).
> 
> I'd like to finish review on the
> <https://bugzilla.tianocore.org/show_bug.cgi?id=3059> series first,
> since that's what I've got mostly in my mind at this point.
> 
> I hope to start reviewing the unplug series in a few days.

Sounds good. Thanks.

Ankur

> 
> Thanks
> Laszlo
> 
>>
>> Thanks
>> Ankur
>>
>> Ankur Arora (9):
>>    OvmfPkg/CpuHotplugSmm: refactor hotplug logic
>>    OvmfPkg/CpuHotplugSmm: collect hot-unplug events
>>    OvmfPkg/CpuHotplugSmm: add Qemu Cpu Status helper
>>    OvmfPkg/CpuHotplugSmm: introduce UnplugCpus()
>>    OvmfPkg/CpuHotplugSmm: define CPU_HOT_EJECT_DATA
>>    OvmfPkg/SmmCpuFeaturesLib: init CPU ejection state
>>    OvmfPkg/CpuHotplugSmm: add CpuEject()
>>    OvmfPkg/CpuHotplugSmm: add worker to do CPU ejection
>>    OvmfPkg/SmmControl2Dxe: negotiate CPU hot-unplug
>>
>>   OvmfPkg/CpuHotplugSmm/CpuHotplug.c                 | 435 ++++++++++++++++-----
>>   OvmfPkg/CpuHotplugSmm/CpuHotplugSmm.inf            |   1 +
>>   OvmfPkg/CpuHotplugSmm/QemuCpuhp.c                  |  58 ++-
>>   OvmfPkg/CpuHotplugSmm/QemuCpuhp.h                  |   6 +
>>   OvmfPkg/Include/IndustryStandard/QemuCpuHotplug.h  |   2 +
>>   OvmfPkg/Include/Library/CpuHotEjectData.h          |  31 ++
>>   .../Library/SmmCpuFeaturesLib/SmmCpuFeaturesLib.c  |  62 +++
>>   .../SmmCpuFeaturesLib/SmmCpuFeaturesLib.inf        |   3 +
>>   OvmfPkg/OvmfPkg.dec                                |   6 +
>>   OvmfPkg/SmmControl2Dxe/SmiFeatures.c               |  25 +-
>>   10 files changed, 527 insertions(+), 102 deletions(-)
>>   create mode 100644 OvmfPkg/Include/Library/CpuHotEjectData.h
>>
> 

Re: [edk2-devel] [PATCH v4 0/9] support CPU hot-unplug

[Laszlo Ersek]

Hi Ankur,

On 01/18/21 19:35, Ankur Arora wrote:
> On 2021-01-18 9:09 a.m., Laszlo Ersek wrote:
>> On 01/18/21 07:34, Ankur Arora wrote:
>>> Hi,
>>>
>>> This series adds support for CPU hot-unplug with OVMF.
>>>
>>> Please see this in conjunction with the QEMU secureboot hot-unplug v2
>>> series posted here (now upstreamed):
>>>   
>>> https://lore.kernel.org/qemu-devel/20201207140739.3829993-1-imammedo@redhat.com/
>>>
>>>
>>> Patches 1 and 3,
>>>    ("OvmfPkg/CpuHotplugSmm: refactor hotplug logic")
>>>    ("OvmfPkg/CpuHotplugSmm: add Qemu Cpu Status helper")
>>> are either refactors or add support functions.
>>>
>>>    OvmfPkg/CpuHotplugSmm: introduce UnplugCpus()
>>>    OvmfPkg/CpuHotplugSmm: add CpuEject()
>>>    OvmfPkg/CpuHotplugSmm: add worker to do CPU ejection
>>>
>>> Patch 2 and 9,
>>>    ("OvmfPkg/CpuHotplugSmm: collect hot-unplug events")
>>>    ("OvmfPkg/SmmControl2Dxe: negotiate CPU hot-unplug")
>>> handle the QEMU protocol logic for collection of CPU hot-unplug events
>>> or the protocol negotiation.
>>>
>>> Patch 4,
>>>    ("OvmfPkg/CpuHotplugSmm: introduce UnplugCpus()")
>>> adds the MMI logic for CPU hot-unplug handling and informing
>>> the PiSmmCpuDxeSmm of CPU removal.
>>>
>>> Patches 5 and 6,
>>>    ("OvmfPkg/CpuHotplugSmm: define CPU_HOT_EJECT_DATA")
>>>    ("OvmfPkg/SmmCpuFeaturesLib: init CPU ejection state")
>>> sets up state for doing the CPU ejection as part of hot-unplug.
>>>
>>> Patches 7, and 8,
>>>    ("OvmfPkg/CpuHotplugSmm: introduce UnplugCpus()")
>>>    ("OvmfPkg/CpuHotplugSmm: add worker to do CPU ejection")
>>> add the CPU ejection logic.
>>>
>>> Testing (with QEMU 5.2.50):
>>>   - Stable with randomized CPU plug/unplug (guest maxcpus=1,8,128)
>>>   - Synthetic tests with simultaneous multi CPU hot-unplug
>>>   - Negotiation with/without CPU hotplug enabled
>>>
>>> Also at:
>>>    github.com/terminus/edk2/ hot-unplug-v4
>>>
>>> Changelog:
>>> v4:
>>>    - Gets rid of unnecessary UefiCpuPkg changes
>>>
>>> v3:
>>>    - Use a saner PCD based interface to share state between
>>> PiSmmCpuDxeSmm
>>>      and OvmfPkg/CpuHotplugSmm
>>>    - Cleaner split of the hot-unplug code
>>>    URL:
>>> https://patchew.org/EDK2/20210115074533.277448-1-ankur.a.arora@oracle.com/
>>>
>>>
>>> v2:
>>>    - Do the ejection via SmmCpuFeaturesRendezvousExit()
>>>    URL:
>>> https://patchew.org/EDK2/20210107195515.106158-1-ankur.a.arora@oracle.com/
>>>
>>>
>>> RFC:
>>>    URL:
>>> https://patchew.org/EDK2/20201208053432.2690694-1-ankur.a.arora@oracle.com/
>>>
>>>
>>>
>>> Please review.
>>
>> I've got this series in my review queue (confirming).
>>
>> I'd like to finish review on the
>> <https://bugzilla.tianocore.org/show_bug.cgi?id=3059> series first,
>> since that's what I've got mostly in my mind at this point.
>>
>> I hope to start reviewing the unplug series in a few days.
> 
> Sounds good. Thanks.

I apologize for coming back with more formalities.

The patches are somewhat incorrectly composed. Looking at the very first
patch email for example, the quoted-printable encoding of the email shows:

- a hard \r in each context line of the patch (encoded as "=0D"),
- no \r character in any newly added line of the patch.

This *inconsistency* is a problem -- once I apply the patch with git-am,
it creates files with mixed LF / CRLF line terminators.

Every source file (new files in particular) should be purely CRLF.

Please update your editor's settings to stick with the line terminator
style of the file that's being edited. (When creating new files, you may
have to switch to CRLF explicitly.)

Furthermore, please convert all of the patches to purely CRLF as follows:
- run a git-rebase with "edit" actions
- at each stage, determine the set of files updated/created by the commit
- run "unix2dos" on all those files
- squash the result into the commit
- continue

(You could script this too, using the "exec" git-rebase action, but
doing it manually could be tolerable as well.)

Now, of course I can do this myself with the patches, on my end, but
(assuming at least one more version of the patch set is going to be
necessary), fixing your settings and the patches both, for future
manipulation, would now be timely.

... For reviewing non-trivial patch series, I tend to apply them first
on a local branch (nothing beats the power of the full git toolkit
during a review); that's how I found this.


For catching such issues early on, please run "PatchCheck.py" before
posting, e.g. as in:

$ python3 BaseTools/Scripts/PatchCheck.py master..topic_branch

(In the present case, PatchCheck reports lots of "not CRLF" errors.)


An even better idea is to push your topic branch (which you're about to
post to the list) to your edk2 fork on github.com, and then submit a
pull request against the "tianocore/edk2" project. The pull request will
be auto-closed in the end (it will never be merged), but the goal is to
trigger a CI run on the patch set, and to give you the error messages if
any. PatchCheck runs as a part of CI, too.

(github.com PRs are used for actual merging by edk2 maintainers, but for
that, they set the "push" label on the subject PRs, and the "push" label
is restricted to maintainers.)

I apologize about the extra delay. Would you be OK posting v5?

Thanks!
Laszlo

Re: [edk2-devel] [PATCH v4 0/9] support CPU hot-unplug

[Ankur Arora]

On 2021-01-21 4:29 a.m., Laszlo Ersek wrote:
> Hi Ankur,
> 
> On 01/18/21 19:35, Ankur Arora wrote:
>> On 2021-01-18 9:09 a.m., Laszlo Ersek wrote:
>>> On 01/18/21 07:34, Ankur Arora wrote:
>>>> Hi,
>>>>
>>>> This series adds support for CPU hot-unplug with OVMF.
>>>>
>>>> Please see this in conjunction with the QEMU secureboot hot-unplug v2
>>>> series posted here (now upstreamed):
>>>>    
>>>> https://lore.kernel.org/qemu-devel/20201207140739.3829993-1-imammedo@redhat.com/
>>>>
>>>>
>>>> Patches 1 and 3,
>>>>     ("OvmfPkg/CpuHotplugSmm: refactor hotplug logic")
>>>>     ("OvmfPkg/CpuHotplugSmm: add Qemu Cpu Status helper")
>>>> are either refactors or add support functions.
>>>>
>>>>     OvmfPkg/CpuHotplugSmm: introduce UnplugCpus()
>>>>     OvmfPkg/CpuHotplugSmm: add CpuEject()
>>>>     OvmfPkg/CpuHotplugSmm: add worker to do CPU ejection
>>>>
>>>> Patch 2 and 9,
>>>>     ("OvmfPkg/CpuHotplugSmm: collect hot-unplug events")
>>>>     ("OvmfPkg/SmmControl2Dxe: negotiate CPU hot-unplug")
>>>> handle the QEMU protocol logic for collection of CPU hot-unplug events
>>>> or the protocol negotiation.
>>>>
>>>> Patch 4,
>>>>     ("OvmfPkg/CpuHotplugSmm: introduce UnplugCpus()")
>>>> adds the MMI logic for CPU hot-unplug handling and informing
>>>> the PiSmmCpuDxeSmm of CPU removal.
>>>>
>>>> Patches 5 and 6,
>>>>     ("OvmfPkg/CpuHotplugSmm: define CPU_HOT_EJECT_DATA")
>>>>     ("OvmfPkg/SmmCpuFeaturesLib: init CPU ejection state")
>>>> sets up state for doing the CPU ejection as part of hot-unplug.
>>>>
>>>> Patches 7, and 8,
>>>>     ("OvmfPkg/CpuHotplugSmm: introduce UnplugCpus()")
>>>>     ("OvmfPkg/CpuHotplugSmm: add worker to do CPU ejection")
>>>> add the CPU ejection logic.
>>>>
>>>> Testing (with QEMU 5.2.50):
>>>>    - Stable with randomized CPU plug/unplug (guest maxcpus=1,8,128)
>>>>    - Synthetic tests with simultaneous multi CPU hot-unplug
>>>>    - Negotiation with/without CPU hotplug enabled
>>>>
>>>> Also at:
>>>>     github.com/terminus/edk2/ hot-unplug-v4
>>>>
>>>> Changelog:
>>>> v4:
>>>>     - Gets rid of unnecessary UefiCpuPkg changes
>>>>
>>>> v3:
>>>>     - Use a saner PCD based interface to share state between
>>>> PiSmmCpuDxeSmm
>>>>       and OvmfPkg/CpuHotplugSmm
>>>>     - Cleaner split of the hot-unplug code
>>>>     URL:
>>>> https://patchew.org/EDK2/20210115074533.277448-1-ankur.a.arora@oracle.com/
>>>>
>>>>
>>>> v2:
>>>>     - Do the ejection via SmmCpuFeaturesRendezvousExit()
>>>>     URL:
>>>> https://patchew.org/EDK2/20210107195515.106158-1-ankur.a.arora@oracle.com/
>>>>
>>>>
>>>> RFC:
>>>>     URL:
>>>> https://patchew.org/EDK2/20201208053432.2690694-1-ankur.a.arora@oracle.com/
>>>>
>>>>
>>>>
>>>> Please review.
>>>
>>> I've got this series in my review queue (confirming).
>>>
>>> I'd like to finish review on the
>>> <https://bugzilla.tianocore.org/show_bug.cgi?id=3059> series first,
>>> since that's what I've got mostly in my mind at this point.
>>>
>>> I hope to start reviewing the unplug series in a few days.
>>
>> Sounds good. Thanks.
> 
> I apologize for coming back with more formalities.
> 
> The patches are somewhat incorrectly composed. Looking at the very first
> patch email for example, the quoted-printable encoding of the email shows:
> 
> - a hard \r in each context line of the patch (encoded as "=0D"),
> - no \r character in any newly added line of the patch.
> 
> This *inconsistency* is a problem -- once I apply the patch with git-am,
> it creates files with mixed LF / CRLF line terminators.
> 
> Every source file (new files in particular) should be purely CRLF.
> 
> Please update your editor's settings to stick with the line terminator
> style of the file that's being edited. (When creating new files, you may
> have to switch to CRLF explicitly.)
> 
> Furthermore, please convert all of the patches to purely CRLF as follows:
> - run a git-rebase with "edit" actions
> - at each stage, determine the set of files updated/created by the commit
> - run "unix2dos" on all those files
> - squash the result into the commit
> - continue
> 
> (You could script this too, using the "exec" git-rebase action, but
> doing it manually could be tolerable as well.)
> 
> Now, of course I can do this myself with the patches, on my end, but
> (assuming at least one more version of the patch set is going to be
> necessary), fixing your settings and the patches both, for future
> manipulation, would now be timely.
> 
> ... For reviewing non-trivial patch series, I tend to apply them first
> on a local branch (nothing beats the power of the full git toolkit
> during a review); that's how I found this.
> 
> 
> For catching such issues early on, please run "PatchCheck.py" before
> posting, e.g. as in:
> 
> $ python3 BaseTools/Scripts/PatchCheck.py master..topic_branch
> 
> (In the present case, PatchCheck reports lots of "not CRLF" errors.)
> 

PatchCheck.py is great. Thanks.

> 
> An even better idea is to push your topic branch (which you're about to
> post to the list) to your edk2 fork on github.com, and then submit a
> pull request against the "tianocore/edk2" project. The pull request will
> be auto-closed in the end (it will never be merged), but the goal is to
> trigger a CI run on the patch set, and to give you the error messages if
> any. PatchCheck runs as a part of CI, too.
> 
> (github.com PRs are used for actual merging by edk2 maintainers, but for
> that, they set the "push" label on the subject PRs, and the "push" label
> is restricted to maintainers.)
> 
> I apologize about the extra delay. Would you be OK posting v5?

Sure. Just a side question which I should have asked you earlier: are
the coding standards listed somewhere?
I had looked for answers to similar questions but the "TianoCore C style
guide" doesn't mention either PatchCheck.py or rules around CRLF.

(Though now that I do look at it, the CRLF rules are listed in the
"EDK II C Coding Standards_2.1 Draft.PDF".)

Thanks for the comments and my apologies for making you review all of
these non-substantive changes.


Ankur

> 
> Thanks!
> Laszlo
> 

Re: [edk2-devel] [PATCH v4 0/9] support CPU hot-unplug

[Laszlo Ersek]

On 01/21/21 20:11, Ankur Arora wrote:
> On 2021-01-21 4:29 a.m., Laszlo Ersek wrote:

>> For catching such issues early on, please run "PatchCheck.py" before
>> posting, e.g. as in:
>>
>> $ python3 BaseTools/Scripts/PatchCheck.py master..topic_branch
>>
>> (In the present case, PatchCheck reports lots of "not CRLF" errors.)
>>
> 
> PatchCheck.py is great. Thanks.
> 
>>
>> An even better idea is to push your topic branch (which you're about to
>> post to the list) to your edk2 fork on github.com, and then submit a
>> pull request against the "tianocore/edk2" project. The pull request will
>> be auto-closed in the end (it will never be merged), but the goal is to
>> trigger a CI run on the patch set, and to give you the error messages if
>> any. PatchCheck runs as a part of CI, too.
>>
>> (github.com PRs are used for actual merging by edk2 maintainers, but for
>> that, they set the "push" label on the subject PRs, and the "push" label
>> is restricted to maintainers.)
>>
>> I apologize about the extra delay. Would you be OK posting v5?
> 
> Sure.

Thanks for your understanding, I'm relieved.

> Just a side question which I should have asked you earlier: are
> the coding standards listed somewhere?

Yes, see "C Coding Standards" at
<https://github.com/tianocore/tianocore.github.io/wiki/EDK-II-Documentation#specifications>.

Although...

> I had looked for answers to similar questions but the "TianoCore C style
> guide" doesn't mention either PatchCheck.py or rules around CRLF.

... you may have found exactly that, already. Unfortunately, like all
documentation in the world, this document is somewhat out of date.

There is also a tool, called ECC, that contains a builtin C language
parser, that enforces various style requirements. It's built into the CI
system and runs on github -- however, we have it disabled for OVMF, in
commit ef3e73c6a0c0 ("OvmfPkg: Disable EccCheck CI until EccCheck issues
are fixed", 2020-12-14). There are two reasons for that:

- Some stuff that ECC enforces is just overzealous -- in some cases, we
relax the coding style under OvmfPkg specifically, but ECC cannot easily
be configured to tolerate those relaxations (for example, you cannot
configure it to ignore a particular error code *regardless* of where it
occurs under OvmfPkg).

- In some cases, ECC rejects C language constructs that are both valid
and conformant to the coding style (i.e., ECC has its own bugs).


So we usually "teach" these quirks to contributors during review. It's
quite a lot of investment, which is part of why we hope that
contributors stick around.


An alternative is this: in your topic branch that you intend to post to
edk2-devel, include an *extra* patch (not to be posted to the list) that
*reverts* commit ef3e73c6a0c0. And submit a github.com PR with this
variant of your topic branch. As a result, ECC will be unleashed on (the
end-stage of) your patch series. If you fix all of the ECC issues, then
reviewers should make next to zero *style* complaints -- on the other
hand, you could be updating code that ECC complains about *incorrectly*
(due to an ECC bug, or because the subject style rule is something we
deem superfluous, or at least "possible to bend", under OvmfPkg).

So perhaps try to investigate this latter avenue, and see if you are
willing to deal with what ECC throws at you. :)

> Thanks for the comments and my apologies for making you review all of
> these non-substantive changes.

It's a standard part of reviewing the first few contributions of a new
community member; it's too bad we don't have better stuff in place. We
do have docs but they are slightly outdated, and we do have ECC but with
warts.

We also have recommendations (not official requirements!) on setting up
git, as a part of
<https://github.com/tianocore/tianocore.github.io/wiki/Laszlo's-unkempt-git-guide-for-edk2-contributors-and-maintainers>
-- which is likely out of date somewhat --; the

  BaseTools/Scripts/SetupGit.py

utility automates more or less the same settings.

We also have

  BaseTools/Scripts/GetMaintainer.py

which lets you determine the Cc:'s you should put in each commit message...

And when you dump all this *process* on a new contributor, they run away
screaming -- kind of justifiedly. :/

Thanks
Laszlo

Re: [edk2-devel] [PATCH v4 0/9] support CPU hot-unplug

[Ankur Arora]

On 2021-01-21 11:51 a.m., Laszlo Ersek wrote:
> On 01/21/21 20:11, Ankur Arora wrote:
>> On 2021-01-21 4:29 a.m., Laszlo Ersek wrote:
> 
>>> For catching such issues early on, please run "PatchCheck.py" before
>>> posting, e.g. as in:
>>>
>>> $ python3 BaseTools/Scripts/PatchCheck.py master..topic_branch
>>>
>>> (In the present case, PatchCheck reports lots of "not CRLF" errors.)
>>>
>>
>> PatchCheck.py is great. Thanks.
>>
>>>
>>> An even better idea is to push your topic branch (which you're about to
>>> post to the list) to your edk2 fork on github.com, and then submit a
>>> pull request against the "tianocore/edk2" project. The pull request will
>>> be auto-closed in the end (it will never be merged), but the goal is to
>>> trigger a CI run on the patch set, and to give you the error messages if
>>> any. PatchCheck runs as a part of CI, too.
>>>
>>> (github.com PRs are used for actual merging by edk2 maintainers, but for
>>> that, they set the "push" label on the subject PRs, and the "push" label
>>> is restricted to maintainers.)
>>>
>>> I apologize about the extra delay. Would you be OK posting v5?
>>
>> Sure.
> 
> Thanks for your understanding, I'm relieved.
> 
>> Just a side question which I should have asked you earlier: are
>> the coding standards listed somewhere?
> 
> Yes, see "C Coding Standards" at
> <https://github.com/tianocore/tianocore.github.io/wiki/EDK-II-Documentation#specifications>.
> 
> Although...
> 
>> I had looked for answers to similar questions but the "TianoCore C style
>> guide" doesn't mention either PatchCheck.py or rules around CRLF.
> 
> ... you may have found exactly that, already. Unfortunately, like all
> documentation in the world, this document is somewhat out of date.
> 
> There is also a tool, called ECC, that contains a builtin C language
> parser, that enforces various style requirements. It's built into the CI
> system and runs on github -- however, we have it disabled for OVMF, in
> commit ef3e73c6a0c0 ("OvmfPkg: Disable EccCheck CI until EccCheck issues
> are fixed", 2020-12-14). There are two reasons for that:
> 
> - Some stuff that ECC enforces is just overzealous -- in some cases, we
> relax the coding style under OvmfPkg specifically, but ECC cannot easily
> be configured to tolerate those relaxations (for example, you cannot
> configure it to ignore a particular error code *regardless* of where it
> occurs under OvmfPkg).
> 
> - In some cases, ECC rejects C language constructs that are both valid
> and conformant to the coding style (i.e., ECC has its own bugs).
> 
> 
> So we usually "teach" these quirks to contributors during review. It's
> quite a lot of investment, which is part of why we hope that
> contributors stick around.

A very understandable sentiment :).

> 
> An alternative is this: in your topic branch that you intend to post to
> edk2-devel, include an *extra* patch (not to be posted to the list) that
> *reverts* commit ef3e73c6a0c0. And submit a github.com PR with this
> variant of your topic branch. As a result, ECC will be unleashed on (the
> end-stage of) your patch series. If you fix all of the ECC issues, then
> reviewers should make next to zero *style* complaints -- on the other
> hand, you could be updating code that ECC complains about *incorrectly*
> (due to an ECC bug, or because the subject style rule is something we
> deem superfluous, or at least "possible to bend", under OvmfPkg).
> 
> So perhaps try to investigate this latter avenue, and see if you are
> willing to deal with what ECC throws at you. :)

Can't guarantee that I'll see it through but my curiosity on what ECC
will throw at me is piqued. Will try.

> 
>> Thanks for the comments and my apologies for making you review all of
>> these non-substantive changes.
> 
> It's a standard part of reviewing the first few contributions of a new
> community member; it's too bad we don't have better stuff in place. We
> do have docs but they are slightly outdated, and we do have ECC but with
> warts.
> 
> We also have recommendations (not official requirements!) on setting up
> git, as a part of
> <https://github.com/tianocore/tianocore.github.io/wiki/Laszlo's-unkempt-git-guide-for-edk2-contributors-and-maintainers>
> -- which is likely out of date somewhat --; the

I guess that's the nature of non-executable bits... out of date
soon as they are written. Thanks, this and the scripts below both
were pretty useful.

>    BaseTools/Scripts/SetupGit.py
> 
> utility automates more or less the same settings.
> 
> We also have
> 
>    BaseTools/Scripts/GetMaintainer.py
> 
> which lets you determine the Cc:'s you should put in each commit message...
> 
> And when you dump all this *process* on a new contributor, they run away
> screaming -- kind of justifiedly. :/

Heh, I think I see the appeal of this gradual introduction of process.

Personally speaking, it is after working on this feature and seeing
things  like SafeUintnMult(), that you see that FW generally and EDK2
specifically really does need to be written differently from, for
instance the kernel.

Thanks
Ankur
  
> 
> Thanks
> Laszlo
> 

Re: [edk2-devel] [PATCH v4 0/9] support CPU hot-unplug

[Laszlo Ersek]

On 01/22/21 07:32, Ankur Arora wrote:
> On 2021-01-21 11:51 a.m., Laszlo Ersek wrote:

>> So perhaps try to investigate this latter avenue, and see if you are
>> willing to deal with what ECC throws at you. :)
> 
> Can't guarantee that I'll see it through but my curiosity on what ECC
> will throw at me is piqued. Will try.

Thanks!
Laszlo