[PATCH v3 08/10] OvmfPkg/CpuHotplugSmm: add worker to do CPU ejection

["Ankur Arora" <ankur.a.arora@oracle.com>]

Designate a worker CPU (we use the one executing the root MMI handler),
which will do the final CPU ejection. This happens via CpuEject().

On the worker CPU, CpuEject() calls QEMU to do the ejection for each CPU
that is unplugged. QEMU handles this by signalling the remote VCPU thread
which forces the SMI AP to context switch out of the SMI, ending with its
QEMU state destroyed.

On the AP, CpuEject() spins around in its holding area until the
context-switch happens. Given that the context switch would end up with
the AP state being cleaned up, this means that the AP will never
return to finish the SMI handling, and thus would not restore some
of the CPU state that it ordinarily would (in SmiRendezvous() and in
SmiEntry.nasm::CommonHandler).

This unrestored state includes FPU state, CET enable, stuffing of
RSB and the final RSM. Given that the CPU state is destroyed by
QEMU on unplug, this should be okay.

Cc: Laszlo Ersek <lersek@redhat.com>
Cc: Jordan Justen <jordan.l.justen@intel.com>
Cc: Ard Biesheuvel <ard.biesheuvel@arm.com>
Cc: Igor Mammedov <imammedo@redhat.com>
Cc: Boris Ostrovsky <boris.ostrovsky@oracle.com>
Cc: Aaron Young <aaron.young@oracle.com>
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3132
Signed-off-by: Ankur Arora <ankur.a.arora@oracle.com>
---
 OvmfPkg/CpuHotplugSmm/CpuHotplug.c | 61 ++++++++++++++++++++++++++++++++++----
 1 file changed, 56 insertions(+), 5 deletions(-)

diff --git a/OvmfPkg/CpuHotplugSmm/CpuHotplug.c b/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
index 4048490783e4..8aa52ebe5dd1 100644
--- a/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
+++ b/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
@@ -266,6 +266,20 @@ UnplugCpus(
     ToUnplugIdx++;
   }
 
+  if (EjectCount) {
+    UINTN  Worker;
+    Status = mMmCpuService->WhoAmI(mMmCpuService, &Worker);
+    ASSERT_EFI_ERROR(Status);
+    //
+    // UnplugCpus() is called via the root MMI handler and thus we are in the
+    // BSP context. Accordingly, mark ourselves as the ejecting CPU.
+    // Note that, the QEMU eject protocol does not specify that only the BSP
+    // can do the ejection, so this should be safe on any CPU (that is not itself
+    // being unplugged.)
+    //
+    mCpuHotEjectData->ApicIdMap[Worker] = CPU_EJECT_WORKER;
+  }
+
   //
   // We've handled this unplug.
   //
@@ -383,11 +397,6 @@ CpuHotplugMmi (
   if (EFI_ERROR (Status)) {
     goto Fatal;
   }
-  if (ToUnplugCount > 0) {
-    DEBUG ((DEBUG_ERROR, "%a: hot-unplug is not supported yet\n",
-      __FUNCTION__));
-    goto Fatal;
-  }
 
   if (PluggedCount > 0) {
     Status = PlugCpus(mPluggedApicIds, PluggedCount);
@@ -430,6 +439,48 @@ CpuEject(
     return;
   }
 
+  if (ApicId == CPU_EJECT_WORKER) {
+    UINT32 CpuIndex;
+    for (CpuIndex = 0; CpuIndex < mCpuHotEjectData->ArrayLength; CpuIndex++) {
+      UINT64 RemoveApicId = mCpuHotEjectData->ApicIdMap[CpuIndex];
+
+      if ((RemoveApicId != CPU_EJECT_INVALID && RemoveApicId != CPU_EJECT_WORKER)) {
+
+        //
+        // The CPUs to be unplugged have received the BSP's signal to exit the
+        // SMI and either will execute SmmCpuFeaturesSmiRendezvousExit()
+        // followed by this callback or are already waiting in the CpuDeadLoop()
+        // below.
+        //
+        // Tell QEMU to put them out of their misery.
+        //
+         QemuCpuhpWriteCpuSelector (mMmCpuIo, RemoveApicId);
+         QemuCpuhpWriteCpuStatus (mMmCpuIo, QEMU_CPUHP_STAT_EJECTED);
+
+         //
+         // Compiler memory barrier to ensure the next store isn't reordered
+         //
+         MemoryFence();
+
+        // Clear the unplug status for CpuIndex to ensure that an invalid SMI
+        // later does not end up trying to unplug it or the newly hotplugged
+        // CpuIndex does not go into the dead loop.
+        //
+        mCpuHotEjectData->ApicIdMap[CpuIndex] = CPU_EJECT_INVALID;
+
+        DEBUG ((DEBUG_INFO, "%a: Unplugged CPU " FMT_APIC_ID "\n",
+               __FUNCTION__, RemoveApicId));
+       }
+     }
+
+    //
+    // Clear our own CPU status to ensure that we don't needlessly enter
+    // the this loop on the next SMI.
+    //
+    mCpuHotEjectData->ApicIdMap[ProcessorNum] = CPU_EJECT_INVALID;
+    return;
+   }
+
   //
   // CPU(s) being unplugged get here from SmmCpuFeaturesSmiRendezvousExit()
   // after having been cleared to exit the SMI by the monarch and thus have
-- 
2.9.3