[PATCH v2 08/10] OvmfCpuPkg/CpuHotplug: add a hot-unplug handler called at SMI exit

["Ankur Arora" <ankur.a.arora@oracle.com>]

Add CpuUnplugExitWork(), to be called from SmmCpuFeaturesRendezvousExit()
to do the final ejection as part of CPU hot-unplug.

On the BSP, CpuUnplugExitWork() calls QEMU to do the ejection for each
CPU that is unplugged. QEMU handles this by signalling the remote VCPU
thread which forces the SMI AP to context switch out of the SMI and
with its QEMU state destroyed.

On the AP, CpuUnplugExitWork() provides a holding area where the CPU
spins until context switched out by QEMU via the BSP. Given that
the context switch would end up with the AP state being cleaned up,
this means that the AP CPU will never return to finish the SMI
handling, and thus would not restore some of the CPU state that it
ordinarily would (in SmiRendezvous() and in
SmiEntry.nasm::CommonHandler).

This unrestored state includes FPU state, CET enable, stuffing of
RSB and the final RSM. Given that the CPU state is destroyed by
QEMU on unplug, this should be okay.

Cc: Laszlo Ersek <lersek@redhat.com>
Cc: Jordan Justen <jordan.l.justen@intel.com>
Cc: Ard Biesheuvel <ard.biesheuvel@arm.com>
Cc: Eric Dong <eric.dong@intel.com>
Cc: Ray Ni <ray.ni@intel.com>
Cc: Rahul Kumar <rahul1.kumar@intel.com>
Cc: Igor Mammedov <imammedo@redhat.com>
Cc: Boris Ostrovsky <boris.ostrovsky@oracle.com>
Cc: Aaron Young <aaron.young@oracle.com>
Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=3132
Signed-off-by: Ankur Arora <ankur.a.arora@oracle.com>
---
 OvmfPkg/CpuHotplugSmm/CpuHotplug.c    | 68 +++++++++++++++++++++++++++++++++++
 UefiCpuPkg/PiSmmCpuDxeSmm/MpService.c |  6 ++++
 2 files changed, 74 insertions(+)

diff --git a/OvmfPkg/CpuHotplugSmm/CpuHotplug.c b/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
index 20d92a35da39..379c9a66f261 100644
--- a/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
+++ b/OvmfPkg/CpuHotplugSmm/CpuHotplug.c
@@ -421,6 +421,69 @@ Fatal:
   return EFI_INTERRUPT_PENDING;
 }
 
+EFI_STATUS
+EFIAPI
+CpuUnplugExitWork(
+  IN UINTN CpuIndex,
+  IN BOOLEAN IsBSP
+  )
+{
+  APIC_ID RemoveApicId;
+
+  RemoveApicId = mHotUnplugWork[CpuIndex];
+
+  if (!IsBSP && RemoveApicId == MAX_UINT32) {
+    return EFI_SUCCESS;
+  }
+
+  if (IsBSP) {
+    UINT32 Idx;
+    for (Idx = 0; Idx < mCpuHotPlugData->ArrayLength; Idx++) {
+      RemoveApicId = mHotUnplugWork[Idx];
+
+      if (RemoveApicId != MAX_UINT32) {
+	//
+	// The CPU(s) to be unplugged have received the BSP's signal to exit the
+	// SMI and either will execute SmmCpuFeaturesSmiRendezvousExit() and this
+	// callback or are waiting here.
+	//
+	// Tell HW to put it out of its misery.
+	//
+        QemuCpuhpWriteCpuSelector (mMmCpuIo, RemoveApicId);
+        QemuCpuhpWriteCpuStatus (mMmCpuIo, QEMU_CPUHP_STAT_EJECTED);
+
+	//
+	// Barrier to ensure that the compiler doesn't reorder the next store
+	//
+	MemoryFence();
+
+	//
+	// Clear the unplug status to make sure that an invalid SMI later
+	// does not try to do an unplug or go to the dead loop.
+	//
+	mHotUnplugWork[Idx] = MAX_UINT32;
+
+        DEBUG ((DEBUG_INFO, "%a: Unplugged CPU " FMT_APIC_ID "\n",
+		__FUNCTION__, RemoveApicId));
+      }
+    }
+    return EFI_SUCCESS;
+  }
+  
+  //
+  // CPU(s) being unplugged get here from SmmCpuFeaturesSmiRendezvousExit()
+  // after having been cleared to exit the SMI by the monarch and thus have
+  // no SMM processing remaining.
+  //
+  // Given that we cannot allow them to escape to the guest, we pen them
+  // here until the SMM monarch tells the HW to unplug them.
+  //
+
+  CpuDeadLoop ();
+
+  return EFI_ABORTED;
+}
+
 
 //
 // Entry point function of this driver.
@@ -573,6 +636,11 @@ CpuHotplugEntry (
   }
 
   //
+  // Register handler for hot-unplugging an AP.
+  //
+  MmRegisterShutdownInterface(CpuUnplugExitWork);
+
+  //
   // Register the handler for the CPU Hotplug MMI.
   //
   Status = gMmst->MmiHandlerRegister (
diff --git a/UefiCpuPkg/PiSmmCpuDxeSmm/MpService.c b/UefiCpuPkg/PiSmmCpuDxeSmm/MpService.c
index fb6aab17de37..f246d730d1e2 100644
--- a/UefiCpuPkg/PiSmmCpuDxeSmm/MpService.c
+++ b/UefiCpuPkg/PiSmmCpuDxeSmm/MpService.c
@@ -1727,6 +1727,12 @@ SmiRendezvous (
     }
   }
 
+  //
+  // Note that the BSP will unplug any CPUs that have been marked for
+  // hot-unplug at any point after it sets AllCpusInSync = FALSE
+  // so it cannot depend on an AP executing code post that point.
+  //
+
 Exit:
   SmmCpuFeaturesRendezvousExit (CpuIndex, IsBsp);
 
-- 
2.9.3