[PATCH 00/16] OvmfPkg: support VCPU hotplug with -D SMM_REQUIRE

["Laszlo Ersek" <lersek@redhat.com>]

Bugzilla: https://bugzilla.tianocore.org/show_bug.cgi?id=1512
Repo:     https://github.com/lersek/edk2.git
Branch:   vcpu_hotplug_smm_bz_1512

This series implements VCPU hotplug with SMM for OVMF, i.e., when OVMF
is built with "-D SMM_REQUIRE".

SEV support and hot-unplug support are out of scope for now.

Patch#13 ("OvmfPkg/CpuHotplugSmm: complete root MMI handler for CPU
hotplug") describes tests and results in the Notes section.

Obviously this is not being proposed for the edk2-stable202002 tag
(which is in hard feature freeze).

QEMU needs patches for this feature, too. I've done the development
against a QEMU patch that Igor hacked up quickly at my request. It was
never posted (it needs some polish for upstreaming), but it has allowed
me to write and test this feature.

The key parts of the QEMU commit message are:

> x68:acpi: trigger SMI before scanning for added/removed CPUs
>
> Firmware should only scan for new CPUs and not modify events in CPU
> hotplug registers.

Igor's patch is based on upstream QEMU commit 418fa86dd465. Until he
decides to post or otherwise share the patch, its effect can be
expressed with a diff, taken in the Linux guest, between decompiled
before/after versions of the QEMU-generated DSDT:

> @@ -81,6 +81,27 @@
>                  Return (Arg3)
>              }
>          }
> +
> +        Device (SMI0)
> +        {
> +            Name (_HID, "PNP0A06" /* Generic Container Device */)  // _HID: Hardware ID
> +            Name (_UID, "SMI resources")  // _UID: Unique ID
> +            Name (_STA, 0x0B)  // _STA: Status
> +            Name (_CRS, ResourceTemplate ()  // _CRS: Current Resource Settings
> +            {
> +                IO (Decode16,
> +                    0x00B2,             // Range Minimum
> +                    0x00B2,             // Range Maximum
> +                    0x01,               // Alignment
> +                    0x01,               // Length
> +                    )
> +            })
> +            OperationRegion (SMIR, SystemIO, 0xB2, One)
> +            Field (SMIR, ByteAcc, NoLock, WriteAsZeros)
> +            {
> +                SMIC,   8
> +            }
> +        }
>      }
>
>      Scope (_SB)
> @@ -3016,6 +3037,7 @@
>              Method (CSCN, 0, Serialized)
>              {
>                  Acquire (\_SB.PCI0.PRES.CPLK, 0xFFFF)
> +                \_SB.SMI0.SMIC = 0x04
>                  Local0 = One
>                  While ((Local0 == One))
>                  {

where the CSCN ("CPU scan") method is the _E02 GPE ("CPU hotplug") event
handler:

>      Method (\_GPE._E02, 0, NotSerialized)  // _Exx: Edge-Triggered GPE, xx=0x00-0xFF
>      {
>          \_SB.CPUS.CSCN ()
>      }

If you'd like to test this series, please ask Igor for the QEMU patch.
:)

The series has been formatted for review with the following options:

  --stat=1000 --stat-graph-width=20 \
  --unified=22 \
  --find-copies=43 --find-copies-harder \
  --base=master \

At every stage in the series:
- the tree builds,
- "PatchCheck.py" is happy,
- and OVMF works without regressions.

(Hotplug is made functional at patch#13, and "S3 after hotplug" is
completed at patch#16. So those actions should not be attempted before
said respective patches.)

Thanks,
Laszlo

Cc: Ard Biesheuvel <ard.biesheuvel@linaro.org>
Cc: Eric Dong <eric.dong@intel.com>
Cc: Hao A Wu <hao.a.wu@intel.com>
Cc: Igor Mammedov <imammedo@redhat.com>
Cc: Jian J Wang <jian.j.wang@intel.com>
Cc: Jiewen Yao <jiewen.yao@intel.com>
Cc: Jordan Justen <jordan.l.justen@intel.com>
Cc: Michael Kinney <michael.d.kinney@intel.com>
Cc: Philippe Mathieu-Daudé <philmd@redhat.com>
Cc: Ray Ni <ray.ni@intel.com>

Thanks
Laszlo

Laszlo Ersek (16):
  MdeModulePkg/PiSmmCore: log SMM image start failure
  UefiCpuPkg/PiSmmCpuDxeSmm: fix S3 Resume for CPU hotplug
  OvmfPkg: clone SmmCpuPlatformHookLib from UefiCpuPkg
  OvmfPkg: enable SMM Monarch Election in PiSmmCpuDxeSmm
  OvmfPkg: enable CPU hotplug support in PiSmmCpuDxeSmm
  OvmfPkg/CpuHotplugSmm: introduce skeleton for CPU Hotplug SMM driver
  OvmfPkg/CpuHotplugSmm: add hotplug register block helper functions
  OvmfPkg/CpuHotplugSmm: define the QEMU_CPUHP_CMD_GET_ARCH_ID macro
  OvmfPkg/CpuHotplugSmm: add function for collecting CPUs with events
  OvmfPkg/CpuHotplugSmm: collect CPUs with events
  OvmfPkg/CpuHotplugSmm: introduce Post-SMM Pen for hot-added CPUs
  OvmfPkg/CpuHotplugSmm: introduce First SMI Handler for hot-added CPUs
  OvmfPkg/CpuHotplugSmm: complete root MMI handler for CPU hotplug
  OvmfPkg: clone CpuS3DataDxe from UefiCpuPkg
  OvmfPkg/CpuS3DataDxe: superficial cleanups
  OvmfPkg/CpuS3DataDxe: enable S3 resume after CPU hotplug

 MdeModulePkg/Core/PiSmmCore/Dispatcher.c                                                                                                              |   6 +
 OvmfPkg/CpuHotplugSmm/ApicId.h                                                                                                                        |  23 ++
 OvmfPkg/CpuHotplugSmm/CpuHotplug.c                                                                                                                    | 426 ++++++++++++++++++++
 OvmfPkg/CpuHotplugSmm/CpuHotplugSmm.inf                                                                                                               |  64 +++
 OvmfPkg/CpuHotplugSmm/FirstSmiHandler.nasm                                                                                                            | 149 +++++++
 OvmfPkg/CpuHotplugSmm/FirstSmiHandlerContext.h                                                                                                        |  41 ++
 OvmfPkg/CpuHotplugSmm/PostSmmPen.nasm                                                                                                                 | 137 +++++++
 OvmfPkg/CpuHotplugSmm/QemuCpuhp.c                                                                                                                     | 301 ++++++++++++++
 OvmfPkg/CpuHotplugSmm/QemuCpuhp.h                                                                                                                     |  61 +++
 OvmfPkg/CpuHotplugSmm/Smbase.c                                                                                                                        | 252 ++++++++++++
 OvmfPkg/CpuHotplugSmm/Smbase.h                                                                                                                        |  46 +++
 OvmfPkg/Include/IndustryStandard/Q35MchIch9.h                                                                                                         |   5 +-
 OvmfPkg/Include/IndustryStandard/QemuCpuHotplug.h                                                                                                     |   3 +
 OvmfPkg/OvmfPkgIa32.dsc                                                                                                                               |   7 +-
 OvmfPkg/OvmfPkgIa32.fdf                                                                                                                               |   3 +-
 OvmfPkg/OvmfPkgIa32X64.dsc                                                                                                                            |   7 +-
 OvmfPkg/OvmfPkgIa32X64.fdf                                                                                                                            |   3 +-
 OvmfPkg/OvmfPkgX64.dsc                                                                                                                                |   7 +-
 OvmfPkg/OvmfPkgX64.fdf                                                                                                                                |   3 +-
 UefiCpuPkg/Library/SmmCpuPlatformHookLibNull/SmmCpuPlatformHookLibNull.c => OvmfPkg/Library/SmmCpuPlatformHookLibQemu/SmmCpuPlatformHookLibQemu.c     |  45 ++-
 UefiCpuPkg/Library/SmmCpuPlatformHookLibNull/SmmCpuPlatformHookLibNull.inf => OvmfPkg/Library/SmmCpuPlatformHookLibQemu/SmmCpuPlatformHookLibQemu.inf |  24 +-
 UefiCpuPkg/PiSmmCpuDxeSmm/CpuS3.c                                                                                                                     |  14 +-
 {UefiCpuPkg => OvmfPkg}/CpuS3DataDxe/CpuS3Data.c                                                                                                      |  99 +++--
 {UefiCpuPkg => OvmfPkg}/CpuS3DataDxe/CpuS3DataDxe.inf                                                                                                 |  30 +-
 24 files changed, 1667 insertions(+), 89 deletions(-)
 copy UefiCpuPkg/Library/SmmCpuPlatformHookLibNull/SmmCpuPlatformHookLibNull.c => OvmfPkg/Library/SmmCpuPlatformHookLibQemu/SmmCpuPlatformHookLibQemu.c (61%)
 copy UefiCpuPkg/Library/SmmCpuPlatformHookLibNull/SmmCpuPlatformHookLibNull.inf => OvmfPkg/Library/SmmCpuPlatformHookLibQemu/SmmCpuPlatformHookLibQemu.inf (43%)
 copy {UefiCpuPkg => OvmfPkg}/CpuS3DataDxe/CpuS3Data.c (77%)
 copy {UefiCpuPkg => OvmfPkg}/CpuS3DataDxe/CpuS3DataDxe.inf (69%)
 create mode 100644 OvmfPkg/CpuHotplugSmm/ApicId.h
 create mode 100644 OvmfPkg/CpuHotplugSmm/CpuHotplug.c
 create mode 100644 OvmfPkg/CpuHotplugSmm/CpuHotplugSmm.inf
 create mode 100644 OvmfPkg/CpuHotplugSmm/FirstSmiHandler.nasm
 create mode 100644 OvmfPkg/CpuHotplugSmm/FirstSmiHandlerContext.h
 create mode 100644 OvmfPkg/CpuHotplugSmm/PostSmmPen.nasm
 create mode 100644 OvmfPkg/CpuHotplugSmm/QemuCpuhp.c
 create mode 100644 OvmfPkg/CpuHotplugSmm/QemuCpuhp.h
 create mode 100644 OvmfPkg/CpuHotplugSmm/Smbase.c
 create mode 100644 OvmfPkg/CpuHotplugSmm/Smbase.h


base-commit: 1d3215fd24f47eaa4877542a59b4bbf5afc0cfe8
-- 
2.19.1.3.g30247aa5d201