Re: [edk2-devel] [PATCH 00/16] OvmfPkg: support VCPU hotplug with -D SMM_REQUIRE

["Laszlo Ersek" <lersek@redhat.com>]

On 07/24/20 08:26, Wu, Jiaxin wrote:
> Hi Laszlo,
>
> Looks OVMF supports the CPU hotplug with those series patches.
>
> Could you provide some guide how to enable the OVMF CPU hotplug
> verification? Is there any general work flow introduction how it
> works? For example, how to do the hot add CPU initialization (e.g.
> Register setting / Microcode update, etc.)? I'm very interested in
> this feature on OVMF.

Long version:
-------------

(1) There are three pieces missing:

(1a) The QEMU side changes for the ACPI (DSDT) content that QEMU
generates for the OS.

The ACPI GPE handler for CPU hotplug is being modified by my colleague
Igor Mammedov to raise the SMI (command value 4) on CPU hotplug.

For developing the OVMF series for TianoCore#1512 (which is now merged),
I used a prototype QEMU patch, from Igor. But that patch is not suitable
for upstreaming to QEMU. SO Igor is now developing the real patches for
QEMU's ACPI generator.

(1b) The related feature negotiation patches in QEMU.

In order for "CPU hotplug with SMM" to work, both OVMF and QEMU need to
perform specific things. In order to deal with cross-version
compatibility problems, the "CPU hotplug with SMI" feature is
dynamically negotiated between OVMF and QEMU. For this negotiation, both
QEMU and OVMF need additional patches. These patches are not related to
the actual plugging activities; instead they control whether plugging is
permitted at all, or not.

Igor's QEMU series covers both purposes (1a) and (1b). It's work in
progress. The first posting was an RFC series:

(1b1) [RFC 0/3] x86: fix cpu hotplug with secure boot
      https://lists.gnu.org/archive/html/qemu-devel/2020-07/msg03746.html
      http://mid.mail-archive.com/20200710161704.309824-1-imammedo@redhat.com

The latest posting has been a PATCH series:

(1b2) [qemu-devel] [PATCH 0/6] x86: fix cpu hotplug with secure boot
      https://lists.gnu.org/archive/html/qemu-devel/2020-07/msg05850.html
      http://mid.mail-archive.com/20200720141610.574308-1-imammedo@redhat.com

(1c) The feature negotiation patch for OVMF is here:

* [edk2-devel] [PATCH] OvmfPkg/SmmControl2Dxe: negotiate ICH9_LPC_SMI_F_CPU_HOTPLUG
  https://edk2.groups.io/g/devel/message/62561
  http://mid.mail-archive.com/20200714184305.9814-1-lersek@redhat.com


(2) Special register setting and microcode stuff are not needed.


(3) As I mentioned before, I strongly suggest using QEMU and OVMF with
libvirt. I had written an article about that here:

  https://github.com/tianocore/tianocore.github.io/wiki/Testing-SMM-with-QEMU,-KVM-and-libvirt

I wrote this article specifically for "Windows-based" developers. The
article is written from such a perspective that you don't need a
personal Linux workstation, only a single Linux workstation *per team*.
So you can continue using a Windows workstation, just set up one Linux
box for your team (if you don't yet have one).

This article remains relevant.

(3a) In order to set up a guest for VCPU hotplug, simply go through the
article, initially.

(3b) Once you're done with that, power down the guest, and modify the
domain XML as follows:

  virsh edit <DOMAIN_NAME>

(3b1) replace the "pc-q35-2.9" machine type with "pc-q35-5.1"

(3b2) replace the following stanza:

  <vcpu placement='static'>4</vcpu>

with:

  <vcpu placement='static' current='2'>4</vcpu>
  <vcpus>
    <vcpu id='0' enabled='yes' hotpluggable='no'/>
    <vcpu id='1' enabled='no' hotpluggable='yes'/>
    <vcpu id='2' enabled='yes' hotpluggable='yes'/>
    <vcpu id='3' enabled='no' hotpluggable='yes'/>
  </vcpus>

This will create a VCPU topology where:

- CPU#0 is present up-front, and is not hot-pluggable (this is a QEMU
requirement),

- CPU#1, CPU#2, and CPU#3 are hot-pluggable,

- CPU#2 is present up-front ("cold-plugged"), while CPU#1 and CPU#3 are
absent initially.


(4) Boot the guest. Once you have a root prompt in the guest, you can
use one of two libvirt commands for hot-plugging a CPU:

(4a) the singular "virsh setvcpu" command:

  virsh setvcpu <DOMAIN_NAME> <PROCESSOR_ID> --enable --live

where you can pass in 1 or 3 for <PROCESSOR_ID>.

This command lets you specify the precise ID of the processor to be
hot-plugged; IOW, the command lets you control topology.

(4b) the plural "virsh setvcpus" command:

  virsh setvcpus <GUEST_NAME> <PROCESSOR_COUNT> --live

This command lets you specify the desired number of total active CPUs.
It does not let you control topology. (My understanding is that it keeps
the topology populated at the "front".)

Regarding the current QEMU status, we need to do more work for
supporting (4b). The RFC series (1b1) enables (4a) to work. The PATCH
series (1b2) intended to make (4b) work, but unfortunately it broke even
(4a). So now we need at least one more version of the QEMU series (I've
given my feedback to Igor already, on qemu-devel).

(4c) Dependent on guest OS configuration, you might have to manually
online the newly plugged CPUs in the guest:

  echo 1 > /sys/devices/system/cpu/cpu2/online
  echo 1 > /sys/devices/system/cpu/cpu3/online

Note that the "cpuN" identifiers seen here are *neither* APIC IDs *nor*
the same IDs as seen in the libvirt domain XML. Instead, these IDs are
assigned in the order the Linux kernel learns about the CPUs (if I
understand correctly).


Short version:
--------------

- apply (1b1) on top of latest QEMU master from git, and build and
install it,

- apply (1c) on latest edk2, and build OVMF with "-D SMM_REQUIRE",

- install a Linux guest on a Linux host (using KVM!) as described in my
Wiki article (3),

- modify the domain XML for the guest as described in (3b),

- use the singular "virsh setvcpu" command (4a) for hot-plugging VCPU#1
and/or VCPU#3,

- if necessary, use (4c) in the guest.


You can do the same with Windows Server guests too, although I'm not
exactly sure what versions support CPU hotplug. For testing I've used
Windows Server 2012 R2. The Wiki article at (3) has a section dedicated
to installing Windows guests too.

Thanks,
Laszlo