Re: [edk2-devel] [PATCH 00/16] OvmfPkg: support VCPU hotplug with -D SMM_REQUIRE

["Wu, Jiaxin" <jiaxin.wu@intel.com>]

Hi Laszlo,

Appreciate your feedback! Thank you very much.

Jiaxin

> -----Original Message-----
> From: Laszlo Ersek <lersek@redhat.com>
> Sent: Saturday, July 25, 2020 12:02 AM
> To: Wu, Jiaxin <jiaxin.wu@intel.com>
> Cc: devel@edk2.groups.io
> Subject: Re: [edk2-devel] [PATCH 00/16] OvmfPkg: support VCPU hotplug
> with -D SMM_REQUIRE
> 
> On 07/24/20 08:26, Wu, Jiaxin wrote:
> > Hi Laszlo,
> >
> > Looks OVMF supports the CPU hotplug with those series patches.
> >
> > Could you provide some guide how to enable the OVMF CPU hotplug
> > verification? Is there any general work flow introduction how it
> > works? For example, how to do the hot add CPU initialization (e.g.
> > Register setting / Microcode update, etc.)? I'm very interested in
> > this feature on OVMF.
> 
> Long version:
> -------------
> 
> (1) There are three pieces missing:
> 
> (1a) The QEMU side changes for the ACPI (DSDT) content that QEMU
> generates for the OS.
> 
> The ACPI GPE handler for CPU hotplug is being modified by my colleague
> Igor Mammedov to raise the SMI (command value 4) on CPU hotplug.
> 
> For developing the OVMF series for TianoCore#1512 (which is now merged),
> I used a prototype QEMU patch, from Igor. But that patch is not suitable
> for upstreaming to QEMU. SO Igor is now developing the real patches for
> QEMU's ACPI generator.
> 
> (1b) The related feature negotiation patches in QEMU.
> 
> In order for "CPU hotplug with SMM" to work, both OVMF and QEMU need
> to
> perform specific things. In order to deal with cross-version
> compatibility problems, the "CPU hotplug with SMI" feature is
> dynamically negotiated between OVMF and QEMU. For this negotiation,
> both
> QEMU and OVMF need additional patches. These patches are not related to
> the actual plugging activities; instead they control whether plugging is
> permitted at all, or not.
> 
> Igor's QEMU series covers both purposes (1a) and (1b). It's work in
> progress. The first posting was an RFC series:
> 
> (1b1) [RFC 0/3] x86: fix cpu hotplug with secure boot
>       https://lists.gnu.org/archive/html/qemu-devel/2020-07/msg03746.html
>       http://mid.mail-archive.com/20200710161704.309824-1-
> imammedo@redhat.com
> 
> The latest posting has been a PATCH series:
> 
> (1b2) [qemu-devel] [PATCH 0/6] x86: fix cpu hotplug with secure boot
>       https://lists.gnu.org/archive/html/qemu-devel/2020-07/msg05850.html
>       http://mid.mail-archive.com/20200720141610.574308-1-
> imammedo@redhat.com
> 
> (1c) The feature negotiation patch for OVMF is here:
> 
> * [edk2-devel] [PATCH] OvmfPkg/SmmControl2Dxe: negotiate
> ICH9_LPC_SMI_F_CPU_HOTPLUG
>   https://edk2.groups.io/g/devel/message/62561
>   http://mid.mail-archive.com/20200714184305.9814-1-lersek@redhat.com
> 
> 
> (2) Special register setting and microcode stuff are not needed.
> 
> 
> (3) As I mentioned before, I strongly suggest using QEMU and OVMF with
> libvirt. I had written an article about that here:
> 
>   https://github.com/tianocore/tianocore.github.io/wiki/Testing-SMM-with-
> QEMU,-KVM-and-libvirt
> 
> I wrote this article specifically for "Windows-based" developers. The
> article is written from such a perspective that you don't need a
> personal Linux workstation, only a single Linux workstation *per team*.
> So you can continue using a Windows workstation, just set up one Linux
> box for your team (if you don't yet have one).
> 
> This article remains relevant.
> 
> (3a) In order to set up a guest for VCPU hotplug, simply go through the
> article, initially.
> 
> (3b) Once you're done with that, power down the guest, and modify the
> domain XML as follows:
> 
>   virsh edit <DOMAIN_NAME>
> 
> (3b1) replace the "pc-q35-2.9" machine type with "pc-q35-5.1"
> 
> (3b2) replace the following stanza:
> 
>   <vcpu placement='static'>4</vcpu>
> 
> with:
> 
>   <vcpu placement='static' current='2'>4</vcpu>
>   <vcpus>
>     <vcpu id='0' enabled='yes' hotpluggable='no'/>
>     <vcpu id='1' enabled='no' hotpluggable='yes'/>
>     <vcpu id='2' enabled='yes' hotpluggable='yes'/>
>     <vcpu id='3' enabled='no' hotpluggable='yes'/>
>   </vcpus>
> 
> This will create a VCPU topology where:
> 
> - CPU#0 is present up-front, and is not hot-pluggable (this is a QEMU
> requirement),
> 
> - CPU#1, CPU#2, and CPU#3 are hot-pluggable,
> 
> - CPU#2 is present up-front ("cold-plugged"), while CPU#1 and CPU#3 are
> absent initially.
> 
> 
> (4) Boot the guest. Once you have a root prompt in the guest, you can
> use one of two libvirt commands for hot-plugging a CPU:
> 
> (4a) the singular "virsh setvcpu" command:
> 
>   virsh setvcpu <DOMAIN_NAME> <PROCESSOR_ID> --enable --live
> 
> where you can pass in 1 or 3 for <PROCESSOR_ID>.
> 
> This command lets you specify the precise ID of the processor to be
> hot-plugged; IOW, the command lets you control topology.
> 
> (4b) the plural "virsh setvcpus" command:
> 
>   virsh setvcpus <GUEST_NAME> <PROCESSOR_COUNT> --live
> 
> This command lets you specify the desired number of total active CPUs.
> It does not let you control topology. (My understanding is that it keeps
> the topology populated at the "front".)
> 
> Regarding the current QEMU status, we need to do more work for
> supporting (4b). The RFC series (1b1) enables (4a) to work. The PATCH
> series (1b2) intended to make (4b) work, but unfortunately it broke even
> (4a). So now we need at least one more version of the QEMU series (I've
> given my feedback to Igor already, on qemu-devel).
> 
> (4c) Dependent on guest OS configuration, you might have to manually
> online the newly plugged CPUs in the guest:
> 
>   echo 1 > /sys/devices/system/cpu/cpu2/online
>   echo 1 > /sys/devices/system/cpu/cpu3/online
> 
> Note that the "cpuN" identifiers seen here are *neither* APIC IDs *nor*
> the same IDs as seen in the libvirt domain XML. Instead, these IDs are
> assigned in the order the Linux kernel learns about the CPUs (if I
> understand correctly).
> 
> 
> Short version:
> --------------
> 
> - apply (1b1) on top of latest QEMU master from git, and build and
> install it,
> 
> - apply (1c) on latest edk2, and build OVMF with "-D SMM_REQUIRE",
> 
> - install a Linux guest on a Linux host (using KVM!) as described in my
> Wiki article (3),
> 
> - modify the domain XML for the guest as described in (3b),
> 
> - use the singular "virsh setvcpu" command (4a) for hot-plugging VCPU#1
> and/or VCPU#3,
> 
> - if necessary, use (4c) in the guest.
> 
> 
> You can do the same with Windows Server guests too, although I'm not
> exactly sure what versions support CPU hotplug. For testing I've used
> Windows Server 2012 R2. The Wiki article at (3) has a section dedicated
> to installing Windows guests too.
> 
> Thanks,
> Laszlo