From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from us-smtp-delivery-1.mimecast.com (us-smtp-delivery-1.mimecast.com [205.139.110.61]) by mx.groups.io with SMTP id smtpd.web10.9222.1595606508548818059 for ; Fri, 24 Jul 2020 09:01:48 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=h9I4EG9Z; spf=pass (domain: redhat.com, ip: 205.139.110.61, mailfrom: lersek@redhat.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1595606507; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=ZBpKwpXM71LT57J522ZgLq3sANB6HbqZ+fL2u/JyDsE=; b=h9I4EG9ZOEgajGTPW2oH2EkXhJtTeHoZcjj0haT8vw7vokXTqg9pAeHBvd0jV5G1TM30Eh MYw17vT0Y+q58lHoSFeTTR4nsofHqmC27Qs6/MzaUKr7NIaOqNZ0i3rCHbho1OTTt5i//U 8JTPgXseZSZfdmrOZefsmYOLk6lbtZ4= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-222-YPlba89NMqCOIczkieQFJw-1; Fri, 24 Jul 2020 12:01:40 -0400 X-MC-Unique: YPlba89NMqCOIczkieQFJw-1 Received: from smtp.corp.redhat.com (int-mx05.intmail.prod.int.phx2.redhat.com [10.5.11.15]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 942CA100CCC4; Fri, 24 Jul 2020 16:01:39 +0000 (UTC) Received: from lacos-laptop-7.usersys.redhat.com (ovpn-112-113.ams2.redhat.com [10.36.112.113]) by smtp.corp.redhat.com (Postfix) with ESMTP id D7FB95BAD5; Fri, 24 Jul 2020 16:01:38 +0000 (UTC) Subject: Re: [edk2-devel] [PATCH 00/16] OvmfPkg: support VCPU hotplug with -D SMM_REQUIRE To: "Wu, Jiaxin" References: <20200223172537.28464-1-lersek@redhat.com> From: "Laszlo Ersek" Cc: "devel@edk2.groups.io" Message-ID: <2679d4af-4034-525e-8189-2d795794ced1@redhat.com> Date: Fri, 24 Jul 2020 18:01:37 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0 Thunderbird/52.9.1 MIME-Version: 1.0 In-Reply-To: X-Scanned-By: MIMEDefang 2.79 on 10.5.11.15 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Language: en-US Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit On 07/24/20 08:26, Wu, Jiaxin wrote: > Hi Laszlo, > > Looks OVMF supports the CPU hotplug with those series patches. > > Could you provide some guide how to enable the OVMF CPU hotplug > verification? Is there any general work flow introduction how it > works? For example, how to do the hot add CPU initialization (e.g. > Register setting / Microcode update, etc.)? I'm very interested in > this feature on OVMF. Long version: ------------- (1) There are three pieces missing: (1a) The QEMU side changes for the ACPI (DSDT) content that QEMU generates for the OS. The ACPI GPE handler for CPU hotplug is being modified by my colleague Igor Mammedov to raise the SMI (command value 4) on CPU hotplug. For developing the OVMF series for TianoCore#1512 (which is now merged), I used a prototype QEMU patch, from Igor. But that patch is not suitable for upstreaming to QEMU. SO Igor is now developing the real patches for QEMU's ACPI generator. (1b) The related feature negotiation patches in QEMU. In order for "CPU hotplug with SMM" to work, both OVMF and QEMU need to perform specific things. In order to deal with cross-version compatibility problems, the "CPU hotplug with SMI" feature is dynamically negotiated between OVMF and QEMU. For this negotiation, both QEMU and OVMF need additional patches. These patches are not related to the actual plugging activities; instead they control whether plugging is permitted at all, or not. Igor's QEMU series covers both purposes (1a) and (1b). It's work in progress. The first posting was an RFC series: (1b1) [RFC 0/3] x86: fix cpu hotplug with secure boot https://lists.gnu.org/archive/html/qemu-devel/2020-07/msg03746.html http://mid.mail-archive.com/20200710161704.309824-1-imammedo@redhat.com The latest posting has been a PATCH series: (1b2) [qemu-devel] [PATCH 0/6] x86: fix cpu hotplug with secure boot https://lists.gnu.org/archive/html/qemu-devel/2020-07/msg05850.html http://mid.mail-archive.com/20200720141610.574308-1-imammedo@redhat.com (1c) The feature negotiation patch for OVMF is here: * [edk2-devel] [PATCH] OvmfPkg/SmmControl2Dxe: negotiate ICH9_LPC_SMI_F_CPU_HOTPLUG https://edk2.groups.io/g/devel/message/62561 http://mid.mail-archive.com/20200714184305.9814-1-lersek@redhat.com (2) Special register setting and microcode stuff are not needed. (3) As I mentioned before, I strongly suggest using QEMU and OVMF with libvirt. I had written an article about that here: https://github.com/tianocore/tianocore.github.io/wiki/Testing-SMM-with-QEMU,-KVM-and-libvirt I wrote this article specifically for "Windows-based" developers. The article is written from such a perspective that you don't need a personal Linux workstation, only a single Linux workstation *per team*. So you can continue using a Windows workstation, just set up one Linux box for your team (if you don't yet have one). This article remains relevant. (3a) In order to set up a guest for VCPU hotplug, simply go through the article, initially. (3b) Once you're done with that, power down the guest, and modify the domain XML as follows: virsh edit (3b1) replace the "pc-q35-2.9" machine type with "pc-q35-5.1" (3b2) replace the following stanza: 4 with: 4 This will create a VCPU topology where: - CPU#0 is present up-front, and is not hot-pluggable (this is a QEMU requirement), - CPU#1, CPU#2, and CPU#3 are hot-pluggable, - CPU#2 is present up-front ("cold-plugged"), while CPU#1 and CPU#3 are absent initially. (4) Boot the guest. Once you have a root prompt in the guest, you can use one of two libvirt commands for hot-plugging a CPU: (4a) the singular "virsh setvcpu" command: virsh setvcpu --enable --live where you can pass in 1 or 3 for . This command lets you specify the precise ID of the processor to be hot-plugged; IOW, the command lets you control topology. (4b) the plural "virsh setvcpus" command: virsh setvcpus --live This command lets you specify the desired number of total active CPUs. It does not let you control topology. (My understanding is that it keeps the topology populated at the "front".) Regarding the current QEMU status, we need to do more work for supporting (4b). The RFC series (1b1) enables (4a) to work. The PATCH series (1b2) intended to make (4b) work, but unfortunately it broke even (4a). So now we need at least one more version of the QEMU series (I've given my feedback to Igor already, on qemu-devel). (4c) Dependent on guest OS configuration, you might have to manually online the newly plugged CPUs in the guest: echo 1 > /sys/devices/system/cpu/cpu2/online echo 1 > /sys/devices/system/cpu/cpu3/online Note that the "cpuN" identifiers seen here are *neither* APIC IDs *nor* the same IDs as seen in the libvirt domain XML. Instead, these IDs are assigned in the order the Linux kernel learns about the CPUs (if I understand correctly). Short version: -------------- - apply (1b1) on top of latest QEMU master from git, and build and install it, - apply (1c) on latest edk2, and build OVMF with "-D SMM_REQUIRE", - install a Linux guest on a Linux host (using KVM!) as described in my Wiki article (3), - modify the domain XML for the guest as described in (3b), - use the singular "virsh setvcpu" command (4a) for hot-plugging VCPU#1 and/or VCPU#3, - if necessary, use (4c) in the guest. You can do the same with Windows Server guests too, although I'm not exactly sure what versions support CPU hotplug. For testing I've used Windows Server 2012 R2. The Wiki article at (3) has a section dedicated to installing Windows guests too. Thanks, Laszlo