[RFC PATCH] OvmfPkg/PlatformCI VS2019: Enable temporary workaround for cpuhp bugfix

[RFC PATCH] OvmfPkg/PlatformCI VS2019: Enable temporary workaround for cpuhp bugfix

[Ard Biesheuvel]

QEMU for x86 has a nasty CPU hotplug bug of which the ramifications are
difficult to oversee, even though KVM acceleration seems to be
unaffected. This has been addressed in QEMU mainline, and will percolate
through the ecosystem at its usual pace. In the mean time, due to the
potential impact on production workloads, we will be updating OVMF to
abort the boot when it detects a QEMU build that is affected.

Tiancore's platform CI uses QEMU in TCG mode, and is therefore impacted
by this mitigation, unless its QEMU builds are updated. This has been
done for Ubuntu-GCC5, but Windows-VS2019 still uses a QEMU build that is
affected.

Aborting the boot upon detecting the QEMU issue will render all boot
tests carried out on Windows-VS2019 broken unless we implement the
'escape hatch' that enables proceed-at-your-own-risk mode, and permits
the boot to proceed even if the QEMU issue is detected.

So let's enable this for Windows-VS2019, and remove it again once it is
no longer needed.

Cc: Laszlo Ersek <lersek@redhat.com>
Cc: Gerd Hoffmann <kraxel@redhat.com>
Cc: Jiewen Yao <jiewen.yao@intel.com>
Cc: Michael Brown <mcb30@ipxe.org>
Cc: Oliver Steffen <osteffen@redhat.com>
Cc: Michael Kubacki <michael.kubacki@microsoft.com>

Bugzilla: https://bugzilla.tianocore.org/show_bug.cgi?id=4250
Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
---
 OvmfPkg/PlatformCI/.azurepipelines/Windows-VS2019.yml |  2 +-
 OvmfPkg/PlatformCI/PlatformBuildLib.py                | 12 ++++++++++++
 2 files changed, 13 insertions(+), 1 deletion(-)

diff --git a/OvmfPkg/PlatformCI/.azurepipelines/Windows-VS2019.yml b/OvmfPkg/PlatformCI/.azurepipelines/Windows-VS2019.yml
index 7e63f419b26b..b3b91aa84ea0 100644
--- a/OvmfPkg/PlatformCI/.azurepipelines/Windows-VS2019.yml
+++ b/OvmfPkg/PlatformCI/.azurepipelines/Windows-VS2019.yml
@@ -24,7 +24,7 @@ jobs:
       package: 'OvmfPkg'
       vm_image: 'windows-2019'
       should_run: true
-      run_flags: "MAKE_STARTUP_NSH=TRUE QEMU_HEADLESS=TRUE"
+      run_flags: "MAKE_STARTUP_NSH=TRUE QEMU_HEADLESS=TRUE QEMU_CPUHP_QUIRK=TRUE"
 
     #Use matrix to speed up the build process
     strategy:
diff --git a/OvmfPkg/PlatformCI/PlatformBuildLib.py b/OvmfPkg/PlatformCI/PlatformBuildLib.py
index bfef9849c749..58dc1189a2cc 100644
--- a/OvmfPkg/PlatformCI/PlatformBuildLib.py
+++ b/OvmfPkg/PlatformCI/PlatformBuildLib.py
@@ -170,6 +170,7 @@ class PlatformBuilder( UefiBuilder, BuildSettingsManager):
         self.env.SetValue("PRODUCT_NAME", "OVMF", "Platform Hardcoded")
         self.env.SetValue("MAKE_STARTUP_NSH", "FALSE", "Default to false")
         self.env.SetValue("QEMU_HEADLESS", "FALSE", "Default to false")
+        self.env.SetValue("QEMU_CPUHP_QUIRK", "FALSE", "Default to false")
         return 0
 
     def PlatformPreBuild(self):
@@ -211,6 +212,17 @@ class PlatformBuilder( UefiBuilder, BuildSettingsManager):
             args += " -pflash " + os.path.join(OutputPath_FV, "OVMF.fd")    # path to firmware
 
 
+        ###
+        ### NOTE This is a temporary workaround to allow platform CI to cope with
+        ###      a QEMU bug in the CPU hotplug code. Once the CI environment has
+        ###      been updated to carry a fixed version of QEMU, this can be
+        ###      removed again
+        ###
+        ### Bugzilla: https://bugzilla.tianocore.org/show_bug.cgi?id=4250
+        ###
+        if (self.env.GetValue("QEMU_CPUHP_QUIRK").upper() == "TRUE"):
+            args += "  -fw_cfg name=opt/org.tianocore/X-Cpuhp-Bugcheck-Override,string=yes"
+
         if (self.env.GetValue("MAKE_STARTUP_NSH").upper() == "TRUE"):
             f = open(os.path.join(VirtualDrive, "startup.nsh"), "w")
             f.write("BOOT SUCCESS !!! \n")
-- 
2.39.0

Re: [RFC PATCH] OvmfPkg/PlatformCI VS2019: Enable temporary workaround for cpuhp bugfix

[Gerd Hoffmann]

On Thu, Jan 19, 2023 at 02:43:02PM +0100, Ard Biesheuvel wrote:
> QEMU for x86 has a nasty CPU hotplug bug of which the ramifications are
> difficult to oversee, even though KVM acceleration seems to be
> unaffected. This has been addressed in QEMU mainline, and will percolate
> through the ecosystem at its usual pace. In the mean time, due to the
> potential impact on production workloads, we will be updating OVMF to
> abort the boot when it detects a QEMU build that is affected.
> 
> Tiancore's platform CI uses QEMU in TCG mode, and is therefore impacted
> by this mitigation, unless its QEMU builds are updated. This has been
> done for Ubuntu-GCC5, but Windows-VS2019 still uses a QEMU build that is
> affected.
> 
> Aborting the boot upon detecting the QEMU issue will render all boot
> tests carried out on Windows-VS2019 broken unless we implement the
> 'escape hatch' that enables proceed-at-your-own-risk mode, and permits
> the boot to proceed even if the QEMU issue is detected.
> 
> So let's enable this for Windows-VS2019, and remove it again once it is
> no longer needed.
> 
> Cc: Laszlo Ersek <lersek@redhat.com>
> Cc: Gerd Hoffmann <kraxel@redhat.com>
> Cc: Jiewen Yao <jiewen.yao@intel.com>
> Cc: Michael Brown <mcb30@ipxe.org>
> Cc: Oliver Steffen <osteffen@redhat.com>
> Cc: Michael Kubacki <michael.kubacki@microsoft.com>
> 
> Bugzilla: https://bugzilla.tianocore.org/show_bug.cgi?id=4250
> Signed-off-by: Ard Biesheuvel <ardb@kernel.org>

Acked-by: Gerd Hoffmann <kraxel@redhat.com>

Re: [edk2-devel] [RFC PATCH] OvmfPkg/PlatformCI VS2019: Enable temporary workaround for cpuhp bugfix

[Michael Kubacki]

Reviewed-by: Michael Kubacki <michael.kubacki@microsoft.com>

On 1/19/2023 8:43 AM, Ard Biesheuvel wrote:
> QEMU for x86 has a nasty CPU hotplug bug of which the ramifications are
> 
> difficult to oversee, even though KVM acceleration seems to be
> 
> unaffected. This has been addressed in QEMU mainline, and will percolate
> 
> through the ecosystem at its usual pace. In the mean time, due to the
> 
> potential impact on production workloads, we will be updating OVMF to
> 
> abort the boot when it detects a QEMU build that is affected.
> 
> 
> 
> Tiancore's platform CI uses QEMU in TCG mode, and is therefore impacted
> 
> by this mitigation, unless its QEMU builds are updated. This has been
> 
> done for Ubuntu-GCC5, but Windows-VS2019 still uses a QEMU build that is
> 
> affected.
> 
> 
> 
> Aborting the boot upon detecting the QEMU issue will render all boot
> 
> tests carried out on Windows-VS2019 broken unless we implement the
> 
> 'escape hatch' that enables proceed-at-your-own-risk mode, and permits
> 
> the boot to proceed even if the QEMU issue is detected.
> 
> 
> 
> So let's enable this for Windows-VS2019, and remove it again once it is
> 
> no longer needed.
> 
> 
> 
> Cc: Laszlo Ersek <lersek@redhat.com>
> 
> Cc: Gerd Hoffmann <kraxel@redhat.com>
> 
> Cc: Jiewen Yao <jiewen.yao@intel.com>
> 
> Cc: Michael Brown <mcb30@ipxe.org>
> 
> Cc: Oliver Steffen <osteffen@redhat.com>
> 
> Cc: Michael Kubacki <michael.kubacki@microsoft.com>
> 
> 
> 
> Bugzilla: https://bugzilla.tianocore.org/show_bug.cgi?id=4250
> 
> Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
> 
> ---
> 
>   OvmfPkg/PlatformCI/.azurepipelines/Windows-VS2019.yml |  2 +-
> 
>   OvmfPkg/PlatformCI/PlatformBuildLib.py                | 12 ++++++++++++
> 
>   2 files changed, 13 insertions(+), 1 deletion(-)
> 
> 
> 
> diff --git a/OvmfPkg/PlatformCI/.azurepipelines/Windows-VS2019.yml b/OvmfPkg/PlatformCI/.azurepipelines/Windows-VS2019.yml
> 
> index 7e63f419b26b..b3b91aa84ea0 100644
> 
> --- a/OvmfPkg/PlatformCI/.azurepipelines/Windows-VS2019.yml
> 
> +++ b/OvmfPkg/PlatformCI/.azurepipelines/Windows-VS2019.yml
> 
> @@ -24,7 +24,7 @@ jobs:
> 
>         package: 'OvmfPkg'
> 
>         vm_image: 'windows-2019'
> 
>         should_run: true
> 
> -      run_flags: "MAKE_STARTUP_NSH=TRUE QEMU_HEADLESS=TRUE"
> 
> +      run_flags: "MAKE_STARTUP_NSH=TRUE QEMU_HEADLESS=TRUE QEMU_CPUHP_QUIRK=TRUE"
> 
>   
> 
>       #Use matrix to speed up the build process
> 
>       strategy:
> 
> diff --git a/OvmfPkg/PlatformCI/PlatformBuildLib.py b/OvmfPkg/PlatformCI/PlatformBuildLib.py
> 
> index bfef9849c749..58dc1189a2cc 100644
> 
> --- a/OvmfPkg/PlatformCI/PlatformBuildLib.py
> 
> +++ b/OvmfPkg/PlatformCI/PlatformBuildLib.py
> 
> @@ -170,6 +170,7 @@ class PlatformBuilder( UefiBuilder, BuildSettingsManager):
> 
>           self.env.SetValue("PRODUCT_NAME", "OVMF", "Platform Hardcoded")
> 
>           self.env.SetValue("MAKE_STARTUP_NSH", "FALSE", "Default to false")
> 
>           self.env.SetValue("QEMU_HEADLESS", "FALSE", "Default to false")
> 
> +        self.env.SetValue("QEMU_CPUHP_QUIRK", "FALSE", "Default to false")
> 
>           return 0
> 
>   
> 
>       def PlatformPreBuild(self):
> 
> @@ -211,6 +212,17 @@ class PlatformBuilder( UefiBuilder, BuildSettingsManager):
> 
>               args += " -pflash " + os.path.join(OutputPath_FV, "OVMF.fd")    # path to firmware
> 
>   
> 
>   
> 
> +        ###
> 
> +        ### NOTE This is a temporary workaround to allow platform CI to cope with
> 
> +        ###      a QEMU bug in the CPU hotplug code. Once the CI environment has
> 
> +        ###      been updated to carry a fixed version of QEMU, this can be
> 
> +        ###      removed again
> 
> +        ###
> 
> +        ### Bugzilla: https://bugzilla.tianocore.org/show_bug.cgi?id=4250
> 
> +        ###
> 
> +        if (self.env.GetValue("QEMU_CPUHP_QUIRK").upper() == "TRUE"):
> 
> +            args += "  -fw_cfg name=opt/org.tianocore/X-Cpuhp-Bugcheck-Override,string=yes"
> 
> +
> 
>           if (self.env.GetValue("MAKE_STARTUP_NSH").upper() == "TRUE"):
> 
>               f = open(os.path.join(VirtualDrive, "startup.nsh"), "w")
> 
>               f.write("BOOT SUCCESS !!! \n")
> 

Re: [edk2-devel] [RFC PATCH] OvmfPkg/PlatformCI VS2019: Enable temporary workaround for cpuhp bugfix

[Yao, Jiewen]

Acked-by: Jiewen Yao <Jiewen.yao@Intel.com>

> -----Original Message-----
> From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Michael
> Kubacki
> Sent: Friday, January 20, 2023 12:51 AM
> To: devel@edk2.groups.io; ardb@kernel.org
> Cc: Laszlo Ersek <lersek@redhat.com>; Gerd Hoffmann
> <kraxel@redhat.com>; Yao, Jiewen <jiewen.yao@intel.com>; Michael Brown
> <mcb30@ipxe.org>; Oliver Steffen <osteffen@redhat.com>; Kubacki, Michael
> <michael.kubacki@microsoft.com>
> Subject: Re: [edk2-devel] [RFC PATCH] OvmfPkg/PlatformCI VS2019: Enable
> temporary workaround for cpuhp bugfix
> 
> Reviewed-by: Michael Kubacki <michael.kubacki@microsoft.com>
> 
> On 1/19/2023 8:43 AM, Ard Biesheuvel wrote:
> > QEMU for x86 has a nasty CPU hotplug bug of which the ramifications are
> >
> > difficult to oversee, even though KVM acceleration seems to be
> >
> > unaffected. This has been addressed in QEMU mainline, and will percolate
> >
> > through the ecosystem at its usual pace. In the mean time, due to the
> >
> > potential impact on production workloads, we will be updating OVMF to
> >
> > abort the boot when it detects a QEMU build that is affected.
> >
> >
> >
> > Tiancore's platform CI uses QEMU in TCG mode, and is therefore impacted
> >
> > by this mitigation, unless its QEMU builds are updated. This has been
> >
> > done for Ubuntu-GCC5, but Windows-VS2019 still uses a QEMU build that
> is
> >
> > affected.
> >
> >
> >
> > Aborting the boot upon detecting the QEMU issue will render all boot
> >
> > tests carried out on Windows-VS2019 broken unless we implement the
> >
> > 'escape hatch' that enables proceed-at-your-own-risk mode, and permits
> >
> > the boot to proceed even if the QEMU issue is detected.
> >
> >
> >
> > So let's enable this for Windows-VS2019, and remove it again once it is
> >
> > no longer needed.
> >
> >
> >
> > Cc: Laszlo Ersek <lersek@redhat.com>
> >
> > Cc: Gerd Hoffmann <kraxel@redhat.com>
> >
> > Cc: Jiewen Yao <jiewen.yao@intel.com>
> >
> > Cc: Michael Brown <mcb30@ipxe.org>
> >
> > Cc: Oliver Steffen <osteffen@redhat.com>
> >
> > Cc: Michael Kubacki <michael.kubacki@microsoft.com>
> >
> >
> >
> > Bugzilla: https://bugzilla.tianocore.org/show_bug.cgi?id=4250
> >
> > Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
> >
> > ---
> >
> >   OvmfPkg/PlatformCI/.azurepipelines/Windows-VS2019.yml |  2 +-
> >
> >   OvmfPkg/PlatformCI/PlatformBuildLib.py                | 12 ++++++++++++
> >
> >   2 files changed, 13 insertions(+), 1 deletion(-)
> >
> >
> >
> > diff --git a/OvmfPkg/PlatformCI/.azurepipelines/Windows-VS2019.yml
> b/OvmfPkg/PlatformCI/.azurepipelines/Windows-VS2019.yml
> >
> > index 7e63f419b26b..b3b91aa84ea0 100644
> >
> > --- a/OvmfPkg/PlatformCI/.azurepipelines/Windows-VS2019.yml
> >
> > +++ b/OvmfPkg/PlatformCI/.azurepipelines/Windows-VS2019.yml
> >
> > @@ -24,7 +24,7 @@ jobs:
> >
> >         package: 'OvmfPkg'
> >
> >         vm_image: 'windows-2019'
> >
> >         should_run: true
> >
> > -      run_flags: "MAKE_STARTUP_NSH=TRUE QEMU_HEADLESS=TRUE"
> >
> > +      run_flags: "MAKE_STARTUP_NSH=TRUE QEMU_HEADLESS=TRUE
> QEMU_CPUHP_QUIRK=TRUE"
> >
> >
> >
> >       #Use matrix to speed up the build process
> >
> >       strategy:
> >
> > diff --git a/OvmfPkg/PlatformCI/PlatformBuildLib.py
> b/OvmfPkg/PlatformCI/PlatformBuildLib.py
> >
> > index bfef9849c749..58dc1189a2cc 100644
> >
> > --- a/OvmfPkg/PlatformCI/PlatformBuildLib.py
> >
> > +++ b/OvmfPkg/PlatformCI/PlatformBuildLib.py
> >
> > @@ -170,6 +170,7 @@ class PlatformBuilder( UefiBuilder,
> BuildSettingsManager):
> >
> >           self.env.SetValue("PRODUCT_NAME", "OVMF", "Platform Hardcoded")
> >
> >           self.env.SetValue("MAKE_STARTUP_NSH", "FALSE", "Default to false")
> >
> >           self.env.SetValue("QEMU_HEADLESS", "FALSE", "Default to false")
> >
> > +        self.env.SetValue("QEMU_CPUHP_QUIRK", "FALSE", "Default to false")
> >
> >           return 0
> >
> >
> >
> >       def PlatformPreBuild(self):
> >
> > @@ -211,6 +212,17 @@ class PlatformBuilder( UefiBuilder,
> BuildSettingsManager):
> >
> >               args += " -pflash " + os.path.join(OutputPath_FV, "OVMF.fd")    #
> path to firmware
> >
> >
> >
> >
> >
> > +        ###
> >
> > +        ### NOTE This is a temporary workaround to allow platform CI to
> cope with
> >
> > +        ###      a QEMU bug in the CPU hotplug code. Once the CI
> environment has
> >
> > +        ###      been updated to carry a fixed version of QEMU, this can be
> >
> > +        ###      removed again
> >
> > +        ###
> >
> > +        ### Bugzilla: https://bugzilla.tianocore.org/show_bug.cgi?id=4250
> >
> > +        ###
> >
> > +        if (self.env.GetValue("QEMU_CPUHP_QUIRK").upper() == "TRUE"):
> >
> > +            args += "  -fw_cfg name=opt/org.tianocore/X-Cpuhp-Bugcheck-
> Override,string=yes"
> >
> > +
> >
> >           if (self.env.GetValue("MAKE_STARTUP_NSH").upper() == "TRUE"):
> >
> >               f = open(os.path.join(VirtualDrive, "startup.nsh"), "w")
> >
> >               f.write("BOOT SUCCESS !!! \n")
> >
> 
> 
> 
> 

Re: [RFC PATCH] OvmfPkg/PlatformCI VS2019: Enable temporary workaround for cpuhp bugfix

[Laszlo Ersek]

On 1/19/23 14:43, Ard Biesheuvel wrote:
> QEMU for x86 has a nasty CPU hotplug bug of which the ramifications are
> difficult to oversee, even though KVM acceleration seems to be
> unaffected. This has been addressed in QEMU mainline, and will percolate
> through the ecosystem at its usual pace. In the mean time, due to the
> potential impact on production workloads, we will be updating OVMF to
> abort the boot when it detects a QEMU build that is affected.
> 
> Tiancore's platform CI uses QEMU in TCG mode, and is therefore impacted
> by this mitigation, unless its QEMU builds are updated. This has been
> done for Ubuntu-GCC5, but Windows-VS2019 still uses a QEMU build that is
> affected.
> 
> Aborting the boot upon detecting the QEMU issue will render all boot
> tests carried out on Windows-VS2019 broken unless we implement the
> 'escape hatch' that enables proceed-at-your-own-risk mode, and permits
> the boot to proceed even if the QEMU issue is detected.
> 
> So let's enable this for Windows-VS2019, and remove it again once it is
> no longer needed.
> 
> Cc: Laszlo Ersek <lersek@redhat.com>
> Cc: Gerd Hoffmann <kraxel@redhat.com>
> Cc: Jiewen Yao <jiewen.yao@intel.com>
> Cc: Michael Brown <mcb30@ipxe.org>
> Cc: Oliver Steffen <osteffen@redhat.com>
> Cc: Michael Kubacki <michael.kubacki@microsoft.com>
> 
> Bugzilla: https://bugzilla.tianocore.org/show_bug.cgi?id=4250
> Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
> ---
>  OvmfPkg/PlatformCI/.azurepipelines/Windows-VS2019.yml |  2 +-
>  OvmfPkg/PlatformCI/PlatformBuildLib.py                | 12 ++++++++++++
>  2 files changed, 13 insertions(+), 1 deletion(-)
> 
> diff --git a/OvmfPkg/PlatformCI/.azurepipelines/Windows-VS2019.yml b/OvmfPkg/PlatformCI/.azurepipelines/Windows-VS2019.yml
> index 7e63f419b26b..b3b91aa84ea0 100644
> --- a/OvmfPkg/PlatformCI/.azurepipelines/Windows-VS2019.yml
> +++ b/OvmfPkg/PlatformCI/.azurepipelines/Windows-VS2019.yml
> @@ -24,7 +24,7 @@ jobs:
>        package: 'OvmfPkg'
>        vm_image: 'windows-2019'
>        should_run: true
> -      run_flags: "MAKE_STARTUP_NSH=TRUE QEMU_HEADLESS=TRUE"
> +      run_flags: "MAKE_STARTUP_NSH=TRUE QEMU_HEADLESS=TRUE QEMU_CPUHP_QUIRK=TRUE"
>  
>      #Use matrix to speed up the build process
>      strategy:
> diff --git a/OvmfPkg/PlatformCI/PlatformBuildLib.py b/OvmfPkg/PlatformCI/PlatformBuildLib.py
> index bfef9849c749..58dc1189a2cc 100644
> --- a/OvmfPkg/PlatformCI/PlatformBuildLib.py
> +++ b/OvmfPkg/PlatformCI/PlatformBuildLib.py
> @@ -170,6 +170,7 @@ class PlatformBuilder( UefiBuilder, BuildSettingsManager):
>          self.env.SetValue("PRODUCT_NAME", "OVMF", "Platform Hardcoded")
>          self.env.SetValue("MAKE_STARTUP_NSH", "FALSE", "Default to false")
>          self.env.SetValue("QEMU_HEADLESS", "FALSE", "Default to false")
> +        self.env.SetValue("QEMU_CPUHP_QUIRK", "FALSE", "Default to false")
>          return 0
>  
>      def PlatformPreBuild(self):
> @@ -211,6 +212,17 @@ class PlatformBuilder( UefiBuilder, BuildSettingsManager):
>              args += " -pflash " + os.path.join(OutputPath_FV, "OVMF.fd")    # path to firmware
>  
>  
> +        ###
> +        ### NOTE This is a temporary workaround to allow platform CI to cope with
> +        ###      a QEMU bug in the CPU hotplug code. Once the CI environment has
> +        ###      been updated to carry a fixed version of QEMU, this can be
> +        ###      removed again
> +        ###
> +        ### Bugzilla: https://bugzilla.tianocore.org/show_bug.cgi?id=4250
> +        ###
> +        if (self.env.GetValue("QEMU_CPUHP_QUIRK").upper() == "TRUE"):
> +            args += "  -fw_cfg name=opt/org.tianocore/X-Cpuhp-Bugcheck-Override,string=yes"
> +
>          if (self.env.GetValue("MAKE_STARTUP_NSH").upper() == "TRUE"):
>              f = open(os.path.join(VirtualDrive, "startup.nsh"), "w")
>              f.write("BOOT SUCCESS !!! \n")

Reviewed-by: Laszlo Ersek <lersek@redhat.com>

Technically speaking, can I merge this *prepended* to my v3 patch set
([PATCH v3 0/2] OvmfPkg/PlatformInitLib: catch QEMU's CPU hotplug reg
block regression), in the *same* PR?

Because then I'll do that, saving us a bit of duplicated work.

I'll then also file the BZ for reverting this (once I know the commit
hash), for when a new QEMU is out for Windows -- whom should I assign
that BZ? Oliver or Ard?

Thanks!
Laszlo

Re: [edk2-devel] [RFC PATCH] OvmfPkg/PlatformCI VS2019: Enable temporary workaround for cpuhp bugfix

[Ard Biesheuvel]

On Fri, 20 Jan 2023 at 10:25, Laszlo Ersek <lersek@redhat.com> wrote:
>
> On 1/19/23 14:43, Ard Biesheuvel wrote:
> > QEMU for x86 has a nasty CPU hotplug bug of which the ramifications are
> > difficult to oversee, even though KVM acceleration seems to be
> > unaffected. This has been addressed in QEMU mainline, and will percolate
> > through the ecosystem at its usual pace. In the mean time, due to the
> > potential impact on production workloads, we will be updating OVMF to
> > abort the boot when it detects a QEMU build that is affected.
> >
> > Tiancore's platform CI uses QEMU in TCG mode, and is therefore impacted
> > by this mitigation, unless its QEMU builds are updated. This has been
> > done for Ubuntu-GCC5, but Windows-VS2019 still uses a QEMU build that is
> > affected.
> >
> > Aborting the boot upon detecting the QEMU issue will render all boot
> > tests carried out on Windows-VS2019 broken unless we implement the
> > 'escape hatch' that enables proceed-at-your-own-risk mode, and permits
> > the boot to proceed even if the QEMU issue is detected.
> >
> > So let's enable this for Windows-VS2019, and remove it again once it is
> > no longer needed.
> >
> > Cc: Laszlo Ersek <lersek@redhat.com>
> > Cc: Gerd Hoffmann <kraxel@redhat.com>
> > Cc: Jiewen Yao <jiewen.yao@intel.com>
> > Cc: Michael Brown <mcb30@ipxe.org>
> > Cc: Oliver Steffen <osteffen@redhat.com>
> > Cc: Michael Kubacki <michael.kubacki@microsoft.com>
> >
> > Bugzilla: https://bugzilla.tianocore.org/show_bug.cgi?id=4250
> > Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
> > ---
> >  OvmfPkg/PlatformCI/.azurepipelines/Windows-VS2019.yml |  2 +-
> >  OvmfPkg/PlatformCI/PlatformBuildLib.py                | 12 ++++++++++++
> >  2 files changed, 13 insertions(+), 1 deletion(-)
> >
> > diff --git a/OvmfPkg/PlatformCI/.azurepipelines/Windows-VS2019.yml b/OvmfPkg/PlatformCI/.azurepipelines/Windows-VS2019.yml
> > index 7e63f419b26b..b3b91aa84ea0 100644
> > --- a/OvmfPkg/PlatformCI/.azurepipelines/Windows-VS2019.yml
> > +++ b/OvmfPkg/PlatformCI/.azurepipelines/Windows-VS2019.yml
> > @@ -24,7 +24,7 @@ jobs:
> >        package: 'OvmfPkg'
> >        vm_image: 'windows-2019'
> >        should_run: true
> > -      run_flags: "MAKE_STARTUP_NSH=TRUE QEMU_HEADLESS=TRUE"
> > +      run_flags: "MAKE_STARTUP_NSH=TRUE QEMU_HEADLESS=TRUE QEMU_CPUHP_QUIRK=TRUE"
> >
> >      #Use matrix to speed up the build process
> >      strategy:
> > diff --git a/OvmfPkg/PlatformCI/PlatformBuildLib.py b/OvmfPkg/PlatformCI/PlatformBuildLib.py
> > index bfef9849c749..58dc1189a2cc 100644
> > --- a/OvmfPkg/PlatformCI/PlatformBuildLib.py
> > +++ b/OvmfPkg/PlatformCI/PlatformBuildLib.py
> > @@ -170,6 +170,7 @@ class PlatformBuilder( UefiBuilder, BuildSettingsManager):
> >          self.env.SetValue("PRODUCT_NAME", "OVMF", "Platform Hardcoded")
> >          self.env.SetValue("MAKE_STARTUP_NSH", "FALSE", "Default to false")
> >          self.env.SetValue("QEMU_HEADLESS", "FALSE", "Default to false")
> > +        self.env.SetValue("QEMU_CPUHP_QUIRK", "FALSE", "Default to false")
> >          return 0
> >
> >      def PlatformPreBuild(self):
> > @@ -211,6 +212,17 @@ class PlatformBuilder( UefiBuilder, BuildSettingsManager):
> >              args += " -pflash " + os.path.join(OutputPath_FV, "OVMF.fd")    # path to firmware
> >
> >
> > +        ###
> > +        ### NOTE This is a temporary workaround to allow platform CI to cope with
> > +        ###      a QEMU bug in the CPU hotplug code. Once the CI environment has
> > +        ###      been updated to carry a fixed version of QEMU, this can be
> > +        ###      removed again
> > +        ###
> > +        ### Bugzilla: https://bugzilla.tianocore.org/show_bug.cgi?id=4250
> > +        ###
> > +        if (self.env.GetValue("QEMU_CPUHP_QUIRK").upper() == "TRUE"):
> > +            args += "  -fw_cfg name=opt/org.tianocore/X-Cpuhp-Bugcheck-Override,string=yes"
> > +
> >          if (self.env.GetValue("MAKE_STARTUP_NSH").upper() == "TRUE"):
> >              f = open(os.path.join(VirtualDrive, "startup.nsh"), "w")
> >              f.write("BOOT SUCCESS !!! \n")
>
> Reviewed-by: Laszlo Ersek <lersek@redhat.com>
>
> Technically speaking, can I merge this *prepended* to my v3 patch set
> ([PATCH v3 0/2] OvmfPkg/PlatformInitLib: catch QEMU's CPU hotplug reg
> block regression), in the *same* PR?
>
> Because then I'll do that, saving us a bit of duplicated work.
>

Yes, please go ahead and merge this if it is all ready to go. I won't
get around to it to later this afternoon.

> I'll then also file the BZ for reverting this (once I know the commit
> hash), for when a new QEMU is out for Windows -- whom should I assign
> that BZ? Oliver or Ard?
>

You can assign it to me. And please put Michael Kubacki on cc as well.

Re: [RFC PATCH] OvmfPkg/PlatformCI VS2019: Enable temporary workaround for cpuhp bugfix

[Laszlo Ersek]

On 1/19/23 14:43, Ard Biesheuvel wrote:
> QEMU for x86 has a nasty CPU hotplug bug of which the ramifications are
> difficult to oversee, even though KVM acceleration seems to be
> unaffected. This has been addressed in QEMU mainline, and will percolate
> through the ecosystem at its usual pace. In the mean time, due to the
> potential impact on production workloads, we will be updating OVMF to
> abort the boot when it detects a QEMU build that is affected.
> 
> Tiancore's platform CI uses QEMU in TCG mode, and is therefore impacted
> by this mitigation, unless its QEMU builds are updated. This has been
> done for Ubuntu-GCC5, but Windows-VS2019 still uses a QEMU build that is
> affected.
> 
> Aborting the boot upon detecting the QEMU issue will render all boot
> tests carried out on Windows-VS2019 broken unless we implement the
> 'escape hatch' that enables proceed-at-your-own-risk mode, and permits
> the boot to proceed even if the QEMU issue is detected.
> 
> So let's enable this for Windows-VS2019, and remove it again once it is
> no longer needed.
> 
> Cc: Laszlo Ersek <lersek@redhat.com>
> Cc: Gerd Hoffmann <kraxel@redhat.com>
> Cc: Jiewen Yao <jiewen.yao@intel.com>
> Cc: Michael Brown <mcb30@ipxe.org>
> Cc: Oliver Steffen <osteffen@redhat.com>
> Cc: Michael Kubacki <michael.kubacki@microsoft.com>
> 
> Bugzilla: https://bugzilla.tianocore.org/show_bug.cgi?id=4250
> Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
> ---
>  OvmfPkg/PlatformCI/.azurepipelines/Windows-VS2019.yml |  2 +-
>  OvmfPkg/PlatformCI/PlatformBuildLib.py                | 12 ++++++++++++
>  2 files changed, 13 insertions(+), 1 deletion(-)

Merged as the first commit in range 51411435d559..bf5678b58026, via
<https://github.com/tianocore/edk2/pull/3935>.

Thank you!
Laszlo

Re: [edk2-devel] [RFC PATCH] OvmfPkg/PlatformCI VS2019: Enable temporary workaround for cpuhp bugfix

[Laszlo Ersek]

On 1/20/23 10:54, Ard Biesheuvel wrote:
> On Fri, 20 Jan 2023 at 10:25, Laszlo Ersek <lersek@redhat.com> wrote:
>>
>> On 1/19/23 14:43, Ard Biesheuvel wrote:
>>> QEMU for x86 has a nasty CPU hotplug bug of which the ramifications are
>>> difficult to oversee, even though KVM acceleration seems to be
>>> unaffected. This has been addressed in QEMU mainline, and will percolate
>>> through the ecosystem at its usual pace. In the mean time, due to the
>>> potential impact on production workloads, we will be updating OVMF to
>>> abort the boot when it detects a QEMU build that is affected.
>>>
>>> Tiancore's platform CI uses QEMU in TCG mode, and is therefore impacted
>>> by this mitigation, unless its QEMU builds are updated. This has been
>>> done for Ubuntu-GCC5, but Windows-VS2019 still uses a QEMU build that is
>>> affected.
>>>
>>> Aborting the boot upon detecting the QEMU issue will render all boot
>>> tests carried out on Windows-VS2019 broken unless we implement the
>>> 'escape hatch' that enables proceed-at-your-own-risk mode, and permits
>>> the boot to proceed even if the QEMU issue is detected.
>>>
>>> So let's enable this for Windows-VS2019, and remove it again once it is
>>> no longer needed.
>>>
>>> Cc: Laszlo Ersek <lersek@redhat.com>
>>> Cc: Gerd Hoffmann <kraxel@redhat.com>
>>> Cc: Jiewen Yao <jiewen.yao@intel.com>
>>> Cc: Michael Brown <mcb30@ipxe.org>
>>> Cc: Oliver Steffen <osteffen@redhat.com>
>>> Cc: Michael Kubacki <michael.kubacki@microsoft.com>
>>>
>>> Bugzilla: https://bugzilla.tianocore.org/show_bug.cgi?id=4250
>>> Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
>>> ---
>>>  OvmfPkg/PlatformCI/.azurepipelines/Windows-VS2019.yml |  2 +-
>>>  OvmfPkg/PlatformCI/PlatformBuildLib.py                | 12 ++++++++++++
>>>  2 files changed, 13 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/OvmfPkg/PlatformCI/.azurepipelines/Windows-VS2019.yml b/OvmfPkg/PlatformCI/.azurepipelines/Windows-VS2019.yml
>>> index 7e63f419b26b..b3b91aa84ea0 100644
>>> --- a/OvmfPkg/PlatformCI/.azurepipelines/Windows-VS2019.yml
>>> +++ b/OvmfPkg/PlatformCI/.azurepipelines/Windows-VS2019.yml
>>> @@ -24,7 +24,7 @@ jobs:
>>>        package: 'OvmfPkg'
>>>        vm_image: 'windows-2019'
>>>        should_run: true
>>> -      run_flags: "MAKE_STARTUP_NSH=TRUE QEMU_HEADLESS=TRUE"
>>> +      run_flags: "MAKE_STARTUP_NSH=TRUE QEMU_HEADLESS=TRUE QEMU_CPUHP_QUIRK=TRUE"
>>>
>>>      #Use matrix to speed up the build process
>>>      strategy:
>>> diff --git a/OvmfPkg/PlatformCI/PlatformBuildLib.py b/OvmfPkg/PlatformCI/PlatformBuildLib.py
>>> index bfef9849c749..58dc1189a2cc 100644
>>> --- a/OvmfPkg/PlatformCI/PlatformBuildLib.py
>>> +++ b/OvmfPkg/PlatformCI/PlatformBuildLib.py
>>> @@ -170,6 +170,7 @@ class PlatformBuilder( UefiBuilder, BuildSettingsManager):
>>>          self.env.SetValue("PRODUCT_NAME", "OVMF", "Platform Hardcoded")
>>>          self.env.SetValue("MAKE_STARTUP_NSH", "FALSE", "Default to false")
>>>          self.env.SetValue("QEMU_HEADLESS", "FALSE", "Default to false")
>>> +        self.env.SetValue("QEMU_CPUHP_QUIRK", "FALSE", "Default to false")
>>>          return 0
>>>
>>>      def PlatformPreBuild(self):
>>> @@ -211,6 +212,17 @@ class PlatformBuilder( UefiBuilder, BuildSettingsManager):
>>>              args += " -pflash " + os.path.join(OutputPath_FV, "OVMF.fd")    # path to firmware
>>>
>>>
>>> +        ###
>>> +        ### NOTE This is a temporary workaround to allow platform CI to cope with
>>> +        ###      a QEMU bug in the CPU hotplug code. Once the CI environment has
>>> +        ###      been updated to carry a fixed version of QEMU, this can be
>>> +        ###      removed again
>>> +        ###
>>> +        ### Bugzilla: https://bugzilla.tianocore.org/show_bug.cgi?id=4250
>>> +        ###
>>> +        if (self.env.GetValue("QEMU_CPUHP_QUIRK").upper() == "TRUE"):
>>> +            args += "  -fw_cfg name=opt/org.tianocore/X-Cpuhp-Bugcheck-Override,string=yes"
>>> +
>>>          if (self.env.GetValue("MAKE_STARTUP_NSH").upper() == "TRUE"):
>>>              f = open(os.path.join(VirtualDrive, "startup.nsh"), "w")
>>>              f.write("BOOT SUCCESS !!! \n")
>>
>> Reviewed-by: Laszlo Ersek <lersek@redhat.com>
>>
>> Technically speaking, can I merge this *prepended* to my v3 patch set
>> ([PATCH v3 0/2] OvmfPkg/PlatformInitLib: catch QEMU's CPU hotplug reg
>> block regression), in the *same* PR?
>>
>> Because then I'll do that, saving us a bit of duplicated work.
>>
> 
> Yes, please go ahead and merge this if it is all ready to go. I won't
> get around to it to later this afternoon.
> 
>> I'll then also file the BZ for reverting this (once I know the commit
>> hash), for when a new QEMU is out for Windows -- whom should I assign
>> that BZ? Oliver or Ard?
>>
> 
> You can assign it to me. And please put Michael Kubacki on cc as well.
> 

https://bugzilla.tianocore.org/show_bug.cgi?id=4324

(I couldn't add Oliver to the CC list; AIUI, he's not managed to get a
BZ account just yet.)

Thanks!
Laszlo