[PATCH v2 1/2] ArmVirtPkg/ArmPlatformLibQemu: Ensure that VFP is on before running C code

[PATCH v2 1/2] ArmVirtPkg/ArmPlatformLibQemu: Ensure that VFP is on before running C code

[Ard Biesheuvel]

Now that we build the early code without strict alignment and without
suppressing the use of SIMD registers, ensure that the VFP unit is on
before entering C code.

While at it, simplyify the mov_i macro, which is only used for 32-bit
quantities.

Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
---
 ArmVirtPkg/Library/ArmPlatformLibQemu/AArch64/ArmPlatformHelper.S | 12 +++++-------
 1 file changed, 5 insertions(+), 7 deletions(-)

diff --git a/ArmVirtPkg/Library/ArmPlatformLibQemu/AArch64/ArmPlatformHelper.S b/ArmVirtPkg/Library/ArmPlatformLibQemu/AArch64/ArmPlatformHelper.S
index 05ccc7f9f043..1787d52fbf51 100644
--- a/ArmVirtPkg/Library/ArmPlatformLibQemu/AArch64/ArmPlatformHelper.S
+++ b/ArmVirtPkg/Library/ArmPlatformLibQemu/AArch64/ArmPlatformHelper.S
@@ -8,9 +8,7 @@
 #include <AsmMacroIoLibV8.h>
 
   .macro mov_i, reg:req, imm:req
-  movz   \reg, :abs_g3:\imm
-  movk   \reg, :abs_g2_nc:\imm
-  movk   \reg, :abs_g1_nc:\imm
+  movz   \reg, :abs_g1:\imm
   movk   \reg, :abs_g0_nc:\imm
   .endm
 
@@ -45,10 +43,9 @@
 
 ASM_FUNC(ArmPlatformPeiBootAction)
   mrs    x0, CurrentEL           // check current exception level
-  tbz    x0, #3, 0f              // bail if above EL1
-  ret
+  tbnz   x0, #3, 0f              // omit early ID map if above EL1
 
-0:mov_i  x0, mairval
+  mov_i  x0, mairval
   mov_i  x1, tcrval
   adrp   x2, idmap
   orr    x2, x2, #0xff << 48     // set non-zero ASID
@@ -87,7 +84,8 @@ ASM_FUNC(ArmPlatformPeiBootAction)
 
   msr    sctlr_el1, x3           // enable MMU and caches
   isb
-  ret
+
+0:b      ArmEnableVFP            // enable SIMD before entering C code
 
 //UINTN
 //ArmPlatformGetCorePosition (
-- 
2.39.0

[PATCH v2 2/2] ArmVirtPkg/ArmVirtQemu: Avoid early ID map on ThunderX

[Ard Biesheuvel]

The early ID map used by ArmVirtQemu uses ASID scoped non-global
mappings, as this allows us to switch to the permanent ID map seamlessly
without the need for explicit TLB maintenance.

However, this triggers a known erratum on ThunderX, which does not
tolerate non-global mappings that are executable at EL1, as this appears
to result in I-cache corruption. (Linux disables the KPTI based Meltdown
mitigation on ThunderX for the same reason)

So work around this, by detecting the CPU implementor and part number,
and proceeding without the early ID map if a ThunderX CPU is detected.

Note that this requires the C code to be built with strict alignment
again, as we may end up executing it with the MMU and caches off.

Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
---
 ArmVirtPkg/ArmVirtQemu.dsc                                        |  5 +++++
 ArmVirtPkg/Library/ArmPlatformLibQemu/AArch64/ArmPlatformHelper.S | 15 +++++++++++++++
 2 files changed, 20 insertions(+)

diff --git a/ArmVirtPkg/ArmVirtQemu.dsc b/ArmVirtPkg/ArmVirtQemu.dsc
index f77443229e8e..5dd8b6104cca 100644
--- a/ArmVirtPkg/ArmVirtQemu.dsc
+++ b/ArmVirtPkg/ArmVirtQemu.dsc
@@ -31,6 +31,7 @@ [Defines]
   DEFINE SECURE_BOOT_ENABLE      = FALSE
   DEFINE TPM2_ENABLE             = FALSE
   DEFINE TPM2_CONFIG_ENABLE      = FALSE
+  DEFINE CAVIUM_ERRATUM_27456    = FALSE
 
   #
   # Network definition
@@ -117,7 +118,11 @@ [LibraryClasses.common.UEFI_DRIVER]
   UefiScsiLib|MdePkg/Library/UefiScsiLib/UefiScsiLib.inf
 
 [BuildOptions]
+!if $(CAVIUM_ERRATUM_27456) == TRUE
+  GCC:*_*_AARCH64_PP_FLAGS = -DCAVIUM_ERRATUM_27456
+!else
   GCC:*_*_AARCH64_CC_XIPFLAGS ==
+!endif
 
 !include NetworkPkg/NetworkBuildOptions.dsc.inc
 
diff --git a/ArmVirtPkg/Library/ArmPlatformLibQemu/AArch64/ArmPlatformHelper.S b/ArmVirtPkg/Library/ArmPlatformLibQemu/AArch64/ArmPlatformHelper.S
index 1787d52fbf51..5ac7c732f6ec 100644
--- a/ArmVirtPkg/Library/ArmPlatformLibQemu/AArch64/ArmPlatformHelper.S
+++ b/ArmVirtPkg/Library/ArmPlatformLibQemu/AArch64/ArmPlatformHelper.S
@@ -42,6 +42,21 @@
 
 
 ASM_FUNC(ArmPlatformPeiBootAction)
+#ifdef CAVIUM_ERRATUM_27456
+  /*
+   * On Cavium ThunderX, using non-global mappings that are executable at EL1
+   * results in I-cache corruption. So just avoid the early ID mapping there.
+   *
+   * MIDR implementor   0x43
+   * MIDR part numbers  0xA1 0xA2 (but not 0xAF)
+   */
+  mrs    x0, midr_el1            // read the MIDR into X0
+  ubfx   x1, x0, #24, #8         // grab implementor id
+  ubfx   x0, x0, #7, #9          // grab part number bits [11:3]
+  cmp    x1, #0x43               // compare implementor id
+  ccmp   x0, #0xA0 >> 3, #0, eq  // compare part# bits [11:3]
+  b.eq   0f
+#endif
   mrs    x0, CurrentEL           // check current exception level
   tbnz   x0, #3, 0f              // omit early ID map if above EL1
 
-- 
2.39.0

Re: [PATCH v2 2/2] ArmVirtPkg/ArmVirtQemu: Avoid early ID map on ThunderX

[dann frazier]

On Thu, Jan 05, 2023 at 05:25:28PM +0100, Ard Biesheuvel wrote:
> The early ID map used by ArmVirtQemu uses ASID scoped non-global
> mappings, as this allows us to switch to the permanent ID map seamlessly
> without the need for explicit TLB maintenance.
> 
> However, this triggers a known erratum on ThunderX, which does not
> tolerate non-global mappings that are executable at EL1, as this appears
> to result in I-cache corruption. (Linux disables the KPTI based Meltdown
> mitigation on ThunderX for the same reason)
> 
> So work around this, by detecting the CPU implementor and part number,
> and proceeding without the early ID map if a ThunderX CPU is detected.
> 
> Note that this requires the C code to be built with strict alignment
> again, as we may end up executing it with the MMU and caches off.
> 
> Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
> ---
>  ArmVirtPkg/ArmVirtQemu.dsc                                        |  5 +++++
>  ArmVirtPkg/Library/ArmPlatformLibQemu/AArch64/ArmPlatformHelper.S | 15 +++++++++++++++
>  2 files changed, 20 insertions(+)

FTR, this v2 series also worked for me.

  -dann

> diff --git a/ArmVirtPkg/ArmVirtQemu.dsc b/ArmVirtPkg/ArmVirtQemu.dsc
> index f77443229e8e..5dd8b6104cca 100644
> --- a/ArmVirtPkg/ArmVirtQemu.dsc
> +++ b/ArmVirtPkg/ArmVirtQemu.dsc
> @@ -31,6 +31,7 @@ [Defines]
>    DEFINE SECURE_BOOT_ENABLE      = FALSE
>    DEFINE TPM2_ENABLE             = FALSE
>    DEFINE TPM2_CONFIG_ENABLE      = FALSE
> +  DEFINE CAVIUM_ERRATUM_27456    = FALSE
>  
>    #
>    # Network definition
> @@ -117,7 +118,11 @@ [LibraryClasses.common.UEFI_DRIVER]
>    UefiScsiLib|MdePkg/Library/UefiScsiLib/UefiScsiLib.inf
>  
>  [BuildOptions]
> +!if $(CAVIUM_ERRATUM_27456) == TRUE
> +  GCC:*_*_AARCH64_PP_FLAGS = -DCAVIUM_ERRATUM_27456
> +!else
>    GCC:*_*_AARCH64_CC_XIPFLAGS ==
> +!endif
>  
>  !include NetworkPkg/NetworkBuildOptions.dsc.inc
>  
> diff --git a/ArmVirtPkg/Library/ArmPlatformLibQemu/AArch64/ArmPlatformHelper.S b/ArmVirtPkg/Library/ArmPlatformLibQemu/AArch64/ArmPlatformHelper.S
> index 1787d52fbf51..5ac7c732f6ec 100644
> --- a/ArmVirtPkg/Library/ArmPlatformLibQemu/AArch64/ArmPlatformHelper.S
> +++ b/ArmVirtPkg/Library/ArmPlatformLibQemu/AArch64/ArmPlatformHelper.S
> @@ -42,6 +42,21 @@
>  
>  
>  ASM_FUNC(ArmPlatformPeiBootAction)
> +#ifdef CAVIUM_ERRATUM_27456
> +  /*
> +   * On Cavium ThunderX, using non-global mappings that are executable at EL1
> +   * results in I-cache corruption. So just avoid the early ID mapping there.
> +   *
> +   * MIDR implementor   0x43
> +   * MIDR part numbers  0xA1 0xA2 (but not 0xAF)
> +   */
> +  mrs    x0, midr_el1            // read the MIDR into X0
> +  ubfx   x1, x0, #24, #8         // grab implementor id
> +  ubfx   x0, x0, #7, #9          // grab part number bits [11:3]
> +  cmp    x1, #0x43               // compare implementor id
> +  ccmp   x0, #0xA0 >> 3, #0, eq  // compare part# bits [11:3]
> +  b.eq   0f
> +#endif
>    mrs    x0, CurrentEL           // check current exception level
>    tbnz   x0, #3, 0f              // omit early ID map if above EL1
>  

Re: [edk2-devel] [PATCH v2 2/2] ArmVirtPkg/ArmVirtQemu: Avoid early ID map on ThunderX

[Oliver Steffen]

[-- Attachment #1: Type: text/plain, Size: 4405 bytes --]

Hi Ard, Hi everyone,

Thanks for the work!

But somehow this patch (as it was merged into master branch) does not
work for me on the ThunderX box we have.

Any idea what could be wrong?
I enabled the erratum during build ;-)

CPU Info:
# lscpu
Architecture:           aarch64
  CPU op-mode(s):       64-bit
  Byte Order:           Little Endian
CPU(s):                 224
  On-line CPU(s) list:  0-223
Vendor ID:              Cavium
  BIOS Vendor ID:       Cavium Inc.
  Model name:           ThunderX2 99xx
    BIOS Model name:    Cavium ThunderX2(R) CPU CN9975 v2.2 @ 2.0GHz
    Model:              2
    Thread(s) per core: 4
    Core(s) per socket: 28
    Socket(s):          2
    Stepping:           0x1
    BogoMIPS:           400.00
    Flags:              fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics
cpuid asimdrdm
Caches (sum of all):
  L1d:                  1.8 MiB (56 instances)
  L1i:                  1.8 MiB (56 instances)
  L2:                   14 MiB (56 instances)
  L3:                   64 MiB (2 instances)
[...]

Thanks a lot!
- Oliver


On Tue, Jan 10, 2023 at 1:08 AM dann frazier <dann.frazier@canonical.com>
wrote:

> On Thu, Jan 05, 2023 at 05:25:28PM +0100, Ard Biesheuvel wrote:
> > The early ID map used by ArmVirtQemu uses ASID scoped non-global
> > mappings, as this allows us to switch to the permanent ID map seamlessly
> > without the need for explicit TLB maintenance.
> >
> > However, this triggers a known erratum on ThunderX, which does not
> > tolerate non-global mappings that are executable at EL1, as this appears
> > to result in I-cache corruption. (Linux disables the KPTI based Meltdown
> > mitigation on ThunderX for the same reason)
> >
> > So work around this, by detecting the CPU implementor and part number,
> > and proceeding without the early ID map if a ThunderX CPU is detected.
> >
> > Note that this requires the C code to be built with strict alignment
> > again, as we may end up executing it with the MMU and caches off.
> >
> > Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
> > ---
> >  ArmVirtPkg/ArmVirtQemu.dsc                                        |  5
> +++++
> >  ArmVirtPkg/Library/ArmPlatformLibQemu/AArch64/ArmPlatformHelper.S | 15
> +++++++++++++++
> >  2 files changed, 20 insertions(+)
>
> FTR, this v2 series also worked for me.
>
>   -dann
>
> > diff --git a/ArmVirtPkg/ArmVirtQemu.dsc b/ArmVirtPkg/ArmVirtQemu.dsc
> > index f77443229e8e..5dd8b6104cca 100644
> > --- a/ArmVirtPkg/ArmVirtQemu.dsc
> > +++ b/ArmVirtPkg/ArmVirtQemu.dsc
> > @@ -31,6 +31,7 @@ [Defines]
> >    DEFINE SECURE_BOOT_ENABLE      = FALSE
> >    DEFINE TPM2_ENABLE             = FALSE
> >    DEFINE TPM2_CONFIG_ENABLE      = FALSE
> > +  DEFINE CAVIUM_ERRATUM_27456    = FALSE
> >
> >    #
> >    # Network definition
> > @@ -117,7 +118,11 @@ [LibraryClasses.common.UEFI_DRIVER]
> >    UefiScsiLib|MdePkg/Library/UefiScsiLib/UefiScsiLib.inf
> >
> >  [BuildOptions]
> > +!if $(CAVIUM_ERRATUM_27456) == TRUE
> > +  GCC:*_*_AARCH64_PP_FLAGS = -DCAVIUM_ERRATUM_27456
> > +!else
> >    GCC:*_*_AARCH64_CC_XIPFLAGS ==
> > +!endif
> >
> >  !include NetworkPkg/NetworkBuildOptions.dsc.inc
> >
> > diff --git
> a/ArmVirtPkg/Library/ArmPlatformLibQemu/AArch64/ArmPlatformHelper.S
> b/ArmVirtPkg/Library/ArmPlatformLibQemu/AArch64/ArmPlatformHelper.S
> > index 1787d52fbf51..5ac7c732f6ec 100644
> > --- a/ArmVirtPkg/Library/ArmPlatformLibQemu/AArch64/ArmPlatformHelper.S
> > +++ b/ArmVirtPkg/Library/ArmPlatformLibQemu/AArch64/ArmPlatformHelper.S
> > @@ -42,6 +42,21 @@
> >
> >
> >  ASM_FUNC(ArmPlatformPeiBootAction)
> > +#ifdef CAVIUM_ERRATUM_27456
> > +  /*
> > +   * On Cavium ThunderX, using non-global mappings that are executable
> at EL1
> > +   * results in I-cache corruption. So just avoid the early ID mapping
> there.
> > +   *
> > +   * MIDR implementor   0x43
> > +   * MIDR part numbers  0xA1 0xA2 (but not 0xAF)
> > +   */
> > +  mrs    x0, midr_el1            // read the MIDR into X0
> > +  ubfx   x1, x0, #24, #8         // grab implementor id
> > +  ubfx   x0, x0, #7, #9          // grab part number bits [11:3]
> > +  cmp    x1, #0x43               // compare implementor id
> > +  ccmp   x0, #0xA0 >> 3, #0, eq  // compare part# bits [11:3]
> > +  b.eq   0f
> > +#endif
> >    mrs    x0, CurrentEL           // check current exception level
> >    tbnz   x0, #3, 0f              // omit early ID map if above EL1
> >
>
>
> 
>
>
>

[-- Attachment #2: Type: text/html, Size: 6463 bytes --]

Re: [edk2-devel] [PATCH v2 2/2] ArmVirtPkg/ArmVirtQemu: Avoid early ID map on ThunderX

[Ard Biesheuvel]

On Tue, 17 Jan 2023 at 13:48, Oliver Steffen <osteffen@redhat.com> wrote:
>
> Hi Ard, Hi everyone,
>
> Thanks for the work!
>
> But somehow this patch (as it was merged into master branch) does not
> work for me on the ThunderX box we have.
>
> Any idea what could be wrong?

I'm not sure I understand the question. The patch targets ThunderX,
and you are using a ThunderX2.

What were you expecting to happen, and what is happening instead?


> I enabled the erratum during build ;-)
>
> CPU Info:
> # lscpu
> Architecture:           aarch64
>   CPU op-mode(s):       64-bit
>   Byte Order:           Little Endian
> CPU(s):                 224
>   On-line CPU(s) list:  0-223
> Vendor ID:              Cavium
>   BIOS Vendor ID:       Cavium Inc.
>   Model name:           ThunderX2 99xx
>     BIOS Model name:    Cavium ThunderX2(R) CPU CN9975 v2.2 @ 2.0GHz
>     Model:              2
>     Thread(s) per core: 4
>     Core(s) per socket: 28
>     Socket(s):          2
>     Stepping:           0x1
>     BogoMIPS:           400.00
>     Flags:              fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics cpuid asimdrdm
> Caches (sum of all):
>   L1d:                  1.8 MiB (56 instances)
>   L1i:                  1.8 MiB (56 instances)
>   L2:                   14 MiB (56 instances)
>   L3:                   64 MiB (2 instances)
> [...]
>
> Thanks a lot!
> - Oliver
>
>
> On Tue, Jan 10, 2023 at 1:08 AM dann frazier <dann.frazier@canonical.com> wrote:
>>
>> On Thu, Jan 05, 2023 at 05:25:28PM +0100, Ard Biesheuvel wrote:
>> > The early ID map used by ArmVirtQemu uses ASID scoped non-global
>> > mappings, as this allows us to switch to the permanent ID map seamlessly
>> > without the need for explicit TLB maintenance.
>> >
>> > However, this triggers a known erratum on ThunderX, which does not
>> > tolerate non-global mappings that are executable at EL1, as this appears
>> > to result in I-cache corruption. (Linux disables the KPTI based Meltdown
>> > mitigation on ThunderX for the same reason)
>> >
>> > So work around this, by detecting the CPU implementor and part number,
>> > and proceeding without the early ID map if a ThunderX CPU is detected.
>> >
>> > Note that this requires the C code to be built with strict alignment
>> > again, as we may end up executing it with the MMU and caches off.
>> >
>> > Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
>> > ---
>> >  ArmVirtPkg/ArmVirtQemu.dsc                                        |  5 +++++
>> >  ArmVirtPkg/Library/ArmPlatformLibQemu/AArch64/ArmPlatformHelper.S | 15 +++++++++++++++
>> >  2 files changed, 20 insertions(+)
>>
>> FTR, this v2 series also worked for me.
>>
>>   -dann
>>
>> > diff --git a/ArmVirtPkg/ArmVirtQemu.dsc b/ArmVirtPkg/ArmVirtQemu.dsc
>> > index f77443229e8e..5dd8b6104cca 100644
>> > --- a/ArmVirtPkg/ArmVirtQemu.dsc
>> > +++ b/ArmVirtPkg/ArmVirtQemu.dsc
>> > @@ -31,6 +31,7 @@ [Defines]
>> >    DEFINE SECURE_BOOT_ENABLE      = FALSE
>> >    DEFINE TPM2_ENABLE             = FALSE
>> >    DEFINE TPM2_CONFIG_ENABLE      = FALSE
>> > +  DEFINE CAVIUM_ERRATUM_27456    = FALSE
>> >
>> >    #
>> >    # Network definition
>> > @@ -117,7 +118,11 @@ [LibraryClasses.common.UEFI_DRIVER]
>> >    UefiScsiLib|MdePkg/Library/UefiScsiLib/UefiScsiLib.inf
>> >
>> >  [BuildOptions]
>> > +!if $(CAVIUM_ERRATUM_27456) == TRUE
>> > +  GCC:*_*_AARCH64_PP_FLAGS = -DCAVIUM_ERRATUM_27456
>> > +!else
>> >    GCC:*_*_AARCH64_CC_XIPFLAGS ==
>> > +!endif
>> >
>> >  !include NetworkPkg/NetworkBuildOptions.dsc.inc
>> >
>> > diff --git a/ArmVirtPkg/Library/ArmPlatformLibQemu/AArch64/ArmPlatformHelper.S b/ArmVirtPkg/Library/ArmPlatformLibQemu/AArch64/ArmPlatformHelper.S
>> > index 1787d52fbf51..5ac7c732f6ec 100644
>> > --- a/ArmVirtPkg/Library/ArmPlatformLibQemu/AArch64/ArmPlatformHelper.S
>> > +++ b/ArmVirtPkg/Library/ArmPlatformLibQemu/AArch64/ArmPlatformHelper.S
>> > @@ -42,6 +42,21 @@
>> >
>> >
>> >  ASM_FUNC(ArmPlatformPeiBootAction)
>> > +#ifdef CAVIUM_ERRATUM_27456
>> > +  /*
>> > +   * On Cavium ThunderX, using non-global mappings that are executable at EL1
>> > +   * results in I-cache corruption. So just avoid the early ID mapping there.
>> > +   *
>> > +   * MIDR implementor   0x43
>> > +   * MIDR part numbers  0xA1 0xA2 (but not 0xAF)
>> > +   */
>> > +  mrs    x0, midr_el1            // read the MIDR into X0
>> > +  ubfx   x1, x0, #24, #8         // grab implementor id
>> > +  ubfx   x0, x0, #7, #9          // grab part number bits [11:3]
>> > +  cmp    x1, #0x43               // compare implementor id
>> > +  ccmp   x0, #0xA0 >> 3, #0, eq  // compare part# bits [11:3]
>> > +  b.eq   0f
>> > +#endif
>> >    mrs    x0, CurrentEL           // check current exception level
>> >    tbnz   x0, #3, 0f              // omit early ID map if above EL1
>> >
>>
>>
>>
>>
>>
> 

Re: [edk2-devel] [PATCH v2 2/2] ArmVirtPkg/ArmVirtQemu: Avoid early ID map on ThunderX

[Oliver Steffen]

[-- Attachment #1: Type: text/plain, Size: 5508 bytes --]

On Tue, Jan 17, 2023 at 3:57 PM Ard Biesheuvel <ardb@kernel.org> wrote:

> On Tue, 17 Jan 2023 at 13:48, Oliver Steffen <osteffen@redhat.com> wrote:
> >
> > Hi Ard, Hi everyone,
> >
> > Thanks for the work!
> >
> > But somehow this patch (as it was merged into master branch) does not
> > work for me on the ThunderX box we have.
> >
> > Any idea what could be wrong?
>
> I'm not sure I understand the question. The patch targets ThunderX,
> and you are using a ThunderX2.
>
> What were you expecting to happen, and what is happening instead?


Firmware does not start at all when using KVM.

Please excuse my limited knowledge of Arm processor variants.
I assumed that ThunderX and ThunderX2 are very similar and hoped
the fix would also work for this case.

The issue was introduced by the same commit that Dann
reported (07be1d34d95460a238fcd0f6693efb747c28b329):
"ArmVirtPkg/ArmVirtQemu: enable initial ID map at early boot".

> I enabled the erratum during build ;-)
> >
> > CPU Info:
> > # lscpu
> > Architecture:           aarch64
> >   CPU op-mode(s):       64-bit
> >   Byte Order:           Little Endian
> > CPU(s):                 224
> >   On-line CPU(s) list:  0-223
> > Vendor ID:              Cavium
> >   BIOS Vendor ID:       Cavium Inc.
> >   Model name:           ThunderX2 99xx
> >     BIOS Model name:    Cavium ThunderX2(R) CPU CN9975 v2.2 @ 2.0GHz
> >     Model:              2
> >     Thread(s) per core: 4
> >     Core(s) per socket: 28
> >     Socket(s):          2
> >     Stepping:           0x1
> >     BogoMIPS:           400.00
> >     Flags:              fp asimd evtstrm aes pmull sha1 sha2 crc32
> atomics cpuid asimdrdm
> > Caches (sum of all):
> >   L1d:                  1.8 MiB (56 instances)
> >   L1i:                  1.8 MiB (56 instances)
> >   L2:                   14 MiB (56 instances)
> >   L3:                   64 MiB (2 instances)
> > [...]
> >
> > Thanks a lot!
> > - Oliver
> >
> >
> > On Tue, Jan 10, 2023 at 1:08 AM dann frazier <dann.frazier@canonical.com>
> wrote:
> >>
> >> On Thu, Jan 05, 2023 at 05:25:28PM +0100, Ard Biesheuvel wrote:
> >> > The early ID map used by ArmVirtQemu uses ASID scoped non-global
> >> > mappings, as this allows us to switch to the permanent ID map
> seamlessly
> >> > without the need for explicit TLB maintenance.
> >> >
> >> > However, this triggers a known erratum on ThunderX, which does not
> >> > tolerate non-global mappings that are executable at EL1, as this
> appears
> >> > to result in I-cache corruption. (Linux disables the KPTI based
> Meltdown
> >> > mitigation on ThunderX for the same reason)
> >> >
> >> > So work around this, by detecting the CPU implementor and part number,
> >> > and proceeding without the early ID map if a ThunderX CPU is detected.
> >> >
> >> > Note that this requires the C code to be built with strict alignment
> >> > again, as we may end up executing it with the MMU and caches off.
> >> >
> >> > Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
> >> > ---
> >> >  ArmVirtPkg/ArmVirtQemu.dsc                                        |
> 5 +++++
> >> >  ArmVirtPkg/Library/ArmPlatformLibQemu/AArch64/ArmPlatformHelper.S |
> 15 +++++++++++++++
> >> >  2 files changed, 20 insertions(+)
> >>
> >> FTR, this v2 series also worked for me.
> >>
> >>   -dann
> >>
> >> > diff --git a/ArmVirtPkg/ArmVirtQemu.dsc b/ArmVirtPkg/ArmVirtQemu.dsc
> >> > index f77443229e8e..5dd8b6104cca 100644
> >> > --- a/ArmVirtPkg/ArmVirtQemu.dsc
> >> > +++ b/ArmVirtPkg/ArmVirtQemu.dsc
> >> > @@ -31,6 +31,7 @@ [Defines]
> >> >    DEFINE SECURE_BOOT_ENABLE      = FALSE
> >> >    DEFINE TPM2_ENABLE             = FALSE
> >> >    DEFINE TPM2_CONFIG_ENABLE      = FALSE
> >> > +  DEFINE CAVIUM_ERRATUM_27456    = FALSE
> >> >
> >> >    #
> >> >    # Network definition
> >> > @@ -117,7 +118,11 @@ [LibraryClasses.common.UEFI_DRIVER]
> >> >    UefiScsiLib|MdePkg/Library/UefiScsiLib/UefiScsiLib.inf
> >> >
> >> >  [BuildOptions]
> >> > +!if $(CAVIUM_ERRATUM_27456) == TRUE
> >> > +  GCC:*_*_AARCH64_PP_FLAGS = -DCAVIUM_ERRATUM_27456
> >> > +!else
> >> >    GCC:*_*_AARCH64_CC_XIPFLAGS ==
> >> > +!endif
> >> >
> >> >  !include NetworkPkg/NetworkBuildOptions.dsc.inc
> >> >
> >> > diff --git
> a/ArmVirtPkg/Library/ArmPlatformLibQemu/AArch64/ArmPlatformHelper.S
> b/ArmVirtPkg/Library/ArmPlatformLibQemu/AArch64/ArmPlatformHelper.S
> >> > index 1787d52fbf51..5ac7c732f6ec 100644
> >> > ---
> a/ArmVirtPkg/Library/ArmPlatformLibQemu/AArch64/ArmPlatformHelper.S
> >> > +++
> b/ArmVirtPkg/Library/ArmPlatformLibQemu/AArch64/ArmPlatformHelper.S
> >> > @@ -42,6 +42,21 @@
> >> >
> >> >
> >> >  ASM_FUNC(ArmPlatformPeiBootAction)
> >> > +#ifdef CAVIUM_ERRATUM_27456
> >> > +  /*
> >> > +   * On Cavium ThunderX, using non-global mappings that are
> executable at EL1
> >> > +   * results in I-cache corruption. So just avoid the early ID
> mapping there.
> >> > +   *
> >> > +   * MIDR implementor   0x43
> >> > +   * MIDR part numbers  0xA1 0xA2 (but not 0xAF)
> >> > +   */
> >> > +  mrs    x0, midr_el1            // read the MIDR into X0
> >> > +  ubfx   x1, x0, #24, #8         // grab implementor id
> >> > +  ubfx   x0, x0, #7, #9          // grab part number bits [11:3]
> >> > +  cmp    x1, #0x43               // compare implementor id
> >> > +  ccmp   x0, #0xA0 >> 3, #0, eq  // compare part# bits [11:3]
> >> > +  b.eq   0f
> >> > +#endif
> >> >    mrs    x0, CurrentEL           // check current exception level
> >> >    tbnz   x0, #3, 0f              // omit early ID map if above EL1
> >> >
>
>

[-- Attachment #2: Type: text/html, Size: 8163 bytes --]

Re: [edk2-devel] [PATCH v2 2/2] ArmVirtPkg/ArmVirtQemu: Avoid early ID map on ThunderX

[Ard Biesheuvel]

On Wed, 18 Jan 2023 at 07:37, Oliver Steffen <osteffen@redhat.com> wrote:
>
> On Tue, Jan 17, 2023 at 3:57 PM Ard Biesheuvel <ardb@kernel.org> wrote:
>>
>> On Tue, 17 Jan 2023 at 13:48, Oliver Steffen <osteffen@redhat.com> wrote:
>> >
>> > Hi Ard, Hi everyone,
>> >
>> > Thanks for the work!
>> >
>> > But somehow this patch (as it was merged into master branch) does not
>> > work for me on the ThunderX box we have.
>> >
>> > Any idea what could be wrong?
>>
>> I'm not sure I understand the question. The patch targets ThunderX,
>> and you are using a ThunderX2.
>>
>> What were you expecting to happen, and what is happening instead?
>
>
> Firmware does not start at all when using KVM.
>
> Please excuse my limited knowledge of Arm processor variants.
> I assumed that ThunderX and ThunderX2 are very similar and hoped
> the fix would also work for this case.
>
> The issue was introduced by the same commit that Dann
> reported (07be1d34d95460a238fcd0f6693efb747c28b329):
> "ArmVirtPkg/ArmVirtQemu: enable initial ID map at early boot".
>

Can you share the QEMU command line that you are using? I use a
ThunderX2 basically 24/7 to do all my Linux and EDK2 development, so
this change was developed on ThunderX2 and so I'm surprised you are
seeing this issue.

Did you try the DEBUG build as well?

Re: [edk2-devel] [PATCH v2 2/2] ArmVirtPkg/ArmVirtQemu: Avoid early ID map on ThunderX

[Oliver Steffen]

Quoting Ard Biesheuvel (2023-01-18 08:34:32)
> On Wed, 18 Jan 2023 at 07:37, Oliver Steffen <osteffen@redhat.com> wrote:
> >
> > On Tue, Jan 17, 2023 at 3:57 PM Ard Biesheuvel <ardb@kernel.org> wrote:
> >>
> >> On Tue, 17 Jan 2023 at 13:48, Oliver Steffen <osteffen@redhat.com> wrote:
> >> >
> >> > Hi Ard, Hi everyone,
> >> >
> >> > Thanks for the work!
> >> >
> >> > But somehow this patch (as it was merged into master branch) does not
> >> > work for me on the ThunderX box we have.
> >> >
> >> > Any idea what could be wrong?
> >>
> >> I'm not sure I understand the question. The patch targets ThunderX,
> >> and you are using a ThunderX2.
> >>
> >> What were you expecting to happen, and what is happening instead?
> >
> >
> > Firmware does not start at all when using KVM.
> >
> > Please excuse my limited knowledge of Arm processor variants.
> > I assumed that ThunderX and ThunderX2 are very similar and hoped
> > the fix would also work for this case.
> >
> > The issue was introduced by the same commit that Dann
> > reported (07be1d34d95460a238fcd0f6693efb747c28b329):
> > "ArmVirtPkg/ArmVirtQemu: enable initial ID map at early boot".
> >
>
> Can you share the QEMU command line that you are using? I use a
> ThunderX2 basically 24/7 to do all my Linux and EDK2 development, so
> this change was developed on ThunderX2 and so I'm surprised you are
> seeing this issue.
>
> Did you try the DEBUG build as well?
Yes, debug is on.

Here is what I have, trying with the master branch from just now
(998ebe5ca0ae5c449e83ede533bee872f97d63af):

# make -C BaseTools && \
  . ./edksetup.sh && \
  build -t GCC5 -a AARCH64 \
    -p ArmVirtPkg/ArmVirtQemu.dsc \
    -DCAVIUM_ERRATUM_27456 \
    -b DEBUG

# /usr/libexec/qemu-kvm \
    -machine accel=kvm -m 1G -boot menu=on \
    -blockdev node-name=code,driver=file,filename="${FW_CODE_RESIZED}",read-only=on
\
    -blockdev node-name=vars,driver=file,filename="${FW_VARS}" \
    -machine pflash0=code \
    -machine pflash1=vars \
    -cpu max \
    -net none \
    -serial stdio

# /usr/libexec/qemu-kvm --version
QEMU emulator version 7.2.0 (qemu-kvm-7.2.0-3.el9)

# uname -r
5.14.0-234.el9.aarch64

# gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/libexec/gcc/aarch64-redhat-linux/11/lto-wrapper
Target: aarch64-redhat-linux
Configured with: ../configure --enable-bootstrap --enable-host-pie
--enable-host-bind-now --enable-languages=c,c++,fortran,lto
--prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info
--with-bugurl=http://bugzilla.redhat.com/bugzilla --enable-shared
--enable-threads=posix --enable-checking=release --with-system-zlib
--enable-__cxa_atexit --disable-libunwind-exceptions
--enable-gnu-unique-object --enable-linker-build-id
--with-gcc-major-version-only --enable-plugin --enable-initfini-array
--without-isl --enable-multilib --with-linker-hash-style=gnu
--enable-gnu-indirect-function --build=aarch64-redhat-linux
--with-build-config=bootstrap-lto --enable-link-serialization=1
Thread model: posix
Supported LTO compression algorithms: zlib zstd
gcc version 11.3.1 20221121 (Red Hat 11.3.1-4) (GCC)


Since you have the same CPU... Might this be a bug in KVM?

Thanks!
-Oliver

Re: [edk2-devel] [PATCH v2 2/2] ArmVirtPkg/ArmVirtQemu: Avoid early ID map on ThunderX

[Ard Biesheuvel]

On Wed, 18 Jan 2023 at 09:28, Oliver Steffen <osteffen@redhat.com> wrote:
>
> Quoting Ard Biesheuvel (2023-01-18 08:34:32)
> > On Wed, 18 Jan 2023 at 07:37, Oliver Steffen <osteffen@redhat.com> wrote:
> > >
> > > On Tue, Jan 17, 2023 at 3:57 PM Ard Biesheuvel <ardb@kernel.org> wrote:
> > >>
> > >> On Tue, 17 Jan 2023 at 13:48, Oliver Steffen <osteffen@redhat.com> wrote:
> > >> >
> > >> > Hi Ard, Hi everyone,
> > >> >
> > >> > Thanks for the work!
> > >> >
> > >> > But somehow this patch (as it was merged into master branch) does not
> > >> > work for me on the ThunderX box we have.
> > >> >
> > >> > Any idea what could be wrong?
> > >>
> > >> I'm not sure I understand the question. The patch targets ThunderX,
> > >> and you are using a ThunderX2.
> > >>
> > >> What were you expecting to happen, and what is happening instead?
> > >
> > >
> > > Firmware does not start at all when using KVM.
> > >
> > > Please excuse my limited knowledge of Arm processor variants.
> > > I assumed that ThunderX and ThunderX2 are very similar and hoped
> > > the fix would also work for this case.
> > >
> > > The issue was introduced by the same commit that Dann
> > > reported (07be1d34d95460a238fcd0f6693efb747c28b329):
> > > "ArmVirtPkg/ArmVirtQemu: enable initial ID map at early boot".
> > >
> >
> > Can you share the QEMU command line that you are using? I use a
> > ThunderX2 basically 24/7 to do all my Linux and EDK2 development, so
> > this change was developed on ThunderX2 and so I'm surprised you are
> > seeing this issue.
> >
> > Did you try the DEBUG build as well?
> Yes, debug is on.
>
> Here is what I have, trying with the master branch from just now
> (998ebe5ca0ae5c449e83ede533bee872f97d63af):
>
> # make -C BaseTools && \
>   . ./edksetup.sh && \
>   build -t GCC5 -a AARCH64 \
>     -p ArmVirtPkg/ArmVirtQemu.dsc \
>     -DCAVIUM_ERRATUM_27456 \
>     -b DEBUG
>
> # /usr/libexec/qemu-kvm \
>     -machine accel=kvm -m 1G -boot menu=on \
>     -blockdev node-name=code,driver=file,filename="${FW_CODE_RESIZED}",read-only=on
> \
>     -blockdev node-name=vars,driver=file,filename="${FW_VARS}" \
>     -machine pflash0=code \
>     -machine pflash1=vars \
>     -cpu max \
>     -net none \
>     -serial stdio
>

My distro does not have qemu-kvm, and using the command line above
results in the following if i try it with qemu-system-aarch64

"""
qemu-system-aarch64: No machine specified, and there is no default
Use -machine help to list supported machines
"""

unless i change it to

qemu-system-aarch64 -machine virt,accel=kvm -m 1G -boot menu=on \
    -blockdev node-name=code,driver=file,filename=$HOME/bin/flash0.img,read-only=on
\
    -blockdev node-name=vars,driver=file,filename=$HOME/bin/flash1.img \
    -machine pflash0=code \
    -machine pflash1=vars \
    -cpu max \
    -net none \
    -nographic

and that works fine with my firmware build.


> # /usr/libexec/qemu-kvm --version
> QEMU emulator version 7.2.0 (qemu-kvm-7.2.0-3.el9)
>
> # uname -r
> 5.14.0-234.el9.aarch64
>

Yeah, that is quite old. One potential issue that comes to mind here
is the one address by the patch below


>
>
> Since you have the same CPU... Might this be a bug in KVM?
>

Indeed. Could you try applying this patch?

commit 406504c7b0405d74d74c15a667cd4c4620c3e7a9
Author: Marc Zyngier <maz@kernel.org>
Date:   Tue Dec 20 14:03:52 2022 +0000

    KVM: arm64: Fix S1PTW handling on RO memslots

Or check whether this is generally reproducible with newer kernels?

Re: [edk2-devel] [PATCH v2 2/2] ArmVirtPkg/ArmVirtQemu: Avoid early ID map on ThunderX

[Ard Biesheuvel]

On Wed, 18 Jan 2023 at 09:48, Ard Biesheuvel <ardb@kernel.org> wrote:
>
> On Wed, 18 Jan 2023 at 09:28, Oliver Steffen <osteffen@redhat.com> wrote:
> >
> > Quoting Ard Biesheuvel (2023-01-18 08:34:32)
> > > On Wed, 18 Jan 2023 at 07:37, Oliver Steffen <osteffen@redhat.com> wrote:
> > > >
> > > > On Tue, Jan 17, 2023 at 3:57 PM Ard Biesheuvel <ardb@kernel.org> wrote:
> > > >>
> > > >> On Tue, 17 Jan 2023 at 13:48, Oliver Steffen <osteffen@redhat.com> wrote:
> > > >> >
> > > >> > Hi Ard, Hi everyone,
> > > >> >
> > > >> > Thanks for the work!
> > > >> >
> > > >> > But somehow this patch (as it was merged into master branch) does not
> > > >> > work for me on the ThunderX box we have.
> > > >> >
> > > >> > Any idea what could be wrong?
> > > >>
> > > >> I'm not sure I understand the question. The patch targets ThunderX,
> > > >> and you are using a ThunderX2.
> > > >>
> > > >> What were you expecting to happen, and what is happening instead?
> > > >
> > > >
> > > > Firmware does not start at all when using KVM.
> > > >
> > > > Please excuse my limited knowledge of Arm processor variants.
> > > > I assumed that ThunderX and ThunderX2 are very similar and hoped
> > > > the fix would also work for this case.
> > > >
> > > > The issue was introduced by the same commit that Dann
> > > > reported (07be1d34d95460a238fcd0f6693efb747c28b329):
> > > > "ArmVirtPkg/ArmVirtQemu: enable initial ID map at early boot".
> > > >
> > >
> > > Can you share the QEMU command line that you are using? I use a
> > > ThunderX2 basically 24/7 to do all my Linux and EDK2 development, so
> > > this change was developed on ThunderX2 and so I'm surprised you are
> > > seeing this issue.
> > >
> > > Did you try the DEBUG build as well?
> > Yes, debug is on.
> >
> > Here is what I have, trying with the master branch from just now
> > (998ebe5ca0ae5c449e83ede533bee872f97d63af):
> >
> > # make -C BaseTools && \
> >   . ./edksetup.sh && \
> >   build -t GCC5 -a AARCH64 \
> >     -p ArmVirtPkg/ArmVirtQemu.dsc \
> >     -DCAVIUM_ERRATUM_27456 \
> >     -b DEBUG
> >
> > # /usr/libexec/qemu-kvm \
> >     -machine accel=kvm -m 1G -boot menu=on \
> >     -blockdev node-name=code,driver=file,filename="${FW_CODE_RESIZED}",read-only=on
> > \
> >     -blockdev node-name=vars,driver=file,filename="${FW_VARS}" \
> >     -machine pflash0=code \
> >     -machine pflash1=vars \
> >     -cpu max \
> >     -net none \
> >     -serial stdio
> >
>
> My distro does not have qemu-kvm, and using the command line above
> results in the following if i try it with qemu-system-aarch64
>
> """
> qemu-system-aarch64: No machine specified, and there is no default
> Use -machine help to list supported machines
> """
>
> unless i change it to
>
> qemu-system-aarch64 -machine virt,accel=kvm -m 1G -boot menu=on \
>     -blockdev node-name=code,driver=file,filename=$HOME/bin/flash0.img,read-only=on
> \
>     -blockdev node-name=vars,driver=file,filename=$HOME/bin/flash1.img \
>     -machine pflash0=code \
>     -machine pflash1=vars \
>     -cpu max \
>     -net none \
>     -nographic
>
> and that works fine with my firmware build.
>
>
> > # /usr/libexec/qemu-kvm --version
> > QEMU emulator version 7.2.0 (qemu-kvm-7.2.0-3.el9)
> >
> > # uname -r
> > 5.14.0-234.el9.aarch64
> >
>
> Yeah, that is quite old. One potential issue that comes to mind here
> is the one address by the patch below
>
>
> >
> >
> > Since you have the same CPU... Might this be a bug in KVM?
> >
>
> Indeed. Could you try applying this patch?
>
> commit 406504c7b0405d74d74c15a667cd4c4620c3e7a9
> Author: Marc Zyngier <maz@kernel.org>
> Date:   Tue Dec 20 14:03:52 2022 +0000
>
>     KVM: arm64: Fix S1PTW handling on RO memslots
>
> Or check whether this is generally reproducible with newer kernels?

Another thing you might try:

- build the firmware with the following hunk applied

"""
diff --git a/ArmVirtPkg/Library/ArmPlatformLibQemu/AArch64/ArmPlatformHelper.S
b/ArmVirtPkg/Library/ArmPlatformLibQemu/AArch64/ArmPlatformHelper.S
index 5ac7c732f6ec..f4e1285beefc 100644
--- a/ArmVirtPkg/Library/ArmPlatformLibQemu/AArch64/ArmPlatformHelper.S
+++ b/ArmVirtPkg/Library/ArmPlatformLibQemu/AArch64/ArmPlatformHelper.S
@@ -40,6 +40,12 @@
  .set    sctlrval, SCTLR_ELx_M | SCTLR_ELx_C | SCTLR_ELx_SA |
SCTLR_EL1_ITD | SCTLR_EL1_SED
  .set    sctlrval, sctlrval | SCTLR_ELx_I | SCTLR_EL1_SPAN | SCTLR_EL1_RES1

+  .align  11
+.Lvectors:
+  .rept   16
+  .align  7
+  b       .
+  .endr

 ASM_FUNC(ArmPlatformPeiBootAction)
 #ifdef CAVIUM_ERRATUM_27456
@@ -90,6 +96,8 @@ ASM_FUNC(ArmPlatformPeiBootAction)
   msr    mair_el1, x0            // set up the 1:1 mapping
   msr    tcr_el1, x1
   msr    ttbr0_el1, x2
+  adr    x0, .Lvectors
+  msr    vbar_el1, x0
   isb

   tlbi   vmalle1                 // invalidate any cached translations
"""

- run qemu with the -s option and let it crash

- connect with gdb and dump the exception context

target remote:1234
set radix 16
p $FAR_EL1
p $ESR_EL1
p $ELR_EL1

That should at least tell us why the crash is occurring.

Re: [edk2-devel] [PATCH v2 2/2] ArmVirtPkg/ArmVirtQemu: Avoid early ID map on ThunderX

[Oliver Steffen]

Quoting Ard Biesheuvel (2023-01-18 10:22:12)
> On Wed, 18 Jan 2023 at 09:48, Ard Biesheuvel <ardb@kernel.org> wrote:
> >
> > On Wed, 18 Jan 2023 at 09:28, Oliver Steffen <osteffen@redhat.com> wrote:
> > >
> > > Quoting Ard Biesheuvel (2023-01-18 08:34:32)
> > > > On Wed, 18 Jan 2023 at 07:37, Oliver Steffen <osteffen@redhat.com> wrote:
> > > > >
> > > > > On Tue, Jan 17, 2023 at 3:57 PM Ard Biesheuvel <ardb@kernel.org> wrote:
> > > > >>
> > > > >> On Tue, 17 Jan 2023 at 13:48, Oliver Steffen <osteffen@redhat.com> wrote:
> > > > >> >
> > > > >> > Hi Ard, Hi everyone,
> > > > >> >
> > > > >> > Thanks for the work!
> > > > >> >
> > > > >> > But somehow this patch (as it was merged into master branch) does not
> > > > >> > work for me on the ThunderX box we have.
> > > > >> >
> > > > >> > Any idea what could be wrong?
> > > > >>
> > > > >> I'm not sure I understand the question. The patch targets ThunderX,
> > > > >> and you are using a ThunderX2.
> > > > >>
> > > > >> What were you expecting to happen, and what is happening instead?
> > > > >
> > > > >
> > > > > Firmware does not start at all when using KVM.
> > > > >
> > > > > Please excuse my limited knowledge of Arm processor variants.
> > > > > I assumed that ThunderX and ThunderX2 are very similar and hoped
> > > > > the fix would also work for this case.
> > > > >
> > > > > The issue was introduced by the same commit that Dann
> > > > > reported (07be1d34d95460a238fcd0f6693efb747c28b329):
> > > > > "ArmVirtPkg/ArmVirtQemu: enable initial ID map at early boot".
> > > > >
> > > >
> > > > Can you share the QEMU command line that you are using? I use a
> > > > ThunderX2 basically 24/7 to do all my Linux and EDK2 development, so
> > > > this change was developed on ThunderX2 and so I'm surprised you are
> > > > seeing this issue.
> > > >
> > > > Did you try the DEBUG build as well?
> > > Yes, debug is on.
> > >
> > > Here is what I have, trying with the master branch from just now
> > > (998ebe5ca0ae5c449e83ede533bee872f97d63af):
> > >
> > > # make -C BaseTools && \
> > >   . ./edksetup.sh && \
> > >   build -t GCC5 -a AARCH64 \
> > >     -p ArmVirtPkg/ArmVirtQemu.dsc \
> > >     -DCAVIUM_ERRATUM_27456 \
> > >     -b DEBUG
> > >
> > > # /usr/libexec/qemu-kvm \
> > >     -machine accel=kvm -m 1G -boot menu=on \
> > >     -blockdev node-name=code,driver=file,filename="${FW_CODE_RESIZED}",read-only=on
> > > \
> > >     -blockdev node-name=vars,driver=file,filename="${FW_VARS}" \
> > >     -machine pflash0=code \
> > >     -machine pflash1=vars \
> > >     -cpu max \
> > >     -net none \
> > >     -serial stdio
> > >
> >
> > My distro does not have qemu-kvm, and using the command line above
> > results in the following if i try it with qemu-system-aarch64
> >
> > """
> > qemu-system-aarch64: No machine specified, and there is no default
> > Use -machine help to list supported machines
> > """
> >
> > unless i change it to
> >
> > qemu-system-aarch64 -machine virt,accel=kvm -m 1G -boot menu=on \
> >     -blockdev node-name=code,driver=file,filename=$HOME/bin/flash0.img,read-only=on
> > \
> >     -blockdev node-name=vars,driver=file,filename=$HOME/bin/flash1.img \
> >     -machine pflash0=code \
> >     -machine pflash1=vars \
> >     -cpu max \
> >     -net none \
> >     -nographic
> >
> > and that works fine with my firmware build.
> >
> >
> > > # /usr/libexec/qemu-kvm --version
> > > QEMU emulator version 7.2.0 (qemu-kvm-7.2.0-3.el9)
> > >
> > > # uname -r
> > > 5.14.0-234.el9.aarch64
> > >
> >
> > Yeah, that is quite old. One potential issue that comes to mind here
> > is the one address by the patch below
> >
> >
> > >
> > >
> > > Since you have the same CPU... Might this be a bug in KVM?
> > >
> >
> > Indeed. Could you try applying this patch?
> >
> > commit 406504c7b0405d74d74c15a667cd4c4620c3e7a9
> > Author: Marc Zyngier <maz@kernel.org>
> > Date:   Tue Dec 20 14:03:52 2022 +0000
> >
> >     KVM: arm64: Fix S1PTW handling on RO memslots
> >
> > Or check whether this is generally reproducible with newer kernels?
>
> Another thing you might try:
>
> - build the firmware with the following hunk applied
>
> """
> diff --git a/ArmVirtPkg/Library/ArmPlatformLibQemu/AArch64/ArmPlatformHelper.S
> b/ArmVirtPkg/Library/ArmPlatformLibQemu/AArch64/ArmPlatformHelper.S
> index 5ac7c732f6ec..f4e1285beefc 100644
> --- a/ArmVirtPkg/Library/ArmPlatformLibQemu/AArch64/ArmPlatformHelper.S
> +++ b/ArmVirtPkg/Library/ArmPlatformLibQemu/AArch64/ArmPlatformHelper.S
> @@ -40,6 +40,12 @@
>   .set    sctlrval, SCTLR_ELx_M | SCTLR_ELx_C | SCTLR_ELx_SA |
> SCTLR_EL1_ITD | SCTLR_EL1_SED
>   .set    sctlrval, sctlrval | SCTLR_ELx_I | SCTLR_EL1_SPAN | SCTLR_EL1_RES1
>
> +  .align  11
> +.Lvectors:
> +  .rept   16
> +  .align  7
> +  b       .
> +  .endr
>
>  ASM_FUNC(ArmPlatformPeiBootAction)
>  #ifdef CAVIUM_ERRATUM_27456
> @@ -90,6 +96,8 @@ ASM_FUNC(ArmPlatformPeiBootAction)
>    msr    mair_el1, x0            // set up the 1:1 mapping
>    msr    tcr_el1, x1
>    msr    ttbr0_el1, x2
> +  adr    x0, .Lvectors
> +  msr    vbar_el1, x0
>    isb
>
>    tlbi   vmalle1                 // invalidate any cached translations
> """
>
> - run qemu with the -s option and let it crash
>
> - connect with gdb and dump the exception context
>
> target remote:1234
> set radix 16
> p $FAR_EL1
> p $ESR_EL1
> p $ELR_EL1
>
> That should at least tell us why the crash is occurring.
>

I tried the most recent Qemu master (v7.2.50) and also v7.0.0,
on the 5.14 (RHEL) kernel and on 6.1.6-200.fc37.aarch64 (from Fedora).
No luck.

I applied the patch and attached gdb, as described (Qemu 7.2.50):

  p $ELR_EL1
  (gdb) p $FAR_EL1
  $1 = 0x6200
  (gdb) p $ESR_EL1
  $2 = 0x86000010
  (gdb) p $ELR_EL1
  $3 = 0x6200

There is no sign of any crash. It seems like it does not even start
running.

Thanks,
 Oliver

Re: [edk2-devel] [PATCH v2 2/2] ArmVirtPkg/ArmVirtQemu: Avoid early ID map on ThunderX

[Ard Biesheuvel]

 (cc Marc)

Context:
- on my TX2 (with the S1PTW r/o memslot fix applied), the new version
of ArmVirtQemu that uses an initial ID map in emulated NOR flash works
fine.
- in Oliver's case (which is a slightly different flavor of TX2), it
crashes extremely early, presumably at the point where this ID map is
activated.

More details at the end.

On Thu, 19 Jan 2023 at 12:03, Oliver Steffen <osteffen@redhat.com> wrote:
>
> Quoting Ard Biesheuvel (2023-01-18 10:22:12)
> > On Wed, 18 Jan 2023 at 09:48, Ard Biesheuvel <ardb@kernel.org> wrote:
> > >
> > > On Wed, 18 Jan 2023 at 09:28, Oliver Steffen <osteffen@redhat.com> wrote:
> > > >
> > > > Quoting Ard Biesheuvel (2023-01-18 08:34:32)
> > > > > On Wed, 18 Jan 2023 at 07:37, Oliver Steffen <osteffen@redhat.com> wrote:
> > > > > >
> > > > > > On Tue, Jan 17, 2023 at 3:57 PM Ard Biesheuvel <ardb@kernel.org> wrote:
> > > > > >>
> > > > > >> On Tue, 17 Jan 2023 at 13:48, Oliver Steffen <osteffen@redhat.com> wrote:
> > > > > >> >
> > > > > >> > Hi Ard, Hi everyone,
> > > > > >> >
> > > > > >> > Thanks for the work!
> > > > > >> >
> > > > > >> > But somehow this patch (as it was merged into master branch) does not
> > > > > >> > work for me on the ThunderX box we have.
> > > > > >> >
> > > > > >> > Any idea what could be wrong?
> > > > > >>
> > > > > >> I'm not sure I understand the question. The patch targets ThunderX,
> > > > > >> and you are using a ThunderX2.
> > > > > >>
> > > > > >> What were you expecting to happen, and what is happening instead?
> > > > > >
> > > > > >
> > > > > > Firmware does not start at all when using KVM.
> > > > > >
> > > > > > Please excuse my limited knowledge of Arm processor variants.
> > > > > > I assumed that ThunderX and ThunderX2 are very similar and hoped
> > > > > > the fix would also work for this case.
> > > > > >
> > > > > > The issue was introduced by the same commit that Dann
> > > > > > reported (07be1d34d95460a238fcd0f6693efb747c28b329):
> > > > > > "ArmVirtPkg/ArmVirtQemu: enable initial ID map at early boot".
> > > > > >
> > > > >
> > > > > Can you share the QEMU command line that you are using? I use a
> > > > > ThunderX2 basically 24/7 to do all my Linux and EDK2 development, so
> > > > > this change was developed on ThunderX2 and so I'm surprised you are
> > > > > seeing this issue.
> > > > >
> > > > > Did you try the DEBUG build as well?
> > > > Yes, debug is on.
> > > >
> > > > Here is what I have, trying with the master branch from just now
> > > > (998ebe5ca0ae5c449e83ede533bee872f97d63af):
> > > >
> > > > # make -C BaseTools && \
> > > >   . ./edksetup.sh && \
> > > >   build -t GCC5 -a AARCH64 \
> > > >     -p ArmVirtPkg/ArmVirtQemu.dsc \
> > > >     -DCAVIUM_ERRATUM_27456 \
> > > >     -b DEBUG
> > > >
> > > > # /usr/libexec/qemu-kvm \
> > > >     -machine accel=kvm -m 1G -boot menu=on \
> > > >     -blockdev node-name=code,driver=file,filename="${FW_CODE_RESIZED}",read-only=on
> > > > \
> > > >     -blockdev node-name=vars,driver=file,filename="${FW_VARS}" \
> > > >     -machine pflash0=code \
> > > >     -machine pflash1=vars \
> > > >     -cpu max \
> > > >     -net none \
> > > >     -serial stdio
> > > >
> > >
> > > My distro does not have qemu-kvm, and using the command line above
> > > results in the following if i try it with qemu-system-aarch64
> > >
> > > """
> > > qemu-system-aarch64: No machine specified, and there is no default
> > > Use -machine help to list supported machines
> > > """
> > >
> > > unless i change it to
> > >
> > > qemu-system-aarch64 -machine virt,accel=kvm -m 1G -boot menu=on \
> > >     -blockdev node-name=code,driver=file,filename=$HOME/bin/flash0.img,read-only=on
> > > \
> > >     -blockdev node-name=vars,driver=file,filename=$HOME/bin/flash1.img \
> > >     -machine pflash0=code \
> > >     -machine pflash1=vars \
> > >     -cpu max \
> > >     -net none \
> > >     -nographic
> > >
> > > and that works fine with my firmware build.
> > >
> > >
> > > > # /usr/libexec/qemu-kvm --version
> > > > QEMU emulator version 7.2.0 (qemu-kvm-7.2.0-3.el9)
> > > >
> > > > # uname -r
> > > > 5.14.0-234.el9.aarch64
> > > >
> > >
> > > Yeah, that is quite old. One potential issue that comes to mind here
> > > is the one address by the patch below
> > >
> > >
> > > >
> > > >
> > > > Since you have the same CPU... Might this be a bug in KVM?
> > > >
> > >
> > > Indeed. Could you try applying this patch?
> > >
> > > commit 406504c7b0405d74d74c15a667cd4c4620c3e7a9
> > > Author: Marc Zyngier <maz@kernel.org>
> > > Date:   Tue Dec 20 14:03:52 2022 +0000
> > >
> > >     KVM: arm64: Fix S1PTW handling on RO memslots
> > >
> > > Or check whether this is generally reproducible with newer kernels?
> >
> > Another thing you might try:
> >
> > - build the firmware with the following hunk applied
> >
> > """
> > diff --git a/ArmVirtPkg/Library/ArmPlatformLibQemu/AArch64/ArmPlatformHelper.S
> > b/ArmVirtPkg/Library/ArmPlatformLibQemu/AArch64/ArmPlatformHelper.S
> > index 5ac7c732f6ec..f4e1285beefc 100644
> > --- a/ArmVirtPkg/Library/ArmPlatformLibQemu/AArch64/ArmPlatformHelper.S
> > +++ b/ArmVirtPkg/Library/ArmPlatformLibQemu/AArch64/ArmPlatformHelper.S
> > @@ -40,6 +40,12 @@
> >   .set    sctlrval, SCTLR_ELx_M | SCTLR_ELx_C | SCTLR_ELx_SA |
> > SCTLR_EL1_ITD | SCTLR_EL1_SED
> >   .set    sctlrval, sctlrval | SCTLR_ELx_I | SCTLR_EL1_SPAN | SCTLR_EL1_RES1
> >
> > +  .align  11
> > +.Lvectors:
> > +  .rept   16
> > +  .align  7
> > +  b       .
> > +  .endr
> >
> >  ASM_FUNC(ArmPlatformPeiBootAction)
> >  #ifdef CAVIUM_ERRATUM_27456
> > @@ -90,6 +96,8 @@ ASM_FUNC(ArmPlatformPeiBootAction)
> >    msr    mair_el1, x0            // set up the 1:1 mapping
> >    msr    tcr_el1, x1
> >    msr    ttbr0_el1, x2
> > +  adr    x0, .Lvectors
> > +  msr    vbar_el1, x0
> >    isb
> >
> >    tlbi   vmalle1                 // invalidate any cached translations
> > """
> >
> > - run qemu with the -s option and let it crash
> >
> > - connect with gdb and dump the exception context
> >
> > target remote:1234
> > set radix 16
> > p $FAR_EL1
> > p $ESR_EL1
> > p $ELR_EL1
> >
> > That should at least tell us why the crash is occurring.
> >
>
> I tried the most recent Qemu master (v7.2.50) and also v7.0.0,
> on the 5.14 (RHEL) kernel and on 6.1.6-200.fc37.aarch64 (from Fedora).
> No luck.
>

Does that include a backport of commit 406504c7b0405d74d74c15a667cd4c4620c3e7a9?

> I applied the patch and attached gdb, as described (Qemu 7.2.50):
>
>   p $ELR_EL1
>   (gdb) p $FAR_EL1
>   $1 = 0x6200
>   (gdb) p $ESR_EL1
>   $2 = 0x86000010
>   (gdb) p $ELR_EL1
>   $3 = 0x6200
>
> There is no sign of any crash. It seems like it does not even start
> running.
>

So 0x6200 is the sync exception vector, which is both the code
location of the crash and the faulting address. This means fetching
the instructions to handle the original exception failed, and so the
original exception reason (ESR) is lost. However, the synchronous
external abort (https://esr.arm64.dev/?#0x86000010) that you are
seeing might point to an issue similar (or the same) that Marc
recently fixed in KVM.

It is quite odd that this does not reproduce *at all* on my TX2.
Fedora kernels don't use 64k pages right?

Re: [edk2-devel] [PATCH v2 2/2] ArmVirtPkg/ArmVirtQemu: Avoid early ID map on ThunderX

[Oliver Steffen]

Quoting Ard Biesheuvel (2023-01-19 12:11:34)
>  (cc Marc)
>
> Context:
> - on my TX2 (with the S1PTW r/o memslot fix applied), the new version
> of ArmVirtQemu that uses an initial ID map in emulated NOR flash works
> fine.
> - in Oliver's case (which is a slightly different flavor of TX2), it
> crashes extremely early, presumably at the point where this ID map is
> activated.
>
> More details at the end.
>
> On Thu, 19 Jan 2023 at 12:03, Oliver Steffen <osteffen@redhat.com> wrote:
> >
> > Quoting Ard Biesheuvel (2023-01-18 10:22:12)
> > > On Wed, 18 Jan 2023 at 09:48, Ard Biesheuvel <ardb@kernel.org> wrote:
> > > >
> > > > On Wed, 18 Jan 2023 at 09:28, Oliver Steffen <osteffen@redhat.com> wrote:
> > > > >
> > > > > Quoting Ard Biesheuvel (2023-01-18 08:34:32)
> > > > > > On Wed, 18 Jan 2023 at 07:37, Oliver Steffen <osteffen@redhat.com> wrote:
> > > > > > >
> > > > > > > On Tue, Jan 17, 2023 at 3:57 PM Ard Biesheuvel <ardb@kernel.org> wrote:
> > > > > > >>
> > > > > > >> On Tue, 17 Jan 2023 at 13:48, Oliver Steffen <osteffen@redhat.com> wrote:
> > > > > > >> >
> > > > > > >> > Hi Ard, Hi everyone,
> > > > > > >> >
> > > > > > >> > Thanks for the work!
> > > > > > >> >
> > > > > > >> > But somehow this patch (as it was merged into master branch) does not
> > > > > > >> > work for me on the ThunderX box we have.
> > > > > > >> >
> > > > > > >> > Any idea what could be wrong?
> > > > > > >>
> > > > > > >> I'm not sure I understand the question. The patch targets ThunderX,
> > > > > > >> and you are using a ThunderX2.
> > > > > > >>
> > > > > > >> What were you expecting to happen, and what is happening instead?
> > > > > > >
> > > > > > >
> > > > > > > Firmware does not start at all when using KVM.
> > > > > > >
> > > > > > > Please excuse my limited knowledge of Arm processor variants.
> > > > > > > I assumed that ThunderX and ThunderX2 are very similar and hoped
> > > > > > > the fix would also work for this case.
> > > > > > >
> > > > > > > The issue was introduced by the same commit that Dann
> > > > > > > reported (07be1d34d95460a238fcd0f6693efb747c28b329):
> > > > > > > "ArmVirtPkg/ArmVirtQemu: enable initial ID map at early boot".
> > > > > > >
> > > > > >
> > > > > > Can you share the QEMU command line that you are using? I use a
> > > > > > ThunderX2 basically 24/7 to do all my Linux and EDK2 development, so
> > > > > > this change was developed on ThunderX2 and so I'm surprised you are
> > > > > > seeing this issue.
> > > > > >
> > > > > > Did you try the DEBUG build as well?
> > > > > Yes, debug is on.
> > > > >
> > > > > Here is what I have, trying with the master branch from just now
> > > > > (998ebe5ca0ae5c449e83ede533bee872f97d63af):
> > > > >
> > > > > # make -C BaseTools && \
> > > > >   . ./edksetup.sh && \
> > > > >   build -t GCC5 -a AARCH64 \
> > > > >     -p ArmVirtPkg/ArmVirtQemu.dsc \
> > > > >     -DCAVIUM_ERRATUM_27456 \
> > > > >     -b DEBUG
> > > > >
> > > > > # /usr/libexec/qemu-kvm \
> > > > >     -machine accel=kvm -m 1G -boot menu=on \
> > > > >     -blockdev node-name=code,driver=file,filename="${FW_CODE_RESIZED}",read-only=on
> > > > > \
> > > > >     -blockdev node-name=vars,driver=file,filename="${FW_VARS}" \
> > > > >     -machine pflash0=code \
> > > > >     -machine pflash1=vars \
> > > > >     -cpu max \
> > > > >     -net none \
> > > > >     -serial stdio
> > > > >
> > > >
> > > > My distro does not have qemu-kvm, and using the command line above
> > > > results in the following if i try it with qemu-system-aarch64
> > > >
> > > > """
> > > > qemu-system-aarch64: No machine specified, and there is no default
> > > > Use -machine help to list supported machines
> > > > """
> > > >
> > > > unless i change it to
> > > >
> > > > qemu-system-aarch64 -machine virt,accel=kvm -m 1G -boot menu=on \
> > > >     -blockdev node-name=code,driver=file,filename=$HOME/bin/flash0.img,read-only=on
> > > > \
> > > >     -blockdev node-name=vars,driver=file,filename=$HOME/bin/flash1.img \
> > > >     -machine pflash0=code \
> > > >     -machine pflash1=vars \
> > > >     -cpu max \
> > > >     -net none \
> > > >     -nographic
> > > >
> > > > and that works fine with my firmware build.
> > > >
> > > >
> > > > > # /usr/libexec/qemu-kvm --version
> > > > > QEMU emulator version 7.2.0 (qemu-kvm-7.2.0-3.el9)
> > > > >
> > > > > # uname -r
> > > > > 5.14.0-234.el9.aarch64
> > > > >
> > > >
> > > > Yeah, that is quite old. One potential issue that comes to mind here
> > > > is the one address by the patch below
> > > >
> > > >
> > > > >
> > > > >
> > > > > Since you have the same CPU... Might this be a bug in KVM?
> > > > >
> > > >
> > > > Indeed. Could you try applying this patch?
> > > >
> > > > commit 406504c7b0405d74d74c15a667cd4c4620c3e7a9
> > > > Author: Marc Zyngier <maz@kernel.org>
> > > > Date:   Tue Dec 20 14:03:52 2022 +0000
> > > >
> > > >     KVM: arm64: Fix S1PTW handling on RO memslots
> > > >
> > > > Or check whether this is generally reproducible with newer kernels?
> > >
> > > Another thing you might try:
> > >
> > > - build the firmware with the following hunk applied
> > >
> > > """
> > > diff --git a/ArmVirtPkg/Library/ArmPlatformLibQemu/AArch64/ArmPlatformHelper.S
> > > b/ArmVirtPkg/Library/ArmPlatformLibQemu/AArch64/ArmPlatformHelper.S
> > > index 5ac7c732f6ec..f4e1285beefc 100644
> > > --- a/ArmVirtPkg/Library/ArmPlatformLibQemu/AArch64/ArmPlatformHelper.S
> > > +++ b/ArmVirtPkg/Library/ArmPlatformLibQemu/AArch64/ArmPlatformHelper.S
> > > @@ -40,6 +40,12 @@
> > >   .set    sctlrval, SCTLR_ELx_M | SCTLR_ELx_C | SCTLR_ELx_SA |
> > > SCTLR_EL1_ITD | SCTLR_EL1_SED
> > >   .set    sctlrval, sctlrval | SCTLR_ELx_I | SCTLR_EL1_SPAN | SCTLR_EL1_RES1
> > >
> > > +  .align  11
> > > +.Lvectors:
> > > +  .rept   16
> > > +  .align  7
> > > +  b       .
> > > +  .endr
> > >
> > >  ASM_FUNC(ArmPlatformPeiBootAction)
> > >  #ifdef CAVIUM_ERRATUM_27456
> > > @@ -90,6 +96,8 @@ ASM_FUNC(ArmPlatformPeiBootAction)
> > >    msr    mair_el1, x0            // set up the 1:1 mapping
> > >    msr    tcr_el1, x1
> > >    msr    ttbr0_el1, x2
> > > +  adr    x0, .Lvectors
> > > +  msr    vbar_el1, x0
> > >    isb
> > >
> > >    tlbi   vmalle1                 // invalidate any cached translations
> > > """
> > >
> > > - run qemu with the -s option and let it crash
> > >
> > > - connect with gdb and dump the exception context
> > >
> > > target remote:1234
> > > set radix 16
> > > p $FAR_EL1
> > > p $ESR_EL1
> > > p $ELR_EL1
> > >
> > > That should at least tell us why the crash is occurring.
> > >
> >
> > I tried the most recent Qemu master (v7.2.50) and also v7.0.0,
> > on the 5.14 (RHEL) kernel and on 6.1.6-200.fc37.aarch64 (from Fedora).
> > No luck.
> >
>
> Does that include a backport of commit 406504c7b0405d74d74c15a667cd4c4620c3e7a9?
>
> > I applied the patch and attached gdb, as described (Qemu 7.2.50):
> >
> >   p $ELR_EL1
> >   (gdb) p $FAR_EL1
> >   $1 = 0x6200
> >   (gdb) p $ESR_EL1
> >   $2 = 0x86000010
> >   (gdb) p $ELR_EL1
> >   $3 = 0x6200
> >
> > There is no sign of any crash. It seems like it does not even start
> > running.
> >
>
> So 0x6200 is the sync exception vector, which is both the code
> location of the crash and the faulting address. This means fetching
> the instructions to handle the original exception failed, and so the
> original exception reason (ESR) is lost. However, the synchronous
> external abort (https://esr.arm64.dev/?#0x86000010) that you are
> seeing might point to an issue similar (or the same) that Marc
> recently fixed in KVM.
>
> It is quite odd that this does not reproduce *at all* on my TX2.
> Fedora kernels don't use 64k pages right?
>

Kernel config says:
CONFIG_ARM64_4K_PAGES=y

Re: [edk2-devel] [PATCH v2 2/2] ArmVirtPkg/ArmVirtQemu: Avoid early ID map on ThunderX

[Marc Zyngier]

On Thu, 19 Jan 2023 11:11:34 +0000,
Ard Biesheuvel <ardb@kernel.org> wrote:
> 
>  (cc Marc)
> 
> Context:
> - on my TX2 (with the S1PTW r/o memslot fix applied), the new version
> of ArmVirtQemu that uses an initial ID map in emulated NOR flash works
> fine.
> - in Oliver's case (which is a slightly different flavor of TX2), it
> crashes extremely early, presumably at the point where this ID map is
> activated.

Oliver seems to have a vintage ThunderX (aka the worst arm64
implementation in history!), so it is indeed a very different beat
from TX2.

Without the kernel patch[1], I can trigger the issue pretty reliably,
specially in the absence of THP. It all depends on the layout of the
EDK2 object and the order in which pages get mapped.

The first course of action would be to make sure that the patch is
applied to the host kernel. If this still fails to boot, I'm happy to
help investigating it.

	M.

[1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/patch?id=406504c7b0405d74d74c15a667cd4c4620c3e7a9

-- 
Without deviation from the norm, progress is not possible.

Re: [edk2-devel] [PATCH v2 2/2] ArmVirtPkg/ArmVirtQemu: Avoid early ID map on ThunderX

[Gerd Hoffmann]

  Hi,

> > I tried the most recent Qemu master (v7.2.50) and also v7.0.0,
> > on the 5.14 (RHEL) kernel and on 6.1.6-200.fc37.aarch64 (from Fedora).
> > No luck.
> >
> 
> Does that include a backport of commit 406504c7b0405d74d74c15a667cd4c4620c3e7a9?

Probably not given that fedora kernels are almost vanilla and the
backport landed in stable in v6.1.7 (commit 9a1195c584321).

Oliver: try pulling the latest kernel directly from koji:
https://koji.fedoraproject.org/koji/buildinfo?buildID=2112315

take care,
  Gerd

Re: [edk2-devel] [PATCH v2 2/2] ArmVirtPkg/ArmVirtQemu: Avoid early ID map on ThunderX

[Ard Biesheuvel]

On Thu, 19 Jan 2023 at 12:56, Marc Zyngier <maz@kernel.org> wrote:
>
> On Thu, 19 Jan 2023 11:11:34 +0000,
> Ard Biesheuvel <ardb@kernel.org> wrote:
> >
> >  (cc Marc)
> >
> > Context:
> > - on my TX2 (with the S1PTW r/o memslot fix applied), the new version
> > of ArmVirtQemu that uses an initial ID map in emulated NOR flash works
> > fine.
> > - in Oliver's case (which is a slightly different flavor of TX2), it
> > crashes extremely early, presumably at the point where this ID map is
> > activated.
>
> Oliver seems to have a vintage ThunderX (aka the worst arm64
> implementation in history!), so it is indeed a very different beat
> from TX2.
>

No, it's a TX2 alright - we disabled this ID map for TX1 already
because it tickles the I-cache corruption erratum (EDK2 runs at EL1
and the initial ID map is ASID scoped with non-global mappings, so
that we can switch to the actual ID map seamlessly)

> Without the kernel patch[1], I can trigger the issue pretty reliably,
> specially in the absence of THP. It all depends on the layout of the
> EDK2 object and the order in which pages get mapped.
>
> The first course of action would be to make sure that the patch is
> applied to the host kernel. If this still fails to boot, I'm happy to
> help investigating it.
>

indeed.

Re: [edk2-devel] [PATCH v2 2/2] ArmVirtPkg/ArmVirtQemu: Avoid early ID map on ThunderX

[Oliver Steffen]

Quoting Gerd Hoffmann (2023-01-19 13:00:21)
>   Hi,
>
> > > I tried the most recent Qemu master (v7.2.50) and also v7.0.0,
> > > on the 5.14 (RHEL) kernel and on 6.1.6-200.fc37.aarch64 (from Fedora).
> > > No luck.
> > >
> >
> > Does that include a backport of commit 406504c7b0405d74d74c15a667cd4c4620c3e7a9?
>
> Probably not given that fedora kernels are almost vanilla and the
> backport landed in stable in v6.1.7 (commit 9a1195c584321).
>
> Oliver: try pulling the latest kernel directly from koji:
> https://koji.fedoraproject.org/koji/buildinfo?buildID=2112315
>
> take care,
>   Gerd
>

Thanks for the koji link, Gerd.

It works with 6.1.7-200.fc37.aarch64.
Perfect.

Thanks Ard, thanks Marc!

-Oliver

Re: [edk2-devel] [PATCH v2 2/2] ArmVirtPkg/ArmVirtQemu: Avoid early ID map on ThunderX

[Ard Biesheuvel]

On Thu, 19 Jan 2023 at 13:55, Oliver Steffen <osteffen@redhat.com> wrote:
>
> Quoting Gerd Hoffmann (2023-01-19 13:00:21)
> >   Hi,
> >
> > > > I tried the most recent Qemu master (v7.2.50) and also v7.0.0,
> > > > on the 5.14 (RHEL) kernel and on 6.1.6-200.fc37.aarch64 (from Fedora).
> > > > No luck.
> > > >
> > >
> > > Does that include a backport of commit 406504c7b0405d74d74c15a667cd4c4620c3e7a9?
> >
> > Probably not given that fedora kernels are almost vanilla and the
> > backport landed in stable in v6.1.7 (commit 9a1195c584321).
> >
> > Oliver: try pulling the latest kernel directly from koji:
> > https://koji.fedoraproject.org/koji/buildinfo?buildID=2112315
> >
> > take care,
> >   Gerd
> >
>
> Thanks for the koji link, Gerd.
>
> It works with 6.1.7-200.fc37.aarch64.
> Perfect.
>
> Thanks Ard, thanks Marc!
>

Thanks for the report. Up to this point, Marc is the only one who had
managed to reproduce this afaik, so having another data point where
the fix works as intended is rather nice.

Re: [edk2-devel] [PATCH v2 2/2] ArmVirtPkg/ArmVirtQemu: Avoid early ID map on ThunderX

[Gerd Hoffmann]

On Thu, Jan 19, 2023 at 02:21:30PM +0100, Ard Biesheuvel wrote:
> On Thu, 19 Jan 2023 at 13:55, Oliver Steffen <osteffen@redhat.com> wrote:
> >
> > Quoting Gerd Hoffmann (2023-01-19 13:00:21)
> > >   Hi,
> > >
> > > > > I tried the most recent Qemu master (v7.2.50) and also v7.0.0,
> > > > > on the 5.14 (RHEL) kernel and on 6.1.6-200.fc37.aarch64 (from Fedora).
> > > > > No luck.
> > > > >
> > > >
> > > > Does that include a backport of commit 406504c7b0405d74d74c15a667cd4c4620c3e7a9?
> > >
> > > Probably not given that fedora kernels are almost vanilla and the
> > > backport landed in stable in v6.1.7 (commit 9a1195c584321).
> > >
> > > Oliver: try pulling the latest kernel directly from koji:
> > > https://koji.fedoraproject.org/koji/buildinfo?buildID=2112315
> > >
> > > take care,
> > >   Gerd
> > >
> >
> > Thanks for the koji link, Gerd.
> >
> > It works with 6.1.7-200.fc37.aarch64.
> > Perfect.
> >
> > Thanks Ard, thanks Marc!
> >
> 
> Thanks for the report. Up to this point, Marc is the only one who had
> managed to reproduce this afaik, so having another data point where
> the fix works as intended is rather nice.

One more data point:  This also gets things working again on the
raspberry pi 3.  Well, sort of.  There still is some timer problem
(it waits forever on the splash screen).  That might also be some
qemu bug though ...

take care,
  Gerd

Re: [edk2-devel] [PATCH v2 2/2] ArmVirtPkg/ArmVirtQemu: Avoid early ID map on ThunderX

[Marc Zyngier]

On Thu, 26 Jan 2023 12:01:51 +0000,
Gerd Hoffmann <kraxel@redhat.com> wrote:
> 
> On Thu, Jan 19, 2023 at 02:21:30PM +0100, Ard Biesheuvel wrote:
> > On Thu, 19 Jan 2023 at 13:55, Oliver Steffen <osteffen@redhat.com> wrote:
> > >
> > > Quoting Gerd Hoffmann (2023-01-19 13:00:21)
> > > >   Hi,
> > > >
> > > > > > I tried the most recent Qemu master (v7.2.50) and also v7.0.0,
> > > > > > on the 5.14 (RHEL) kernel and on 6.1.6-200.fc37.aarch64 (from Fedora).
> > > > > > No luck.
> > > > > >
> > > > >
> > > > > Does that include a backport of commit 406504c7b0405d74d74c15a667cd4c4620c3e7a9?
> > > >
> > > > Probably not given that fedora kernels are almost vanilla and the
> > > > backport landed in stable in v6.1.7 (commit 9a1195c584321).
> > > >
> > > > Oliver: try pulling the latest kernel directly from koji:
> > > > https://koji.fedoraproject.org/koji/buildinfo?buildID=2112315
> > > >
> > > > take care,
> > > >   Gerd
> > > >
> > >
> > > Thanks for the koji link, Gerd.
> > >
> > > It works with 6.1.7-200.fc37.aarch64.
> > > Perfect.
> > >
> > > Thanks Ard, thanks Marc!
> > >
> > 
> > Thanks for the report. Up to this point, Marc is the only one who had
> > managed to reproduce this afaik, so having another data point where
> > the fix works as intended is rather nice.
> 
> One more data point:  This also gets things working again on the
> raspberry pi 3.  Well, sort of.  There still is some timer problem
> (it waits forever on the splash screen).  That might also be some
> qemu bug though ...

RPi3 as a host? Using the QEMU-provided GIC emulation? I wouldn't be
surprised if that broke ages ago as nobody really tests this
configuration. If you can pinpoint the last state where this actually
worked, I'll try and have a look.

But this sounds unrelated to the page fault handling that my patch was
fixing.

Thanks,

	M.

-- 
Without deviation from the norm, progress is not possible.

Re: [edk2-devel] [PATCH v2 2/2] ArmVirtPkg/ArmVirtQemu: Avoid early ID map on ThunderX

[Gerd Hoffmann]

On Thu, Jan 26, 2023 at 01:48:12PM +0000, Marc Zyngier wrote:
> On Thu, 26 Jan 2023 12:01:51 +0000,
> Gerd Hoffmann <kraxel@redhat.com> wrote:
> > 
> > > > Thanks for the koji link, Gerd.
> > > >
> > > > It works with 6.1.7-200.fc37.aarch64.
> > > > Perfect.
> > > >
> > > > Thanks Ard, thanks Marc!
> > > >
> > > 
> > > Thanks for the report. Up to this point, Marc is the only one who had
> > > managed to reproduce this afaik, so having another data point where
> > > the fix works as intended is rather nice.
> > 
> > One more data point:  This also gets things working again on the
> > raspberry pi 3.  Well, sort of.  There still is some timer problem
> > (it waits forever on the splash screen).  That might also be some
> > qemu bug though ...
> 
> RPi3 as a host? Using the QEMU-provided GIC emulation?

Yes.

> I wouldn't be surprised if that broke ages ago as nobody really tests
> this configuration.

Understandable.  1G is enough for simple smoke tests but not much
beyond that.  And it is the odd child in the gang which hasn't a gic.

> If you can pinpoint the last state where this actually worked, I'll
> try and have a look.

Don't have that at hand.  Need try downgrade stuff and see what I find.
Tried with edk2 already back a year or so without success.  Didn't try
downgrading qemu / kernel yet.

> But this sounds unrelated to the page fault handling that my patch was
> fixing.

Yes, unrelated bug.  But the page fault bug reproduces on the rpi3 too,
without the fix the firmware hangs before printing the first log line.
Wanted report that because Ard mentioned it is hard to reproduce.

take care,
  Gerd

Re: [edk2-devel] [PATCH v2 2/2] ArmVirtPkg/ArmVirtQemu: Avoid early ID map on ThunderX

[Marc Zyngier]

On Thu, 26 Jan 2023 14:46:13 +0000,
Gerd Hoffmann <kraxel@redhat.com> wrote:
> 
> On Thu, Jan 26, 2023 at 01:48:12PM +0000, Marc Zyngier wrote:
> > On Thu, 26 Jan 2023 12:01:51 +0000,
> > Gerd Hoffmann <kraxel@redhat.com> wrote:
> > > 
> > > > > Thanks for the koji link, Gerd.
> > > > >
> > > > > It works with 6.1.7-200.fc37.aarch64.
> > > > > Perfect.
> > > > >
> > > > > Thanks Ard, thanks Marc!
> > > > >
> > > > 
> > > > Thanks for the report. Up to this point, Marc is the only one who had
> > > > managed to reproduce this afaik, so having another data point where
> > > > the fix works as intended is rather nice.
> > > 
> > > One more data point:  This also gets things working again on the
> > > raspberry pi 3.  Well, sort of.  There still is some timer problem
> > > (it waits forever on the splash screen).  That might also be some
> > > qemu bug though ...
> > 
> > RPi3 as a host? Using the QEMU-provided GIC emulation?
> 
> Yes.
> 
> > I wouldn't be surprised if that broke ages ago as nobody really tests
> > this configuration.
> 
> Understandable.  1G is enough for simple smoke tests but not much
> beyond that.  And it is the odd child in the gang which hasn't a gic.

I suspect that it is that last point that has broken. The kernel
handling of that configuration has always been super sketchy, and it
would take almost nothing for that to break.

I'm almost thinking of writing an in-kernel GICv2 emulation just to
get rid of the rest of the hacks that have proliferated in the timer
and PMU code. But is it worth it for such an odd machine?

The alternative would be to move on and stop supporting KVM on this
wonky setup. :-/

> > If you can pinpoint the last state where this actually worked, I'll
> > try and have a look.
> 
> Don't have that at hand.  Need try downgrade stuff and see what I find.
> Tried with edk2 already back a year or so without success.  Didn't try
> downgrading qemu / kernel yet.

Right. Let me know if you get somewhere with that at some point.

> > But this sounds unrelated to the page fault handling that my patch was
> > fixing.
> 
> Yes, unrelated bug.  But the page fault bug reproduces on the rpi3 too,
> without the fix the firmware hangs before printing the first log line.
> Wanted report that because Ard mentioned it is hard to reproduce.

Right, thanks for that. Very interesting data point indeed.

Cheers,

	M.

-- 
Without deviation from the norm, progress is not possible.

Re: [edk2-devel] [PATCH v2 2/2] ArmVirtPkg/ArmVirtQemu: Avoid early ID map on ThunderX

[Oliver Steffen]

[-- Attachment #1: Type: text/plain, Size: 6877 bytes --]

Hi everyone!

On Thu, Jan 19, 2023 at 2:21 PM Ard Biesheuvel <ardb@kernel.org> wrote:
>
> On Thu, 19 Jan 2023 at 13:55, Oliver Steffen <osteffen@redhat.com> wrote:
> >
> > Quoting Gerd Hoffmann (2023-01-19 13:00:21)
> > >   Hi,
> > >
> > > > > I tried the most recent Qemu master (v7.2.50) and also v7.0.0,
> > > > > on the 5.14 (RHEL) kernel and on 6.1.6-200.fc37.aarch64 (from
Fedora).
> > > > > No luck.
> > > > >
> > > >
> > > > Does that include a backport of commit
406504c7b0405d74d74c15a667cd4c4620c3e7a9?
> > >
> > > Probably not given that fedora kernels are almost vanilla and the
> > > backport landed in stable in v6.1.7 (commit 9a1195c584321).
> > >
> > > Oliver: try pulling the latest kernel directly from koji:
> > > https://koji.fedoraproject.org/koji/buildinfo?buildID=2112315
> > >
> > > take care,
> > >   Gerd
> > >
> >
> > Thanks for the koji link, Gerd.
> >
> > It works with 6.1.7-200.fc37.aarch64.
> > Perfect.
> >
> > Thanks Ard, thanks Marc!
> >
>
> Thanks for the report. Up to this point, Marc is the only one who had
> managed to reproduce this afaik, so having another data point where
> the fix works as intended is rather nice.
>

I am sorry, this story does not seem to be over yet.

We are using the Erratum patch and also included the commit 406504c7 in
the kernel.
Now the firmware crashes sometimes (10 out of 89 tests).

Any hints are very welcome!

Here is the serial output of one case:
----------------------------------------------------------------
UEFI firmware (version edk2-20221207gitfff6d81270b5-4.el9.test built at
00:00:00 on Jan 18 2023)
SyncPcrAllocationsAndPcrMask!


Synchronous Exception at 0x000000037FD5BDE0
PC 0x00037FD5BDE0 (0x00037FD59000+0x00002DE0) [ 0] ArmCpuDxe.dll
PC 0x00037FD5BDE0 (0x00037FD59000+0x00002DE0) [ 0] ArmCpuDxe.dll
PC 0x00037FD5BCE0 (0x00037FD59000+0x00002CE0) [ 0] ArmCpuDxe.dll
PC 0x00037FD5C054 (0x00037FD59000+0x00003054) [ 0] ArmCpuDxe.dll
PC 0x0000476F08EC (0x0000476EE000+0x000028EC) [ 1] DxeCore.dll
PC 0x0000476F65C0 (0x0000476EE000+0x000085C0) [ 1] DxeCore.dll
PC 0x0000476FB5B8 (0x0000476EE000+0x0000D5B8) [ 1] DxeCore.dll
PC 0x0000476F5DB8 (0x0000476EE000+0x00007DB8) [ 1] DxeCore.dll
PC 0x00037FD5CD50 (0x00037FD59000+0x00003D50) [ 2] ArmCpuDxe.dll
PC 0x0000476F5758 (0x0000476EE000+0x00007758) [ 3] DxeCore.dll
PC 0x000047702B18 (0x0000476EE000+0x00014B18) [ 3] DxeCore.dll
PC 0x0000476F8D70 (0x0000476EE000+0x0000AD70) [ 3] DxeCore.dll

[ 0]
/builddir/build/BUILD/edk2-fff6d81270b5/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/ArmPkg/Drivers/CpuDxe/CpuDxe/DEBUG/ArmCpuDxe.dll
[ 1]
/builddir/build/BUILD/edk2-fff6d81270b5/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/MdeModulePkg/Core/Dxe/DxeMain/DEBUG/DxeCore.dll
[ 2]
/builddir/build/BUILD/edk2-fff6d81270b5/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/ArmPkg/Drivers/CpuDxe/CpuDxe/DEBUG/ArmCpuDxe.dll
[ 3]
/builddir/build/BUILD/edk2-fff6d81270b5/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/MdeModulePkg/Core/Dxe/DxeMain/DEBUG/DxeCore.dll

  X0 0x0000000047FFE068   X1 0x000000037F179003   X2 0x0000000000340000
X3 0x0000000000000000
  X4 0x0000000000000200   X5 0x0000000000000004   X6 0x0060000000000000
X7 0xFF9F000000000F3F
  X8 0x000000037FFFF008   X9 0x0000000400000000  X10 0x000000037F177000
 X11 0x000000037FA37FFF
 X12 0x0000000000000000  X13 0x0000000000000008  X14 0x0000000000000000
 X15 0x0000000000000000
 X16 0x000000037FD5A208  X17 0x00000000007979D0  X18 0x0000000000000000
 X19 0x0000000340000000
 X20 0x0000000000000001  X21 0x000000037F179003  X22 0x0000000047FFE000
 X23 0x0000000000000068
 X24 0x000000003FFFFFFF  X25 0x000000037CAA0000  X26 0x0000000000000002
 X27 0x000000037F179000
 X28 0x0000000047FFE068   FP 0x00000000476ED840   LR 0x000000037FD5BDE0

  V0 0x0000000000000000 0000000000000000   V1 0x0000000000000000
0000000000000000
  V2 0x0000000000000000 0000000000000000   V3 0x0000000000000000
0000000000000000
  V4 0x0000000000000000 0000000000000000   V5 0x0000000000000000
0000000000000000
  V6 0x0000000000000000 0000000000000000   V7 0x0000000000000000
0000000000000000
  V8 0x0000000000000000 0000000000000000   V9 0x0000000000000000
0000000000000000
 V10 0x0000000000000000 0000000000000000  V11 0x0000000000000000
0000000000000000
 V12 0x0000000000000000 0000000000000000  V13 0x0000000000000000
0000000000000000
 V14 0x0000000000000000 0000000000000000  V15 0x0000000000000000
0000000000000000
 V16 0x0000000000000000 0000000000000000  V17 0x0000000000000000
0000000000000000
 V18 0x0000000000000000 0000000000000000  V19 0x0000000000000000
0000000000000000
 V20 0x0000000000000000 0000000000000000  V21 0x0000000000000000
0000000000000000
 V22 0x0000000000000000 0000000000000000  V23 0x0000000000000000
0000000000000000
 V24 0x0000000000000000 0000000000000000  V25 0x0000000000000000
0000000000000000
 V26 0x0000000000000000 0000000000000000  V27 0x0000000000000000
0000000000000000
 V28 0x0000000000000000 0000000000000000  V29 0x0000000000000000
0000000000000000
 V30 0x0000000000000000 0000000000000000  V31 0x0000000000000000
0000000000000000

  SP 0x00000000476ED840  ELR 0x000000037FD5BDE0  SPSR 0x80000205  FPSR
0x00000000
 ESR 0x86000005          FAR 0x000000037FD5BDE0

 ESR : EC 0x21  IL 0x1  ISS 0x00000005

Instruction abort: Translation fault, first level

Stack dump:
  00000476ED740: 000000037CAA0000 000000037CAA0000 0060000000000000
FF9F000000000F3F
  00000476ED760: 00000000476ED790 000000037FD610D8 0060000000000003
0000000C00000001
  00000476ED780: 000000037CA0070D 000000037F179000 00000000476ED840
000000037FD5BCE0
  00000476ED7A0: 0000000340000000 0000000000000001 000000037F179000
0000000047FFE000
  00000476ED7C0: 0000000000000068 000000003FFFFFFF 000000037CAA0000
0000000000000002
  00000476ED7E0: 000000037F179000 0000000047FFE068 000000037CC00000
000000037CAA0000
  00000476ED800: 0060000000000000 FF9F000000000F3F 00000000476ED840
000000037FD610D8
  00000476ED820: 0060000000000001 0000001500000001 000000034000070D
000000037F177000
> 00000476ED840: 00000000476ED8F0 000000037FD5BCE0 0000000047FFA000
0000000000000000
  00000476ED860: 0000000047FFE000 0000000047FFF000 0000000000000000
0000007FFFFFFFFF
  00000476ED880: 000000037CAA0000 0000000000000001 0000000047717588
0000000047FFF000
  00000476ED8A0: 0000000380000000 000000037CAA0000 0060000000000000
FF9F000000000F3F
  00000476ED8C0: 000000017FD605DD 000000037FD610D8 0060000000000001
0000001E00000001
  00000476ED8E0: 0060000000000000 000000037F179000 00000000476ED9A0
000000037FD5C054
  00000476ED900: 002000000000041C 0000000000000000 0000000047FFA000
0000000000004000
  00000476ED920: 0000000334AA6000 0000000047FFF000 000000037F17A238
0000000047717000
ASSERT [ArmCpuDxe]
/builddir/build/BUILD/edk2-fff6d81270b5/ArmPkg/Library/DefaultExceptionHandlerLib/AArch64/DefaultExceptionHandler.c(333):
((BOOLEAN)(0==1))
----------------------------------------------------------------
Thanks,
 Oliver

[-- Attachment #2: Type: text/html, Size: 8869 bytes --]

Re: [edk2-devel] [PATCH v2 2/2] ArmVirtPkg/ArmVirtQemu: Avoid early ID map on ThunderX

[Ard Biesheuvel]

On Wed, 1 Feb 2023 at 10:14, Oliver Steffen <osteffen@redhat.com> wrote:
>
> Hi everyone!
>
> On Thu, Jan 19, 2023 at 2:21 PM Ard Biesheuvel <ardb@kernel.org> wrote:
> >
> > On Thu, 19 Jan 2023 at 13:55, Oliver Steffen <osteffen@redhat.com> wrote:
> > >
> > > Quoting Gerd Hoffmann (2023-01-19 13:00:21)
> > > >   Hi,
> > > >
> > > > > > I tried the most recent Qemu master (v7.2.50) and also v7.0.0,
> > > > > > on the 5.14 (RHEL) kernel and on 6.1.6-200.fc37.aarch64 (from Fedora).
> > > > > > No luck.
> > > > > >
> > > > >
> > > > > Does that include a backport of commit 406504c7b0405d74d74c15a667cd4c4620c3e7a9?
> > > >
> > > > Probably not given that fedora kernels are almost vanilla and the
> > > > backport landed in stable in v6.1.7 (commit 9a1195c584321).
> > > >
> > > > Oliver: try pulling the latest kernel directly from koji:
> > > > https://koji.fedoraproject.org/koji/buildinfo?buildID=2112315
> > > >
> > > > take care,
> > > >   Gerd
> > > >
> > >
> > > Thanks for the koji link, Gerd.
> > >
> > > It works with 6.1.7-200.fc37.aarch64.
> > > Perfect.
> > >
> > > Thanks Ard, thanks Marc!
> > >
> >
> > Thanks for the report. Up to this point, Marc is the only one who had
> > managed to reproduce this afaik, so having another data point where
> > the fix works as intended is rather nice.
> >
>
> I am sorry, this story does not seem to be over yet.
>
> We are using the Erratum patch and also included the commit 406504c7 in
> the kernel.
> Now the firmware crashes sometimes (10 out of 89 tests).
>

Thanks for the report. Is this still on ThunderX2?

> Any hints are very welcome!
>

Do  you have access to those build artifacts?
Can you share more of the log output?

If you can rebuild, can you run it with DEBUG_VERBOSE set on ArmCpuDxe. E.g.,

--- a/ArmVirtPkg/ArmVirtQemu.dsc
+++ b/ArmVirtPkg/ArmVirtQemu.dsc
@@ -383,7 +383,11 @@ [Components.common]
   #
   # Architectural Protocols
   #
-  ArmPkg/Drivers/CpuDxe/CpuDxe.inf
+  ArmPkg/Drivers/CpuDxe/CpuDxe.inf {
+    <PcdsFixedAtBuild>
+      gEfiMdePkgTokenSpaceGuid.PcdDebugPrintErrorLevel|0x8040004F
+  }
+
   MdeModulePkg/Core/RuntimeDxe/RuntimeDxe.inf
   MdeModulePkg/Universal/Variable/RuntimeDxe/VariableRuntimeDxe.inf {
     <LibraryClasses>


> Here is the serial output of one case:
> ----------------------------------------------------------------
> UEFI firmware (version edk2-20221207gitfff6d81270b5-4.el9.test built at 00:00:00 on Jan 18 2023)
> SyncPcrAllocationsAndPcrMask!
>
>
> Synchronous Exception at 0x000000037FD5BDE0
> PC 0x00037FD5BDE0 (0x00037FD59000+0x00002DE0) [ 0] ArmCpuDxe.dll
> PC 0x00037FD5BDE0 (0x00037FD59000+0x00002DE0) [ 0] ArmCpuDxe.dll
> PC 0x00037FD5BCE0 (0x00037FD59000+0x00002CE0) [ 0] ArmCpuDxe.dll

This, along with

>   SP 0x00000000476ED840  ELR 0x000000037FD5BDE0  SPSR 0x80000205  FPSR 0x00000000
>  ESR 0x86000005          FAR 0x000000037FD5BDE0

taken from below suggests that UpdateRegionMappingRecursive () is
unmapping itself, likely when splitting a block entry.

This affects all ARM builds, not just ArmVIrtQemu so I would like to
get to the bottom of this.


Thanks,
Ard.



> PC 0x00037FD5C054 (0x00037FD59000+0x00003054) [ 0] ArmCpuDxe.dll
> PC 0x0000476F08EC (0x0000476EE000+0x000028EC) [ 1] DxeCore.dll
> PC 0x0000476F65C0 (0x0000476EE000+0x000085C0) [ 1] DxeCore.dll
> PC 0x0000476FB5B8 (0x0000476EE000+0x0000D5B8) [ 1] DxeCore.dll
> PC 0x0000476F5DB8 (0x0000476EE000+0x00007DB8) [ 1] DxeCore.dll
> PC 0x00037FD5CD50 (0x00037FD59000+0x00003D50) [ 2] ArmCpuDxe.dll
> PC 0x0000476F5758 (0x0000476EE000+0x00007758) [ 3] DxeCore.dll
> PC 0x000047702B18 (0x0000476EE000+0x00014B18) [ 3] DxeCore.dll
> PC 0x0000476F8D70 (0x0000476EE000+0x0000AD70) [ 3] DxeCore.dll
>
> [ 0] /builddir/build/BUILD/edk2-fff6d81270b5/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/ArmPkg/Drivers/CpuDxe/CpuDxe/DEBUG/ArmCpuDxe.dll
> [ 1] /builddir/build/BUILD/edk2-fff6d81270b5/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/MdeModulePkg/Core/Dxe/DxeMain/DEBUG/DxeCore.dll
> [ 2] /builddir/build/BUILD/edk2-fff6d81270b5/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/ArmPkg/Drivers/CpuDxe/CpuDxe/DEBUG/ArmCpuDxe.dll
> [ 3] /builddir/build/BUILD/edk2-fff6d81270b5/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/MdeModulePkg/Core/Dxe/DxeMain/DEBUG/DxeCore.dll
>
>   X0 0x0000000047FFE068   X1 0x000000037F179003   X2 0x0000000000340000   X3 0x0000000000000000
>   X4 0x0000000000000200   X5 0x0000000000000004   X6 0x0060000000000000   X7 0xFF9F000000000F3F
>   X8 0x000000037FFFF008   X9 0x0000000400000000  X10 0x000000037F177000  X11 0x000000037FA37FFF
>  X12 0x0000000000000000  X13 0x0000000000000008  X14 0x0000000000000000  X15 0x0000000000000000
>  X16 0x000000037FD5A208  X17 0x00000000007979D0  X18 0x0000000000000000  X19 0x0000000340000000
>  X20 0x0000000000000001  X21 0x000000037F179003  X22 0x0000000047FFE000  X23 0x0000000000000068
>  X24 0x000000003FFFFFFF  X25 0x000000037CAA0000  X26 0x0000000000000002  X27 0x000000037F179000
>  X28 0x0000000047FFE068   FP 0x00000000476ED840   LR 0x000000037FD5BDE0
>
>   V0 0x0000000000000000 0000000000000000   V1 0x0000000000000000 0000000000000000
>   V2 0x0000000000000000 0000000000000000   V3 0x0000000000000000 0000000000000000
>   V4 0x0000000000000000 0000000000000000   V5 0x0000000000000000 0000000000000000
>   V6 0x0000000000000000 0000000000000000   V7 0x0000000000000000 0000000000000000
>   V8 0x0000000000000000 0000000000000000   V9 0x0000000000000000 0000000000000000
>  V10 0x0000000000000000 0000000000000000  V11 0x0000000000000000 0000000000000000
>  V12 0x0000000000000000 0000000000000000  V13 0x0000000000000000 0000000000000000
>  V14 0x0000000000000000 0000000000000000  V15 0x0000000000000000 0000000000000000
>  V16 0x0000000000000000 0000000000000000  V17 0x0000000000000000 0000000000000000
>  V18 0x0000000000000000 0000000000000000  V19 0x0000000000000000 0000000000000000
>  V20 0x0000000000000000 0000000000000000  V21 0x0000000000000000 0000000000000000
>  V22 0x0000000000000000 0000000000000000  V23 0x0000000000000000 0000000000000000
>  V24 0x0000000000000000 0000000000000000  V25 0x0000000000000000 0000000000000000
>  V26 0x0000000000000000 0000000000000000  V27 0x0000000000000000 0000000000000000
>  V28 0x0000000000000000 0000000000000000  V29 0x0000000000000000 0000000000000000
>  V30 0x0000000000000000 0000000000000000  V31 0x0000000000000000 0000000000000000
>
>   SP 0x00000000476ED840  ELR 0x000000037FD5BDE0  SPSR 0x80000205  FPSR 0x00000000
>  ESR 0x86000005          FAR 0x000000037FD5BDE0
>
>  ESR : EC 0x21  IL 0x1  ISS 0x00000005
>
> Instruction abort: Translation fault, first level
>
> Stack dump:
>   00000476ED740: 000000037CAA0000 000000037CAA0000 0060000000000000 FF9F000000000F3F
>   00000476ED760: 00000000476ED790 000000037FD610D8 0060000000000003 0000000C00000001
>   00000476ED780: 000000037CA0070D 000000037F179000 00000000476ED840 000000037FD5BCE0
>   00000476ED7A0: 0000000340000000 0000000000000001 000000037F179000 0000000047FFE000
>   00000476ED7C0: 0000000000000068 000000003FFFFFFF 000000037CAA0000 0000000000000002
>   00000476ED7E0: 000000037F179000 0000000047FFE068 000000037CC00000 000000037CAA0000
>   00000476ED800: 0060000000000000 FF9F000000000F3F 00000000476ED840 000000037FD610D8
>   00000476ED820: 0060000000000001 0000001500000001 000000034000070D 000000037F177000
> > 00000476ED840: 00000000476ED8F0 000000037FD5BCE0 0000000047FFA000 0000000000000000
>   00000476ED860: 0000000047FFE000 0000000047FFF000 0000000000000000 0000007FFFFFFFFF
>   00000476ED880: 000000037CAA0000 0000000000000001 0000000047717588 0000000047FFF000
>   00000476ED8A0: 0000000380000000 000000037CAA0000 0060000000000000 FF9F000000000F3F
>   00000476ED8C0: 000000017FD605DD 000000037FD610D8 0060000000000001 0000001E00000001
>   00000476ED8E0: 0060000000000000 000000037F179000 00000000476ED9A0 000000037FD5C054
>   00000476ED900: 002000000000041C 0000000000000000 0000000047FFA000 0000000000004000
>   00000476ED920: 0000000334AA6000 0000000047FFF000 000000037F17A238 0000000047717000
> ASSERT [ArmCpuDxe] /builddir/build/BUILD/edk2-fff6d81270b5/ArmPkg/Library/DefaultExceptionHandlerLib/AArch64/DefaultExceptionHandler.c(333): ((BOOLEAN)(0==1))
> ----------------------------------------------------------------
> Thanks,
>  Oliver
>

Re: [edk2-devel] [PATCH v2 2/2] ArmVirtPkg/ArmVirtQemu: Avoid early ID map on ThunderX

[Oliver Steffen]

[-- Attachment #1: Type: text/plain, Size: 9087 bytes --]

On Wed, Feb 1, 2023 at 12:52 PM Ard Biesheuvel <ardb@kernel.org> wrote:

> On Wed, 1 Feb 2023 at 10:14, Oliver Steffen <osteffen@redhat.com> wrote:
> >
> > Hi everyone!
> >
> > On Thu, Jan 19, 2023 at 2:21 PM Ard Biesheuvel <ardb@kernel.org> wrote:
> > >
> > > On Thu, 19 Jan 2023 at 13:55, Oliver Steffen <osteffen@redhat.com>
> wrote:
> > > >
> > > > Quoting Gerd Hoffmann (2023-01-19 13:00:21)
> > > > >   Hi,
> > > > >
> > > > > > > I tried the most recent Qemu master (v7.2.50) and also v7.0.0,
> > > > > > > on the 5.14 (RHEL) kernel and on 6.1.6-200.fc37.aarch64 (from
> Fedora).
> > > > > > > No luck.
> > > > > > >
> > > > > >
> > > > > > Does that include a backport of commit
> 406504c7b0405d74d74c15a667cd4c4620c3e7a9?
> > > > >
> > > > > Probably not given that fedora kernels are almost vanilla and the
> > > > > backport landed in stable in v6.1.7 (commit 9a1195c584321).
> > > > >
> > > > > Oliver: try pulling the latest kernel directly from koji:
> > > > > https://koji.fedoraproject.org/koji/buildinfo?buildID=2112315
> > > > >
> > > > > take care,
> > > > >   Gerd
> > > > >
> > > >
> > > > Thanks for the koji link, Gerd.
> > > >
> > > > It works with 6.1.7-200.fc37.aarch64.
> > > > Perfect.
> > > >
> > > > Thanks Ard, thanks Marc!
> > > >
> > >
> > > Thanks for the report. Up to this point, Marc is the only one who had
> > > managed to reproduce this afaik, so having another data point where
> > > the fix works as intended is rather nice.
> > >
> >
> > I am sorry, this story does not seem to be over yet.
> >
> > We are using the Erratum patch and also included the commit 406504c7 in
> > the kernel.
> > Now the firmware crashes sometimes (10 out of 89 tests).
> >
>
> Thanks for the report. Is this still on ThunderX2?
>
> > Any hints are very welcome!
> >
>
> Do  you have access to those build artifacts?
>

https://kojihub.stream.centos.org/kojifiles/work/tasks/5251/1835251/edk2-aarch64-20221207gitfff6d81270b5-4.el9.test.noarch.rpm

and/or here:

https://kojihub.stream.centos.org/koji/taskinfo?taskID=1835251

Source for reference:
https://gitlab.com/redhat/centos-stream/src/edk2/-/merge_requests/24

Can you share more of the log output?
> If you can rebuild, can you run it with DEBUG_VERBOSE set on ArmCpuDxe.
> E.g.,
>
> --- a/ArmVirtPkg/ArmVirtQemu.dsc
> +++ b/ArmVirtPkg/ArmVirtQemu.dsc
> @@ -383,7 +383,11 @@ [Components.common]
>    #
>    # Architectural Protocols
>    #
> -  ArmPkg/Drivers/CpuDxe/CpuDxe.inf
> +  ArmPkg/Drivers/CpuDxe/CpuDxe.inf {
> +    <PcdsFixedAtBuild>
> +      gEfiMdePkgTokenSpaceGuid.PcdDebugPrintErrorLevel|0x8040004F
> +  }
> +
>    MdeModulePkg/Core/RuntimeDxe/RuntimeDxe.inf
>    MdeModulePkg/Universal/Variable/RuntimeDxe/VariableRuntimeDxe.inf {
>      <LibraryClasses>
>
I'll try to do that.



> > Here is the serial output of one case:
> > ----------------------------------------------------------------
> > UEFI firmware (version edk2-20221207gitfff6d81270b5-4.el9.test built at
> 00:00:00 on Jan 18 2023)
> > SyncPcrAllocationsAndPcrMask!
> >
> >
> > Synchronous Exception at 0x000000037FD5BDE0
> > PC 0x00037FD5BDE0 (0x00037FD59000+0x00002DE0) [ 0] ArmCpuDxe.dll
> > PC 0x00037FD5BDE0 (0x00037FD59000+0x00002DE0) [ 0] ArmCpuDxe.dll
> > PC 0x00037FD5BCE0 (0x00037FD59000+0x00002CE0) [ 0] ArmCpuDxe.dll
>
> This, along with
>
> >   SP 0x00000000476ED840  ELR 0x000000037FD5BDE0  SPSR 0x80000205  FPSR
> 0x00000000
> >  ESR 0x86000005          FAR 0x000000037FD5BDE0
>
> taken from below suggests that UpdateRegionMappingRecursive () is
> unmapping itself, likely when splitting a block entry.
>
> This affects all ARM builds, not just ArmVIrtQemu so I would like to
> get to the bottom of this.
>
>
> Thanks,
> Ard.
>
>
>
> > PC 0x00037FD5C054 (0x00037FD59000+0x00003054) [ 0] ArmCpuDxe.dll
> > PC 0x0000476F08EC (0x0000476EE000+0x000028EC) [ 1] DxeCore.dll
> > PC 0x0000476F65C0 (0x0000476EE000+0x000085C0) [ 1] DxeCore.dll
> > PC 0x0000476FB5B8 (0x0000476EE000+0x0000D5B8) [ 1] DxeCore.dll
> > PC 0x0000476F5DB8 (0x0000476EE000+0x00007DB8) [ 1] DxeCore.dll
> > PC 0x00037FD5CD50 (0x00037FD59000+0x00003D50) [ 2] ArmCpuDxe.dll
> > PC 0x0000476F5758 (0x0000476EE000+0x00007758) [ 3] DxeCore.dll
> > PC 0x000047702B18 (0x0000476EE000+0x00014B18) [ 3] DxeCore.dll
> > PC 0x0000476F8D70 (0x0000476EE000+0x0000AD70) [ 3] DxeCore.dll
> >
> > [ 0]
> /builddir/build/BUILD/edk2-fff6d81270b5/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/ArmPkg/Drivers/CpuDxe/CpuDxe/DEBUG/ArmCpuDxe.dll
> > [ 1]
> /builddir/build/BUILD/edk2-fff6d81270b5/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/MdeModulePkg/Core/Dxe/DxeMain/DEBUG/DxeCore.dll
> > [ 2]
> /builddir/build/BUILD/edk2-fff6d81270b5/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/ArmPkg/Drivers/CpuDxe/CpuDxe/DEBUG/ArmCpuDxe.dll
> > [ 3]
> /builddir/build/BUILD/edk2-fff6d81270b5/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/MdeModulePkg/Core/Dxe/DxeMain/DEBUG/DxeCore.dll
> >
> >   X0 0x0000000047FFE068   X1 0x000000037F179003   X2 0x0000000000340000
>  X3 0x0000000000000000
> >   X4 0x0000000000000200   X5 0x0000000000000004   X6 0x0060000000000000
>  X7 0xFF9F000000000F3F
> >   X8 0x000000037FFFF008   X9 0x0000000400000000  X10 0x000000037F177000
> X11 0x000000037FA37FFF
> >  X12 0x0000000000000000  X13 0x0000000000000008  X14 0x0000000000000000
> X15 0x0000000000000000
> >  X16 0x000000037FD5A208  X17 0x00000000007979D0  X18 0x0000000000000000
> X19 0x0000000340000000
> >  X20 0x0000000000000001  X21 0x000000037F179003  X22 0x0000000047FFE000
> X23 0x0000000000000068
> >  X24 0x000000003FFFFFFF  X25 0x000000037CAA0000  X26 0x0000000000000002
> X27 0x000000037F179000
> >  X28 0x0000000047FFE068   FP 0x00000000476ED840   LR 0x000000037FD5BDE0
> >
> >   V0 0x0000000000000000 0000000000000000   V1 0x0000000000000000
> 0000000000000000
> >   V2 0x0000000000000000 0000000000000000   V3 0x0000000000000000
> 0000000000000000
> >   V4 0x0000000000000000 0000000000000000   V5 0x0000000000000000
> 0000000000000000
> >   V6 0x0000000000000000 0000000000000000   V7 0x0000000000000000
> 0000000000000000
> >   V8 0x0000000000000000 0000000000000000   V9 0x0000000000000000
> 0000000000000000
> >  V10 0x0000000000000000 0000000000000000  V11 0x0000000000000000
> 0000000000000000
> >  V12 0x0000000000000000 0000000000000000  V13 0x0000000000000000
> 0000000000000000
> >  V14 0x0000000000000000 0000000000000000  V15 0x0000000000000000
> 0000000000000000
> >  V16 0x0000000000000000 0000000000000000  V17 0x0000000000000000
> 0000000000000000
> >  V18 0x0000000000000000 0000000000000000  V19 0x0000000000000000
> 0000000000000000
> >  V20 0x0000000000000000 0000000000000000  V21 0x0000000000000000
> 0000000000000000
> >  V22 0x0000000000000000 0000000000000000  V23 0x0000000000000000
> 0000000000000000
> >  V24 0x0000000000000000 0000000000000000  V25 0x0000000000000000
> 0000000000000000
> >  V26 0x0000000000000000 0000000000000000  V27 0x0000000000000000
> 0000000000000000
> >  V28 0x0000000000000000 0000000000000000  V29 0x0000000000000000
> 0000000000000000
> >  V30 0x0000000000000000 0000000000000000  V31 0x0000000000000000
> 0000000000000000
> >
> >   SP 0x00000000476ED840  ELR 0x000000037FD5BDE0  SPSR 0x80000205  FPSR
> 0x00000000
> >  ESR 0x86000005          FAR 0x000000037FD5BDE0
> >
> >  ESR : EC 0x21  IL 0x1  ISS 0x00000005
> >
> > Instruction abort: Translation fault, first level
> >
> > Stack dump:
> >   00000476ED740: 000000037CAA0000 000000037CAA0000 0060000000000000
> FF9F000000000F3F
> >   00000476ED760: 00000000476ED790 000000037FD610D8 0060000000000003
> 0000000C00000001
> >   00000476ED780: 000000037CA0070D 000000037F179000 00000000476ED840
> 000000037FD5BCE0
> >   00000476ED7A0: 0000000340000000 0000000000000001 000000037F179000
> 0000000047FFE000
> >   00000476ED7C0: 0000000000000068 000000003FFFFFFF 000000037CAA0000
> 0000000000000002
> >   00000476ED7E0: 000000037F179000 0000000047FFE068 000000037CC00000
> 000000037CAA0000
> >   00000476ED800: 0060000000000000 FF9F000000000F3F 00000000476ED840
> 000000037FD610D8
> >   00000476ED820: 0060000000000001 0000001500000001 000000034000070D
> 000000037F177000
> > > 00000476ED840: 00000000476ED8F0 000000037FD5BCE0 0000000047FFA000
> 0000000000000000
> >   00000476ED860: 0000000047FFE000 0000000047FFF000 0000000000000000
> 0000007FFFFFFFFF
> >   00000476ED880: 000000037CAA0000 0000000000000001 0000000047717588
> 0000000047FFF000
> >   00000476ED8A0: 0000000380000000 000000037CAA0000 0060000000000000
> FF9F000000000F3F
> >   00000476ED8C0: 000000017FD605DD 000000037FD610D8 0060000000000001
> 0000001E00000001
> >   00000476ED8E0: 0060000000000000 000000037F179000 00000000476ED9A0
> 000000037FD5C054
> >   00000476ED900: 002000000000041C 0000000000000000 0000000047FFA000
> 0000000000004000
> >   00000476ED920: 0000000334AA6000 0000000047FFF000 000000037F17A238
> 0000000047717000
> > ASSERT [ArmCpuDxe]
> /builddir/build/BUILD/edk2-fff6d81270b5/ArmPkg/Library/DefaultExceptionHandlerLib/AArch64/DefaultExceptionHandler.c(333):
> ((BOOLEAN)(0==1))
> > ----------------------------------------------------------------
> > Thanks,
> >  Oliver
> >
>
>

[-- Attachment #2: Type: text/html, Size: 12310 bytes --]

Re: [edk2-devel] [PATCH v2 2/2] ArmVirtPkg/ArmVirtQemu: Avoid early ID map on ThunderX

[Ard Biesheuvel]

On Wed, 1 Feb 2023 at 13:59, Oliver Steffen <osteffen@redhat.com> wrote:
>
>
>
> On Wed, Feb 1, 2023 at 12:52 PM Ard Biesheuvel <ardb@kernel.org> wrote:
>>
>> On Wed, 1 Feb 2023 at 10:14, Oliver Steffen <osteffen@redhat.com> wrote:
>> >
>> > Hi everyone!
>> >
>> > On Thu, Jan 19, 2023 at 2:21 PM Ard Biesheuvel <ardb@kernel.org> wrote:
>> > >
>> > > On Thu, 19 Jan 2023 at 13:55, Oliver Steffen <osteffen@redhat.com> wrote:
>> > > >
>> > > > Quoting Gerd Hoffmann (2023-01-19 13:00:21)
>> > > > >   Hi,
>> > > > >
>> > > > > > > I tried the most recent Qemu master (v7.2.50) and also v7.0.0,
>> > > > > > > on the 5.14 (RHEL) kernel and on 6.1.6-200.fc37.aarch64 (from Fedora).
>> > > > > > > No luck.
>> > > > > > >
>> > > > > >
>> > > > > > Does that include a backport of commit 406504c7b0405d74d74c15a667cd4c4620c3e7a9?
>> > > > >
>> > > > > Probably not given that fedora kernels are almost vanilla and the
>> > > > > backport landed in stable in v6.1.7 (commit 9a1195c584321).
>> > > > >
>> > > > > Oliver: try pulling the latest kernel directly from koji:
>> > > > > https://koji.fedoraproject.org/koji/buildinfo?buildID=2112315
>> > > > >
>> > > > > take care,
>> > > > >   Gerd
>> > > > >
>> > > >
>> > > > Thanks for the koji link, Gerd.
>> > > >
>> > > > It works with 6.1.7-200.fc37.aarch64.
>> > > > Perfect.
>> > > >
>> > > > Thanks Ard, thanks Marc!
>> > > >
>> > >
>> > > Thanks for the report. Up to this point, Marc is the only one who had
>> > > managed to reproduce this afaik, so having another data point where
>> > > the fix works as intended is rather nice.
>> > >
>> >
>> > I am sorry, this story does not seem to be over yet.
>> >
>> > We are using the Erratum patch and also included the commit 406504c7 in
>> > the kernel.
>> > Now the firmware crashes sometimes (10 out of 89 tests).
>> >
>>
>> Thanks for the report. Is this still on ThunderX2?
>>
>> > Any hints are very welcome!
>> >
>>
>> Do  you have access to those build artifacts?
>
>
> https://kojihub.stream.centos.org/kojifiles/work/tasks/5251/1835251/edk2-aarch64-20221207gitfff6d81270b5-4.el9.test.noarch.rpm
>
> and/or here:
>
> https://kojihub.stream.centos.org/koji/taskinfo?taskID=1835251
>
> Source for reference:
> https://gitlab.com/redhat/centos-stream/src/edk2/-/merge_requests/24
>

Any chance the .dll files (which are actually ELF executables) have
been preserved somewhere?

>> Can you share more of the log output?
>> If you can rebuild, can you run it with DEBUG_VERBOSE set on ArmCpuDxe. E.g.,
>>
>> --- a/ArmVirtPkg/ArmVirtQemu.dsc
>> +++ b/ArmVirtPkg/ArmVirtQemu.dsc
>> @@ -383,7 +383,11 @@ [Components.common]
>>    #
>>    # Architectural Protocols
>>    #
>> -  ArmPkg/Drivers/CpuDxe/CpuDxe.inf
>> +  ArmPkg/Drivers/CpuDxe/CpuDxe.inf {
>> +    <PcdsFixedAtBuild>
>> +      gEfiMdePkgTokenSpaceGuid.PcdDebugPrintErrorLevel|0x8040004F
>> +  }
>> +
>>    MdeModulePkg/Core/RuntimeDxe/RuntimeDxe.inf
>>    MdeModulePkg/Universal/Variable/RuntimeDxe/VariableRuntimeDxe.inf {
>>      <LibraryClasses>
>
> I'll try to do that.
>
>
>>
>> > Here is the serial output of one case:
>> > ----------------------------------------------------------------
>> > UEFI firmware (version edk2-20221207gitfff6d81270b5-4.el9.test built at 00:00:00 on Jan 18 2023)
>> > SyncPcrAllocationsAndPcrMask!
>> >
>> >
>> > Synchronous Exception at 0x000000037FD5BDE0
>> > PC 0x00037FD5BDE0 (0x00037FD59000+0x00002DE0) [ 0] ArmCpuDxe.dll
>> > PC 0x00037FD5BDE0 (0x00037FD59000+0x00002DE0) [ 0] ArmCpuDxe.dll
>> > PC 0x00037FD5BCE0 (0x00037FD59000+0x00002CE0) [ 0] ArmCpuDxe.dll
>>
>> This, along with
>>
>> >   SP 0x00000000476ED840  ELR 0x000000037FD5BDE0  SPSR 0x80000205  FPSR 0x00000000
>> >  ESR 0x86000005          FAR 0x000000037FD5BDE0
>>
>> taken from below suggests that UpdateRegionMappingRecursive () is
>> unmapping itself, likely when splitting a block entry.
>>
>> This affects all ARM builds, not just ArmVIrtQemu so I would like to
>> get to the bottom of this.
>>
>>
>> Thanks,
>> Ard.
>>
>>
>>
>> > PC 0x00037FD5C054 (0x00037FD59000+0x00003054) [ 0] ArmCpuDxe.dll
>> > PC 0x0000476F08EC (0x0000476EE000+0x000028EC) [ 1] DxeCore.dll
>> > PC 0x0000476F65C0 (0x0000476EE000+0x000085C0) [ 1] DxeCore.dll
>> > PC 0x0000476FB5B8 (0x0000476EE000+0x0000D5B8) [ 1] DxeCore.dll
>> > PC 0x0000476F5DB8 (0x0000476EE000+0x00007DB8) [ 1] DxeCore.dll
>> > PC 0x00037FD5CD50 (0x00037FD59000+0x00003D50) [ 2] ArmCpuDxe.dll
>> > PC 0x0000476F5758 (0x0000476EE000+0x00007758) [ 3] DxeCore.dll
>> > PC 0x000047702B18 (0x0000476EE000+0x00014B18) [ 3] DxeCore.dll
>> > PC 0x0000476F8D70 (0x0000476EE000+0x0000AD70) [ 3] DxeCore.dll
>> >
>> > [ 0] /builddir/build/BUILD/edk2-fff6d81270b5/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/ArmPkg/Drivers/CpuDxe/CpuDxe/DEBUG/ArmCpuDxe.dll
>> > [ 1] /builddir/build/BUILD/edk2-fff6d81270b5/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/MdeModulePkg/Core/Dxe/DxeMain/DEBUG/DxeCore.dll
>> > [ 2] /builddir/build/BUILD/edk2-fff6d81270b5/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/ArmPkg/Drivers/CpuDxe/CpuDxe/DEBUG/ArmCpuDxe.dll
>> > [ 3] /builddir/build/BUILD/edk2-fff6d81270b5/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/MdeModulePkg/Core/Dxe/DxeMain/DEBUG/DxeCore.dll
>> >
>> >   X0 0x0000000047FFE068   X1 0x000000037F179003   X2 0x0000000000340000   X3 0x0000000000000000
>> >   X4 0x0000000000000200   X5 0x0000000000000004   X6 0x0060000000000000   X7 0xFF9F000000000F3F
>> >   X8 0x000000037FFFF008   X9 0x0000000400000000  X10 0x000000037F177000  X11 0x000000037FA37FFF
>> >  X12 0x0000000000000000  X13 0x0000000000000008  X14 0x0000000000000000  X15 0x0000000000000000
>> >  X16 0x000000037FD5A208  X17 0x00000000007979D0  X18 0x0000000000000000  X19 0x0000000340000000
>> >  X20 0x0000000000000001  X21 0x000000037F179003  X22 0x0000000047FFE000  X23 0x0000000000000068
>> >  X24 0x000000003FFFFFFF  X25 0x000000037CAA0000  X26 0x0000000000000002  X27 0x000000037F179000
>> >  X28 0x0000000047FFE068   FP 0x00000000476ED840   LR 0x000000037FD5BDE0
>> >
>> >   V0 0x0000000000000000 0000000000000000   V1 0x0000000000000000 0000000000000000
>> >   V2 0x0000000000000000 0000000000000000   V3 0x0000000000000000 0000000000000000
>> >   V4 0x0000000000000000 0000000000000000   V5 0x0000000000000000 0000000000000000
>> >   V6 0x0000000000000000 0000000000000000   V7 0x0000000000000000 0000000000000000
>> >   V8 0x0000000000000000 0000000000000000   V9 0x0000000000000000 0000000000000000
>> >  V10 0x0000000000000000 0000000000000000  V11 0x0000000000000000 0000000000000000
>> >  V12 0x0000000000000000 0000000000000000  V13 0x0000000000000000 0000000000000000
>> >  V14 0x0000000000000000 0000000000000000  V15 0x0000000000000000 0000000000000000
>> >  V16 0x0000000000000000 0000000000000000  V17 0x0000000000000000 0000000000000000
>> >  V18 0x0000000000000000 0000000000000000  V19 0x0000000000000000 0000000000000000
>> >  V20 0x0000000000000000 0000000000000000  V21 0x0000000000000000 0000000000000000
>> >  V22 0x0000000000000000 0000000000000000  V23 0x0000000000000000 0000000000000000
>> >  V24 0x0000000000000000 0000000000000000  V25 0x0000000000000000 0000000000000000
>> >  V26 0x0000000000000000 0000000000000000  V27 0x0000000000000000 0000000000000000
>> >  V28 0x0000000000000000 0000000000000000  V29 0x0000000000000000 0000000000000000
>> >  V30 0x0000000000000000 0000000000000000  V31 0x0000000000000000 0000000000000000
>> >
>> >   SP 0x00000000476ED840  ELR 0x000000037FD5BDE0  SPSR 0x80000205  FPSR 0x00000000
>> >  ESR 0x86000005          FAR 0x000000037FD5BDE0
>> >
>> >  ESR : EC 0x21  IL 0x1  ISS 0x00000005
>> >
>> > Instruction abort: Translation fault, first level
>> >
>> > Stack dump:
>> >   00000476ED740: 000000037CAA0000 000000037CAA0000 0060000000000000 FF9F000000000F3F
>> >   00000476ED760: 00000000476ED790 000000037FD610D8 0060000000000003 0000000C00000001
>> >   00000476ED780: 000000037CA0070D 000000037F179000 00000000476ED840 000000037FD5BCE0
>> >   00000476ED7A0: 0000000340000000 0000000000000001 000000037F179000 0000000047FFE000
>> >   00000476ED7C0: 0000000000000068 000000003FFFFFFF 000000037CAA0000 0000000000000002
>> >   00000476ED7E0: 000000037F179000 0000000047FFE068 000000037CC00000 000000037CAA0000
>> >   00000476ED800: 0060000000000000 FF9F000000000F3F 00000000476ED840 000000037FD610D8
>> >   00000476ED820: 0060000000000001 0000001500000001 000000034000070D 000000037F177000
>> > > 00000476ED840: 00000000476ED8F0 000000037FD5BCE0 0000000047FFA000 0000000000000000
>> >   00000476ED860: 0000000047FFE000 0000000047FFF000 0000000000000000 0000007FFFFFFFFF
>> >   00000476ED880: 000000037CAA0000 0000000000000001 0000000047717588 0000000047FFF000
>> >   00000476ED8A0: 0000000380000000 000000037CAA0000 0060000000000000 FF9F000000000F3F
>> >   00000476ED8C0: 000000017FD605DD 000000037FD610D8 0060000000000001 0000001E00000001
>> >   00000476ED8E0: 0060000000000000 000000037F179000 00000000476ED9A0 000000037FD5C054
>> >   00000476ED900: 002000000000041C 0000000000000000 0000000047FFA000 0000000000004000
>> >   00000476ED920: 0000000334AA6000 0000000047FFF000 000000037F17A238 0000000047717000
>> > ASSERT [ArmCpuDxe] /builddir/build/BUILD/edk2-fff6d81270b5/ArmPkg/Library/DefaultExceptionHandlerLib/AArch64/DefaultExceptionHandler.c(333): ((BOOLEAN)(0==1))
>> > ----------------------------------------------------------------
>> > Thanks,
>> >  Oliver
>> >
>>
> 

Re: [edk2-devel] [PATCH v2 2/2] ArmVirtPkg/ArmVirtQemu: Avoid early ID map on ThunderX

[Oliver Steffen]

[-- Attachment #1: Type: text/plain, Size: 8684 bytes --]

On Wed, Feb 1, 2023 at 2:29 PM Ard Biesheuvel <ardb@kernel.org> wrote:

> On Wed, 1 Feb 2023 at 13:59, Oliver Steffen <osteffen@redhat.com> wrote:
> >
> > On Wed, Feb 1, 2023 at 12:52 PM Ard Biesheuvel <ardb@kernel.org> wrote:
> >>
> >> On Wed, 1 Feb 2023 at 10:14, Oliver Steffen <osteffen@redhat.com>
> wrote:
> >> >
>
[...]

> >> > I am sorry, this story does not seem to be over yet.
> >> >
> >> > We are using the Erratum patch and also included the commit 406504c7
> in
> >> > the kernel.
> >> > Now the firmware crashes sometimes (10 out of 89 tests).
> >> >
> >>
> >> Thanks for the report. Is this still on ThunderX2?
> >>
> >> > Any hints are very welcome!
> >> >
> >>
> >> Do  you have access to those build artifacts?
> >
> >
> >
> https://kojihub.stream.centos.org/kojifiles/work/tasks/5251/1835251/edk2-aarch64-20221207gitfff6d81270b5-4.el9.test.noarch.rpm
> >
> > and/or here:
> >
> > https://kojihub.stream.centos.org/koji/taskinfo?taskID=1835251
> >
> > Source for reference:
> > https://gitlab.com/redhat/centos-stream/src/edk2/-/merge_requests/24
> >
>
> Any chance the .dll files (which are actually ELF executables) have
> been preserved somewhere?
>
Here is the build folder (~90MB):
https://gitlab.com/osteffen/thunderx2-debug/-/raw/main/armvirt-thunderx2-issue.tar.xz

I am waiting for the tests with the additional debug output to run.


> >> Can you share more of the log output?
> >> If you can rebuild, can you run it with DEBUG_VERBOSE set on ArmCpuDxe.
> E.g.,
> >>
> >> --- a/ArmVirtPkg/ArmVirtQemu.dsc
> >> +++ b/ArmVirtPkg/ArmVirtQemu.dsc
> >> @@ -383,7 +383,11 @@ [Components.common]
> >>    #
> >>    # Architectural Protocols
> >>    #
> >> -  ArmPkg/Drivers/CpuDxe/CpuDxe.inf
> >> +  ArmPkg/Drivers/CpuDxe/CpuDxe.inf {
> >> +    <PcdsFixedAtBuild>
> >> +      gEfiMdePkgTokenSpaceGuid.PcdDebugPrintErrorLevel|0x8040004F
> >> +  }
> >> +
> >>    MdeModulePkg/Core/RuntimeDxe/RuntimeDxe.inf
> >>    MdeModulePkg/Universal/Variable/RuntimeDxe/VariableRuntimeDxe.inf {
> >>      <LibraryClasses>
> >
> > I'll try to do that.
> >
> >
> >>
> >> > Here is the serial output of one case:
> >> > ----------------------------------------------------------------
> >> > UEFI firmware (version edk2-20221207gitfff6d81270b5-4.el9.test built
> at 00:00:00 on Jan 18 2023)
> >> > SyncPcrAllocationsAndPcrMask!
> >> >
> >> >
> >> > Synchronous Exception at 0x000000037FD5BDE0
> >> > PC 0x00037FD5BDE0 (0x00037FD59000+0x00002DE0) [ 0] ArmCpuDxe.dll
> >> > PC 0x00037FD5BDE0 (0x00037FD59000+0x00002DE0) [ 0] ArmCpuDxe.dll
> >> > PC 0x00037FD5BCE0 (0x00037FD59000+0x00002CE0) [ 0] ArmCpuDxe.dll
> >>
> >> This, along with
> >>
> >> >   SP 0x00000000476ED840  ELR 0x000000037FD5BDE0  SPSR 0x80000205
> FPSR 0x00000000
> >> >  ESR 0x86000005          FAR 0x000000037FD5BDE0
> >>
> >> taken from below suggests that UpdateRegionMappingRecursive () is
> >> unmapping itself, likely when splitting a block entry.
> >>
> >> This affects all ARM builds, not just ArmVIrtQemu so I would like to
> >> get to the bottom of this.
> >>
> >>
> >> Thanks,
> >> Ard.
> >>
> >>
> >>
> >> > PC 0x00037FD5C054 (0x00037FD59000+0x00003054) [ 0] ArmCpuDxe.dll
> >> > PC 0x0000476F08EC (0x0000476EE000+0x000028EC) [ 1] DxeCore.dll
> >> > PC 0x0000476F65C0 (0x0000476EE000+0x000085C0) [ 1] DxeCore.dll
> >> > PC 0x0000476FB5B8 (0x0000476EE000+0x0000D5B8) [ 1] DxeCore.dll
> >> > PC 0x0000476F5DB8 (0x0000476EE000+0x00007DB8) [ 1] DxeCore.dll
> >> > PC 0x00037FD5CD50 (0x00037FD59000+0x00003D50) [ 2] ArmCpuDxe.dll
> >> > PC 0x0000476F5758 (0x0000476EE000+0x00007758) [ 3] DxeCore.dll
> >> > PC 0x000047702B18 (0x0000476EE000+0x00014B18) [ 3] DxeCore.dll
> >> > PC 0x0000476F8D70 (0x0000476EE000+0x0000AD70) [ 3] DxeCore.dll
> >> >
> >> > [ 0]
> /builddir/build/BUILD/edk2-fff6d81270b5/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/ArmPkg/Drivers/CpuDxe/CpuDxe/DEBUG/ArmCpuDxe.dll
> >> > [ 1]
> /builddir/build/BUILD/edk2-fff6d81270b5/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/MdeModulePkg/Core/Dxe/DxeMain/DEBUG/DxeCore.dll
> >> > [ 2]
> /builddir/build/BUILD/edk2-fff6d81270b5/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/ArmPkg/Drivers/CpuDxe/CpuDxe/DEBUG/ArmCpuDxe.dll
> >> > [ 3]
> /builddir/build/BUILD/edk2-fff6d81270b5/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/MdeModulePkg/Core/Dxe/DxeMain/DEBUG/DxeCore.dll
> >> >
> >> >   X0 0x0000000047FFE068   X1 0x000000037F179003   X2
> 0x0000000000340000   X3 0x0000000000000000
> >> >   X4 0x0000000000000200   X5 0x0000000000000004   X6
> 0x0060000000000000   X7 0xFF9F000000000F3F
> >> >   X8 0x000000037FFFF008   X9 0x0000000400000000  X10
> 0x000000037F177000  X11 0x000000037FA37FFF
> >> >  X12 0x0000000000000000  X13 0x0000000000000008  X14
> 0x0000000000000000  X15 0x0000000000000000
> >> >  X16 0x000000037FD5A208  X17 0x00000000007979D0  X18
> 0x0000000000000000  X19 0x0000000340000000
> >> >  X20 0x0000000000000001  X21 0x000000037F179003  X22
> 0x0000000047FFE000  X23 0x0000000000000068
> >> >  X24 0x000000003FFFFFFF  X25 0x000000037CAA0000  X26
> 0x0000000000000002  X27 0x000000037F179000
> >> >  X28 0x0000000047FFE068   FP 0x00000000476ED840   LR
> 0x000000037FD5BDE0
> >> >
> >> >   V0 0x0000000000000000 0000000000000000   V1 0x0000000000000000
> 0000000000000000
> >> >   V2 0x0000000000000000 0000000000000000   V3 0x0000000000000000
> 0000000000000000
> >> >   V4 0x0000000000000000 0000000000000000   V5 0x0000000000000000
> 0000000000000000
> >> >   V6 0x0000000000000000 0000000000000000   V7 0x0000000000000000
> 0000000000000000
> >> >   V8 0x0000000000000000 0000000000000000   V9 0x0000000000000000
> 0000000000000000
> >> >  V10 0x0000000000000000 0000000000000000  V11 0x0000000000000000
> 0000000000000000
> >> >  V12 0x0000000000000000 0000000000000000  V13 0x0000000000000000
> 0000000000000000
> >> >  V14 0x0000000000000000 0000000000000000  V15 0x0000000000000000
> 0000000000000000
> >> >  V16 0x0000000000000000 0000000000000000  V17 0x0000000000000000
> 0000000000000000
> >> >  V18 0x0000000000000000 0000000000000000  V19 0x0000000000000000
> 0000000000000000
> >> >  V20 0x0000000000000000 0000000000000000  V21 0x0000000000000000
> 0000000000000000
> >> >  V22 0x0000000000000000 0000000000000000  V23 0x0000000000000000
> 0000000000000000
> >> >  V24 0x0000000000000000 0000000000000000  V25 0x0000000000000000
> 0000000000000000
> >> >  V26 0x0000000000000000 0000000000000000  V27 0x0000000000000000
> 0000000000000000
> >> >  V28 0x0000000000000000 0000000000000000  V29 0x0000000000000000
> 0000000000000000
> >> >  V30 0x0000000000000000 0000000000000000  V31 0x0000000000000000
> 0000000000000000
> >> >
> >> >   SP 0x00000000476ED840  ELR 0x000000037FD5BDE0  SPSR 0x80000205
> FPSR 0x00000000
> >> >  ESR 0x86000005          FAR 0x000000037FD5BDE0
> >> >
> >> >  ESR : EC 0x21  IL 0x1  ISS 0x00000005
> >> >
> >> > Instruction abort: Translation fault, first level
> >> >
> >> > Stack dump:
> >> >   00000476ED740: 000000037CAA0000 000000037CAA0000 0060000000000000
> FF9F000000000F3F
> >> >   00000476ED760: 00000000476ED790 000000037FD610D8 0060000000000003
> 0000000C00000001
> >> >   00000476ED780: 000000037CA0070D 000000037F179000 00000000476ED840
> 000000037FD5BCE0
> >> >   00000476ED7A0: 0000000340000000 0000000000000001 000000037F179000
> 0000000047FFE000
> >> >   00000476ED7C0: 0000000000000068 000000003FFFFFFF 000000037CAA0000
> 0000000000000002
> >> >   00000476ED7E0: 000000037F179000 0000000047FFE068 000000037CC00000
> 000000037CAA0000
> >> >   00000476ED800: 0060000000000000 FF9F000000000F3F 00000000476ED840
> 000000037FD610D8
> >> >   00000476ED820: 0060000000000001 0000001500000001 000000034000070D
> 000000037F177000
> >> > > 00000476ED840: 00000000476ED8F0 000000037FD5BCE0 0000000047FFA000
> 0000000000000000
> >> >   00000476ED860: 0000000047FFE000 0000000047FFF000 0000000000000000
> 0000007FFFFFFFFF
> >> >   00000476ED880: 000000037CAA0000 0000000000000001 0000000047717588
> 0000000047FFF000
> >> >   00000476ED8A0: 0000000380000000 000000037CAA0000 0060000000000000
> FF9F000000000F3F
> >> >   00000476ED8C0: 000000017FD605DD 000000037FD610D8 0060000000000001
> 0000001E00000001
> >> >   00000476ED8E0: 0060000000000000 000000037F179000 00000000476ED9A0
> 000000037FD5C054
> >> >   00000476ED900: 002000000000041C 0000000000000000 0000000047FFA000
> 0000000000004000
> >> >   00000476ED920: 0000000334AA6000 0000000047FFF000 000000037F17A238
> 0000000047717000
> >> > ASSERT [ArmCpuDxe]
> /builddir/build/BUILD/edk2-fff6d81270b5/ArmPkg/Library/DefaultExceptionHandlerLib/AArch64/DefaultExceptionHandler.c(333):
> ((BOOLEAN)(0==1))
> >> > ----------------------------------------------------------------
> >> > Thanks,
> >> >  Oliver
> >> >
>
>

[-- Attachment #2: Type: text/html, Size: 12040 bytes --]

Re: [edk2-devel] [PATCH v2 2/2] ArmVirtPkg/ArmVirtQemu: Avoid early ID map on ThunderX

[Oliver Steffen]

[-- Attachment #1: Type: text/plain, Size: 2120 bytes --]

On Thu, Feb 2, 2023 at 12:09 PM Oliver Steffen <osteffen@redhat.com> wrote:

>
> On Wed, Feb 1, 2023 at 2:29 PM Ard Biesheuvel <ardb@kernel.org> wrote:
>
>> On Wed, 1 Feb 2023 at 13:59, Oliver Steffen <osteffen@redhat.com> wrote:
>> >
>> > On Wed, Feb 1, 2023 at 12:52 PM Ard Biesheuvel <ardb@kernel.org> wrote:
>> >>
>> >> On Wed, 1 Feb 2023 at 10:14, Oliver Steffen <osteffen@redhat.com>
>> wrote:
>> >> >
>>
> [...]
>
>> >> > I am sorry, this story does not seem to be over yet.
>> >> >
>> >> > We are using the Erratum patch and also included the commit 406504c7
>> in
>> >> > the kernel.
>> >> > Now the firmware crashes sometimes (10 out of 89 tests).
>> >> >
>> >>
>> >> Thanks for the report. Is this still on ThunderX2?
>> >>
>> >> > Any hints are very welcome!
>> >> >
>> >>
>> >> Do  you have access to those build artifacts?
>> >
>> >
>> >
>> https://kojihub.stream.centos.org/kojifiles/work/tasks/5251/1835251/edk2-aarch64-20221207gitfff6d81270b5-4.el9.test.noarch.rpm
>> >
>> > and/or here:
>> >
>> > https://kojihub.stream.centos.org/koji/taskinfo?taskID=1835251
>> >
>> > Source for reference:
>> > https://gitlab.com/redhat/centos-stream/src/edk2/-/merge_requests/24
>> >
>>
>> Any chance the .dll files (which are actually ELF executables) have
>> been preserved somewhere?
>>
> Here is the build folder (~90MB):
>
> https://gitlab.com/osteffen/thunderx2-debug/-/raw/main/armvirt-thunderx2-issue.tar.xz
>
> I am waiting for the tests with the additional debug output to run.
>

We reran the test suite with the Erratum and the additional debug
output enabled.  Strangely, the problem does not occur anymore, the
firmware boots up normally.

We retried the tests without the additional debug output.
RHEL ships two firmware flavors for AARCH64: a silent and a verbose
version. Both were tried. We see no problems with the verbose
one. The silent one fails noticeably more often if a software TPM device
is present.

Could this be related to how much stuff is going on in the early phase
of the firmware (when logging is enabled: formatting of messages and
sending to serial port...) ?

Thanks,
  Oliver

[-- Attachment #2: Type: text/html, Size: 4299 bytes --]

Re: [edk2-devel] [PATCH v2 2/2] ArmVirtPkg/ArmVirtQemu: Avoid early ID map on ThunderX

[Ard Biesheuvel]

On Tue, 7 Feb 2023 at 11:51, Oliver Steffen <osteffen@redhat.com> wrote:
>
> On Thu, Feb 2, 2023 at 12:09 PM Oliver Steffen <osteffen@redhat.com> wrote:
>>
>>
>> On Wed, Feb 1, 2023 at 2:29 PM Ard Biesheuvel <ardb@kernel.org> wrote:
>>>
>>> On Wed, 1 Feb 2023 at 13:59, Oliver Steffen <osteffen@redhat.com> wrote:
>>> >
>>> > On Wed, Feb 1, 2023 at 12:52 PM Ard Biesheuvel <ardb@kernel.org> wrote:
>>> >>
>>> >> On Wed, 1 Feb 2023 at 10:14, Oliver Steffen <osteffen@redhat.com> wrote:
>>> >> >
>>
>> [...]
>>>
>>> >> > I am sorry, this story does not seem to be over yet.
>>> >> >
>>> >> > We are using the Erratum patch and also included the commit 406504c7 in
>>> >> > the kernel.
>>> >> > Now the firmware crashes sometimes (10 out of 89 tests).
>>> >> >
>>> >>
>>> >> Thanks for the report. Is this still on ThunderX2?
>>> >>
>>> >> > Any hints are very welcome!
>>> >> >
>>> >>
>>> >> Do  you have access to those build artifacts?
>>> >
>>> >
>>> > https://kojihub.stream.centos.org/kojifiles/work/tasks/5251/1835251/edk2-aarch64-20221207gitfff6d81270b5-4.el9.test.noarch.rpm
>>> >
>>> > and/or here:
>>> >
>>> > https://kojihub.stream.centos.org/koji/taskinfo?taskID=1835251
>>> >
>>> > Source for reference:
>>> > https://gitlab.com/redhat/centos-stream/src/edk2/-/merge_requests/24
>>> >
>>>
>>> Any chance the .dll files (which are actually ELF executables) have
>>> been preserved somewhere?
>>
>> Here is the build folder (~90MB):
>> https://gitlab.com/osteffen/thunderx2-debug/-/raw/main/armvirt-thunderx2-issue.tar.xz
>>
>> I am waiting for the tests with the additional debug output to run.
>
>
> We reran the test suite with the Erratum and the additional debug
> output enabled.  Strangely, the problem does not occur anymore, the
> firmware boots up normally.
>
> We retried the tests without the additional debug output.
> RHEL ships two firmware flavors for AARCH64: a silent and a verbose
> version.

Are these RELEASE vs DEBUG builds?

> Both were tried. We see no problems with the verbose
> one. The silent one fails noticeably more often if a software TPM device
> is present.
>

This smells like some missing cache or TLB maintenance - the verbose
one exits to the host much more often, and likely relies on cache/TLB
maintenance occurring in the hypervisor.

So the build always includes TPM support but the issue only occurs
when the sw TPM is actually exposed by QEMU?

> Could this be related to how much stuff is going on in the early phase
> of the firmware (when logging is enabled: formatting of messages and
> sending to serial port...) ?
>

I'll try to see if I can rig something up that logs into a buffer
rather than straight to the serial, and dump it all out when handling
the crash

Re: [edk2-devel] [PATCH v2 2/2] ArmVirtPkg/ArmVirtQemu: Avoid early ID map on ThunderX

[Oliver Steffen]

[-- Attachment #1: Type: text/plain, Size: 3233 bytes --]

On Tue, Feb 7, 2023 at 12:57 PM Ard Biesheuvel <ardb@kernel.org> wrote:

> On Tue, 7 Feb 2023 at 11:51, Oliver Steffen <osteffen@redhat.com> wrote:
> >
> > On Thu, Feb 2, 2023 at 12:09 PM Oliver Steffen <osteffen@redhat.com>
> wrote:
> >>
> >>
> >> On Wed, Feb 1, 2023 at 2:29 PM Ard Biesheuvel <ardb@kernel.org> wrote:
> >>>
> >>> On Wed, 1 Feb 2023 at 13:59, Oliver Steffen <osteffen@redhat.com>
> wrote:
> >>> >
> >>> > On Wed, Feb 1, 2023 at 12:52 PM Ard Biesheuvel <ardb@kernel.org>
> wrote:
> >>> >>
> >>> >> On Wed, 1 Feb 2023 at 10:14, Oliver Steffen <osteffen@redhat.com>
> wrote:
> >>> >> >
> >>
> >> [...]
> >>>
> >>> >> > I am sorry, this story does not seem to be over yet.
> >>> >> >
> >>> >> > We are using the Erratum patch and also included the commit
> 406504c7 in
> >>> >> > the kernel.
> >>> >> > Now the firmware crashes sometimes (10 out of 89 tests).
> >>> >> >
> >>> >>
> >>> >> Thanks for the report. Is this still on ThunderX2?
> >>> >>
> >>> >> > Any hints are very welcome!
> >>> >> >
> >>> >>
> >>> >> Do  you have access to those build artifacts?
> >>> >
> >>> >
> >>> >
> https://kojihub.stream.centos.org/kojifiles/work/tasks/5251/1835251/edk2-aarch64-20221207gitfff6d81270b5-4.el9.test.noarch.rpm
> >>> >
> >>> > and/or here:
> >>> >
> >>> > https://kojihub.stream.centos.org/koji/taskinfo?taskID=1835251
> >>> >
> >>> > Source for reference:
> >>> > https://gitlab.com/redhat/centos-stream/src/edk2/-/merge_requests/24
> >>> >
> >>>
> >>> Any chance the .dll files (which are actually ELF executables) have
> >>> been preserved somewhere?
> >>
> >> Here is the build folder (~90MB):
> >>
> https://gitlab.com/osteffen/thunderx2-debug/-/raw/main/armvirt-thunderx2-issue.tar.xz
> >>
> >> I am waiting for the tests with the additional debug output to run.
> >
> >
> > We reran the test suite with the Erratum and the additional debug
> > output enabled.  Strangely, the problem does not occur anymore, the
> > firmware boots up normally.
> >
> > We retried the tests without the additional debug output.
> > RHEL ships two firmware flavors for AARCH64: a silent and a verbose
> > version.
>
> Are these RELEASE vs DEBUG builds?
>

All builds are DEBUG, just the amount of information printed on
the serial is different (almost zero for the "silent" one.)


> > Both were tried. We see no problems with the verbose
> > one. The silent one fails noticeably more often if a software TPM device
> > is present.
> >
>
> This smells like some missing cache or TLB maintenance - the verbose
> one exits to the host much more often, and likely relies on cache/TLB
> maintenance occurring in the hypervisor.
>
> So the build always includes TPM support but the issue only occurs
> when the sw TPM is actually exposed by QEMU?
>

Yes.
All builds include support for TPM, but the issue occurs more frequently
if a sw TPM is exposed by QEMU.


> > Could this be related to how much stuff is going on in the early phase
> > of the firmware (when logging is enabled: formatting of messages and
> > sending to serial port...) ?
> >
>
> I'll try to see if I can rig something up that logs into a buffer
> rather than straight to the serial, and dump it all out when handling
> the crash
>
> Awesome.

Thanks,
 Oliver

[-- Attachment #2: Type: text/html, Size: 6233 bytes --]

Re: [edk2-devel] [PATCH v2 2/2] ArmVirtPkg/ArmVirtQemu: Avoid early ID map on ThunderX

[Ard Biesheuvel]

On Tue, 7 Feb 2023 at 13:58, Oliver Steffen <osteffen@redhat.com> wrote:
>
> On Tue, Feb 7, 2023 at 12:57 PM Ard Biesheuvel <ardb@kernel.org> wrote:
>>
>> On Tue, 7 Feb 2023 at 11:51, Oliver Steffen <osteffen@redhat.com> wrote:
>> >
>> > On Thu, Feb 2, 2023 at 12:09 PM Oliver Steffen <osteffen@redhat.com> wrote:
>> >>
>> >>
>> >> On Wed, Feb 1, 2023 at 2:29 PM Ard Biesheuvel <ardb@kernel.org> wrote:
>> >>>
>> >>> On Wed, 1 Feb 2023 at 13:59, Oliver Steffen <osteffen@redhat.com> wrote:
>> >>> >
>> >>> > On Wed, Feb 1, 2023 at 12:52 PM Ard Biesheuvel <ardb@kernel.org> wrote:
>> >>> >>
>> >>> >> On Wed, 1 Feb 2023 at 10:14, Oliver Steffen <osteffen@redhat.com> wrote:
>> >>> >> >
>> >>
>> >> [...]
>> >>>
>> >>> >> > I am sorry, this story does not seem to be over yet.
>> >>> >> >
>> >>> >> > We are using the Erratum patch and also included the commit 406504c7 in
>> >>> >> > the kernel.
>> >>> >> > Now the firmware crashes sometimes (10 out of 89 tests).
>> >>> >> >
>> >>> >>
>> >>> >> Thanks for the report. Is this still on ThunderX2?
>> >>> >>
>> >>> >> > Any hints are very welcome!
>> >>> >> >
>> >>> >>
>> >>> >> Do  you have access to those build artifacts?
>> >>> >
>> >>> >
>> >>> > https://kojihub.stream.centos.org/kojifiles/work/tasks/5251/1835251/edk2-aarch64-20221207gitfff6d81270b5-4.el9.test.noarch.rpm
>> >>> >
>> >>> > and/or here:
>> >>> >
>> >>> > https://kojihub.stream.centos.org/koji/taskinfo?taskID=1835251
>> >>> >
>> >>> > Source for reference:
>> >>> > https://gitlab.com/redhat/centos-stream/src/edk2/-/merge_requests/24
>> >>> >
>> >>>
>> >>> Any chance the .dll files (which are actually ELF executables) have
>> >>> been preserved somewhere?
>> >>
>> >> Here is the build folder (~90MB):
>> >> https://gitlab.com/osteffen/thunderx2-debug/-/raw/main/armvirt-thunderx2-issue.tar.xz
>> >>
>> >> I am waiting for the tests with the additional debug output to run.
>> >
>> >
>> > We reran the test suite with the Erratum and the additional debug
>> > output enabled.  Strangely, the problem does not occur anymore, the
>> > firmware boots up normally.
>> >
>> > We retried the tests without the additional debug output.
>> > RHEL ships two firmware flavors for AARCH64: a silent and a verbose
>> > version.
>>
>> Are these RELEASE vs DEBUG builds?
>
>
> All builds are DEBUG, just the amount of information printed on
> the serial is different (almost zero for the "silent" one.)
>
>>
>> > Both were tried. We see no problems with the verbose
>> > one. The silent one fails noticeably more often if a software TPM device
>> > is present.
>> >
>>
>> This smells like some missing cache or TLB maintenance - the verbose
>> one exits to the host much more often, and likely relies on cache/TLB
>> maintenance occurring in the hypervisor.
>>
>> So the build always includes TPM support but the issue only occurs
>> when the sw TPM is actually exposed by QEMU?
>
>
> Yes.
> All builds include support for TPM, but the issue occurs more frequently
> if a sw TPM is exposed by QEMU.
>

Any chance you could provide a specific command line for launching
QEMU? I am trying to reproduce this, but I am not making any progress.

>>
>> > Could this be related to how much stuff is going on in the early phase
>> > of the firmware (when logging is enabled: formatting of messages and
>> > sending to serial port...) ?
>> >
>>
>> I'll try to see if I can rig something up that logs into a buffer
>> rather than straight to the serial, and dump it all out when handling
>> the crash
>>

This takes a bit more time than I can afford to spend on this atm, and
I'd like to be able to reproduce before I go down this rabbit hole.

Re: [edk2-devel] [PATCH v2 2/2] ArmVirtPkg/ArmVirtQemu: Avoid early ID map on ThunderX

[Ard Biesheuvel]

On Thu, 9 Feb 2023 at 16:15, Ard Biesheuvel <ardb@kernel.org> wrote:
>
> On Tue, 7 Feb 2023 at 13:58, Oliver Steffen <osteffen@redhat.com> wrote:
> >
> > On Tue, Feb 7, 2023 at 12:57 PM Ard Biesheuvel <ardb@kernel.org> wrote:
> >>
> >> On Tue, 7 Feb 2023 at 11:51, Oliver Steffen <osteffen@redhat.com> wrote:
> >> >
> >> > On Thu, Feb 2, 2023 at 12:09 PM Oliver Steffen <osteffen@redhat.com> wrote:
> >> >>
> >> >>
> >> >> On Wed, Feb 1, 2023 at 2:29 PM Ard Biesheuvel <ardb@kernel.org> wrote:
> >> >>>
> >> >>> On Wed, 1 Feb 2023 at 13:59, Oliver Steffen <osteffen@redhat.com> wrote:
> >> >>> >
> >> >>> > On Wed, Feb 1, 2023 at 12:52 PM Ard Biesheuvel <ardb@kernel.org> wrote:
> >> >>> >>
> >> >>> >> On Wed, 1 Feb 2023 at 10:14, Oliver Steffen <osteffen@redhat.com> wrote:
> >> >>> >> >
> >> >>
> >> >> [...]
> >> >>>
> >> >>> >> > I am sorry, this story does not seem to be over yet.
> >> >>> >> >
> >> >>> >> > We are using the Erratum patch and also included the commit 406504c7 in
> >> >>> >> > the kernel.
> >> >>> >> > Now the firmware crashes sometimes (10 out of 89 tests).
> >> >>> >> >
> >> >>> >>
> >> >>> >> Thanks for the report. Is this still on ThunderX2?
> >> >>> >>
> >> >>> >> > Any hints are very welcome!
> >> >>> >> >
> >> >>> >>
> >> >>> >> Do  you have access to those build artifacts?
> >> >>> >
> >> >>> >
> >> >>> > https://kojihub.stream.centos.org/kojifiles/work/tasks/5251/1835251/edk2-aarch64-20221207gitfff6d81270b5-4.el9.test.noarch.rpm
> >> >>> >
> >> >>> > and/or here:
> >> >>> >
> >> >>> > https://kojihub.stream.centos.org/koji/taskinfo?taskID=1835251
> >> >>> >
> >> >>> > Source for reference:
> >> >>> > https://gitlab.com/redhat/centos-stream/src/edk2/-/merge_requests/24
> >> >>> >
> >> >>>
> >> >>> Any chance the .dll files (which are actually ELF executables) have
> >> >>> been preserved somewhere?
> >> >>
> >> >> Here is the build folder (~90MB):
> >> >> https://gitlab.com/osteffen/thunderx2-debug/-/raw/main/armvirt-thunderx2-issue.tar.xz
> >> >>
> >> >> I am waiting for the tests with the additional debug output to run.
> >> >
> >> >
> >> > We reran the test suite with the Erratum and the additional debug
> >> > output enabled.  Strangely, the problem does not occur anymore, the
> >> > firmware boots up normally.
> >> >
> >> > We retried the tests without the additional debug output.
> >> > RHEL ships two firmware flavors for AARCH64: a silent and a verbose
> >> > version.
> >>
> >> Are these RELEASE vs DEBUG builds?
> >
> >
> > All builds are DEBUG, just the amount of information printed on
> > the serial is different (almost zero for the "silent" one.)
> >
> >>
> >> > Both were tried. We see no problems with the verbose
> >> > one. The silent one fails noticeably more often if a software TPM device
> >> > is present.
> >> >
> >>
> >> This smells like some missing cache or TLB maintenance - the verbose
> >> one exits to the host much more often, and likely relies on cache/TLB
> >> maintenance occurring in the hypervisor.
> >>
> >> So the build always includes TPM support but the issue only occurs
> >> when the sw TPM is actually exposed by QEMU?
> >
> >
> > Yes.
> > All builds include support for TPM, but the issue occurs more frequently
> > if a sw TPM is exposed by QEMU.
> >
>
> Any chance you could provide a specific command line for launching
> QEMU? I am trying to reproduce this, but I am not making any progress.
>
> >>
> >> > Could this be related to how much stuff is going on in the early phase
> >> > of the firmware (when logging is enabled: formatting of messages and
> >> > sending to serial port...) ?
> >> >
> >>
> >> I'll try to see if I can rig something up that logs into a buffer
> >> rather than straight to the serial, and dump it all out when handling
> >> the crash
> >>
>
> This takes a bit more time than I can afford to spend on this atm, and
> I'd like to be able to reproduce before I go down this rabbit hole.

Have there been any developments regarding this issue?

Re: [edk2-devel] [PATCH v2 2/2] ArmVirtPkg/ArmVirtQemu: Avoid early ID map on ThunderX

[Oliver Steffen]

[-- Attachment #1: Type: text/plain, Size: 4620 bytes --]

On Thu, Mar 2, 2023 at 11:50 AM Ard Biesheuvel <ardb@kernel.org> wrote:

> On Thu, 9 Feb 2023 at 16:15, Ard Biesheuvel <ardb@kernel.org> wrote:
> >
> > On Tue, 7 Feb 2023 at 13:58, Oliver Steffen <osteffen@redhat.com> wrote:
> > >
> > > On Tue, Feb 7, 2023 at 12:57 PM Ard Biesheuvel <ardb@kernel.org>
> wrote:
> > >>
> > >> On Tue, 7 Feb 2023 at 11:51, Oliver Steffen <osteffen@redhat.com>
> wrote:
> > >> >
> > >> > On Thu, Feb 2, 2023 at 12:09 PM Oliver Steffen <osteffen@redhat.com>
> wrote:
> > >> >>
> > >> >>
> > >> >> On Wed, Feb 1, 2023 at 2:29 PM Ard Biesheuvel <ardb@kernel.org>
> wrote:
> > >> >>>
> > >> >>> On Wed, 1 Feb 2023 at 13:59, Oliver Steffen <osteffen@redhat.com>
> wrote:
> > >> >>> >
> > >> >>> > On Wed, Feb 1, 2023 at 12:52 PM Ard Biesheuvel <ardb@kernel.org>
> wrote:
> > >> >>> >>
> > >> >>> >> On Wed, 1 Feb 2023 at 10:14, Oliver Steffen <
> osteffen@redhat.com> wrote:
> > >> >>> >> >
> > >> >>
> > >> >> [...]
> > >> >>>
> > >> >>> >> > I am sorry, this story does not seem to be over yet.
> > >> >>> >> >
> > >> >>> >> > We are using the Erratum patch and also included the commit
> 406504c7 in
> > >> >>> >> > the kernel.
> > >> >>> >> > Now the firmware crashes sometimes (10 out of 89 tests).
> > >> >>> >> >
> > >> >>> >>
> > >> >>> >> Thanks for the report. Is this still on ThunderX2?
> > >> >>> >>
> > >> >>> >> > Any hints are very welcome!
> > >> >>> >> >
> > >> >>> >>
> > >> >>> >> Do  you have access to those build artifacts?
> > >> >>> >
> > >> >>> >
> > >> >>> >
> https://kojihub.stream.centos.org/kojifiles/work/tasks/5251/1835251/edk2-aarch64-20221207gitfff6d81270b5-4.el9.test.noarch.rpm
> > >> >>> >
> > >> >>> > and/or here:
> > >> >>> >
> > >> >>> > https://kojihub.stream.centos.org/koji/taskinfo?taskID=1835251
> > >> >>> >
> > >> >>> > Source for reference:
> > >> >>> >
> https://gitlab.com/redhat/centos-stream/src/edk2/-/merge_requests/24
> > >> >>> >
> > >> >>>
> > >> >>> Any chance the .dll files (which are actually ELF executables)
> have
> > >> >>> been preserved somewhere?
> > >> >>
> > >> >> Here is the build folder (~90MB):
> > >> >>
> https://gitlab.com/osteffen/thunderx2-debug/-/raw/main/armvirt-thunderx2-issue.tar.xz
> > >> >>
> > >> >> I am waiting for the tests with the additional debug output to run.
> > >> >
> > >> >
> > >> > We reran the test suite with the Erratum and the additional debug
> > >> > output enabled.  Strangely, the problem does not occur anymore, the
> > >> > firmware boots up normally.
> > >> >
> > >> > We retried the tests without the additional debug output.
> > >> > RHEL ships two firmware flavors for AARCH64: a silent and a verbose
> > >> > version.
> > >>
> > >> Are these RELEASE vs DEBUG builds?
> > >
> > >
> > > All builds are DEBUG, just the amount of information printed on
> > > the serial is different (almost zero for the "silent" one.)
> > >
> > >>
> > >> > Both were tried. We see no problems with the verbose
> > >> > one. The silent one fails noticeably more often if a software TPM
> device
> > >> > is present.
> > >> >
> > >>
> > >> This smells like some missing cache or TLB maintenance - the verbose
> > >> one exits to the host much more often, and likely relies on cache/TLB
> > >> maintenance occurring in the hypervisor.
> > >>
> > >> So the build always includes TPM support but the issue only occurs
> > >> when the sw TPM is actually exposed by QEMU?
> > >
> > >
> > > Yes.
> > > All builds include support for TPM, but the issue occurs more
> frequently
> > > if a sw TPM is exposed by QEMU.
> > >
> >
> > Any chance you could provide a specific command line for launching
> > QEMU? I am trying to reproduce this, but I am not making any progress.
> >
> > >>
> > >> > Could this be related to how much stuff is going on in the early
> phase
> > >> > of the firmware (when logging is enabled: formatting of messages and
> > >> > sending to serial port...) ?
> > >> >
> > >>
> > >> I'll try to see if I can rig something up that logs into a buffer
> > >> rather than straight to the serial, and dump it all out when handling
> > >> the crash
> > >>
> >
> > This takes a bit more time than I can afford to spend on this atm, and
> > I'd like to be able to reproduce before I go down this rabbit hole.
>
> Have there been any developments regarding this issue?
>

Nothing from my side.  I tried to come up with a more reliable/faster
reproducer
but then stopped because of other stuff.

If you have any idea what I could try next let me know.

-Oliver

[-- Attachment #2: Type: text/html, Size: 7956 bytes --]

Re: [edk2-devel] [PATCH v2 2/2] ArmVirtPkg/ArmVirtQemu: Avoid early ID map on ThunderX

[Oliver Steffen]

Quoting Oliver Steffen (2023-03-02 14:29:43)
> On Thu, Mar 2, 2023 at 11:50 AM Ard Biesheuvel <[1]ardb@kernel.org> wrote:
>
>     On Thu, 9 Feb 2023 at 16:15, Ard Biesheuvel <[2]ardb@kernel.org> wrote:
>     >
>     > On Tue, 7 Feb 2023 at 13:58, Oliver Steffen <[3]osteffen@redhat.com>
>     wrote:
>     > >
>     > > On Tue, Feb 7, 2023 at 12:57 PM Ard Biesheuvel <[4]ardb@kernel.org>
>     wrote:
>     > >>
>     > >> On Tue, 7 Feb 2023 at 11:51, Oliver Steffen <[5]osteffen@redhat.com>
>     wrote:
>     > >> >
>     > >> > On Thu, Feb 2, 2023 at 12:09 PM Oliver Steffen <[6]
>     osteffen@redhat.com> wrote:
>     > >> >>
>     > >> >>
>     > >> >> On Wed, Feb 1, 2023 at 2:29 PM Ard Biesheuvel <[7]ardb@kernel.org>
>     wrote:
>     > >> >>>
>     > >> >>> On Wed, 1 Feb 2023 at 13:59, Oliver Steffen <[8]
>     osteffen@redhat.com> wrote:
>     > >> >>> >
>     > >> >>> > On Wed, Feb 1, 2023 at 12:52 PM Ard Biesheuvel <[9]
>     ardb@kernel.org> wrote:
>     > >> >>> >>
>     > >> >>> >> On Wed, 1 Feb 2023 at 10:14, Oliver Steffen <[10]
>     osteffen@redhat.com> wrote:
>     > >> >>> >> >
>     > >> >>
>     > >> >> [...]
>     > >> >>>
>     > >> >>> >> > I am sorry, this story does not seem to be over yet.
>     > >> >>> >> >
>     > >> >>> >> > We are using the Erratum patch and also included the commit
>     406504c7 in
>     > >> >>> >> > the kernel.
>     > >> >>> >> > Now the firmware crashes sometimes (10 out of 89 tests).
>     > >> >>> >> >
>     > >> >>> >>
>     > >> >>> >> Thanks for the report. Is this still on ThunderX2?
>     > >> >>> >>
>     > >> >>> >> > Any hints are very welcome!
>     > >> >>> >> >
>     > >> >>> >>
>     > >> >>> >> Do  you have access to those build artifacts?
>     > >> >>> >
>     > >> >>> >
>     > >> >>> > [11]https://kojihub.stream.centos.org/kojifiles/work/tasks/5251/
>     1835251/edk2-aarch64-20221207gitfff6d81270b5-4.el9.test.noarch.rpm
>     > >> >>> >
>     > >> >>> > and/or here:
>     > >> >>> >
>     > >> >>> > [12]https://kojihub.stream.centos.org/koji/taskinfo?taskID=
>     1835251
>     > >> >>> >
>     > >> >>> > Source for reference:
>     > >> >>> > [13]https://gitlab.com/redhat/centos-stream/src/edk2/-/
>     merge_requests/24
>     > >> >>> >
>     > >> >>>
>     > >> >>> Any chance the .dll files (which are actually ELF executables)
>     have
>     > >> >>> been preserved somewhere?
>     > >> >>
>     > >> >> Here is the build folder (~90MB):
>     > >> >> [14]https://gitlab.com/osteffen/thunderx2-debug/-/raw/main/
>     armvirt-thunderx2-issue.tar.xz
>     > >> >>
>     > >> >> I am waiting for the tests with the additional debug output to run.
>     > >> >
>     > >> >
>     > >> > We reran the test suite with the Erratum and the additional debug
>     > >> > output enabled.  Strangely, the problem does not occur anymore, the
>     > >> > firmware boots up normally.
>     > >> >
>     > >> > We retried the tests without the additional debug output.
>     > >> > RHEL ships two firmware flavors for AARCH64: a silent and a verbose
>     > >> > version.
>     > >>
>     > >> Are these RELEASE vs DEBUG builds?
>     > >
>     > >
>     > > All builds are DEBUG, just the amount of information printed on
>     > > the serial is different (almost zero for the "silent" one.)
>     > >
>     > >>
>     > >> > Both were tried. We see no problems with the verbose
>     > >> > one. The silent one fails noticeably more often if a software TPM
>     device
>     > >> > is present.
>     > >> >
>     > >>
>     > >> This smells like some missing cache or TLB maintenance - the verbose
>     > >> one exits to the host much more often, and likely relies on cache/TLB
>     > >> maintenance occurring in the hypervisor.
>     > >>
>     > >> So the build always includes TPM support but the issue only occurs
>     > >> when the sw TPM is actually exposed by QEMU?
>     > >
>     > >
>     > > Yes.
>     > > All builds include support for TPM, but the issue occurs more
>     frequently
>     > > if a sw TPM is exposed by QEMU.
>     > >
>     >
>     > Any chance you could provide a specific command line for launching
>     > QEMU? I am trying to reproduce this, but I am not making any progress.
>     >
>     > >>
>     > >> > Could this be related to how much stuff is going on in the early
>     phase
>     > >> > of the firmware (when logging is enabled: formatting of messages and
>     > >> > sending to serial port...) ?
>     > >> >
>     > >>
>     > >> I'll try to see if I can rig something up that logs into a buffer
>     > >> rather than straight to the serial, and dump it all out when handling
>     > >> the crash
>     > >>
>     >
>     > This takes a bit more time than I can afford to spend on this atm, and
>     > I'd like to be able to reproduce before I go down this rabbit hole.
>
>     Have there been any developments regarding this issue?
>
>
> Nothing from my side.  I tried to come up with a more reliable/faster
> reproducer
> but then stopped because of other stuff.
>
> If you have any idea what I could try next let me know.
>
> -Oliver
# Summary for Email 2

Hi all,

I had another look at this and I can now reproduce the issue consistently,
with a quite minimal setup, on recent Linux kernel, Qemu, and EDK2.
It requires rebooting the guest in a tight loop. It happens in silent
and verbose
builds alike, but since the verbose ones are slowed down by the serial
output, it
takes longer to hit the issue.
It is possible to reproduce it with the silent builds within a few minutes.
For the verbose case I recommend running multiple Qemu instances in parallel (as
many as the machine allows, in my case ~100).

Details:

CPU: Cavium ThunderX2(R) CPU CN9975
Tested on 3 different machines:
    HPE apache, HPE apollo, Gigabyte R181
Kernels tested:
 - 6.2.15-100.fc36.aarch64
 - 5.14.0-312.el9.aarch64
   (contains 406504c7b0405d74d74c15a667cd4c4620c3e7a9,
   "KVM: arm64: Fix S1PTW handling on RO memslots")
Qemu v8.0.0 (RHEL version and build from upstream repo)
EDK2: master branch from 2023-05-16 (cafb4f3f)
gcc 11.3.1

EDK2 build command line:
build \
  -a AARCH64
  -p ArmVirtPkg/ArmVirtQemu.dsc
  -t GCC5 -b DEBUG \
  -D NETWORK_IP6_ENABLE \
  -D NETWORK_HTTP_BOOT_ENABLE \
  -D NETWORK_TLS_ENABLE \
  -D NETWORK_ISCSI_ENABLE \
  -D NETWORK_ALLOW_HTTP_CONNECTIONS \
  -D CAVIUM_ERRATUM_27456=TRUE \
  -D TPM2_ENABLE=TRUE \
  -D TPM1_ENABLE=FALSE \
  -D DEBUG_PRINT_ERROR_LEVEL=0x80000000  \
  -D BUILD_SHELL=TRUE \
  --pcd="gEfiShellPkgTokenSpaceGuid.PcdShellDefaultDelay=0" \
  --pcd="gEfiMdePkgTokenSpaceGuid.PcdPlatformBootTimeOut=0" \
  --hash --cmd-len=65536

To reproduce the issue I launched the firmware in Qemu and have it do
a reboot once it finished booting up
via a startup.nsh on the ESP.

Qemu command line:
qemu-system-aarch64 \
    -machine virt,accel=kvm -m 13G \
    -boot menu=off \
    -cpu host \
    -blockdev node-name=code,driver=file,filename="${FW_CODE}",read-only=on \
    -blockdev node-name=vars,driver=file,filename="${FW_VARS}" \
    -machine pflash0=code \
    -machine pflash1=vars \
    -serial stdio \
    -net none \
    -drive file=esp.img,snapshot=on

Other things like number of CPUs or the presence of a vTPM have no
influence. I did not try different amounts of RAM yet.

Serial output:
[...]
InitializeDxeNxMemoryProtectionPolicy: StackBase = 0x00000000476C5000
StackSize = 0x0000000000020000
InitializeDxeNxMemoryProtectionPolicy: applying strict permissions to
active memory regions
SetUefiImageMemoryAttributes - 0x0000000040000000 - 0x00000000076E5000
(0x0000000000004000)
UpdateRegionMappingRecursive(0): 40000000 - 476E5000 set
60000000000400 clr FF9F000000000B3F
UpdateRegionMappingRecursive(1): 40000000 - 476E5000 set
60000000000400 clr FF9F000000000B3F
UpdateRegionMappingRecursive(2): 40000000 - 476E5000 set
60000000000400 clr FF9F000000000B3F
UpdateRegionMappingRecursive(3): 47600000 - 476E5000 set
60000000000400 clr FF9F000000000B3F
SetUefiImageMemoryAttributes - 0x00000000476C5000 - 0x0000000000001000
(0x0000000000006000)
UpdateRegionMappingRecursive(0): 476C5000 - 476C6000 set
60000000000000 clr FF9F000000000B3F
UpdateRegionMappingRecursive(1): 476C5000 - 476C6000 set
60000000000000 clr FF9F000000000B3F
UpdateRegionMappingRecursive(2): 476C5000 - 476C6000 set
60000000000000 clr FF9F000000000B3F
UpdateRegionMappingRecursive(3): 476C5000 - 476C6000 set
60000000000000 clr FF9F000000000B3F
SetUefiImageMemoryAttributes - 0x000000004772B000 - 0x00000000007C0000
(0x0000000000004000)
UpdateRegionMappingRecursive(0): 4772B000 - 47EEB000 set
60000000000400 clr FF9F000000000B3F
UpdateRegionMappingRecursive(1): 4772B000 - 47EEB000 set
60000000000400 clr FF9F000000000B3F
UpdateRegionMappingRecursive(2): 4772B000 - 47EEB000 set
60000000000400 clr FF9F000000000B3F
UpdateRegionMappingRecursive(3): 4772B000 - 47800000 set
60000000000400 clr FF9F000000000B3F
UpdateRegionMappingRecursive(3): 47E00000 - 47EEB000 set
60000000000400 clr FF9F000000000B3F
SetUefiImageMemoryAttributes - 0x0000000047EF3000 - 0x0000000000101000
(0x0000000000004000)
UpdateRegionMappingRecursive(0): 47EF3000 - 47FF4000 set
60000000000400 clr FF9F000000000B3F
UpdateRegionMappingRecursive(1): 47EF3000 - 47FF4000 set
60000000000400 clr FF9F000000000B3F
UpdateRegionMappingRecursive(2): 47EF3000 - 47FF4000 set
60000000000400 clr FF9F000000000B3F
UpdateRegionMappingRecursive(3): 47EF3000 - 47FF4000 set
60000000000400 clr FF9F000000000B3F
SetUefiImageMemoryAttributes - 0x0000000047FFA000 - 0x0000000334AA6000
(0x0000000000004000)
UpdateRegionMappingRecursive(0): 47FFA000 - 37CAA0000 set
60000000000400 clr FF9F000000000B3F
UpdateRegionMappingRecursive(1): 47FFA000 - 37CAA0000 set
60000000000400 clr FF9F000000000B3F
UpdateRegionMappingRecursive(2): 47FFA000 - 80000000 set
60000000000400 clr FF9F000000000B3F
UpdateRegionMappingRecursive(3): 47FFA000 - 48000000 set
60000000000400 clr FF9F000000000B3F
UpdateRegionMappingRecursive(2): 340000000 - 380000000 set 70C clr 0
UpdateRegionMappingRecursive(3): 37F000000 - 37F200000 set 70C clr 0
UpdateRegionMappingRecursive(2): 340000000 - 37CAA0000 set
60000000000400 clr FF9F000000000B3F
UpdateRegionMappingRecursive(3): 37CA00000 - 37CC00000 set 70C clr 0
UpdateRegionMappingRecursive(3): 37CA00000 - 37CAA0000 set
60000000000400 clr FF9F000000000B3F
SetUefiImageMemoryAttributes - 0x000000037CB40000 - 0x00000000031F9000
(0x0000000000004000)
UpdateRegionMappingRecursive(0): 37CB40000 - 37FD39000 set
60000000000400 clr FF9F000000000B3F
UpdateRegionMappingRecursive(1): 37CB40000 - 37FD39000 set
60000000000400 clr FF9F000000000B3F
UpdateRegionMappingRecursive(2): 37CB40000 - 37FD39000 set
60000000000400 clr FF9F000000000B3F
UpdateRegionMappingRecursive(3): 37CB40000 - 37CC00000 set
60000000000400 clr FF9F000000000B3F
UpdateRegionMappingRecursive(3): 37F000000 - 37F200000 set
60000000000400 clr FF9F000000000B3F
UpdateRegionMappingRecursive(3): 37FC00000 - 37FE00000 set 70C clr 0
UpdateRegionMappingRecursive(3): 37FC00000 - 37FD39000 set
60000000000400 clr FF9F000000000B3F


Synchronous Exception at 0x000000037FD3C0A8
PC 0x00037FD3C0A8 (0x00037FD39000+0x000030A8) [ 0] ArmCpuDxe.dll
PC 0x00037FD3C0A8 (0x00037FD39000+0x000030A8) [ 0] ArmCpuDxe.dll
PC 0x00037FD3BE70 (0x00037FD39000+0x00002E70) [ 0] ArmCpuDxe.dll
PC 0x00037FD3BE70 (0x00037FD39000+0x00002E70) [ 0] ArmCpuDxe.dll
PC 0x00037FD3C2E4 (0x00037FD39000+0x000032E4) [ 0] ArmCpuDxe.dll
PC 0x0000476E78F8 (0x0000476E5000+0x000028F8) [ 1] DxeCore.dll
PC 0x0000476ED680 (0x0000476E5000+0x00008680) [ 1] DxeCore.dll
PC 0x0000476F2744 (0x0000476E5000+0x0000D744) [ 1] DxeCore.dll
PC 0x0000476ECDE8 (0x0000476E5000+0x00007DE8) [ 1] DxeCore.dll
PC 0x00037FD3D2DC (0x00037FD39000+0x000042DC) [ 2] ArmCpuDxe.dll
PC 0x0000476EC788 (0x0000476E5000+0x00007788) [ 3] DxeCore.dll
PC 0x0000476F9CA8 (0x0000476E5000+0x00014CA8) [ 3] DxeCore.dll
PC 0x0000476EFEF0 (0x0000476E5000+0x0000AEF0) [ 3] DxeCore.dll

[ 0] /root/edk2/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/ArmPkg/Drivers/CpuDxe/CpuDxe/DEBUG/ArmCpuDxe.dll
[ 1] /root/edk2/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/MdeModulePkg/Core/Dxe/DxeMain/DEBUG/DxeCore.dll
[ 2] /root/edk2/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/ArmPkg/Drivers/CpuDxe/CpuDxe/DEBUG/ArmCpuDxe.dll
[ 3] /root/edk2/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/MdeModulePkg/Core/Dxe/DxeMain/DEBUG/DxeCore.dll

  X0 0x000000037F10BFF0   X1 0x000000037F106003   X2
0x000000000037FC00   X3 0x0000000000000000
  X4 0x0000000000000200   X5 0x0000000000000004   X6
0x0000000000000000   X7 0x000000037FD3F4B5
  X8 0x0000000000000000   X9 0x0000000000000002  X10
0x0000000000000000  X11 0x0000000000000000
 X12 0x0000000000000002  X13 0x0000000000000002  X14
0x0000000000000001  X15 0x0000000000000002
 X16 0x000000037FD3A268  X17 0x00000000007AFA10  X18
0x0000000000000000  X19 0x000000037FC00000
 X20 0x0000000000000002  X21 0x000000037F106003  X22
0x000000037F10B000  X23 0x000000037FD42000
 X24 0x00000000001FFFFF  X25 0x000000037FD39000  X26
0x000000037F106000  X27 0x0000000000000003
 X28 0x000000037F10BFF0   FP 0x00000000476E4780   LR 0x000000037FD3C0A8

  V0 0x0000000000000000 0000000000000000   V1 0x0000000000000000
0000000000000000
  V2 0x0000000000000000 0000000000000000   V3 0x0000000000000000
0000000000000000
  V4 0x0000000000000000 0000000000000000   V5 0x0000000000000000
0000000000000000
  V6 0x0000000000000000 0000000000000000   V7 0x0000000000000000
0000000000000000
  V8 0x0000000000000000 0000000000000000   V9 0x0000000000000000
0000000000000000
 V10 0x0000000000000000 0000000000000000  V11 0x0000000000000000
0000000000000000
 V12 0x0000000000000000 0000000000000000  V13 0x0000000000000000
0000000000000000
 V14 0x0000000000000000 0000000000000000  V15 0x0000000000000000
0000000000000000
 V16 0x0000000000000000 0000000000000000  V17 0x0000000000000000
0000000000000000
 V18 0x0000000000000000 0000000000000000  V19 0x0000000000000000
0000000000000000
 V20 0x0000000000000000 0000000000000000  V21 0x0000000000000000
0000000000000000
 V22 0x0000000000000000 0000000000000000  V23 0x0000000000000000
0000000000000000
 V24 0x0000000000000000 0000000000000000  V25 0x0000000000000000
0000000000000000
 V26 0x0000000000000000 0000000000000000  V27 0x0000000000000000
0000000000000000
 V28 0x0000000000000000 0000000000000000  V29 0x0000000000000000
0000000000000000
 V30 0x0000000000000000 0000000000000000  V31 0x0000000000000000
0000000000000000

  SP 0x00000000476E4780  ELR 0x000000037FD3C0A8  SPSR 0x80000205  FPSR
0x00000000
 ESR 0x86000006          FAR 0x000000037FD3C0A8

 ESR : EC 0x21  IL 0x1  ISS 0x00000006

Instruction abort: Translation fault, second level

Stack dump:
  00000476E4680: 0000000000000001 0000000000000004 00000000476E4700
00000000476F3980
  00000476E46A0: 000000037FD40CBD 0000000000000003 000000037FC00000
000000037FD39000
  00000476E46C0: 0060000000000400 FF9F000000000B3F 00000000476E4780
000000037FD3BE70
  00000476E46E0: 000000037FC00000 0000000000000002 000000037F106000
000000037F10B000
  00000476E4700: 0000000000000FF0 00000000001FFFFF 000000037FD39000
000000037F106000
  00000476E4720: 0000000000000003 000000037F10BFF0 0060000000000400
FF9F000000000B3F
  00000476E4740: 000000037FD39000 000000037FD39000 00000000476E4780
0060000000000403
  00000476E4760: 0000000C00000001 000000037FD3F90E 0000000000000400
000000037F10B000
> 00000476E4780: 00000000476E4830 000000037FD3BE70 000000037CB40000 0000000000000001
  00000476E47A0: 000000037F10B000 0000000047FFE000 0000000000000068
000000003FFFFFFF
  00000476E47C0: 000000037FD39000 000000037F10C528 0000000000000002
0000000047FFE068
  00000476E47E0: 0060000000000400 FF9F000000000B3F 0000000300000001
000000037FD39000
  00000476E4800: 000000017FD40CBD 0060000000000401 0000001500000001
000000037FD3F90E
  00000476E4820: 0060000000000400 000000037F106000 00000000476E48E0
000000037FD3BE70
  00000476E4840: 000000037CB40000 0000000000000000 0000000047FFE000
0000000047FFF000
  00000476E4860: 0000000000000000 0000007FFFFFFFFF 000000037FD39000
000000037F10C528
ASSERT [ArmCpuDxe]
/root/edk2/ArmPkg/Library/DefaultExceptionHandlerLib/AArch64/DefaultExceptionHandler.c(333):
((BOOLEAN)(0==1))



The full log is available here:
https://gitlab.com/osteffen/thunderx2-debug/-/raw/main/2023-05-19/85.log?inline=false

Debug files, firmware binaries, and the full build tree are here:
https://gitlab.com/osteffen/thunderx2-debug/-/tree/main/2023-05-19

I am able to reproduce this quickly, so any ideas for what I can try
are welcome :-)

Thanks
-Oliver

Re: [edk2-devel] [PATCH v2 2/2] ArmVirtPkg/ArmVirtQemu: Avoid early ID map on ThunderX

[Ard Biesheuvel]

On Fri, 19 May 2023 at 18:32, Oliver Steffen <osteffen@redhat.com> wrote:
>
>
> Hi all,
>
> I had another look at this and I can now reproduce the issue consistently,
> with a quite minimal setup, on recent Linux kernel, Qemu, and EDK2.
> It requires rebooting the guest in a tight loop. It happens in silent
> and verbose
> builds alike, but since the verbose ones are slowed down by the serial
> output, it
> takes longer to hit the issue.
> It is possible to reproduce it with the silent builds within a few minutes.
> For the verbose case I recommend running multiple Qemu instances in parallel (as
> many as the machine allows, in my case ~100).
>

Thanks a lot for all these details, this is extremely helpful.

So what appears to be happening is that we split the 2M block mapping
that covers the code that we were called from, and hit a level 2
translation fault because the updated page table entry is still
observed to be in its transient 'invalid' state as we return to it.

Could you please check whether this makes a difference?

--- a/ArmPkg/Library/ArmMmuLib/AArch64/ArmMmuLibReplaceEntry.S
+++ b/ArmPkg/Library/ArmMmuLib/AArch64/ArmMmuLibReplaceEntry.S
@@ -65,6 +65,7 @@
   // write updated entry
   str   x1, [x0]
   dsb   nshst
+  isb

 .L2_\@:
   .endm



> Details:
>
> CPU: Cavium ThunderX2(R) CPU CN9975
> Tested on 3 different machines:
>     HPE apache, HPE apollo, Gigabyte R181
> Kernels tested:
>  - 6.2.15-100.fc36.aarch64
>  - 5.14.0-312.el9.aarch64
>    (contains 406504c7b0405d74d74c15a667cd4c4620c3e7a9,
>    "KVM: arm64: Fix S1PTW handling on RO memslots")
> Qemu v8.0.0 (RHEL version and build from upstream repo)
> EDK2: master branch from 2023-05-16 (cafb4f3f)
> gcc 11.3.1
>
> EDK2 build command line:
> build \
>   -a AARCH64
>   -p ArmVirtPkg/ArmVirtQemu.dsc
>   -t GCC5 -b DEBUG \
>   -D NETWORK_IP6_ENABLE \
>   -D NETWORK_HTTP_BOOT_ENABLE \
>   -D NETWORK_TLS_ENABLE \
>   -D NETWORK_ISCSI_ENABLE \
>   -D NETWORK_ALLOW_HTTP_CONNECTIONS \
>   -D CAVIUM_ERRATUM_27456=TRUE \
>   -D TPM2_ENABLE=TRUE \
>   -D TPM1_ENABLE=FALSE \
>   -D DEBUG_PRINT_ERROR_LEVEL=0x80000000  \
>   -D BUILD_SHELL=TRUE \
>   --pcd="gEfiShellPkgTokenSpaceGuid.PcdShellDefaultDelay=0" \
>   --pcd="gEfiMdePkgTokenSpaceGuid.PcdPlatformBootTimeOut=0" \
>   --hash --cmd-len=65536
>
> To reproduce the issue I launched the firmware in Qemu and have it do
> a reboot once it finished booting up
> via a startup.nsh on the ESP.
>
> Qemu command line:
> qemu-system-aarch64 \
>     -machine virt,accel=kvm -m 13G \
>     -boot menu=off \
>     -cpu host \
>     -blockdev node-name=code,driver=file,filename="${FW_CODE}",read-only=on \
>     -blockdev node-name=vars,driver=file,filename="${FW_VARS}" \
>     -machine pflash0=code \
>     -machine pflash1=vars \
>     -serial stdio \
>     -net none \
>     -drive file=esp.img,snapshot=on
>
> Other things like number of CPUs or the presence of a vTPM have no
> influence. I did not try different amounts of RAM yet.
>
> Serial output:
> [...]
> InitializeDxeNxMemoryProtectionPolicy: StackBase = 0x00000000476C5000
> StackSize = 0x0000000000020000
> InitializeDxeNxMemoryProtectionPolicy: applying strict permissions to
> active memory regions
> SetUefiImageMemoryAttributes - 0x0000000040000000 - 0x00000000076E5000
> (0x0000000000004000)
> UpdateRegionMappingRecursive(0): 40000000 - 476E5000 set
> 60000000000400 clr FF9F000000000B3F
> UpdateRegionMappingRecursive(1): 40000000 - 476E5000 set
> 60000000000400 clr FF9F000000000B3F
> UpdateRegionMappingRecursive(2): 40000000 - 476E5000 set
> 60000000000400 clr FF9F000000000B3F
> UpdateRegionMappingRecursive(3): 47600000 - 476E5000 set
> 60000000000400 clr FF9F000000000B3F
> SetUefiImageMemoryAttributes - 0x00000000476C5000 - 0x0000000000001000
> (0x0000000000006000)
> UpdateRegionMappingRecursive(0): 476C5000 - 476C6000 set
> 60000000000000 clr FF9F000000000B3F
> UpdateRegionMappingRecursive(1): 476C5000 - 476C6000 set
> 60000000000000 clr FF9F000000000B3F
> UpdateRegionMappingRecursive(2): 476C5000 - 476C6000 set
> 60000000000000 clr FF9F000000000B3F
> UpdateRegionMappingRecursive(3): 476C5000 - 476C6000 set
> 60000000000000 clr FF9F000000000B3F
> SetUefiImageMemoryAttributes - 0x000000004772B000 - 0x00000000007C0000
> (0x0000000000004000)
> UpdateRegionMappingRecursive(0): 4772B000 - 47EEB000 set
> 60000000000400 clr FF9F000000000B3F
> UpdateRegionMappingRecursive(1): 4772B000 - 47EEB000 set
> 60000000000400 clr FF9F000000000B3F
> UpdateRegionMappingRecursive(2): 4772B000 - 47EEB000 set
> 60000000000400 clr FF9F000000000B3F
> UpdateRegionMappingRecursive(3): 4772B000 - 47800000 set
> 60000000000400 clr FF9F000000000B3F
> UpdateRegionMappingRecursive(3): 47E00000 - 47EEB000 set
> 60000000000400 clr FF9F000000000B3F
> SetUefiImageMemoryAttributes - 0x0000000047EF3000 - 0x0000000000101000
> (0x0000000000004000)
> UpdateRegionMappingRecursive(0): 47EF3000 - 47FF4000 set
> 60000000000400 clr FF9F000000000B3F
> UpdateRegionMappingRecursive(1): 47EF3000 - 47FF4000 set
> 60000000000400 clr FF9F000000000B3F
> UpdateRegionMappingRecursive(2): 47EF3000 - 47FF4000 set
> 60000000000400 clr FF9F000000000B3F
> UpdateRegionMappingRecursive(3): 47EF3000 - 47FF4000 set
> 60000000000400 clr FF9F000000000B3F
> SetUefiImageMemoryAttributes - 0x0000000047FFA000 - 0x0000000334AA6000
> (0x0000000000004000)
> UpdateRegionMappingRecursive(0): 47FFA000 - 37CAA0000 set
> 60000000000400 clr FF9F000000000B3F
> UpdateRegionMappingRecursive(1): 47FFA000 - 37CAA0000 set
> 60000000000400 clr FF9F000000000B3F
> UpdateRegionMappingRecursive(2): 47FFA000 - 80000000 set
> 60000000000400 clr FF9F000000000B3F
> UpdateRegionMappingRecursive(3): 47FFA000 - 48000000 set
> 60000000000400 clr FF9F000000000B3F
> UpdateRegionMappingRecursive(2): 340000000 - 380000000 set 70C clr 0
> UpdateRegionMappingRecursive(3): 37F000000 - 37F200000 set 70C clr 0
> UpdateRegionMappingRecursive(2): 340000000 - 37CAA0000 set
> 60000000000400 clr FF9F000000000B3F
> UpdateRegionMappingRecursive(3): 37CA00000 - 37CC00000 set 70C clr 0
> UpdateRegionMappingRecursive(3): 37CA00000 - 37CAA0000 set
> 60000000000400 clr FF9F000000000B3F
> SetUefiImageMemoryAttributes - 0x000000037CB40000 - 0x00000000031F9000
> (0x0000000000004000)
> UpdateRegionMappingRecursive(0): 37CB40000 - 37FD39000 set
> 60000000000400 clr FF9F000000000B3F
> UpdateRegionMappingRecursive(1): 37CB40000 - 37FD39000 set
> 60000000000400 clr FF9F000000000B3F
> UpdateRegionMappingRecursive(2): 37CB40000 - 37FD39000 set
> 60000000000400 clr FF9F000000000B3F
> UpdateRegionMappingRecursive(3): 37CB40000 - 37CC00000 set
> 60000000000400 clr FF9F000000000B3F
> UpdateRegionMappingRecursive(3): 37F000000 - 37F200000 set
> 60000000000400 clr FF9F000000000B3F
> UpdateRegionMappingRecursive(3): 37FC00000 - 37FE00000 set 70C clr 0
> UpdateRegionMappingRecursive(3): 37FC00000 - 37FD39000 set
> 60000000000400 clr FF9F000000000B3F
>
>
> Synchronous Exception at 0x000000037FD3C0A8
> PC 0x00037FD3C0A8 (0x00037FD39000+0x000030A8) [ 0] ArmCpuDxe.dll
> PC 0x00037FD3C0A8 (0x00037FD39000+0x000030A8) [ 0] ArmCpuDxe.dll
> PC 0x00037FD3BE70 (0x00037FD39000+0x00002E70) [ 0] ArmCpuDxe.dll
> PC 0x00037FD3BE70 (0x00037FD39000+0x00002E70) [ 0] ArmCpuDxe.dll
> PC 0x00037FD3C2E4 (0x00037FD39000+0x000032E4) [ 0] ArmCpuDxe.dll
> PC 0x0000476E78F8 (0x0000476E5000+0x000028F8) [ 1] DxeCore.dll
> PC 0x0000476ED680 (0x0000476E5000+0x00008680) [ 1] DxeCore.dll
> PC 0x0000476F2744 (0x0000476E5000+0x0000D744) [ 1] DxeCore.dll
> PC 0x0000476ECDE8 (0x0000476E5000+0x00007DE8) [ 1] DxeCore.dll
> PC 0x00037FD3D2DC (0x00037FD39000+0x000042DC) [ 2] ArmCpuDxe.dll
> PC 0x0000476EC788 (0x0000476E5000+0x00007788) [ 3] DxeCore.dll
> PC 0x0000476F9CA8 (0x0000476E5000+0x00014CA8) [ 3] DxeCore.dll
> PC 0x0000476EFEF0 (0x0000476E5000+0x0000AEF0) [ 3] DxeCore.dll
>
> [ 0] /root/edk2/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/ArmPkg/Drivers/CpuDxe/CpuDxe/DEBUG/ArmCpuDxe.dll
> [ 1] /root/edk2/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/MdeModulePkg/Core/Dxe/DxeMain/DEBUG/DxeCore.dll
> [ 2] /root/edk2/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/ArmPkg/Drivers/CpuDxe/CpuDxe/DEBUG/ArmCpuDxe.dll
> [ 3] /root/edk2/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/MdeModulePkg/Core/Dxe/DxeMain/DEBUG/DxeCore.dll
>
>   X0 0x000000037F10BFF0   X1 0x000000037F106003   X2
> 0x000000000037FC00   X3 0x0000000000000000
>   X4 0x0000000000000200   X5 0x0000000000000004   X6
> 0x0000000000000000   X7 0x000000037FD3F4B5
>   X8 0x0000000000000000   X9 0x0000000000000002  X10
> 0x0000000000000000  X11 0x0000000000000000
>  X12 0x0000000000000002  X13 0x0000000000000002  X14
> 0x0000000000000001  X15 0x0000000000000002
>  X16 0x000000037FD3A268  X17 0x00000000007AFA10  X18
> 0x0000000000000000  X19 0x000000037FC00000
>  X20 0x0000000000000002  X21 0x000000037F106003  X22
> 0x000000037F10B000  X23 0x000000037FD42000
>  X24 0x00000000001FFFFF  X25 0x000000037FD39000  X26
> 0x000000037F106000  X27 0x0000000000000003
>  X28 0x000000037F10BFF0   FP 0x00000000476E4780   LR 0x000000037FD3C0A8
>
>   V0 0x0000000000000000 0000000000000000   V1 0x0000000000000000
> 0000000000000000
>   V2 0x0000000000000000 0000000000000000   V3 0x0000000000000000
> 0000000000000000
>   V4 0x0000000000000000 0000000000000000   V5 0x0000000000000000
> 0000000000000000
>   V6 0x0000000000000000 0000000000000000   V7 0x0000000000000000
> 0000000000000000
>   V8 0x0000000000000000 0000000000000000   V9 0x0000000000000000
> 0000000000000000
>  V10 0x0000000000000000 0000000000000000  V11 0x0000000000000000
> 0000000000000000
>  V12 0x0000000000000000 0000000000000000  V13 0x0000000000000000
> 0000000000000000
>  V14 0x0000000000000000 0000000000000000  V15 0x0000000000000000
> 0000000000000000
>  V16 0x0000000000000000 0000000000000000  V17 0x0000000000000000
> 0000000000000000
>  V18 0x0000000000000000 0000000000000000  V19 0x0000000000000000
> 0000000000000000
>  V20 0x0000000000000000 0000000000000000  V21 0x0000000000000000
> 0000000000000000
>  V22 0x0000000000000000 0000000000000000  V23 0x0000000000000000
> 0000000000000000
>  V24 0x0000000000000000 0000000000000000  V25 0x0000000000000000
> 0000000000000000
>  V26 0x0000000000000000 0000000000000000  V27 0x0000000000000000
> 0000000000000000
>  V28 0x0000000000000000 0000000000000000  V29 0x0000000000000000
> 0000000000000000
>  V30 0x0000000000000000 0000000000000000  V31 0x0000000000000000
> 0000000000000000
>
>   SP 0x00000000476E4780  ELR 0x000000037FD3C0A8  SPSR 0x80000205  FPSR
> 0x00000000
>  ESR 0x86000006          FAR 0x000000037FD3C0A8
>
>  ESR : EC 0x21  IL 0x1  ISS 0x00000006
>
> Instruction abort: Translation fault, second level
>
> Stack dump:
>   00000476E4680: 0000000000000001 0000000000000004 00000000476E4700
> 00000000476F3980
>   00000476E46A0: 000000037FD40CBD 0000000000000003 000000037FC00000
> 000000037FD39000
>   00000476E46C0: 0060000000000400 FF9F000000000B3F 00000000476E4780
> 000000037FD3BE70
>   00000476E46E0: 000000037FC00000 0000000000000002 000000037F106000
> 000000037F10B000
>   00000476E4700: 0000000000000FF0 00000000001FFFFF 000000037FD39000
> 000000037F106000
>   00000476E4720: 0000000000000003 000000037F10BFF0 0060000000000400
> FF9F000000000B3F
>   00000476E4740: 000000037FD39000 000000037FD39000 00000000476E4780
> 0060000000000403
>   00000476E4760: 0000000C00000001 000000037FD3F90E 0000000000000400
> 000000037F10B000
> > 00000476E4780: 00000000476E4830 000000037FD3BE70 000000037CB40000 0000000000000001
>   00000476E47A0: 000000037F10B000 0000000047FFE000 0000000000000068
> 000000003FFFFFFF
>   00000476E47C0: 000000037FD39000 000000037F10C528 0000000000000002
> 0000000047FFE068
>   00000476E47E0: 0060000000000400 FF9F000000000B3F 0000000300000001
> 000000037FD39000
>   00000476E4800: 000000017FD40CBD 0060000000000401 0000001500000001
> 000000037FD3F90E
>   00000476E4820: 0060000000000400 000000037F106000 00000000476E48E0
> 000000037FD3BE70
>   00000476E4840: 000000037CB40000 0000000000000000 0000000047FFE000
> 0000000047FFF000
>   00000476E4860: 0000000000000000 0000007FFFFFFFFF 000000037FD39000
> 000000037F10C528
> ASSERT [ArmCpuDxe]
> /root/edk2/ArmPkg/Library/DefaultExceptionHandlerLib/AArch64/DefaultExceptionHandler.c(333):
> ((BOOLEAN)(0==1))
>
>
>
> The full log is available here:
> https://gitlab.com/osteffen/thunderx2-debug/-/raw/main/2023-05-19/85.log?inline=false
>
> Debug files, firmware binaries, and the full build tree are here:
> https://gitlab.com/osteffen/thunderx2-debug/-/tree/main/2023-05-19
>
> I am able to reproduce this quickly, so any ideas for what I can try
> are welcome :-)
>
> Thanks
> -Oliver
>

Re: [edk2-devel] [PATCH v2 2/2] ArmVirtPkg/ArmVirtQemu: Avoid early ID map on ThunderX

[Oliver Steffen]

Quoting Ard Biesheuvel (2023-05-19 23:36:53)
> On Fri, 19 May 2023 at 18:32, Oliver Steffen <osteffen@redhat.com> wrote:
> >
> >
> > Hi all,
> >
> > I had another look at this and I can now reproduce the issue consistently,
> > with a quite minimal setup, on recent Linux kernel, Qemu, and EDK2.
> > It requires rebooting the guest in a tight loop. It happens in silent
> > and verbose
> > builds alike, but since the verbose ones are slowed down by the serial
> > output, it
> > takes longer to hit the issue.
> > It is possible to reproduce it with the silent builds within a few minutes.
> > For the verbose case I recommend running multiple Qemu instances in parallel (as
> > many as the machine allows, in my case ~100).
> >
>
> Thanks a lot for all these details, this is extremely helpful.
>
> So what appears to be happening is that we split the 2M block mapping
> that covers the code that we were called from, and hit a level 2
> translation fault because the updated page table entry is still
> observed to be in its transient 'invalid' state as we return to it.
>
> Could you please check whether this makes a difference?
>
> --- a/ArmPkg/Library/ArmMmuLib/AArch64/ArmMmuLibReplaceEntry.S
> +++ b/ArmPkg/Library/ArmMmuLib/AArch64/ArmMmuLibReplaceEntry.S
> @@ -65,6 +65,7 @@
>    // write updated entry
>    str   x1, [x0]
>    dsb   nshst
> +  isb
>
>  .L2_\@:
>    .endm

That fixes it - no crash observed within 150k iterations.
Thanks, Ard!

- Oliver

>
>
> > Details:
> >
> > CPU: Cavium ThunderX2(R) CPU CN9975
> > Tested on 3 different machines:
> >     HPE apache, HPE apollo, Gigabyte R181
> > Kernels tested:
> >  - 6.2.15-100.fc36.aarch64
> >  - 5.14.0-312.el9.aarch64
> >    (contains 406504c7b0405d74d74c15a667cd4c4620c3e7a9,
> >    "KVM: arm64: Fix S1PTW handling on RO memslots")
> > Qemu v8.0.0 (RHEL version and build from upstream repo)
> > EDK2: master branch from 2023-05-16 (cafb4f3f)
> > gcc 11.3.1
> >
> > EDK2 build command line:
> > build \
> >   -a AARCH64
> >   -p ArmVirtPkg/ArmVirtQemu.dsc
> >   -t GCC5 -b DEBUG \
> >   -D NETWORK_IP6_ENABLE \
> >   -D NETWORK_HTTP_BOOT_ENABLE \
> >   -D NETWORK_TLS_ENABLE \
> >   -D NETWORK_ISCSI_ENABLE \
> >   -D NETWORK_ALLOW_HTTP_CONNECTIONS \
> >   -D CAVIUM_ERRATUM_27456=TRUE \
> >   -D TPM2_ENABLE=TRUE \
> >   -D TPM1_ENABLE=FALSE \
> >   -D DEBUG_PRINT_ERROR_LEVEL=0x80000000  \
> >   -D BUILD_SHELL=TRUE \
> >   --pcd="gEfiShellPkgTokenSpaceGuid.PcdShellDefaultDelay=0" \
> >   --pcd="gEfiMdePkgTokenSpaceGuid.PcdPlatformBootTimeOut=0" \
> >   --hash --cmd-len=65536
> >
> > To reproduce the issue I launched the firmware in Qemu and have it do
> > a reboot once it finished booting up
> > via a startup.nsh on the ESP.
> >
> > Qemu command line:
> > qemu-system-aarch64 \
> >     -machine virt,accel=kvm -m 13G \
> >     -boot menu=off \
> >     -cpu host \
> >     -blockdev node-name=code,driver=file,filename="${FW_CODE}",read-only=on \
> >     -blockdev node-name=vars,driver=file,filename="${FW_VARS}" \
> >     -machine pflash0=code \
> >     -machine pflash1=vars \
> >     -serial stdio \
> >     -net none \
> >     -drive file=esp.img,snapshot=on
> >
> > Other things like number of CPUs or the presence of a vTPM have no
> > influence. I did not try different amounts of RAM yet.
> >
> > Serial output:
> > [...]
> > InitializeDxeNxMemoryProtectionPolicy: StackBase = 0x00000000476C5000
> > StackSize = 0x0000000000020000
> > InitializeDxeNxMemoryProtectionPolicy: applying strict permissions to
> > active memory regions
> > SetUefiImageMemoryAttributes - 0x0000000040000000 - 0x00000000076E5000
> > (0x0000000000004000)
> > UpdateRegionMappingRecursive(0): 40000000 - 476E5000 set
> > 60000000000400 clr FF9F000000000B3F
> > UpdateRegionMappingRecursive(1): 40000000 - 476E5000 set
> > 60000000000400 clr FF9F000000000B3F
> > UpdateRegionMappingRecursive(2): 40000000 - 476E5000 set
> > 60000000000400 clr FF9F000000000B3F
> > UpdateRegionMappingRecursive(3): 47600000 - 476E5000 set
> > 60000000000400 clr FF9F000000000B3F
> > SetUefiImageMemoryAttributes - 0x00000000476C5000 - 0x0000000000001000
> > (0x0000000000006000)
> > UpdateRegionMappingRecursive(0): 476C5000 - 476C6000 set
> > 60000000000000 clr FF9F000000000B3F
> > UpdateRegionMappingRecursive(1): 476C5000 - 476C6000 set
> > 60000000000000 clr FF9F000000000B3F
> > UpdateRegionMappingRecursive(2): 476C5000 - 476C6000 set
> > 60000000000000 clr FF9F000000000B3F
> > UpdateRegionMappingRecursive(3): 476C5000 - 476C6000 set
> > 60000000000000 clr FF9F000000000B3F
> > SetUefiImageMemoryAttributes - 0x000000004772B000 - 0x00000000007C0000
> > (0x0000000000004000)
> > UpdateRegionMappingRecursive(0): 4772B000 - 47EEB000 set
> > 60000000000400 clr FF9F000000000B3F
> > UpdateRegionMappingRecursive(1): 4772B000 - 47EEB000 set
> > 60000000000400 clr FF9F000000000B3F
> > UpdateRegionMappingRecursive(2): 4772B000 - 47EEB000 set
> > 60000000000400 clr FF9F000000000B3F
> > UpdateRegionMappingRecursive(3): 4772B000 - 47800000 set
> > 60000000000400 clr FF9F000000000B3F
> > UpdateRegionMappingRecursive(3): 47E00000 - 47EEB000 set
> > 60000000000400 clr FF9F000000000B3F
> > SetUefiImageMemoryAttributes - 0x0000000047EF3000 - 0x0000000000101000
> > (0x0000000000004000)
> > UpdateRegionMappingRecursive(0): 47EF3000 - 47FF4000 set
> > 60000000000400 clr FF9F000000000B3F
> > UpdateRegionMappingRecursive(1): 47EF3000 - 47FF4000 set
> > 60000000000400 clr FF9F000000000B3F
> > UpdateRegionMappingRecursive(2): 47EF3000 - 47FF4000 set
> > 60000000000400 clr FF9F000000000B3F
> > UpdateRegionMappingRecursive(3): 47EF3000 - 47FF4000 set
> > 60000000000400 clr FF9F000000000B3F
> > SetUefiImageMemoryAttributes - 0x0000000047FFA000 - 0x0000000334AA6000
> > (0x0000000000004000)
> > UpdateRegionMappingRecursive(0): 47FFA000 - 37CAA0000 set
> > 60000000000400 clr FF9F000000000B3F
> > UpdateRegionMappingRecursive(1): 47FFA000 - 37CAA0000 set
> > 60000000000400 clr FF9F000000000B3F
> > UpdateRegionMappingRecursive(2): 47FFA000 - 80000000 set
> > 60000000000400 clr FF9F000000000B3F
> > UpdateRegionMappingRecursive(3): 47FFA000 - 48000000 set
> > 60000000000400 clr FF9F000000000B3F
> > UpdateRegionMappingRecursive(2): 340000000 - 380000000 set 70C clr 0
> > UpdateRegionMappingRecursive(3): 37F000000 - 37F200000 set 70C clr 0
> > UpdateRegionMappingRecursive(2): 340000000 - 37CAA0000 set
> > 60000000000400 clr FF9F000000000B3F
> > UpdateRegionMappingRecursive(3): 37CA00000 - 37CC00000 set 70C clr 0
> > UpdateRegionMappingRecursive(3): 37CA00000 - 37CAA0000 set
> > 60000000000400 clr FF9F000000000B3F
> > SetUefiImageMemoryAttributes - 0x000000037CB40000 - 0x00000000031F9000
> > (0x0000000000004000)
> > UpdateRegionMappingRecursive(0): 37CB40000 - 37FD39000 set
> > 60000000000400 clr FF9F000000000B3F
> > UpdateRegionMappingRecursive(1): 37CB40000 - 37FD39000 set
> > 60000000000400 clr FF9F000000000B3F
> > UpdateRegionMappingRecursive(2): 37CB40000 - 37FD39000 set
> > 60000000000400 clr FF9F000000000B3F
> > UpdateRegionMappingRecursive(3): 37CB40000 - 37CC00000 set
> > 60000000000400 clr FF9F000000000B3F
> > UpdateRegionMappingRecursive(3): 37F000000 - 37F200000 set
> > 60000000000400 clr FF9F000000000B3F
> > UpdateRegionMappingRecursive(3): 37FC00000 - 37FE00000 set 70C clr 0
> > UpdateRegionMappingRecursive(3): 37FC00000 - 37FD39000 set
> > 60000000000400 clr FF9F000000000B3F
> >
> >
> > Synchronous Exception at 0x000000037FD3C0A8
> > PC 0x00037FD3C0A8 (0x00037FD39000+0x000030A8) [ 0] ArmCpuDxe.dll
> > PC 0x00037FD3C0A8 (0x00037FD39000+0x000030A8) [ 0] ArmCpuDxe.dll
> > PC 0x00037FD3BE70 (0x00037FD39000+0x00002E70) [ 0] ArmCpuDxe.dll
> > PC 0x00037FD3BE70 (0x00037FD39000+0x00002E70) [ 0] ArmCpuDxe.dll
> > PC 0x00037FD3C2E4 (0x00037FD39000+0x000032E4) [ 0] ArmCpuDxe.dll
> > PC 0x0000476E78F8 (0x0000476E5000+0x000028F8) [ 1] DxeCore.dll
> > PC 0x0000476ED680 (0x0000476E5000+0x00008680) [ 1] DxeCore.dll
> > PC 0x0000476F2744 (0x0000476E5000+0x0000D744) [ 1] DxeCore.dll
> > PC 0x0000476ECDE8 (0x0000476E5000+0x00007DE8) [ 1] DxeCore.dll
> > PC 0x00037FD3D2DC (0x00037FD39000+0x000042DC) [ 2] ArmCpuDxe.dll
> > PC 0x0000476EC788 (0x0000476E5000+0x00007788) [ 3] DxeCore.dll
> > PC 0x0000476F9CA8 (0x0000476E5000+0x00014CA8) [ 3] DxeCore.dll
> > PC 0x0000476EFEF0 (0x0000476E5000+0x0000AEF0) [ 3] DxeCore.dll
> >
> > [ 0] /root/edk2/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/ArmPkg/Drivers/CpuDxe/CpuDxe/DEBUG/ArmCpuDxe.dll
> > [ 1] /root/edk2/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/MdeModulePkg/Core/Dxe/DxeMain/DEBUG/DxeCore.dll
> > [ 2] /root/edk2/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/ArmPkg/Drivers/CpuDxe/CpuDxe/DEBUG/ArmCpuDxe.dll
> > [ 3] /root/edk2/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/MdeModulePkg/Core/Dxe/DxeMain/DEBUG/DxeCore.dll
> >
> >   X0 0x000000037F10BFF0   X1 0x000000037F106003   X2
> > 0x000000000037FC00   X3 0x0000000000000000
> >   X4 0x0000000000000200   X5 0x0000000000000004   X6
> > 0x0000000000000000   X7 0x000000037FD3F4B5
> >   X8 0x0000000000000000   X9 0x0000000000000002  X10
> > 0x0000000000000000  X11 0x0000000000000000
> >  X12 0x0000000000000002  X13 0x0000000000000002  X14
> > 0x0000000000000001  X15 0x0000000000000002
> >  X16 0x000000037FD3A268  X17 0x00000000007AFA10  X18
> > 0x0000000000000000  X19 0x000000037FC00000
> >  X20 0x0000000000000002  X21 0x000000037F106003  X22
> > 0x000000037F10B000  X23 0x000000037FD42000
> >  X24 0x00000000001FFFFF  X25 0x000000037FD39000  X26
> > 0x000000037F106000  X27 0x0000000000000003
> >  X28 0x000000037F10BFF0   FP 0x00000000476E4780   LR 0x000000037FD3C0A8
> >
> >   V0 0x0000000000000000 0000000000000000   V1 0x0000000000000000
> > 0000000000000000
> >   V2 0x0000000000000000 0000000000000000   V3 0x0000000000000000
> > 0000000000000000
> >   V4 0x0000000000000000 0000000000000000   V5 0x0000000000000000
> > 0000000000000000
> >   V6 0x0000000000000000 0000000000000000   V7 0x0000000000000000
> > 0000000000000000
> >   V8 0x0000000000000000 0000000000000000   V9 0x0000000000000000
> > 0000000000000000
> >  V10 0x0000000000000000 0000000000000000  V11 0x0000000000000000
> > 0000000000000000
> >  V12 0x0000000000000000 0000000000000000  V13 0x0000000000000000
> > 0000000000000000
> >  V14 0x0000000000000000 0000000000000000  V15 0x0000000000000000
> > 0000000000000000
> >  V16 0x0000000000000000 0000000000000000  V17 0x0000000000000000
> > 0000000000000000
> >  V18 0x0000000000000000 0000000000000000  V19 0x0000000000000000
> > 0000000000000000
> >  V20 0x0000000000000000 0000000000000000  V21 0x0000000000000000
> > 0000000000000000
> >  V22 0x0000000000000000 0000000000000000  V23 0x0000000000000000
> > 0000000000000000
> >  V24 0x0000000000000000 0000000000000000  V25 0x0000000000000000
> > 0000000000000000
> >  V26 0x0000000000000000 0000000000000000  V27 0x0000000000000000
> > 0000000000000000
> >  V28 0x0000000000000000 0000000000000000  V29 0x0000000000000000
> > 0000000000000000
> >  V30 0x0000000000000000 0000000000000000  V31 0x0000000000000000
> > 0000000000000000
> >
> >   SP 0x00000000476E4780  ELR 0x000000037FD3C0A8  SPSR 0x80000205  FPSR
> > 0x00000000
> >  ESR 0x86000006          FAR 0x000000037FD3C0A8
> >
> >  ESR : EC 0x21  IL 0x1  ISS 0x00000006
> >
> > Instruction abort: Translation fault, second level
> >
> > Stack dump:
> >   00000476E4680: 0000000000000001 0000000000000004 00000000476E4700
> > 00000000476F3980
> >   00000476E46A0: 000000037FD40CBD 0000000000000003 000000037FC00000
> > 000000037FD39000
> >   00000476E46C0: 0060000000000400 FF9F000000000B3F 00000000476E4780
> > 000000037FD3BE70
> >   00000476E46E0: 000000037FC00000 0000000000000002 000000037F106000
> > 000000037F10B000
> >   00000476E4700: 0000000000000FF0 00000000001FFFFF 000000037FD39000
> > 000000037F106000
> >   00000476E4720: 0000000000000003 000000037F10BFF0 0060000000000400
> > FF9F000000000B3F
> >   00000476E4740: 000000037FD39000 000000037FD39000 00000000476E4780
> > 0060000000000403
> >   00000476E4760: 0000000C00000001 000000037FD3F90E 0000000000000400
> > 000000037F10B000
> > > 00000476E4780: 00000000476E4830 000000037FD3BE70 000000037CB40000 0000000000000001
> >   00000476E47A0: 000000037F10B000 0000000047FFE000 0000000000000068
> > 000000003FFFFFFF
> >   00000476E47C0: 000000037FD39000 000000037F10C528 0000000000000002
> > 0000000047FFE068
> >   00000476E47E0: 0060000000000400 FF9F000000000B3F 0000000300000001
> > 000000037FD39000
> >   00000476E4800: 000000017FD40CBD 0060000000000401 0000001500000001
> > 000000037FD3F90E
> >   00000476E4820: 0060000000000400 000000037F106000 00000000476E48E0
> > 000000037FD3BE70
> >   00000476E4840: 000000037CB40000 0000000000000000 0000000047FFE000
> > 0000000047FFF000
> >   00000476E4860: 0000000000000000 0000007FFFFFFFFF 000000037FD39000
> > 000000037F10C528
> > ASSERT [ArmCpuDxe]
> > /root/edk2/ArmPkg/Library/DefaultExceptionHandlerLib/AArch64/DefaultExceptionHandler.c(333):
> > ((BOOLEAN)(0==1))
> >
> >
> >
> > The full log is available here:
> > https://gitlab.com/osteffen/thunderx2-debug/-/raw/main/2023-05-19/85.log?inline=false
> >
> > Debug files, firmware binaries, and the full build tree are here:
> > https://gitlab.com/osteffen/thunderx2-debug/-/tree/main/2023-05-19
> >
> > I am able to reproduce this quickly, so any ideas for what I can try
> > are welcome :-)
> >
> > Thanks
> > -Oliver
> >
>

--
🎩Oliver Steffen (he/him) - Software Engineer, Virtualization
Red Hat GmbH <https://www.redhat.com/de/global/dach>,
Registered seat: Werner-von-Siemens-Ring 12, D-85630 Grasbrunn, Germany
Commercial register: Amtsgericht München/Munich, HRB 153243,
Managing Directors: Ryan Barnhart, Charles Cachera, Michael O'Neill,
Amy Ross

Everyone has different working hours… Please do not feel obligated to
reply outside of your normal work schedule.

Re: [edk2-devel] [PATCH v2 2/2] ArmVirtPkg/ArmVirtQemu: Avoid early ID map on ThunderX

[Ard Biesheuvel]

On Sat, 20 May 2023 at 10:37, Oliver Steffen <osteffen@redhat.com> wrote:
>
> Quoting Ard Biesheuvel (2023-05-19 23:36:53)
> > On Fri, 19 May 2023 at 18:32, Oliver Steffen <osteffen@redhat.com> wrote:
> > >
> > >
> > > Hi all,
> > >
> > > I had another look at this and I can now reproduce the issue consistently,
> > > with a quite minimal setup, on recent Linux kernel, Qemu, and EDK2.
> > > It requires rebooting the guest in a tight loop. It happens in silent
> > > and verbose
> > > builds alike, but since the verbose ones are slowed down by the serial
> > > output, it
> > > takes longer to hit the issue.
> > > It is possible to reproduce it with the silent builds within a few minutes.
> > > For the verbose case I recommend running multiple Qemu instances in parallel (as
> > > many as the machine allows, in my case ~100).
> > >
> >
> > Thanks a lot for all these details, this is extremely helpful.
> >
> > So what appears to be happening is that we split the 2M block mapping
> > that covers the code that we were called from, and hit a level 2
> > translation fault because the updated page table entry is still
> > observed to be in its transient 'invalid' state as we return to it.
> >
> > Could you please check whether this makes a difference?
> >
> > --- a/ArmPkg/Library/ArmMmuLib/AArch64/ArmMmuLibReplaceEntry.S
> > +++ b/ArmPkg/Library/ArmMmuLib/AArch64/ArmMmuLibReplaceEntry.S
> > @@ -65,6 +65,7 @@
> >    // write updated entry
> >    str   x1, [x0]
> >    dsb   nshst
> > +  isb
> >
> >  .L2_\@:
> >    .endm
>
> That fixes it - no crash observed within 150k iterations.
> Thanks, Ard!
>

Fantastic! Thanks a lot for all the effort in tracking this down.