On Wed, Feb 1, 2023 at 2:29 PM Ard Biesheuvel wrote: > On Wed, 1 Feb 2023 at 13:59, Oliver Steffen wrote: > > > > On Wed, Feb 1, 2023 at 12:52 PM Ard Biesheuvel wrote: > >> > >> On Wed, 1 Feb 2023 at 10:14, Oliver Steffen > wrote: > >> > > [...] > >> > I am sorry, this story does not seem to be over yet. > >> > > >> > We are using the Erratum patch and also included the commit 406504c7 > in > >> > the kernel. > >> > Now the firmware crashes sometimes (10 out of 89 tests). > >> > > >> > >> Thanks for the report. Is this still on ThunderX2? > >> > >> > Any hints are very welcome! > >> > > >> > >> Do you have access to those build artifacts? > > > > > > > https://kojihub.stream.centos.org/kojifiles/work/tasks/5251/1835251/edk2-aarch64-20221207gitfff6d81270b5-4.el9.test.noarch.rpm > > > > and/or here: > > > > https://kojihub.stream.centos.org/koji/taskinfo?taskID=1835251 > > > > Source for reference: > > https://gitlab.com/redhat/centos-stream/src/edk2/-/merge_requests/24 > > > > Any chance the .dll files (which are actually ELF executables) have > been preserved somewhere? > Here is the build folder (~90MB): https://gitlab.com/osteffen/thunderx2-debug/-/raw/main/armvirt-thunderx2-issue.tar.xz I am waiting for the tests with the additional debug output to run. > >> Can you share more of the log output? > >> If you can rebuild, can you run it with DEBUG_VERBOSE set on ArmCpuDxe. > E.g., > >> > >> --- a/ArmVirtPkg/ArmVirtQemu.dsc > >> +++ b/ArmVirtPkg/ArmVirtQemu.dsc > >> @@ -383,7 +383,11 @@ [Components.common] > >> # > >> # Architectural Protocols > >> # > >> - ArmPkg/Drivers/CpuDxe/CpuDxe.inf > >> + ArmPkg/Drivers/CpuDxe/CpuDxe.inf { > >> + > >> + gEfiMdePkgTokenSpaceGuid.PcdDebugPrintErrorLevel|0x8040004F > >> + } > >> + > >> MdeModulePkg/Core/RuntimeDxe/RuntimeDxe.inf > >> MdeModulePkg/Universal/Variable/RuntimeDxe/VariableRuntimeDxe.inf { > >> > > > > I'll try to do that. > > > > > >> > >> > Here is the serial output of one case: > >> > ---------------------------------------------------------------- > >> > UEFI firmware (version edk2-20221207gitfff6d81270b5-4.el9.test built > at 00:00:00 on Jan 18 2023) > >> > SyncPcrAllocationsAndPcrMask! > >> > > >> > > >> > Synchronous Exception at 0x000000037FD5BDE0 > >> > PC 0x00037FD5BDE0 (0x00037FD59000+0x00002DE0) [ 0] ArmCpuDxe.dll > >> > PC 0x00037FD5BDE0 (0x00037FD59000+0x00002DE0) [ 0] ArmCpuDxe.dll > >> > PC 0x00037FD5BCE0 (0x00037FD59000+0x00002CE0) [ 0] ArmCpuDxe.dll > >> > >> This, along with > >> > >> > SP 0x00000000476ED840 ELR 0x000000037FD5BDE0 SPSR 0x80000205 > FPSR 0x00000000 > >> > ESR 0x86000005 FAR 0x000000037FD5BDE0 > >> > >> taken from below suggests that UpdateRegionMappingRecursive () is > >> unmapping itself, likely when splitting a block entry. > >> > >> This affects all ARM builds, not just ArmVIrtQemu so I would like to > >> get to the bottom of this. > >> > >> > >> Thanks, > >> Ard. > >> > >> > >> > >> > PC 0x00037FD5C054 (0x00037FD59000+0x00003054) [ 0] ArmCpuDxe.dll > >> > PC 0x0000476F08EC (0x0000476EE000+0x000028EC) [ 1] DxeCore.dll > >> > PC 0x0000476F65C0 (0x0000476EE000+0x000085C0) [ 1] DxeCore.dll > >> > PC 0x0000476FB5B8 (0x0000476EE000+0x0000D5B8) [ 1] DxeCore.dll > >> > PC 0x0000476F5DB8 (0x0000476EE000+0x00007DB8) [ 1] DxeCore.dll > >> > PC 0x00037FD5CD50 (0x00037FD59000+0x00003D50) [ 2] ArmCpuDxe.dll > >> > PC 0x0000476F5758 (0x0000476EE000+0x00007758) [ 3] DxeCore.dll > >> > PC 0x000047702B18 (0x0000476EE000+0x00014B18) [ 3] DxeCore.dll > >> > PC 0x0000476F8D70 (0x0000476EE000+0x0000AD70) [ 3] DxeCore.dll > >> > > >> > [ 0] > /builddir/build/BUILD/edk2-fff6d81270b5/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/ArmPkg/Drivers/CpuDxe/CpuDxe/DEBUG/ArmCpuDxe.dll > >> > [ 1] > /builddir/build/BUILD/edk2-fff6d81270b5/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/MdeModulePkg/Core/Dxe/DxeMain/DEBUG/DxeCore.dll > >> > [ 2] > /builddir/build/BUILD/edk2-fff6d81270b5/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/ArmPkg/Drivers/CpuDxe/CpuDxe/DEBUG/ArmCpuDxe.dll > >> > [ 3] > /builddir/build/BUILD/edk2-fff6d81270b5/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/MdeModulePkg/Core/Dxe/DxeMain/DEBUG/DxeCore.dll > >> > > >> > X0 0x0000000047FFE068 X1 0x000000037F179003 X2 > 0x0000000000340000 X3 0x0000000000000000 > >> > X4 0x0000000000000200 X5 0x0000000000000004 X6 > 0x0060000000000000 X7 0xFF9F000000000F3F > >> > X8 0x000000037FFFF008 X9 0x0000000400000000 X10 > 0x000000037F177000 X11 0x000000037FA37FFF > >> > X12 0x0000000000000000 X13 0x0000000000000008 X14 > 0x0000000000000000 X15 0x0000000000000000 > >> > X16 0x000000037FD5A208 X17 0x00000000007979D0 X18 > 0x0000000000000000 X19 0x0000000340000000 > >> > X20 0x0000000000000001 X21 0x000000037F179003 X22 > 0x0000000047FFE000 X23 0x0000000000000068 > >> > X24 0x000000003FFFFFFF X25 0x000000037CAA0000 X26 > 0x0000000000000002 X27 0x000000037F179000 > >> > X28 0x0000000047FFE068 FP 0x00000000476ED840 LR > 0x000000037FD5BDE0 > >> > > >> > V0 0x0000000000000000 0000000000000000 V1 0x0000000000000000 > 0000000000000000 > >> > V2 0x0000000000000000 0000000000000000 V3 0x0000000000000000 > 0000000000000000 > >> > V4 0x0000000000000000 0000000000000000 V5 0x0000000000000000 > 0000000000000000 > >> > V6 0x0000000000000000 0000000000000000 V7 0x0000000000000000 > 0000000000000000 > >> > V8 0x0000000000000000 0000000000000000 V9 0x0000000000000000 > 0000000000000000 > >> > V10 0x0000000000000000 0000000000000000 V11 0x0000000000000000 > 0000000000000000 > >> > V12 0x0000000000000000 0000000000000000 V13 0x0000000000000000 > 0000000000000000 > >> > V14 0x0000000000000000 0000000000000000 V15 0x0000000000000000 > 0000000000000000 > >> > V16 0x0000000000000000 0000000000000000 V17 0x0000000000000000 > 0000000000000000 > >> > V18 0x0000000000000000 0000000000000000 V19 0x0000000000000000 > 0000000000000000 > >> > V20 0x0000000000000000 0000000000000000 V21 0x0000000000000000 > 0000000000000000 > >> > V22 0x0000000000000000 0000000000000000 V23 0x0000000000000000 > 0000000000000000 > >> > V24 0x0000000000000000 0000000000000000 V25 0x0000000000000000 > 0000000000000000 > >> > V26 0x0000000000000000 0000000000000000 V27 0x0000000000000000 > 0000000000000000 > >> > V28 0x0000000000000000 0000000000000000 V29 0x0000000000000000 > 0000000000000000 > >> > V30 0x0000000000000000 0000000000000000 V31 0x0000000000000000 > 0000000000000000 > >> > > >> > SP 0x00000000476ED840 ELR 0x000000037FD5BDE0 SPSR 0x80000205 > FPSR 0x00000000 > >> > ESR 0x86000005 FAR 0x000000037FD5BDE0 > >> > > >> > ESR : EC 0x21 IL 0x1 ISS 0x00000005 > >> > > >> > Instruction abort: Translation fault, first level > >> > > >> > Stack dump: > >> > 00000476ED740: 000000037CAA0000 000000037CAA0000 0060000000000000 > FF9F000000000F3F > >> > 00000476ED760: 00000000476ED790 000000037FD610D8 0060000000000003 > 0000000C00000001 > >> > 00000476ED780: 000000037CA0070D 000000037F179000 00000000476ED840 > 000000037FD5BCE0 > >> > 00000476ED7A0: 0000000340000000 0000000000000001 000000037F179000 > 0000000047FFE000 > >> > 00000476ED7C0: 0000000000000068 000000003FFFFFFF 000000037CAA0000 > 0000000000000002 > >> > 00000476ED7E0: 000000037F179000 0000000047FFE068 000000037CC00000 > 000000037CAA0000 > >> > 00000476ED800: 0060000000000000 FF9F000000000F3F 00000000476ED840 > 000000037FD610D8 > >> > 00000476ED820: 0060000000000001 0000001500000001 000000034000070D > 000000037F177000 > >> > > 00000476ED840: 00000000476ED8F0 000000037FD5BCE0 0000000047FFA000 > 0000000000000000 > >> > 00000476ED860: 0000000047FFE000 0000000047FFF000 0000000000000000 > 0000007FFFFFFFFF > >> > 00000476ED880: 000000037CAA0000 0000000000000001 0000000047717588 > 0000000047FFF000 > >> > 00000476ED8A0: 0000000380000000 000000037CAA0000 0060000000000000 > FF9F000000000F3F > >> > 00000476ED8C0: 000000017FD605DD 000000037FD610D8 0060000000000001 > 0000001E00000001 > >> > 00000476ED8E0: 0060000000000000 000000037F179000 00000000476ED9A0 > 000000037FD5C054 > >> > 00000476ED900: 002000000000041C 0000000000000000 0000000047FFA000 > 0000000000004000 > >> > 00000476ED920: 0000000334AA6000 0000000047FFF000 000000037F17A238 > 0000000047717000 > >> > ASSERT [ArmCpuDxe] > /builddir/build/BUILD/edk2-fff6d81270b5/ArmPkg/Library/DefaultExceptionHandlerLib/AArch64/DefaultExceptionHandler.c(333): > ((BOOLEAN)(0==1)) > >> > ---------------------------------------------------------------- > >> > Thanks, > >> > Oliver > >> > > >