From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by mx.groups.io with SMTP id smtpd.web10.31112.1684513940473814488 for ; Fri, 19 May 2023 09:32:20 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=MufVDKTj; spf=pass (domain: redhat.com, ip: 170.10.133.124, mailfrom: osteffen@redhat.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684513939; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=lyOrzT3O+lIzYvZB+k4xSBJECioeqBAuoPnqMA6qPJ4=; b=MufVDKTjqdEQg7a4edP6lVTFiHpzyA0+y+qUzIPlDdEDusYDcF5uftoPqyeW6a0aYgHWau Wv5ANQswigCxpue0VbQ7pE/aFfVB23vqYNeBqSrq6j2LaUcQuy/l6s1LoVi71MdRtfPUmO l9kVRqgpO0mZaBzKJd6qGdwhbtoeyE8= Received: from mail-lf1-f69.google.com (mail-lf1-f69.google.com [209.85.167.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-60-EkyjN-KMPFKevA_qWn13hQ-1; Fri, 19 May 2023 12:32:16 -0400 X-MC-Unique: EkyjN-KMPFKevA_qWn13hQ-1 Received: by mail-lf1-f69.google.com with SMTP id 2adb3069b0e04-4edc7ab63ccso2166380e87.3 for ; Fri, 19 May 2023 09:32:16 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684513935; x=1687105935; h=content-transfer-encoding:cc:to:subject:message-id:date:user-agent :from:references:in-reply-to:mime-version:x-gm-message-state:from:to :cc:subject:date:message-id:reply-to; bh=lyOrzT3O+lIzYvZB+k4xSBJECioeqBAuoPnqMA6qPJ4=; b=eAI2e93MGfBoITwZYOWn4kQafnqrf9Qe4JDxbmXPpRYGGUfaaZirkXa8lOZgsjTABu unbyCTdJN7De3uum5uCrGX3Xx67+1keIwud0M+Bj3tqbNmOFrHFCenl977xsOaVKlFyl 0btKHAdMk2ENHThM98SrCr0+Q58tX/+l9uGBO9k6jxxbyrz+OjS7cm4Q9IGdpsqAqRyr ItXvrKTe2N3F6nfPGiGlUYdby5VPWdhmz70aBGN9pm2Dai2EY9MkPdlxJFpYD6hf5NAx eoJjmQVDB4tiiKP5LCF3L8cY0bcnACyzjxcEjDB+Kxv8ogWGCf+c+uhVrk9QRJLFsgk+ bmhQ== X-Gm-Message-State: AC+VfDxWHckNTArbw5FJLhcxJclJqg+WiRQSIExMo/ZlHK/ydyEkW0B4 ezXF29UUcW0b54HLUNmgCWVSfZO9xbSYvthVhF6Vxu/Ei5xBvpuK+4dT8DfuzJLdVxvNLtJwKzk lEdQKkPcMgyABsTDTcph/di2kJ90OBw== X-Received: by 2002:ac2:5550:0:b0:4eb:79:fa5 with SMTP id l16-20020ac25550000000b004eb00790fa5mr893132lfk.25.1684513934825; Fri, 19 May 2023 09:32:14 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ6MWkm40MnIMuXpAlPL7vfpezQn9mw/fN2erv4vJvPXnoxt8f0xQiONECu5lqo8MowQnJRJF2+NIlST2Zdb4JA= X-Received: by 2002:ac2:5550:0:b0:4eb:79:fa5 with SMTP id l16-20020ac25550000000b004eb00790fa5mr893117lfk.25.1684513934401; Fri, 19 May 2023 09:32:14 -0700 (PDT) Received: from 567203818698 named unknown by gmailapi.google.com with HTTPREST; Fri, 19 May 2023 12:32:13 -0400 MIME-Version: 1.0 In-Reply-To: <17489D498A098DB9.9697@groups.io> References: <173FFD60429C89C3.3213@groups.io> <17489D498A098DB9.9697@groups.io> From: "Oliver Steffen" User-Agent: alot/0.8.1 Date: Fri, 19 May 2023 12:32:13 -0400 Message-ID: Subject: Re: [edk2-devel] [PATCH v2 2/2] ArmVirtPkg/ArmVirtQemu: Avoid early ID map on ThunderX To: Oliver Steffen , ardb@kernel.org, devel@edk2.groups.io Cc: Gerd Hoffmann , Marc Zyngier , dann.frazier@canonical.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Quoting Oliver Steffen (2023-03-02 14:29:43) > On Thu, Mar 2, 2023 at 11:50=E2=80=AFAM Ard Biesheuvel <[1]ardb@kernel.or= g> wrote: > > On Thu, 9 Feb 2023 at 16:15, Ard Biesheuvel <[2]ardb@kernel.org> wrot= e: > > > > On Tue, 7 Feb 2023 at 13:58, Oliver Steffen <[3]osteffen@redhat.com= > > wrote: > > > > > > On Tue, Feb 7, 2023 at 12:57 PM Ard Biesheuvel <[4]ardb@kernel.or= g> > wrote: > > >> > > >> On Tue, 7 Feb 2023 at 11:51, Oliver Steffen <[5]osteffen@redhat.= com> > wrote: > > >> > > > >> > On Thu, Feb 2, 2023 at 12:09 PM Oliver Steffen <[6] > osteffen@redhat.com> wrote: > > >> >> > > >> >> > > >> >> On Wed, Feb 1, 2023 at 2:29 PM Ard Biesheuvel <[7]ardb@kernel= .org> > wrote: > > >> >>> > > >> >>> On Wed, 1 Feb 2023 at 13:59, Oliver Steffen <[8] > osteffen@redhat.com> wrote: > > >> >>> > > > >> >>> > On Wed, Feb 1, 2023 at 12:52 PM Ard Biesheuvel <[9] > ardb@kernel.org> wrote: > > >> >>> >> > > >> >>> >> On Wed, 1 Feb 2023 at 10:14, Oliver Steffen <[10] > osteffen@redhat.com> wrote: > > >> >>> >> > > > >> >> > > >> >> [...] > > >> >>> > > >> >>> >> > I am sorry, this story does not seem to be over yet. > > >> >>> >> > > > >> >>> >> > We are using the Erratum patch and also included the co= mmit > 406504c7 in > > >> >>> >> > the kernel. > > >> >>> >> > Now the firmware crashes sometimes (10 out of 89 tests)= . > > >> >>> >> > > > >> >>> >> > > >> >>> >> Thanks for the report. Is this still on ThunderX2? > > >> >>> >> > > >> >>> >> > Any hints are very welcome! > > >> >>> >> > > > >> >>> >> > > >> >>> >> Do=C2=A0 you have access to those build artifacts? > > >> >>> > > > >> >>> > > > >> >>> > [11]https://kojihub.stream.centos.org/kojifiles/work/tasks= /5251/ > 1835251/edk2-aarch64-20221207gitfff6d81270b5-4.el9.test.noarch.rpm > > >> >>> > > > >> >>> > and/or here: > > >> >>> > > > >> >>> > [12]https://kojihub.stream.centos.org/koji/taskinfo?taskID= =3D > 1835251 > > >> >>> > > > >> >>> > Source for reference: > > >> >>> > [13]https://gitlab.com/redhat/centos-stream/src/edk2/-/ > merge_requests/24 > > >> >>> > > > >> >>> > > >> >>> Any chance the .dll files (which are actually ELF executable= s) > have > > >> >>> been preserved somewhere? > > >> >> > > >> >> Here is the build folder (~90MB): > > >> >> [14]https://gitlab.com/osteffen/thunderx2-debug/-/raw/main/ > armvirt-thunderx2-issue.tar.xz > > >> >> > > >> >> I am waiting for the tests with the additional debug output t= o run. > > >> > > > >> > > > >> > We reran the test suite with the Erratum and the additional de= bug > > >> > output enabled.=C2=A0 Strangely, the problem does not occur an= ymore, the > > >> > firmware boots up normally. > > >> > > > >> > We retried the tests without the additional debug output. > > >> > RHEL ships two firmware flavors for AARCH64: a silent and a ve= rbose > > >> > version. > > >> > > >> Are these RELEASE vs DEBUG builds? > > > > > > > > > All builds are DEBUG, just the amount of information printed on > > > the serial is different (almost zero for the "silent" one.) > > > > > >> > > >> > Both were tried. We see no problems with the verbose > > >> > one. The silent one fails noticeably more often if a software = TPM > device > > >> > is present. > > >> > > > >> > > >> This smells like some missing cache or TLB maintenance - the ver= bose > > >> one exits to the host much more often, and likely relies on cach= e/TLB > > >> maintenance occurring in the hypervisor. > > >> > > >> So the build always includes TPM support but the issue only occu= rs > > >> when the sw TPM is actually exposed by QEMU? > > > > > > > > > Yes. > > > All builds include support for TPM, but the issue occurs more > frequently > > > if a sw TPM is exposed by QEMU. > > > > > > > Any chance you could provide a specific command line for launching > > QEMU? I am trying to reproduce this, but I am not making any progre= ss. > > > > >> > > >> > Could this be related to how much stuff is going on in the ear= ly > phase > > >> > of the firmware (when logging is enabled: formatting of messag= es and > > >> > sending to serial port...) ? > > >> > > > >> > > >> I'll try to see if I can rig something up that logs into a buffe= r > > >> rather than straight to the serial, and dump it all out when han= dling > > >> the crash > > >> > > > > This takes a bit more time than I can afford to spend on this atm, = and > > I'd like to be able to reproduce before I go down this rabbit hole. > > Have there been any developments regarding this issue? > > > Nothing from my side.=C2=A0 I tried to come up with a more reliable/faste= r > reproducer > but then stopped because of other stuff. > > If you have any idea what I could try next let me know. > > -Oliver # Summary for Email 2 Hi all, I had another look at this and I can now reproduce the issue consistently, with a quite minimal setup, on recent Linux kernel, Qemu, and EDK2. It requires rebooting the guest in a tight loop. It happens in silent and verbose builds alike, but since the verbose ones are slowed down by the serial output, it takes longer to hit the issue. It is possible to reproduce it with the silent builds within a few minutes. For the verbose case I recommend running multiple Qemu instances in paralle= l (as many as the machine allows, in my case ~100). Details: CPU: Cavium ThunderX2(R) CPU CN9975 Tested on 3 different machines: HPE apache, HPE apollo, Gigabyte R181 Kernels tested: - 6.2.15-100.fc36.aarch64 - 5.14.0-312.el9.aarch64 (contains 406504c7b0405d74d74c15a667cd4c4620c3e7a9, "KVM: arm64: Fix S1PTW handling on RO memslots") Qemu v8.0.0 (RHEL version and build from upstream repo) EDK2: master branch from 2023-05-16 (cafb4f3f) gcc 11.3.1 EDK2 build command line: build \ -a AARCH64 -p ArmVirtPkg/ArmVirtQemu.dsc -t GCC5 -b DEBUG \ -D NETWORK_IP6_ENABLE \ -D NETWORK_HTTP_BOOT_ENABLE \ -D NETWORK_TLS_ENABLE \ -D NETWORK_ISCSI_ENABLE \ -D NETWORK_ALLOW_HTTP_CONNECTIONS \ -D CAVIUM_ERRATUM_27456=3DTRUE \ -D TPM2_ENABLE=3DTRUE \ -D TPM1_ENABLE=3DFALSE \ -D DEBUG_PRINT_ERROR_LEVEL=3D0x80000000 \ -D BUILD_SHELL=3DTRUE \ --pcd=3D"gEfiShellPkgTokenSpaceGuid.PcdShellDefaultDelay=3D0" \ --pcd=3D"gEfiMdePkgTokenSpaceGuid.PcdPlatformBootTimeOut=3D0" \ --hash --cmd-len=3D65536 To reproduce the issue I launched the firmware in Qemu and have it do a reboot once it finished booting up via a startup.nsh on the ESP. Qemu command line: qemu-system-aarch64 \ -machine virt,accel=3Dkvm -m 13G \ -boot menu=3Doff \ -cpu host \ -blockdev node-name=3Dcode,driver=3Dfile,filename=3D"${FW_CODE}",read-o= nly=3Don \ -blockdev node-name=3Dvars,driver=3Dfile,filename=3D"${FW_VARS}" \ -machine pflash0=3Dcode \ -machine pflash1=3Dvars \ -serial stdio \ -net none \ -drive file=3Desp.img,snapshot=3Don Other things like number of CPUs or the presence of a vTPM have no influence. I did not try different amounts of RAM yet. Serial output: [...] InitializeDxeNxMemoryProtectionPolicy: StackBase =3D 0x00000000476C5000 StackSize =3D 0x0000000000020000 InitializeDxeNxMemoryProtectionPolicy: applying strict permissions to active memory regions SetUefiImageMemoryAttributes - 0x0000000040000000 - 0x00000000076E5000 (0x0000000000004000) UpdateRegionMappingRecursive(0): 40000000 - 476E5000 set 60000000000400 clr FF9F000000000B3F UpdateRegionMappingRecursive(1): 40000000 - 476E5000 set 60000000000400 clr FF9F000000000B3F UpdateRegionMappingRecursive(2): 40000000 - 476E5000 set 60000000000400 clr FF9F000000000B3F UpdateRegionMappingRecursive(3): 47600000 - 476E5000 set 60000000000400 clr FF9F000000000B3F SetUefiImageMemoryAttributes - 0x00000000476C5000 - 0x0000000000001000 (0x0000000000006000) UpdateRegionMappingRecursive(0): 476C5000 - 476C6000 set 60000000000000 clr FF9F000000000B3F UpdateRegionMappingRecursive(1): 476C5000 - 476C6000 set 60000000000000 clr FF9F000000000B3F UpdateRegionMappingRecursive(2): 476C5000 - 476C6000 set 60000000000000 clr FF9F000000000B3F UpdateRegionMappingRecursive(3): 476C5000 - 476C6000 set 60000000000000 clr FF9F000000000B3F SetUefiImageMemoryAttributes - 0x000000004772B000 - 0x00000000007C0000 (0x0000000000004000) UpdateRegionMappingRecursive(0): 4772B000 - 47EEB000 set 60000000000400 clr FF9F000000000B3F UpdateRegionMappingRecursive(1): 4772B000 - 47EEB000 set 60000000000400 clr FF9F000000000B3F UpdateRegionMappingRecursive(2): 4772B000 - 47EEB000 set 60000000000400 clr FF9F000000000B3F UpdateRegionMappingRecursive(3): 4772B000 - 47800000 set 60000000000400 clr FF9F000000000B3F UpdateRegionMappingRecursive(3): 47E00000 - 47EEB000 set 60000000000400 clr FF9F000000000B3F SetUefiImageMemoryAttributes - 0x0000000047EF3000 - 0x0000000000101000 (0x0000000000004000) UpdateRegionMappingRecursive(0): 47EF3000 - 47FF4000 set 60000000000400 clr FF9F000000000B3F UpdateRegionMappingRecursive(1): 47EF3000 - 47FF4000 set 60000000000400 clr FF9F000000000B3F UpdateRegionMappingRecursive(2): 47EF3000 - 47FF4000 set 60000000000400 clr FF9F000000000B3F UpdateRegionMappingRecursive(3): 47EF3000 - 47FF4000 set 60000000000400 clr FF9F000000000B3F SetUefiImageMemoryAttributes - 0x0000000047FFA000 - 0x0000000334AA6000 (0x0000000000004000) UpdateRegionMappingRecursive(0): 47FFA000 - 37CAA0000 set 60000000000400 clr FF9F000000000B3F UpdateRegionMappingRecursive(1): 47FFA000 - 37CAA0000 set 60000000000400 clr FF9F000000000B3F UpdateRegionMappingRecursive(2): 47FFA000 - 80000000 set 60000000000400 clr FF9F000000000B3F UpdateRegionMappingRecursive(3): 47FFA000 - 48000000 set 60000000000400 clr FF9F000000000B3F UpdateRegionMappingRecursive(2): 340000000 - 380000000 set 70C clr 0 UpdateRegionMappingRecursive(3): 37F000000 - 37F200000 set 70C clr 0 UpdateRegionMappingRecursive(2): 340000000 - 37CAA0000 set 60000000000400 clr FF9F000000000B3F UpdateRegionMappingRecursive(3): 37CA00000 - 37CC00000 set 70C clr 0 UpdateRegionMappingRecursive(3): 37CA00000 - 37CAA0000 set 60000000000400 clr FF9F000000000B3F SetUefiImageMemoryAttributes - 0x000000037CB40000 - 0x00000000031F9000 (0x0000000000004000) UpdateRegionMappingRecursive(0): 37CB40000 - 37FD39000 set 60000000000400 clr FF9F000000000B3F UpdateRegionMappingRecursive(1): 37CB40000 - 37FD39000 set 60000000000400 clr FF9F000000000B3F UpdateRegionMappingRecursive(2): 37CB40000 - 37FD39000 set 60000000000400 clr FF9F000000000B3F UpdateRegionMappingRecursive(3): 37CB40000 - 37CC00000 set 60000000000400 clr FF9F000000000B3F UpdateRegionMappingRecursive(3): 37F000000 - 37F200000 set 60000000000400 clr FF9F000000000B3F UpdateRegionMappingRecursive(3): 37FC00000 - 37FE00000 set 70C clr 0 UpdateRegionMappingRecursive(3): 37FC00000 - 37FD39000 set 60000000000400 clr FF9F000000000B3F Synchronous Exception at 0x000000037FD3C0A8 PC 0x00037FD3C0A8 (0x00037FD39000+0x000030A8) [ 0] ArmCpuDxe.dll PC 0x00037FD3C0A8 (0x00037FD39000+0x000030A8) [ 0] ArmCpuDxe.dll PC 0x00037FD3BE70 (0x00037FD39000+0x00002E70) [ 0] ArmCpuDxe.dll PC 0x00037FD3BE70 (0x00037FD39000+0x00002E70) [ 0] ArmCpuDxe.dll PC 0x00037FD3C2E4 (0x00037FD39000+0x000032E4) [ 0] ArmCpuDxe.dll PC 0x0000476E78F8 (0x0000476E5000+0x000028F8) [ 1] DxeCore.dll PC 0x0000476ED680 (0x0000476E5000+0x00008680) [ 1] DxeCore.dll PC 0x0000476F2744 (0x0000476E5000+0x0000D744) [ 1] DxeCore.dll PC 0x0000476ECDE8 (0x0000476E5000+0x00007DE8) [ 1] DxeCore.dll PC 0x00037FD3D2DC (0x00037FD39000+0x000042DC) [ 2] ArmCpuDxe.dll PC 0x0000476EC788 (0x0000476E5000+0x00007788) [ 3] DxeCore.dll PC 0x0000476F9CA8 (0x0000476E5000+0x00014CA8) [ 3] DxeCore.dll PC 0x0000476EFEF0 (0x0000476E5000+0x0000AEF0) [ 3] DxeCore.dll [ 0] /root/edk2/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/ArmPkg/Drivers= /CpuDxe/CpuDxe/DEBUG/ArmCpuDxe.dll [ 1] /root/edk2/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/MdeModulePkg/C= ore/Dxe/DxeMain/DEBUG/DxeCore.dll [ 2] /root/edk2/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/ArmPkg/Drivers= /CpuDxe/CpuDxe/DEBUG/ArmCpuDxe.dll [ 3] /root/edk2/Build/ArmVirtQemu-AARCH64/DEBUG_GCC5/AARCH64/MdeModulePkg/C= ore/Dxe/DxeMain/DEBUG/DxeCore.dll X0 0x000000037F10BFF0 X1 0x000000037F106003 X2 0x000000000037FC00 X3 0x0000000000000000 X4 0x0000000000000200 X5 0x0000000000000004 X6 0x0000000000000000 X7 0x000000037FD3F4B5 X8 0x0000000000000000 X9 0x0000000000000002 X10 0x0000000000000000 X11 0x0000000000000000 X12 0x0000000000000002 X13 0x0000000000000002 X14 0x0000000000000001 X15 0x0000000000000002 X16 0x000000037FD3A268 X17 0x00000000007AFA10 X18 0x0000000000000000 X19 0x000000037FC00000 X20 0x0000000000000002 X21 0x000000037F106003 X22 0x000000037F10B000 X23 0x000000037FD42000 X24 0x00000000001FFFFF X25 0x000000037FD39000 X26 0x000000037F106000 X27 0x0000000000000003 X28 0x000000037F10BFF0 FP 0x00000000476E4780 LR 0x000000037FD3C0A8 V0 0x0000000000000000 0000000000000000 V1 0x0000000000000000 0000000000000000 V2 0x0000000000000000 0000000000000000 V3 0x0000000000000000 0000000000000000 V4 0x0000000000000000 0000000000000000 V5 0x0000000000000000 0000000000000000 V6 0x0000000000000000 0000000000000000 V7 0x0000000000000000 0000000000000000 V8 0x0000000000000000 0000000000000000 V9 0x0000000000000000 0000000000000000 V10 0x0000000000000000 0000000000000000 V11 0x0000000000000000 0000000000000000 V12 0x0000000000000000 0000000000000000 V13 0x0000000000000000 0000000000000000 V14 0x0000000000000000 0000000000000000 V15 0x0000000000000000 0000000000000000 V16 0x0000000000000000 0000000000000000 V17 0x0000000000000000 0000000000000000 V18 0x0000000000000000 0000000000000000 V19 0x0000000000000000 0000000000000000 V20 0x0000000000000000 0000000000000000 V21 0x0000000000000000 0000000000000000 V22 0x0000000000000000 0000000000000000 V23 0x0000000000000000 0000000000000000 V24 0x0000000000000000 0000000000000000 V25 0x0000000000000000 0000000000000000 V26 0x0000000000000000 0000000000000000 V27 0x0000000000000000 0000000000000000 V28 0x0000000000000000 0000000000000000 V29 0x0000000000000000 0000000000000000 V30 0x0000000000000000 0000000000000000 V31 0x0000000000000000 0000000000000000 SP 0x00000000476E4780 ELR 0x000000037FD3C0A8 SPSR 0x80000205 FPSR 0x00000000 ESR 0x86000006 FAR 0x000000037FD3C0A8 ESR : EC 0x21 IL 0x1 ISS 0x00000006 Instruction abort: Translation fault, second level Stack dump: 00000476E4680: 0000000000000001 0000000000000004 00000000476E4700 00000000476F3980 00000476E46A0: 000000037FD40CBD 0000000000000003 000000037FC00000 000000037FD39000 00000476E46C0: 0060000000000400 FF9F000000000B3F 00000000476E4780 000000037FD3BE70 00000476E46E0: 000000037FC00000 0000000000000002 000000037F106000 000000037F10B000 00000476E4700: 0000000000000FF0 00000000001FFFFF 000000037FD39000 000000037F106000 00000476E4720: 0000000000000003 000000037F10BFF0 0060000000000400 FF9F000000000B3F 00000476E4740: 000000037FD39000 000000037FD39000 00000000476E4780 0060000000000403 00000476E4760: 0000000C00000001 000000037FD3F90E 0000000000000400 000000037F10B000 > 00000476E4780: 00000000476E4830 000000037FD3BE70 000000037CB40000 0000000= 000000001 00000476E47A0: 000000037F10B000 0000000047FFE000 0000000000000068 000000003FFFFFFF 00000476E47C0: 000000037FD39000 000000037F10C528 0000000000000002 0000000047FFE068 00000476E47E0: 0060000000000400 FF9F000000000B3F 0000000300000001 000000037FD39000 00000476E4800: 000000017FD40CBD 0060000000000401 0000001500000001 000000037FD3F90E 00000476E4820: 0060000000000400 000000037F106000 00000000476E48E0 000000037FD3BE70 00000476E4840: 000000037CB40000 0000000000000000 0000000047FFE000 0000000047FFF000 00000476E4860: 0000000000000000 0000007FFFFFFFFF 000000037FD39000 000000037F10C528 ASSERT [ArmCpuDxe] /root/edk2/ArmPkg/Library/DefaultExceptionHandlerLib/AArch64/DefaultExcepti= onHandler.c(333): ((BOOLEAN)(0=3D=3D1)) The full log is available here: https://gitlab.com/osteffen/thunderx2-debug/-/raw/main/2023-05-19/85.log?in= line=3Dfalse Debug files, firmware binaries, and the full build tree are here: https://gitlab.com/osteffen/thunderx2-debug/-/tree/main/2023-05-19 I am able to reproduce this quickly, so any ideas for what I can try are welcome :-) Thanks -Oliver