From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by mx.groups.io with SMTP id smtpd.web10.195031.1673959680373346318 for ; Tue, 17 Jan 2023 04:48:00 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@redhat.com header.s=mimecast20190719 header.b=QnyvtUin; spf=pass (domain: redhat.com, ip: 170.10.129.124, mailfrom: osteffen@redhat.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1673959679; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=Uob8NHu1ZiOJP880QC7FiPPGjbvL96g3xXn0K+sBFKA=; b=QnyvtUinOPiP5UMbVZ+oPI+Ef7hLCTG/tKxCbAS7EiEbtl08mjoSDQKsR3LOkmeF5BU/xW 7SH/BWHu731PwckpsbPiqVGF3jGdCT7dlCwvxhNp+n0ZHCa/X4AP4iwJNtwdDXFAD0gcbq Nk4lgWQ5vY+YgSNjeNRvhhjFqiqygz8= Received: from mail-lf1-f72.google.com (mail-lf1-f72.google.com [209.85.167.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-151-XWPbqe4RPIaoxUYpK02wvA-1; Tue, 17 Jan 2023 07:47:58 -0500 X-MC-Unique: XWPbqe4RPIaoxUYpK02wvA-1 Received: by mail-lf1-f72.google.com with SMTP id g28-20020a0565123b9c00b004cc8a085997so8034328lfv.13 for ; Tue, 17 Jan 2023 04:47:58 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=Uob8NHu1ZiOJP880QC7FiPPGjbvL96g3xXn0K+sBFKA=; b=hYHI7Ci++azWFE2Aw05APCJq6eFi75eeJeanl7/w7SP2w5M1ClkTPjK+ZFcYELzbyP nl1aww2ivKcSJmczCBgfUMPSkrU/iD9K3s+7p97Lz61UOYggFUzvg+7VUQ645X/LMb8S lRaxCtBJwSu4atqvFGG9fDtCfbKc71HRNalgDw3PQlDRnj0OX9hCwBGbf+4XvAD3OoOL H3B5Pu3oxkBQXJh+GKZnHoYM/wwDEWQJGgY6crvIkQg6WbkAlu3CcLCVn1sYqA8d9TJP 5y43WzVDxaYddzHUszSDKMQaJYhcy/WFjCQn8OoXhBNFO2R2b8Wy4hLC2CneVW/SNs9V mv4A== X-Gm-Message-State: AFqh2kplYgRu5HS/TvmkpZWaBPXVmy0YTwXLuwGsm5DWnYY/uCUXclR0 NB09f+q6kNZlcaiyjoQ/ThKoN+82V6IxPP/dooN5/vkvopKFJWBlUPWWvMSSrmvgchq11RCVags FGzTxp8AeVu8J++nQo0CNCXk+IrQiQQ== X-Received: by 2002:a2e:8750:0:b0:28b:6377:219a with SMTP id q16-20020a2e8750000000b0028b6377219amr273226ljj.455.1673959676034; Tue, 17 Jan 2023 04:47:56 -0800 (PST) X-Google-Smtp-Source: AMrXdXvGuSclFgB2+gGeOQAyoDHohplbW7Cj2gG2KszpcZWb9kwEptfL6uEA8yhzBPmTlOLsAUu/C6Y+U9I4uNfNMQE= X-Received: by 2002:a2e:8750:0:b0:28b:6377:219a with SMTP id q16-20020a2e8750000000b0028b6377219amr273220ljj.455.1673959675599; Tue, 17 Jan 2023 04:47:55 -0800 (PST) MIME-Version: 1.0 References: <20230105162528.1430368-1-ardb@kernel.org> <20230105162528.1430368-2-ardb@kernel.org> In-Reply-To: From: "Oliver Steffen" Date: Tue, 17 Jan 2023 13:47:44 +0100 Message-ID: Subject: Re: [edk2-devel] [PATCH v2 2/2] ArmVirtPkg/ArmVirtQemu: Avoid early ID map on ThunderX To: devel@edk2.groups.io, dann.frazier@canonical.com Cc: Ard Biesheuvel X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: multipart/alternative; boundary="000000000000d53afe05f2751bcd" --000000000000d53afe05f2751bcd Content-Type: text/plain; charset="UTF-8" Hi Ard, Hi everyone, Thanks for the work! But somehow this patch (as it was merged into master branch) does not work for me on the ThunderX box we have. Any idea what could be wrong? I enabled the erratum during build ;-) CPU Info: # lscpu Architecture: aarch64 CPU op-mode(s): 64-bit Byte Order: Little Endian CPU(s): 224 On-line CPU(s) list: 0-223 Vendor ID: Cavium BIOS Vendor ID: Cavium Inc. Model name: ThunderX2 99xx BIOS Model name: Cavium ThunderX2(R) CPU CN9975 v2.2 @ 2.0GHz Model: 2 Thread(s) per core: 4 Core(s) per socket: 28 Socket(s): 2 Stepping: 0x1 BogoMIPS: 400.00 Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics cpuid asimdrdm Caches (sum of all): L1d: 1.8 MiB (56 instances) L1i: 1.8 MiB (56 instances) L2: 14 MiB (56 instances) L3: 64 MiB (2 instances) [...] Thanks a lot! - Oliver On Tue, Jan 10, 2023 at 1:08 AM dann frazier wrote: > On Thu, Jan 05, 2023 at 05:25:28PM +0100, Ard Biesheuvel wrote: > > The early ID map used by ArmVirtQemu uses ASID scoped non-global > > mappings, as this allows us to switch to the permanent ID map seamlessly > > without the need for explicit TLB maintenance. > > > > However, this triggers a known erratum on ThunderX, which does not > > tolerate non-global mappings that are executable at EL1, as this appears > > to result in I-cache corruption. (Linux disables the KPTI based Meltdown > > mitigation on ThunderX for the same reason) > > > > So work around this, by detecting the CPU implementor and part number, > > and proceeding without the early ID map if a ThunderX CPU is detected. > > > > Note that this requires the C code to be built with strict alignment > > again, as we may end up executing it with the MMU and caches off. > > > > Signed-off-by: Ard Biesheuvel > > --- > > ArmVirtPkg/ArmVirtQemu.dsc | 5 > +++++ > > ArmVirtPkg/Library/ArmPlatformLibQemu/AArch64/ArmPlatformHelper.S | 15 > +++++++++++++++ > > 2 files changed, 20 insertions(+) > > FTR, this v2 series also worked for me. > > -dann > > > diff --git a/ArmVirtPkg/ArmVirtQemu.dsc b/ArmVirtPkg/ArmVirtQemu.dsc > > index f77443229e8e..5dd8b6104cca 100644 > > --- a/ArmVirtPkg/ArmVirtQemu.dsc > > +++ b/ArmVirtPkg/ArmVirtQemu.dsc > > @@ -31,6 +31,7 @@ [Defines] > > DEFINE SECURE_BOOT_ENABLE = FALSE > > DEFINE TPM2_ENABLE = FALSE > > DEFINE TPM2_CONFIG_ENABLE = FALSE > > + DEFINE CAVIUM_ERRATUM_27456 = FALSE > > > > # > > # Network definition > > @@ -117,7 +118,11 @@ [LibraryClasses.common.UEFI_DRIVER] > > UefiScsiLib|MdePkg/Library/UefiScsiLib/UefiScsiLib.inf > > > > [BuildOptions] > > +!if $(CAVIUM_ERRATUM_27456) == TRUE > > + GCC:*_*_AARCH64_PP_FLAGS = -DCAVIUM_ERRATUM_27456 > > +!else > > GCC:*_*_AARCH64_CC_XIPFLAGS == > > +!endif > > > > !include NetworkPkg/NetworkBuildOptions.dsc.inc > > > > diff --git > a/ArmVirtPkg/Library/ArmPlatformLibQemu/AArch64/ArmPlatformHelper.S > b/ArmVirtPkg/Library/ArmPlatformLibQemu/AArch64/ArmPlatformHelper.S > > index 1787d52fbf51..5ac7c732f6ec 100644 > > --- a/ArmVirtPkg/Library/ArmPlatformLibQemu/AArch64/ArmPlatformHelper.S > > +++ b/ArmVirtPkg/Library/ArmPlatformLibQemu/AArch64/ArmPlatformHelper.S > > @@ -42,6 +42,21 @@ > > > > > > ASM_FUNC(ArmPlatformPeiBootAction) > > +#ifdef CAVIUM_ERRATUM_27456 > > + /* > > + * On Cavium ThunderX, using non-global mappings that are executable > at EL1 > > + * results in I-cache corruption. So just avoid the early ID mapping > there. > > + * > > + * MIDR implementor 0x43 > > + * MIDR part numbers 0xA1 0xA2 (but not 0xAF) > > + */ > > + mrs x0, midr_el1 // read the MIDR into X0 > > + ubfx x1, x0, #24, #8 // grab implementor id > > + ubfx x0, x0, #7, #9 // grab part number bits [11:3] > > + cmp x1, #0x43 // compare implementor id > > + ccmp x0, #0xA0 >> 3, #0, eq // compare part# bits [11:3] > > + b.eq 0f > > +#endif > > mrs x0, CurrentEL // check current exception level > > tbnz x0, #3, 0f // omit early ID map if above EL1 > > > > > > > > --000000000000d53afe05f2751bcd Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi = Ard, Hi everyone,

Thanks = for the work!
<= br>
But somehow= this patch (as it was merged into master branch) does not
work for me on the ThunderX bo= x we have.

=
Any idea what = could be wrong?
I enabled the erratum during build ;-)

CPU Info:
# lscpu
Architecture: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= aarch64
=C2=A0 CPU op-mode(s): =C2=A0 =C2=A0 =C2=A0 64-bit
=C2=A0 By= te Order: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Little Endian
CPU(s): =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 224
=C2=A0 On-line = CPU(s) list: =C2=A00-223
Vendor ID: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0Cavium
=C2=A0 BIOS Vendor ID: =C2=A0 =C2=A0 =C2=A0 Cavium I= nc.
=C2=A0 Model name: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 ThunderX2 99xx=
=C2=A0 =C2=A0 BIOS Model name: =C2=A0 =C2=A0Cavium ThunderX2(R) CPU CN9= 975 v2.2 @ 2.0GHz
=C2=A0 =C2=A0 Model: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A02
=C2=A0 =C2=A0 Thread(s) per core: 4
=C2=A0 =C2=A0 = Core(s) per socket: 28
=C2=A0 =C2=A0 Socket(s): =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A02
=C2=A0 =C2=A0 Stepping: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 0= x1
=C2=A0 =C2=A0 BogoMIPS: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 400.00
= =C2=A0 =C2=A0 Flags: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0fp asi= md evtstrm aes pmull sha1 sha2 crc32 atomics cpuid asimdrdm
Caches (sum = of all):
=C2=A0 L1d: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A01.8 MiB (56 instances)
=C2=A0 L1i: =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A01.8 MiB (56 instances)
=C2=A0 L2: = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 14 MiB (56 i= nstances)
=C2=A0 L3: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 64 MiB (2 instances)
[...]

Thanks a lot!
- Oliver


On Tue, Jan 10, 2023= at 1:08 AM dann frazier <dann.frazier@canonical.com> wrote:
On Thu, Jan 05, 2023 at 05:25:2= 8PM +0100, Ard Biesheuvel wrote:
> The early ID map used by ArmVirtQemu uses ASID scoped non-global
> mappings, as this allows us to switch to the permanent ID map seamless= ly
> without the need for explicit TLB maintenance.
>
> However, this triggers a known erratum on ThunderX, which does not
> tolerate non-global mappings that are executable at EL1, as this appea= rs
> to result in I-cache corruption. (Linux disables the KPTI based Meltdo= wn
> mitigation on ThunderX for the same reason)
>
> So work around this, by detecting the CPU implementor and part number,=
> and proceeding without the early ID map if a ThunderX CPU is detected.=
>
> Note that this requires the C code to be built with strict alignment > again, as we may end up executing it with the MMU and caches off.
>
> Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
> ---
>=C2=A0 ArmVirtPkg/ArmVirtQemu.dsc=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 |=C2=A0 5 +++++
>=C2=A0 ArmVirtPkg/Library/ArmPlatformLibQemu/AArch64/ArmPlatformHelper.= S | 15 +++++++++++++++
>=C2=A0 2 files changed, 20 insertions(+)

FTR, this v2 series also worked for me.

=C2=A0 -dann

> diff --git a/ArmVirtPkg/ArmVirtQemu.dsc b/ArmVirtPkg/ArmVirtQemu.dsc > index f77443229e8e..5dd8b6104cca 100644
> --- a/ArmVirtPkg/ArmVirtQemu.dsc
> +++ b/ArmVirtPkg/ArmVirtQemu.dsc
> @@ -31,6 +31,7 @@ [Defines]
>=C2=A0 =C2=A0 DEFINE SECURE_BOOT_ENABLE=C2=A0 =C2=A0 =C2=A0 =3D FALSE >=C2=A0 =C2=A0 DEFINE TPM2_ENABLE=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0=3D FALSE
>=C2=A0 =C2=A0 DEFINE TPM2_CONFIG_ENABLE=C2=A0 =C2=A0 =C2=A0 =3D FALSE > +=C2=A0 DEFINE CAVIUM_ERRATUM_27456=C2=A0 =C2=A0 =3D FALSE
>=C2=A0
>=C2=A0 =C2=A0 #
>=C2=A0 =C2=A0 # Network definition
> @@ -117,7 +118,11 @@ [LibraryClasses.common.UEFI_DRIVER]
>=C2=A0 =C2=A0 UefiScsiLib|MdePkg/Library/UefiScsiLib/UefiScsiLib.inf >=C2=A0
>=C2=A0 [BuildOptions]
> +!if $(CAVIUM_ERRATUM_27456) =3D=3D TRUE
> +=C2=A0 GCC:*_*_AARCH64_PP_FLAGS =3D -DCAVIUM_ERRATUM_27456
> +!else
>=C2=A0 =C2=A0 GCC:*_*_AARCH64_CC_XIPFLAGS =3D=3D
> +!endif
>=C2=A0
>=C2=A0 !include NetworkPkg/NetworkBuildOptions.dsc.inc
>=C2=A0
> diff --git a/ArmVirtPkg/Library/ArmPlatformLibQemu/AArch64/ArmPlatform= Helper.S b/ArmVirtPkg/Library/ArmPlatformLibQemu/AArch64/ArmPlatformHelper.= S
> index 1787d52fbf51..5ac7c732f6ec 100644
> --- a/ArmVirtPkg/Library/ArmPlatformLibQemu/AArch64/ArmPlatformHelper.= S
> +++ b/ArmVirtPkg/Library/ArmPlatformLibQemu/AArch64/ArmPlatformHelper.= S
> @@ -42,6 +42,21 @@
>=C2=A0
>=C2=A0
>=C2=A0 ASM_FUNC(ArmPlatformPeiBootAction)
> +#ifdef CAVIUM_ERRATUM_27456
> +=C2=A0 /*
> +=C2=A0 =C2=A0* On Cavium ThunderX, using non-global mappings that are= executable at EL1
> +=C2=A0 =C2=A0* results in I-cache corruption. So just avoid the early= ID mapping there.
> +=C2=A0 =C2=A0*
> +=C2=A0 =C2=A0* MIDR implementor=C2=A0 =C2=A00x43
> +=C2=A0 =C2=A0* MIDR part numbers=C2=A0 0xA1 0xA2 (but not 0xAF)
> +=C2=A0 =C2=A0*/
> +=C2=A0 mrs=C2=A0 =C2=A0 x0, midr_el1=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 // read the MIDR into X0
> +=C2=A0 ubfx=C2=A0 =C2=A0x1, x0, #24, #8=C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0// grab implementor id
> +=C2=A0 ubfx=C2=A0 =C2=A0x0, x0, #7, #9=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 // grab part number bits [11:3]
> +=C2=A0 cmp=C2=A0 =C2=A0 x1, #0x43=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0// compare implementor id
> +=C2=A0 ccmp=C2=A0 =C2=A0x0, #0xA0 >> 3, #0, eq=C2=A0 // compare= part# bits [11:3]
> +=C2=A0 b.eq=C2=A0 =C2=A00f
> +#endif
>=C2=A0 =C2=A0 mrs=C2=A0 =C2=A0 x0, CurrentEL=C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0// check current exception level
>=C2=A0 =C2=A0 tbnz=C2=A0 =C2=A0x0, #3, 0f=C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 // omit early ID map if above EL1
>=C2=A0





--000000000000d53afe05f2751bcd--