Hi Ard, Hi everyone,

Thanks for the work!

But somehow this patch (as it was merged into master branch) does not

work for me on the ThunderX box we have.

Any idea what could be wrong?

I enabled the erratum during build ;-)

CPU Info:

# lscpu
Architecture: aarch64
CPU op-mode(s): 64-bit
Byte Order: Little Endian
CPU(s): 224
On-line CPU(s) list: 0-223
Vendor ID: Cavium
BIOS Vendor ID: Cavium Inc.
Model name: ThunderX2 99xx
BIOS Model name: Cavium ThunderX2(R) CPU CN9975 v2.2 @ 2.0GHz
Model: 2
Thread(s) per core: 4
Core(s) per socket: 28
Socket(s): 2
Stepping: 0x1
BogoMIPS: 400.00
Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics cpuid asimdrdm
Caches (sum of all):
L1d: 1.8 MiB (56 instances)
L1i: 1.8 MiB (56 instances)
L2: 14 MiB (56 instances)
L3: 64 MiB (2 instances)

[...]

Thanks a lot!

- Oliver

On Tue, Jan 10, 2023 at 1:08 AM dann frazier <dann.frazier@canonical.com> wrote:

On Thu, Jan 05, 2023 at 05:25:28PM +0100, Ard Biesheuvel wrote:
> The early ID map used by ArmVirtQemu uses ASID scoped non-global
> mappings, as this allows us to switch to the permanent ID map seamlessly
> without the need for explicit TLB maintenance.
>
> However, this triggers a known erratum on ThunderX, which does not
> tolerate non-global mappings that are executable at EL1, as this appears
> to result in I-cache corruption. (Linux disables the KPTI based Meltdown
> mitigation on ThunderX for the same reason)
>
> So work around this, by detecting the CPU implementor and part number,
> and proceeding without the early ID map if a ThunderX CPU is detected.
>
> Note that this requires the C code to be built with strict alignment
> again, as we may end up executing it with the MMU and caches off.
>
> Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
> ---
> ArmVirtPkg/ArmVirtQemu.dsc | 5 +++++
> ArmVirtPkg/Library/ArmPlatformLibQemu/AArch64/ArmPlatformHelper.S | 15 +++++++++++++++
> 2 files changed, 20 insertions(+)

FTR, this v2 series also worked for me.

-dann

> diff --git a/ArmVirtPkg/ArmVirtQemu.dsc b/ArmVirtPkg/ArmVirtQemu.dsc
> index f77443229e8e..5dd8b6104cca 100644
> --- a/ArmVirtPkg/ArmVirtQemu.dsc
> +++ b/ArmVirtPkg/ArmVirtQemu.dsc
> @@ -31,6 +31,7 @@ [Defines]
> DEFINE SECURE_BOOT_ENABLE = FALSE
> DEFINE TPM2_ENABLE = FALSE
> DEFINE TPM2_CONFIG_ENABLE = FALSE
> + DEFINE CAVIUM_ERRATUM_27456 = FALSE
>
> #
> # Network definition
> @@ -117,7 +118,11 @@ [LibraryClasses.common.UEFI_DRIVER]
> UefiScsiLib|MdePkg/Library/UefiScsiLib/UefiScsiLib.inf
>
> [BuildOptions]
> +!if $(CAVIUM_ERRATUM_27456) == TRUE
> + GCC:*_*_AARCH64_PP_FLAGS = -DCAVIUM_ERRATUM_27456
> +!else
> GCC:*_*_AARCH64_CC_XIPFLAGS ==
> +!endif
>
> !include NetworkPkg/NetworkBuildOptions.dsc.inc
>
> diff --git a/ArmVirtPkg/Library/ArmPlatformLibQemu/AArch64/ArmPlatformHelper.S b/ArmVirtPkg/Library/ArmPlatformLibQemu/AArch64/ArmPlatformHelper.S
> index 1787d52fbf51..5ac7c732f6ec 100644
> --- a/ArmVirtPkg/Library/ArmPlatformLibQemu/AArch64/ArmPlatformHelper.S
> +++ b/ArmVirtPkg/Library/ArmPlatformLibQemu/AArch64/ArmPlatformHelper.S
> @@ -42,6 +42,21 @@
>
>
> ASM_FUNC(ArmPlatformPeiBootAction)
> +#ifdef CAVIUM_ERRATUM_27456
> + /*
> + * On Cavium ThunderX, using non-global mappings that are executable at EL1
> + * results in I-cache corruption. So just avoid the early ID mapping there.
> + *
> + * MIDR implementor 0x43
> + * MIDR part numbers 0xA1 0xA2 (but not 0xAF)
> + */
> + mrs x0, midr_el1 // read the MIDR into X0
> + ubfx x1, x0, #24, #8 // grab implementor id
> + ubfx x0, x0, #7, #9 // grab part number bits [11:3]
> + cmp x1, #0x43 // compare implementor id
> + ccmp x0, #0xA0 >> 3, #0, eq // compare part# bits [11:3]
> + b.eq 0f
> +#endif
> mrs x0, CurrentEL // check current exception level
> tbnz x0, #3, 0f // omit early ID map if above EL1
>