Re: [edk2-devel] [PATCH v3 12/16] ArmVirtPkg/ArmVirtQemu: enable initial ID map at early boot

["dann frazier" <dann.frazier@canonical.com>]

On Tue, Jan 03, 2023 at 10:02:27AM +0100, Ard Biesheuvel wrote:
> On Thu, 29 Dec 2022 at 22:10, dann frazier <dann.frazier@canonical.com> wrote:
> >
> > On Mon, Sep 26, 2022 at 10:25:07AM +0200, Ard Biesheuvel wrote:
> > > Now that we have all the pieces in place, switch the AArch64 version of
> > > ArmVirtQemu to a mode where the first thing it does out of reset is
> > > enable a preliminary ID map that covers the NOR flash and sufficient
> > > DRAM to create the UEFI page tables as usual.
> > >
> > > The advantage of this is that no manipulation of memory occurs any
> > > longer before the MMU is enabled, which removes the need for explicit
> > > coherency management, which is cumbersome and bad for performance.
> > >
> > > It also means we no longer need to build all components that may execute
> > > with the MMU off (including BASE libraries) with strict alignment.
> >
> > After this switch, I'm seeing a Synchronous Exception when launching a
> > VM, though only on old Cavium ThunderX (CN88XX) systems. I used print
> > debugging to narrow it down to ArmSetTTBR0(). Initially I thought it
> > might be related to Cavium Erratum 27456, but that doesn't seem to
> > make sense because the instruction cache isn't enabled until
> > later. I tried implementing the same workaround as Linux does anyway
> > (flush caches after the setting ttbr0) without any luck.
> >
> > Any idea what is going on there?
> >
> 
> I suspect it is in fact the same erratum - the I-cache does get
> enabled almost immediately after reset, and the use of ASIDs for EL1
> mappings looks suspiciously like failure mode that required us to
> disable KPTI for ThunderX on Linux.
> 
> It is a bit disappointing to have to add workarounds for obsolete
> platforms in new code, so if I fix this, it will be gated by a -D
> build option - is that acceptable to you?

It is. fwiw, here's what I tried w/o luck:

diff --git a/ArmPkg/Library/ArmLib/AArch64/ArmLibSupport.S b/ArmPkg/Library/ArmLib/AArch64/ArmLibSupport.S
index ba0ec5682b..1a94a9782c 100644
--- a/ArmPkg/Library/ArmLib/AArch64/ArmLibSupport.S
+++ b/ArmPkg/Library/ArmLib/AArch64/ArmLibSupport.S
@@ -56,10 +56,30 @@ ASM_FUNC(ArmReadAuxCr)
 ASM_FUNC(ArmSetTTBR0)
   EL1_OR_EL2_OR_EL3(x1)
 1:msr     ttbr0_el1, x0      // Translation Table Base Reg 0 (TTBR0)
+  isb
+  nop
+  nop
+  nop
+  ic iallu
+  dsb nsh
+  isb
   b       4f
 2:msr     ttbr0_el2, x0      // Translation Table Base Reg 0 (TTBR0)
+  isb
+  nop
+  nop
+  nop
+  ic iallu
+  dsb nsh
+  isb
   b       4f
 3:msr     ttbr0_el3, x0      // Translation Table Base Reg 0 (TTBR0)
+  isb
+  nop
+  nop
+  nop
+  ic iallu
+  dsb nsh
 4:isb
   ret