Re: [PATCH 1/1] ArmPlatformPkg/PrePi: replace set/way cache ops with by-VA ones

["Ard Biesheuvel" <ard.biesheuvel@linaro.org>]

On Tue, 25 Feb 2020 at 19:28, Ard Biesheuvel <ard.biesheuvel@linaro.org> wrote:
>
> Cache maintenance operations by set/way are only intended to be used
> in the context of on/offlining a core, while it has been taken out of
> the coherency domain. Any use intended to ensure that the contents of
> the cache have made it to main memory is unreliable, since cacheline
> migration and non-architected system caches may cause these contents
> to linger elsewhere, without being visible in main memory once the
> MMU and caches are disabled.
>
> In KVM on Linux, there are horrid hacks in place to ensure that such
> set/way operations are trapped, and replaced with a single by-VA
> clean/invalidate of the entire guest VA space once the MMU state
> changes, which can be costly, and is unnecessary if we manage the
> caches a bit more carefully, and perform maintenance by virtual
> address only.
>
> So let's get rid of the call to ArmInvalidateDataCache () in the
> PrePeiCore startup code, and instead, invalidate the UEFI memory
> region by virtual address, which is the only memory region we will
> be touching with the caches and MMU both disabled and enabled.
> (This will lead to data corruption if data written with the MMU off
> is shadowed by clean, stale cachelines that stick around when the
> MMU is enabled again.)
>
> Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
> ---

Forgot to add a note that this is the *PrePi* version, not the
PrePeiCore one that I sent before.

@Pete: this might affect RPi3 and RPi4, and I am currently not able to
test it. If it's not too much trouble, I'd appreciate a Tested-by. If
not, I'll test it myself, but it may take me a while to get around to
it.

Thanks,

>  ArmPlatformPkg/PrePi/PeiMPCore.inf  | 1 +
>  ArmPlatformPkg/PrePi/PeiUniCore.inf | 1 +
>  ArmPlatformPkg/PrePi/PrePi.c        | 8 +++++---
>  3 files changed, 7 insertions(+), 3 deletions(-)
>
> diff --git a/ArmPlatformPkg/PrePi/PeiMPCore.inf b/ArmPlatformPkg/PrePi/PeiMPCore.inf
> index 9c5da0d42a7b..053f9fd9e616 100644
> --- a/ArmPlatformPkg/PrePi/PeiMPCore.inf
> +++ b/ArmPlatformPkg/PrePi/PeiMPCore.inf
> @@ -37,6 +37,7 @@ [Packages]
>
>  [LibraryClasses]
>    BaseLib
> +  CacheMaintenanceLib
>    DebugLib
>    DebugAgentLib
>    ArmLib
> diff --git a/ArmPlatformPkg/PrePi/PeiUniCore.inf b/ArmPlatformPkg/PrePi/PeiUniCore.inf
> index ee9b05b25337..78d218ae09ca 100644
> --- a/ArmPlatformPkg/PrePi/PeiUniCore.inf
> +++ b/ArmPlatformPkg/PrePi/PeiUniCore.inf
> @@ -37,6 +37,7 @@ [Packages]
>
>  [LibraryClasses]
>    BaseLib
> +  CacheMaintenanceLib
>    DebugLib
>    DebugAgentLib
>    ArmLib
> diff --git a/ArmPlatformPkg/PrePi/PrePi.c b/ArmPlatformPkg/PrePi/PrePi.c
> index 2bb144958139..254fb331733e 100644
> --- a/ArmPlatformPkg/PrePi/PrePi.c
> +++ b/ArmPlatformPkg/PrePi/PrePi.c
> @@ -8,6 +8,7 @@
>
>  #include <PiPei.h>
>
> +#include <Library/CacheMaintenanceLib.h>
>  #include <Library/DebugAgentLib.h>
>  #include <Library/PrePiLib.h>
>  #include <Library/PrintLib.h>
> @@ -178,8 +179,6 @@ CEntryPoint (
>
>    // Data Cache enabled on Primary core when MMU is enabled.
>    ArmDisableDataCache ();
> -  // Invalidate Data cache
> -  ArmInvalidateDataCache ();
>    // Invalidate instruction cache
>    ArmInvalidateInstructionCache ();
>    // Enable Instruction Caches on all cores.
> @@ -200,6 +199,10 @@ CEntryPoint (
>
>    // If not primary Jump to Secondary Main
>    if (ArmPlatformIsPrimaryCore (MpId)) {
> +
> +    InvalidateDataCacheRange ((VOID *)UefiMemoryBase,
> +                              FixedPcdGet32(PcdSystemMemoryUefiRegionSize));
> +
>      // Goto primary Main.
>      PrimaryMain (UefiMemoryBase, StacksBase, StartTimeStamp);
>    } else {
> @@ -209,4 +212,3 @@ CEntryPoint (
>    // DXE Core should always load and never return
>    ASSERT (FALSE);
>  }
> -
> --
> 2.20.1
>