[PATCH 1/1] ArmPlatformPkg/PrePi: replace set/way cache ops with by-VA ones

[PATCH 1/1] ArmPlatformPkg/PrePi: replace set/way cache ops with by-VA ones

[Ard Biesheuvel]

Cache maintenance operations by set/way are only intended to be used
in the context of on/offlining a core, while it has been taken out of
the coherency domain. Any use intended to ensure that the contents of
the cache have made it to main memory is unreliable, since cacheline
migration and non-architected system caches may cause these contents
to linger elsewhere, without being visible in main memory once the
MMU and caches are disabled.

In KVM on Linux, there are horrid hacks in place to ensure that such
set/way operations are trapped, and replaced with a single by-VA
clean/invalidate of the entire guest VA space once the MMU state
changes, which can be costly, and is unnecessary if we manage the
caches a bit more carefully, and perform maintenance by virtual
address only.

So let's get rid of the call to ArmInvalidateDataCache () in the
PrePeiCore startup code, and instead, invalidate the UEFI memory
region by virtual address, which is the only memory region we will
be touching with the caches and MMU both disabled and enabled.
(This will lead to data corruption if data written with the MMU off
is shadowed by clean, stale cachelines that stick around when the
MMU is enabled again.)

Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
---
 ArmPlatformPkg/PrePi/PeiMPCore.inf  | 1 +
 ArmPlatformPkg/PrePi/PeiUniCore.inf | 1 +
 ArmPlatformPkg/PrePi/PrePi.c        | 8 +++++---
 3 files changed, 7 insertions(+), 3 deletions(-)

diff --git a/ArmPlatformPkg/PrePi/PeiMPCore.inf b/ArmPlatformPkg/PrePi/PeiMPCore.inf
index 9c5da0d42a7b..053f9fd9e616 100644
--- a/ArmPlatformPkg/PrePi/PeiMPCore.inf
+++ b/ArmPlatformPkg/PrePi/PeiMPCore.inf
@@ -37,6 +37,7 @@ [Packages]
 
 [LibraryClasses]
   BaseLib
+  CacheMaintenanceLib
   DebugLib
   DebugAgentLib
   ArmLib
diff --git a/ArmPlatformPkg/PrePi/PeiUniCore.inf b/ArmPlatformPkg/PrePi/PeiUniCore.inf
index ee9b05b25337..78d218ae09ca 100644
--- a/ArmPlatformPkg/PrePi/PeiUniCore.inf
+++ b/ArmPlatformPkg/PrePi/PeiUniCore.inf
@@ -37,6 +37,7 @@ [Packages]
 
 [LibraryClasses]
   BaseLib
+  CacheMaintenanceLib
   DebugLib
   DebugAgentLib
   ArmLib
diff --git a/ArmPlatformPkg/PrePi/PrePi.c b/ArmPlatformPkg/PrePi/PrePi.c
index 2bb144958139..254fb331733e 100644
--- a/ArmPlatformPkg/PrePi/PrePi.c
+++ b/ArmPlatformPkg/PrePi/PrePi.c
@@ -8,6 +8,7 @@
 
 #include <PiPei.h>
 
+#include <Library/CacheMaintenanceLib.h>
 #include <Library/DebugAgentLib.h>
 #include <Library/PrePiLib.h>
 #include <Library/PrintLib.h>
@@ -178,8 +179,6 @@ CEntryPoint (
 
   // Data Cache enabled on Primary core when MMU is enabled.
   ArmDisableDataCache ();
-  // Invalidate Data cache
-  ArmInvalidateDataCache ();
   // Invalidate instruction cache
   ArmInvalidateInstructionCache ();
   // Enable Instruction Caches on all cores.
@@ -200,6 +199,10 @@ CEntryPoint (
 
   // If not primary Jump to Secondary Main
   if (ArmPlatformIsPrimaryCore (MpId)) {
+
+    InvalidateDataCacheRange ((VOID *)UefiMemoryBase,
+                              FixedPcdGet32(PcdSystemMemoryUefiRegionSize));
+
     // Goto primary Main.
     PrimaryMain (UefiMemoryBase, StacksBase, StartTimeStamp);
   } else {
@@ -209,4 +212,3 @@ CEntryPoint (
   // DXE Core should always load and never return
   ASSERT (FALSE);
 }
-
-- 
2.20.1

Re: [PATCH 1/1] ArmPlatformPkg/PrePi: replace set/way cache ops with by-VA ones

[Ard Biesheuvel]

On Tue, 25 Feb 2020 at 19:28, Ard Biesheuvel <ard.biesheuvel@linaro.org> wrote:
>
> Cache maintenance operations by set/way are only intended to be used
> in the context of on/offlining a core, while it has been taken out of
> the coherency domain. Any use intended to ensure that the contents of
> the cache have made it to main memory is unreliable, since cacheline
> migration and non-architected system caches may cause these contents
> to linger elsewhere, without being visible in main memory once the
> MMU and caches are disabled.
>
> In KVM on Linux, there are horrid hacks in place to ensure that such
> set/way operations are trapped, and replaced with a single by-VA
> clean/invalidate of the entire guest VA space once the MMU state
> changes, which can be costly, and is unnecessary if we manage the
> caches a bit more carefully, and perform maintenance by virtual
> address only.
>
> So let's get rid of the call to ArmInvalidateDataCache () in the
> PrePeiCore startup code, and instead, invalidate the UEFI memory
> region by virtual address, which is the only memory region we will
> be touching with the caches and MMU both disabled and enabled.
> (This will lead to data corruption if data written with the MMU off
> is shadowed by clean, stale cachelines that stick around when the
> MMU is enabled again.)
>
> Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
> ---

Forgot to add a note that this is the *PrePi* version, not the
PrePeiCore one that I sent before.

@Pete: this might affect RPi3 and RPi4, and I am currently not able to
test it. If it's not too much trouble, I'd appreciate a Tested-by. If
not, I'll test it myself, but it may take me a while to get around to
it.

Thanks,

>  ArmPlatformPkg/PrePi/PeiMPCore.inf  | 1 +
>  ArmPlatformPkg/PrePi/PeiUniCore.inf | 1 +
>  ArmPlatformPkg/PrePi/PrePi.c        | 8 +++++---
>  3 files changed, 7 insertions(+), 3 deletions(-)
>
> diff --git a/ArmPlatformPkg/PrePi/PeiMPCore.inf b/ArmPlatformPkg/PrePi/PeiMPCore.inf
> index 9c5da0d42a7b..053f9fd9e616 100644
> --- a/ArmPlatformPkg/PrePi/PeiMPCore.inf
> +++ b/ArmPlatformPkg/PrePi/PeiMPCore.inf
> @@ -37,6 +37,7 @@ [Packages]
>
>  [LibraryClasses]
>    BaseLib
> +  CacheMaintenanceLib
>    DebugLib
>    DebugAgentLib
>    ArmLib
> diff --git a/ArmPlatformPkg/PrePi/PeiUniCore.inf b/ArmPlatformPkg/PrePi/PeiUniCore.inf
> index ee9b05b25337..78d218ae09ca 100644
> --- a/ArmPlatformPkg/PrePi/PeiUniCore.inf
> +++ b/ArmPlatformPkg/PrePi/PeiUniCore.inf
> @@ -37,6 +37,7 @@ [Packages]
>
>  [LibraryClasses]
>    BaseLib
> +  CacheMaintenanceLib
>    DebugLib
>    DebugAgentLib
>    ArmLib
> diff --git a/ArmPlatformPkg/PrePi/PrePi.c b/ArmPlatformPkg/PrePi/PrePi.c
> index 2bb144958139..254fb331733e 100644
> --- a/ArmPlatformPkg/PrePi/PrePi.c
> +++ b/ArmPlatformPkg/PrePi/PrePi.c
> @@ -8,6 +8,7 @@
>
>  #include <PiPei.h>
>
> +#include <Library/CacheMaintenanceLib.h>
>  #include <Library/DebugAgentLib.h>
>  #include <Library/PrePiLib.h>
>  #include <Library/PrintLib.h>
> @@ -178,8 +179,6 @@ CEntryPoint (
>
>    // Data Cache enabled on Primary core when MMU is enabled.
>    ArmDisableDataCache ();
> -  // Invalidate Data cache
> -  ArmInvalidateDataCache ();
>    // Invalidate instruction cache
>    ArmInvalidateInstructionCache ();
>    // Enable Instruction Caches on all cores.
> @@ -200,6 +199,10 @@ CEntryPoint (
>
>    // If not primary Jump to Secondary Main
>    if (ArmPlatformIsPrimaryCore (MpId)) {
> +
> +    InvalidateDataCacheRange ((VOID *)UefiMemoryBase,
> +                              FixedPcdGet32(PcdSystemMemoryUefiRegionSize));
> +
>      // Goto primary Main.
>      PrimaryMain (UefiMemoryBase, StacksBase, StartTimeStamp);
>    } else {
> @@ -209,4 +212,3 @@ CEntryPoint (
>    // DXE Core should always load and never return
>    ASSERT (FALSE);
>  }
> -
> --
> 2.20.1
>

Re: [PATCH 1/1] ArmPlatformPkg/PrePi: replace set/way cache ops with by-VA ones

[Ard Biesheuvel]

On Tue, 25 Feb 2020 at 19:31, Ard Biesheuvel <ard.biesheuvel@linaro.org> wrote:
>
> On Tue, 25 Feb 2020 at 19:28, Ard Biesheuvel <ard.biesheuvel@linaro.org> wrote:
> >
> > Cache maintenance operations by set/way are only intended to be used
> > in the context of on/offlining a core, while it has been taken out of
> > the coherency domain. Any use intended to ensure that the contents of
> > the cache have made it to main memory is unreliable, since cacheline
> > migration and non-architected system caches may cause these contents
> > to linger elsewhere, without being visible in main memory once the
> > MMU and caches are disabled.
> >
> > In KVM on Linux, there are horrid hacks in place to ensure that such
> > set/way operations are trapped, and replaced with a single by-VA
> > clean/invalidate of the entire guest VA space once the MMU state
> > changes, which can be costly, and is unnecessary if we manage the
> > caches a bit more carefully, and perform maintenance by virtual
> > address only.
> >
> > So let's get rid of the call to ArmInvalidateDataCache () in the
> > PrePeiCore startup code, and instead, invalidate the UEFI memory
> > region by virtual address, which is the only memory region we will
> > be touching with the caches and MMU both disabled and enabled.
> > (This will lead to data corruption if data written with the MMU off
> > is shadowed by clean, stale cachelines that stick around when the
> > MMU is enabled again.)
> >
> > Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
> > ---
>
> Forgot to add a note that this is the *PrePi* version, not the
> PrePeiCore one that I sent before.
>
> @Pete: this might affect RPi3 and RPi4, and I am currently not able to
> test it. If it's not too much trouble, I'd appreciate a Tested-by. If
> not, I'll test it myself, but it may take me a while to get around to
> it.
>

Actually, I decided to dedicate some of my evening to have a go at this myself.

This patch works for me on RPi4, as well as the BCM GENET with the
latest changes, and the version of the driver that turned up in
net-next today

I am getting around 100 - 150 Mbit/s - is that what I should expect?

Re: [PATCH 1/1] ArmPlatformPkg/PrePi: replace set/way cache ops with by-VA ones

[Pete Batard]

Hi Ard,

On 2020.02.25 22:27, Ard Biesheuvel wrote:
> On Tue, 25 Feb 2020 at 19:31, Ard Biesheuvel <ard.biesheuvel@linaro.org> wrote:
>>
>> On Tue, 25 Feb 2020 at 19:28, Ard Biesheuvel <ard.biesheuvel@linaro.org> wrote:
>>>
>>> Cache maintenance operations by set/way are only intended to be used
>>> in the context of on/offlining a core, while it has been taken out of
>>> the coherency domain. Any use intended to ensure that the contents of
>>> the cache have made it to main memory is unreliable, since cacheline
>>> migration and non-architected system caches may cause these contents
>>> to linger elsewhere, without being visible in main memory once the
>>> MMU and caches are disabled.
>>>
>>> In KVM on Linux, there are horrid hacks in place to ensure that such
>>> set/way operations are trapped, and replaced with a single by-VA
>>> clean/invalidate of the entire guest VA space once the MMU state
>>> changes, which can be costly, and is unnecessary if we manage the
>>> caches a bit more carefully, and perform maintenance by virtual
>>> address only.
>>>
>>> So let's get rid of the call to ArmInvalidateDataCache () in the
>>> PrePeiCore startup code, and instead, invalidate the UEFI memory
>>> region by virtual address, which is the only memory region we will
>>> be touching with the caches and MMU both disabled and enabled.
>>> (This will lead to data corruption if data written with the MMU off
>>> is shadowed by clean, stale cachelines that stick around when the
>>> MMU is enabled again.)
>>>
>>> Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
>>> ---
>>
>> Forgot to add a note that this is the *PrePi* version, not the
>> PrePeiCore one that I sent before.
>>
>> @Pete: this might affect RPi3 and RPi4, and I am currently not able to
>> test it. If it's not too much trouble, I'd appreciate a Tested-by. If
>> not, I'll test it myself, but it may take me a while to get around to
>> it.
>>
> 
> Actually, I decided to dedicate some of my evening to have a go at this myself.
> 
> This patch works for me on RPi4, as well as the BCM GENET with the
> latest changes, and the version of the driver that turned up in
> net-next today
> 
> I am getting around 100 - 150 Mbit/s - is that what I should expect?

Not sure what device you're running your system from, because I'm easily 
reaching 700 Mbit/s when running a large Samba copy from a USB 3.0 
drive, which I hope should work well enough as a stress test.

Tested both for read and write, with and without the patch (but only on 
Pi 4). No noticeable reduction in speed as far as I can tell.

With this:

Tested-By: Pete Batard <pete@akeo.ie>

Regards,

/Pete

Re: [PATCH 1/1] ArmPlatformPkg/PrePi: replace set/way cache ops with by-VA ones

[Ard Biesheuvel]

On Wed, 26 Feb 2020 at 02:34, Pete Batard <pete@akeo.ie> wrote:
>
> Hi Ard,
>
> On 2020.02.25 22:27, Ard Biesheuvel wrote:
> > On Tue, 25 Feb 2020 at 19:31, Ard Biesheuvel <ard.biesheuvel@linaro.org> wrote:
> >>
> >> On Tue, 25 Feb 2020 at 19:28, Ard Biesheuvel <ard.biesheuvel@linaro.org> wrote:
> >>>
> >>> Cache maintenance operations by set/way are only intended to be used
> >>> in the context of on/offlining a core, while it has been taken out of
> >>> the coherency domain. Any use intended to ensure that the contents of
> >>> the cache have made it to main memory is unreliable, since cacheline
> >>> migration and non-architected system caches may cause these contents
> >>> to linger elsewhere, without being visible in main memory once the
> >>> MMU and caches are disabled.
> >>>
> >>> In KVM on Linux, there are horrid hacks in place to ensure that such
> >>> set/way operations are trapped, and replaced with a single by-VA
> >>> clean/invalidate of the entire guest VA space once the MMU state
> >>> changes, which can be costly, and is unnecessary if we manage the
> >>> caches a bit more carefully, and perform maintenance by virtual
> >>> address only.
> >>>
> >>> So let's get rid of the call to ArmInvalidateDataCache () in the
> >>> PrePeiCore startup code, and instead, invalidate the UEFI memory
> >>> region by virtual address, which is the only memory region we will
> >>> be touching with the caches and MMU both disabled and enabled.
> >>> (This will lead to data corruption if data written with the MMU off
> >>> is shadowed by clean, stale cachelines that stick around when the
> >>> MMU is enabled again.)
> >>>
> >>> Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
> >>> ---
> >>
> >> Forgot to add a note that this is the *PrePi* version, not the
> >> PrePeiCore one that I sent before.
> >>
> >> @Pete: this might affect RPi3 and RPi4, and I am currently not able to
> >> test it. If it's not too much trouble, I'd appreciate a Tested-by. If
> >> not, I'll test it myself, but it may take me a while to get around to
> >> it.
> >>
> >
> > Actually, I decided to dedicate some of my evening to have a go at this myself.
> >
> > This patch works for me on RPi4, as well as the BCM GENET with the
> > latest changes, and the version of the driver that turned up in
> > net-next today
> >
> > I am getting around 100 - 150 Mbit/s - is that what I should expect?
>
> Not sure what device you're running your system from, because I'm easily
> reaching 700 Mbit/s when running a large Samba copy from a USB 3.0
> drive, which I hope should work well enough as a stress test.
>
> Tested both for read and write, with and without the patch (but only on
> Pi 4). No noticeable reduction in speed as far as I can tell.
>
> With this:
>
> Tested-By: Pete Batard <pete@akeo.ie>
>

Thanks Pete.

I realize I implied by my answer that this patch might have any effect
on the GENET performance, but these are unrelated. I just meant to say
that this is the first time I unboxed the RPi4 since early December,
to test this patch, but also, to test GENET myself in ACPI mode for
the first time (or at all, for that matter)

This patch [if done wrong] would cause boot issue very early on, which
it apparently doesn't, so yay!

Thanks,
Ard.

Re: [edk2-devel] [PATCH 1/1] ArmPlatformPkg/PrePi: replace set/way cache ops with by-VA ones

[Laszlo Ersek]

On 02/25/20 19:28, Ard Biesheuvel wrote:
> Cache maintenance operations by set/way are only intended to be used
> in the context of on/offlining a core, while it has been taken out of
> the coherency domain. Any use intended to ensure that the contents of
> the cache have made it to main memory is unreliable, since cacheline
> migration and non-architected system caches may cause these contents
> to linger elsewhere, without being visible in main memory once the
> MMU and caches are disabled.
> 
> In KVM on Linux, there are horrid hacks in place to ensure that such
> set/way operations are trapped, and replaced with a single by-VA
> clean/invalidate of the entire guest VA space once the MMU state
> changes, which can be costly, and is unnecessary if we manage the
> caches a bit more carefully, and perform maintenance by virtual
> address only.
> 
> So let's get rid of the call to ArmInvalidateDataCache () in the
> PrePeiCore startup code, and instead, invalidate the UEFI memory
> region by virtual address, which is the only memory region we will
> be touching with the caches and MMU both disabled and enabled.
> (This will lead to data corruption if data written with the MMU off
> is shadowed by clean, stale cachelines that stick around when the
> MMU is enabled again.)
> 
> Signed-off-by: Ard Biesheuvel <ard.biesheuvel@linaro.org>
> ---
>  ArmPlatformPkg/PrePi/PeiMPCore.inf  | 1 +
>  ArmPlatformPkg/PrePi/PeiUniCore.inf | 1 +
>  ArmPlatformPkg/PrePi/PrePi.c        | 8 +++++---
>  3 files changed, 7 insertions(+), 3 deletions(-)
> 
> diff --git a/ArmPlatformPkg/PrePi/PeiMPCore.inf b/ArmPlatformPkg/PrePi/PeiMPCore.inf
> index 9c5da0d42a7b..053f9fd9e616 100644
> --- a/ArmPlatformPkg/PrePi/PeiMPCore.inf
> +++ b/ArmPlatformPkg/PrePi/PeiMPCore.inf
> @@ -37,6 +37,7 @@ [Packages]
>  
>  [LibraryClasses]
>    BaseLib
> +  CacheMaintenanceLib
>    DebugLib
>    DebugAgentLib
>    ArmLib
> diff --git a/ArmPlatformPkg/PrePi/PeiUniCore.inf b/ArmPlatformPkg/PrePi/PeiUniCore.inf
> index ee9b05b25337..78d218ae09ca 100644
> --- a/ArmPlatformPkg/PrePi/PeiUniCore.inf
> +++ b/ArmPlatformPkg/PrePi/PeiUniCore.inf
> @@ -37,6 +37,7 @@ [Packages]
>  
>  [LibraryClasses]
>    BaseLib
> +  CacheMaintenanceLib
>    DebugLib
>    DebugAgentLib
>    ArmLib
> diff --git a/ArmPlatformPkg/PrePi/PrePi.c b/ArmPlatformPkg/PrePi/PrePi.c
> index 2bb144958139..254fb331733e 100644
> --- a/ArmPlatformPkg/PrePi/PrePi.c
> +++ b/ArmPlatformPkg/PrePi/PrePi.c
> @@ -8,6 +8,7 @@
>  
>  #include <PiPei.h>
>  
> +#include <Library/CacheMaintenanceLib.h>
>  #include <Library/DebugAgentLib.h>
>  #include <Library/PrePiLib.h>
>  #include <Library/PrintLib.h>
> @@ -178,8 +179,6 @@ CEntryPoint (
>  
>    // Data Cache enabled on Primary core when MMU is enabled.
>    ArmDisableDataCache ();
> -  // Invalidate Data cache
> -  ArmInvalidateDataCache ();
>    // Invalidate instruction cache
>    ArmInvalidateInstructionCache ();
>    // Enable Instruction Caches on all cores.
> @@ -200,6 +199,10 @@ CEntryPoint (
>  
>    // If not primary Jump to Secondary Main
>    if (ArmPlatformIsPrimaryCore (MpId)) {
> +
> +    InvalidateDataCacheRange ((VOID *)UefiMemoryBase,
> +                              FixedPcdGet32(PcdSystemMemoryUefiRegionSize));
> +
>      // Goto primary Main.
>      PrimaryMain (UefiMemoryBase, StacksBase, StartTimeStamp);
>    } else {
> @@ -209,4 +212,3 @@ CEntryPoint (
>    // DXE Core should always load and never return
>    ASSERT (FALSE);
>  }
> -
> 

Acked-by: Laszlo Ersek <lersek@redhat.com>