From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from NAM11-DM6-obe.outbound.protection.outlook.com (NAM11-DM6-obe.outbound.protection.outlook.com [40.107.223.57]) by mx.groups.io with SMTP id smtpd.web10.11010.1645637783261247548 for ; Wed, 23 Feb 2022 09:36:23 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@nvidia.com header.s=selector2 header.b=KhLBg0we; spf=permerror, err=parse error for token &{10 18 %{i}._ip.%{h}._ehlo.%{d}._spf.vali.email}: invalid domain name (domain: nvidia.com, ip: 40.107.223.57, mailfrom: ashishsingha@nvidia.com) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=lywG2bzF87HwWuIzIzaRebZI9xFVlukQYIJc/j/C+U9zNgqugfXFJDFIZVksh62kItzA2XMImNoPrYmIEW6HMZ+vWp6pdrb+TS7Mq5786gBgoJMsfgbiVmRmWDN1lj9b1tyWRL4KzuqrILCY+Y1xEykS8EQiD/OpFTsMtu6GWH1YgBe5/LshXe4iTXygPMzOY0QF8yXNRrxAzybm5OxP555qHrCSyQPQ6RPUT59YKSyqbAoxR2u+907padfgU0mjBc4E0FtoKjKgFOKOGcIOHapYz6gl8NPd1SkLe0KFAGucbqd8tR/iceGu0nGHZHnGfZ2zfOwun6LCinqo0pW3fg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=IxzAbaXbK3RLz37bmCCCs0UTRzfDJqQIqXCFYP6xDW0=; b=AL6D1uiS6Mcz2xbIuRqND60Kdl9BnqTombE/hajsiMSxJAVXQa8CmXk5kCuyH2h+yaUUEfJyHq+P4Fm6sDHYvpuvr86midFdO4YYe3nzuLRHyKl20+wqvhngh2fcitb86Dv51wdZp7V+MG4f7sTL6wpoAC3dlkdGakC61czI6u0gI1Y0p+BpGcJD3lI+vRMWHJJn8YP2b7MKIdpvntxe7AlsWPt7BLOE4kiHiqTZhzZSOWdlkKtjGgIvUYSa5apzkZle+eIC711aPRuY+5BgnS0sHHwVsLrislPKnlx42GxEevSjgS37i3EHrfS5qtTnnMJBvJXiVJpuQR16gFnIZA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=IxzAbaXbK3RLz37bmCCCs0UTRzfDJqQIqXCFYP6xDW0=; b=KhLBg0we1Sz+GAZT2CdOcTQP2BlCPv2DLnEtRaCwuuYqNwTSwYE6YvNVeccDuqVz78XrxbXSzGxStk2pRdaXtB/TS2ykpSlFyvxzqQOveyYiiR3sC9evCncj6GvMtx+YNOJ5I2PS4fSRA7JCNipWzYV/aaavlya8lgvV3NgJI8/pcPA3hb7BlvCnEYfN9y8U27NWU1OWLz/fj4FmiERKgtZWxz03/iCSlpe2t3rm2ZzbUIIOsVK42agmTTyfpKAy4zur0407umd1Fnt0jKT7tGenj3486yCSuLw6AFexCq/+FTb0Fm4c7woyNugZ57GYkSmy88xdkuD0PczY96SI6Q== Received: from CH2PR12MB5546.namprd12.prod.outlook.com (2603:10b6:610:63::14) by DM5PR12MB4661.namprd12.prod.outlook.com (2603:10b6:4:aa::32) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5017.21; Wed, 23 Feb 2022 17:36:20 +0000 Received: from CH2PR12MB5546.namprd12.prod.outlook.com ([fe80::2003:f63d:1875:ff76]) by CH2PR12MB5546.namprd12.prod.outlook.com ([fe80::2003:f63d:1875:ff76%8]) with mapi id 15.20.4995.027; Wed, 23 Feb 2022 17:36:20 +0000 From: "Ashish Singhal" To: Marc Zyngier , Ard Biesheuvel CC: edk2-devel-groups-io , Sami Mujawar , Ard Biesheuvel , Leif Lindholm Subject: Re: [PATCH] ArmPkg: Invalidate Instruction Cache On MMU Enable Thread-Topic: [PATCH] ArmPkg: Invalidate Instruction Cache On MMU Enable Thread-Index: AQHYJ5XQvRPcrKvPP0GmL2mPrc+CGKygt3cAgAAgeICAAIPkTg== Date: Wed, 23 Feb 2022 17:36:20 +0000 Message-ID: References: <122c32bb19ed0730ef166b9f46d3b112bc9ed937.1645497637.git.ashishsingha@nvidia.com> <877d9m3qny.wl-maz@kernel.org> In-Reply-To: <877d9m3qny.wl-maz@kernel.org> Accept-Language: en-US X-Mentions: maz@kernel.org X-MS-Has-Attach: X-MS-TNEF-Correlator: suggested_attachment_session_id: 1bd2f815-91de-0522-0b7e-3501eaca2bf3 authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 3b53e392-8c04-4252-4952-08d9f6f300e5 x-ms-traffictypediagnostic: DM5PR12MB4661:EE_ x-microsoft-antispam-prvs: x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: QGw4d03Dn+kb1cOaQF7O6Sy8jZwjaVNhkwIiVIvPocbEwCer9IVI4opdEsBDenfPsQ63jPrXfQyEm0mFYv5SBGCj4qEoPccxKoZs2Tvf56gp88rm5LnPLvz82+kZQDvu4JSFnshvGSCD88b4/VgK1IxR9Diio302S59F/75kQsLLDS/sAV7ux3bkAUHZOyyAmKGal/9RQ8dBOse9QK8nrdgUo/K68FQeHhP5QnniPc31x0CY9r5pMksJqKyqPjuWNfACgDX3S1Fx3efir2LSfM0qBVikjJVhdDzg5BPdr54xPxGw0C4UZHdXXlUq8viUTsbwdnHF4v4A8bYcedW/5g9UM0QH7YM8Cc0Q3pJ94kYsiAeXns2UL0DZ0u05X+GVorpk1CXkHZvbaVIVTkZX3Gdwg04FbYcSzP1pVm5dvRQaGs8itJrO9aZR82OuasJeU9QmjKxwZS5ISj1v+ob9vfK3U+M9DS0WoCKCWKbRAMGgce2oKuUyCEEIJdU6GAOsada774D7WzL4vmCCX4ZviJBd2RhfhKv0qqaNddsDtcdwg5Vb1E0qgfwA5wu3Nvrkr6XLe3GiSq21zMbCBEN64OKoXPHiszz/siZFClXGFPLfhjrWLoMwWEnL21x3pg9lUu/nIOKUbt5R8JQ4Z+KWjn2fArB3MLKQNcatnmA2e+d5Adb5cwiyTRqKON+synET/9nSxslpLEQG2Xxn5lww+w== x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CH2PR12MB5546.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230001)(4636009)(366004)(19627235002)(38070700005)(19627405001)(316002)(71200400001)(26005)(38100700002)(110136005)(55016003)(54906003)(186003)(122000001)(2906002)(33656002)(66946007)(8936002)(8676002)(66556008)(66476007)(7696005)(66446008)(64756008)(86362001)(4326008)(9686003)(508600001)(83380400001)(52536014)(6506007)(76116006)(91956017)(5660300002)(53546011);DIR:OUT;SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?iso-8859-1?Q?hnGIfomsk1YIynxxrxJ4rdCWxUWJGDWAuZXdhoGqi/IpFDGkDAzZnlWgwx?= =?iso-8859-1?Q?5qxo8Q74AF/GOg6qZzsk6YEQbl6972c+//kwWDgfWF17D1G6PcDonhUEwM?= =?iso-8859-1?Q?GMrsr+oGGuGJnXIbzsSDWOvjY0HtQOIQ5RlInriBNmmJvXrhgxpaJcHymH?= =?iso-8859-1?Q?ky1kCg6H42lixKToSXyKjZMyHRXyiE7ECWq6lEeBIfsurDwf3YMUSHj3TM?= =?iso-8859-1?Q?MXcG5tNj4HqchozU8rHnD2CP+ZVBinB03hjZPkCv/BOnBfrJLprfNzlXuV?= =?iso-8859-1?Q?bR5aSrBlxZmfkr/pMDGEvzf0fxIWIqRFonT2lyYhMr4XA4iymcTCGp4CRK?= =?iso-8859-1?Q?3jxWT0np60djfeQjayLJHVNGMa3TFpzusju2B0VQlEl/D2HAvUyeoKY2n4?= =?iso-8859-1?Q?Oas96KOKC6nxGzUDq2lcG85Z4QciuPjvDoPsIEDylk0InWOkJrN1GUe/2a?= =?iso-8859-1?Q?/vnzuNcEuhmHk56pa2DU1nIdSy3jV3LmGhnEi8IqfKfjglAQ56NPUhd4ZU?= =?iso-8859-1?Q?026L+C81ozGvqlg/PorY8hRsHdTcwb0aWH8vTbcl629VQq+1yNBWSgl0E3?= =?iso-8859-1?Q?fneAfvDq0IwkHkkGR8w27ZcdG9xQsLYjiqpcXSSVhBYfPCqKJzBuZZlTs5?= =?iso-8859-1?Q?8SKvaXPOLXqtMRiJkdpev/6DWeznIP+JJimzBNp0XNN2qTgLOQhGhnBgYw?= =?iso-8859-1?Q?uZV/3DJ0fYqrIqs8FFecLIRaUlF0UR0Jh4n4sii/ouVsMlgdDo62H6gBqX?= =?iso-8859-1?Q?dssFlVcGfMlkw5xlecyGnZWbO8TD7AmHFEyZznX9qZ44fDwJm9SMEEzTOn?= =?iso-8859-1?Q?N+QJM0E0c1VEBphddeHepsoiVE9slVGVo/h4q/UpPFoNIIjjNxGrOZk1PE?= =?iso-8859-1?Q?nKs0lhfUcizi8sBUJSa43uNvrmLDIh7AcKRETR1ApcrsCWsMS6Ac14X438?= =?iso-8859-1?Q?WRA5iNZDAHOsVW9gI/f7W9BN9POpIr2zfNEOFguooq45dvESnq4lFRiL82?= =?iso-8859-1?Q?rmTzzTFCtau9Xi0gCOtWltIrbrCTIMaC6On43OKcz7MVydIGTz6cidKyLS?= =?iso-8859-1?Q?QyU/FcDrpj6GsPY8b5nC0WFunB3I61iaTaJskVexuVeZHnj6NsgzV8ujZa?= =?iso-8859-1?Q?568NWyfRPJgGt46TVHw2RKqITN5u1ib+tCluDLEKtvhP/4dIzysqv223FO?= =?iso-8859-1?Q?HVoQuZlPnnVcpDQ+GXX6GDr2WJ8CGoqAEpZrHFMDM8NX9DFDSl8Kn/Zs+Q?= =?iso-8859-1?Q?/p2vTJACgReJag5zBzZzjcsFe7Vm0tJYhvv88s89PJ0NQY+qHXVF2qE82z?= =?iso-8859-1?Q?KFP41wiyyHfY4Umzl3nj6q3KD+PVh/2EfM9mYYbWBfQq2Dr/iT05+iViCv?= =?iso-8859-1?Q?80rj/De1eEPrGpT/oqAzikxuG0xPIAb/bNlnNy6TtxQHxPGBGeXvZYw542?= =?iso-8859-1?Q?TEABG6PqbvMjsgM6AeSqPCdSD3brSyeE1eDy4fvPSmUkbTasbKZ1Mq9iHA?= =?iso-8859-1?Q?e7ZBhTH3LGxQ+qaxs+McEB4Obi5KLHFmNTC/vZ3r1JkBA1cXAJtIuc8wDp?= =?iso-8859-1?Q?VhXKNHdBBg6g2sZSDKfY0Lrt2t2S/SRFghUfUd95s/fdZDrAXZdtuS3aG8?= =?iso-8859-1?Q?Dr5fYKq8Ko2NATyOdvGQE7flYnZYC64norzfSrWRFle9XPbuuAr7vyCvAG?= =?iso-8859-1?Q?OLItDoGbP1IkNxDLfR8=3D?= MIME-Version: 1.0 X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: CH2PR12MB5546.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 3b53e392-8c04-4252-4952-08d9f6f300e5 X-MS-Exchange-CrossTenant-originalarrivaltime: 23 Feb 2022 17:36:20.1210 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: 8KFYYkFhb7KYzz3lWvb8uxHZ9Q+w+EQ8PBVuyzV76JWgCq/mAmlY+1vI07hTXOJhHDwO2KXOYxgWkCfsxbUYgQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5PR12MB4661 Content-Language: en-US Content-Type: multipart/alternative; boundary="_000_CH2PR12MB5546685B7807F54DFE7269F3BA3C9CH2PR12MB5546namp_" --_000_CH2PR12MB5546685B7807F54DFE7269F3BA3C9CH2PR12MB5546namp_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Hello Ard and Marc, I apologize for not providing the background on this in the commit message = and I understand the commit message is not very clear as well. Let me try t= o summarize the problem. In our UEFI implementation, we are doing the following as part of the initi= al MMU setup: 1. Set the applicable device memory as nGnRnE memory. 2. Set the whole DRAM as normal memory that translates to RW and executa= ble memory. 3. Enable caches and MMU. 4. At this time, the memory map looks correct when I check that from DS-= 5. When we start dispatching drivers, DxeCore dispatches a driver and marks it= s code area as RO and executable and its data region as RW and non-executab= le. What we are seeing randomly is that some of the page tables (using DS-5= ) have invalid output address that leads to the correct input address from = UEFI being translated to an unavailable memory location causing a crash som= etime in EL2 or sometimes as a RAS error in EL3. When I reached out to the CPU team here, they said Arm=AE Cortex=AE-A78AE i= s a highly speculative core and we need to have appropriate barriers in pla= ce so that there is consistency in the way an address is accessed especiall= y if it is done right after there is a change in translation tables. Based = on this, I started some experimentation wrt caches whenever MMUs are enable= d and I found that invalidating the instruction cache after enabling MMUs s= olves this problem. Please note that I could be wrong with my hypothesis here and I may just be= masking the issue. If that is the case, please let me know what I should b= e trying as I am out of ideas at this point. Also, the same UEFI works on N= VIDIA's Xavier Silicon that has Carmel cores but shows this issue on Orin S= ilicon that has Arm=AE Cortex=AE-A78AE v8.2 64-bit CPU. @Marc Zyngier, I agree with your concern about using "dsb sy" and we should replace that w= ith "dsb nsh" as we are running this only on a single core during boot. Thanks Ashish ________________________________ From: Marc Zyngier Sent: Wednesday, February 23, 2022 1:58 AM To: Ard Biesheuvel ; Ashish Singhal Cc: edk2-devel-groups-io ; Sami Mujawar ; Ard Biesheuvel ; Leif Lindholm Subject: Re: [PATCH] ArmPkg: Invalidate Instruction Cache On MMU Enable External email: Use caution opening links or attachments On Wed, 23 Feb 2022 07:02:28 +0000, Ard Biesheuvel wrote: > > (+ Marc) > > On Tue, 22 Feb 2022 at 03:42, Ashish Singhal wr= ote: > > > > Even with MMU turned off, instruction cache can speculate > > and fetch instructions. This can cause a crash if region > > being executed has been modified recently. With this patch, > > we ensure that instruction cache is invalidated right after > > MMU has been enabled and any potentially stale instruction > > fetched earlier has been discarded. Are you doing code patching while the MMU is off? > > > > I don't follow this reasoning. Every time the UEFI image loader loads > an image into memory, it does so with MMU and caches enabled, and the > code regions are cleaned to the PoU before being invalidated in the > I-cache. So how could these regions be stale? The only code that needs > special care here is the little trampoline that executes with the MMU > off, but this is being taken care of explicitly, by cleaning the code > to the PoC before invoking it. > > > This is specially helpful when the memory attributes of a > > region in MMU are being changed and some instructions > > operating on the region are prefetched in the instruction > > cache. > > Huh, this is way too vague. How do you expect anyone to understand your problem based on this? > > This sounds like a TLB problem not a cache problem. For the sake of > posterity, could you include a more detailed description of the issue > in the commit log? IIRC, this is about speculative instruction fetches > hitting device memory locations? > > As I mentioned before, the architecture does not permit speculative > instruction fetches for regions mapped with the XN attribute. > > Is the issue occurring under virtualization? Is there a stage 2 > mapping that lacks the XN attribute for the device memory region in > question? > > > Signed-off-by: Ashish Singhal > > --- > > ArmPkg/Library/ArmLib/AArch64/AArch64Support.S | 4 +++- > > ArmPkg/Library/ArmMmuLib/AArch64/ArmMmuLibReplaceEntry.S | 2 ++ > > 2 files changed, 5 insertions(+), 1 deletion(-) > > > > diff --git a/ArmPkg/Library/ArmLib/AArch64/AArch64Support.S b/ArmPkg/Li= brary/ArmLib/AArch64/AArch64Support.S > > index d3cc1e8671..9648245182 100644 > > --- a/ArmPkg/Library/ArmLib/AArch64/AArch64Support.S > > +++ b/ArmPkg/Library/ArmLib/AArch64/AArch64Support.S > > @@ -89,7 +89,9 @@ ASM_FUNC(ArmEnableMmu) > > dsb nsh > > isb > > msr sctlr_el3, x0 // Write back > > -4: isb > > +4: ic iallu > > + dsb sy Why DSB SY? Given that you are only invalidating on a single CPU, DSB NSH should be enough. > > + isb > > ret > > > > > > diff --git a/ArmPkg/Library/ArmMmuLib/AArch64/ArmMmuLibReplaceEntry.S b= /ArmPkg/Library/ArmMmuLib/AArch64/ArmMmuLibReplaceEntry.S > > index 66ebca571e..56cc2dd73f 100644 > > --- a/ArmPkg/Library/ArmMmuLib/AArch64/ArmMmuLibReplaceEntry.S > > +++ b/ArmPkg/Library/ArmMmuLib/AArch64/ArmMmuLibReplaceEntry.S > > @@ -37,6 +37,8 @@ > > > > // re-enable the MMU > > msr sctlr_el\el, x8 > > + ic iallu > > + dsb sy Same thing here. M. -- Without deviation from the norm, progress is not possible. --_000_CH2PR12MB5546685B7807F54DFE7269F3BA3C9CH2PR12MB5546namp_ Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable
Hello Ard and Marc,

I apologize for not providing the background on this in the commit message = and I understand the commit message is not very clear as well. Let me try t= o summarize the problem.

In our UEFI implementation, we are doing the following as part of t= he initial MMU setup:
  1. Set the applicable device memory as nGnRnE memory.
  2. Set the whole DRAM as normal memory that translates to RW and= executable memory.
  3. Enable caches and MMU.
  4. At this time, the memory map looks correct when I check that = from DS-5.
When we start dispatching dr= ivers, DxeCore dispatches a driver and marks its code area as RO and execut= able and its data region as RW and non-executable. What we are seeing randomly is that some of the page table= s (using DS-5) have invalid output address that leads to the correct input = address from UEFI being translated to an unavailable memory location causin= g a crash sometime in EL2 or sometimes as a RAS error in EL3.

When I reached out to the CPU team here, they said Arm=AE Cortex=AE-A78AE is a highly = speculative core and we need to have appropriate barriers in place so that there is consistency in the way an address is ac= cessed especially if it is done right after there is a change in translatio= n tables. Based on this, I started some experimentation wrt caches whenever= MMUs are enabled and I found that invalidating the instruction cache after enabling MMUs solves this problem= .

Please note that I could be wrong with my hypothesis here and I may just be= masking the issue. If that is the case, please let me know what I sho= uld be trying as I am out of ideas at this point. Also, the same UEFI works on NVIDIA's Xavier Silicon that has Carmel cores but shows= this issue on Orin Silicon that has Arm=AE Cortex=AE-A78AE v8.2 64-bit CPU.


I agree with your concern about using "dsb sy" and we sh= ould replace that with "dsb nsh" as we are running this only on a single core during boot.

Thanks
Ashish
From: Marc Zyngier <maz@= kernel.org>
Sent: Wednesday, February 23, 2022 1:58 AM
To: Ard Biesheuvel <ardb@kernel.org>; Ashish Singhal <ashis= hsingha@nvidia.com>
Cc: edk2-devel-groups-io <devel@edk2.groups.io>; Sami Mujawar = <sami.mujawar@arm.com>; Ard Biesheuvel <ardb+tianocore@kernel.org&= gt;; Leif Lindholm <quic_llindhol@quicinc.com>
Subject: Re: [PATCH] ArmPkg: Invalidate Instruction Cache On MMU Ena= ble
 
External email: Use caution opening links or attac= hments


On Wed, 23 Feb 2022 07:02:28 +0000,
Ard Biesheuvel <ardb@kernel.org> wrote:
>
> (+ Marc)
>
> On Tue, 22 Feb 2022 at 03:42, Ashish Singhal <ashishsingha@nvidia.c= om> wrote:
> >
> > Even with MMU turned off, instruction cache can speculate
> > and fetch instructions. This can cause a crash if region
> > being executed has been modified recently. With this patch,
> > we ensure that instruction cache is invalidated right after
> > MMU has been enabled and any potentially stale instruction
> > fetched earlier has been discarded.

Are you doing code patching while the MMU is off?

> >
>
> I don't follow this reasoning. Every time the UEFI image loader loads<= br> > an image into memory, it does so with MMU and caches enabled, and the<= br> > code regions are cleaned to the PoU before being invalidated in the > I-cache. So how could these regions be stale? The only code that needs=
> special care here is the little trampoline that executes with the MMU<= br> > off, but this is being taken care of explicitly, by cleaning the code<= br> > to the PoC before invoking it.
>
> > This is specially helpful when the memory attributes of a
> > region in MMU are being changed and some instructions
> > operating on the region are prefetched in the instruction
> > cache.
> >

Huh, this is way too vague. How do you expect anyone to understand
your problem based on this?

>
> This sounds like a TLB problem not a cache problem. For the sake of > posterity, could you include a more detailed description of the issue<= br> > in the commit log? IIRC, this is about speculative instruction fetches=
> hitting device memory locations?
>
> As I mentioned before, the architecture does not permit speculative > instruction fetches for regions mapped with the XN attribute.
>
> Is the issue occurring under virtualization? Is there a stage 2
> mapping that lacks the XN attribute for the device memory region in > question?
>
> > Signed-off-by: Ashish Singhal <ashishsingha@nvidia.com>
> > ---
> >  ArmPkg/Library/ArmLib/AArch64/AArch64Support.S  &= nbsp;        | 4 +++-
> >  ArmPkg/Library/ArmMmuLib/AArch64/ArmMmuLibReplaceEntry.S | = 2 ++
> >  2 files changed, 5 insertions(+), 1 deletion(-)
> >
> > diff --git a/ArmPkg/Library/ArmLib/AArch64/AArch64Support.S b/Arm= Pkg/Library/ArmLib/AArch64/AArch64Support.S
> > index d3cc1e8671..9648245182 100644
> > --- a/ArmPkg/Library/ArmLib/AArch64/AArch64Support.S
> > +++ b/ArmPkg/Library/ArmLib/AArch64/AArch64Support.S
> > @@ -89,7 +89,9 @@ ASM_FUNC(ArmEnableMmu)
> >     dsb     nsh
> >     isb
> >     msr     sctlr_el3, x0=        // Write back
> > -4: isb
> > +4: ic      iallu
> > +   dsb     sy

Why DSB SY? Given that you are only invalidating on a single CPU,
DSB NSH should be enough.

> > +   isb
> >     ret
> >
> >
> > diff --git a/ArmPkg/Library/ArmMmuLib/AArch64/ArmMmuLibReplaceEnt= ry.S b/ArmPkg/Library/ArmMmuLib/AArch64/ArmMmuLibReplaceEntry.S
> > index 66ebca571e..56cc2dd73f 100644
> > --- a/ArmPkg/Library/ArmMmuLib/AArch64/ArmMmuLibReplaceEntry.S > > +++ b/ArmPkg/Library/ArmMmuLib/AArch64/ArmMmuLibReplaceEntry.S > > @@ -37,6 +37,8 @@
> >
> >    // re-enable the MMU
> >    msr   sctlr_el\el, x8
> > +  ic    iallu
> > +  dsb   sy

Same thing here.

        M.

--
Without deviation from the norm, progress is not possible.
--_000_CH2PR12MB5546685B7807F54DFE7269F3BA3C9CH2PR12MB5546namp_--