From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from NAM11-CO1-obe.outbound.protection.outlook.com (NAM11-CO1-obe.outbound.protection.outlook.com [40.107.220.53]) by mx.groups.io with SMTP id smtpd.web10.11544.1645640039875669377 for ; Wed, 23 Feb 2022 10:14:00 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@nvidia.com header.s=selector2 header.b=Tk9xXrw9; spf=permerror, err=parse error for token &{10 18 %{i}._ip.%{h}._ehlo.%{d}._spf.vali.email}: invalid domain name (domain: nvidia.com, ip: 40.107.220.53, mailfrom: ashishsingha@nvidia.com) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=csQa9yGHLrcOEuPLQMcQICg9i+M/3wacFsBZI/LhsKZWGrs1fmYSF6r7kmhh7i7mvlyMHwWEcqLzcxKgBTUanatuZGE6Xo26+oQS10v9hLiJZ/jd7e7gi6WRm46T1RbkSntzhRLRqBm3Fg3HTeJ9yrQf4rSn8BhFPDpst10dKnSffR/m+mOSJpC6rXLlumrhF7j5kCePJfDozrNcJuwhoCrR2WBrFafyWBu+y81rtZ14P3JorAFYEMvdWf3f0f/VASb9SvjWE3VWetowEDfSaqEZus0wA/iVhcEGX+hgQkdUtuWxCl85e2vVeRffVoVF6P0LtFp3yU6MmeLc1lbVpA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Qg4vndkZeNW1mVPqI9jJET16EABZ0FEwmHVlE7TC84k=; b=J9spdZy0vKc0fXl21Dn1Yt/RWAWrdIm6757zX2pBNz+Sv1V0Ww4jWKUMSJS2+4na9bWNwJHQbuqveFmLbp6pMZ/mVwyPxY5fp9M8k/YGpPliVvr+DIFv0cLNzQXv5BTugdkGPeOaj3zn8Sq4v2+QLA2AHGvkh0aWyyRpk1wPc8pytw6c4a1asob9eCKCp00XjKepjT226jvkGxUKhGig2nUNJch1vKjLvtzR6TqD2y3SWLbs0sOWu4fMo/QOPuKP/bL1eqyWfcKEC97rH4arZgeootMLR9FGCae9x497N8m6NJ7iuPhfnz6/rg/XQKhbppt3j7KuHfXnMFhQugEQ3Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Qg4vndkZeNW1mVPqI9jJET16EABZ0FEwmHVlE7TC84k=; b=Tk9xXrw9L5lEbyc3dj88iUHa6laEP2UJhtqQmEECPKvIeZV0o8O0ZY3qdvMJU5paSZ02RZG+VZcSV6usQVzHjKtfkmF4vdmwViM68HiwfLFXGJ9PQni1auU2QTtsKvGNFVjC3dmJIV2XCNZyFuLzS5QgmpyrHehwWe9KGAufVqTqwrQG45eQaO2vXJNXF40edwDC0QxqECYdz7VSwkaMFSBU9S3Cm5awqjn3T5U7QfrRAN0tYSSyPnhtfAt6Z9VxzBvvojpgdaZcu56RfstIKXYl7zawM/l+BIgIg1RwZkffrcpoF1mlVSayNZQKKvLPT3fCkSJMZDbvhgX1k5TQvQ== Received: from CH2PR12MB5546.namprd12.prod.outlook.com (2603:10b6:610:63::14) by BN7PR12MB2722.namprd12.prod.outlook.com (2603:10b6:408:2e::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5017.21; Wed, 23 Feb 2022 18:13:57 +0000 Received: from CH2PR12MB5546.namprd12.prod.outlook.com ([fe80::2003:f63d:1875:ff76]) by CH2PR12MB5546.namprd12.prod.outlook.com ([fe80::2003:f63d:1875:ff76%8]) with mapi id 15.20.4995.027; Wed, 23 Feb 2022 18:13:57 +0000 From: "Ashish Singhal" To: Ard Biesheuvel , edk2-devel-groups-io CC: Marc Zyngier , Sami Mujawar , Ard Biesheuvel , Leif Lindholm Subject: Re: [edk2-devel] [PATCH] ArmPkg: Invalidate Instruction Cache On MMU Enable Thread-Topic: [edk2-devel] [PATCH] ArmPkg: Invalidate Instruction Cache On MMU Enable Thread-Index: AQHYJ5XQvRPcrKvPP0GmL2mPrc+CGKygt3cAgAAgeICAAIPkToAADgqAgAAB9z4= Date: Wed, 23 Feb 2022 18:13:57 +0000 Message-ID: References: <122c32bb19ed0730ef166b9f46d3b112bc9ed937.1645497637.git.ashishsingha@nvidia.com> <877d9m3qny.wl-maz@kernel.org> In-Reply-To: Accept-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: suggested_attachment_session_id: 2b735d18-bf70-b0f3-1f10-dd2bc468cbf1 authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 79eac699-5274-4c12-7f72-08d9f6f8426c x-ms-traffictypediagnostic: BN7PR12MB2722:EE_ x-microsoft-antispam-prvs: x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: 3VGeWmnSEEGa+3ox3f602hnsJiGDYtdwsezTBj4TNLsCw1bPBEv8EzrZ85Ww7q2PsY02eDa5OGtsTrF6No0/ijqF++Mpm5KgU5szZxZDzlE2+rVfRtwOPsI9tNGwWfY4mIegbNaToStJa2Pg044KleZOaDcBIVjW/PtBGnafn1onUzwtsBQvMSgAj/hovTCgUoBG/jHNYYPxw83OPENb7vNOrT5v3NloNZXtwxKHeOszM7dYvVDMO8eTWTazqdBQZe73PFSXl/CSyopsWJ8PmWxaU4zeBcn8LxnnvQTkzWRc/RvQuhPQtDzSlHljYwCA15Z8w8b8rSgbLNBbekJVU1f3yyO2w2mfEHZiboIvUVh8I+/ELv/kxE72XNz+Zg1wC3sOJvYrkgGZVrFB1D4Ra5okCuOVCsHSqyhVJw5g63YaSNccbto5zjMYkBqgDDuWTZuz7ZDUkiBYP7xt9fx8uO0oLscr7OcH0m74jN3hdAd8tdvE3XeV4soyUma3LlZ0LdKCw3OeKIAfhBBgWvlbFftZkgZKKnJHBm8DPK9u+Eap2CWD4UaVxZfEpRtP2A3sAorTHEIRde4xhDJzoM/d9oxMU+uhhpdU0LdmBHGVvQbQJvW5xOggqJr1Eqk3EhfISyU7amTEQp8uopK61QTcgsBtfw2K50jSCssOpcsbRN1BTVFqE4o7UYSS4TvWwa9Hrzw4vt1VNcodK4lOpMW4nA== x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:CH2PR12MB5546.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230001)(4636009)(366004)(33656002)(38070700005)(86362001)(38100700002)(122000001)(66446008)(8936002)(508600001)(66476007)(2906002)(52536014)(64756008)(66556008)(76116006)(66946007)(186003)(316002)(19627405001)(4326008)(110136005)(54906003)(5660300002)(8676002)(83380400001)(55016003)(53546011)(9686003)(7696005)(6506007)(91956017)(26005)(71200400001);DIR:OUT;SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?iso-8859-1?Q?cg0lv6IIBWhz9NQpIzu4imU59MWYUP8jrsSD9Eg5GA5MtxL7h32QhzkkLM?= =?iso-8859-1?Q?nJq5uJVStVoIh8gZACj27VYOFBjM1Z8flI5sM+AHC+Upm7ItiF02kNHcVk?= =?iso-8859-1?Q?kcpkAyzSJZ3+a4+EMhKJ7lO5Powgz+UE41fEMx34if1GaxuZJ7+h0tAUTo?= =?iso-8859-1?Q?qTZRyXni0tZi0nUGCKnFDe2Ul2VE455ditx0Ckh5AohOxXgcL8P3umbMnU?= =?iso-8859-1?Q?Utwcu6Ltuqqu3ERhGFSteETjJrob8XbTcKqLjNbxzRFjCEQGdTMdfwcul1?= =?iso-8859-1?Q?F6P60t3z/zGsflkfBy0IVTdUeq6EoX8l++qT3EMBytA5sLfyxsjZ6n4+RV?= =?iso-8859-1?Q?kNjRcn5DN1evHSOtqMim3EoFD+ObhbUc1/zMUuNu7cMk5Cjs1jTowvl3wD?= =?iso-8859-1?Q?6nieuF+zi3Nw6mmK6Nge0W+yBMzNgfhdhWRc82UEw1SYXBUpTbVoHNelY1?= =?iso-8859-1?Q?ZwrUpPbBcysok3AqXRqp5T+UltuW1a+oohOt8n2zZtdO+b5d0K+pYAZ4k6?= =?iso-8859-1?Q?mEMz6d8HdRgec1w+9BBI0y9xoZm17m5CnYVqG2uA2MG59QaPAXf5N+GiEp?= =?iso-8859-1?Q?iKho7Nlcba4STz4s+9GPydlFFwrvX+8I9eN5MiLeLrJc/ie/mw0S0AdoVz?= =?iso-8859-1?Q?+wpjVEECYGJcdswsK5rOtw0ObMtfVLOEwnsWO5AuoqlhhBgLXY00a3ZDwN?= =?iso-8859-1?Q?zBpuWaN+4RTf/ui5UKuliX772ymvsR1NhBoxyzkoZVHNRqBarWUAi42Wof?= =?iso-8859-1?Q?0f7pNmFFlmJXJJ22p82fYIeO7To9vR/Po2Xbl7EhV2h7HM6WEX19L4Etdm?= =?iso-8859-1?Q?CJaM71P1CCsQpiu1kG2apxZq9/khQmm3oMk+OZLrQPNWkp4CgDY+EoSInS?= =?iso-8859-1?Q?XUdCYS9eAtPjF1v239vlz7cG835ePWNOmShvUjvIxvsC+jb6ku73MLY4VD?= =?iso-8859-1?Q?PlyVn6koIefJnuXUNq7mms9Su/O6UHVE0fC/2btEinTID57W5ynW4cmXy8?= =?iso-8859-1?Q?dtYrbwLpomouwFqWf9e4i4ZtrePGaNRprSAF9RxEnJY8Lad4rGTDZPMlQb?= =?iso-8859-1?Q?aCZfypqWgF14scRNJGV/Czq0+rJBp425Yms6aZK3lx2E+oe5hJqcpPEh2X?= =?iso-8859-1?Q?3XztOJCO/d1dCFlzBO/mDCjTmbktTtoDl2HpGb6fJKg/L6TnytHxFi9HMh?= =?iso-8859-1?Q?G3QWYqJx7gycGs55AA44EhnkSA+AesZTcqsDUGsuF8snQ7FlRmiFBreY2x?= =?iso-8859-1?Q?UV9BIWPtJ7FOXaBBhmm1jc6lnkjnoDJ1VXQG6JzAqfNogZNBHIHBbyFxlW?= =?iso-8859-1?Q?NYT63EQkjSoIZArjkWYPDthOAsG6/J1DhAY16lLwW1e7VUIQaHLcg5z56Z?= =?iso-8859-1?Q?/SdmZaS/BCO3fqBsWltmlROlNlSiYQPnA1b872iQ/52D3XDTK25ZzNoGnA?= =?iso-8859-1?Q?kZaKEResX73SBijCQMOfDeRRrqtTzwSngXYpr40yJ4ucxSnzMHhJADxlfM?= =?iso-8859-1?Q?2n80puS0Wm9EwgxIxkGPwG6bDZf+W6LG1HYIeD2cUTFQWMqTuTHyxfnOLE?= =?iso-8859-1?Q?+XP8Iwd3qJPJIiSXtVetztm6whzO7sKyLWl5Cpq/m3MGueSjNNPXKMpUuP?= =?iso-8859-1?Q?D3YyZp4MbmP8Xze3T02pfHLVBChh5bY7geS6grXmMBMgBx9+JynbRvWVct?= =?iso-8859-1?Q?KqvmycgNqlu+F/UHHzY=3D?= MIME-Version: 1.0 X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: CH2PR12MB5546.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 79eac699-5274-4c12-7f72-08d9f6f8426c X-MS-Exchange-CrossTenant-originalarrivaltime: 23 Feb 2022 18:13:57.5584 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: AWnTpGn8K+fnum0UcIvowIN+W3rg1aXvbLxiLDWctsIXn80ZtPwPU3dzP3YAfV/1i/rlraQC5avH402nXdEMKg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN7PR12MB2722 Content-Language: en-US Content-Type: multipart/alternative; boundary="_000_CH2PR12MB5546FB7F34DCE25C2B23F2A7BA3C9CH2PR12MB5546namp_" --_000_CH2PR12MB5546FB7F34DCE25C2B23F2A7BA3C9CH2PR12MB5546namp_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Ard, During PrePi, I setup the initial memory map by calling into ArmConfigureMm= u function with my memory table where device memory regions have attribute = of ARM_MEMORY_REGION_ATTRIBUTE_DEVICE and DRAM regions have attribute of AR= M_MEMORY_REGION_ATTRIBUTE_WRITE_BACK. For device memory, XN bit is set by ArmMemoryAttributeToPageAttribute funct= ion. After PrePi, when I add a region of memory to device memory from a DXE= driver, I call gDS->AddMemorySpace with EfiGcdMemoryTypeMemoryMappedIo and= EFI_MEMORY_UC | EFI_MEMORY_RUNTIME followed by gDS->SetMemorySpaceAttribut= es with EFI_MEMORY_UC. Please let me know in case I have still not understood your question. In our case, UEFI is running in NS-EL2. I understand what I am proposing ma= y not be a proper fix. I am looking for help on what could be a fix for thi= s as we are using all upstream code for memory management. Thanks Ashish ________________________________ From: Ard Biesheuvel Sent: Wednesday, February 23, 2022 10:40 AM To: edk2-devel-groups-io ; Ashish Singhal Cc: Marc Zyngier ; Sami Mujawar ; Ard= Biesheuvel ; Leif Lindholm Subject: Re: [edk2-devel] [PATCH] ArmPkg: Invalidate Instruction Cache On M= MU Enable External email: Use caution opening links or attachments On Wed, 23 Feb 2022 at 18:36, Ashish Singhal via groups.io wrote: > > Hello Ard and Marc, > > I apologize for not providing the background on this in the commit messag= e and I understand the commit message is not very clear as well. Let me try= to summarize the problem. > > In our UEFI implementation, we are doing the following as part of the ini= tial MMU setup: > > Set the applicable device memory as nGnRnE memory. > Set the whole DRAM as normal memory that translates to RW and executable = memory. > Enable caches and MMU. > At this time, the memory map looks correct when I check that from DS-5. > I have asked you a number of times about the XN attribute. How and where are you setting it for the device regions? > When we start dispatching drivers, DxeCore dispatches a driver and marks = its code area as RO and executable and its data region as RW and non-execut= able. What we are seeing randomly is that some of the page tables (using DS= -5) have invalid output address that leads to the correct input address fro= m UEFI being translated to an unavailable memory location causing a crash s= ometime in EL2 or sometimes as a RAS error in EL3. > At which EL does UEFI run? If at EL1, is it running under a stage2 mapping? If so, does the stage 2 mapping set the XN attribute appropriately for the device mappings? > When I reached out to the CPU team here, they said Arm=AE Cortex=AE-A78AE= is a highly speculative core and we need to have appropriate barriers in p= lace so that there is consistency in the way an address is accessed especia= lly if it is done right after there is a change in translation tables. Base= d on this, I started some experimentation wrt caches whenever MMUs are enab= led and I found that invalidating the instruction cache after enabling MMUs= solves this problem. > It may hide it, but I don't think it is a proper fix. > Please note that I could be wrong with my hypothesis here and I may just = be masking the issue. If that is the case, please let me know what I should= be trying as I am out of ideas at this point. Also, the same UEFI works on= NVIDIA's Xavier Silicon that has Carmel cores but shows this issue on Orin= Silicon that has Arm=AE Cortex=AE-A78AE v8.2 64-bit CPU. > Thanks, Ard. --_000_CH2PR12MB5546FB7F34DCE25C2B23F2A7BA3C9CH2PR12MB5546namp_ Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable
Ard,

During PrePi, I setup the initial memory map by calling into ArmConfig= ureMmu function with my memory table where device memory regions have attri= bute of ARM_MEMORY_REGION_ATTRIBUTE_DEVICE and DRAM regions have attri= bute of ARM_MEMORY_REGION_ATTRIBUTE_WRITE_BACK.

For device memory, XN bit is set by ArmMemoryAttributeToPageAttribute = function. After PrePi, when I add a region of memory to device memory from = a DXE driver, I call gDS->AddMemorySpace with EfiGcdMemoryType= MemoryMappedIo and EFI_MEMORY_UC | EFI_MEMORY_RUNTIME followed by gDS->SetMemorySpaceAttributes with EFI_MEMORY_UC.=

Please let me know in case I have still not understood your question.

In our case, UEFI is running in NS-EL2. I understand what I am proposing may not be a proper fix. I am looking for= help on what could be a fix for this as we are using all upstream code for memory management.

Thanks
Ashish
From: Ard Biesheuvel <ar= db@kernel.org>
Sent: Wednesday, February 23, 2022 10:40 AM
To: edk2-devel-groups-io <devel@edk2.groups.io>; Ashish Singha= l <ashishsingha@nvidia.com>
Cc: Marc Zyngier <maz@kernel.org>; Sami Mujawar <sami.mujaw= ar@arm.com>; Ard Biesheuvel <ardb+tianocore@kernel.org>; Leif Lind= holm <quic_llindhol@quicinc.com>
Subject: Re: [edk2-devel] [PATCH] ArmPkg: Invalidate Instruction Cac= he On MMU Enable
 
External email: Use caution opening links or attac= hments


On Wed, 23 Feb 2022 at 18:36, Ashish Singhal via groups.io
<ashishsingha=3Dnvidia.com@groups.io> wrote:
>
> Hello Ard and Marc,
>
> I apologize for not providing the background on this in the commit mes= sage and I understand the commit message is not very clear as well. Let me = try to summarize the problem.
>
> In our UEFI implementation, we are doing the following as part of the = initial MMU setup:
>
> Set the applicable device memory as nGnRnE memory.
> Set the whole DRAM as normal memory that translates to RW and executab= le memory.
> Enable caches and MMU.
> At this time, the memory map looks correct when I check that from DS-5= .
>

I have asked you a number of times about the XN attribute. How and
where are you setting it for the device regions?

> When we start dispatching drivers, DxeCore dispatches a driver and mar= ks its code area as RO and executable and its data region as RW and non-exe= cutable. What we are seeing randomly is that some of the page tables (using= DS-5) have invalid output address that leads to the correct input address from UEFI being translated to an u= navailable memory location causing a crash sometime in EL2 or sometimes as = a RAS error in EL3.
>

At which EL does UEFI run? If at EL1, is it running under a stage2
mapping? If so, does the stage 2 mapping set the XN attribute
appropriately for the device mappings?


> When I reached out to the CPU team here, they said Arm=AE Cortex=AE-A7= 8AE is a highly speculative core and we need to have appropriate barriers i= n place so that there is consistency in the way an address is accessed espe= cially if it is done right after there is a change in translation tables. Based on this, I started some experimen= tation wrt caches whenever MMUs are enabled and I found that invalidating t= he instruction cache after enabling MMUs solves this problem.
>

It may hide it, but I don't think it is a proper fix.

> Please note that I could be wrong with my hypothesis here and I may ju= st be masking the issue. If that is the case, please let me know what I sho= uld be trying as I am out of ideas at this point. Also, the same UEFI works= on NVIDIA's Xavier Silicon that has Carmel cores but shows this issue on Orin Silicon that has Arm=AE Cortex= =AE-A78AE v8.2 64-bit CPU.
>

Thanks,
Ard.
--_000_CH2PR12MB5546FB7F34DCE25C2B23F2A7BA3C9CH2PR12MB5546namp_--