From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from NAM12-DM6-obe.outbound.protection.outlook.com (NAM12-DM6-obe.outbound.protection.outlook.com [40.107.243.63]) by mx.groups.io with SMTP id smtpd.web08.3384.1645850808290811101 for ; Fri, 25 Feb 2022 20:46:48 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@nvidia.com header.s=selector2 header.b=IZ4A9AN7; spf=permerror, err=parse error for token &{10 18 %{i}._ip.%{h}._ehlo.%{d}._spf.vali.email}: invalid domain name (domain: nvidia.com, ip: 40.107.243.63, mailfrom: ashishsingha@nvidia.com) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=KJOcoyGgRVLedKWYkmEwSppi2Q+mKhnF5U855v7YYNBse+Qp3RZsY/9K8wRFtMVZpwLDmL6WpGKdGSTe9Z2tne5rKB9HE8V/GXx0qq4wsOUkWFHjs7VT51Iuf987ZXp7/jet66WyAPIKh9uhyr+mTQBGjyZpaeHiqygaMuXmA7YFa0h6HCLnoxF3SPde20TBXPYRCxw70xkNJZOmfGiLY4f7f4CsvMV7IrRrIgCUIh5RwI5SN8/laEb8nP/XS2QTbvl7BMc1xdCDo4R6a7J5nSVlNxX2lcwC6sRRYJmSCtrKRMMIiv/ocYXiRjyhZuLHuFae2NZodnW2BB+F3Qy/AA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=X8LSjgX8YGfwsISCO1fUhhNYqcaS4jzWt+QoBet692g=; b=QwdNFrsRkx3vMiS3W6y47erBv2hJKuF0q6ViRPEIPyO42u0ZT2T/yVueKMj5ji8yBlfDQNAn02SxuD6b8COMbEBflnVG05H+v+AJEJnhdEwh4uNhi+Zuau2PkoJmCy9H0lilyQ6x44O4i/DEZ36WjQGmQmV/BokSsgDrisvTNgvTxWs0AHUy0JIwHJxmp4z8t7kZaUNWZ0A84WzkVLDi9nVXpyprs2RFqIHHvLZKJ9mJO9LFNkdhc7HspokI1moVPd6YneRU8QKvIFBTJnmWx44nEHWOTsoSTt/ZI/jGDIjlGj0mYzmoESqUhwJ/fy7svFV5gEvCHfPf2hO9rzs1Pg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=X8LSjgX8YGfwsISCO1fUhhNYqcaS4jzWt+QoBet692g=; b=IZ4A9AN7DfBrTF1aisEnaDVedXLSDPHpbze4bcVlrmjWyckzOiFUaKXq0Q9YPce4LPbacPlx0v0MM3VfPZvmvGdfP5YRwqnXq6UGakhOMl6kAY+lV/OcoQUfb4q+QOUuGN21rEmocRJt7KsqtCXtHJnz7cSenlx5Ice05prpPihMpQxQoHzgFd+Ktvkl8ZQAWfo9Wz+mtOrVOO3KQUAgKVWlmFCLCSYLo7ZYeYIQXwLgVqN2BYsHrG7UZE0p+2GTmJQuSX9QD/nkIw5N+J14yDqL+8DojhUjxbnzxHQj8God92uuuEGDaX1JK+M4kve2UFttlSWH+jg5kYexz+hKAA== Received: from BY5PR12MB5544.namprd12.prod.outlook.com (2603:10b6:a03:1d9::22) by CY4PR1201MB0213.namprd12.prod.outlook.com (2603:10b6:910:21::17) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5017.22; Sat, 26 Feb 2022 04:46:45 +0000 Received: from BY5PR12MB5544.namprd12.prod.outlook.com ([fe80::1cc3:2137:2949:b0a7]) by BY5PR12MB5544.namprd12.prod.outlook.com ([fe80::1cc3:2137:2949:b0a7%3]) with mapi id 15.20.5017.026; Sat, 26 Feb 2022 04:46:45 +0000 From: "Ashish Singhal" To: Ard Biesheuvel CC: edk2-devel-groups-io , Marc Zyngier , Sami Mujawar , Ard Biesheuvel , Leif Lindholm Subject: Re: [edk2-devel] [PATCH] ArmPkg: Invalidate Instruction Cache On MMU Enable Thread-Topic: [edk2-devel] [PATCH] ArmPkg: Invalidate Instruction Cache On MMU Enable Thread-Index: AQHYJ5XQvRPcrKvPP0GmL2mPrc+CGKygt3cAgAAgeICAAIPkToAADgqAgAAB9z6AAFWaAIAAdXSAgAMQ3OA= Date: Sat, 26 Feb 2022 04:46:45 +0000 Message-ID: References: <122c32bb19ed0730ef166b9f46d3b112bc9ed937.1645497637.git.ashishsingha@nvidia.com> <877d9m3qny.wl-maz@kernel.org> In-Reply-To: Accept-Language: en-US X-Mentions: maz@kernel.org,ardb@kernel.org X-MS-Has-Attach: X-MS-TNEF-Correlator: suggested_attachment_session_id: 74ca9710-2830-cd21-0ca4-417ddb43d9bd authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 369f9646-2e16-40b7-d0a6-08d9f8e2fe00 x-ms-traffictypediagnostic: CY4PR1201MB0213:EE_ x-microsoft-antispam-prvs: x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: +6S6INF6O26HlDbHFaE7+CC6YS+emdCHaZexI0xCFmZ9Kc5IhcoJKuVbpGfPYWr69c5gbYjtPZqaCTfhKRuy+vZPqE6llSFJC0zA6JZEAvGkQePQn2yG5m+5qlzKZNpRD+5tEvOpe9BY8LBGVOiTjRe3asBrKlTmqt073QGANKPkbLWzDRyJHd5ZAx5/ebslIrp+ZGZNtzUTvRw6T+OnXM/64eDGk4hoHu7E/TEUQa3nuzEILFdAtm7W6sDCeHDR8CuCTLIVfVEL5NW9qbRQcOZ7TXLwFd/9ONyBcRcJT2hCpV8BKwwJ6pYMFAsuDcxnj3H1/wFcG+/5FjgDD1kbxhYHD01EBDJpQEbgxJr6B6s7vP53/MRP+UD9Z0N8jCaZ71rneY2VxFgTKzxzbmdg4fSlEmpYCaTDjUzvnwbfFgH/PEVDnUJKWm44H/kSqlmT5qIqEkCz70br8dH4ztOSMNXS6/7CU95uMvvfE7gpktEmIqE7c6kyfoP5R0Ak+1IxO5QwWR2Dd+SN5IO5MvmceifyoOoPPFzFsob8D+5WBceO2J6Wiox571TC12o3YaGMa2orVTfZXeOWWR+9AGX+5V790Yd9ob+er0e38uyVF4x1gcr87ki+Trj2Ucf6GhT7Rp1ZKGZp7EewOYYcf8JyKeTzzmFH7RUwPWiF0CUsuxZ1yvZGxNRAZSJSBjKEYy4ujS2h4RbEO2KNnjO0O00ImhwAHMppUaZ3rZZh/h9vxNcmcWTTo/DGPcpIOGcHUTd/2MXKHVbsg6WNcslvsip58t1PluEihHC0QrDWAI1bGxg= x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR12MB5544.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230001)(4636009)(366004)(76116006)(4326008)(8676002)(91956017)(5660300002)(83380400001)(64756008)(19627405001)(55016003)(66446008)(66946007)(8936002)(66476007)(66556008)(2906002)(52536014)(33656002)(7696005)(6506007)(38100700002)(53546011)(71200400001)(86362001)(166002)(9686003)(122000001)(316002)(186003)(38070700005)(26005)(6916009)(508600001)(54906003);DIR:OUT;SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?8VOv+SPyQ8E7kQ43XXktjhXrLDC3/kZ/kayD3Jaix5q8V/k+B0c6IpMcCem6?= =?us-ascii?Q?0U6PvtudWbVlh1LurqMBKN/Eq+4wrJPMtCp6Yl+dfpLUjyYccafIcAcQqj2r?= =?us-ascii?Q?xOgllh5ZuyRK6+neSsut+Bjh6lXAgdBEqCDGIXiJFJ7XJAJsJUH/4yT3XsPq?= =?us-ascii?Q?7+QC5b2bFIcZIIfMHbYPdSohRKnneeWIYzELGlRLGWAo3VrW9EdigjDqYwi4?= =?us-ascii?Q?K9DtZvGmCk/s3i1WkEYgsEwMJ2qTjVFJDSiXyJct67ceGZK2dPKtGs5U13M3?= =?us-ascii?Q?LozTXweO8i+tWeExZcStokHL1+gnN1+5yfgmq0PiW4kRtjLM1nJL/OsfT/ye?= =?us-ascii?Q?jD9irLaLg+f5y4UlxngMbJcXUJ5C2S6EoIKO5y0hU70n/HW46pc+FQEwiiwE?= =?us-ascii?Q?YeL7uBN3Xg/yvCHr2PhLP2x1n74anf8SLRIT6vkiOp5kE7vqcSmgXuJ5UA3Q?= =?us-ascii?Q?YYW6d1qXov1DEuzFt1t4zc4aBnSEMj+uMqQlP4m5QYogXYfoRXxS7ITAYb1c?= =?us-ascii?Q?QHEueXMJYKYpifXkPCiRMsnNb7/FIyg1LSvJ5cs2wX/EF5zzUvM3ka5nNUMh?= =?us-ascii?Q?qDfbjmehm/OsicgnRxV/bGpWMkzOpys75Be1qgQd42WVxpkr9O1hDkMLt199?= =?us-ascii?Q?x17aOOVMWk8xtWUffIg56jQHFbwrePX2iyMzBS1Dl3ZZ/DNFNia11L6tL9Af?= =?us-ascii?Q?ztPXfLuSjprmCLxQrqHVDvio/SrCsJB+6Yo/wxhovXhlkK+ml2WCijG3pr0v?= =?us-ascii?Q?yMFGCn1dmJD2F1kZMP5FCJB3HKJhACMjX7W1IkGowNNgDg/X1h+8mHvxxcH3?= =?us-ascii?Q?nQGruUDzm2ooA4ftNSVtc+rBaQpYfGB7faMfUGA1m5Noe0tejHSP6H5JbDqh?= =?us-ascii?Q?t1SwrtvWU6lRjfWVliZvGwTtJQguFgtFmiBfEG5VGp0hgC2j2qsEA8Ugdnm0?= =?us-ascii?Q?NwXiJNwYQooW40Lszd9otEswjFJRenrvsghfZtzZ9Ritv5nNDazDYsc+rZuf?= =?us-ascii?Q?4lLzwBVPhTmnL9JTvCbc1BabwFdX/Tzmf1MWTlYXtIDJjQzdRE7KY2Q1Qu2u?= =?us-ascii?Q?3f5trvskd3ahOSaNTT7pJua9jOPU7xiqDq7Ms8tyPrZprQRcZvOWyiWWF4mm?= =?us-ascii?Q?xi9cMmoBqSm1IeD6BouzmEc58cf5G9Tcd2WqEwy4795paIGYQWTdohx/avho?= =?us-ascii?Q?d70uQWOBllMn/GvvsawbqWHzrv8ehJ/0xy2W7G3IO2PBkgiSHPHyZjcnvcG0?= =?us-ascii?Q?dfQtApH1Odm9h8G3ety4Z+fxwGxlHh2p+6LJJzgWgNE2rBChsGL9FbydbjT4?= =?us-ascii?Q?yhVtdGQSuudIGEHs8AnAJJzpqDl9Y2WCMCjg58HqJKW9Wfmbjv/3LZ1OgUMl?= =?us-ascii?Q?Zi7AEZudTETTozh2bMoUjddO+im99BrC6knKBiyBiqGTnlhKOaoWF6sep9tf?= =?us-ascii?Q?jJAv/WpLiS/3P09wb7gppefNONwH8Sjzf7lDigr5MLrtIR/RA1sNL/GQBJav?= =?us-ascii?Q?eCzJOgrSXbhydvJPJmfo2E5VLPSJdZduQA59XSAlty3Ut+rjHwTewZ687mFp?= =?us-ascii?Q?+yg2eVeYUZHphRy0eekik1jEDOwLaMjKUi+cUG8RXn4R2BEeKIYIlF3q17LS?= =?us-ascii?Q?pTqL46Hf5F3frlp67k1l1cM=3D?= MIME-Version: 1.0 X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: BY5PR12MB5544.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 369f9646-2e16-40b7-d0a6-08d9f8e2fe00 X-MS-Exchange-CrossTenant-originalarrivaltime: 26 Feb 2022 04:46:45.6462 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: +f+Nd0Vp7KI0AS95lbJFkWli0PyAv4PIN0FuLPHpVcrG4f+0qg/LnW1a6FXk3CcRoIOfNAru9DpK2pRQbmW1Dg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY4PR1201MB0213 Content-Language: en-US Content-Type: multipart/alternative; boundary="_000_BY5PR12MB5544BD743639BD85C3489667BA3F9BY5PR12MB5544namp_" --_000_BY5PR12MB5544BD743639BD85C3489667BA3F9BY5PR12MB5544namp_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable @Marc Zyngier I have addressed your comment regarding the type of data synchronization ba= rrier and have posted v2 of the patch set. @Ard Biesheuvel I am still looking for a better solution to the problem and I am open to wo= rking with you or anyone else from ARM on that. Please let me know if you w= ant me to try something else. Thanks Ashish ________________________________ From: Ashish Singhal Sent: Wednesday, February 23, 2022 11:01 PM To: Ard Biesheuvel Cc: edk2-devel-groups-io ; Marc Zyngier ; Sami Mujawar ; Ard Biesheuvel ; Leif Lindholm Subject: Re: [edk2-devel] [PATCH] ArmPkg: Invalidate Instruction Cache On M= MU Enable Hello Ard, When we had a discussion on this topic earlier, maybe a few weeks back, we = thought device memory is being accessed in a speculative manner. Our latest= debug where we focussed on MMU page tables at the time of error tells that= the issue is actually DRAM mapping in page tables getting corrupt (as per = DS-5) where descriptor for a page seems to be something garbage. What this = causes is that a valid input address in DRAM may get translated to an addre= ss in a different region of DRAM or some address in device memory. When I invalidate the instruction cache after enabling MMUs, this issue see= ms to be getting resolved. Again, I am not saying this is a fix but this is= something that solves the issue while we are looking for suggestions from = you for a proper fix. I have tried to summarize the problem based on the latest findings a few em= ails down the trail if you want to have a look at that again. Thanks Ashish ________________________________ From: Ard Biesheuvel Sent: Wednesday, February 23, 2022 3:54 PM To: Ashish Singhal Cc: edk2-devel-groups-io ; Marc Zyngier ; Sami Mujawar ; Ard Biesheuvel ; Leif Lindholm Subject: Re: [edk2-devel] [PATCH] ArmPkg: Invalidate Instruction Cache On M= MU Enable External email: Use caution opening links or attachments On Wed, 23 Feb 2022 at 19:14, Ashish Singhal wrot= e: > > Ard, > > During PrePi, I setup the initial memory map by calling into ArmConfigure= Mmu function with my memory table where device memory regions have attribut= e of ARM_MEMORY_REGION_ATTRIBUTE_DEVICE and DRAM regions have attribute of = ARM_MEMORY_REGION_ATTRIBUTE_WRITE_BACK. > > For device memory, XN bit is set by ArmMemoryAttributeToPageAttribute fun= ction. After PrePi, when I add a region of memory to device memory from a D= XE driver, I call gDS->AddMemorySpace with EfiGcdMemoryTypeMemoryMappedIo a= nd EFI_MEMORY_UC | EFI_MEMORY_RUNTIME followed by gDS->SetMemorySpaceAttrib= utes with EFI_MEMORY_UC. > > Please let me know in case I have still not understood your question. > This all looks ok. But the real question is whether the address that the speculative access targets is mapped using the XN attribute or not, so you will need to find a way to check that. So there are a couple of options: - The XN attribute is set correctly, but the CPU is speculatively fetching instructions anyway. This would imply a severe hardware bug, and flushing the I-cache is unlikely to make a difference. - The speculative access is not the result of an instruction fetch, in which case I-cache maintenance is unlikely to help either. - The XN bit is not being set correctly, and so the MMU code needs to be fi= xed. Papering over this by adding I-cache maintenance doesn't seem the best course of action tbh. -- Ard. --_000_BY5PR12MB5544BD743639BD85C3489667BA3F9BY5PR12MB5544namp_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

I have addressed your comment regarding the type of data synchronization ba= rrier and have posted v2 of the patch set.


I am still looking for a better solution to the problem and I am open to wo= rking with you or anyone else from ARM on that. Please let me know if you w= ant me to try something else.

Thanks
Ashish
From: Ashish Singhal <as= hishsingha@nvidia.com>
Sent: Wednesday, February 23, 2022 11:01 PM
To: Ard Biesheuvel <ardb@kernel.org>
Cc: edk2-devel-groups-io <devel@edk2.groups.io>; Marc Zyngier = <maz@kernel.org>; Sami Mujawar <sami.mujawar@arm.com>; Ard Bies= heuvel <ardb+tianocore@kernel.org>; Leif Lindholm <quic_llindhol@q= uicinc.com>
Subject: Re: [edk2-devel] [PATCH] ArmPkg: Invalidate Instruction Cac= he On MMU Enable
 
Hello Ard,

When we had a discussion on this topic earlier, maybe= a few weeks back, we thought device memory is being accessed in a speculat= ive manner. Our latest debug where we focussed on MMU page tables at the time of error tells that the issue i= s actually DRAM mapping in page tables getting corrupt (as per DS-5) where = descriptor for a page seems to be something garbage. What this causes is th= at a valid input address in DRAM may get translated to an address in a different region of DRAM or some add= ress in device memory.

When I invalidate the instruction cache after enablin= g MMUs, this issue seems to be getting resolved. Again, I am not saying thi= s is a fix but this is something that solves the issue while we are looking for suggestions from you for a prope= r fix.

I have tried to summarize the problem based on the la= test findings a few emails down the trail if you want to have a look at tha= t again.

Thanks
Ashish
From: Ard Biesheuvel <= ardb@kernel.org>
Sent: Wednesday, February 23, 2022 3:54 PM
To: Ashish Singhal <ashishsingha@nvidia.com>
Cc: edk2-devel-groups-io <devel@edk2.groups.io>; Marc Zyngier = <maz@kernel.org>; Sami Mujawar <sami.mujawar@arm.com>; Ard Bies= heuvel <ardb+tianocore@kernel.org>; Leif Lindholm <quic_llindhol@q= uicinc.com>
Subject: Re: [edk2-devel] [PATCH] ArmPkg: Invalidate Instruction Cac= he On MMU Enable
 
External email: Use caution opening links or att= achments


On Wed, 23 Feb 2022 at 19:14, Ashish Singhal <ashishsingha@nvidia.com>= ; wrote:
>
> Ard,
>
> During PrePi, I setup the initial memory map by calling into ArmConfig= ureMmu function with my memory table where device memory regions have attri= bute of ARM_MEMORY_REGION_ATTRIBUTE_DEVICE and DRAM regions have attribute = of ARM_MEMORY_REGION_ATTRIBUTE_WRITE_BACK.
>
> For device memory, XN bit is set by ArmMemoryAttributeToPageAttribute = function. After PrePi, when I add a region of memory to device memory from = a DXE driver, I call gDS->AddMemorySpace with EfiGcdMemoryTypeMemoryMapp= edIo and EFI_MEMORY_UC | EFI_MEMORY_RUNTIME followed by gDS->SetMemorySpaceAttributes with EFI_MEMORY_UC.
>
> Please let me know in case I have still not understood your question.<= br> >

This all looks ok. But the real question is whether the address that
the speculative access targets is mapped using the XN attribute or
not, so you will need to find a way to check that.

So there are a couple of options:
- The XN attribute is set correctly, but the CPU is speculatively
fetching instructions anyway. This would imply a severe hardware bug,
and flushing the I-cache is unlikely to make a difference.
- The speculative access is not the result of an instruction fetch, in
which case I-cache maintenance is unlikely to help either.
- The XN bit is not being set correctly, and so the MMU code needs to be fi= xed.

Papering over this by adding I-cache maintenance doesn't seem the best
course of action tbh.

--
Ard.
--_000_BY5PR12MB5544BD743639BD85C3489667BA3F9BY5PR12MB5544namp_--