From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from NAM11-DM6-obe.outbound.protection.outlook.com (NAM11-DM6-obe.outbound.protection.outlook.com [40.107.223.63]) by mx.groups.io with SMTP id smtpd.web09.7171.1645682520664297608 for ; Wed, 23 Feb 2022 22:02:01 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@nvidia.com header.s=selector2 header.b=TECkthMW; spf=permerror, err=parse error for token &{10 18 %{i}._ip.%{h}._ehlo.%{d}._spf.vali.email}: invalid domain name (domain: nvidia.com, ip: 40.107.223.63, mailfrom: ashishsingha@nvidia.com) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=HW86OixXl77yjfdDT9jSmbfrHkB5f7Gfduqyc+vhOp+4wxHBSl1gvyB5gz4KSI0ovD0dq5/d+dgZuUUG2EP6UUOth6ngNwIod6lhvgjISSXb6j87J86UqcSxZWl7+eLYSwA6XBvc6WzEw2mVe2JQUJPL57gqu7Z0xwW/W55zHD2hSLQU6S144gb8+3VecKqji350x7GMjmIhBCjT2gLZAyFUVdcyl6+wSzz1DVrk37GtAeQ2CQ+OtU9yv9IHKL5tx5LsP1fUUmO5rEwnYpyvNAWwOtCwiqhgOa1cNWh4HytHdNWcID7PPxuFiHF82z+Ux82eBNsabRmmcRh5UkNVsw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=fiGRb27t9G6VsMCV8tAMv7ymNzoGuQrWwBLq25yiBvQ=; b=X//DVkURz/yq5CdxzwxAzQFu6pctGBeK/1hJuaY9KyMvSFhCHaFRO/XmUWR+XR8ZgIGyAEmmCZC6Uz/oRb410VtXp++ugSWr9/ebyESTsKdSR0XZzVRFizfR5YlYcKIEY3SU7eY+ruo7U8pQ8BJcbvh/Bqpn6SMVcwzETFd9eBZulP0YDJ3wGfY/eB0nS8fY1WDqAx+noF/aSy4u5kHXAqvGJkeI8tTYWbTRoW8AVZR7u0cWoz8bISh8NyfePSvDiHvHrnPcaGS1MsNIbVHnOSyssUrwcf3mKLW0fLB+ZdRHW+rrLq3fOUbNd9vTp3iqaNCGjjSfqvsrzJYetX7a1w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=fiGRb27t9G6VsMCV8tAMv7ymNzoGuQrWwBLq25yiBvQ=; b=TECkthMWsIWQs21Ies+2YB1pkRlpFlwSd0xNs3IF2Uarj4+wl5PLuSt5Yo0tm520yCsmfTeCEAmedUUo7Vw4BvcWsnxmpLoSCznU2e47N57qcEwFcfrCTzHjVQ+bKX39yG17ci2JDlYyUSoMpFxl/Cs8MxcHERfu3lwxSffdnPDrLBiCsmfDPh+YZ4Belh2OKc7x73gHOtdjqT83qxoAO2nzyymNRA7J2ADjGamkcuh0CoZgTC1RG1/q5djanbziBvuxRERdorpK+xaG9G8gi2HyvyucgkqKwOV8T0Xn3jI2n8RqQoXOSRddoG/tXLnCMbp2fVzfhuXtmxIlh1KivA== Received: from BY5PR12MB5544.namprd12.prod.outlook.com (2603:10b6:a03:1d9::22) by MN2PR12MB2896.namprd12.prod.outlook.com (2603:10b6:208:ab::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4995.26; Thu, 24 Feb 2022 06:01:57 +0000 Received: from BY5PR12MB5544.namprd12.prod.outlook.com ([fe80::1cc3:2137:2949:b0a7]) by BY5PR12MB5544.namprd12.prod.outlook.com ([fe80::1cc3:2137:2949:b0a7%3]) with mapi id 15.20.5017.024; Thu, 24 Feb 2022 06:01:57 +0000 From: "Ashish Singhal" To: Ard Biesheuvel CC: edk2-devel-groups-io , Marc Zyngier , Sami Mujawar , Ard Biesheuvel , Leif Lindholm Subject: Re: [edk2-devel] [PATCH] ArmPkg: Invalidate Instruction Cache On MMU Enable Thread-Topic: [edk2-devel] [PATCH] ArmPkg: Invalidate Instruction Cache On MMU Enable Thread-Index: AQHYJ5XQvRPcrKvPP0GmL2mPrc+CGKygt3cAgAAgeICAAIPkToAADgqAgAAB9z6AAFWaAIAAdXSA Date: Thu, 24 Feb 2022 06:01:57 +0000 Message-ID: References: <122c32bb19ed0730ef166b9f46d3b112bc9ed937.1645497637.git.ashishsingha@nvidia.com> <877d9m3qny.wl-maz@kernel.org> In-Reply-To: Accept-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: suggested_attachment_session_id: 033962ca-9980-e248-7bef-6cd65cfe18dc authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 1ae8573f-e60e-45ca-32e9-08d9f75b2a3b x-ms-traffictypediagnostic: MN2PR12MB2896:EE_ x-microsoft-antispam-prvs: x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: /rEW03lYTwqvVumiZ6jV/nx+nFzAIBSajKx8rbqBOiR7mjQC0CyUnm5nlqb8li1KKRICosHE3uHthd0m9CF4YBB7OmP9cdRMr8Gc2NBIvqoGo9jdLatPhynEBC/d7RCujUEebidPngb2RUI6D3Tz4fh0kbKkMEaQLYP9z44SgY/ixVZqlpUdOGzXJE0p4Q9apBZilR/MmGLwHDt6ADJ3uLmv+yRLPJJQlXR9l5UQlqt9I26bSqxGgpbTi3QyDKgYVesw5nfNC4M/ubNZQWMEldP5mdsY5w/vKz2ZhlX2mV2EhCS2eah6q3X5iDP16ON2YmowjoET8ofY7fnEdLj/PtHoyIKGvjZk+63CzS6MZ9qqs9sU6QweQN3WhxyH4Lw++xfQw4LF8f9BXBvoMCmBZcbBpSGxnTWZiR6QYU8qfJirshylaulKKYm5IcJh8d9NN5WINpG9BYeO8+lGKZuxmUEfsL1ZShIcibVB89hfucLFSfjSjKcPL4CIzj4JiD751Z94VSKcI4AACKg+9npyhHRwnPZIqEcFcYC07ueMjjIreTzcoOu5BiQQ6Cj47pyeG5IEKSlKaz81rIf5zMWtmnozPta1Nn457CS0RuJXnHFlfS/QxTBAqLs2S+UVh9mVn5F5IFUexWAmdiUhh1zZrX5Jk/dy70eHcfp4d6kpDQVh9RxTo9sQbA69AuyPZZo3Z2GkU/ef/zCFVqnhjDxdSg== x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR12MB5544.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230001)(4636009)(366004)(86362001)(508600001)(19627405001)(122000001)(55016003)(6916009)(38100700002)(2906002)(54906003)(83380400001)(71200400001)(33656002)(7696005)(6506007)(9686003)(316002)(8936002)(5660300002)(66946007)(66556008)(66476007)(66446008)(64756008)(4326008)(52536014)(8676002)(186003)(26005)(76116006)(91956017)(53546011)(38070700005);DIR:OUT;SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?I2yGMRXPlCW3SZIGUZMdHKevvse3NmIiHJyOa2s3DFgTmFd+m29HDnS7PBoP?= =?us-ascii?Q?mI8kOSv9iRern6pMNvg8C+j52NLbyP1BeSN0ZvJ1tT9DjJ2bIyjMZHjf9///?= =?us-ascii?Q?VvdHOGwss6GPQ9gLXewRlytrkYepvNUae2UNXiwQOmpJu6lbSuU1BCfOBoif?= =?us-ascii?Q?dIKNIbsMvv+7Qfw7I8/xTOJWq7/RLNvCoV+heVM6OHACYMY7UB2ukzw9Zji+?= =?us-ascii?Q?Vy8ozlzq0LcEmjG7MIfGY5JqWAFOVh62vv4G7YQV1SZErBQpAeQ5TLkFfmDa?= =?us-ascii?Q?j9ppTNhXLQ1WcsHQNSUsCncFy7sXmmG8O7ZxGPblpVCwTVOexSSNwfL4pENg?= =?us-ascii?Q?WO5oMPXRxDi+BitWYLXuUUp0jXqYbo/7GaQ1YhmywIqe9kDFGMGdhV68jX/D?= =?us-ascii?Q?nYduCW8wWUxnS2UZJwBEPx5hRf/NJ98I5Z1cLr3HYHeXwgYij9vJTslnxV5D?= =?us-ascii?Q?mNOO8oYi/lpHUq53wQlQfbGbhW+Fe8vEjsdfEb+QTu5ue4NfSrzoBeYOsgJY?= =?us-ascii?Q?ORDeSXvdsQRawszZM4zUkjYY/HqTAHN/ZBI1Nm5OfMATIpioljm9YXwahDcU?= =?us-ascii?Q?5djkly/8VcO9Om5WPZGTinEM2GtXtWy87n9iNmMcZ+wfTT2zMSZM7piOaAyZ?= =?us-ascii?Q?J5ypgAQiJmlxoZo3XoQdiigZ9q7L92sfe4Xg/9PmwbMGJMdR5zTPbKBASBaz?= =?us-ascii?Q?iqJGr+wbKLHVQQWZS+8oDKeOlTdVNe7khNOeHmBEvdLkR9biuSSf+CfCPRx1?= =?us-ascii?Q?yV200HSkO2pTpzsK2ApRVsqR14z6qVdoZi/rrcOCMnQsp2gcABxz9c/huIur?= =?us-ascii?Q?BFzsWqLJoND3rpE2hC9oCMLWKqBrNyLFbd3rYlJrEKnoLYD6rr3xXy1pRV7z?= =?us-ascii?Q?zhD3gZ24E4B99ykk21KeuvN29PivYHvb2EY8YSk880T848aoZ56uoYUXmmDe?= =?us-ascii?Q?+OiDlk0blSlblIVB11cQsmsqmJcEcHxxOvvnLMijeaaZZ0Fum0FeU02fn5JD?= =?us-ascii?Q?cZmckEIpMVuRwpavEHyXGlza0zw3kWIx9gc7Scl5qEmIB3A9mQLAj7HObGoU?= =?us-ascii?Q?RhTq7/ezXiYGHhgsSxjIP2k2RKgUJFkIwxFZIqYLrUe1IrHSstah9c5SEZFg?= =?us-ascii?Q?5IOd8oRZm4Hqywhh7qaBzH+nljN6D/MVZfPoiwsdlHC/8LFM6/nRq4yTqZqV?= =?us-ascii?Q?lsmiQ0E/cgzONqVcqzzEwlQk4dR+LxU+y6wLXPMu6R0oEOOVSf5TOCNhsoMY?= =?us-ascii?Q?oX2tR/FigTM+JdzqibASohOBcGeqETwfmwvfGDbDZwTpC9NFQYivQd87lKmx?= =?us-ascii?Q?NaazDBCUC/t4eck8+CCqbTJ9w9AvO6xOi7XErmX3YE9S5dA8jBLStdIzXjgH?= =?us-ascii?Q?NMLAYzWbye4cMka+roRuna7CGGK2yC/vFRCda8VVtst2oXvKT0lhniINGqBp?= =?us-ascii?Q?BFF+F5vZlLv7TKjEJg87Vc/D8cfwE3/OyG6NXXaPHH+RKBJDsGv4SjrD1NY1?= =?us-ascii?Q?NP+4CFZHuWi1gJqs/CCBAVtGt/YnfR3U87shbjFZazgmvCljIXabUmGoybCP?= =?us-ascii?Q?O+++mjlwEj1zYOlwHZuvYyR4fqAwXsw/9a8kLTU7AHY9EZhLRICV6DvAqpbE?= =?us-ascii?Q?/g=3D=3D?= MIME-Version: 1.0 X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: BY5PR12MB5544.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 1ae8573f-e60e-45ca-32e9-08d9f75b2a3b X-MS-Exchange-CrossTenant-originalarrivaltime: 24 Feb 2022 06:01:57.1462 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: FYi8MJ8Q9nhHwmt+W+ofXA4zXU1qRwaQvCzSXZk78pKtqlSA3H4lgZlEVKWOddyjG1m99KM2c5g42UU9PSsxoA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR12MB2896 Content-Language: en-US Content-Type: multipart/alternative; boundary="_000_BY5PR12MB554453090338EADAD6695525BA3D9BY5PR12MB5544namp_" --_000_BY5PR12MB554453090338EADAD6695525BA3D9BY5PR12MB5544namp_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hello Ard, When we had a discussion on this topic earlier, maybe a few weeks back, we = thought device memory is being accessed in a speculative manner. Our latest= debug where we focussed on MMU page tables at the time of error tells that= the issue is actually DRAM mapping in page tables getting corrupt (as per = DS-5) where descriptor for a page seems to be something garbage. What this = causes is that a valid input address in DRAM may get translated to an addre= ss in a different region of DRAM or some address in device memory. When I invalidate the instruction cache after enabling MMUs, this issue see= ms to be getting resolved. Again, I am not saying this is a fix but this is= something that solves the issue while we are looking for suggestions from = you for a proper fix. I have tried to summarize the problem based on the latest findings a few em= ails down the trail if you want to have a look at that again. Thanks Ashish ________________________________ From: Ard Biesheuvel Sent: Wednesday, February 23, 2022 3:54 PM To: Ashish Singhal Cc: edk2-devel-groups-io ; Marc Zyngier ; Sami Mujawar ; Ard Biesheuvel ; Leif Lindholm Subject: Re: [edk2-devel] [PATCH] ArmPkg: Invalidate Instruction Cache On M= MU Enable External email: Use caution opening links or attachments On Wed, 23 Feb 2022 at 19:14, Ashish Singhal wrot= e: > > Ard, > > During PrePi, I setup the initial memory map by calling into ArmConfigure= Mmu function with my memory table where device memory regions have attribut= e of ARM_MEMORY_REGION_ATTRIBUTE_DEVICE and DRAM regions have attribute of = ARM_MEMORY_REGION_ATTRIBUTE_WRITE_BACK. > > For device memory, XN bit is set by ArmMemoryAttributeToPageAttribute fun= ction. After PrePi, when I add a region of memory to device memory from a D= XE driver, I call gDS->AddMemorySpace with EfiGcdMemoryTypeMemoryMappedIo a= nd EFI_MEMORY_UC | EFI_MEMORY_RUNTIME followed by gDS->SetMemorySpaceAttrib= utes with EFI_MEMORY_UC. > > Please let me know in case I have still not understood your question. > This all looks ok. But the real question is whether the address that the speculative access targets is mapped using the XN attribute or not, so you will need to find a way to check that. So there are a couple of options: - The XN attribute is set correctly, but the CPU is speculatively fetching instructions anyway. This would imply a severe hardware bug, and flushing the I-cache is unlikely to make a difference. - The speculative access is not the result of an instruction fetch, in which case I-cache maintenance is unlikely to help either. - The XN bit is not being set correctly, and so the MMU code needs to be fi= xed. Papering over this by adding I-cache maintenance doesn't seem the best course of action tbh. -- Ard. --_000_BY5PR12MB554453090338EADAD6695525BA3D9BY5PR12MB5544namp_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable
Hello Ard,

When we had a discussion on this topic earli= er, maybe a few weeks back, we thought device memory is being accessed in a= speculative manner. Our latest debug where we focussed on MMU page tables at the time of error tells that the i= ssue is actually DRAM mapping in page tables getting corrupt (as per DS-5) = where descriptor for a page seems to be something garbage. What this causes= is that a valid input address in DRAM may get translated to an address in a different region of DRAM or som= e address in device memory.

When I invalidate the instruction cache afte= r enabling MMUs, this issue seems to be getting resolved. Again, I am not s= aying this is a fix but this is something that solves the issue while we are looking for suggestions from you for a = proper fix.

I have tried to summarize the problem based = on the latest findings a few emails down the trail if you want to have a lo= ok at that again.

Thanks
Ashish
From: Ard Biesheuvel <ar= db@kernel.org>
Sent: Wednesday, February 23, 2022 3:54 PM
To: Ashish Singhal <ashishsingha@nvidia.com>
Cc: edk2-devel-groups-io <devel@edk2.groups.io>; Marc Zyngier = <maz@kernel.org>; Sami Mujawar <sami.mujawar@arm.com>; Ard Bies= heuvel <ardb+tianocore@kernel.org>; Leif Lindholm <quic_llindhol@q= uicinc.com>
Subject: Re: [edk2-devel] [PATCH] ArmPkg: Invalidate Instruction Cac= he On MMU Enable
 
External email: Use caution opening links or attac= hments


On Wed, 23 Feb 2022 at 19:14, Ashish Singhal <ashishsingha@nvidia.com>= ; wrote:
>
> Ard,
>
> During PrePi, I setup the initial memory map by calling into ArmConfig= ureMmu function with my memory table where device memory regions have attri= bute of ARM_MEMORY_REGION_ATTRIBUTE_DEVICE and DRAM regions have attribute = of ARM_MEMORY_REGION_ATTRIBUTE_WRITE_BACK.
>
> For device memory, XN bit is set by ArmMemoryAttributeToPageAttribute = function. After PrePi, when I add a region of memory to device memory from = a DXE driver, I call gDS->AddMemorySpace with EfiGcdMemoryTypeMemoryMapp= edIo and EFI_MEMORY_UC | EFI_MEMORY_RUNTIME followed by gDS->SetMemorySpaceAttributes with EFI_MEMORY_UC.
>
> Please let me know in case I have still not understood your question.<= br> >

This all looks ok. But the real question is whether the address that
the speculative access targets is mapped using the XN attribute or
not, so you will need to find a way to check that.

So there are a couple of options:
- The XN attribute is set correctly, but the CPU is speculatively
fetching instructions anyway. This would imply a severe hardware bug,
and flushing the I-cache is unlikely to make a difference.
- The speculative access is not the result of an instruction fetch, in
which case I-cache maintenance is unlikely to help either.
- The XN bit is not being set correctly, and so the MMU code needs to be fi= xed.

Papering over this by adding I-cache maintenance doesn't seem the best
course of action tbh.

--
Ard.
--_000_BY5PR12MB554453090338EADAD6695525BA3D9BY5PR12MB5544namp_--