From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from NAM12-BN8-obe.outbound.protection.outlook.com (NAM12-BN8-obe.outbound.protection.outlook.com [40.107.237.62]) by mx.groups.io with SMTP id smtpd.web09.9765.1645976197567087801 for ; Sun, 27 Feb 2022 07:36:38 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@nvidia.com header.s=selector2 header.b=n+5l8T95; spf=permerror, err=parse error for token &{10 18 %{i}._ip.%{h}._ehlo.%{d}._spf.vali.email}: invalid domain name (domain: nvidia.com, ip: 40.107.237.62, mailfrom: ashishsingha@nvidia.com) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=amOU5/4hPsRBQj845ZA6EgLILP9wMdRfkuYpD4Lu+oj2Px1ZIfgQMSf87rJcLyzPJSJ3sc//9o1j0Mz4N3nrJoJNhV0PMQrEiDgC7Bv9ZZ9KqiuGADgjmGRKuVi+ga3VOFvSNSZpu5FSeDsuYgy2SRYNrDAa+Vgmv7ovBxpj0uhXvC3sLfYJUPAyEKE7mZ8StUFmpikob3H8bDv7r1z7TCWF0v49RfNG7vC32PsuKiLu9Y/OSKaQ1gF06qz2fIbhO98iYV7R1Pg0vZO7aGwnhSmhBGkhy8PtSCsd5/+mJFapKjSWH3Yoe0F5TJ0fzvHxqIIsEuocIC9iVmxUsLzaiA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=JVwjlxuPVUyJtBTpqcUdqLRRsyKD7EOXLPr4J7oCwW4=; b=GGLqsSg6seZIaHJSFUCuay4oraS7uBII4Ut8MczwlQWe/IZXwVynjKUUEGp42q9zH8A8ZHoQxwvxZNtbcmy5BWPRASpggEIpkYkrgoIdtuIPnx2/bWGDbmWwGs5F8jFDhKDKauZGrmyaSKcbooB1mUUuuP30C0Ji574y6G92kGXN0CFfEqLmk32zDzxKkx2Jv9Ropg6PgnjJM6N4xEZit1b1rf6MnYtBlGq+GGpnOVE0JiMZWevRSWgVR+fY82EkQGlduSyHYtg5pkVuY6edENCiY2UrzDKbZRBW+PeIOTaltgmBECuV/Tnu8VKziZcmEdDf1PyHMIR44RR2MK9cEw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nvidia.com; dmarc=pass action=none header.from=nvidia.com; dkim=pass header.d=nvidia.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=JVwjlxuPVUyJtBTpqcUdqLRRsyKD7EOXLPr4J7oCwW4=; b=n+5l8T95HcZckzWTNuNB+PQSc9LkSAiJd/QlW0P96H7CP17OvL8Fb1kpsJudsLKKQHAam2nlba9pa8asewW7PhZXOyBy3EtEkz/VwF03RI3XuMcrFTcjupciBXON71s0l7Y7KaoHd2eJWRecTdIZ7CvWdoR3Xjn/OnE/q8UoXvfabSmzzcsRokN+RZMpmG/BcAGYjbfa5N32/BE3vhkDcOU1DBIoUD6xHLIhOKOGvuGcvJUPKGBG/me0pJDshpWP6iX4tN8DBgPw91CtQjDFlRtLtkAC0yfqb4/7VIsXI0ujjAgfCu97IY1IVutRmlkvOpSTMzoEHc4J50vnNaj5pw== Received: from BY5PR12MB5544.namprd12.prod.outlook.com (2603:10b6:a03:1d9::22) by DM5PR12MB1292.namprd12.prod.outlook.com (2603:10b6:3:73::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5017.25; Sun, 27 Feb 2022 15:36:33 +0000 Received: from BY5PR12MB5544.namprd12.prod.outlook.com ([fe80::1cc3:2137:2949:b0a7]) by BY5PR12MB5544.namprd12.prod.outlook.com ([fe80::1cc3:2137:2949:b0a7%3]) with mapi id 15.20.5017.026; Sun, 27 Feb 2022 15:36:33 +0000 From: "Ashish Singhal" To: Marc Zyngier CC: "devel@edk2.groups.io" , "quic_llindhol@quicinc.com" , "ardb+tianocore@kernel.org" , "sami.mujawar@arm.com" Subject: Re: [PATCH v2] ArmPkg: Invalidate Instruction Cache On MMU Enable Thread-Topic: [PATCH v2] ArmPkg: Invalidate Instruction Cache On MMU Enable Thread-Index: AQHYKst6QKpOdXi4RUWQpgoiJ0nb3qymVzsAgAAGEiOAASxdDQ== Date: Sun, 27 Feb 2022 15:36:33 +0000 Message-ID: References: <9f95ba0bb19fd034af27f4f564e5eeff0ec19fff.1645850486.git.ashishsingha@nvidia.com> <87zgmd5nth.wl-maz@kernel.org> In-Reply-To: Accept-Language: en-US X-Mentions: maz@kernel.org,ardb+tianocore@kernel.org X-MS-Has-Attach: X-MS-TNEF-Correlator: suggested_attachment_session_id: 6bf1baea-b24d-f418-9d8f-321a38b11f7e authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nvidia.com; x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 4c8cfcc2-0b87-4f5a-5ec3-08d9fa06ef03 x-ms-traffictypediagnostic: DM5PR12MB1292:EE_ x-microsoft-antispam-prvs: x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: vUSD6R0BE+7hid9TzYVW9oJGKz1GK4GvQHPw7lKyrNqyUbnAG7J/dvdCZfKO22Hu3FhHxI5LI3GzGmZmB0XSvpdKTch5Z/TEHpWYwZysqtqIJRFyvB17l9hDnXQ3UYu5OUKNMPeG3DD9mGSDes9v2HKk3NCfvsFyGs8N2ClmHCT9UrK8bxKr6co0vvneHxD4JpBMZpHIycyZIg1YL4nuZ3ldd7W4qx0L5d3AhB+fEKuiDJ8vXPT2KrLNQCAom4e9jI6yggljtnjT3A9/Rc8DeY1o/nU7LusDnvT5oOM2CE8DgAHnniAKjVVesF0UuWr5AUD517Tsf0WPFBej5X26AwMgOiVTuzy1kcSk8muf1j6btsFEu2MmiwX2t215fLQ5z5ZbtB8b0BBS4V0eP0amRGv1C5B0p1+2QJpJRCBXDcknEh5nU69rzjWY6DsL11TEP3mWpdCPksXouBIfUJ26DlIkMYJjunlVM14AxIEYrvl/Bx6rsTE+f9SXPAwWNmN1bQxGiHBW/ZqC5RsRND3HoYjhYhnk/Rf4qUsnYsBAphbEEdxtGR7OjAO4atSpuAv4KzuPU2eH0mojKOp94kGrl/8w59zziuPohszkH2N73soTlcFmnEEzq8EuqlBe6A1rCedPBLGrQRnZO0qjgMI7wAvRBC9nNb5YoCFjqHTFdbrVfrNICNqyO4062uWh6vRzi8Tjq9/TrE85UNmNCzQDtg== x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BY5PR12MB5544.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230001)(4636009)(366004)(54906003)(316002)(6916009)(83380400001)(53546011)(86362001)(8936002)(186003)(26005)(5660300002)(52536014)(66446008)(4326008)(2906002)(76116006)(33656002)(66946007)(19627405001)(122000001)(66556008)(8676002)(38070700005)(9686003)(7696005)(6506007)(38100700002)(66476007)(71200400001)(508600001)(55016003)(64756008);DIR:OUT;SFP:1101; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?Windows-1252?Q?JAklZ+r9NUhuPPy/Nztm83fRFkI4izR0VwUigoYYaiQkx11CIaVxiJLM?= =?Windows-1252?Q?xJximogCRHWqzUnpEjAXDLdWKo6NR1lLwJikrTpdc4OcntevY1aQS7BB?= =?Windows-1252?Q?y/F88wy75dj/52067lN3KTUh4R+2KEeHJ3qvumfRzvKNid2IuZNU3SOz?= =?Windows-1252?Q?b1meIiVM1BpE7/mip6BYKrq/Ti10MQm9lEe5SwcTbikdfHlFq8XQEt6P?= =?Windows-1252?Q?uSHivlnfpd3wzIPhad/oPxqMzuzd4hyBSHQV65TUZMajapUFWhLEWr5B?= =?Windows-1252?Q?PG2vswnj0xiBO0sgnWxT/z0RJN2M5al9OtRWz8rfuzoVdHtzw8vCHF1O?= =?Windows-1252?Q?Tc1/iy1gGX9nJ+kq2B+QSpljvRS1WXAB//f4hxmUDAdIP79sPBuNiD3X?= =?Windows-1252?Q?z2BYuajr/8Q+T+agZkzOVWVop8Rjzui3U4OLuLClB1aRPV4a3bxDWDFR?= =?Windows-1252?Q?SDhY/Mc2f4+i5KVx91VNNJkFeh9uIBzEIZzW/JM4mAO9Yk8iJf/1s97r?= =?Windows-1252?Q?sIC9p4lEQMUTZES/0CzKJNrVR/cFho4P1pjjZlJUBkkp4V5x4ntnXXUp?= =?Windows-1252?Q?dM5jneWI8kYhk1iC3zQcbEvM2KP7EXAyEnz76uBJlJB1lWWG/ZDFaDFB?= =?Windows-1252?Q?FzKr2Wk9ufIiIFWdLJvr5bFaJwyA6+9QnA0gJTbFjbD937r53B3A2XUl?= =?Windows-1252?Q?c4iGOMVUmNCdHspSPb6GBtam8CQ6wQWlZw1QDEVcJ/vPj8kk0SscDTnn?= =?Windows-1252?Q?xQf5TxkIwul6WBbHpmRxJGb5UQz8JgR9T8M025HB6C2sBjzMWe1MAjdd?= =?Windows-1252?Q?qWn057+66NSO2TdOZMETweMg9NRkLsFdBzUOMPwj2Gp8rS3Hb3+CvLx2?= =?Windows-1252?Q?X+6P3sU/bgG9NI+AzuRZoU3QLbcjAulOPtZm2bYdOiHoSYALuVHidqLB?= =?Windows-1252?Q?Y7Rbq7pow6l85nVaga1+xniDHdnRGxgdPHXAMU+YLeNKRJTD0mgi0HHO?= =?Windows-1252?Q?hEs8AK+X5KqdAgTG9LRRO8DfdNQM6RSYoRjk74ALmA9nFeHKyFsw+cn5?= =?Windows-1252?Q?Ha+fvZab2f8n9Ox+sI+eaAGha8a1MUXG04RoclblbJkC0i+k//vH0lhS?= =?Windows-1252?Q?FHruR5ARj+SPYbsq/tVU5n9hkaDivq6amFKlQI51cuXBjTAVqheAERhq?= =?Windows-1252?Q?BIWMHS69nox5YG6ZgMMeJINhu1mC9A1i5rUKYWvg9kJiosuGwkA6+0k0?= =?Windows-1252?Q?BE+1UQwWIddehMnr/be2HJKB4siceMrNDwzYk4/UMtwKe/Dtnz4oEUxO?= =?Windows-1252?Q?vrQFM0XbI10bFInPj5GVz+DidDupecU0LfS/Z2l2y4ChMx6j/tJLED1g?= =?Windows-1252?Q?L36QTJ0Xv7TPOkZcALFNlApCf0cRgDx3qlMXe91uiEckorYRUpOfkoSn?= =?Windows-1252?Q?1TBSJ/lziqiSIc7rCwAc6I56L135hT5HSZNGJcpX3Oo3cWSEmlfD/iw/?= =?Windows-1252?Q?W8olUAsL+/c0xeXXhcb10NGEWZQag0ctmRQTb34o4+6T05zs/1eb3WiE?= =?Windows-1252?Q?9Jt+BIWZIyzGT9NdfYXSJyah/tLKfoKRyqwCXZDC5rBGLVM+uHfDaNqT?= =?Windows-1252?Q?WU7nFI0LU4BitlzuFo5w/31PTJwfhBGcrv4ZzTXEtVPz7KjDfHwQMOOV?= =?Windows-1252?Q?TDIjPozGbJVb3Rzh30ROW4cO+GINCwuxXQCuHM9fxcEhFlrbT99PVA?= =?Windows-1252?Q?=3D=3D?= MIME-Version: 1.0 X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: BY5PR12MB5544.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 4c8cfcc2-0b87-4f5a-5ec3-08d9fa06ef03 X-MS-Exchange-CrossTenant-originalarrivaltime: 27 Feb 2022 15:36:33.5265 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: G0yDaDQuUSHS3+L6aNiTpoA58BvzHOucaidCNPVFWrIHmaMrCDzGSQfj/luPj6ZystjIj8Sjb0k3fpiIdICMDA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5PR12MB1292 Content-Language: en-US Content-Type: multipart/alternative; boundary="_000_BY5PR12MB55445ABEBD23846AE84E3DC6BA009BY5PR12MB5544namp_" --_000_BY5PR12MB55445ABEBD23846AE84E3DC6BA009BY5PR12MB5544namp_ Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable @Marc Zyngier @ardb+tianocore@kernel.org I am going to look at things a bit more deeply and t= ry to find out the exact scenario when the issue is occurring and seek help= based on that. Meanwhile, you can ignore both the patches I have sent for = this issue till now. Thanks Ashish ________________________________ From: Ashish Singhal Sent: Saturday, February 26, 2022 2:48 PM To: Marc Zyngier Cc: devel@edk2.groups.io ; quic_llindhol@quicinc.com = ; ardb+tianocore@kernel.org ; sami.mujawar@arm.com Subject: Re: [PATCH v2] ArmPkg: Invalidate Instruction Cache On MMU Enable @Marc Zyngier Thanks for your response. I have provi= ded some comments inline. Please note that I am still not saying what I am = doing is a solution and I could still just be masking the problem at hand. = Please provide me with some suggestions that I can try for the problem bei= ng discussed on the email chain for the first version of this patch where y= ou are already there. Thanks Ashish ________________________________ From: Marc Zyngier Sent: Saturday, February 26, 2022 2:18 PM To: Ashish Singhal Cc: devel@edk2.groups.io ; quic_llindhol@quicinc.com = ; ardb+tianocore@kernel.org ; sami.mujawar@arm.com Subject: Re: [PATCH v2] ArmPkg: Invalidate Instruction Cache On MMU Enable External email: Use caution opening links or attachments On Sat, 26 Feb 2022 04:43:37 +0000, Ashish Singhal wrote: > > Even with MMU turned off, instruction cache can speculate > and fetch instructions. This can cause a crash if region > being executed has been modified recently. With this patch, Modified by what? [Singhal, Ashish]: Modified by MMU code in terms of their memory attributes= . > we ensure that instruction cache is invalidated right after > MMU has been enabled and any potentially stale instruction > fetched earlier has been discarded. > > This is specially helpful when the memory attributes of a > region in MMU are being changed and some instructions Changed from what to what else? Are you concerned with the content of the memory being changed? Or by the attribute being changed? Or both? [Singhal, Ashish]: I am concerned with the attributes being changed. UEFI drivers are dispatched in RWE memory and then memory attributes are changed to ROE for the code section and RWnE for the data section. > operating on the region are prefetched in the instruction > cache. I don't see how this fixes anything. Yes, speculation occurs. But if your icache contains crap, how is it safe to first enable the MMU first and then nuke the icache? You could well be executing garbage at that point. Worse case, and assuming that you have an aliasing VIVT icache, this will invalidate fetches that would alias with the layout of the memory once the MMU is on. But as far as I know, EDK2 is entirely identity mapped. I also don't think it uses instruction patching. Finally, if you see speculative accesses on regions that shouldn't be accessed as such, it could well be because the code is placed too close to such a region, as mentioned in the ARM ARM (DDI0487H_a, page D5-4828): Behavior of instruction fetches when all associated stages of translation are disabled [...] To ensure architectural compliance, software must ensure that both of the following apply: =95 Instructions that will be executed when all associated stages of address translation are disabled are located in blocks of the address space, of the translation granule size, that contain only memory that is tolerant to speculative accesses. =95 Each block of the address space, of the translation granule size, that immediately follows a similar block that holds instructions that will be executed when all associated stages address translation are disabled, contains only memory that is tolerant to speculative accesses. [Singhal, Ashish]: I do not think this is an issue here. Thanks, M -- Without deviation from the norm, progress is not possible. --_000_BY5PR12MB55445ABEBD23846AE84E3DC6BA009BY5PR12MB5544namp_ Content-Type: text/html; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable
@Marc Zyngier @ardb+tianocore@kernel.org I am going to look at things a bit more deeply and try to find out the exact= scenario when the issue is occurring and seek help based on that. Meanwhil= e, you can ignore both the patches I have sent for this issue till now.

Thanks
Ashish
From: Ashish Singhal <as= hishsingha@nvidia.com>
Sent: Saturday, February 26, 2022 2:48 PM
To: Marc Zyngier <maz@kernel.org>
Cc: devel@edk2.groups.io <devel@edk2.groups.io>; quic_llindhol= @quicinc.com <quic_llindhol@quicinc.com>; ardb+tianocore@kernel.org &= lt;ardb+tianocore@kernel.org>; sami.mujawar@arm.com <sami.mujawar@arm= .com>
Subject: Re: [PATCH v2] ArmPkg: Invalidate Instruction Cache On MMU = Enable
 
@Marc Zyngier Thanks= for your response. I have provided some comments inline. Please note that = I am still not saying what I am doing is a solution and I could still just be masking the problem at hand.  Plea= se provide me with some suggestions that I can try for the problem being di= scussed on the email chain for the first version of this patch where you ar= e already there.

Thanks
Ashish

From: Marc Zyngier <ma= z@kernel.org>
Sent: Saturday, February 26, 2022 2:18 PM
To: Ashish Singhal <ashishsingha@nvidia.com>
Cc: devel@edk2.groups.io <devel@edk2.groups.io>; quic_llindhol= @quicinc.com <quic_llindhol@quicinc.com>; ardb+tianocore@kernel.org &= lt;ardb+tianocore@kernel.org>; sami.mujawar@arm.com <sami.mujawar@arm= .com>
Subject: Re: [PATCH v2] ArmPkg: Invalidate Instruction Cache On MMU = Enable
 
External email: Use caution opening links or att= achments


On Sat, 26 Feb 2022 04:43:37 +0000,
Ashish Singhal <ashishsingha@nvidia.com> wrote:
>
> Even with MMU turned off, instruction cache can speculate
> and fetch instructions. This can cause a crash if region
> being executed has been modified recently. With this patch,

Modified by what?
[Singhal, Ashish]: Modified by MMU code in terms= of their memory attributes.

> we ensure that instruction cache is invalidated right after
> MMU has been enabled and any potentially stale instruction
> fetched earlier has been discarded.
>
> This is specially helpful when the memory attributes of a
> region in MMU are being changed and some instructions

Changed from what to what else? Are you concerned with the content of
the memory being changed? Or by the attribute being changed? Or both?
[Singhal, Ashish]: I am concerned with the attr= ibutes being changed. UEFI
drivers are dispatched in RWE memory and then m= emory attributes are
changed to ROE for the code section and RWnE fo= r the data section.

> operating on the region are prefetched in the instruction
> cache.

I don't see how this fixes anything. Yes, speculation occurs. But if
your icache contains crap, how is it safe to first enable the MMU
first and then nuke the icache? You could well be executing garbage at
that point.

Worse case, and assuming that you have an aliasing VIVT icache, this
will invalidate fetches that would alias with the layout of the memory
once the MMU is on. But as far as I know, EDK2 is entirely identity
mapped. I also don't think it uses instruction patching.

Finally, if you see speculative accesses on regions that shouldn't be
accessed as such, it could well be because the code is placed too
close to such a region, as mentioned in the ARM ARM (DDI0487H_a, page
D5-4828):

<quote>
Behavior of instruction fetches when all associated stages of
translation are disabled

[...]

To ensure architectural compliance, software must ensure that both of
the following apply:

=95 Instructions that will be executed when all associated stages of
  address translation are disabled are located in blocks of the
  address space, of the translation granule size, that contain only   memory that is tolerant to speculative accesses.

=95 Each block of the address space, of the translation granule size,
  that immediately follows a similar block that holds instructions
  that will be executed when all associated stages address translation=
  are disabled, contains only memory that is tolerant to speculative   accesses.
</quote>

[Singhal, Ashish]: I do not think this is an issue here.

Thanks,

        M

--
Without deviation from the norm, progress is not possible.
--_000_BY5PR12MB55445ABEBD23846AE84E3DC6BA009BY5PR12MB5544namp_--