From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from NAM04-DM6-obe.outbound.protection.outlook.com (NAM04-DM6-obe.outbound.protection.outlook.com [40.107.102.62]) by mx.groups.io with SMTP id smtpd.web09.1144.1609886911107821029 for ; Tue, 05 Jan 2021 14:48:31 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@amdcloud.onmicrosoft.com header.s=selector2-amdcloud-onmicrosoft-com header.b=nYGISI0K; spf=permerror, err=parse error for token &{10 18 %{i}._ip.%{h}._ehlo.%{d}._spf.vali.email}: invalid domain name (domain: amd.com, ip: 40.107.102.62, mailfrom: thomas.lendacky@amd.com) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=iG1SNNmCzEC0p1m8pnHFDcqjTlXsFt5bt2eYQOOmN9DBXrrJz1vjtwPbvYOcn5IlWDGzx3jm4DdNUQHLsJ2UB63CegowBBX2/fJPWdCMAmRbtP2ymV7wZwmxUGl2iWMUOAcu7Rzzrr/7s1jIzParsH8XAQmJGZmj6bDpI4mb0hi9ymIofOGUyLHCbKYrP0njRwQQgajcHVu3va1DA/JBiKelN+uirDum9fYN6OGbZNeqa4pUqWLOLh8/5xPEUtJvH0hCY7Ffi5fbtMAdD2SoJubj37B9+AGI30hbxwhK7EugCVQjjHtQSCdc55q76ItDCN59Gq+qhLI7isOBaj9u2Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=mjrWYiTuDIGDV3meQsNvWyeurCccV5ffqDpI6Y6bHdY=; b=YhLRjMJtJ2eCwEzJHd370E6R7Im56kbgxdjC8Apd2Px82SfULfObJoc3DqGHFeASyJM1xwm/sQ73QvGM4h7ZR3W/9Z0C5/C3NeDI4Q2IoGB9Gl6uluh5nUNULemgKcJeTWYAqrx8JdZGRsqFZITJnsVB0eUbMjr9F3QQxed/KqbxpCo2R/Ft2zLm/m1WK+ueyXFhI76+sG8fy8un0nYGQTsNHZcISSH/26lWERvjFAMDJjBPxmoMq7Tkg6mIlmorl+0sThC/2Bbh7ae6H+EXvaIs3VfH12nqlOvlEBaVCCC3tL8pIwjQ1HuYJOC5bbGYQ8gd8E2xoPirYIUWIHyjqA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=amd.com; dmarc=pass action=none header.from=amd.com; dkim=pass header.d=amd.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amdcloud.onmicrosoft.com; s=selector2-amdcloud-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=mjrWYiTuDIGDV3meQsNvWyeurCccV5ffqDpI6Y6bHdY=; b=nYGISI0K+xwGOgRVvFAVemKKI/Hajecci98emgbrxbgIeL69GmYPMZIZJmznSafZ8F7uuN2zIxHqDGnQaxJMdLMEROS9MbUJXpSVactkaKFC7/b0M7olMiUbaknb9KKEL94BURBQ+ywrIm8JUEmvNF1/pe5ak4kFkZiJWHTHZzI= Authentication-Results: arm.com; dkim=none (message not signed) header.d=none;arm.com; dmarc=none action=none header.from=amd.com; Received: from DM5PR12MB1355.namprd12.prod.outlook.com (2603:10b6:3:6e::7) by DM5PR1201MB0121.namprd12.prod.outlook.com (2603:10b6:4:56::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3721.23; Tue, 5 Jan 2021 22:48:29 +0000 Received: from DM5PR12MB1355.namprd12.prod.outlook.com ([fe80::d95e:b9d:1d6a:e845]) by DM5PR12MB1355.namprd12.prod.outlook.com ([fe80::d95e:b9d:1d6a:e845%12]) with mapi id 15.20.3721.024; Tue, 5 Jan 2021 22:48:28 +0000 Subject: Re: [edk2-devel] [PATCH 06/12] OvmfPkg/AmdSevDxe: Clear encryption bit on PCIe MMCONFIG range To: Laszlo Ersek , devel@edk2.groups.io Cc: Brijesh Singh , James Bottomley , Jordan Justen , Ard Biesheuvel References: <90152d0505354d270cc3af9e5838010e4dcbe114.1608065471.git.thomas.lendacky@amd.com> <5daddb82-ad8e-7de7-49df-f3d18907bfe1@redhat.com> From: "Lendacky, Thomas" Message-ID: <946164e0-38f4-aff6-b2dc-3f2348c0d97d@amd.com> Date: Tue, 5 Jan 2021 16:48:26 -0600 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.10.0 In-Reply-To: <5daddb82-ad8e-7de7-49df-f3d18907bfe1@redhat.com> X-Originating-IP: [165.204.77.1] X-ClientProxiedBy: SN1PR12CA0089.namprd12.prod.outlook.com (2603:10b6:802:21::24) To DM5PR12MB1355.namprd12.prod.outlook.com (2603:10b6:3:6e::7) Return-Path: thomas.lendacky@amd.com MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from [10.236.30.118] (165.204.77.1) by SN1PR12CA0089.namprd12.prod.outlook.com (2603:10b6:802:21::24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3742.6 via Frontend Transport; Tue, 5 Jan 2021 22:48:27 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-HT: Tenant X-MS-Office365-Filtering-Correlation-Id: 85ba7bba-e1b2-477b-2779-08d8b1cc04b8 X-MS-TrafficTypeDiagnostic: DM5PR1201MB0121: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:10000; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: iBZc9iO4xWjHXobc3D7R3s6AQQz0WZoaTbPmGH4wgqwOR/LfHt2PGTYwuv4HBelJdpgIGz9RBQwmdkam8PH1jadPVw3J+GD2shkjlRWhQzg4ep4RciMw7n2+SflO+O/0h+u8OrCX1JiMFcpXZLf1tfyG4Vs3XcgwQ5rp+ddzcTaeBRvuRtXrOGttRaUCCKEpbHq3rS+2JmwUTOsPTsS8LMFtK/S50qVM9VWImZAWf9uGq71LxxD0VBQgCeb3lTg7fzR19mz0h3Q+JvxzZQiIxXxQK6GVPbN5PfisDsTWVV+YS/Ccs7svOWJNZ49wNjnTkzt0x9EWoOaF3kbea2+0RS3A1JiBC92LNP3zPY6seYc6BPWHAAV7eSzL8t6YUt5pq8xiD5O1d4J5bmt5bWrsg6/LCm+gmfDG5NfbHYlrTpdKGB45D0iSz1xGgof9DlKMdRkUgPrT2EvESyAutEFoHBVVeYZozlM+PnjQygxEOX7Mox4d0jRIg6AVp2yvg6vgnzF4chJDIm64spsMKs9auhh4PVVI9B2ER9eS0p8H8iDpQZ4YHv8ly8Zf5jyqzw0P X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM5PR12MB1355.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(4636009)(39860400002)(396003)(346002)(366004)(376002)(136003)(6486002)(8936002)(5660300002)(2906002)(66946007)(66476007)(956004)(966005)(8676002)(31696002)(45080400002)(478600001)(86362001)(53546011)(186003)(52116002)(36756003)(31686004)(54906003)(316002)(16526019)(16576012)(4326008)(83380400001)(66556008)(2616005)(26005)(43740500002)(45980500001);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: =?utf-8?B?WlFHUXI4ZXEvMk52bHZNTVhTTlVmd0E0VjNHMit2RHlHMTd3c1FZS05TOG14?= =?utf-8?B?VzJOa0JxRzQ5ZUU3MEE3Zk1BcnVWcnZRWmhkZFU0RUNuVTRVY0c4M2hyMWFp?= =?utf-8?B?UWVkZmRZMzN2WmpEaUgwQnZhOTloNGE5bHdHcTlaUU1ZUkpFOUo0dlgxL0Np?= =?utf-8?B?NHlJYVpUN0xqK2t0bitReitLS2xpZnlJQlRlbm1ZVWVOU2VOKzJBUUpaVUE1?= =?utf-8?B?SjRFNmlNcFFjK0FoTktrMVFxTjJWSGlEYTM5UFFDSTh6Y0g1NTBMeWp5M2ov?= =?utf-8?B?UHFMZ05nYnJrNFBRN0NwS2tSYlZ3UXpHQzRCRGVISmRhUXJ6aHhTS2c2T05s?= =?utf-8?B?WSsxYjNBWnJ6NUdpcHBRVWlWTTREUm9rSWpuRXVWTXFiZm5yU3FhVVhuakt3?= =?utf-8?B?QW84WHdjN3ROUDQzTUVrMWlhQ0N1MDE2T1U1TlVGVC8rclVLQnoxQ0Z1OWMr?= =?utf-8?B?K0Q3WFBZa2cvNFNUYjYrTFA3RTNaOS8zV2xVZjdjSXJzbTI2RlkvcXUzaTE1?= =?utf-8?B?QUx1UkRHN0pjOFQyWWlYR3ZSNlFEM1drMWV4NjJBRGswSmZqL2g1bWJ0SUVZ?= =?utf-8?B?K0FFRG1mcnBmTGRLbU43VENmQlZreEZYV25yQnZRTG5xL2cxdlVzZzZ2VG0y?= =?utf-8?B?QXc5WmVPSzlvWVNXdmhzNVQrVWh6WUVQT0xMSlIxUExWclZmdFFQOXhSdlZl?= =?utf-8?B?aGlrWER2SDA1M2k3d0RjL1A3MFdpSk5KV2ZVMUwyMk02cVJEYzhLeVQ5NjNH?= =?utf-8?B?emVqdFNEbGFSUmtDckdDdFpXN0F4MlBxVE9OWkIwOC9rQWdJaG8wL0xMMWF3?= =?utf-8?B?dkJRYkM2RTJ0aW54Y0ptSDI1cyszWGlOZnorZklCWEhHK1d2YklzU01mMmhE?= =?utf-8?B?TXUwd2VxWnpFaVByWnl5NVdtQ09taDl2RUZrRzNVSGNkcFlDRGRYS2IvRXJR?= =?utf-8?B?UmI1enIyY2lBdUF2cGhsUm1FeVNveDVqOFBFWTI2NHlEY20vbzVQWEhpUEQz?= =?utf-8?B?OFI2Wkk3VnY1Rm5WN1FsMjZrMUNIc0RYZFErMGovYnZtVm1zRHNydWxBQjVK?= =?utf-8?B?N0xQR0pVU0djSFFxVzkxZnlyOW5DOUR2RFJVWUdac1pBbEsvc3FBdURoNDZH?= =?utf-8?B?a2xOK202V0ltRmJYa3cvL0RaM0xwZzVoL2lqRjd2eWQyTzIyREJ0UTNOZGI3?= =?utf-8?B?N0FQK2ZOUER2L0xlK1JHUTg4YmlPTHUwbTdrQldtRVRRSDFFYVFraXZaaHlB?= =?utf-8?B?VGM1dWpvdGh1aTQ5d1ZuMFVyRk13UmZNc3dWamZYekdRTTkzMHUyZS9GRWd4?= =?utf-8?Q?Jvnq0A1l3nMfVfLHLEpug+7wXkQ2HRrhF+?= X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-AuthSource: DM5PR12MB1355.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 05 Jan 2021 22:48:28.3057 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-Network-Message-Id: 85ba7bba-e1b2-477b-2779-08d8b1cc04b8 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: flbFl5g4xx+kZ2ABaGJiyINZ3zS3ayWpbYrgYvrZGAFjvf02Htix/WFKPIHE6rhDyt5WQLfYj7VYxy//dMZ/Ww== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5PR1201MB0121 Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit On 1/4/21 3:04 PM, Laszlo Ersek wrote: > On 12/15/20 21:51, Lendacky, Thomas wrote: >> From: Tom Lendacky >> >> BZ: https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugzilla.tianocore.org%2Fshow_bug.cgi%3Fid%3D3108&data=04%7C01%7Cthomas.lendacky%40amd.com%7Cf35ac4fb20264b713aa108d8b0f45717%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637453910773208310%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=wZg2UIdJ%2FZ2HLGbWcfli3SVzl1cSMkyI%2FvREVldOB9M%3D&reserved=0 >> >> The PCIe MMCONFIG range should be treated as an MMIO range. However, >> there is a comment in the code explaining why AddIoMemoryBaseSizeHob() >> is not called. The AmdSevDxe walks the GCD map looking for MemoryMappedIo >> or NonExistent type memory and will clear the encryption bit for these >> ranges. >> >> Since the MMCONFIG range does not have one of these types, the encryption >> bit is not cleared for this range. Add support to detect the presence of >> the MMCONFIG range and clear the encryption bit. This will be needed for >> follow-on support that will validate MMIO under SEV-ES. >> >> Cc: Jordan Justen >> Cc: Laszlo Ersek >> Cc: Ard Biesheuvel >> Cc: Brijesh Singh >> Signed-off-by: Tom Lendacky >> --- >> OvmfPkg/AmdSevDxe/AmdSevDxe.inf | 8 +++++++- >> OvmfPkg/AmdSevDxe/AmdSevDxe.c | 20 +++++++++++++++++++- >> 2 files changed, 26 insertions(+), 2 deletions(-) >> >> diff --git a/OvmfPkg/AmdSevDxe/AmdSevDxe.inf b/OvmfPkg/AmdSevDxe/AmdSevDxe.inf >> index dd9ecc789a20..0676fcc5b6a4 100644 >> --- a/OvmfPkg/AmdSevDxe/AmdSevDxe.inf >> +++ b/OvmfPkg/AmdSevDxe/AmdSevDxe.inf >> @@ -2,7 +2,7 @@ >> # >> # Driver clears the encryption attribute from MMIO regions when SEV is enabled >> # >> -# Copyright (c) 2017, AMD Inc. All rights reserved.
>> +# Copyright (c) 2017 - 2020, AMD Inc. All rights reserved.
>> # >> # SPDX-License-Identifier: BSD-2-Clause-Patent >> # >> @@ -39,3 +39,9 @@ [Depex] >> >> [FeaturePcd] >> gUefiOvmfPkgTokenSpaceGuid.PcdSmmSmramRequire >> + >> +[FixedPcd] >> + gEfiMdePkgTokenSpaceGuid.PcdPciExpressBaseAddress >> + >> +[Pcd] >> + gUefiOvmfPkgTokenSpaceGuid.PcdOvmfHostBridgePciDevId >> diff --git a/OvmfPkg/AmdSevDxe/AmdSevDxe.c b/OvmfPkg/AmdSevDxe/AmdSevDxe.c >> index 595586617882..ed516fcdf956 100644 >> --- a/OvmfPkg/AmdSevDxe/AmdSevDxe.c >> +++ b/OvmfPkg/AmdSevDxe/AmdSevDxe.c >> @@ -4,7 +4,7 @@ >> in APRIORI. It clears C-bit from MMIO and NonExistent Memory space when SEV >> is enabled. >> >> - Copyright (c) 2017, AMD Inc. All rights reserved.
>> + Copyright (c) 2017 - 2020, AMD Inc. All rights reserved.
>> >> SPDX-License-Identifier: BSD-2-Clause-Patent >> >> @@ -17,6 +17,7 @@ >> #include >> #include >> #include >> +#include > > (1) Please keep the #include list alphabetically sorted. Will fix. > >> >> EFI_STATUS >> EFIAPI >> @@ -65,6 +66,23 @@ AmdSevDxeEntryPoint ( >> FreePool (AllDescMap); >> } >> >> + // >> + // If PCI Express is enabled, the MMCONFIG area has been reserved, rather >> + // than marked as MMIO, and so the C-bit won't be cleared by the above walk >> + // through the GCD map. Check for the MMCONFIG area and clear the C-bit for >> + // the range. >> + // >> + if (PcdGet16 (PcdOvmfHostBridgePciDevId) == INTEL_Q35_MCH_DEVICE_ID) { >> + Status = MemEncryptSevClearPageEncMask ( >> + 0, >> + FixedPcdGet64 (PcdPciExpressBaseAddress), >> + EFI_SIZE_TO_PAGES (SIZE_256MB), >> + FALSE >> + ); >> + >> + ASSERT_EFI_ERROR (Status); >> + } >> + >> // >> // When SMM is enabled, clear the C-bit from SMM Saved State Area >> // >> > > Very interesting. One wonders why, without this change, MMCONFIG > accesses work at all on SEV. > > But then... this guest phys area is not backed by RAM in the first > place. Whenever the guest accesses it, we trap to QEMU unconditionally. > And so memory encryption plays no role in practice, I must think. > > It's different for the flash, because the flash is backed by RAM, and > whether an access to it traps to QEMU or not depends on both the access > (r/w/x) and the mode the flash is in (programming mode on vs. off). > > I now wonder whether the comment in the leading context (not visible > above), namely the one that references the root bridge MMIO aperture, > from which the PCI MMIO BARs are allocated, is accurate. Perhaps that > area would work in fact even if we didn't clear the C bit for them > (considering just the accesses themselves under SEV; not SEV-ES). > > (2) Please include a sentence in the commit message about the fact that > MMCONFIG is not backed by a KVM memory slot, and so actual memory > encryption does not take place, and that's why MMCONFIG accesses do not > break currently under SEV / SEV-ES. (This is at least what I think happens.) Since that address range is marked as MMIO in the nested page tables by KVM (reserved bits set), accessing that address range will always trigger a nested page fault (NPF). For SEV, the hardware clears the encryption bit from the GPA provided for the NPF, so KVM/Qemu see the base address and everything just works. For SEV-ES, the NPF triggers a #VC. Since we run identity mapped (VA == PA), I use the virtual address in the VMGEXIT, which doesn't contain the encryption bit, so, again, everything just works. The SEV-ES check for the encryption bit being set is what uncovered this condition. I'll write that up in the commit. Thanks, Tom > > With (1) and (2) addressed: > > Reviewed-by: Laszlo Ersek > > Thanks > Laszlo >