From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.158.5]) by mx.groups.io with SMTP id smtpd.web10.652.1614797935481623881 for ; Wed, 03 Mar 2021 10:58:55 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@ibm.com header.s=pp1 header.b=LIPSgi3p; spf=none, err=permanent DNS error (domain: linux.vnet.ibm.com, ip: 148.163.158.5, mailfrom: dovmurik@linux.vnet.ibm.com) Received: from pps.filterd (m0098416.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.16.0.43/8.16.0.43) with SMTP id 123IXZ72180838; Wed, 3 Mar 2021 13:58:53 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=subject : to : cc : references : from : message-id : date : mime-version : in-reply-to : content-type : content-transfer-encoding; s=pp1; bh=wFvVNbQMg4JCtWnvAiyNtdNTZTeEIFbHQmhToxmX3Is=; b=LIPSgi3pFD17HwSzj6+dhu5BF7QG2haHSjVtCpx38/ynCqVcSJ2wJ1rD7YYGOq6Y7d5z /4RdxRQC2sI1buKAmvybov0XFazv/ud9a23aur0qXTv4L/XELzFOuENgCuz352H54dgJ tmT0Aa7DUok6869sSRIvjhyIA+lrtb2COtF2P178O4rTJCkdNW9qVbCY64KMgpc7GnVf qW/9dX8v+zYovtyQBDcIghIlg5R/rJD+Sa1w81uclWv/KjhwOkipbS9sA0OLNJBqwY9V TOJxtiwxEjZ3WiJgFM2kVwY/Es95CJhFLk3iCPl0tnpF9q2OtnueRGqpqANKgjaObkVc Fg== Received: from ppma03ams.nl.ibm.com (62.31.33a9.ip4.static.sl-reverse.com [169.51.49.98]) by mx0b-001b2d01.pphosted.com with ESMTP id 372f82t4v6-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 03 Mar 2021 13:58:53 -0500 Received: from pps.filterd (ppma03ams.nl.ibm.com [127.0.0.1]) by ppma03ams.nl.ibm.com (8.16.0.42/8.16.0.42) with SMTP id 123Ivfmc022676; Wed, 3 Mar 2021 18:58:51 GMT Received: from b06cxnps4076.portsmouth.uk.ibm.com (d06relay13.portsmouth.uk.ibm.com [9.149.109.198]) by ppma03ams.nl.ibm.com with ESMTP id 371162j3q0-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Wed, 03 Mar 2021 18:58:51 +0000 Received: from d06av26.portsmouth.uk.ibm.com (d06av26.portsmouth.uk.ibm.com [9.149.105.62]) by b06cxnps4076.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 123IwmoC31785328 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 3 Mar 2021 18:58:48 GMT Received: from d06av26.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 62539AE045; Wed, 3 Mar 2021 18:58:48 +0000 (GMT) Received: from d06av26.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 60764AE04D; Wed, 3 Mar 2021 18:58:45 +0000 (GMT) Received: from [9.160.115.38] (unknown [9.160.115.38]) by d06av26.portsmouth.uk.ibm.com (Postfix) with ESMTP; Wed, 3 Mar 2021 18:58:45 +0000 (GMT) Subject: Re: [RFC PATCH 11/14] OvmfPkg/AmdSev: Build page table for migration handler To: Ashish Kalra , Tobin Feldman-Fitzthum Cc: devel@edk2.groups.io, Tobin Feldman-Fitzthum , James Bottomley , Hubertus Franke , Brijesh Singh , Jon Grimm , Tom Lendacky References: <20210302204839.82042-1-tobin@linux.ibm.com> <20210302204839.82042-12-tobin@linux.ibm.com> <20210303163202.GA31638@ashkalra_ubuntu_server> From: "Dov Murik" Message-ID: Date: Wed, 3 Mar 2021 20:58:43 +0200 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.8.0 MIME-Version: 1.0 In-Reply-To: <20210303163202.GA31638@ashkalra_ubuntu_server> X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.369,18.0.761 definitions=2021-03-03_06:2021-03-03,2021-03-03 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 phishscore=0 spamscore=0 mlxscore=0 malwarescore=0 suspectscore=0 priorityscore=1501 impostorscore=0 adultscore=0 clxscore=1015 bulkscore=0 lowpriorityscore=0 mlxlogscore=999 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2103030129 Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit On 03/03/2021 18:32, Ashish Kalra wrote: > On Tue, Mar 02, 2021 at 03:48:36PM -0500, Tobin Feldman-Fitzthum wrote: >> From: Dov Murik >> >> The migration handler builds its own page tables and switches >> to them. The MH pagetables are reserved as runtime memory. >> >> When the hypervisor asks the MH to import/export a page, the HV >> writes the guest physical address of the page in question to the >> mailbox. The MH uses an identity mapping so that it can read/write >> whatever GPA is requested by the HV. The hypervisor only asks the >> MH to import/export encrypted pages. Thus, the C-Bit can be set >> for every page in the identity map. >> >> The MH also needs to read shared pages, such as the mailbox. >> These are mapped at an offset. The offset must be added to >> the physical address before it can be resolved. >> >> Signed-off-by: Tobin Feldman-Fitzthum >> Signed-off-by: Dov Murik >> --- >> .../ConfidentialMigrationDxe.inf | 1 + >> .../ConfidentialMigration/VirtualMemory.h | 177 ++++++++++++++++++ >> .../ConfidentialMigrationDxe.c | 88 ++++++++- >> 3 files changed, 265 insertions(+), 1 deletion(-) >> create mode 100644 OvmfPkg/AmdSev/ConfidentialMigration/VirtualMemory.h >> >> diff --git a/OvmfPkg/AmdSev/ConfidentialMigration/ConfidentialMigrationDxe.inf b/OvmfPkg/AmdSev/ConfidentialMigration/ConfidentialMigrationDxe.inf >> index 49457d5d17..8dadfd1d13 100644 >> --- a/OvmfPkg/AmdSev/ConfidentialMigration/ConfidentialMigrationDxe.inf >> +++ b/OvmfPkg/AmdSev/ConfidentialMigration/ConfidentialMigrationDxe.inf >> @@ -15,6 +15,7 @@ >> >> [Sources] >> ConfidentialMigrationDxe.c >> + VirtualMemory.h >> >> [Packages] >> MdePkg/MdePkg.dec >> diff --git a/OvmfPkg/AmdSev/ConfidentialMigration/VirtualMemory.h b/OvmfPkg/AmdSev/ConfidentialMigration/VirtualMemory.h >> new file mode 100644 >> index 0000000000..c50cb64c63 >> --- /dev/null >> +++ b/OvmfPkg/AmdSev/ConfidentialMigration/VirtualMemory.h >> @@ -0,0 +1,177 @@ >> +/** @file >> + Virtual Memory Management Services to set or clear the memory encryption bit >> + Copyright (c) 2006 - 2016, Intel Corporation. All rights reserved.
>> + Copyright (c) 2017, AMD Incorporated. All rights reserved.
>> + SPDX-License-Identifier: BSD-2-Clause-Patent >> + Code is derived from OvmfPkg/Library/BaseMemEncryptSevLib/X64/VirtualMemory.h >> + >> +**/ >> + >> +#ifndef __VIRTUAL_MEMORY__ >> +#define __VIRTUAL_MEMORY__ >> + >> +#include >> +#include >> +#include >> +#include >> +#include >> +#include >> + >> +#define SYS_CODE64_SEL 0x38 >> + >> +#pragma pack(1) >> + >> +// >> +// Page-Map Level-4 Offset (PML4) and >> +// Page-Directory-Pointer Offset (PDPE) entries 4K & 2MB >> +// >> + >> +typedef union { >> + struct { >> + UINT64 Present:1; // 0 = Not present in memory, >> + // 1 = Present in memory >> + UINT64 ReadWrite:1; // 0 = Read-Only, 1= Read/Write >> + UINT64 UserSupervisor:1; // 0 = Supervisor, 1=User >> + UINT64 WriteThrough:1; // 0 = Write-Back caching, >> + // 1 = Write-Through caching >> + UINT64 CacheDisabled:1; // 0 = Cached, 1=Non-Cached >> + UINT64 Accessed:1; // 0 = Not accessed, >> + // 1 = Accessed (set by CPU) >> + UINT64 Reserved:1; // Reserved >> + UINT64 MustBeZero:2; // Must Be Zero >> + UINT64 Available:3; // Available for use by system software >> + UINT64 PageTableBaseAddress:40; // Page Table Base Address >> + UINT64 AvabilableHigh:11; // Available for use by system software >> + UINT64 Nx:1; // No Execute bit >> + } Bits; >> + UINT64 Uint64; >> +} PAGE_MAP_AND_DIRECTORY_POINTER; >> + >> +// >> +// Page Table Entry 4KB >> +// >> +typedef union { >> + struct { >> + UINT64 Present:1; // 0 = Not present in memory, >> + // 1 = Present in memory >> + UINT64 ReadWrite:1; // 0 = Read-Only, 1= Read/Write >> + UINT64 UserSupervisor:1; // 0 = Supervisor, 1=User >> + UINT64 WriteThrough:1; // 0 = Write-Back caching, >> + // 1 = Write-Through caching >> + UINT64 CacheDisabled:1; // 0 = Cached, 1=Non-Cached >> + UINT64 Accessed:1; // 0 = Not accessed, >> + // 1 = Accessed (set by CPU) >> + UINT64 Dirty:1; // 0 = Not Dirty, 1 = written by >> + // processor on access to page >> + UINT64 PAT:1; // >> + UINT64 Global:1; // 0 = Not global page, 1 = global page >> + // TLB not cleared on CR3 write >> + UINT64 Available:3; // Available for use by system software >> + UINT64 PageTableBaseAddress:40; // Page Table Base Address >> + UINT64 AvabilableHigh:11; // Available for use by system software >> + UINT64 Nx:1; // 0 = Execute Code, >> + // 1 = No Code Execution >> + } Bits; >> + UINT64 Uint64; >> +} PAGE_TABLE_4K_ENTRY; >> + >> +// >> +// Page Table Entry 2MB >> +// >> +typedef union { >> + struct { >> + UINT64 Present:1; // 0 = Not present in memory, >> + // 1 = Present in memory >> + UINT64 ReadWrite:1; // 0 = Read-Only, 1= Read/Write >> + UINT64 UserSupervisor:1; // 0 = Supervisor, 1=User >> + UINT64 WriteThrough:1; // 0 = Write-Back caching, >> + // 1=Write-Through caching >> + UINT64 CacheDisabled:1; // 0 = Cached, 1=Non-Cached >> + UINT64 Accessed:1; // 0 = Not accessed, >> + // 1 = Accessed (set by CPU) >> + UINT64 Dirty:1; // 0 = Not Dirty, 1 = written by >> + // processor on access to page >> + UINT64 MustBe1:1; // Must be 1 >> + UINT64 Global:1; // 0 = Not global page, 1 = global page >> + // TLB not cleared on CR3 write >> + UINT64 Available:3; // Available for use by system software >> + UINT64 PAT:1; // >> + UINT64 MustBeZero:8; // Must be zero; >> + UINT64 PageTableBaseAddress:31; // Page Table Base Address >> + UINT64 AvabilableHigh:11; // Available for use by system software >> + UINT64 Nx:1; // 0 = Execute Code, >> + // 1 = No Code Execution >> + } Bits; >> + UINT64 Uint64; >> +} PAGE_TABLE_ENTRY; >> + >> +// >> +// Page Table Entry 1GB >> +// >> +typedef union { >> + struct { >> + UINT64 Present:1; // 0 = Not present in memory, >> + // 1 = Present in memory >> + UINT64 ReadWrite:1; // 0 = Read-Only, 1= Read/Write >> + UINT64 UserSupervisor:1; // 0 = Supervisor, 1=User >> + UINT64 WriteThrough:1; // 0 = Write-Back caching, >> + // 1 = Write-Through caching >> + UINT64 CacheDisabled:1; // 0 = Cached, 1=Non-Cached >> + UINT64 Accessed:1; // 0 = Not accessed, >> + // 1 = Accessed (set by CPU) >> + UINT64 Dirty:1; // 0 = Not Dirty, 1 = written by >> + // processor on access to page >> + UINT64 MustBe1:1; // Must be 1 >> + UINT64 Global:1; // 0 = Not global page, 1 = global page >> + // TLB not cleared on CR3 write >> + UINT64 Available:3; // Available for use by system software >> + UINT64 PAT:1; // >> + UINT64 MustBeZero:17; // Must be zero; >> + UINT64 PageTableBaseAddress:22; // Page Table Base Address >> + UINT64 AvabilableHigh:11; // Available for use by system software >> + UINT64 Nx:1; // 0 = Execute Code, >> + // 1 = No Code Execution >> + } Bits; >> + UINT64 Uint64; >> +} PAGE_TABLE_1G_ENTRY; >> + >> +#pragma pack() >> + >> +#define IA32_PG_P BIT0 >> +#define IA32_PG_RW BIT1 >> +#define IA32_PG_PS BIT7 >> + >> +#define PAGING_PAE_INDEX_MASK 0x1FF >> + >> +#define PAGING_4K_ADDRESS_MASK_64 0x000FFFFFFFFFF000ull >> +#define PAGING_2M_ADDRESS_MASK_64 0x000FFFFFFFE00000ull >> +#define PAGING_1G_ADDRESS_MASK_64 0x000FFFFFC0000000ull >> + >> +#define PAGING_L1_ADDRESS_SHIFT 12 >> +#define PAGING_L2_ADDRESS_SHIFT 21 >> +#define PAGING_L3_ADDRESS_SHIFT 30 >> +#define PAGING_L4_ADDRESS_SHIFT 39 >> + >> +#define PAGING_PML4E_NUMBER 4 >> + >> +#define PAGETABLE_ENTRY_MASK ((1UL << 9) - 1) >> +#define PML4_OFFSET(x) ( (x >> 39) & PAGETABLE_ENTRY_MASK) >> +#define PDP_OFFSET(x) ( (x >> 30) & PAGETABLE_ENTRY_MASK) >> +#define PDE_OFFSET(x) ( (x >> 21) & PAGETABLE_ENTRY_MASK) >> +#define PTE_OFFSET(x) ( (x >> 12) & PAGETABLE_ENTRY_MASK) >> +#define PAGING_1G_ADDRESS_MASK_64 0x000FFFFFC0000000ull >> + >> +#define PAGE_TABLE_POOL_ALIGNMENT BASE_2MB >> +#define PAGE_TABLE_POOL_UNIT_SIZE SIZE_2MB >> +#define PAGE_TABLE_POOL_UNIT_PAGES \ >> + EFI_SIZE_TO_PAGES (PAGE_TABLE_POOL_UNIT_SIZE) >> +#define PAGE_TABLE_POOL_ALIGN_MASK \ >> + (~(EFI_PHYSICAL_ADDRESS)(PAGE_TABLE_POOL_ALIGNMENT - 1)) >> + >> +typedef struct { >> + VOID *NextPool; >> + UINTN Offset; >> + UINTN FreePages; >> +} PAGE_TABLE_POOL; >> + >> +#endif >> diff --git a/OvmfPkg/AmdSev/ConfidentialMigration/ConfidentialMigrationDxe.c b/OvmfPkg/AmdSev/ConfidentialMigration/ConfidentialMigrationDxe.c >> index 8402fcc4fa..3df3b09732 100644 >> --- a/OvmfPkg/AmdSev/ConfidentialMigration/ConfidentialMigrationDxe.c >> +++ b/OvmfPkg/AmdSev/ConfidentialMigration/ConfidentialMigrationDxe.c >> @@ -11,6 +11,7 @@ >> #include >> #include >> >> +#include "VirtualMemory.h" >> // >> // Functions implemented by the migration handler >> // >> @@ -43,6 +44,83 @@ typedef volatile struct { >> UINT32 done; >> } MH_COMMAND_PARAMETERS; >> >> +// >> +// Addresses for MH page table. >> +// >> +STATIC PAGE_TABLE_POOL *mPageTablePool = NULL; >> +STATIC PHYSICAL_ADDRESS mMigrationHelperPageTables = 0; >> + >> +// >> +// Offset for non-cbit mapping. >> +// >> +#define UNENC_VIRT_ADDR_BASE 0xffffff8000000000ULL >> + >> + >> +/** >> + Allocates and fills in custom page tables for Migration Handler. >> + The MH must be able to write to any encrypted page. Thus, it >> + uses an identity map where the C-bit is set for every page. The >> + HV should never ask the MH to import/export a shared page. The >> + MH must also be able to read some shared pages. The first 1GB >> + of memory is mapped at offset UNENC_VIRT_ADDR_BASE. >> + >> +**/ >> +VOID >> +PrepareMigrationHandlerPageTables ( >> + VOID >> + ) >> +{ >> + UINTN PoolPages; >> + VOID *Buffer; >> + VOID *Start; >> + PAGE_MAP_AND_DIRECTORY_POINTER *PageMapLevel4Entry; >> + PAGE_TABLE_1G_ENTRY *PageDirectory1GEntry; >> + PAGE_TABLE_1G_ENTRY *Unenc1GEntry; >> + UINT64 AddressEncMask; >> + >> + PoolPages = 1 + 10; >> + Buffer = AllocateAlignedRuntimePages (PoolPages, PAGE_TABLE_POOL_ALIGNMENT); >> + mPageTablePool = Buffer; >> + mPageTablePool->NextPool = mPageTablePool; >> + mPageTablePool->FreePages = PoolPages - 1; >> + mPageTablePool->Offset = EFI_PAGES_TO_SIZE (1); >> + >> + Start = (UINT8 *)mPageTablePool + mPageTablePool->Offset; >> + ZeroMem(Start, mPageTablePool->FreePages * EFI_PAGE_SIZE); >> + >> + AddressEncMask = 1ULL << 47; >> + > > Preferably getting the encryption bit location from SEV CPUID > information. You're right; I'll fix. > >> + PageMapLevel4Entry = Start; >> + PageDirectory1GEntry = (PAGE_TABLE_1G_ENTRY*)((UINT8*)Start + EFI_PAGE_SIZE); >> + Unenc1GEntry = (PAGE_TABLE_1G_ENTRY*)((UINT8*)Start + 2 * EFI_PAGE_SIZE); >> + >> + PageMapLevel4Entry = Start; >> + PageMapLevel4Entry += PML4_OFFSET(0x0ULL); >> + PageMapLevel4Entry->Uint64 = (UINT64)PageDirectory1GEntry | AddressEncMask | 0x23; >> + >> + PageMapLevel4Entry = Start; >> + PageMapLevel4Entry += PML4_OFFSET(UNENC_VIRT_ADDR_BASE); // should be 511 >> + PageMapLevel4Entry->Uint64 = (UINT64)Unenc1GEntry | AddressEncMask | 0x23; >> + >> + UINT64 PageAddr = 0; >> + for (int i = 0; i < 512; i++, PageAddr += SIZE_1GB) { >> + PAGE_TABLE_1G_ENTRY *e = PageDirectory1GEntry + i; >> + e->Uint64 = PageAddr | AddressEncMask | 0xe3; // 1GB page >> + } >> + > > Changing encryption attributes of a page requires to flush it from > the caches, you may need to do a clflush here. Thanks for pointing that out. I think I didn't see that in the code that builds the page tables for encrypted memory (which inspired most of this function), but maybe I missed it. Thanks, Dov > > Thanks, > Ashish > >> + UINT64 UnencPageAddr = 0; >> + Unenc1GEntry->Uint64 = UnencPageAddr | 0xe3; // 1GB page unencrypted >> + >> + mMigrationHelperPageTables = (UINT64)Start | AddressEncMask; >> +} >> + >> +VOID >> +SwitchToMigrationHelperPageTables(VOID) >> +{ >> + AsmWriteCr3(mMigrationHelperPageTables); >> +} >> + >> + >> >> VOID >> EFIAPI >> @@ -56,7 +134,12 @@ MigrationHandlerMain ( >> >> DebugPrint (DEBUG_INFO,"MIGRATION Handler Started\n"); >> >> - params_base = PcdGet32 (PcdConfidentialMigrationMailboxBase); >> + SwitchToMigrationHelperPageTables(); >> + >> + // >> + // Shared pages must be offset by UNENC_VIRT_ADDR_BASE. >> + // >> + params_base = PcdGet32 (PcdConfidentialMigrationMailboxBase) + UNENC_VIRT_ADDR_BASE; >> params = (VOID *)params_base; >> page_va = (VOID *)params_base + 0x1000; >> >> @@ -134,6 +217,8 @@ LaunchMigrationHandler ( >> >> MigrationHandlerCpuIndex = NumProc - 1; >> >> + PrepareMigrationHandlerPageTables(); >> + >> EFI_EVENT Event; >> MpProto->GetProcessorInfo (MpProto, MigrationHandlerCpuIndex, &Tcb); >> if (Tcb.StatusFlag != 7) { >> @@ -154,6 +239,7 @@ LaunchMigrationHandler ( >> if (PcdGetBool(PcdIsConfidentialMigrationTarget)) { >> DebugPrint (DEBUG_INFO,"Waiting for incoming confidential migration.\n"); >> DisableInterrupts (); >> + SwitchToMigrationHelperPageTables(); >> CpuDeadLoop (); >> } >> >> -- >> 2.20.1 >>