From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from NAM11-BN8-obe.outbound.protection.outlook.com (NAM11-BN8-obe.outbound.protection.outlook.com [40.107.236.40]) by mx.groups.io with SMTP id smtpd.web10.77415.1670947461248641294 for ; Tue, 13 Dec 2022 08:04:21 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@amd.com header.s=selector1 header.b=glRWEtfm; spf=permerror, err=parse error for token &{10 18 %{i}._ip.%{h}._ehlo.%{d}._spf.vali.email}: invalid domain name (domain: amd.com, ip: 40.107.236.40, mailfrom: thomas.lendacky@amd.com) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Wy6CSbhrhRLyf7uisTJvbNLu8YGh5FGSj2JuchH8olcqEOGyykwReZWCF+ix99W1qni9iIWju2zPtfqcsFTFnJdqRU6sLRIQ2xz85k2+ODxH5bWbnU2vcnxHKITxwBg5DiKiN8ibVPeoKmXPhxSXA/RswSJId1+PJVCzPyTiP72iIiZ7UqZoaDNWm/Co0JUKictoughgQQ1D+CpbSLrSknjszSHUxv5MzlburceDLZeUDSDGA+UkJlzwDlDTZ3Vo8TNrxkVrofOp31wPCdRoRkZRWUepcZPYuefJnxRMeSUhYHYGCz6Zvo8kupR7SX/SMoq2l2KUUNLYvcadJTHHUQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=2vgngKByfDB2uROX4feyUE3t2byC7r/f6/88D/mZsvY=; b=Qzwtxv4arHz9zXEDTaVrXl2Xmw345e+imvhbWnfUVeGKSsV0d7+MuAzpelztjMtTLczjTreD3P3XjtcBMUbPu/EfGWS7joTGVlpVP37sHRjSod/Fg4MG89EiZGw2hmYHVddH8CgcSb6R4E61hIxQLw3FWQrWDH81jq2IHzw14PU7333Jzpsve4JojM63so2/V91NkQYHMMb4mDVW/mF0PmLNLSj+JDTualq/kz5BJQlWoorjZDpi9OmahIvvT8IBJqohMcIdFSE0rY1kOzfQg8wYKtd1Py3Zl0b2pb5P0EQmEhkAgF3zkMKS83Xxo8W8SZPRgbUk6ft1a5Xl7ARibg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=amd.com; dmarc=pass action=none header.from=amd.com; dkim=pass header.d=amd.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=2vgngKByfDB2uROX4feyUE3t2byC7r/f6/88D/mZsvY=; b=glRWEtfmQru1eJTDNPLOI6Q7T31T+pD7d6HHHcrBngUbqpYOepv6pLHq0M7gg6TNVKBNm9WlegCsGwHuUIRFYSvXwKIboPEsNTtocrsekMFCjWp6lh7Gdl6mfIjT3b6XWbSNX6wlRAmHPHXfvU1XYAtk+0u6q7Yh2KzwbNI6cR0= Authentication-Results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=amd.com; Received: from DM4PR12MB5229.namprd12.prod.outlook.com (2603:10b6:5:398::12) by IA0PR12MB7602.namprd12.prod.outlook.com (2603:10b6:208:43a::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.5880.19; Tue, 13 Dec 2022 16:04:18 +0000 Received: from DM4PR12MB5229.namprd12.prod.outlook.com ([fe80::8200:4042:8db4:63d7]) by DM4PR12MB5229.namprd12.prod.outlook.com ([fe80::8200:4042:8db4:63d7%4]) with mapi id 15.20.5880.019; Tue, 13 Dec 2022 16:04:17 +0000 Message-ID: Date: Tue, 13 Dec 2022 10:04:15 -0600 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.4.2 Subject: Re: [PATCH V2 1/4] OvmfPkg/IoMmuDxe: Reserve shared memory region for DMA operation To: Min Xu , devel@edk2.groups.io Cc: Erdem Aktas , James Bottomley , Jiewen Yao , Gerd Hoffmann References: <20221213054824.53-1-min.m.xu@intel.com> <20221213054824.53-2-min.m.xu@intel.com> From: "Lendacky, Thomas" In-Reply-To: <20221213054824.53-2-min.m.xu@intel.com> X-ClientProxiedBy: BLAPR03CA0034.namprd03.prod.outlook.com (2603:10b6:208:32d::9) To DM4PR12MB5229.namprd12.prod.outlook.com (2603:10b6:5:398::12) Return-Path: Thomas.Lendacky@amd.com MIME-Version: 1.0 X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: DM4PR12MB5229:EE_|IA0PR12MB7602:EE_ X-MS-Office365-Filtering-Correlation-Id: ff469bc0-7e45-4a79-28cb-08dadd23b03e X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: Qpfkw6XrqhpMBcQemCJCjL50BIpynKBVQu/QmgY2k2zuvvoXlYN8tqX9myuAonYNUzFgxBiJQqlgcVPk1cq1dYdOxDJOuQ+cnruBljYIzJ3kbmCmFYqih053q70KsDxCMk+7CnUsVJNKRn7+35deycRzWskjRyoCfqP4mB4pk+XMq5khk3IvIQx23Teb0RhivPeKE/XTsf6VTwMtrLfLbALFilbspzDArd8+DLuYyTXjCHT4ywFIF7P7AgqTrlD48Y4e9onFfJt8WBEIvepYey3FyShFSqECvhq3jhy0UsF3jw6jkkaYMfNy1XYHGSTwh+RPwUSyqZL/0wedCfrJeZUS4Gt6EBmdLYY6mZWGch85nte2cuIPRUFxqyICgqFKsCel7FyiNa2vzmwDpk6qPTjtv45eDIYlIq504iCzfomeRVPl9MOf+ZTqHR1MeCJ5ToMgyqK6yzNFX10jLpcp03YMXoLuwHh/qHlQLri9XF/18HUpLf8YxGLrjDAhA10lU+ml6krsJjyuGr2oTmeW/aZA6RArrCdZrTDQTM3Wvfegn4lN7/ZXaNMAC6BgQvdgzazTIo1c/29jilX6At9C9nSjCXVRuLK1pSaLax/x1+jyGP4m2lRBYZGzSZF+Y7xFwK9E2EMYetBRpasmx+qQebREaU5NsnVanZFrJlzrPg498xgl8sGGU7ia4ZwNLz0sVar44zovDjA0JLFfObrEC8hQiB071jxYg2OwqTgV3U27sQ0KctAvkU2WJVSI/huwbcOVg6dr/H8KLlxd9ADLuw== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DM4PR12MB5229.namprd12.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230022)(6029001)(4636009)(376002)(396003)(366004)(39860400002)(136003)(346002)(451199015)(31686004)(83380400001)(6486002)(966005)(36756003)(31696002)(86362001)(478600001)(53546011)(26005)(38100700002)(186003)(6506007)(2616005)(4326008)(8676002)(66556008)(66476007)(66946007)(6512007)(5660300002)(54906003)(2906002)(316002)(41300700001)(8936002)(45980500001)(43740500002);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData-ChunkCount: 1 X-MS-Exchange-AntiSpam-MessageData-0: =?utf-8?B?R3RqNXRVVUlOMWNhbEVKdTR5WXloU0NBTVVia2F0d0NHeUhWVE1IOWs3Mm9r?= =?utf-8?B?aUxTZTVDS0ZmVU1NM1g5N2l1OHZKMHBWK2c0RzZVenRGbU5EZWRmYW9XTnNX?= =?utf-8?B?OWdiL1p5RmdrdWdpeVc5alNNeDkyYlQ1YXBxWGUzQllpSjJuWUg5ek1RNmNK?= =?utf-8?B?dFZCTzZmbTZlMUlSRElWclFQTkx4WmRRVGtrL2xxbHFkWFE4UjdORVdFUmh1?= =?utf-8?B?RUN6ZFhsNlphMWhkTVNhcmthT240dWUzMkFIL0xzeE0xZ0FZd0JNT09xOHVG?= =?utf-8?B?L0hoTDcyQ2NLTHRZbjE5U2pwNWlqL1JpYzhkcmNVVFRVekJhdmlYdjdvSFF5?= =?utf-8?B?QWYzdWdFZnpHYkpWNWRoUHMzRWpmYkNuWHBvdm5zZTVBZ09GSVZ0S2JMaFND?= =?utf-8?B?K3NGM3VCZzJFdDFza2tpUFRnRmxPSnJUMXowUjRWQUplU2JaOVZKdjFIVFV6?= =?utf-8?B?TlBEUDFwWElIdUpIY3Fpc0M5M2o5L2xvK2l1dWN4NWExVmQvem9qVW1SaFdw?= =?utf-8?B?QVhGclkxSWFqZ0oyNzhMRnM0dnhmaEJUcXNCWjVWc0l5S2VoTTFUM2p4eEc0?= =?utf-8?B?ZGQzQzgycDRPK1dqZ1A5T0hUcnBucHRqZzlMbGdRQ3NCb09rek5UR0F3ZjB0?= =?utf-8?B?QmRXcUQ1cEp4alUyS3k2d1BNemRZQ1NhaHQwK25zbnhjM3orOU9uVlVnL0k2?= =?utf-8?B?bU5XZ0EzR3JFemNzcnN0ODBlV0xMTXJlenFIMXduSTJ5bStZRXdSZWVvNFQ2?= =?utf-8?B?ZXQ2ekFyQ1p3Y0h0ZkVUYlFJZ0YzTWdENVNoTk85amRvNGJHSnoxajB0bnJ5?= =?utf-8?B?STFuc3VEdjE1emp1VVJ4MTlaTC9sSHlYSzYyZ1IwZGQzSWZvQlNyRGI1bmNQ?= =?utf-8?B?Z2wvNWNCMGRuanZaWDhIK0xHbTlLbWkwVmU5OHZ1aE5sQ3BJa1dTNFhYbmJQ?= =?utf-8?B?VlZhVjB1ckhSZmNxSGZQa1d3aGwzNEc5c3dFbmd1RDNqNmY0SVE1RlE1cTJX?= =?utf-8?B?MlRxc2tBZjVuOXhUcForVFJzbTYwOTl1VGZkWDRldE5aM3J2VUpOL00rMTVs?= =?utf-8?B?bkV0RTBsWDZYZHRjdVJHVFArSU9FaSsxZXRETWlLbmlVWmJKcFhhc0NBYjBo?= =?utf-8?B?TU9uSEpnTlJOU2NwRlhLT2I4M0wvNEdqenVNNW9LYTFkOUZ0M0xsZlBYUWRk?= =?utf-8?B?Z3hsUzk5WEM1bEpqNmp2WWk2NzhOdmw5THhwcVdMcUV4eTlMcjFKalFpdzFs?= =?utf-8?B?MStnN0JyYmxQRElLQXV1dnhXNWlpb284d0FQcmJZS3BmRnpzN3EvS1dIV1Jp?= =?utf-8?B?enBQU2FQZ3JWN3pKSTA4dzdaWlJUdmQyV1VBL2NnOGtHVkZqbWxhUU1WNUJ5?= =?utf-8?B?QnBoaTZCNmpUbzRzelpJMmxpK21sbThjNDh6L0tsdkkvTWh2QmJDQ1V3aEdL?= =?utf-8?B?SU5qVlgwb0hIOStJNkRzc2xjTys5bytIaDhMOE1wQ3V2SFl3cStiNHZVUnVE?= =?utf-8?B?dGJYTEtCdUJLQm1XTFVaVTNsTTltYlFtK0dOZDNSU2ZyVUJxektaWHRuWGJU?= =?utf-8?B?Tk1mSUozUFpOZUJEd3J4aWdsSElpVEUxVXluMjlwQ0ZqQ3R3ZjU4OGI5QlFD?= =?utf-8?B?bFBvYmtRUE9BWE80MnlZM1dGSWRrWVFrYVo0RnhtRGcyYXRQbm95WlJ4Lzkr?= =?utf-8?B?eTZSQ3JndDQ0RXBnUEdIMkYzeGhBWXZXRXpPdGsvUXpNRHZDTTYwWTlteTZl?= =?utf-8?B?NktkM3pQSzhvcTlRaXZKdmxxcWM5cmk4MWxZaW1Sc2hFRlI0ZWttWENqSkVu?= =?utf-8?B?QUJsdXRiQlRxelMxeVNHK0hVNjdCR0grc3BYdU1SYk94SVlSVlZzRFJsUHBh?= =?utf-8?B?ZFJHYnRlWFk0aWQ2MXJocmFrM3pkeHBlK3ZiQUxBWHpudFh3dUoxMklUR0tZ?= =?utf-8?B?THdpa1psSGJyQ3dKMWdLem5SS1FNdnc1VDV2Tm9icHI0NU5EbENRVHBja0tp?= =?utf-8?B?Kzh4VFFjc1d0TW92YzAwSHIxWkdNYXY0WC9TNWR3S1NDbHdvYk15dWdCSUgv?= =?utf-8?B?QVlCWEY0VmFSblNWdU9EcnZORmpiWGRMY1NJV0NTUWdPeGhybDdKME1yTDMv?= =?utf-8?Q?ElKy2rS4hk2R6nVULJg0RzpTL?= X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-Network-Message-Id: ff469bc0-7e45-4a79-28cb-08dadd23b03e X-MS-Exchange-CrossTenant-AuthSource: DM4PR12MB5229.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 13 Dec 2022 16:04:17.8344 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: wmKpP+bc8aCXhKmvmG+G4St1QGRaEqdpPRJz5QuDJBFoJCQn75HWvaxO/Fji2w5uDngtnRTEt9bY5t1LcrjjsA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA0PR12MB7602 Content-Language: en-US Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit On 12/12/22 23:48, Min Xu wrote: > From: Min M Xu > > BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=4171 > > A typical QEMU fw_cfg read bytes with IOMMU for td guest is that: > (QemuFwCfgReadBytes@QemuFwCfgLib.c is the example) > 1) Allocate DMA Access buffer > 2) Map actual data buffer > 3) start the transfer and wait for the transfer to complete > 4) Free DMA Access buffer > 5) Un-map actual data buffer > > In step 1/2, Private memories are allocated, converted to shared memories. > In Step 4/5 the shared memories are converted to private memories and > accepted again. The final step is to free the pages. > > This is time-consuming and impacts td guest's boot perf (both direct boot > and grub boot) badly. > > In a typical grub boot, there are about 5000 calls of page allocation and > private/share conversion. Most of page size is less than 32KB. > > This patch allocates a memory region and initializes it into pieces of > memory with different sizes. A piece of such memory consists of 2 parts: > the first page is of private memory, and the other pages are shared > memory. This is to meet the layout of common buffer. > > When allocating bounce buffer in IoMmuMap(), IoMmuAllocateBounceBuffer() > is called to allocate the buffer. Accordingly when freeing bounce buffer > in IoMmuUnmapWorker(), IoMmuFreeBounceBuffer() is called to free the > bounce buffer. CommonBuffer is allocated by IoMmuAllocateCommonBuffer > and accordingly freed by IoMmuFreeCommonBuffer. > > This feature is tested in Intel TDX pre-production platform. It saves up > to hundreds of ms in a grub boot. This patch causes crashes for SEV guests and breaks bisect-ability of the EDK2 tree. See below... > > Cc: Erdem Aktas > Cc: James Bottomley > Cc: Jiewen Yao > Cc: Tom Lendacky > Cc: Gerd Hoffmann > Reviewed-by: Jiewen Yao > Signed-off-by: Min Xu > --- > OvmfPkg/IoMmuDxe/AmdSevIoMmu.c | 142 +++++----- > OvmfPkg/IoMmuDxe/IoMmuBuffer.c | 459 +++++++++++++++++++++++++++++++ > OvmfPkg/IoMmuDxe/IoMmuDxe.inf | 1 + > OvmfPkg/IoMmuDxe/IoMmuInternal.h | 179 ++++++++++++ > 4 files changed, 710 insertions(+), 71 deletions(-) > create mode 100644 OvmfPkg/IoMmuDxe/IoMmuBuffer.c > create mode 100644 OvmfPkg/IoMmuDxe/IoMmuInternal.h > > diff --git a/OvmfPkg/IoMmuDxe/AmdSevIoMmu.c b/OvmfPkg/IoMmuDxe/AmdSevIoMmu.c > index 6b65897f032a..77e46bbf4a60 100644 > --- a/OvmfPkg/IoMmuDxe/AmdSevIoMmu.c > +++ b/OvmfPkg/IoMmuDxe/AmdSevIoMmu.c > @@ -633,8 +614,9 @@ IoMmuAllocateBuffer ( > CommonBufferHeader = (VOID *)(UINTN)PhysicalAddress; > PhysicalAddress += EFI_PAGE_SIZE; > > - CommonBufferHeader->Signature = COMMON_BUFFER_SIG; > - CommonBufferHeader->StashBuffer = StashBuffer; > + CommonBufferHeader->Signature = COMMON_BUFFER_SIG; > + CommonBufferHeader->StashBuffer = StashBuffer; > + CommonBufferHeader->ReservedMemBitmap = ReservedMemBitmap; > > *HostAddress = (VOID *)(UINTN)PhysicalAddress; > > @@ -707,7 +689,7 @@ IoMmuFreeBuffer ( > // Release the common buffer itself. Unmap() has re-encrypted it in-place, so > // no need to zero it. > // > - return gBS->FreePages ((UINTN)CommonBufferHeader, CommonBufferPages); > + return IoMmuFreeCommonBuffer (CommonBufferHeader, CommonBufferPages); > } > > /** > @@ -878,6 +860,11 @@ IoMmuUnmapAllMappings ( > TRUE // MemoryMapLocked > ); > } > + > + // > + // Release the reserved shared memory as well. > + // > + IoMmuReleaseReservedSharedMem (TRUE); This call is the reason for the crash. You'll need to check for whether there has been any shared memory reserved before attempting to free it (in the case of SEV that doesn't happen until patch #3 of the series). I think adding the check in IoMmuReleaseReservedSharedMem() itself might be best, since you can also experience this crash should the below allocation fail, too. Thanks, Tom > } > > /** > @@ -936,6 +923,19 @@ InstallIoMmuProtocol ( > goto CloseExitBootEvent; > } > > + // > + // Currently only Tdx guest support Reserved shared memory for DMA operation. > + // > + if (CC_GUEST_IS_TDX (PcdGet64 (PcdConfidentialComputingGuestAttr))) { > + mReservedSharedMemSupported = TRUE; > + Status = IoMmuInitReservedSharedMem (); > + if (EFI_ERROR (Status)) { > + mReservedSharedMemSupported = FALSE; > + } else { > + DEBUG ((DEBUG_INFO, "%a: Feature of reserved memory for DMA is supported.\n", __FUNCTION__)); > + } > + } > + > return EFI_SUCCESS; > > CloseExitBootEvent: