From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ml01.01.org (Postfix) with ESMTPS id 86F1B21EA35D7 for ; Tue, 5 Sep 2017 05:38:26 -0700 (PDT) Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 0145975707; Tue, 5 Sep 2017 12:41:15 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mx1.redhat.com 0145975707 Authentication-Results: ext-mx03.extmail.prod.ext.phx2.redhat.com; dmarc=none (p=none dis=none) header.from=redhat.com Authentication-Results: ext-mx03.extmail.prod.ext.phx2.redhat.com; spf=fail smtp.mailfrom=lersek@redhat.com Received: from lacos-laptop-7.usersys.redhat.com (ovpn-120-185.rdu2.redhat.com [10.10.120.185]) by smtp.corp.redhat.com (Postfix) with ESMTP id 5C34984D17; Tue, 5 Sep 2017 12:41:13 +0000 (UTC) To: Brijesh Singh , edk2-devel@lists.01.org Cc: Jordan Justen , Tom Lendacky , Ard Biesheuvel References: <1504265045-19008-1-git-send-email-brijesh.singh@amd.com> <1504265045-19008-5-git-send-email-brijesh.singh@amd.com> From: Laszlo Ersek Message-ID: <09699a1f-9bc9-02d1-f7d0-99cd9d42051e@redhat.com> Date: Tue, 5 Sep 2017 14:41:12 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.3.0 MIME-Version: 1.0 In-Reply-To: <1504265045-19008-5-git-send-email-brijesh.singh@amd.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.27]); Tue, 05 Sep 2017 12:41:15 +0000 (UTC) Subject: Re: [PATCH 4/5] OvmfPkg/VirtioNetDxe: map virtio-net transmit request buffer X-BeenThere: edk2-devel@lists.01.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: EDK II Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Sep 2017 12:38:26 -0000 Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit On 09/01/17 13:24, Brijesh Singh wrote: > When device is behind the IOMMU, driver is require to pass the device > address of transmit buffer for the bus master operations. > > The patch uses VirtioMapAllBytesInSharedBuffer() to map transmit buffer > system physical address to the device address. > > Since the transmit buffers are returned back to caller in > VirtioNetGetStatus() hence we use OrderCollection library interface to > save the host to device address mapping. After the buffer is succesfully > transmited we do reverse lookup in OrderCollection data structure to get > the host address for the transmitted device address. > > Cc: Ard Biesheuvel > Cc: Jordan Justen > Cc: Tom Lendacky > Cc: Laszlo Ersek > Contributed-under: TianoCore Contribution Agreement 1.1 > Signed-off-by: Brijesh Singh > --- > OvmfPkg/VirtioNetDxe/VirtioNet.inf | 1 + > OvmfPkg/VirtioNetDxe/VirtioNet.h | 19 +++ > OvmfPkg/VirtioNetDxe/SnpGetStatus.c | 30 +++- > OvmfPkg/VirtioNetDxe/SnpSharedHelpers.c | 157 ++++++++++++++++++++ > OvmfPkg/VirtioNetDxe/SnpTransmit.c | 37 ++++- > 5 files changed, 232 insertions(+), 12 deletions(-) I have some preliminary comments for this patch. This is by no means a full review. (1) Using "SnpSharedHelpers.c" for this kind of functionality is appropriate, but then please add a comment block between - the current group of functions (VirtioNetShutdownRx, VirtioNetShutdownTx, and the upcoming VirtioNetUninitRing), - and between these other data types and functions. The current comment block only covers the first group of functions. (2) Please use more sensible names for the data structures and their fields. I'm not sure just yet if I'm going to agree with the current usage, but USER_STRUCT and "Value" are not good names. Please add documentation to the fields. (3) The "HostAddress" field and parameter can be / should be (VOID*). (4) You don't need to define a structure for USER_KEY if it has only one field. (5) The ordered collection should be initialized separately, in VirtioNetInitTx(). (6) Currently OrderedCollectionUninit() is not called anywhere. The collection should be torn down in VirtioNetShutdownTx(). Beware, tearing down the transmit mappings takes more work than just calling OrderedCollectionUninit(). You'll have to iterate over the structure, remove each node, destroy (unmap + free) each shared buffer, free each user structure, and finally call OrderedCollectionUninit(). Generally it looks like this, for (Entry = OrderedCollectionMin (Collection); Entry != NULL; Entry = Entry2) { Entry2 = OrderedCollectionNext (Entry); OrderedCollectionDelete (Collection, Entry, &UserStruct); // // now unmap and free the shared buffer identified by UserStruct // // finally, free UserStruct itself // } OrderedCollectionUninit (Collection); (7) We'll have to document the new model in "TechNotes.txt". (8) VirtioOperationBusMasterRead is not appropriate for mappings that remain in existence after we return from an async protocol member function, such as SNP.Transmit(). Namely, if ExitBootServices() is called next, we have to tear down all the mappings (outgoing buffers) *without* releasing memory, and BusMasterRead is not OK for that. Only CommonBuffer is. (9) Which makes me realize that VirtioNetExitBoot() is not updated in this patch at all. It should basically do the same as (5), except nodes from the collection should not be deleted, shared buffers should only be unmapped, not freed, and user structures should not be deleted. Something like this: for (Entry = OrderedCollectionMin (Collection); Entry != NULL; Entry = OrderedCollectionNext (Entry)) { UserStruct = OrderedCollectionUserStruct (Entry); // // now unmap the shared buffer identified by UserStruct // } (10) This patch is going to be large (even larger than now); can we split it up? First, update the documentation. Second, just introduce the data structure and the helper functions. Third, update the current code to set up the data structure, call the helpers whenever appropriate, and tear down the data structure. (11) Regarding points (8) and (9), i.e., unmapping at ExitBootServices(), have you been following the related discussion in: [edk2] [PATCH 0/4] MdeModulePkg: some PCI HC drivers: unmap common buffers at ExitBootServices() ? Let's return to this patch when we reach it with the gradual advancement that I'm requesting. Thank you, Laszlo > diff --git a/OvmfPkg/VirtioNetDxe/VirtioNet.inf b/OvmfPkg/VirtioNetDxe/VirtioNet.inf > index a855ad4ac154..9ff6d87e6190 100644 > --- a/OvmfPkg/VirtioNetDxe/VirtioNet.inf > +++ b/OvmfPkg/VirtioNetDxe/VirtioNet.inf > @@ -49,6 +49,7 @@ [LibraryClasses] > DebugLib > DevicePathLib > MemoryAllocationLib > + OrderedCollectionLib > UefiBootServicesTableLib > UefiDriverEntryPoint > UefiLib > diff --git a/OvmfPkg/VirtioNetDxe/VirtioNet.h b/OvmfPkg/VirtioNetDxe/VirtioNet.h > index 3efe95a735d8..d931969af795 100644 > --- a/OvmfPkg/VirtioNetDxe/VirtioNet.h > +++ b/OvmfPkg/VirtioNetDxe/VirtioNet.h > @@ -269,6 +269,25 @@ VirtioNetShutdownTx ( > IN OUT VNET_DEV *Dev > ); > > +EFI_STATUS > +EFIAPI > +VirtioMapTxBuf ( > + IN VNET_DEV *Dev, > + IN UINT16 Index, > + IN EFI_PHYSICAL_ADDRESS HostAddress, > + IN UINTN NumberOfBytes, > + OUT EFI_PHYSICAL_ADDRESS *DeviceAddress > + ); > + > +EFI_STATUS > +EFIAPI > +VirtioUnmapTxBuf ( > + IN VNET_DEV *Dev, > + IN UINT16 Index, > + OUT EFI_PHYSICAL_ADDRESS *HostAddress, > + IN EFI_PHYSICAL_ADDRESS DeviceAddress > + ); > + > // > // event callbacks > // > diff --git a/OvmfPkg/VirtioNetDxe/SnpGetStatus.c b/OvmfPkg/VirtioNetDxe/SnpGetStatus.c > index 694940ea1d97..10ca26de6d7a 100644 > --- a/OvmfPkg/VirtioNetDxe/SnpGetStatus.c > +++ b/OvmfPkg/VirtioNetDxe/SnpGetStatus.c > @@ -5,6 +5,7 @@ > > Copyright (C) 2013, Red Hat, Inc. > Copyright (c) 2006 - 2014, Intel Corporation. All rights reserved.
> + Copyright (c) 2017, AMD Inc, All rights reserved.
> > This program and the accompanying materials are licensed and made available > under the terms and conditions of the BSD License which accompanies this > @@ -47,7 +48,8 @@ > @retval EFI_INVALID_PARAMETER One or more of the parameters has an > unsupported value. > @retval EFI_DEVICE_ERROR The command could not be sent to the network > - interface. > + interface or failed to map TxBuf to bus master > + address. > @retval EFI_UNSUPPORTED This function is not supported by the network > interface. > > @@ -126,8 +128,10 @@ VirtioNetGetStatus ( > *TxBuf = NULL; > } > else { > - UINT16 UsedElemIdx; > - UINT32 DescIdx; > + UINT16 UsedElemIdx; > + UINT32 DescIdx; > + EFI_PHYSICAL_ADDRESS BufferAddress; > + EFI_PHYSICAL_ADDRESS DeviceAddress; > > // > // fetch the first descriptor among those that the hypervisor reports > @@ -141,9 +145,27 @@ VirtioNetGetStatus ( > ASSERT (DescIdx < (UINT32) (2 * Dev->TxMaxPending - 1)); > > // > + // Ring descriptor contains the device address for the caller buffer. > + // Lets unmap the device address and find its corresponding system > + // physical address. > + // > + DeviceAddress = Dev->TxRing.Desc[DescIdx + 1].Addr; > + Status = VirtioUnmapTxBuf ( > + Dev, > + DescIdx + 1, > + &BufferAddress, > + DeviceAddress > + ); > + if (EFI_ERROR (Status)) { > + Status = EFI_DEVICE_ERROR; > + goto Exit; > + } > + > + // > + // > // report buffer address to caller that has been enqueued by caller > // > - *TxBuf = (VOID *)(UINTN) Dev->TxRing.Desc[DescIdx + 1].Addr; > + *TxBuf = (VOID *)(UINTN) BufferAddress; > > // > // now this descriptor can be used again to enqueue a transmit buffer > diff --git a/OvmfPkg/VirtioNetDxe/SnpSharedHelpers.c b/OvmfPkg/VirtioNetDxe/SnpSharedHelpers.c > index aeb9e6605f0d..b91ddd3e4ede 100644 > --- a/OvmfPkg/VirtioNetDxe/SnpSharedHelpers.c > +++ b/OvmfPkg/VirtioNetDxe/SnpSharedHelpers.c > @@ -14,10 +14,25 @@ > > **/ > > +#include > #include > +#include > > #include "VirtioNet.h" > > +typedef struct { > + UINT16 Value; > +} USER_KEY; > + > +typedef struct { > + USER_KEY Key; > + EFI_PHYSICAL_ADDRESS HostAddress; > + EFI_PHYSICAL_ADDRESS DeviceAddress; > + VOID *BufMap; > +} USER_STRUCT; > + > +STATIC ORDERED_COLLECTION *Collection; > + > /** > Release RX and TX resources on the boundary of the > EfiSimpleNetworkInitialized state. > @@ -63,3 +78,145 @@ VirtioNetShutdownTx ( > > FreePool (Dev->TxFreeStack); > } > + > +STATIC > +INTN > +EFIAPI > +KeyCompare ( > + IN CONST VOID *StandaloneKey, > + IN CONST VOID *UserStruct > + ) > +{ > + CONST USER_KEY *CmpKey; > + CONST USER_STRUCT *CmpStruct; > + > + CmpKey = StandaloneKey; > + CmpStruct = UserStruct; > + > + return CmpKey->Value < CmpStruct->Key.Value ? -1 : > + CmpKey->Value > CmpStruct->Key.Value ? 1 : > + 0; > +} > + > +STATIC > +INTN > +EFIAPI > +UserStructCompare ( > + IN CONST VOID *UserStruct1, > + IN CONST VOID *UserStruct2 > + ) > +{ > + CONST USER_STRUCT *CmpStruct1; > + > + CmpStruct1 = UserStruct1; > + > + return KeyCompare (&CmpStruct1->Key, UserStruct2); > +} > + > +EFI_STATUS > +EFIAPI > +VirtioMapTxBuf ( > + IN VNET_DEV *Dev, > + IN UINT16 Index, > + IN EFI_PHYSICAL_ADDRESS HostAddress, > + IN UINTN NumberOfBytes, > + OUT EFI_PHYSICAL_ADDRESS *DeviceAddress > + ) > +{ > + EFI_STATUS Status; > + ORDERED_COLLECTION_ENTRY *Entry; > + USER_STRUCT *UserStruct; > + EFI_PHYSICAL_ADDRESS Address; > + VOID *Mapping; > + > + // > + // If ordered collection is not initialized then initialize it. > + // > + if (Collection == NULL) { > + Collection = OrderedCollectionInit (UserStructCompare, KeyCompare); > + if (Collection == NULL) { > + return EFI_OUT_OF_RESOURCES; > + } > + } > + > + UserStruct = AllocatePool (sizeof (*UserStruct)); > + if (UserStruct == NULL) { > + return EFI_OUT_OF_RESOURCES; > + } > + > + Status = VirtioMapAllBytesInSharedBuffer ( > + Dev->VirtIo, > + VirtioOperationBusMasterRead, > + (VOID *) (UINTN) HostAddress, > + NumberOfBytes, > + &Address, > + &Mapping > + ); > + if (EFI_ERROR (Status)) { > + goto ReleaseUserStruct; > + } > + > + UserStruct->Key.Value = Index; > + UserStruct->HostAddress = HostAddress; > + UserStruct->DeviceAddress = Address; > + UserStruct->BufMap = Mapping; > + > + Status = OrderedCollectionInsert (Collection, &Entry, UserStruct); > + switch (Status) { > + case RETURN_OUT_OF_RESOURCES: > + Status = EFI_OUT_OF_RESOURCES; > + goto UnmapBuffer; > + case RETURN_ALREADY_STARTED: > + Status = EFI_INVALID_PARAMETER; > + goto UnmapBuffer; > + default: > + ASSERT (Status == RETURN_SUCCESS); > + break; > + } > + > + ASSERT (OrderedCollectionUserStruct (Entry) == UserStruct); > + > + *DeviceAddress = Address; > + return EFI_SUCCESS; > + > +UnmapBuffer: > + Dev->VirtIo->UnmapSharedBuffer (Dev->VirtIo, Mapping); > + > +ReleaseUserStruct: > + FreePool (UserStruct); > + return Status; > +} > + > +EFI_STATUS > +EFIAPI > +VirtioUnmapTxBuf ( > + IN VNET_DEV *Dev, > + IN UINT16 Index, > + OUT EFI_PHYSICAL_ADDRESS *HostAddress, > + IN EFI_PHYSICAL_ADDRESS DeviceAddress > + ) > +{ > + USER_KEY StandaloneKey; > + ORDERED_COLLECTION_ENTRY *Entry; > + USER_STRUCT *UserStruct; > + VOID *Ptr; > + EFI_STATUS Status; > + > + StandaloneKey.Value = Index; > + Entry = OrderedCollectionFind (Collection, &StandaloneKey); > + if (Entry == NULL) { > + return EFI_INVALID_PARAMETER; > + } > + > + OrderedCollectionDelete (Collection, Entry, &Ptr); > + > + UserStruct = Ptr; > + ASSERT (UserStruct->DeviceAddress == DeviceAddress); > + ASSERT (UserStruct->Key.Value == Index); > + > + *HostAddress = UserStruct->HostAddress; > + Status = Dev->VirtIo->UnmapSharedBuffer (Dev->VirtIo, UserStruct->BufMap); > + FreePool (UserStruct); > + > + return Status; > +} > diff --git a/OvmfPkg/VirtioNetDxe/SnpTransmit.c b/OvmfPkg/VirtioNetDxe/SnpTransmit.c > index 7ca40d5d0650..1bd6e0d70d7c 100644 > --- a/OvmfPkg/VirtioNetDxe/SnpTransmit.c > +++ b/OvmfPkg/VirtioNetDxe/SnpTransmit.c > @@ -55,7 +55,8 @@ > @retval EFI_INVALID_PARAMETER One or more of the parameters has an > unsupported value. > @retval EFI_DEVICE_ERROR The command could not be sent to the network > - interface. > + interface or failed to map the Buffer to > + bus master address. > @retval EFI_UNSUPPORTED This function is not supported by the network > interface. > > @@ -73,11 +74,12 @@ VirtioNetTransmit ( > IN UINT16 *Protocol OPTIONAL > ) > { > - VNET_DEV *Dev; > - EFI_TPL OldTpl; > - EFI_STATUS Status; > - UINT16 DescIdx; > - UINT16 AvailIdx; > + VNET_DEV *Dev; > + EFI_TPL OldTpl; > + EFI_STATUS Status; > + UINT16 DescIdx; > + UINT16 AvailIdx; > + EFI_PHYSICAL_ADDRESS DeviceAddress; > > if (This == NULL || BufferSize == 0 || Buffer == NULL) { > return EFI_INVALID_PARAMETER; > @@ -144,10 +146,29 @@ VirtioNetTransmit ( > } > > // > - // virtio-0.9.5, 2.4.1 Supplying Buffers to The Device > + // Get the available descriptor > // > DescIdx = Dev->TxFreeStack[Dev->TxCurPending++]; > - Dev->TxRing.Desc[DescIdx + 1].Addr = (UINTN) Buffer; > + > + // > + // Map the transmit buffer system physical address to device address. > + // > + Status = VirtioMapTxBuf ( > + Dev, > + DescIdx + 1, > + (EFI_PHYSICAL_ADDRESS) (UINTN) Buffer, > + BufferSize, > + &DeviceAddress > + ); > + if (EFI_ERROR (Status)) { > + Status = EFI_DEVICE_ERROR; > + goto Exit; > + } > + > + // > + // virtio-0.9.5, 2.4.1 Supplying Buffers to The Device > + // > + Dev->TxRing.Desc[DescIdx + 1].Addr = DeviceAddress; > Dev->TxRing.Desc[DescIdx + 1].Len = (UINT32) BufferSize; > > // >