From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: mx.groups.io; dkim=missing; spf=pass (domain: redhat.com, ip: 209.132.183.28, mailfrom: lersek@redhat.com) Received: from mx1.redhat.com (mx1.redhat.com [209.132.183.28]) by groups.io with SMTP; Wed, 10 Jul 2019 07:05:06 -0700 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 768B83084029; Wed, 10 Jul 2019 14:05:05 +0000 (UTC) Received: from lacos-laptop-7.usersys.redhat.com (ovpn-117-153.ams2.redhat.com [10.36.117.153]) by smtp.corp.redhat.com (Postfix) with ESMTP id 9B68083798; Wed, 10 Jul 2019 14:05:03 +0000 (UTC) Subject: Re: [PATCH v3 33/35] OvmfPkg: Introduce XenIoPvhDxe to initialize Grant Tables To: Anthony PERARD , devel@edk2.groups.io Cc: xen-devel@lists.xenproject.org, Ard Biesheuvel , Jordan Justen , Julien Grall References: <20190704144233.27968-1-anthony.perard@citrix.com> <20190704144233.27968-34-anthony.perard@citrix.com> From: "Laszlo Ersek" Message-ID: <4badd535-c23d-c64d-7bb3-fb42bbbf538a@redhat.com> Date: Wed, 10 Jul 2019 16:05:02 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.9.1 MIME-Version: 1.0 In-Reply-To: <20190704144233.27968-34-anthony.perard@citrix.com> X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.5.110.40]); Wed, 10 Jul 2019 14:05:05 +0000 (UTC) Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit On 07/04/19 16:42, Anthony PERARD wrote: > XenIoPvhDxe use XenIoMmioLib to reserve some space to be use by the > Grant Tables. > > The call is only done if it is necessary, we simply detect if the > guest is PVH, as in this case there is currently no PCI bus, and no > PCI Xen platform device which would start the XenIoPciDxe and allocate > the space for the Grant Tables. > > Ref: https://bugzilla.tianocore.org/show_bug.cgi?id=1689 > Signed-off-by: Anthony PERARD > --- > > Notes: > v3: > - downgrade type to DXE_DRIVER > - use SPDX > - rework InitializeXenIoPvhDxe, and handle errors properly. > - Free the reserved allocation in ExitBootServices even if the XenIo > protocol could successfully been uninstalled. > > v2: > - do allocation in EntryPoint like the other user of XenIoMmioLib. > - allocate memory instead of hardcoded addr. > - cleanup, add copyright > - detect if we are running in PVH mode > > OvmfPkg/OvmfXen.dsc | 2 + > OvmfPkg/OvmfXen.fdf | 1 + > OvmfPkg/XenIoPvhDxe/XenIoPvhDxe.inf | 34 +++++++++ > OvmfPkg/XenIoPvhDxe/XenIoPvhDxe.c | 108 ++++++++++++++++++++++++++++ > 4 files changed, 145 insertions(+) > create mode 100644 OvmfPkg/XenIoPvhDxe/XenIoPvhDxe.inf > create mode 100644 OvmfPkg/XenIoPvhDxe/XenIoPvhDxe.c > > diff --git a/OvmfPkg/OvmfXen.dsc b/OvmfPkg/OvmfXen.dsc > index 487bada64d..af92ce3ed2 100644 > --- a/OvmfPkg/OvmfXen.dsc > +++ b/OvmfPkg/OvmfXen.dsc > @@ -196,6 +196,7 @@ [LibraryClasses] > OrderedCollectionLib|MdePkg/Library/BaseOrderedCollectionRedBlackTreeLib/BaseOrderedCollectionRedBlackTreeLib.inf > XenHypercallLib|OvmfPkg/Library/XenHypercallLib/XenHypercallLib.inf > XenPlatformLib|OvmfPkg/Library/XenPlatformLib/XenPlatformLib.inf > + XenIoMmioLib|OvmfPkg/Library/XenIoMmioLib/XenIoMmioLib.inf > > Tcg2PhysicalPresenceLib|OvmfPkg/Library/Tcg2PhysicalPresenceLibNull/DxeTcg2PhysicalPresenceLib.inf > > @@ -583,6 +584,7 @@ [Components] > NULL|OvmfPkg/Csm/LegacyBootMaintUiLib/LegacyBootMaintUiLib.inf > !endif > } > + OvmfPkg/XenIoPvhDxe/XenIoPvhDxe.inf > OvmfPkg/XenIoPciDxe/XenIoPciDxe.inf > OvmfPkg/XenBusDxe/XenBusDxe.inf > OvmfPkg/XenPvBlkDxe/XenPvBlkDxe.inf > diff --git a/OvmfPkg/OvmfXen.fdf b/OvmfPkg/OvmfXen.fdf > index 5c1a925d6a..517a492f14 100644 > --- a/OvmfPkg/OvmfXen.fdf > +++ b/OvmfPkg/OvmfXen.fdf > @@ -309,6 +309,7 @@ [FV.DXEFV] > INF MdeModulePkg/Universal/Metronome/Metronome.inf > INF PcAtChipsetPkg/PcatRealTimeClockRuntimeDxe/PcatRealTimeClockRuntimeDxe.inf > > +INF OvmfPkg/XenIoPvhDxe/XenIoPvhDxe.inf > INF OvmfPkg/XenIoPciDxe/XenIoPciDxe.inf > INF OvmfPkg/XenBusDxe/XenBusDxe.inf > INF OvmfPkg/XenPvBlkDxe/XenPvBlkDxe.inf > diff --git a/OvmfPkg/XenIoPvhDxe/XenIoPvhDxe.inf b/OvmfPkg/XenIoPvhDxe/XenIoPvhDxe.inf > new file mode 100644 > index 0000000000..a093d48fde > --- /dev/null > +++ b/OvmfPkg/XenIoPvhDxe/XenIoPvhDxe.inf > @@ -0,0 +1,34 @@ > +## @file > +# Driver for the XenIo protocol > +# > +# Copyright (c) 2019, Citrix Systems, Inc. > +# > +# SPDX-License-Identifier: BSD-2-Clause-Patent > +# > +## > + > +[Defines] > + INF_VERSION = 0x00010005 > + BASE_NAME = XenIoPvhDxe > + FILE_GUID = 7a567cc4-0e75-4d7a-a305-c3db109b53ad > + MODULE_TYPE = DXE_DRIVER > + VERSION_STRING = 1.0 > + ENTRY_POINT = InitializeXenIoPvhDxe > + > +[Packages] > + MdePkg/MdePkg.dec > + OvmfPkg/OvmfPkg.dec > + > +[Sources] > + XenIoPvhDxe.c > + > +[LibraryClasses] > + DebugLib > + MemoryAllocationLib > + UefiBootServicesTableLib > + UefiDriverEntryPoint > + XenIoMmioLib > + XenPlatformLib > + > +[Depex] > + TRUE > diff --git a/OvmfPkg/XenIoPvhDxe/XenIoPvhDxe.c b/OvmfPkg/XenIoPvhDxe/XenIoPvhDxe.c > new file mode 100644 > index 0000000000..88a394bf91 > --- /dev/null > +++ b/OvmfPkg/XenIoPvhDxe/XenIoPvhDxe.c > @@ -0,0 +1,108 @@ > +/** @file > + > + Driver for the XenIo protocol > + > + This driver simply allocate space for the grant tables. > + > + Copyright (c) 2019, Citrix Systems, Inc. > + > + SPDX-License-Identifier: BSD-2-Clause-Patent > + > +**/ > + > +#include > +#include > +#include > +#include > +#include > + > +typedef struct { > + EFI_HANDLE XenIoHandle; > + EFI_EVENT ExitBootEvent; > + VOID *Allocation; > +} XEN_IO_PVH_STATE; > + > +// > +// Value should be the same as NR_GRANT_FRAMES in XenBusDxe > +// > +#define XEN_GRANT_FRAMES 4 In v2, this was a "naked" integer constant, and I didn't realize it was supposed to match XenBusDxe. (1) Please insert a separate patch before this patch, that turns the value 4 in XenBusDxe into a FixedAtBuild PCD, in "OvmfPkg.dec": [PcdsFixedAtBuild] ## Number of page frames to use for storing grant table entries. # Must be less than or equal to what is configured in the Xen HV. gUefiOvmfPkgTokenSpaceGuid.PcdXenGrantFrames|4|UINT32|0x33 In "XenBusDxe.inf", introduce a [FixedPcd] section, for expressing the dependency on the PCD. And in the code, use FixedPcdGet32 (PcdXenGrantFrames) This will expand to an integer constant, and so it is suitable for use in further #define's. > + > +STATIC > +VOID > +EFIAPI > +XenIoPvhDxeNotifyExitBoot ( > + IN EFI_EVENT Event, > + IN VOID *Context > + ) > +{ > + XEN_IO_PVH_STATE *State; > + EFI_STATUS Status; > + > + State = Context; > + > + gBS->CloseEvent(&State->ExitBootEvent); > + Status = XenIoMmioUninstall(State->XenIoHandle); > + if (Status == EFI_SUCCESS) { > + // > + // Only free the reserved space for grant table if no driver is using it. > + // > + FreePages (State->Allocation, XEN_GRANT_FRAMES); > + } > + FreePool (State); > +} In my v2 review at http://mid.mail-archive.com/2c386393-b886-a7e7-e8b9-c5e339c94727@redhat.com I asked, in point (6), whether we needed the memory allocation to be "reserved". I guess this change (in v3) is supposed to address that. (And, indeed, version 3 of this patch correctly addresses all other points from my v2 review.) However: an ExitBootServices notification function *must not* perform actions that could change the UEFI memory map. Thus, you can not release memory there -- you can't call CloseEvent(), UninstallMultipleProtocolInterfaces(), FreePool(), FreePages(), and so on. In my earlier review, I wrote, > [...] then we could allocate EfiBootServicesData type memory here -- > and perhaps add an ExitBootServices() callback to disconnect from Xen > [...] By "disconnect", I meant actions that do *not* include memory allocation / release, but only make the hypervisor *forget* its guest RAM references that it currently holds. ExitBootServices() handlers (notification functions) are different from normal "teardown" functions. A "teardown" function generally mirrors a "construct" function, where you remove all references, and release all memory allocations, in precise reverse order of construction. ExitBootServices() handlers perform a *subsequence* of that (usually preserving the same relative order between the steps): you only erase external references to RAM -- that is, RAM references that are held by external entities, such as PCI devices (in-flight DMA), the hypervisor, and so on. The referred-to RAM, originally allocated as "boot services data", will be released later, whole-sale, by the OS. In that sense, ExitBootServices() callbacks implement a kind of ownership transfer. You no longer own the UEFI memory map, you just have to kick out such external references into it that the OS would not know about. To summarize: - If this memory *needs* to be reserved past ExitBootServices(), then stick with AllocateReservedPages(). Otherwise, use AllocatePages(); then the pages will be freed automatically by the OS, soon after it is started. - In the latter case, if necessary, you can make the hypervisor forget about the area, as part of ExitBootServices(), in a notification function. But that can only happen without allocating or releasing memory in the notification function. ... I guess you could make a XENMEM_remove_from_physmap hypercall, for example. But, it's *entirely* possible that said hypercall doesn't belong in *this* driver. After all, this is not the driver that calls XENMEM_add_to_physmap either! ... In fact, I think XenBusDxe has a bug. Right now, you have the following call chain, when the OS calls ExitBootServices(): NotifyExitBoot() gBS->DisconnectController() XenBusDxeDriverBindingStop() XenGrantTableDeinit() XenHypercallMemoryOp (XENMEM_remove_from_physmap) Unfortunately, this call tree releases memory left and right, and so it is not valid on the stack of gBS->ExitBootServices(). Instead, NotifyExitBoot() should collect just the subsequence of XenBusDxeDriverBindingStop() -- interpreting that function as a flatened sequence of operations -- that removes external references to RAM. Making hypercalls (using parameters constructed on the stack) is fine. Using dynamically *pre*allocated memory (for constructing hypercall arguments) is also fine. Dynamically allocating or releasing memory *on the spot* is not fine. (2) So, for now, I'd recommend simply removing XenIoPvhDxeNotifyExitBoot(), in this patch. You might want to fix XenBusDxe's NotifyExitBoot(), separately. > + > +EFI_STATUS > +EFIAPI > +InitializeXenIoPvhDxe ( > + IN EFI_HANDLE ImageHandle, > + IN EFI_SYSTEM_TABLE *SystemTable > + ) > +{ > + VOID *Allocation; > + EFI_STATUS Status; > + XEN_IO_PVH_STATE *State; > + > + State = NULL; > + Allocation = NULL; > + > + if (! XenPvhDetected ()) { (3) Please drop the space character. It's not invalid, but quite foreign in edk2, I'd say. > + return EFI_UNSUPPORTED; > + } > + > + State = AllocatePool (sizeof (*State)); (4) What do we gain by allocating "State" dynamically? IMO, it could be a local variable (= "automatic storage duration"). > + if (State == NULL) { > + Status = EFI_OUT_OF_RESOURCES; > + goto Error; > + } > + > + Allocation = AllocateReservedPages (XEN_GRANT_FRAMES); (5) So, again, please evaluate if this could simply be AllocatePages(). Thanks Laszlo > + if (Allocation == NULL) { > + Status = EFI_OUT_OF_RESOURCES; > + goto Error; > + } > + > + State->XenIoHandle = NULL; > + Status = XenIoMmioInstall (&State->XenIoHandle, (UINTN) Allocation); > + if (EFI_ERROR (Status)) { > + goto Error; > + } > + > + State->Allocation = Allocation; > + Status = gBS->CreateEvent (EVT_SIGNAL_EXIT_BOOT_SERVICES, TPL_CALLBACK, > + XenIoPvhDxeNotifyExitBoot, State, &State->ExitBootEvent); > + ASSERT_EFI_ERROR (Status); > + > + return EFI_SUCCESS; > + > +Error: > + if (State != NULL && State->XenIoHandle != NULL) { > + XenIoMmioUninstall(State->XenIoHandle); > + } > + if (Allocation != NULL) { > + FreePages (Allocation, XEN_GRANT_FRAMES); > + } > + if (State != NULL) { > + FreePool (State); > + } > + return Status; > +} >