From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from out04.hibox.biz (out04.hibox.biz [210.71.195.44]) by mx.groups.io with SMTP id smtpd.web08.33680.1629997661555632136 for ; Thu, 26 Aug 2021 10:07:42 -0700 Authentication-Results: mx.groups.io; dkim=missing; spf=pass (domain: insyde.com, ip: 210.71.195.44, mailfrom: tim.lewis@insyde.com) IronPort-SDR: qCsKenE0hJyW0ryCSQlI2smxKqINWe1hnfq8ihThRnRdFZMyVlOB9OQu96X8I8ZIv5qnuPoEaC DdId0DIa8JNQ== X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: =?us-ascii?q?A2B2AQBhxydh/ww0GKxaHAEBAQEBAQc?= =?us-ascii?q?BARIBAQQEAQFACYE/BAEBCwEBgyBWF1WEAUaJBIgGAzgBj0JcGosYgREDVAs?= =?us-ascii?q?BAQEBAQEBAQEJOQUBAgQBASyBS4IwPwQCAoIxJjcGDgECBAEBARIBAQYBAQE?= =?us-ascii?q?BAQYEgSSFaA1AARABhXABAQIDCAIZJgUIIwwBBQYDDQQEAQEBAgImAhsYHgg?= =?us-ascii?q?CBAESCwULAgSCUYJVAy8PqXN6gTEaAmWDTQGDUxmCPIEQKgGBZIUchnmCUIE?= =?us-ascii?q?VgihRMD6BTwGBEgEBAhiBEByDMYJkBIU9E0gdFzQGFBMBGwoEAgUdDSEDPQg?= =?us-ascii?q?BAhQWCA0EChUFAgQHLgsgD5EYIoMBRosnm0g+gRgHjW2HQAeMUyyDZYFIkEg?= =?us-ascii?q?DF5BtkF+FOIxCk2eFKIF3a4ETcFCCaQk2ERmNVGcWgj+BEYJkggUrgxmCUSQ?= =?us-ascii?q?wAgktAgYBCgEBAwkzhS0IFQGCcIZYAQE?= X-IronPort-AV: E=Sophos;i="5.84,354,1620662400"; d="scan'208";a="56644036" Received: from unknown (HELO hb3-BKT202.hibox.biz) ([172.24.52.12]) by out04.hibox.biz with ESMTP; 27 Aug 2021 01:07:37 +0800 IronPort-SDR: V9nywqotnmq5SjefRwEL03+Goj16XH8gaKK3HPLUTClnlb0IifKPc5Xsq6hpRBmfwJJ+fdpZAM 7Sel6Btw5ynQ== Received: from unknown (HELO hb3-BKT103.hibox.biz) ([172.24.51.13]) by hb3-BKT202.hibox.biz with ESMTP; 27 Aug 2021 01:07:38 +0800 IronPort-SDR: cvacCsW9Rm6P+Zxe6jQKi5+jLP0vIrKwBBJBwrqpHNwgEhMP+StwSceJcfgMQV9ZmEWz634d8e I9f+wVKwXERQ== Received: from unknown (HELO hb3-IN01.hibox.biz) ([172.24.12.11]) by hb3-BKT103.hibox.biz with ESMTP; 27 Aug 2021 01:07:38 +0800 IronPort-SDR: WdTqEKryFADxVNZGXqqiKwpL7rB6TlhWdFMAF8tW6RkCVwS0HIiWmtiGcfpGRTJjQnGiziufJZ 5PPqE2WK9Q4Dv5BFqz1Jeub6097J+cKP8= X-Remote-IP: 73.116.1.175 X-Remote-Host: c-73-116-1-175.hsd1.ca.comcast.net X-SBRS: -10.0 X-MID: 60538257 X-Auth-ID: tim.lewis@insyde.com X-EnvelopeFrom: tim.lewis@insyde.com hiBox-Sender: 1 IronPort-HdrOrdr: =?us-ascii?q?A9a23=3AyBD6aahIxfCzWV8z9HjCa8Nr8XBQXu0ji2?= =?us-ascii?q?hC6mlwRA09TyVXra2TdZMgpHrJYVcqKRMdsPqHP7SNRm6Z8JZz75UYM7vKZn?= =?us-ascii?q?iBhILGFuBfBOfZqAEIXheQygc/79YCT0EdMr3N5DFB5K7HCUuDf+rIq+PozE?= =?us-ascii?q?ncv5a4854Cd2tXgtlbnmNENjo=3D?= Received: from c-73-116-1-175.hsd1.ca.comcast.net (HELO DESKTOPHG9V3E8) ([73.116.1.175]) by hb3-IN01.hibox.biz with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 27 Aug 2021 01:07:33 +0800 From: "Tim Lewis" To: =?utf-8?Q?'Marvin_H=C3=A4user'?= , , Cc: "'Andrew Fish'" , , "'Ni, Ray'" , "'Gao, Zhichao'" , "'Wang, Jian J'" , "'Wu, Hao A'" , "'Bi, Dandan'" , "'Dong, Eric'" , "'Bret Barkelew'" , "'Vitaly Cheptsov'" In-Reply-To: <8ea3fa57-3091-b2c5-dba2-c3eca153b697@posteo.de> Subject: Re: [edk2-devel] [RFC] Expose HII package list via C variables Date: Thu, 26 Aug 2021 10:07:32 -0700 Message-ID: <1f9c01d79a9c$dec69190$9c53b4b0$@insyde.com> MIME-Version: 1.0 X-Mailer: Microsoft Outlook 16.0 Thread-Index: AQHq4zyRw9MvmLhmIMjnn5ZbrLSwqqtfcycg Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Content-language: en-us Hi Marvin -- Sorry, nothing I can share. Thanks, Tim -----Original Message----- From: Marvin H=C3=A4user =20 Sent: Thursday, August 26, 2021 9:07 AM To: tim.lewis@insyde.com; devel@edk2.groups.io; michael.d.kinney@intel.com Cc: 'Andrew Fish' ; leif@nuviainc.com; 'Ni, Ray' ; 'Gao, Zhichao' ; 'Wang, Jian J' ; 'Wu, Hao A' ; 'Bi, Dandan' ; 'Dong, Eric' ; 'Bret Barkelew' ; 'Vitaly Cheptsov' Subject: Re: [edk2-devel] [RFC] Expose HII package list via C variables Hey Tim, Interesting, do you happen to have some tool or code that performs such edi= ts at hand to take a look at? Seems like most modules already use variables= and thus cannot be modified in such a way even right now? That's the kind of information I am looking for, thanks a lot! Best regards, Marvin On 26/08/2021 18:01, tim.lewis@insyde.com wrote: > Hi Marvin -- > > I would like to add some historical perspective on this. One of the desig= n requirements back when HII was first introduced into the UEFI specificati= on after Intel's initial contribution was that of binary editability. In or= der to be able to reliably find, extract and then re-insert the HII data in= to the binary, it needed to be discoverable and not affect the offsets of t= he code and data in the binary. > > While it was possible to put some sort of signature in the read-only data= sections of the binary and leave padding for possible future edits, it was= felt that using a PE/COFF section similar to the resource sections was bet= ter. Resource sections are commonly in use for PE/COFF files (in Windows) a= nd the similar idea existed in the then-extant Mac binary format (resource = fork?). > > Thanks, > > Tim Lewis > CTO, Insyde Software > www.insyde.com > > -----Original Message----- > From: devel@edk2.groups.io On Behalf Of Michael=20 > D Kinney > Sent: Thursday, August 26, 2021 7:37 AM > To: devel@edk2.groups.io; mhaeuser@posteo.de; Kinney, Michael D=20 > > Cc: Andrew Fish ; leif@nuviainc.com; Ni, Ray=20 > ; Gao, Zhichao ; Wang, Jian J=20 > ; Wu, Hao A ; Bi, Dandan=20 > ; Dong, Eric ; Bret Barkelew=20 > ; Vitaly Cheptsov=20 > > Subject: Re: [edk2-devel] [RFC] Expose HII package list via C=20 > variables > > Marvin, > > One constraint in this discussion is that the HII content in a PE/COFF re= source section is defined in the UEFI Specification, Which means UEFI Apps = and UEFI Drivers from media and option ROMs that are not part of the system= FW image are allowed to use this feature, This means the system FW PE/COF= F loader must support loading HII content from this PE/COFF resource sectio= n to be UEFI conformant. So we cannot remove this feature from the PE/COFF= loader without changes to the UEFI Specification. Even if changes to the = UEFI Specification we made, we would have to continue to support this featu= re for backward compatibility with existing UEFI Apps/Drivers that may be u= sing this feature. > > Thanks, > > Mike > >> -----Original Message----- >> From: devel@edk2.groups.io On Behalf Of Marvin=20 >> H=C3=A4user >> Sent: Thursday, August 26, 2021 1:51 AM >> To: devel@edk2.groups.io; Kinney, Michael D=20 >> >> Cc: devel@edk2.groups.io; Andrew Fish ;=20 >> leif@nuviainc.com; Ni, Ray ; Gao, Zhichao=20 >> ; Wang, Jian J ; Wu,=20 >> Hao A ; Bi, Dandan ; Dong,=20 >> Eric ; Bret Barkelew=20 >> ; Vitaly Cheptsov=20 >> >> Subject: Re: [edk2-devel] [RFC] Expose HII package list via C=20 >> variables >> >> Hey Mike, >> >> Thanks for your reply! >> >> Well, this switch is not well-documented. Looking now at both the=20 >> generation code and the emitted code, it does not generate a package=20 >> list like my code does, but separate data variables (strings and >> images) that cannot easily be passed to HiiDatabase as-is. However=20 >> apparently there are drivers that use this functionality successfully=20 >> by composing the package list at runtime [1]. >> >> Looking with this information now at the pattern of using HII C=20 >> variables (which I did not know existed before) vs the PE/COFF HII=20 >> section, most that use latter are Shell applications, which I guess=20 >> means the section has actually been introduced to resolve D.? There=20 >> are exceptions such as LogoDxe [2], which use the PE/COFF section while = D. >> is not a problem, hence I got confused, sorry. I think these modules=20 >> should be updated in any case. Do you agree? >> >> So, for modules that use C variables already, my patch would save=20 >> some runtime generation code and dynamic memory allocation for the=20 >> HII package list. This was not my goal (as I said, I didn't realise=20 >> HII C variables already were a thing in the first place), but the=20 >> changes are small enough that it might be worth considering anyway, in m= y opinion. >> If a HII package list is generated for both Shell and non-Shell apps,=20 >> this also means code paths can be unified. For example, there could=20 >> be a library class with constructor and destructor to add/remove=20 >> packages from the HII database for all modules that use such, Shell=20 >> or not. For BaseTools it means that there is no real need for=20 >> separate Python and C paths as ideally they just generate the exact same= data. >> >> Now to D., the only usage for this seems to be that Shell can locate=20 >> the help text in the executable without executing it, yet it is fully=20 >> loaded anyway [3]. To be honest, I find it hard to justify loading an=20 >> executable (PE/COFF loading, memory permission application, the full >> process) to retrieve a help text and then unloading it again,=20 >> especially with the HII code being on a core dispatcher level. 1. to=20 >> 7. still hold true in my opinion. Was there any discussion I could=20 >> read through why Shell apps cannot simply support a "--help" or "-?" >> command and output the string themselves? Pushing the burden to the=20 >> Shell apps does preserve the "drawback" that a full loading is=20 >> required (which honestly does not matter for a debugging application=20 >> like Shell), however it does relieve the burden of PE/COFF HII=20 >> parsing from the core dispatcher (which matters a lot in my opinion=20 >> to keep the core simple). It would simply be a normal Shell app=20 >> execution as any other however. If someone wants to avoid the PE/COFF=20 >> burden altogether, they can still provide a .man file. >> >> As for my points 6. and 7., maybe I should provide some context. Due=20 >> to many issues with TE files, platforms started abandoning them and=20 >> returned to PE/COFF Images. I think a big reason for this is that TE=20 >> is not really a sound and complete format, but a stripped version of=20 >> PE/COFF with none of the necessary fixups applied. I'm currently=20 >> sketching a possible alternative [4], and I would really like to not=20 >> having to specify a HII section type, while still preserving=20 >> compatibility with all of the UEFI Image types and use-cases [4]. >> >> Thanks again! >> >> Best regards, >> Marvin >> >> >> [1] >> https://github.com/tianocore/edk2/blob/7b4a99be8a39c12d3a7fc4b8db9f0e >> a b4ac688d5/MdeModulePkg/Application/BootManagerMenuAp >> p/BootManagerMenu.c#L929-L934 >> >> [2] >> https://github.com/tianocore/edk2/blob/7b4a99be8a39c12d3a7fc4b8db9f0e >> a >> b4ac688d5/MdeModulePkg/Logo/LogoDxe.inf#L23 >> >> [3] >> https://github.com/tianocore/edk2/blob/7b4a99be8a39c12d3a7fc4b8db9f0eab4= ac688d5/ShellPkg/Application/Shell/ShellManParser. >> c#L646-L671 >> >> [4] >> https://github.com/mhaeuser/edk2/blob/ue_poc/MdePkg/Include/IndustryS >> t >> andard/UeImage.h >> >> 26.08.2021 00:34:12 Michael D Kinney : >> >>> Hi Marvin, >>> >>> I think this feature is already there and supported. >>> >>> HII can either be in a global variable or in a PE/COFF resource section= . >>> The default is a global variable because HII was implemented before=20 >>> the PE/COFF resource section feature was added to the UEFI Specificatio= n. >>> >>> There is an INF [Defines] section statement to enable the PE/COFF=20 >>> section. See UefiHiiResource in the following link. >>> >>> https://tianocore-docs.github.io/edk2-InfSpecification/draft/3_edk_i >>> i_inf_file_format/34_[defines]_section.html#34- >> defines-section >>> How is your proposal different than this existing capability? >>> >>> Thanks, >>> >>> Mike >>> >>>> -----Original Message----- >>>> From: devel@edk2.groups.io On Behalf Of=20 >>>> Marvin H=C3=A4user >>>> Sent: Wednesday, August 25, 2021 2:21 PM >>>> To: devel@edk2.groups.io >>>> Cc: Andrew Fish ; leif@nuviainc.com; Kinney,=20 >>>> Michael D ; Ni, Ray ;=20 >>>> Gao, Zhichao ; Wang, Jian J=20 >>>> ; Wu, Hao A ; Bi, Dandan=20 >>>> ; Dong, Eric ; Bret=20 >>>> Barkelew ; Vitaly Cheptsov=20 >>>> >>>> Subject: [edk2-devel] [RFC] Expose HII package list via C variables >>>> >>>> Good day everyone, >>>> >>>> Currently, the HII package list is stored in a PE/COFF resource=20 >>>> section [1]. I propose to store it in a C variable (byte array with=20 >>>> a pointer to it and its size exposed) instead. DxeCore would have a=20 >>>> guard to toggle the deprecated support for the automatic protocol=20 >>>> installation. This has the following advantages: >>>> >>>> 1. Fixes BZ (incl. future toolchains): >>>> https://bugzilla.tianocore.org/show_bug.cgi?id=3D557 >>>> 2. Universal method across all toolchains and output file formats=20 >>>> 3. Saves error-prone parsing work 4. Saves protocol install/locate=20 >>>> work, the data is available right away 5. The omission of a=20 >>>> dedicated section can save space 6. Terse file formats can support=20 >>>> this and remain terse :) 7. Removes a dependency on the PE/COFF=20 >>>> format specifically >>>> >>>> A *very rough* PoC diff can be found here: >>>> https://github.com/mhaeuser/edk2/compare/master...wip_hii_cvar >>>> If the feedback is positive, I will clean it up of course. OVMF=20 >>>> boots with everything working fine. >>>> >>>> I'd explicitly like feedback on the following: >>>> A. Is this an acceptable solution to BZ 557 (Andrew?)? >>>> B. Is this an acceptable solution for the "HII workflow" (MdeModule=20 >>>> maintainers?)? >>>> C. Is it acceptable to make support UEFI-side support for the old=20 >>>> mechanism optional (Stewards?)? >>>> D. Can an acceptable alternative be found for the removed ShellPkg=20 >>>> code (Shell maintainers?)? >>>> >>>> As you can see the BaseTools part also is rough, but that is more a=20 >>>> question of "how" rather than "whether", so I'll postpone asking about= it. >>>> >>>> Thanks for your time and feedback! >>>> >>>> Best regards, >>>> Marvin >>>> >>>> >>>> [1] "Once the image is loaded, LoadImage() installs=20 >>>> EFI_HII_PACKAGE_LIST_PROTOCOL on the handle if the image contains a=20 >>>> custom PE/COFF resource with the type 'HII'." >>>> - UEFI 2.9, 7.4, "EFI_BOOT_SERVICES.LoadImage()" >>>> >>>> >>>> >>>> >>> >>> >>> >> >> >> > > >=20 > > >