From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from rn-mailsvcp-ppex-lapp35.apple.com (rn-mailsvcp-ppex-lapp35.apple.com [17.179.253.44]) by mx.groups.io with SMTP id smtpd.web09.1190.1643331343429980968 for ; Thu, 27 Jan 2022 16:55:43 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@apple.com header.s=20180706 header.b=Zct1IgdK; spf=pass (domain: apple.com, ip: 17.179.253.44, mailfrom: afish@apple.com) Received: from pps.filterd (rn-mailsvcp-ppex-lapp35.rno.apple.com [127.0.0.1]) by rn-mailsvcp-ppex-lapp35.rno.apple.com (8.16.1.2/8.16.1.2) with SMTP id 20S0meTc003632; Thu, 27 Jan 2022 16:55:40 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=apple.com; h=from : message-id : content-type : mime-version : subject : date : in-reply-to : cc : to : references; s=20180706; bh=6eMRTaBt9By3HE4TuMek1FJ2OD+1WZxpYM/tP19cUd8=; b=Zct1IgdKuR4oL0xFGiPRIaHjQ2D676UDo9cBVUlcYtT/CJw+0AFakMl/Gg0vauMpaZvl WHpybKrrfDf+9/lNkcDz22/y2sYbavjgEoI74X9clE7TCWrrvEtFlRJ8Do6wX7WtVkvv ocoeURvJAsG0hAjBZj7Xnysq9IN4d/9ZdY6SOJzGI8hxU0QDhab2sFc1uIurVPjdrzQ4 IsSscUwplu6KTVn0eZ9W1q12YdLvwjEou+ETWkcqwNzSMAdNPxaLKzS7zWYRt6VB7n5T 7T3dlvdbUSbOhU6o/P1uXFVaOypNnZL1vEdWnrwUlErudicnNjSg/cdQ3bZXmh0nHwkq pQ== Received: from rn-mailsvcp-mta-lapp03.rno.apple.com (rn-mailsvcp-mta-lapp03.rno.apple.com [10.225.203.151]) by rn-mailsvcp-ppex-lapp35.rno.apple.com with ESMTP id 3drebc2g02-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO); Thu, 27 Jan 2022 16:55:39 -0800 Received: from rn-mailsvcp-mmp-lapp02.rno.apple.com (rn-mailsvcp-mmp-lapp02.rno.apple.com [17.179.253.15]) by rn-mailsvcp-mta-lapp03.rno.apple.com (Oracle Communications Messaging Server 8.1.0.12.20210903 64bit (built Sep 3 2021)) with ESMTPS id <0R6E00GKPAKROKM0@rn-mailsvcp-mta-lapp03.rno.apple.com>; Thu, 27 Jan 2022 16:55:39 -0800 (PST) Received: from process_milters-daemon.rn-mailsvcp-mmp-lapp02.rno.apple.com by rn-mailsvcp-mmp-lapp02.rno.apple.com (Oracle Communications Messaging Server 8.1.0.12.20210903 64bit (built Sep 3 2021)) id <0R6E01000ADF1M00@rn-mailsvcp-mmp-lapp02.rno.apple.com>; Thu, 27 Jan 2022 16:55:39 -0800 (PST) X-Va-A: X-Va-T-CD: 173a35b795a4f575213dfa7d1cb9411a X-Va-E-CD: 730a34adac2a0aaa593a853de85f4f8b X-Va-R-CD: 5ed789e713db4cabc19cb3c2df9607b2 X-Va-CD: 0 X-Va-ID: 51492704-b435-4a21-975c-e7a091a8c2bd X-V-A: X-V-T-CD: 173a35b795a4f575213dfa7d1cb9411a X-V-E-CD: 730a34adac2a0aaa593a853de85f4f8b X-V-R-CD: 5ed789e713db4cabc19cb3c2df9607b2 X-V-CD: 0 X-V-ID: ee279ed7-a97d-4667-822d-6fd2b821da03 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.425,18.0.816 definitions=2022-01-27_06:2022-01-27,2022-01-27 signatures=0 Received: from smtpclient.apple (unknown [17.235.48.130]) by rn-mailsvcp-mmp-lapp02.rno.apple.com (Oracle Communications Messaging Server 8.1.0.12.20210903 64bit (built Sep 3 2021)) with ESMTPSA id <0R6E00EI6AKPZF00@rn-mailsvcp-mmp-lapp02.rno.apple.com>; Thu, 27 Jan 2022 16:55:39 -0800 (PST) From: "Andrew Fish" Message-id: <71737233-2840-42CE-A65E-1895E936B4CB@apple.com> MIME-version: 1.0 (Mac OS X Mail 15.0 \(3693.20.0.1.32\)) Subject: Re: [edk2-devel] [PATCH 00/24] CryptoPkg/openssl: update openssl submodule to v3.0 Date: Thu, 27 Jan 2022 16:55:37 -0800 In-reply-to: Cc: "kraxel@redhat.com" , Mike Kinney , "Yao, Jiewen" , Sean Brogan , Bret Barkelew , "Wang, Jian J" , "Jiang, Guomin" , Pawel Polawski , "Lu, XiaoyuX" , Pedro Falcato To: edk2-devel-groups-io , "KILIAN_KEGEL@outlook.com" References: <20211203160748.866150-1-kraxel@redhat.com> <20220117114627.ji5cyqxkca6bmiaf@sirius.home.kraxel.org> <20220121083035.dsqzu3akshonliza@sirius.home.kraxel.org> <20220126110244.klk24znojvdtirzw@sirius.home.kraxel.org> X-Mailer: Apple Mail (2.3693.20.0.1.32) X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.425,18.0.816 definitions=2022-01-27_06:2022-01-27,2022-01-27 signatures=0 Content-type: multipart/alternative; boundary="Apple-Mail=_2E5DCBE6-CC8D-417C-920C-08BABBE60071" --Apple-Mail=_2E5DCBE6-CC8D-417C-920C-08BABBE60071 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Very cool idea but =E2=80=A6.. 1) We don=E2=80=99t always use the systems native compiler and sometimes we= use a cross compiler so making assumptions about system libs is not always= valid. On a Mac with Xcode clang I=E2=80=99ve got full SysV ABI libs (not supper h= elpful for EFI), but not EFI/MSFT x86_64 ABI. For bonus point i386 has been= obsoleted in Xcode. ~/work/Compiler/Math>cat hello.c =20 #include int main(int argc, char **argv) { unsigned long long ulldiv =3D ((unsigned long long) argc)/3; // prevent the optimizer to calculate result at build time printf("ulldiv %lld\n", ulldiv ); } ~/work/Compiler/Math>clang hello.c Thus I can link Sys V ABI with the wrong calling convention for EFI. But for the EFI ABIs not so much=E2=80=A6. ~/work/Compiler/Math>clang -target x86_64-pc-win32-macho hello.c clang: warning: unable to find a Visual Studio installation; try running Cl= ang from a developer command prompt [-Wmsvc-not-found] hello.c:1:10: fatal error: 'stdio.h' file not found #include ^~~~~~~~~ 1 error generated. ~/work/Compiler/Math>clang -arch i386 hello.c =20 ld: warning: The i386 architecture is deprecated for macOS (remove from the= Xcode build setting: ARCHS) ld: warning: ignoring file /Applications/Xcode.app/Contents/Developer/Platf= orms/MacOSX.platform/Developer/SDKs/MacOSX12.0.sdk/usr/lib/libSystem.tbd, m= issing required architecture i386 in file /Applications/Xcode.app/Contents/= Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.0.sdk/usr/lib/l= ibSystem.tbd (3 slices) Undefined symbols for architecture i386: "_printf", referenced from: _main in hello-380c0a.o ld: symbol(s) not found for architecture i386 clang: error: linker command failed with exit code 1 (use -v to see invocat= ion) 2) Are these system libs architecturally defined to be free standing? Can t= hey make assumptions about the runtime? If so seems like they could do rand= om bad things to make them not work in EFI like a system call or make other= assumptions like writing to a magic address to generate a trap. Just inspe= cting them tells us the implementation, not the architecture of the lib. 3) Are the paths to these libs architectural or are they arbitrary implemen= tation that is normal abstracted by how the compiler is released? Could som= e packaging change break the magic paths to libs? 4) I asked the Xcode/clang team a long time ago what to do for free standin= g match and they pointed me at some open source implementation of these mat= h libs that had been implemented in C. They did not want us using their lib= s that shipped with macOS.=20 For the GCC/clang tools seems like we are better off just providing the cod= e.=20 We have magic for compiler specific inline assembly=20 https://github.com/tianocore/edk2/blob/master/MdePkg/Library/BaseLib/Ia32/G= ccInline.c We have magic to abstract some compiler intrinsics today: https://github.com/tianocore/edk2/blob/master/MdePkg/Library/BaseIoLibIntri= nsic/IoLibMsc.c https://github.com/tianocore/edk2/blob/master/MdePkg/Library/BaseIoLibIntri= nsic/IoLibGcc.c While we try to avoid it when at all possible the build system does support= doing things differently for different compilers if we have to https://github.com/tianocore/edk2/blob/master/MdePkg/Library/BaseIoLibIntri= nsic/BaseIoLibIntrinsic.inf#L38 [Sources.IA32] IoLibGcc.c | GCC IoLibMsc.c | MSFT IoLib.c Thanks, Andrew Fish PS The compiler still works like you think we just don=E2=80=99t have the l= ibs you might need.=20 ~/work/Compiler/Math>cat i386.c =20 typedef unsigned long long UINT64; UINT64 main2(int argc, char **argv) { UINT64 ulldiv =3D ((UINT64) argc)/3; // prevent the optimizer to calculate result at build time return ulldiv; } ~/work/Compiler/Math>clang -arch i386 i386.c -S ~/work/Compiler/Math>cat i386.S =20 .section __TEXT,__text,regular,pure_instructions .build_version macos, 12, 0 sdk_version 12, 0 .globl _main2 ## -- Begin function main2 .p2align 4, 0x90 _main2: ## @main2 .cfi_startproc ## %bb.0: pushl %ebp .cfi_def_cfa_offset 8 .cfi_offset %ebp, -8 movl %esp, %ebp .cfi_def_cfa_register %ebp subl $24, %esp movl 12(%ebp), %eax movl 8(%ebp), %eax movl 8(%ebp), %ecx movl %ecx, %edx sarl $31, %edx movl %esp, %eax movl %edx, 4(%eax) movl %ecx, (%eax) movl $0, 12(%eax) movl $3, 8(%eax) calll ___udivdi3 movl %edx, -4(%ebp) movl %eax, -8(%ebp) movl -8(%ebp), %eax movl -4(%ebp), %edx addl $24, %esp popl %ebp retl .cfi_endproc ## -- End function .subsections_via_symbols PPS Fun story about ABI differences as the macOS i386 ABI requires 16-byte = aligned stack accesses and that is more strict than EFI. Luckily it does no= t break EFI, but it means you can NOT call macOS code from EFI code. To mak= e the emulator work on macOS I had to write assembly gaskets to align the s= tacks to make calls between the worlds possible. Not my finest hour but it = works=E2=80=A6. https://github.com/tianocore/edk2/blob/master/EmulatorPkg/Unix/Host/Ia32/Ga= sket.S This crazy is an example about how assumptions that are not EFI centric can= leak into the generic libs produced for the compiler. > On Jan 27, 2022, at 2:26 PM, Kilian Kegel wrot= e: >=20 > Hi Gerd, > =20 > >* On my system the gcc intrinsics are only available as shared library, > > so the "just unpack the lib and use the object files" idea is not > > going to work. > > =20 > This little C program makes an unsigned 64Bit division on PC compilers. > Running a 32Bit code generator, it usually invokes an intrinsic function. > MSFT: __aulldiv() > GCC: __udivdi3() > =20 > On my 32Bit Ubuntu standard installation I ran > cc - Xlinker -Map=3Dstatic.map hello.c -static > cc -Xlinker -Map=3Dshared.map hello.c > =20 > The first .OBJ file mentioned in the .MAP file is in both cases: > /usr/lib/gcc/i686-linux-gnu/6/libgcc.a(_udivdi3.o) > > =20 > <377AC53F424C47F794809BA1A5953904.png> > =20 > Then for each a.out I did: > objdump -d a.out > static.dis > objdump -d a.out > shared.dis > =20 > In both cases the intrinsic function is fully linked into the .ELF execut= able. > > =20 > >so the "just unpack the lib and use the object files" idea is not > >going to work. > It seems to me that GNU holds the intrinsic functions in a separate libra= ry > that can be used without any change, and is always correct by definition. > =20 > For Microsoft that is only true when a SDK is installed (INT64.LIB). > Without SDK the intrinsic functions were included in LIBCMT.LIB and > must be isolated manually. > =20 > Gerd, can you please doublecheck in your GCC build, if that works: > add a 64Bit div to an x86 PEI module like: > <500D8F2283CD4FAE9B0E45647823894A.png> > =20 > add libgcc.a as a search library, adjust the conf\tools_def.txt like: > DEBUG_GCCxx_IA32_DLINK_FLAGS =3D =E2=80=A6predefined parameter =E2=80= =A6 /usr/lib/gcc/i686-linux-gnu/6/libgcc.a > to match your build system > build the BIOS=20 > if the build gets ready, check the .MAP file whether it contains __udivd= i3() or not > =20 > >* I have my doubts that compiler's builtin libraries are optimized for > > size, so I'd suspect we would see a noticeable size grow from that. > Please check the size of __udivdi3() and whether the tianocore reimplemen= tation is smaller or not > =20 > If this works for all build platforms, independently of using the tianoco= re reimplementation or > using the original compiler intrinsics, this is correct location to place= the address of the intrinsic library. > For all optimization modes, operation modes (64Bit/32Bit) the intrinsic l= ibrary is available for the compiler. > =20 > GNU lists the intrinsics: > https://gcc.gnu.org/onlinedocs/gccint/Libgcc.html#Libgcc > =20 > The intrinsic library belongs to the compiler not to the build system. > If the build system changes from EDK2 to VS2022/MSBUILD, the compiler > expects the same intrinsic library. > =20 > Leave the intrinsic restrictions behind and just provide all required int= rinsics the compiler needs > to fulfil the C-Standard! > =20 > Thanks a lot, > Kilian > https://github.com/tianocore/edk2-staging/tree/CdePkg#cdepkgblog > =20 > From: kraxel@redhat.com > Sent: Wednesday, January 26, 2022 12:02 PM > To: Pedro Falcato > Cc: edk2-devel-groups-io ; Kinney, Michael D= ; Kilian Kegel ; Yao, Jiewen ; Sean Brogan ; Bret Barkelew ; Wang, Jian J ; Jiang, Guomin ; Pawel Polawski ; Lu, Xiaoy= uX > Subject: Re: [edk2-devel] [PATCH 00/24] CryptoPkg/openssl: update openssl= submodule to v3.0 > =20 > Hi, >=20 > > I think adding intrinsic libraries is a mixed bag, for the following > > reasons: > >=20 > > 1) The intrinsic libraries are completely internal to the compilers. > > Breaking down that toolchain/code barrier is not a good idea if we want= the > > project to compile using a good variety of compilers. The API is unstab= le > > (as it's internal to the compiler + version) and sometimes undocumented= ; > > while in reality the ABI doesn't really change (at least in the LLVM/GC= C > > world, not sure about the other compilers), it's something to consider. >=20 > Yes. But apparently there is no way around them. We have them for arm. > We have them for openssl. IntelUndiPkg in edk2-staging has some too > (see IntelUndiPkg/LibC). And I wouldn't be surprised if there are more > cases ... >=20 > Having a policy to outlaw Intrinsics, but then hand out exceptions left > and right doesn't look like a good idea to me. I think we should revisit > that and accept that there simply is no way around Intrinsics in some > cases. >=20 > I think it makes sense to consolidate all the Intrinsics we have, i.e. > move them over to MdePkg, make everybody use that, so we have only a > single version to maintain. >=20 > I think it also makes sense to restrict Intrinsics to the cases where we > have no other option, to keep them as small as possible and also make it > as easy as possible to maintain them. >=20 > > 2) Linking the compiler's builtin libraries would fix our issues, excep= t > > that it doesn't work in cases where object files are tagged with ABI (s= uch > > as hard FP vs soft FP). >=20 > Also: >=20 > * On my system the gcc intrinsics are only available as shared library, > so the "just unpack the lib and use the object files" idea is not > going to work. >=20 > * I have my doubts that compiler's builtin libraries are optimized for > size, so I'd suspect we would see a noticeable size grow from that. >=20 > * I'd very much prefer to continue with the current approach to have > source code for the Intrinsics we need. In case we run into trouble > things tend to be much easier to fix when you have the source code at > hand. That's actually part of the open source success story. >=20 > take care, > Gerd >=20 > =20 > Sent from Mail for Win= dows > =20 > From: Kilian Kegel > Sent: Tuesday, January 25, 2022 09:06 PM > To: Kinney, Michael D ; devel@edk2.gro= ups.io ; kraxel@redhat.com ; Yao, Jiewen ; Sean Brogan ; Bret Barkelew > Cc: devel@edk2.groups.io ; Wang, Jian J ; Jiang, Guomin ; = Pawel Polawski ; Lu, XiaoyuX > Subject: Re: [edk2-devel] [PATCH 00/24] CryptoPkg/openssl: update openssl= submodule to v3.0 > =20 > Hi Mike, > =20 > thank you for your explanation. I understand all the technical aspects. > But let me go into the details of my approach, that skips step 2) to 5) > and adds step 1.5) > =20 > >I think in practice, the intrinsic APIs we are seeing from use > >of C code from submodules is a very limited set that do not > >change across compiler releases, > I totally agree. E.g INT64.LIB is the same since 2004 (WinXP DDK). > =20 > But my perspective is different anyway: > an intrinsic library belongs to a particular compiler/linker couple = =20 > an intrinsic library does not belong to a UEFI tianocore or commercial BI= OS feature set and should be managed > outside from any EDK2 specific build description (.INF, .DSC) > =20 > Let=E2=80=99s assume there is a C build environment fully installed, e.g= . Microsoft VS2022, on an EDK2 build machine > and all legal aspects were fulfilled. > =20 > In that case the advanced developer is able to locate the library, that h= olds the intrinsic functions > (intrinsic .OBJ modules) we needed to extract. (simply by checking the .M= AP of a 32bit executable > that pulls in the particular intrinsics)=20 > This is step 1) > =20 > >One of the challenges is that compilers are allowed to add/remove/modify= intrinsic APIs > >across compiler releases. We would need to define a solution that will = work if there are > >these types of changes, which would potentially mean a different instanc= e of the intrinsic > >library for each tool chain tag. =20 > =20 > Here comes step 1.5): > In case of Microsoft build it is LIBCMT.LIB that can be found when walkin= g through the > LIB environment string. > =20 > It is easy to extend EDKSETUP.BAT to generate MSFTINTRINx86-32.LIB each t= ime: > locate LIBCMT.LIB > extract the identified .OBJ modules from step 1: =E2=80=9CLIB.EXE /extrac= t:full_path_name.obj /out:name.obj LIBCMT.LIB=E2=80=9D > bind extracted .OBJ to the MSFTINTRINx86-32.LIB: =E2=80=9CLIB.EXE *.obj /= out:%CONF_PATH%\MSFTINTRINx86-32.lib=E2=80=9D > =20 > Now MSFTINTRINx86-32.LIB is located in the conf directory. > =20 > Adjust the DLINK_FLAGS in tools_def.txt to hold MSFTINTRINx86-32.LIB as a= search library: > =20 > DEBUG_VS2015_IA32_DLINK_FLAGS =3D /NOLOGO /NODEFAULTLIB /IGNORE:4001 = /OPT:REF /OPT:ICF=3D10 /MAP /ALIGN:32 /SECTION:.xdata,D /SECTION:.pdata,D /= MACHINE:X86 /LTCG /DLL /ENTRY:$(IMAGE_ENTRY_POINT) /SUBSYSTEM:EFI_BOOT_SERV= ICE_DRIVER /SAFESEH:NO /BASE:0 /DRIVER /DEBUG %CONF_PATH%\MSFTINTRINx86-32.= LIB > =20 > RELEASE_VS2015_IA32_DLINK_FLAGS =3D /NOLOGO /NODEFAULTLIB /IGNORE:4001 = /IGNORE:4254 /OPT:REF /OPT:ICF=3D10 /MAP /ALIGN:32 /SECTION:.xdata,D /SECTI= ON:.pdata,D /MACHINE:X86 /LTCG /DLL /ENTRY:$(IMAGE_ENTRY_POINT) /SUBSYSTEM:= EFI_BOOT_SERVICE_DRIVER /SAFESEH:NO /BASE:0 /DRIVER /MERGE:.rdata=3D.data %= CONF_PATH%\MSFTINTRINx86-32.LIB > =20 > From now on the intrinsics are available for all 32Bit components. > =20 > With that procedure it is guaranteed by design, that the intrinsics are a= lways available and match a particular compiler/linker release. > it saves storage space since source code or binary modules don=E2=80=99t = need to be kept > because they are not kept, they don=E2=80=99t need maintainance > no one needs to understand and document (in math details) the internals a= nd the interface > no one needs to test for the math functions, as it is necessary for tiano= core re-implementations > the compiler vendor itself is in charge in a court case > =20 > The script below is a demonstration of the above arguments. It additional= ly adds memcpy() and memcmp() to MSFTINTRINx86-32.LIB, > that the compiler sometimes needs, depending on optimization style, array= size, instruction set, whatsoever =E2=80=A6 > > =20 > I have checked some more .OBJ from LIBCMT.LIB and some of them (ftol3.obj= to get __ltod3() long to double needed for difftime())=20 > seem not to be space optimized for BIOS usage, because the ftol3.obj hold= s multiple functions > (and so violates the ODR one definition rule). > =20 > But also such cases could be handled automatically by a script (I wrote a= C program < 200 lines) > that disassembles (using Microsoft DUMPBIN.EXE) the .OBJ, splits by simpl= e text processing into > single-function-.ASM-files that could be assembled back to multiple .OBJ = of smaller code size. > =20 > For this approach compiler, library manager, disassembler (DUMPBIN, OBJDU= MP) were needed, > that are available on all build machines by definition. > =20 > Best regards > Kilian > =20 > =20 > From: Kinney, Michael D > Sent: Monday, January 24, 2022 06:28 PM > To: Kilian Kegel ; devel@edk2.groups.io = ; kraxel@redhat.com = ; Yao, Jiewen ; Sean Brogan ; Bret Barkelew ; Kinne= y, Michael D > Cc: devel@edk2.groups.io ; Wang, Jian J ; Jiang, Guomin ; = Pawel Polawski ; Lu, XiaoyuX > Subject: RE: [edk2-devel] [PATCH 00/24] CryptoPkg/openssl: update openssl= submodule to v3.0 > =20 > Hi Kilian, > =20 > I am in favor of an intrinsic lib to improve the EDK II development envir= onment. > =20 > This has already been done for ARM compilers. The solution should mirror= that approach. > =20 > It would be best if we had source code (either in the edk2 repo or throug= h a submodule) for > the required intrinsic APIs. If source code is not possible and we have = to use a binary, then > that must be accessed through a submodule. The edk2 repo does not host b= inaries. We use > repos such as edk2-non-osi for binaries. > =20 > We also have to provide a solution that works with supported compilers (V= S, GCC, CLANG, XCODE). > =20 > One of the challenges is that compilers are allowed to add/remove/modify = intrinsic APIs > across compiler releases. We would need to define a solution that will w= ork if there are > these types of changes, which would potentially mean a different instance= of the intrinsic > library for each tool chain tag. I think in practice, the intrinsic APIs= we are seeing from use > of C code from submodules is a very limited set that do not change across= compiler releases, > so the maintenance of these intrinsic libs would be manageable. > =20 > If we go down the source code path, we can break it up into the following= tasks: > Identify the specific subset of intrinsic APIs from each compiler that is= required for the edk2 use cases.=20 > Obtain the function prototype and full documentation for each intrinsic A= PI to support implementation and unit tests. > Implement the APIs for all compilers. > Implement unit tests for all APIs for all compilers using UnitTestFramewo= rkPkg unit tests. > Update MdeLibs.dsc.inc with the NULL instances for the intrinsic libs > Remove intrinsic APIs from EDK II modules that currently maintain their o= wn implementations of intrinsic APIs. > =20 > Best regards, > =20 > Mike > =20 > From: Kilian Kegel >=20 > Sent: Monday, January 24, 2022 8:25 AM > To: devel@edk2.groups.io ; Kinney, Michael D= >; kraxel@r= edhat.com ; Yao, Jiewen >; Sean Brogan >; Bret Barkelew > > Cc: devel@edk2.groups.io ; Wang, Jian J >; Jiang, Guomin >; Pawel Polawski >; Lu, XiaoyuX > > Subject: RE: [edk2-devel] [PATCH 00/24] CryptoPkg/openssl: update openssl= submodule to v3.0 > =20 > The 64-bit integer math intrinsics and other intrinsic > problems could be solved easily for ever: > =20 > Putting all .OBJ files together from LIBCMT.H or INT64.LIB (for ll*.obj a= nd ull*.obj only) > ltod3.obj > ftol2.obj > lldiv.obj > lldvrm.obj > llmul.obj > llrem.obj > llshl.obj > llshr.obj > ulldiv.obj > ulldvrm.obj > ullrem.obj > ullshr.obj > memcmp.obj > memcpycpy.obj > and adjust for usability in EDK2 (remove / solve further = internal dependencies or rewrite =E2=80=9C*cpy=E2=80=9D and =E2=80=9C*cmp= =E2=80=9D functions) > This is already done in IntrinsicLib.lib for some of the above functions,= just complete this task! > Put all the .OBJ into a e.g. edk2\Conf \=E2=80=9CMSFTINTRINx86-32.lib=E2=80=9D > Update the MSFT_DEF.txt tool chain definition path > DEBUG_*_IA32_DLINK_FLAGS =3D %CONF_PATH%\ MSFTINTRINx86-32.lib > RELEASE_*_IA32_DLINK_FLAGS =3D %CONF_PATH%\ MSFTINTRINx86-32.lib > Resolve build conflicts with other existing intrinsic libraries from Cryp= toPkg, RedfishPkg=E2=80=A6 =E2=80=93 remove these libraries > =20 > From now on all existing 32Bit components have access to their own compil= er intrinsics without > touching any .INF file and the problem is instantly gone. > =20 > Please do the same for=20 > GCCINTRINx86-32.lib > =20 > Leave the intrinsic restrictions behind and just provide all required int= rinsics the compiler needs > to fulfil the C-Standard! > =20 > UEFI shall conform the execution environment described in the C Specifica= tion > http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1256.pdf#page=3D23 > and shall not try to create a new restricted =E2=80=9CUEFI execution envi= ronment=E2=80=9D > that currently prohibits some =E2=80=9Cexpressions=E2=80=9D (shift << >> = , divide / % ) on some =E2=80=9Cdata types=E2=80=9D (64bit =E2=80=9Clong lo= ng=E2=80=9D) > but maybe in the future will prohibit some more =E2=80=9Cexpressions=E2= =80=9D (logical AND &&, relational-expression < >) on > still speculative =E2=80=9Cdata types=E2=80=9D (e.g. a 128bit =E2=80=9Cex= tended long=E2=80=9D) or just because a new compiler > (version) with some new optimization(ultra slow)/security(specdown/meltre= ) capabilities introduces > some new intrinsic functions. > Who knows=E2=80=A6 > =20 > In contrast to: > =E2=80=9CI think we shouldn't add any intrinsics unless we are absolutely= forced > to. I do agree however that, for those intrinsics that we cannot at all > avoid reimplementing, we should at least collect them in a common > library. > (In theory, I can also imagine reimplementing all possible intrinsics > *if* the edk2 coding style spec / requirements are updated in parallel, > permitting all new code to universally rely on the intrinsics, rather > than the BaseLib / BaseMemoryLib functions.)=E2=80=9D > https://bugzilla.tianocore.org/show_bug.cgi?id=3D1516#c2 > =20 > This mindset violates edk2 coding style spec too:=20 > https://edk2-docs.gitbook.io/edk-ii-c-coding-standards-specification/2_gu= iding_principles > Maintainability > Extensibility > Intellectual manageability > Portability > Reusability > Standard techniques > =20 > Have fun, > Kilian > =20 > From: Michael D Kinney > Sent: Friday, January 21, 2022 05:39 PM > To: kraxel@redhat.com ; Yao, Jiewen ; Sean Brogan ; Bret Ba= rkelew ; Kinney, Michael D > Cc: devel@edk2.groups.io ; Wang, Jian J ; Jiang, Guomin ; = Pawel Polawski ; Lu, XiaoyuX > Subject: Re: [edk2-devel] [PATCH 00/24] CryptoPkg/openssl: update openssl= submodule to v3.0 > =20 > Comments below. >=20 > Mike >=20 > > -----Original Message----- > > From: kraxel@redhat.com > > > Sent: Friday, January 21, 2022 12:31 AM > > To: Yao, Jiewen > > > Cc: devel@edk2.groups.io ; Kinney, Michael= D >; Wang, = Jian J >; Jiang, Guomi= n > > >; Pawel Polawsk= i >; Lu, XiaoyuX > > > Subject: Re: [edk2-devel] [PATCH 00/24] CryptoPkg/openssl: update opens= sl submodule to v3.0 > >=20 > > > > No changes in SEC and PEI. > > > [Jiewen] Do you mean the Crypto consumer in PEI has no size differenc= e? Such as > > > https://github.com/tianocore/edk2/tree/master/SecurityPkg/Tcg/Tcg2Pei= , > > > https://github.com/tianocore/edk2/tree/master/SecurityPkg/FvReportPei= , > > > https://github.com/tianocore/edk2/tree/master/SignedCapsulePkg/Univer= sal/RecoveryModuleLoadPei linking > > https://github.com/tianocore/edk2/tree/master/SecurityPkg/Library/FmpAu= thenticationLibRsa2048Sha256 . > >=20 > > PEI has this (OvmfIa32X64Pkg build): > >=20 > > 7062 TpmMmioSevDecryptPei > > 7830 StatusCodeHandlerPei > > 7902 ReportStatusCodeRouterPei > > 8470 FaultTolerantWritePei > > 9734 SmmAccessPei > > 11206 Tcg2ConfigPei > > 11842 PeiVariable > > 14730 Tcg2PlatformPei > > 17274 TcgPei > > 18438 S3Resume2Pei > > 18682 DxeIpl > > 18938 PcdPeim > > 38014 CpuMpPei > > 39554 PlatformPei > > 45050 PeiCore > > 49274 Tcg2Pei > >=20 > > No size change for Tcg2Pei. > >=20 > > The other modules are not there. Seems they are related to firmware > > updates. We don't have that on ovmf as we can simply update the > > firmware image files on the host machine ... > >=20 > > Is there some target I could use to test-build those modules? > >=20 > > > > INFO - OpensslLibCrypto.lib(rsa_lib.obj) : error LNK2001: unresolve= d external > > > > symbol __allmul > > > > INFO - OpensslLibCrypto.lib(rsa_lib.obj) : error LNK2001: unresolve= d external > > > > symbol __aulldiv > > > > INFO - OpensslLibCrypto.lib(bio_print.obj) : error LNK2001: unresol= ved external > > > > symbol __aulldvrm > > > > INFO - OpensslLibCrypto.lib(bio_print.obj) : error LNK2001: unresol= ved external > > > > symbol __ftol2_sse > > > > > > > > Those symbols look like they reference helper functions to do 64bit= math > > > > on 32bit architecture. Any hints how to fix that? > > > [Jiewen] Please add them to https://github.com/tianocore/edk2/tree/ma= ster/CryptoPkg/Library/IntrinsicLib > >=20 > > Any hints where I could get them? Given this happens on windows builds > > it's probably somewhere in the microsoft standard C library? Is that > > available as open source somewhere? >=20 > Sean and Bret may be able to help with these. >=20 > There is also a BZ on this topic. >=20 > https://bugzilla.tianocore.org/show_bug.cgi?id=3D1516 >=20 > >=20 > > > > (3) Some NOOPT builds are failing due to the size growing ... > > > [Jiewen] Size becomes big challenge... > > > Have you tried to use https://github.com/tianocore/edk2/tree/master/C= ryptoPkg/Driver solution? > >=20 > > Seems the idea is to have only one openssl copy in the dxe image by > > calling a protocol instead of linking a lib. Makes sense. > >=20 > > Is this documented somewhere? Is there some easy way to use that as > > drop-in replacement? Or do we have to change all crypto users to call > > the driver instead of linking the lib? > >=20 > > take care, > > Gerd >=20 >=20 >=20 > =20 > =20 > =20 >=20 > --Apple-Mail=_2E5DCBE6-CC8D-417C-920C-08BABBE60071 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 Very cool idea but =E2=80= =A6..

1) We don=E2=80= =99t always use the systems native compiler and sometimes we use a cross co= mpiler so making assumptions about system libs is not always valid.

On a Mac with Xcode clang I=E2= =80=99ve got full SysV ABI libs (not supper helpful for EFI), but not EFI/M= SFT x86_64 ABI. For bonus point i386 has been obsoleted in Xcode.

~/work/Compiler/Math>cat hello.c         =                      = ;  
#include <s= tdio.h>

i= nt main(int argc, char **argv)
{
  unsign= ed long long ulldiv =3D ((unsigned long long) argc)/3;

  // prevent the optimizer = to calculate result at build time
  printf("ulldiv %lld\n", ulldiv );
}
~/work/Compiler/Math>clang hello.c
T= hus I can link Sys V ABI with the wrong calling convention for EFI.<= /span>

<= /div>
But for the EFI ABIs not so much=E2=80=A6.
~/work/Compiler/Math>clang -target x86_64-pc-win32-macho hell= o.c
clang: warning: unable to find a Visual = Studio installation; try running Clang from a developer command prompt [-Wm= svc-not-found]
= hello.c:1:10: fatal error: <= /b>'stdio.h' file not found
#include <stdio.h>
 =         ^~~~~~~~~
1 error generated.
~/work/Compiler/Math>clang -arch i386 hello.c        &nb= sp;          
ld: warning: The i386 architecture is deprecated for macOS = (remove from the Xcode build setting: ARCHS)
ld: warning: ignoring file /Applications/Xcode.app/Co= ntents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.0.sdk/us= r/lib/libSystem.tbd, missing required architecture i386 in file /Applicatio= ns/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/Ma= cOSX12.0.sdk/usr/lib/libSystem.tbd (3 slices)
Undefined symbols for architecture i386:
  "_printf", referenced from:=
      _= main in hello-380c0a.o
ld: symbol(s) not found for architecture i386
clang: error: linker command failed with exit code 1 (use -v to see i= nvocation)


2) Are these= system libs architecturally defined to be free standing? Can they make ass= umptions about the runtime? If so seems like they could do random bad thing= s to make them not work in EFI like a system call or make other assumptions= like writing to a magic address to generate a trap. Just inspecting them t= ells us the implementation, not the architecture of the lib.

3) Are the paths to these libs = architectural or are they arbitrary implementation that is normal abstracte= d by how the compiler is released? Could some packaging change break the ma= gic paths to libs?

4) I asked the Xcode/clang team a long time ago what to do for free stand= ing match and they pointed me at some open source implementation of these m= ath libs that had been implemented in C. They did not want us using their l= ibs that shipped with macOS. 


For the GCC/clang to= ols seems like we are better off just providing the code. 

We have magic for compiler s= pecific inline assembly 
https://github.com/tianocore/edk2/blob/master/MdePkg/Library/Base= Lib/Ia32/GccInline.c

We have magic to abstract some compiler intrinsics today:
https://github.com/tiano= core/edk2/blob/master/MdePkg/Library/BaseIoLibIntrinsic/IoLibMsc.c
https://github.com/= tianocore/edk2/blob/master/MdePkg/Library/BaseIoLibIntrinsic/IoLibGcc.c=

While we try to = avoid it when at all possible the build system does support doing things di= fferently for different compilers if we have to
https://github.com/tianoco= re/edk2/blob/master/MdePkg/Library/BaseIoLibIntrinsic/BaseIoLibIntrinsic.in= f#L38
[Sources.IA32]
IoLibGcc.c | GCC
IoLibMsc.c | MSFT
IoLib.c

Thanks,

Andrew Fish

PS The compiler still works like you think we just don= =E2=80=99t have the libs you might need. 

<= span style=3D"font-variant-ligatures: no-common-ligatures; color: #b42419" = class=3D"">~/work/Compiler/Math>= cat i386.c             
typedef unsigned long long UINT64;
UINT64
main2(int argc, char **argv)
{
  UINT64 ulldiv =3D ((UINT64) argc)/3;

  // pre= vent the optimizer to calculate result at build time
  return  ulldiv;
}
~/work/Compiler/Math>clang -arch i386 i386.c -S
~/work/Compiler/Math>cat i386.S           &= nbsp;    
.section __TEXT,__text,r= egular,pure_instructions
.build_v= ersion macos, 12, 0 sdk_version 12, 0
.glo= bl _main2&n= bsp;                     =     ## -- Begin function main2
= .p2align <= /span>4, 0x90
_main2: &= nbsp;                    =           ## @main2
.cfi_startproc
## %bb.0:
pushl %ebp
.cfi_def_cfa_offset 8
.cfi_offset %ebp, -8
movl %es= p, %ebp
.cfi_def_cfa_register %eb= p
subl $24, %esp
movl 12(%ebp), %eax
movl 8(%ebp), %eax<= /span>
movl 8(%ebp), %ecx
movl %ecx, %edx
<= span class=3D"Apple-tab-span" style=3D"white-space:pre"> sarl $31, %edx<= /div>
movl %esp, %eax
= movl %edx, 4(%eax)
movl %ecx, (%eax)
movl $0, 12(%eax)
movl $= 3, 8(%eax)
calll ___udivdi3
movl %edx, -4(%ebp)
movl %= eax, -8(%ebp)
movl -8(%ebp), %eax
movl -4(%ebp), %edx
= addl $24, %esp
popl %ebp
retl
.cfi_end= proc
    &nbs= p;                     &n= bsp;             ## -- End function
.subsections_via_symbols<= /div>

PPS= Fun story about ABI differences as the macOS i386 ABI requires 16-byte ali= gned stack accesses and that is more strict than EFI. Luckily it does not b= reak EFI, but it means you can NOT call macOS code from EFI code. To make t= he emulator work on macOS I had to write assembly gaskets to align the stac= ks to make calls between the worlds possible. Not my finest hour but it wor= ks=E2=80=A6.
<= br class=3D"">
This crazy is an example about how assumptions tha= t are not EFI centric can leak into the generic libs produced for the compi= ler.

On Jan 27, 2022, at 2:26 PM, Kilian Kegel <KILIAN_KEGEL@OUTLOOK.COM&g= t; wrote:

Hi Gerd,
 
>* On my system the gcc in= trinsics are only available as shared library,
> &nbs= p; so the "just unpack the lib and use the object files" idea is not
>   going to work.
<D175366159184B98= AC9B192EC485505B.png>
<= o:p class=3D""> 
This little C program makes= an unsigned 64Bit division on PC compilers.
Running a 32Bit code generator, it usually invokes an intrinsic function.
  • MSFT: __aulldiv()
  • GCC: __udivdi3()
 
On my 32Bit Ub= untu standard installation I ran
  1. cc - Xlinker -Map=3Dstatic.map hello.c -static
  2. cc  -Xlinker -Map=3Dshared.map hello.c
 
The fi= rst .OBJ file mentioned in the .MAP file is in both cases:<= /o:p>
/usr/lib/gcc/i686-linux-gnu/6/libgc= c.a(_udivdi3.o)
<DC03DFFFFFF14308B3A615804A1BF474.png= >
 
<377AC53F424C47F794809BA1A5953904.p= ng>
<= o:p class=3D""> 
Then for each a.out I did:<= o:p class=3D"">
  • objdump -d a.out = > static.dis
  • objdump -d a.out > sha= red.dis
 
In both cases the intrinsic function is fully linked into the .ELF ex= ecutable.
<B5A273DA2C3A4898BAEB0A354C667FE5.png>
=  
>so the "just unpack th= e lib and use the object files" idea is not
>going to work= .
It seems to me that GNU holds t= he intrinsic functions in a separate library
that can be used without any change, and is always correct by d= efinition.
 =
For Microsoft that is only true when a SDK is in= stalled (INT64.LIB).
Without SDK = the intrinsic functions were included in LIBCMT.LIB and
must be isolated manually.
 
Gerd, can= you please doublecheck in your GCC build, if that works:
  1. add a 64Bit div to an x86= PEI module like:
<500D8F2283CD4FAE9B0E45647823894A.png>
 
<= ol start=3D"2" type=3D"1" style=3D"margin-bottom: 0in; margin-top: 0in;" cl= ass=3D"">
  • add libgcc.a as a search library, adjust the conf\tools_def.txt like:
    • DEBUG_GCCxx_IA32_DLINK_FLAGS   =3D =E2=80= =A6predefined parameter =E2=80=A6&nbs= p;/usr/lib/gcc/= i686-linux-gnu/6/libgcc.a
    to match your build sy= stem
    1. build the BIOS 
    2. if the build gets ready,= check the .MAP file whether it contains  __udivdi3() or not
     
    >= ;* I have my doubts that compiler's builtin libraries are optimized for
    >   size, so I'd suspect we would see a noticeable = size grow from that.
    Please check= the size of __udivdi3() and whether the tianocore reimplementation is smaller or not
     
    If this works for all build platforms, independently of using = the tianocore reimplementation or
    using the original compiler intrinsics, this is correct location to place = the address of the intrinsic library.
    For all optimization modes, operation modes (64Bit/32Bit) the intrins= ic library is available for the compiler.
     
    GNU lists the int= rinsics:
    https://gcc.gnu.org/onlinedocs/gccint/Libgc= c.html#Libgcc
     
    The intrinsic= library belongs to the compiler not to the build system.
    If the build s= ystem changes from EDK2 to VS2022/MSBUILD, the compiler
    expects the same= intrinsic library.
     
    Leave the intrinsic restrictions behind and just provide all required i= ntrinsics the com= piler needs
    to = fulfil the C-Standard!
     
    Thanks a lot,
    Kilian
    https://github.com/tianocore/edk2-staging/tree/CdePkg#c= depkgblog
     
    From: kraxel@redhat.com
    Sent: Wednesday= , January 26, 2022 12:02 PM
    To: Pedro = Falcato
    Cc: edk2-devel-groups-io;=  Kinney, Michael D; = Kilian Kegel; Yao, Jiewen;<= span class=3D"Apple-converted-space"> Sean Brogan; <= a href=3D"mailto:Bret.Barkelew@microsoft.com" style=3D"color: blue; text-de= coration: underline;" class=3D"">Bret Barkelew; Wang, Jian J;<= span class=3D"Apple-converted-space"> Jiang, Guomin; = Pawel Polawski; Lu, XiaoyuX
    Subject: Re: [edk2-devel] [PATCH 00/24] CryptoPkg/openssl: update openssl submo= dule to v3.0
     
      Hi,

    > I= think adding intrinsic libraries is a mixed bag, for the following
    > reasons:
    > 
    > 1) The intrinsic libraries are completel= y internal to the compilers.
    > Breaking down that toolchai= n/code barrier is not a good idea if we want the
    > project= to compile using a good variety of compilers. The API is unstable
    > (as it's internal to the compiler + version) and sometimes undoc= umented;
    > while in reality the ABI doesn't really change = (at least in the LLVM/GCC
    > world, not sure about the othe= r compilers), it's something to consider.

    Yes.=   But apparently there is no way around them.  We have them for a= rm.
    We have them for openssl.  IntelUndiPkg in edk2-stag= ing has some too
    (see IntelUndiPkg/LibC).  And I wouldn'= t be surprised if there are more
    cases ...

    Having a policy to outlaw Intrinsics, but then hand out exceptio= ns left
    and right doesn't look like a good idea to me.  = I think we should revisit
    that and accept that there simply i= s no way around Intrinsics in some
    cases.

    I think it makes sense to consolidate all the Intrinsics we have,= i.e.
    move them over to MdePkg, make everybody use that, so w= e have only a
    single version to maintain.

    I think it also makes sense to restrict Intrinsics to the cases w= here we
    have no other option, to keep them as small as possib= le and also make it
    as easy as possible to maintain them.

    > 2) Linking the compiler's builtin libraries = would fix our issues, except
    > that it doesn't work in cas= es where object files are tagged with ABI (such
    > as hard = FP vs soft FP).

    Also:

     * On my system the gcc intrinsics are only available as shared = library,
       so the "just unpack the lib and use the= object files" idea is not
       going to work.

     * I have my doubts that compiler's builtin lib= raries are optimized for
       size, so I'd suspect we= would see a noticeable size grow from that.

    &= nbsp;* I'd very much prefer to continue with the current approach to have   source code for the Intrinsics we need.  In = case we run into trouble
       things tend to be much = easier to fix when you have the source code at
       h= and.  That's actually part of the open source success story.

    take care,
      Gerd

     
    Sen= t from Mail for Windows
     <= /div>
    From: Kilian = Kegel
    Sent: Tuesday, January 25, 2022 09:06 PM
    To: Kinney, Michael D; devel@edk2.groups.io; kraxel@redhat.com; = Yao, Jiewen; Sean Brogan; Bret Barkelew
    Cc: devel@edk2.groups= .io; Wang, Jian J; Jiang, Guomin; Pawel Polawski; Lu, XiaoyuX
    Subject: Re: [edk2-devel] [PATCH 00/24] CryptoPkg/o= penssl: update openssl submodule to v3.0
     
    Hi Mike,
     
    thank you= for your explanation. I understand all the technical aspects.
    But let me go into the details of my approach= , that skips step 2) to 5)
    and ad= ds step 1.5)
    &nbs= p;
    >I think in practice, the intrinsic APIs we= are seeing from use
    >of C cod= e from submodules is a very limited set that do not
    >change across compiler releases,
    I totally agree. E.g INT64.LIB is the same since 200= 4 (WinXP DDK).
    &n= bsp;
    But my perspective is different anyway:
    1. an intrinsic= library belongs to a particular compiler/linker couple       
      2. an intrinsic library does no= t belong to a UEFI tianocore or commercial BIOS feature set and should be managed
    outside fr= om any EDK2 specific build description (.INF, .DSC)
     
    Let= =E2=80=99s assume there is a C  build environment fully installed, e.g= . Microsoft VS2022, on an EDK2 build machine
    and all legal aspects were fulfilled.
     
    In that c= ase the advanced developer is able to locate the library, that holds the in= trinsic functions
    (intrinsic .OBJ= modules) we needed to extract. (simply by checking the .MAP of a 32bit exe= cutable
    that pulls in the particu= lar intrinsics) 
    This is step 1)
     
    >One o= f the challenges is that compilers are allowed to add/remove/modify intrins= ic APIs
    >across compiler relea= ses.  We would need to define a solution that will work if there are
    >these types of changes, which = would potentially mean a different instance of the intrinsic
    >library for each tool chain tag.  
    <= div style=3D"margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif= ;" class=3D""> 
    Here comes s= tep 1.5):
    In case of Microsoft bu= ild it is LIBCMT.LIB that can be found when walking through the
    LIB environment string.
     
    It= is easy to extend EDKSETUP.BAT = to generate MSFTINTRINx86-32.LIB each time:
    1. locate LIBCMT.LIB
    2. extract the identified .OBJ modules from step 1: =E2= =80=9CLIB.EXE /extract:full_path_name.obj /out:name.obj LIBCM= T.LIB=E2=80=9D
    3. = bind extracted .OBJ to the MSFTINTRINx86-32.LIB:=  =E2=80=9CLIB.EXE *.obj /out:%C= ONF_PATH%\MSFTINTRINx86-32.lib<= i class=3D"">=E2=80=9D
     
    Now MSFTINTRINx86-32.LIB is located in  the conf directory.
     
    Adjust the DLINK_FLAGS in tools_def.txt to hold MSFTINTRINx86-32.LIB as a search library:
     
      DEBUG_VS2015= _IA32_DLINK_FLAGS   =3D /NOLOGO /NODEFAULTLIB /IGNORE:4001 /OPT:R= EF /OPT:ICF=3D10 /MAP /ALIGN:32 /SECTION:.xdata,D /SECTION:.pdata,D /MACHIN= E:X86 /LTCG /DLL /ENTRY:$(IMAGE_ENTRY_POINT) /SUBSYSTEM:EFI_BOOT_SERVICE_DR= IVER /SAFESEH:NO /BASE:0 /DRIVER /DEBUG %CONF_P= ATH%\MSFTINTRINx86-32.LIB
     
    RELEASE_VS2015_IA32_DLINK_= FLAGS   =3D /NOLOGO /NODEFAULTLIB /IGNORE:4001 /IGNORE:4254 /OPT:= REF /OPT:ICF=3D10 /MAP /ALIGN:32 /SECTION:.xdata,D /SECTION:.pdata,D /MACHI= NE:X86 /LTCG /DLL /ENTRY:$(IMAGE_ENTRY_POINT) /SUBSYSTEM:EFI_BOOT_SERVICE_D= RIVER /SAFESEH:NO /BASE:0 /DRIVER /MERGE:.rdata=3D.data %CONF_PATH%\MSFTINTRINx86-32.LIB<= /span>
     
    From now on the intrinsics are available for a= ll 32Bit components.
    &nbs= p;
    With that procedure it is guaranteed= by design, that the intrinsics are always available and match a= particular compiler/linker release.
    it saves storage space since source code or bin= ary modules don=E2=80=99t need to be kept
  • because they are not kept, they don=E2=80=99t need m= aintainance
  • no one= needs to understand and document (in math details) the internals and the i= nterface
  • no one nee= ds to test for the math functions, as it is necessary for tianocore re-impl= ementations
  • the co= mpiler vendor itself is in charge in a court case
     
    The s= cript below is a demonstration of the above arguments. It additionally adds= memcpy() and memcmp() to MSFTINTRINx86-32.LIB,
    =
    that the compiler sometimes needs, depending on optimization= style, array size, instruction set, whatsoever =E2=80=A6
    &l= t;A1B2261BD5014BBF968AF0AA25EF9349.png>
     
    I have ch= ecked some more .OBJ from LIBCMT.LIB and some of them (ftol3.obj to get __l= tod3() long= to double needed for= difftime()) 
    seem not to be space optimized for BIOS us= age, because the ftol3.obj holds multiple functions
    (and so violates the ODR one definition rule).
     
    But also such cases could be handled automatically by a script (I wr= ote a C program < 200 lines)
    t= hat disassembles (using Microsoft DUMPBIN.EXE) the .OBJ, splits by simple t= ext processing into
    single-functi= on-.ASM-files that could be assembled back to multiple .OBJ of smaller code= size.
     
    For this approach compil= er, library manager, disassembler (DUMPBIN, OBJDUMP) were needed,
    that a= re available on all build machines by definition.=
     
    B= est regards
    Kilian
     
     
    From: Kinney, Michael D
    Sent: Monday, January 24, 2022 06:28 = PM
    To:&nb= sp;Kilian Kegel; devel@ed= k2.groups.io; kraxel@redhat.com; Yao, Jiewen; Sean Brog= an; Bret Barkelew;&n= bsp;Kinney, Michael D
    Cc: <= /b>devel@edk2.groups.io; Wang, Jian J= ; Jiang, Guomin; = Pawel Polawski; Lu, XiaoyuX
    Subject: RE: [edk2-devel] [PATCH 00/24] CryptoPkg/openssl: update openssl submo= dule to v3.0
    &nbs= p;
    Hi Kilian,
     
    I am in favor of an= intrinsic lib to improve the EDK II development environment.
     
    This has already been done for ARM compilers.  The solution shou= ld mirror that approach.
     
    It would be best if we had source = code (either in the edk2 repo or through a submodule) for
    the required intrinsic APIs.  If source code = is not possible and we have to use a binary, then
    that must be accessed through a submodule.  The edk2 = repo does not host binaries.  We use
    repos such as edk2-non-osi for binaries.
     
    We also h= ave to provide a solution that works with supported compilers (VS, GCC, CLA= NG, XCODE).
     = ;
    One of the challenges is that compilers are all= owed to add/remove/modify intrinsic APIs
    across compiler releases.  We would need to define a solution = that will work if there are
    these= types of changes, which would potentially mean a different instance of the= intrinsic
    library for each tool = chain tag.  I think in practice, the intrinsic APIs we are seeing from= use
    of C code from submodules is= a very limited set that do not change across compiler releases,
    so the maintenance of these intrinsic libs= would be manageable.
     
    If we go down the source code path, = we can break it up into the following tasks:
  • Identify the specific subset of intrin= sic APIs from each compiler that is required for the edk2 use cases. <= o:p class=3D"">
  • Obtain the function= prototype and full documentation for each intrinsic API to support impleme= ntation and unit tests.
  • Implement the APIs for all compilers.
  • Implement unit tests for all APIs for all compilers= using UnitTestFrameworkPkg unit tests.
  • Update MdeLibs.dsc.inc with the NULL instances for the= intrinsic libs
  • Rem= ove intrinsic APIs from EDK II modules that currently maintain their own im= plementations of intrinsic APIs.
    •  
    Best regards,
     
    Mike
    &n= bsp;
    From: Kilian Kegel <kilian_kegel@outlook.com>&n= bsp;
    Sent: Monday, January 24, 2022 8:25 AM
    To: devel@edk2.groups.io; Kinney, Michael D <michael.d.kinney@intel.com>; kraxel@redh= at.com; Yao, Jiewen <jiewen.yao@intel.c= om>; Sean Brogan <sean.brogan@mic= rosoft.com>; Bret Barkelew <Bre= t.Barkelew@microsoft.com>
    Cc: deve= l@edk2.groups.io; Wang, Jian J <jian.j.w= ang@intel.com>; Jiang, Guomin <guomi= n.jiang@intel.com>; Pawel Polawski <ppo= lawsk@redhat.com>; Lu, XiaoyuX <xiaoyu= x.lu@intel.com>
    Subject: RE: [edk2-devel] [PATCH 00/24] Cryp= toPkg/openssl: update openssl submodule to v3.0
    =
     
    The 64-bit = integer math intrinsics and other intrinsic
    problems could be solved easily for ever:<= /div>
     
    1. Putting all .OBJ files together from LIBCMT.H or INT64= .LIB (for ll*.obj and ull*.obj only)
    ltod3.obj
    ftol2.obj
    lldiv.obj
    lldvrm.obj
    llmul.obj
    llrem.obj
    llshl= .obj
    llshr.obj
    ulldiv.obj
    ulldvrm.obj
    ullrem.obj
    ullshr.obj
    memcmp.obj
    mem= cpycpy.obj
           &n= bsp;        and adjust for usability in = EDK2 (remove / solve further internal dependencies or rewrite =E2=80=9C*cpy= =E2=80=9D and =E2=80=9C*cmp=E2=80=9D functions)
    =
    This is already done in IntrinsicLib.lib= for some of the above functions, just complete this task!<= /o:p>
    1. Put all the .OBJ into a = e.g. edk2\C= onf \=E2=80=9CMSFTINTRINx86-32<compilerversion>.lib=E2=80=9D
    2. Update th= e MSFT_DEF.txt tool chain definition path
      3. DEBUG_*_IA32_DLINK_FLAGS   &n= bsp; =3D %CONF_PATH%\&n= bsp;MSFTINTRINx86-32<compilerversion>.lib
      RELEASE_*_IA32_DLINK_FLAGS   =3D %CONF_PATH%\<= b class=3D""> MSFTINTRINx= 86-32<compilerversion>.lib
      1. Resolve build conflicts with other existing intrinsic libraries fro= m CryptoPkg, RedfishPkg=E2=80=A6 =E2=80=93 remove these libraries
       
      From now on all existing 32Bit compo= nents have access to their own compiler intrinsics without
    touching any = .INF file and the problem is instantly gone.<= /div>
     
    Please= do the same for 
    • GCCINTRINx= 86-32<compilerversion>.lib
     
    Leav= e the intrinsic restrictions behind and just provide all required intrin= sics the compiler= needs
    to fulfi= l the C-Standard!
     
    UEFI shall conform the execution envi= ronment described in the C Specification
    and shall not try to create a new restric= ted =E2=80=9CUEFI execution environment=E2=80=9D
    that currently prohibits some =E2=80=9Cexpressions=E2=80=9D= (shift << >> , divide / % ) on some =E2=80=9Cdata types= =E2=80=9D (64bit =E2=80=9Clong long=E2=80=9D)
    but maybe in the future will prohibit some more =E2=80=9Cexpre= ssions=E2=80=9D (logical AND &&, relational-expression < >) o= n
    still speculative =E2=80=9Cdata= types=E2=80=9D (e.g. a 128bit =E2=80=9Cextended long=E2=80=9D) or just bec= ause a new compiler
    (version) wit= h some new optimization(ultra slow)/security(specdown/meltre) capabilities = introduces
    some new intrinsic fun= ctions.
    Who knows=E2=80=A6
     
    In contrast to:
    =E2=80=9CI think we shouldn't add any intrinsics unless we are absol= utely forced
    to. I do agree however that,= for those intrinsics that we cannot at all
    avoid reimplementing, we should at least collect them in a common
    <= div style=3D"margin: 0in 0in 0in 0.5in; font-size: 11pt; font-family: Calib= ri, sans-serif;" class=3D"">library.
    (In = theory, I can also imagine reimplementing all possible intrinsics
    *if* the edk2 coding style spec / requirements are= updated in parallel,
    permitting all new = code to universally rely on the intrinsics, rather
    than the BaseLib / BaseMemoryLib functions.)=E2=80=9D
    h= ttps://bugzilla.tianocore.org/show_bug.cgi?id=3D1516#c2=
     
    This mindset violates edk2 coding style spec too: 
    https:/= /edk2-docs.gitbook.io/edk-ii-c-coding-standards-specification/2_guiding_pri= nciples
    • Maint= ainability
    • Extens= ibility
    • Intellectua= l manageability
    • Por= tability
    • Reusabilit= y
    • Standard techniqu= es
     
    Have fun,
    = Kilian
     
    From: Michael D Kinney
    Sent:&= nbsp;Friday, January 21, 2022 05:39 PM
    To:     kraxel@redhat.com; = ;Yao, Jiewen; Sean Brogan;=  Bret Barkelew; Kinney, Michael D
    Cc:     devel@edk2.groups.io; Wang, Jian J; Ji= ang, Guomin; Pawel Polawski; Lu, XiaoyuX
    Subject: Re= : [edk2-devel] [PATCH 00/24] CryptoPkg/openssl: update openssl submodule to= v3.0
     

    Comments below.

    Mike

    > -----Original Message-----> From: <= a href=3D"mailto:kraxel@redhat.com" style=3D"color: blue; text-decoration: = underline;" class=3D"">kraxel@redhat.com <kraxel@redhat.com>> Sent: Friday, January 21, 2022 12:31 AM
    >= ; To: Yao, Jiewen <jiewen.yao@intel.com&g= t;
    > Cc: devel@edk2.groups.io; Kinney, Michael D <= ;michael.d.kinney@intel.com>; Wang,= Jian J <jian.j.wang@intel.com>; Jian= g, Guomin
    > <guomin.jiang@= intel.com>; Pawel Polawski <ppolawsk@re= dhat.com>; Lu, XiaoyuX <xiaoyux.lu@int= el.com>
    > Subject: Re: [edk2-devel] [PATCH 00/24] C= ryptoPkg/openssl: update openssl submodule to v3.0
    > 
    > > > = No changes in SEC and PEI.
    > > [Jiewen] Do you mean the= Crypto consumer in PEI has no size difference? Such as
    > = > https://github.com/tianoco= re/edk2/tree/master/SecurityPkg/Tcg/Tcg2Pei ,
    > > https://github.com/tianocore/edk2/tree/master/SecurityPkg= /FvReportPei ,
    > > https://github.com/tianocore/edk2/tree/master/SignedCapsulePkg/= Universal/RecoveryModuleLoadPei&n= bsp;linking
    >=  https://github.com/tianocore/edk2/tree= /master/SecurityPkg/Library/FmpAuthenticationLibRsa2048Sha256.
    > 
    > PEI has this (OvmfIa32X64Pkg build):
    > 
    >   &= nbsp; 7062 TpmMmioSevDecryptPei
    >     = 7830 StatusCodeHandlerPei
    >     7902 R= eportStatusCodeRouterPei
    >     8470 Fa= ultTolerantWritePei
    >     9734 SmmAcce= ssPei
    >    11206 Tcg2ConfigPei
    >    11842 PeiVariable
    >  &n= bsp; 14730 Tcg2PlatformPei
    >    17274 TcgPe= i
    >    18438 S3Resume2Pei
    >= ;    18682 DxeIpl
    >    18938= PcdPeim
    >    38014 CpuMpPei
    = >    39554 PlatformPei
    >  &nbs= p; 45050 PeiCore
    >    49274 Tcg2Pei
    > 
    > No size change for Tcg2Pei.
    > 
    > The other modules are not = there.  Seems they are related to firmware
    > updates.=   We don't have that on ovmf as we can simply update the
    > firmware image files on the host machine ...
    > 
    > Is there s= ome target I could use to test-build those modules?
    > 
    > > >= INFO - OpensslLibCrypto.lib(rsa_lib.obj) : error LNK2001: unresolved exter= nal
    > > > symbol __allmul
    > > &g= t; INFO - OpensslLibCrypto.lib(rsa_lib.obj) : error LNK2001: unresolved ext= ernal
    > > > symbol __aulldiv
    > >= > INFO - OpensslLibCrypto.lib(bio_print.obj) : error LNK2001: unresolve= d external
    > > > symbol __aulldvrm
    >= ; > > INFO - OpensslLibCrypto.lib(bio_print.obj) : error LNK2001: unr= esolved external
    > > > symbol __ftol2_sse
    > > >
    > > > Those symbols look like t= hey reference helper functions to do 64bit math
    > > >= ; on 32bit architecture.  Any hints how to fix that?
    >= ; > [Jiewen] Please add them to&nb= sp;https://github.com/tianocore/edk2/tree/master/CryptoPkg/Library/= IntrinsicLib
    >&n= bsp;
    > Any hints where I could get them?  Give= n this happens on windows builds
    > it's probably somewhere= in the microsoft standard C library?  Is that
    > avai= lable as open source somewhere?

    Sean and Bret = may be able to help with these.

    There is also = a BZ on this topic.

    https://bugzilla.tianocore.org/show_bug.cgi?id= =3D1516

    > 
    > > > (3) Some NOOPT builds ar= e failing due to the size growing ...
    > > [Jiewen] Size= becomes big challenge...
    > > Have you tried to use https://github.com/tianocore/edk2/tree/ma= ster/CryptoPkg/Driver solution?
    > 
    > Seems the idea is to have only one openssl copy in= the dxe image by
    > calling a protocol instead of linking = a lib.  Makes sense.
    > 
    > Is this documented somewhere? = Is there some easy way to use that as
    > drop-in replaceme= nt?  Or do we have to change all crypto users to call
    &g= t; the driver instead of linking the lib?
    > 
    > take care,
    >   Gerd


     
     

     
    <shared.dis><shared.map><static.dis><static.map>

    --Apple-Mail=_2E5DCBE6-CC8D-417C-920C-08BABBE60071--