From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from ma-mailsvcp-mx-lapp01.apple.com (ma-mailsvcp-mx-lapp01.apple.com [17.32.222.22]) by mx.groups.io with SMTP id smtpd.web11.721.1677115191603164762 for ; Wed, 22 Feb 2023 17:19:51 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@apple.com header.s=20180706 header.b=dOKD5O3g; spf=pass (domain: apple.com, ip: 17.32.222.22, mailfrom: afish@apple.com) Received: from rn-mailsvcp-mta-lapp01.rno.apple.com (rn-mailsvcp-mta-lapp01.rno.apple.com [10.225.203.149]) by ma-mailsvcp-mx-lapp01.apple.com (Oracle Communications Messaging Server 8.1.0.20.20220923 64bit (built Sep 23 2022)) with ESMTPS id <0RQI00K8NED0S600@ma-mailsvcp-mx-lapp01.apple.com> for devel@edk2.groups.io; Wed, 22 Feb 2023 17:19:50 -0800 (PST) X-Proofpoint-GUID: 6wcUTuzITDaU2cD94h4WiC09xASf0IZW X-Proofpoint-ORIG-GUID: 6wcUTuzITDaU2cD94h4WiC09xASf0IZW X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.562,18.0.930 definitions=2023-02-22_12:2023-02-22,2023-02-22 signatures=0 X-Proofpoint-Spam-Details: rule=interactive_user_notspam policy=interactive_user score=0 suspectscore=0 mlxlogscore=999 bulkscore=0 malwarescore=0 spamscore=0 mlxscore=0 adultscore=0 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2212070000 definitions=main-2302230009 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=apple.com; h=from : content-type : mime-version : subject : date : references : to : in-reply-to : message-id; s=20180706; bh=EyxM1QARn2KeF5WraC3VYhC009EKAV0wB3atQV79fpU=; b=dOKD5O3g2Pn+rbUQgba5IwfK9GwKd0xREhZOo3PZrdTkKHa9nofB3LAR11IVjuSAL3Cm npmF1CmSoNomSlSBqLscq/yzRqZnpNA+D65j4z1OiwZm3t0Mbyb0Gx/pEn9u1iJ0Th2i btRPVEJyGRZFGsqtztaWvqxC2+w6ArLVfo+DkQukmXfuvkOgcVgB5DI/dEA28w6TjGhI xmme0z7IFfDBXjG4ehSrbL+JbpBYEFV4JSxFLUc5vfpqt30w7cNFRiIIrI6RsAIZ7hKZ Zu14/oyAOSsh3c+xVprXGoboES1MUqf101VpcOOzt9DS5f5CzOIWbHwP4K4cxeWMgZAN WQ== Received: from rn-mailsvcp-mmp-lapp04.rno.apple.com (rn-mailsvcp-mmp-lapp04.rno.apple.com [17.179.253.17]) by rn-mailsvcp-mta-lapp01.rno.apple.com (Oracle Communications Messaging Server 8.1.0.20.20220923 64bit (built Sep 23 2022)) with ESMTPS id <0RQI010QMED1H2I0@rn-mailsvcp-mta-lapp01.rno.apple.com>; Wed, 22 Feb 2023 17:19:49 -0800 (PST) Received: from process_milters-daemon.rn-mailsvcp-mmp-lapp04.rno.apple.com by rn-mailsvcp-mmp-lapp04.rno.apple.com (Oracle Communications Messaging Server 8.1.0.20.20220923 64bit (built Sep 23 2022)) id <0RQI00800EBUBO00@rn-mailsvcp-mmp-lapp04.rno.apple.com>; Wed, 22 Feb 2023 17:19:49 -0800 (PST) X-Va-A: X-Va-T-CD: f839a2fe8738a6cacb8020cb6080bc6b X-Va-E-CD: 71066acee33f65e9a96141518692d84b X-Va-R-CD: 361631305329de6bf4d7d51ee80c4690 X-Va-ID: eb14913e-d8d0-424c-a4d0-1f4943348530 X-Va-CD: 0 X-V-A: X-V-T-CD: f839a2fe8738a6cacb8020cb6080bc6b X-V-E-CD: 71066acee33f65e9a96141518692d84b X-V-R-CD: 361631305329de6bf4d7d51ee80c4690 X-V-ID: b3e1a40c-cf19-4095-bc35-e9f4044ff941 X-V-CD: 0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.562,18.0.930 definitions=2023-02-22_12:2023-02-22,2023-02-22 signatures=0 Received: from smtpclient.apple (unknown [17.11.209.77]) by rn-mailsvcp-mmp-lapp04.rno.apple.com (Oracle Communications Messaging Server 8.1.0.20.20220923 64bit (built Sep 23 2022)) with ESMTPSA id <0RQI00EDXED0VD00@rn-mailsvcp-mmp-lapp04.rno.apple.com>; Wed, 22 Feb 2023 17:19:48 -0800 (PST) From: "Andrew Fish" MIME-version: 1.0 (Mac OS X Mail 16.0 \(3731.500.161\)) Subject: Re: [edk2-devel] DSC nightmare Date: Wed, 22 Feb 2023 17:19:37 -0800 References: To: edk2-devel-groups-io , crawford.benjamin15@gmail.com In-reply-to: Message-id: <53FA79D9-3B40-4A10-A265-384FDFAFCF38@apple.com> X-Mailer: Apple Mail (2.3731.500.161) Content-type: multipart/alternative; boundary="Apple-Mail=_216ABBBB-FCF3-4B4E-8B9C-4FD0E3E53A11" --Apple-Mail=_216ABBBB-FCF3-4B4E-8B9C-4FD0E3E53A11 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Ben, I=E2=80=99d say the tools are optimized to be general purpose. So they are = not really designed to be the easy button. In terms of things like SEC/PEI or DXE it is very common for this code on x= 86 to be compiled for different architectures. SEC/PEI is usually i386 and = DXE is x86_64. Crazy I know but historically on x86 you can=E2=80=99t be in= 64-bit mode without paging being enabled, and you can=E2=80=99t (legally) = put page tables in ROM. So the early code at the reset vector (SEC) and the= code to turn on memory (PEI) is 32-bit i386. PEI got a lot bigger than we = envisioned mainly due to it taking a PhD in antenna design to turn on memor= y on an Intel chipset. Then all the power on engineers got good at writing = PEI code and moved more code to PEI, but I digress=E2=80=A6. OK let us leave behind binary compatibility and talk about the different co= ntracts that exist in the different phases. For example how to allocate mem= ory. There is no standard way in SEC (People use PHIT HOB), there is a PEI = specific API[1], and a DXE/UEFI specific API[2]. So you can end up with dif= ferent instances of the same library API coded with the same API, but imple= mented differently. This was a conscious choice on our part to make it much= easier to port code between the different worlds. We also support the conc= ept of multiple instances of a give library that use different hardware, or= are optimized for different things. For example we have some libs that pro= duce our version of memcpy etc that have different instances for PEI[3] and= DXE[4]. Seems when you are running from ROM vs. running from memory the op= timal optimization schemes are different. Thus we give the user a lot of po= wer over which instance of a library they can pick. When you need it, it is= very handy. You also don=E2=80=99t have other people randomly changing the= rules on you, which can be important for things like security reviews etc.= =20 So a little lingo sometimes helps. A library class is the contract, or the = header file that defines the public interface for the library. The library = instance is the implementation, as I mentioned above there can be more than= one for many reasons. Base means the code does not depend on any of the bo= ot phases. SEC/PEI/DXE etc are phases defined by the UEFI PI Spec [1]. I guess we could try to combine libraries of different phases together to s= implify things, but at this point we have lots of customers with (as you po= int out) complex DSC files that would all break if we refactored the libs. = So kind of makes fixing things a little harder.=20 In terms of tips I find it I useful to use a little git foo to find instanc= es of libraries. The library INF file defines which library class the libra= ry implements so you can filter by LIBRARY_CLASS to get the instances (impl= ementations) of a given library class.=20 /Volumes/Case/edk2(master)>git grep BaseMemoryLib -- \*.inf | grep LIBRARY_= CLASS MdePkg/Library/BaseMemoryLib/BaseMemoryLib.inf:20: LIBRARY_CLASS = =3D BaseMemoryLib MdePkg/Library/BaseMemoryLibMmx/BaseMemoryLibMmx.inf:21: LIBRARY_CLASS = =3D BaseMemoryLib MdePkg/Library/BaseMemoryLibOptDxe/BaseMemoryLibOptDxe.inf:21: LIBRARY_CLA= SS =3D BaseMemoryLib MdePkg/Library/BaseMemoryLibOptDxe/BaseMemoryLibOptDxe.inf:79: LIBRARY_CLA= SS =3D BaseMemoryLib|DXE_CORE DXE_DRIVER DXE_RUNTIME_DRIVER UEFI_DRIVER UEF= I_APPLICATION MdePkg/Library/BaseMemoryLibOptPei/BaseMemoryLibOptPei.inf:21: LIBRARY_CLA= SS =3D BaseMemoryLib MdePkg/Library/BaseMemoryLibRepStr/BaseMemoryLibRepStr.inf:21: LIBRARY_CLA= SS =3D BaseMemoryLib MdePkg/Library/BaseMemoryLibSse2/BaseMemoryLibSse2.inf:20: LIBRARY_CLASS = =3D BaseMemoryLib MdePkg/Library/PeiMemoryLib/PeiMemoryLib.inf:21: LIBRARY_CLASS = =3D BaseMemoryLib|PEIM MdePkg/Library/UefiMemoryLib/UefiMemoryLib.inf:21: LIBRARY_CLASS = =3D BaseMemoryLib|DXE_DRIVER DXE_RUNTIME_DRIVER DXE_SMM_DRIVER UEFI= _APPLICATION UEFI_DRIVER /Volumes/Case/edk2(master)>git grep SerialPortLib -- \*.inf | grep LIBRARY= _CLASS ArmPkg/Library/SemiHostingSerialPortLib/SemiHostingSerialPortLib.inf:17: L= IBRARY_CLASS =3D SerialPortLib ArmPlatformPkg/Library/PL011SerialPortLib/PL011SerialPortLib.inf:17: LIBRA= RY_CLASS =3D SerialPortLib ArmVirtPkg/Library/FdtPL011SerialPortLib/EarlyFdtPL011SerialPortLib.inf:17:= LIBRARY_CLASS =3D SerialPortLib|SEC PEI_CORE PEIM ArmVirtPkg/Library/FdtPL011SerialPortLib/FdtPL011SerialPortLib.inf:17: LIB= RARY_CLASS =3D SerialPortLib|DXE_CORE DXE_DRIVER UEFI_DRIV= ER DXE_RUNTIME_DRIVER UEFI_APPLICATION EmulatorPkg/Library/DxeEmuSerialPortLib/DxeEmuSerialPortLib.inf:18: LIBRAR= Y_CLASS =3D SerialPortLib| DXE_CORE DXE_DRIVER DXE_RUNTIME= _DRIVER DXE_SMM_DRIVER SMM_CORE UEFI_APPLICATION UEFI_DRIVE EmulatorPkg/Library/DxeEmuStdErrSerialPortLib/DxeEmuStdErrSerialPortLib.inf= :18: LIBRARY_CLASS =3D SerialPortLib| DXE_CORE DXE_DRIVER= DXE_RUNTIME_DRIVER DXE_SMM_DRIVER SMM_CORE UEFI_APPLICATION UEFI_DRIVE EmulatorPkg/Library/PeiEmuSerialPortLib/PeiEmuSerialPortLib.inf:18: LIBRAR= Y_CLASS =3D SerialPortLib| PEI_CORE PEIM SEC MdeModulePkg/Library/BaseSerialPortLib16550/BaseSerialPortLib16550.inf:18: = LIBRARY_CLASS =3D SerialPortLib MdePkg/Library/BaseSerialPortLibNull/BaseSerialPortLibNull.inf:18: LIBRARY= _CLASS =3D SerialPortLib OvmfPkg/Library/XenConsoleSerialPortLib/XenConsoleSerialPortLib.inf:17: LI= BRARY_CLASS =3D SerialPortLib PcAtChipsetPkg/Library/SerialIoLib/SerialIoLib.inf:16: LIBRARY_CLASS = =3D SerialPortLib UefiPayloadPkg/Library/BaseSerialPortLibHob/BaseSerialPortLibHob.inf:16: L= IBRARY_CLASS =3D SerialPortLib UefiPayloadPkg/Library/CbSerialPortLib/CbSerialPortLib.inf:15: LIBRARY_CLA= SS =3D SerialPortLib [1] https://github.com/tianocore/edk2/blob/master/MdePkg/Library/PeiMemoryA= llocationLib/MemoryAllocationLib.c#L373 [2] https://github.com/tianocore/edk2/blob/master/MdePkg/Library/UefiMemory= AllocationLib/MemoryAllocationLib.c#L376 [3] https://github.com/tianocore/edk2/tree/master/MdePkg/Library/BaseMemory= LibOptPei [4] https://github.com/tianocore/edk2/tree/master/MdePkg/Library/BaseMemory= LibOptDxe [5] https://uefi.org/specifications Thanks, Andrew Fish > On Feb 19, 2023, at 5:36 PM, Benjamin Mordaunt wrote: >=20 > Hi, >=20 > I am working on implementing a new Arm platform in edk2-platforms, and ha= ve reached the stage of tackling writing a DSC to describe it, along with a= "dsc.inc" which contains defines etc. common to platforms sharing the same= SoC. I'm using, as reference the platforms currently present in the reposi= tory, such as RPi, Beagleboard and HiKey. >=20 > However, it's a horrible process and I can't wrap my head around why I ne= ed to list hundreds of libraries (in the LibraryClasses.X) sections, seemin= gly by hand. By the looks of it, most of them pull from ArmPkg, ArmPlatform= Pkg, MdePkg etc., being either the default implementations of these librari= es, or "Null" versions which stub out the functionality. Only a few are pla= tform specific, such as some drivers I've implemented (display, serial etc.= ). >=20 > Not only that, I then need to list them all over again for each EFI boot = stage, SEC, PEI_CORE, PEIM and so on... why?! Why can't I just say "Hey, th= is is an aarch64 platform with the following drivers/libraries/quirks for m= y platform. It's a huge pain, not to mention a maintenance nightmare... >=20 > Am I looking at this wrong? This seems like an obvious problem and greatl= y increases the labour and maintenance effort required to implement a new p= latform. >=20 > --- >=20 > Ben >=20 --Apple-Mail=_216ABBBB-FCF3-4B4E-8B9C-4FD0E3E53A11 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 Ben,

I=E2=80= =99d say the tools are optimized to be general purpose. So they are not rea= lly designed to be the easy button.

In terms of th= ings like SEC/PEI or DXE it is very common for this code on x86 to be compi= led for different architectures. SEC/PEI is usually i386 and DXE is x86_64.= Crazy I know but historically on x86 you can=E2=80=99t be in 64-bit mode w= ithout paging being enabled, and you can=E2=80=99t (legally) put page table= s in ROM. So the early code at the reset vector (SEC) and the code to turn = on memory (PEI) is 32-bit i386. PEI got a lot bigger than we envisioned mai= nly due to it taking a PhD in antenna design to turn on memory on an Intel = chipset. Then all the power on engineers got good at writing PEI code and m= oved more code to PEI, but I digress=E2=80=A6.

OK = let us leave behind binary compatibility and talk about the different contr= acts that exist in the different phases. For example how to allocate memory= . There is no standard way in SEC (People use PHIT HOB), there is a PEI spe= cific API[1], and a DXE/UEFI specific API[2]. So you can end up with differ= ent instances of the same library API coded with the same API, but implemen= ted differently. This was a conscious choice on our part to make it mu= ch easier to port code between the different worlds. We also support the co= ncept of multiple instances of a give library that use different hardware, = or are optimized for different things. For example we have some libs that p= roduce our version of memcpy etc that have different instances for PEI[3] a= nd DXE[4]. Seems when you are running from ROM vs. running from memory the = optimal optimization schemes are different. Thus we give the user a lot of = power over which instance of a library they can pick. When you need it, it = is very handy. You also don=E2=80=99t have other people randomly changing t= he rules on you, which can be important for things like security reviews et= c. 

So a little lingo sometimes helps. A libr= ary class is the contract, or the header file that defines the public inter= face for the library. The library instance is the implementation, as I ment= ioned above there can be more than one for many reasons. Base means the cod= e does not depend on any of the boot phases. SEC/PEI/DXE etc are phases def= ined by the UEFI PI Spec [1].

I guess we could try= to combine libraries of different phases together to simplify things, but = at this point we have lots of customers with (as you point out) complex DSC= files that would all break if we refactored the libs. So kind of makes fix= ing things a little harder. 

In terms of tips= I find it I useful to use a little git foo to find instances of libraries.= The library INF file defines which library class the library implements so= you can filter by <= span style=3D"font-variant-ligatures: no-common-ligatures; color: #b42419">= /Volumes/Case/edk2(master)>git grep BaseMemoryLib -- \*.i= nf | grep LIBRARY_CLASS

MdePkg/Library/BaseMemoryLib/BaseMemoryLib.inf:20:  LIBRARY_= CLASS                  =3D Bas= eMemoryLib

MdePkg/Library/BaseMemoryLibMmx/BaseMemoryLibMmx.inf:21:  LI= BRARY_CLASS                  = =3D BaseMemoryLib

MdePkg/Library/BaseMemoryLibOptDxe/BaseMemoryLibOptDxe.inf:21:&nb= sp; LIBRARY_CLASS                &n= bsp; =3D BaseMemoryLib

MdePkg/Library/BaseMemoryLibOptDxe/BaseMemoryLibOptDxe.inf:79:&nb= sp; LIBRARY_CLASS =3D BaseMemoryLib|DXE_CORE DXE_DRIVER DXE_RUNTIME_DRIVER = UEFI_DRIVER UEFI_APPLICATION

MdePkg/Library/BaseMemoryLibOptPei/BaseMemoryLibOptPei.inf:21:&nb= sp; LIBRARY_CLASS                &n= bsp; =3D BaseMemoryLib

MdePkg/Library/BaseMemoryLibRepStr/BaseMemoryLibRepStr.inf:21:&nb= sp; LIBRARY_CLASS                &n= bsp; =3D BaseMemoryLib

MdePkg/Library/BaseMemoryLibSse2/BaseMemoryLibSse2.inf:20:  = LIBRARY_CLASS                 = =3D BaseMemoryLib

MdePkg/Library/PeiMemoryLib/PeiMemoryLib.inf:21:  LIBRARY_CL= ASS                  =3D BaseM= emoryLib|PEIM

MdePkg/Library/UefiMemoryLib/UefiMemoryLib.inf:21:  LIBRARY_= CLASS                  =3D Bas= eMemoryLib|DXE_DRIVER DXE_RUNTIME_DRIVER DXE_SMM_DRIVER UEFI_APPLICATION UE= FI_DRIVER


/Volumes/Case/edk2<= span style=3D"font-variant-ligatures: no-common-ligatures; color: #2fb41d">= (master)>git grep SerialPortLib  -- \*.inf | grep LIBRARY_CL= ASS

ArmPkg/Library/SemiHostingSerialPortLib/SemiHostingSerialPortLib.= inf:17:  LIBRARY_CLASS             =     =3D SerialPortLib

ArmPlatformPkg/Library/PL011SerialPortLib/PL011SerialPortLib.inf:= 17:  LIBRARY_CLASS              &nb= sp;   =3D SerialPortLib

ArmVirtPkg/Library/FdtPL011SerialPortLib/EarlyFdtPL011SerialPortL= ib.inf:17:  LIBRARY_CLASS            &nb= sp;     =3D SerialPortLib|SEC PEI_CORE PEIM

ArmVirtPkg/Library/FdtPL011SerialPortLib/FdtPL011SerialPortLib.in= f:17:  LIBRARY_CLASS              &= nbsp;   =3D SerialPortLib|DXE_CORE DXE_DRIVER UEFI_DRIVER DXE_RUNTIME_= DRIVER UEFI_APPLICATION

EmulatorPkg/Library/DxeEmuSerialPortLib/DxeEmuSerialPortLib.inf:1= 8:  LIBRARY_CLASS              &nbs= p;   =3D SerialPortLib| DXE_CORE DXE_DRIVER DXE_RUNTIME_DRIVER DXE_SMM= _DRIVER SMM_CORE UEFI_APPLICATION UEFI_DRIVE

EmulatorPkg/Library/DxeEmuStdErrSerialPortLib/DxeEmuStdErrSerialP= ortLib.inf:18:  LIBRARY_CLASS           =       =3D SerialPortLib| DXE_CORE DXE_DRIVER DXE_RUNTIME_DR= IVER DXE_SMM_DRIVER SMM_CORE UEFI_APPLICATION UEFI_DRIVE

EmulatorPkg/Library/PeiEmuSerialPortLib/PeiEmuSerialPortLib.inf:1= 8:  LIBRARY_CLASS              &nbs= p;   =3D SerialPortLib| PEI_CORE PEIM SEC

MdeModulePkg/Library/BaseSerialPortLib16550/BaseSerialPortLib1655= 0.inf:18:  LIBRARY_CLASS            &nbs= p;     =3D SerialPortLib

MdePkg/Library/BaseSerialPortLibNull/BaseSerialPortLibNull.inf:18= :  LIBRARY_CLASS               = ;   =3D SerialPortLib

OvmfPkg/Library/XenConsoleSerialPortLib/XenConsoleSerialPortLib.i= nf:17:  LIBRARY_CLASS              =     =3D SerialPortLib

PcAtChipsetPkg/Library/SerialIoLib/SerialIoLib.inf:16:  LIBR= ARY_CLASS                  =3D= SerialPortLib

UefiPayloadPkg/Library/BaseSerialPortLibHob/BaseSerialPortLibHob.= inf:16:  LIBRARY_CLASS             =     =3D SerialPortLib

UefiPayloadPkg/Library/CbSerialPortLib/CbSerialPortLib.inf:15:&nb= sp; LIBRARY_CLASS                &n= bsp; =3D SerialPortLib




<= div>[3] https://github.com/tianocore/edk2/tree/maste= r/MdePkg/Library/BaseMemoryLibOptPei


Thanks,

Andrew Fish

On Feb 19, 2023, at 5:36 PM, Benjamin Mordaunt <crawford.= benjamin15@gmail.com> wrote:

Hi,

I am working on implementing a new Arm platform in edk2-p= latforms, and have reached the stage of tackling writing a DSC to describe = it, along with a "dsc.inc" which contains defines etc. common to platforms = sharing the same SoC. I'm using, as reference the platforms currently prese= nt in the repository, such as RPi, Beagleboard and HiKey.

However, i= t's a horrible process and I can't wrap my head around why I need to list h= undreds of libraries (in the LibraryClasses.X) sections, seemingly by hand.= By the looks of it, most of them pull from ArmPkg, ArmPlatformPkg, MdePkg = etc., being either the default implementations of these libraries, or "Null= " versions which stub out the functionality. Only a few are platform specif= ic, such as some drivers I've implemented (display, serial etc.).

No= t only that, I then need to list them all over again for each EFI boot stag= e, SEC, PEI_CORE, PEIM and so on... why?! Why can't I just say "Hey, this i= s an aarch64 platform with the following drivers/libraries/quirks for my pl= atform. It's a huge pain, not to mention a maintenance nightmare...

= Am I looking at this wrong? This seems like an obvious problem and greatly = increases the labour and maintenance effort required to implement a new pla= tform.

---

Ben

--Apple-Mail=_216ABBBB-FCF3-4B4E-8B9C-4FD0E3E53A11--