Re: [edk2-devel] [PATCH v5 00/13] BaseTools,CryptoPkg,EmulatorPkg,MdePkg,others: Delete CLANG35,CLANG38,VS2008-2013,EBC, deprecate GCC48,GCC49,GCC5, add GCC and GCCNOLTO, update CLANGDWARF

["Michael D Kinney" <michael.d.kinney@intel.com>]

Hi Rebecca,

There is an issue in the updates to build.py.

If -t <ToolChainTag> is provided in the build command line, it works.

If -t <ToolsChainTag> is not provided and the tools chain tag is parsed from Conf/target.txt, then python gets a stack trace.

Traceback (most recent call last):
  File "/home/mdkinney/GitHub/tianocore/edk2/BaseTools/BinWrappers/PosixLike/../../Source/Python/build/build.py", line 2798, in <module>
    r = Main()
  File "/home/mdkinney/GitHub/tianocore/edk2/BaseTools/BinWrappers/PosixLike/../../Source/Python/build/build.py", line 2779, in Main
    if any(tc in Option.ToolChain for tc in gDeprecatedToolChains):
  File "/home/mdkinney/GitHub/tianocore/edk2/BaseTools/BinWrappers/PosixLike/../../Source/Python/build/build.py", line 2779, in <genexpr>
    if any(tc in Option.ToolChain for tc in gDeprecatedToolChains):
TypeError: argument of type 'NoneType' is not iterable

I think in this case, Option.ToolChain in None and this this check for use of deprecated choo chain tags, you need to look at both the command line options and tool chain tags used from target.txt.

Thanks,

Mike


> -----Original Message-----
> From: Kinney, Michael D <michael.d.kinney@intel.com>
> Sent: Wednesday, April 26, 2023 8:10 AM
> To: Rebecca Cran <rebecca@bsdio.com>; devel@edk2.groups.io; Gao, Liming
> <gaoliming@byosoft.com.cn>; Feng, Bob C <bob.c.feng@intel.com>; Chen,
> Christine <yuwei.chen@intel.com>; Michael Kubacki
> <mikuback@linux.microsoft.com>; Sean Brogan
> <sean.brogan@microsoft.com>; Chiu, Chasel <chasel.chiu@intel.com>;
> Desimone, Nathaniel L <nathaniel.l.desimone@intel.com>; Zeng, Star
> <star.zeng@intel.com>; Andrew Fish <afish@apple.com>
> Cc: Ni, Ray <ray.ni@intel.com>; Leif Lindholm <quic_llindhol@quicinc.com>;
> Liu, Zhiguang <zhiguang.liu@intel.com>; Wang, Jian J
> <jian.j.wang@intel.com>; Lu, Xiaoyu1 <xiaoyu1.lu@intel.com>; Jiang,
> Guomin <guomin.jiang@intel.com>; Guo, Gua <gua.guo@intel.com>; Ard
> Biesheuvel <ardb@kernel.org>; Pedro Falcato <pedro.falcato@gmail.com>;
> Gerd Hoffmann <kraxel@redhat.com>; Marvin Häuser
> <mhaeuser@posteo.de>; Kinney, Michael D <michael.d.kinney@intel.com>
> Subject: RE: [edk2-devel] [PATCH v5 00/13]
> BaseTools,CryptoPkg,EmulatorPkg,MdePkg,others: Delete
> CLANG35,CLANG38,VS2008-2013,EBC, deprecate GCC48,GCC49,GCC5, add
> GCC and GCCNOLTO, update CLANGDWARF
> 
> Yes.
> 
> Actually 2 issues.  Working on ways to reproduce both in edk2 repo.
> 
> One is in build.py that is generating a python traceback when a tool chain tag
> is used that will be retired in the future.
> 
> The 2nd one is in CLANGDWARF changes.  Main break is that one of the
> PATHs used to map to echo and how it is mapped to a path objcopy-llvm.
> There are other differences and need to determine if they are on purpose or
> not.
> 
> Mike
> 
> > -----Original Message-----
> > From: Rebecca Cran <rebecca@bsdio.com>
> > Sent: Wednesday, April 26, 2023 6:28 AM
> > To: devel@edk2.groups.io; Kinney, Michael D
> <michael.d.kinney@intel.com>; Gao, Liming <gaoliming@byosoft.com.cn>;
> Feng, Bob C
> > <bob.c.feng@intel.com>; Chen, Christine <yuwei.chen@intel.com>;
> Michael Kubacki <mikuback@linux.microsoft.com>; Sean Brogan
> > <sean.brogan@microsoft.com>; Chiu, Chasel <chasel.chiu@intel.com>;
> Desimone, Nathaniel L <nathaniel.l.desimone@intel.com>; Zeng,
> > Star <star.zeng@intel.com>; Andrew Fish <afish@apple.com>
> > Cc: Ni, Ray <ray.ni@intel.com>; Leif Lindholm <quic_llindhol@quicinc.com>;
> Liu, Zhiguang <zhiguang.liu@intel.com>; Wang, Jian J
> > <jian.j.wang@intel.com>; Lu, Xiaoyu1 <xiaoyu1.lu@intel.com>; Jiang,
> Guomin <guomin.jiang@intel.com>; Guo, Gua
> > <gua.guo@intel.com>; Ard Biesheuvel <ardb@kernel.org>; Pedro Falcato
> <pedro.falcato@gmail.com>; Gerd Hoffmann
> > <kraxel@redhat.com>; Marvin Häuser <mhaeuser@posteo.de>
> > Subject: Re: [edk2-devel] [PATCH v5 00/13]
> BaseTools,CryptoPkg,EmulatorPkg,MdePkg,others: Delete
> CLANG35,CLANG38,VS2008-
> > 2013,EBC, deprecate GCC48,GCC49,GCC5, add GCC and GCCNOLTO, update
> CLANGDWARF
> >
> > Do you have an update?
> >
> >
> > --
> >
> > Rebecca Cran
> >
> >
> > On 4/25/23 09:26, Michael D Kinney wrote:
> > > Made some progress on root cause.  Related to rename of CLANG38 to
> CLANGDWARF definitions for IA32.
> > >
> > > Wil give an update in a few hours after some more testing.
> > >
> > > Mike
> > >
> > >> -----Original Message-----
> > >> From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of
> Rebecca Cran
> > >> Sent: Tuesday, April 25, 2023 6:10 AM
> > >> To: devel@edk2.groups.io; Kinney, Michael D
> <michael.d.kinney@intel.com>; Gao, Liming <gaoliming@byosoft.com.cn>;
> Feng, Bob C
> > >> <bob.c.feng@intel.com>; Chen, Christine <yuwei.chen@intel.com>;
> Michael Kubacki <mikuback@linux.microsoft.com>; Sean Brogan
> > >> <sean.brogan@microsoft.com>; Chiu, Chasel <chasel.chiu@intel.com>;
> Desimone, Nathaniel L <nathaniel.l.desimone@intel.com>;
> > Zeng,
> > >> Star <star.zeng@intel.com>; Andrew Fish <afish@apple.com>
> > >> Cc: Ni, Ray <ray.ni@intel.com>; Leif Lindholm
> <quic_llindhol@quicinc.com>; Liu, Zhiguang <zhiguang.liu@intel.com>;
> Wang, Jian
> > J
> > >> <jian.j.wang@intel.com>; Lu, Xiaoyu1 <xiaoyu1.lu@intel.com>; Jiang,
> Guomin <guomin.jiang@intel.com>; Guo, Gua
> > >> <gua.guo@intel.com>; Ard Biesheuvel <ardb@kernel.org>; Pedro
> Falcato <pedro.falcato@gmail.com>; Gerd Hoffmann
> > >> <kraxel@redhat.com>; Marvin Häuser <mhaeuser@posteo.de>
> > >> Subject: Re: [edk2-devel] [PATCH v5 00/13]
> BaseTools,CryptoPkg,EmulatorPkg,MdePkg,others: Delete
> CLANG35,CLANG38,VS2008-
> > >> 2013,EBC, deprecate GCC48,GCC49,GCC5, add GCC and GCCNOLTO,
> update CLANGDWARF
> > >>
> > >> I've merged the OBJCOPY fix. My understanding is that you're seeing a
> > >> problem on Windows with an internal build: if you could share part of
> > >> the log file with filenames etc. changed to protect proprietary
> > >> information that could be useful so we can try and find where the
> > >> program is trying to be run that needs quotes around it.
> > >>
> > >>
> > >> --
> > >>
> > >> Rebecca Cran
> > >>
> > >>
> > >> On 4/24/23 00:14, Michael D Kinney wrote:
> > >>> Hi Rebecca,
> > >>>
> > >>> I have seen some issues with CLANGDWARF and need a little time to
> investigate further.
> > >>>
> > >>> Mike
> > >>>
> > >>>> -----Original Message-----
> > >>>> From: Rebecca Cran <rebecca@bsdio.com>
> > >>>> Sent: Friday, April 21, 2023 10:15 AM
> > >>>> To: devel@edk2.groups.io; Gao, Liming
> <gaoliming@byosoft.com.cn>; Feng, Bob C <bob.c.feng@intel.com>; Chen,
> Christine
> > >>>> <yuwei.chen@intel.com>; Kinney, Michael D
> <michael.d.kinney@intel.com>; Michael Kubacki
> <mikuback@linux.microsoft.com>;
> > Sean
> > >>>> Brogan <sean.brogan@microsoft.com>; Chiu, Chasel
> <chasel.chiu@intel.com>; Desimone, Nathaniel L
> > >>>> <nathaniel.l.desimone@intel.com>; Zeng, Star
> <star.zeng@intel.com>; Andrew Fish <afish@apple.com>
> > >>>> Cc: Rebecca Cran <rebecca@bsdio.com>; Ni, Ray <ray.ni@intel.com>;
> Leif Lindholm <quic_llindhol@quicinc.com>; Liu, Zhiguang
> > >>>> <zhiguang.liu@intel.com>; Wang, Jian J <jian.j.wang@intel.com>; Lu,
> Xiaoyu1 <xiaoyu1.lu@intel.com>; Jiang, Guomin
> > >>>> <guomin.jiang@intel.com>; Guo, Gua <gua.guo@intel.com>; Ard
> Biesheuvel <ardb@kernel.org>; Pedro Falcato
> > >>>> <pedro.falcato@gmail.com>; Gerd Hoffmann <kraxel@redhat.com>;
> Marvin Häuser <mhaeuser@posteo.de>
> > >>>> Subject: [PATCH v5 00/13]
> BaseTools,CryptoPkg,EmulatorPkg,MdePkg,others: Delete
> CLANG35,CLANG38,VS2008-2013,EBC, deprecate
> > >>>> GCC48,GCC49,GCC5, add GCC and GCCNOLTO, update CLANGDWARF
> > >>>>
> > >>>> Update the toolchain definitions:
> > >>>>
> > >>>> - Delete the CLANG35 and CLANG38 toolchains, and replace CLANG38
> with
> > >>>>     CLANGDWARF, updating it to support ARM and AARCH64 in
> addition to X64
> > >>>> and IA32.
> > >>>>
> > >>>> - Mark GCC48, GCC49 and GCC5 as deprecated.
> > >>>>
> > >>>> - Add GCC and GCCNOLTO toolchain definitions.
> > >>>>
> > >>>> - Remove VS2008, VS2010, VS2012 and VS2013.
> > >>>>
> > >>>> - Remove EBC compiler definitions. Full removal of EBC support from
> the
> > >>>>     various packages etc. will be done in a follow-up patch series.
> > >>>>
> > >>>> - Remove unused IPHONE_TOOLS and SOURCERY_CYGWIN_TOOLS
> definitions.
> > >>>>
> > >>>> - Remove unused CYGWIN_ definitions.
> > >>>>
> > >>>> Personal GitHub PR: https://github.com/tianocore/edk2/pull/4240
> > >>>> GitHub branch: https://github.com/bcran/edk2/tree/clangdwarf
> > >>>>
> > >>>> Note: CI is now passing.
> > >>>>
> > >>>> Changes from v1 to v2
> > >>>> =====================
> > >>>>
> > >>>> - Added a commit to delete GCC48 and GCC49, rename GCC5 to GCC
> and
> > >>>>     update the flags for other toolchains to work with the new GCC
> > >>>>     definitions.
> > >>>>
> > >>>> - Bumped VERSION from 2.00 to 3.00 to inform users that they should
> > >>>>     update their Conf/tools_def.txt.
> > >>>>
> > >>>> Changes from v2 to v3
> > >>>> =====================
> > >>>>
> > >>>> - Keep GCC48, GCC49 and GCC5 but mark them deprecated, including
> with
> > >>>>     warnings at the start and end of running a build.
> > >>>>
> > >>>> - Dropped the commit fixing the IA32 build of UefiPayloadPkg,
> because it
> > >>>>     causes CI to fail due to building both IA32 and X64 at the same time:
> > >>>>
> > >>>>     "Module built under multiple ARCHs [IA32, X64]. Not able to
> determine which output to put into flash for Module..."
> > >>>>
> > >>>> - Added more information to the VERSION section in
> tools_def.template.
> > >>>>
> > >>>> - Various fixes to the CLANGDWARF flags.
> > >>>>
> > >>>> Changes from v3 to v4
> > >>>> =====================
> > >>>>
> > >>>> - Use lld for -a ARM -t CLANGDWARF.
> > >>>>
> > >>>> Changes from v4 to v5
> > >>>> =====================
> > >>>>
> > >>>> - Fixed CLANGDWARF on Windows when llvm is installed with the
> llvm.org
> > >>>>     installer and isn't installed in Visual Studio: use llvm-objcopy and the
> > >>>>     tools prefix.
> > >>>>
> > >>>> - Added flags for GCCNOLTO to UnitTestFrameworkPkgHost.dsc.inc
> > >>>>
> > >>>> - Added GCC flags to CryptoPkg/Library/OpensslLib/*.inf
> > >>>>
> > >>>> Rebecca Cran (13):
> > >>>>     BaseTools,CryptoPkg: Update CLANGDWARF, remove CLANG 35/38
> toolchains
> > >>>>     BaseTools: Remove VS2008, 2010, 2012 and 2013 toolchain
> definitions
> > >>>>     BaseTools: Remove VS2008-VS2013 remnants
> > >>>>     MdePkg: Remove VS2008-VS2013 remnants
> > >>>>     edksetup.bat: Remove VS2008-VS2013 remnants
> > >>>>     BaseTools: Remove unused IPHONE_TOOLS and
> SOURCERY_CYGWIN_TOOLS defs
> > >>>>     BaseTools: Remove EBC (EFI Byte Code) compiler definitions
> > >>>>     BaseTools: Update VS toolchain descriptions in
> tools_def.txt.template
> > >>>>     BaseTools/Conf/tools_def.template: Add GCC and GCCNOLTO
> toolchains
> > >>>>     BaseTools: Only call LoadConfiguration once in build.py
> > >>>>     BaseTools: Add a deprecation warning for GCC48,GCC49,GCC5 to
> build.py
> > >>>>     BaseTools/Conf/tools_def.template: Add section for deprecated
> > >>>>       toolchains
> > >>>>     BaseTools/Conf/tools_def.template: Bump VERSION to 3.00
> > >>>>
> > >>>>    UnitTestFrameworkPkg/UnitTestFrameworkPkgHost.dsc.inc       |    4
> ++++
> > >>>>    IntelFsp2Pkg/Tools/Tests/QemuFspPkg.dsc                     |    9
> ++++++++-
> > >>>>    CryptoPkg/Library/BaseCryptLib/BaseCryptLib.inf             |    3 +--
> > >>>>    CryptoPkg/Library/BaseCryptLib/PeiCryptLib.inf              |    3 +--
> > >>>>    CryptoPkg/Library/BaseCryptLib/RuntimeCryptLib.inf          |    3 +--
> > >>>>    CryptoPkg/Library/BaseCryptLib/SecCryptLib.inf              |    3 +--
> > >>>>    CryptoPkg/Library/BaseCryptLib/SmmCryptLib.inf              |    3 +--
> > >>>>    CryptoPkg/Library/BaseCryptLib/UnitTestHostBaseCryptLib.inf |    3
> +--
> > >>>>    CryptoPkg/Library/OpensslLib/OpensslLib.inf                 |    4 ++--
> > >>>>    CryptoPkg/Library/OpensslLib/OpensslLibAccel.inf            |    4 ++--
> > >>>>    CryptoPkg/Library/OpensslLib/OpensslLibCrypto.inf           |    4 ++--
> > >>>>    CryptoPkg/Library/OpensslLib/OpensslLibFull.inf             |    4 ++--
> > >>>>    CryptoPkg/Library/OpensslLib/OpensslLibFullAccel.inf        |    5 +++--
> > >>>>    EmulatorPkg/Unix/Host/Host.inf                              |    2 ++
> > >>>>    MdePkg/Include/Ia32/ProcessorBind.h                         |    8 ++------
> > >>>>    MdePkg/Include/X64/ProcessorBind.h                          |    8 ++------
> > >>>>    BaseTools/Conf/tools_def.template                           | 1993
> > >>>>
> > >>
> >
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> ++++++++++++
> > >>
> >
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> ++++++++++++
> > >>
> >
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> ++++++++++++
> > >>
> >
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> ++++++++++++
> > >>>>
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> ++++++++++++-----------------------------------------------------
> > --
> > >> ---
> > >>>> ------------------------------------------------------------------------------------
> ---------------------------------------
> > --
> > >> ---
> > >>>> ------------------------------------------------------------------------------------
> ---------------------------------------
> > --
> > >> ---
> > >>>> ------------------------------------------------------------------------------------
> ---------
> > >>>> ------------------------------------------------------------------------------------
> ---------------------------------------
> > --
> > >> ---
> > >>>> ------------------------------------------------------------------------------------
> ---------------------------------------
> > --
> > >> ---
> > >>>> ------------------------------------------------------------------------------------
> ---------------------------------------
> > --
> > >> ---
> > >>>> ------------------------------------------------------------------------------------
> ---------------------------------------
> > --
> > >> ---
> > >>>> ------------------------------------------------------------------------------------
> ---------------------------------------
> > --
> > >> ---
> > >>>> ------------------------------------------------------------------------------------
> ---------------------------------------
> > --
> > >> ---
> > >>>> ------------------------------------------------------------------------------------
> ---------------------------------------
> > --
> > >> ---
> > >>>> ------------------------------------------------------------------------------------
> ---------
> > >>>> ---------------
> > >>>>    BaseTools/Scripts/SetVisualStudio.bat                       |   22 +--------------
> -------
> > >>>>    BaseTools/Scripts/ShowEnvironment.bat                       |   44 ------------
> --------------------------------
> > >>>>    BaseTools/Source/Python/build/build.py                      |   18
> +++++++++++++++---
> > >>>>    BaseTools/get_vsvars.bat                                    |   13 -------------
> > >>>>    BaseTools/set_vsprefix_envs.bat                             |   64 ------------------
> -----------------------------------
> > ---
> > >> ----
> > >>>> ----
> > >>>>    BaseTools/toolsetup.bat                                     |   24 +----------------------
> -
> > >>>>    edksetup.bat                                                |    6 +-----
> > >>>>    24 files changed, 635 insertions(+), 1619 deletions(-)
> > >>>>
> > >>>> --
> > >>>> 2.39.2
> > >>>
> > >>>
> > >>>
> > >>>
> > >>
> > >>
> > >>
> > >
> > >
> > > 
> > >
> > >