BaseTools: build failure when specifying multiple BUILDTARGET

BaseTools: build failure when specifying multiple BUILDTARGET

[Leif Lindholm]

Hi Bob, Liming,

With the latest BaseTools (current HEAD, 6c61ec4c62), building
multiple targets from a single command line crashes.

To reproduce:
build -a IA32 -t GCC5 -b RELEASE -b NOOPT -p OvmfPkg/OvmfPkgIa32.dsc
(I first built with -n32, but dropped that to see if it would make a
difference - it does not.)

The first target specified builds successfully. When starting on the
second, the output is as below, and build exits.

/
    Leif

Architecture(s)  = IA32
Build target     = NOOPT
Toolchain        = GCC5

Active Platform          = /work/git/edk2/OvmfPkg/OvmfPkgIa32.dsc
Flash Image Definition   = /work/git/edk2/OvmfPkg/OvmfPkgIa32.fdf

Processing meta-data ...


build.py...
 : error C0DE: Unknown fatal error when processing [/work/git/edk2/OvmfPkg/OvmfPkgIa32.dsc]

(Please send email to edk2-devel@lists.01.org for help, attaching following call stack trace!)

(Python 3.5.3 on linux) Traceback (most recent call last):
  File "/work/git/edk2/BaseTools/BinWrappers/PosixLike/../../Source/Python/build/build.py", line 2387, in Main
    MyBuild.Launch()
  File "/work/git/edk2/BaseTools/BinWrappers/PosixLike/../../Source/Python/build/build.py", line 2141, in Launch
    self._MultiThreadBuildPlatform()
  File "/work/git/edk2/BaseTools/BinWrappers/PosixLike/../../Source/Python/build/build.py", line 1921, in _MultiThreadBuildPlatform
    self.Progress
  File "/work/git/edk2/BaseTools/Source/Python/AutoGen/AutoGen.py", line 304, in __init__
    self._InitWorker(Workspace, MetaFile, Target, Toolchain, Arch, *args, **kwargs)
  File "/work/git/edk2/BaseTools/Source/Python/AutoGen/AutoGen.py", line 477, in _InitWorker
    for BuildData in PGen.BuildDatabase._CACHE_.values():
RuntimeError: dictionary changed size during iteration


- Failed -

Re: BaseTools: build failure when specifying multiple BUILDTARGET

[Philippe Mathieu-Daudé]

Hi,

On 2/4/19 10:58 AM, Leif Lindholm wrote:
> Hi Bob, Liming,
> 
> With the latest BaseTools (current HEAD, 6c61ec4c62), building
> multiple targets from a single command line crashes.
> 
> To reproduce:
> build -a IA32 -t GCC5 -b RELEASE -b NOOPT -p OvmfPkg/OvmfPkgIa32.dsc
> (I first built with -n32, but dropped that to see if it would make a
> difference - it does not.)
> 
> The first target specified builds successfully. When starting on the
> second, the output is as below, and build exits.
> 
> /
>     Leif
> 
> Architecture(s)  = IA32
> Build target     = NOOPT
> Toolchain        = GCC5
> 
> Active Platform          = /work/git/edk2/OvmfPkg/OvmfPkgIa32.dsc
> Flash Image Definition   = /work/git/edk2/OvmfPkg/OvmfPkgIa32.fdf
> 
> Processing meta-data ...
> 
> 
> build.py...
>  : error C0DE: Unknown fatal error when processing [/work/git/edk2/OvmfPkg/OvmfPkgIa32.dsc]
> 
> (Please send email to edk2-devel@lists.01.org for help, attaching following call stack trace!)
> 
> (Python 3.5.3 on linux) Traceback (most recent call last):
>   File "/work/git/edk2/BaseTools/BinWrappers/PosixLike/../../Source/Python/build/build.py", line 2387, in Main
>     MyBuild.Launch()
>   File "/work/git/edk2/BaseTools/BinWrappers/PosixLike/../../Source/Python/build/build.py", line 2141, in Launch
>     self._MultiThreadBuildPlatform()
>   File "/work/git/edk2/BaseTools/BinWrappers/PosixLike/../../Source/Python/build/build.py", line 1921, in _MultiThreadBuildPlatform
>     self.Progress
>   File "/work/git/edk2/BaseTools/Source/Python/AutoGen/AutoGen.py", line 304, in __init__
>     self._InitWorker(Workspace, MetaFile, Target, Toolchain, Arch, *args, **kwargs)
>   File "/work/git/edk2/BaseTools/Source/Python/AutoGen/AutoGen.py", line 477, in _InitWorker
>     for BuildData in PGen.BuildDatabase._CACHE_.values():
> RuntimeError: dictionary changed size during iteration

I believe the culprit is f8d11e5a4aaa.

Sadly the tree isn't bisectable around that commit:

HEAD is now at fd2d74007b BaseTools:Update build tool to print python
version information
$ build -a IA32 -t GCC5 -b RELEASE -b NOOPT -p OvmfPkg/OvmfPkgIa32.dsc
build.py...
 : error C0DE: Unknown fatal error when processing []
(Python 2.7.15 on linux2) Traceback (most recent call last):
  File
"BaseTools/BinWrappers/PosixLike/../../Source/Python/build/build.py",
line 2385, in Main
    MyBuild = Build(Target, Workspace, Option)
  File
"BaseTools/BinWrappers/PosixLike/../../Source/Python/build/build.py",
line 799, in __init__
    EdkLogger.quiet("%-16s = %s" % ("PYTHON", os.environ["PYTHON"]))
  File "/usr/lib64/python2.7/UserDict.py", line 40, in __getitem__
    raise KeyError(key)
KeyError: 'PYTHON'

HEAD is now at 4a3773e578 BaseTools:Fixed Rsa issue and a set define issue.
$ build -a IA32 -t GCC5 -b RELEASE -b NOOPT -p OvmfPkg/OvmfPkgIa32.dsc
Traceback (most recent call last):
  File
"BaseTools/BinWrappers/PosixLike/../../Source/Python/build/build.py",
line 44, in <module>
    from AutoGen.AutoGen import *
  File "BaseTools/Source/Python/AutoGen/AutoGen.py", line 27, in <module>
    from . import GenC
  File "BaseTools/Source/Python/AutoGen/GenC.py", line 26, in <module>
    from .StrGather import *
  File "BaseTools/Source/Python/AutoGen/StrGather.py", line 21, in <module>
    from .UniClassObject import *
  File "BaseTools/Source/Python/AutoGen/UniClassObject.py", line 430
    UniStr = ('\u' + (Line[StartPos + 2 : EndPos])).decode('unicode_escape')
             ^
SyntaxError: (unicode error) 'unicodeescape' codec can't decode bytes in
position 0-1: truncated \uXXXX escape

HEAD is now at 7fa0e68afd BaseTools:ord() don't match in py2 and py3
$ build -a IA32 -t GCC5 -b RELEASE -b NOOPT -p OvmfPkg/OvmfPkgIa32.dsc
build.py...
(Python 3.7.2 on linux) Traceback (most recent call last):
  File
"BaseTools/BinWrappers/PosixLike/../../Source/Python/build/build.py",
line 2387, in Main
    MyBuild.Launch()
  File
"BaseTools/BinWrappers/PosixLike/../../Source/Python/build/build.py",
line 2141, in Launch
    self._MultiThreadBuildPlatform()
  File
"BaseTools/BinWrappers/PosixLike/../../Source/Python/build/build.py",
line 1921, in _MultiThreadBuildPlatform
    self.Progress
  File "BaseTools/Source/Python/AutoGen/AutoGen.py", line 304, in __init__
    self._InitWorker(Workspace, MetaFile, Target, Toolchain, Arch,
*args, **kwargs)
  File "BaseTools/Source/Python/AutoGen/AutoGen.py", line 501, in
_InitWorker
    if MGen in PGen.LibraryAutoGenList:
  File "BaseTools/Source/Python/Common/caching.py", line 34, in __get__
    Value = obj.__dict__[self._function.__name__] = self._function(obj)
  File "BaseTools/Source/Python/AutoGen/AutoGen.py", line 2092, in
LibraryAutoGenList
    for La in Ma.LibraryAutoGenList:
  File "BaseTools/Source/Python/Common/caching.py", line 34, in __get__
    Value = obj.__dict__[self._function.__name__] = self._function(obj)
  File "BaseTools/Source/Python/AutoGen/AutoGen.py", line 4087, in
LibraryAutoGenList
    for Lib in La.CodaTargetList:
  File "BaseTools/Source/Python/Common/caching.py", line 34, in __get__
    Value = obj.__dict__[self._function.__name__] = self._function(obj)
  File "BaseTools/Source/Python/AutoGen/AutoGen.py", line 3259, in
CodaTargetList
    self.Targets
  File "BaseTools/Source/Python/Common/caching.py", line 34, in __get__
    Value = obj.__dict__[self._function.__name__] = self._function(obj)
  File "BaseTools/Source/Python/AutoGen/AutoGen.py", line 3245, in Targets
    self.SourceFileList
  File "BaseTools/Source/Python/Common/caching.py", line 34, in __get__
    Value = obj.__dict__[self._function.__name__] = self._function(obj)
  File "BaseTools/Source/Python/AutoGen/AutoGen.py", line 3074, in
SourceFileList
    self._MatchBuildRuleOrder(RetVal)
  File "BaseTools/Source/Python/AutoGen/AutoGen.py", line 3082, in
_MatchBuildRuleOrder
    self.BuildOption
  File "BaseTools/Source/Python/Common/caching.py", line 34, in __get__
    Value = obj.__dict__[self._function.__name__] = self._function(obj)
  File "BaseTools/Source/Python/AutoGen/AutoGen.py", line 2987, in
BuildOption
    RetVal, self.BuildRuleOrder =
self.PlatformInfo.ApplyBuildOption(self.Module)
  File "BaseTools/Source/Python/AutoGen/AutoGen.py", line 2477, in
ApplyBuildOption
    PlatformModuleOptions.keys() + ModuleTypeOptions.keys() +
TypeError: unsupported operand type(s) for +: 'dict_keys' and 'dict_keys'

I could narrow down to last good commit: 9c2d68c0a299,
first commit reporting Leif symptoms: d943b0c339fe.

Regards,

Phil.

Re: BaseTools: build failure when specifying multiple BUILDTARGET

[Philippe Mathieu-Daudé]

[-- Attachment #1.1: Type: text/plain, Size: 2832 bytes --]

On 2/4/19 12:54 PM, Philippe Mathieu-Daudé wrote:
> On 2/4/19 10:58 AM, Leif Lindholm wrote:
>> Hi Bob, Liming,
>>
>> With the latest BaseTools (current HEAD, 6c61ec4c62), building
>> multiple targets from a single command line crashes.
>>
>> To reproduce:
>> build -a IA32 -t GCC5 -b RELEASE -b NOOPT -p OvmfPkg/OvmfPkgIa32.dsc
>> (I first built with -n32, but dropped that to see if it would make a
>> difference - it does not.)
>>
>> The first target specified builds successfully. When starting on the
>> second, the output is as below, and build exits.
>>
>> /
>>     Leif
>>
>> Architecture(s)  = IA32
>> Build target     = NOOPT
>> Toolchain        = GCC5
>>
>> Active Platform          = /work/git/edk2/OvmfPkg/OvmfPkgIa32.dsc
>> Flash Image Definition   = /work/git/edk2/OvmfPkg/OvmfPkgIa32.fdf
>>
>> Processing meta-data ...
>>
>>
>> build.py...
>>  : error C0DE: Unknown fatal error when processing [/work/git/edk2/OvmfPkg/OvmfPkgIa32.dsc]
>>
>> (Please send email to edk2-devel@lists.01.org for help, attaching following call stack trace!)
>>
>> (Python 3.5.3 on linux) Traceback (most recent call last):
>>   File "/work/git/edk2/BaseTools/BinWrappers/PosixLike/../../Source/Python/build/build.py", line 2387, in Main
>>     MyBuild.Launch()
>>   File "/work/git/edk2/BaseTools/BinWrappers/PosixLike/../../Source/Python/build/build.py", line 2141, in Launch
>>     self._MultiThreadBuildPlatform()
>>   File "/work/git/edk2/BaseTools/BinWrappers/PosixLike/../../Source/Python/build/build.py", line 1921, in _MultiThreadBuildPlatform
>>     self.Progress
>>   File "/work/git/edk2/BaseTools/Source/Python/AutoGen/AutoGen.py", line 304, in __init__
>>     self._InitWorker(Workspace, MetaFile, Target, Toolchain, Arch, *args, **kwargs)
>>   File "/work/git/edk2/BaseTools/Source/Python/AutoGen/AutoGen.py", line 477, in _InitWorker
>>     for BuildData in PGen.BuildDatabase._CACHE_.values():
>> RuntimeError: dictionary changed size during iteration
> 
> I believe the culprit is f8d11e5a4aaa.

With Python3 the dict.value() method returns an iterator.
Using list() to force a copy works for me.

Since I'm not a Python expert (and less with joys of py2/py3
conversion), I'll start sharing a snippet rather than a formal patch :)

-- >8 --
--- a/BaseTools/Source/Python/AutoGen/AutoGen.py
+++ b/BaseTools/Source/Python/AutoGen/AutoGen.py
@@ -475,5 +475,5 @@ class WorkspaceAutoGen(AutoGen):
             # generate the SourcePcdDict and BinaryPcdDict
             PGen = PlatformAutoGen(self, self.MetaFile, Target,
Toolchain, Arch)
-            for BuildData in PGen.BuildDatabase._CACHE_.values():
+            for BuildData in list(PGen.BuildDatabase._CACHE_.values()):
                 if BuildData.Arch != Arch:
                     continue
---


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: BaseTools: build failure when specifying multiple BUILDTARGET

[Leif Lindholm]

On Mon, Feb 04, 2019 at 01:22:03PM +0100, Philippe Mathieu-Daudé wrote:
> On 2/4/19 12:54 PM, Philippe Mathieu-Daudé wrote:
> > On 2/4/19 10:58 AM, Leif Lindholm wrote:
> >> Hi Bob, Liming,
> >>
> >> With the latest BaseTools (current HEAD, 6c61ec4c62), building
> >> multiple targets from a single command line crashes.
> >>
> >> To reproduce:
> >> build -a IA32 -t GCC5 -b RELEASE -b NOOPT -p OvmfPkg/OvmfPkgIa32.dsc
> >> (I first built with -n32, but dropped that to see if it would make a
> >> difference - it does not.)
> >>
> >> The first target specified builds successfully. When starting on the
> >> second, the output is as below, and build exits.
> >>
> >> /
> >>     Leif
> >>
> >> Architecture(s)  = IA32
> >> Build target     = NOOPT
> >> Toolchain        = GCC5
> >>
> >> Active Platform          = /work/git/edk2/OvmfPkg/OvmfPkgIa32.dsc
> >> Flash Image Definition   = /work/git/edk2/OvmfPkg/OvmfPkgIa32.fdf
> >>
> >> Processing meta-data ...
> >>
> >>
> >> build.py...
> >>  : error C0DE: Unknown fatal error when processing [/work/git/edk2/OvmfPkg/OvmfPkgIa32.dsc]
> >>
> >> (Please send email to edk2-devel@lists.01.org for help, attaching following call stack trace!)
> >>
> >> (Python 3.5.3 on linux) Traceback (most recent call last):
> >>   File "/work/git/edk2/BaseTools/BinWrappers/PosixLike/../../Source/Python/build/build.py", line 2387, in Main
> >>     MyBuild.Launch()
> >>   File "/work/git/edk2/BaseTools/BinWrappers/PosixLike/../../Source/Python/build/build.py", line 2141, in Launch
> >>     self._MultiThreadBuildPlatform()
> >>   File "/work/git/edk2/BaseTools/BinWrappers/PosixLike/../../Source/Python/build/build.py", line 1921, in _MultiThreadBuildPlatform
> >>     self.Progress
> >>   File "/work/git/edk2/BaseTools/Source/Python/AutoGen/AutoGen.py", line 304, in __init__
> >>     self._InitWorker(Workspace, MetaFile, Target, Toolchain, Arch, *args, **kwargs)
> >>   File "/work/git/edk2/BaseTools/Source/Python/AutoGen/AutoGen.py", line 477, in _InitWorker
> >>     for BuildData in PGen.BuildDatabase._CACHE_.values():
> >> RuntimeError: dictionary changed size during iteration
> > 
> > I believe the culprit is f8d11e5a4aaa.
> 
> With Python3 the dict.value() method returns an iterator.
> Using list() to force a copy works for me.
> 
> Since I'm not a Python expert (and less with joys of py2/py3
> conversion), I'll start sharing a snippet rather than a formal patch :)

I can confirm this change *appears* to fix the problem with both
python3.5 and python2.7. So from my end:
Tested-by: Leif Lindholm <leif.lindholm@linaro.org>

Thanks!

> 
> -- >8 --
> --- a/BaseTools/Source/Python/AutoGen/AutoGen.py
> +++ b/BaseTools/Source/Python/AutoGen/AutoGen.py
> @@ -475,5 +475,5 @@ class WorkspaceAutoGen(AutoGen):
>              # generate the SourcePcdDict and BinaryPcdDict
>              PGen = PlatformAutoGen(self, self.MetaFile, Target,
> Toolchain, Arch)
> -            for BuildData in PGen.BuildDatabase._CACHE_.values():
> +            for BuildData in list(PGen.BuildDatabase._CACHE_.values()):
>                  if BuildData.Arch != Arch:
>                      continue
> ---
>