* BaseTools: build failure when specifying multiple BUILDTARGET
@ 2019-02-04 9:58 Leif Lindholm
2019-02-04 11:54 ` Philippe Mathieu-Daudé
0 siblings, 1 reply; 4+ messages in thread
From: Leif Lindholm @ 2019-02-04 9:58 UTC (permalink / raw)
To: edk2-devel; +Cc: Bob Feng, Liming Gao
Hi Bob, Liming,
With the latest BaseTools (current HEAD, 6c61ec4c62), building
multiple targets from a single command line crashes.
To reproduce:
build -a IA32 -t GCC5 -b RELEASE -b NOOPT -p OvmfPkg/OvmfPkgIa32.dsc
(I first built with -n32, but dropped that to see if it would make a
difference - it does not.)
The first target specified builds successfully. When starting on the
second, the output is as below, and build exits.
/
Leif
Architecture(s) = IA32
Build target = NOOPT
Toolchain = GCC5
Active Platform = /work/git/edk2/OvmfPkg/OvmfPkgIa32.dsc
Flash Image Definition = /work/git/edk2/OvmfPkg/OvmfPkgIa32.fdf
Processing meta-data ...
build.py...
: error C0DE: Unknown fatal error when processing [/work/git/edk2/OvmfPkg/OvmfPkgIa32.dsc]
(Please send email to edk2-devel@lists.01.org for help, attaching following call stack trace!)
(Python 3.5.3 on linux) Traceback (most recent call last):
File "/work/git/edk2/BaseTools/BinWrappers/PosixLike/../../Source/Python/build/build.py", line 2387, in Main
MyBuild.Launch()
File "/work/git/edk2/BaseTools/BinWrappers/PosixLike/../../Source/Python/build/build.py", line 2141, in Launch
self._MultiThreadBuildPlatform()
File "/work/git/edk2/BaseTools/BinWrappers/PosixLike/../../Source/Python/build/build.py", line 1921, in _MultiThreadBuildPlatform
self.Progress
File "/work/git/edk2/BaseTools/Source/Python/AutoGen/AutoGen.py", line 304, in __init__
self._InitWorker(Workspace, MetaFile, Target, Toolchain, Arch, *args, **kwargs)
File "/work/git/edk2/BaseTools/Source/Python/AutoGen/AutoGen.py", line 477, in _InitWorker
for BuildData in PGen.BuildDatabase._CACHE_.values():
RuntimeError: dictionary changed size during iteration
- Failed -
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: BaseTools: build failure when specifying multiple BUILDTARGET
2019-02-04 9:58 BaseTools: build failure when specifying multiple BUILDTARGET Leif Lindholm
@ 2019-02-04 11:54 ` Philippe Mathieu-Daudé
2019-02-04 12:22 ` Philippe Mathieu-Daudé
0 siblings, 1 reply; 4+ messages in thread
From: Philippe Mathieu-Daudé @ 2019-02-04 11:54 UTC (permalink / raw)
To: Liming Gao, Bob Feng; +Cc: Leif Lindholm, edk2-devel
Hi,
On 2/4/19 10:58 AM, Leif Lindholm wrote:
> Hi Bob, Liming,
>
> With the latest BaseTools (current HEAD, 6c61ec4c62), building
> multiple targets from a single command line crashes.
>
> To reproduce:
> build -a IA32 -t GCC5 -b RELEASE -b NOOPT -p OvmfPkg/OvmfPkgIa32.dsc
> (I first built with -n32, but dropped that to see if it would make a
> difference - it does not.)
>
> The first target specified builds successfully. When starting on the
> second, the output is as below, and build exits.
>
> /
> Leif
>
> Architecture(s) = IA32
> Build target = NOOPT
> Toolchain = GCC5
>
> Active Platform = /work/git/edk2/OvmfPkg/OvmfPkgIa32.dsc
> Flash Image Definition = /work/git/edk2/OvmfPkg/OvmfPkgIa32.fdf
>
> Processing meta-data ...
>
>
> build.py...
> : error C0DE: Unknown fatal error when processing [/work/git/edk2/OvmfPkg/OvmfPkgIa32.dsc]
>
> (Please send email to edk2-devel@lists.01.org for help, attaching following call stack trace!)
>
> (Python 3.5.3 on linux) Traceback (most recent call last):
> File "/work/git/edk2/BaseTools/BinWrappers/PosixLike/../../Source/Python/build/build.py", line 2387, in Main
> MyBuild.Launch()
> File "/work/git/edk2/BaseTools/BinWrappers/PosixLike/../../Source/Python/build/build.py", line 2141, in Launch
> self._MultiThreadBuildPlatform()
> File "/work/git/edk2/BaseTools/BinWrappers/PosixLike/../../Source/Python/build/build.py", line 1921, in _MultiThreadBuildPlatform
> self.Progress
> File "/work/git/edk2/BaseTools/Source/Python/AutoGen/AutoGen.py", line 304, in __init__
> self._InitWorker(Workspace, MetaFile, Target, Toolchain, Arch, *args, **kwargs)
> File "/work/git/edk2/BaseTools/Source/Python/AutoGen/AutoGen.py", line 477, in _InitWorker
> for BuildData in PGen.BuildDatabase._CACHE_.values():
> RuntimeError: dictionary changed size during iteration
I believe the culprit is f8d11e5a4aaa.
Sadly the tree isn't bisectable around that commit:
HEAD is now at fd2d74007b BaseTools:Update build tool to print python
version information
$ build -a IA32 -t GCC5 -b RELEASE -b NOOPT -p OvmfPkg/OvmfPkgIa32.dsc
build.py...
: error C0DE: Unknown fatal error when processing []
(Python 2.7.15 on linux2) Traceback (most recent call last):
File
"BaseTools/BinWrappers/PosixLike/../../Source/Python/build/build.py",
line 2385, in Main
MyBuild = Build(Target, Workspace, Option)
File
"BaseTools/BinWrappers/PosixLike/../../Source/Python/build/build.py",
line 799, in __init__
EdkLogger.quiet("%-16s = %s" % ("PYTHON", os.environ["PYTHON"]))
File "/usr/lib64/python2.7/UserDict.py", line 40, in __getitem__
raise KeyError(key)
KeyError: 'PYTHON'
HEAD is now at 4a3773e578 BaseTools:Fixed Rsa issue and a set define issue.
$ build -a IA32 -t GCC5 -b RELEASE -b NOOPT -p OvmfPkg/OvmfPkgIa32.dsc
Traceback (most recent call last):
File
"BaseTools/BinWrappers/PosixLike/../../Source/Python/build/build.py",
line 44, in <module>
from AutoGen.AutoGen import *
File "BaseTools/Source/Python/AutoGen/AutoGen.py", line 27, in <module>
from . import GenC
File "BaseTools/Source/Python/AutoGen/GenC.py", line 26, in <module>
from .StrGather import *
File "BaseTools/Source/Python/AutoGen/StrGather.py", line 21, in <module>
from .UniClassObject import *
File "BaseTools/Source/Python/AutoGen/UniClassObject.py", line 430
UniStr = ('\u' + (Line[StartPos + 2 : EndPos])).decode('unicode_escape')
^
SyntaxError: (unicode error) 'unicodeescape' codec can't decode bytes in
position 0-1: truncated \uXXXX escape
HEAD is now at 7fa0e68afd BaseTools:ord() don't match in py2 and py3
$ build -a IA32 -t GCC5 -b RELEASE -b NOOPT -p OvmfPkg/OvmfPkgIa32.dsc
build.py...
(Python 3.7.2 on linux) Traceback (most recent call last):
File
"BaseTools/BinWrappers/PosixLike/../../Source/Python/build/build.py",
line 2387, in Main
MyBuild.Launch()
File
"BaseTools/BinWrappers/PosixLike/../../Source/Python/build/build.py",
line 2141, in Launch
self._MultiThreadBuildPlatform()
File
"BaseTools/BinWrappers/PosixLike/../../Source/Python/build/build.py",
line 1921, in _MultiThreadBuildPlatform
self.Progress
File "BaseTools/Source/Python/AutoGen/AutoGen.py", line 304, in __init__
self._InitWorker(Workspace, MetaFile, Target, Toolchain, Arch,
*args, **kwargs)
File "BaseTools/Source/Python/AutoGen/AutoGen.py", line 501, in
_InitWorker
if MGen in PGen.LibraryAutoGenList:
File "BaseTools/Source/Python/Common/caching.py", line 34, in __get__
Value = obj.__dict__[self._function.__name__] = self._function(obj)
File "BaseTools/Source/Python/AutoGen/AutoGen.py", line 2092, in
LibraryAutoGenList
for La in Ma.LibraryAutoGenList:
File "BaseTools/Source/Python/Common/caching.py", line 34, in __get__
Value = obj.__dict__[self._function.__name__] = self._function(obj)
File "BaseTools/Source/Python/AutoGen/AutoGen.py", line 4087, in
LibraryAutoGenList
for Lib in La.CodaTargetList:
File "BaseTools/Source/Python/Common/caching.py", line 34, in __get__
Value = obj.__dict__[self._function.__name__] = self._function(obj)
File "BaseTools/Source/Python/AutoGen/AutoGen.py", line 3259, in
CodaTargetList
self.Targets
File "BaseTools/Source/Python/Common/caching.py", line 34, in __get__
Value = obj.__dict__[self._function.__name__] = self._function(obj)
File "BaseTools/Source/Python/AutoGen/AutoGen.py", line 3245, in Targets
self.SourceFileList
File "BaseTools/Source/Python/Common/caching.py", line 34, in __get__
Value = obj.__dict__[self._function.__name__] = self._function(obj)
File "BaseTools/Source/Python/AutoGen/AutoGen.py", line 3074, in
SourceFileList
self._MatchBuildRuleOrder(RetVal)
File "BaseTools/Source/Python/AutoGen/AutoGen.py", line 3082, in
_MatchBuildRuleOrder
self.BuildOption
File "BaseTools/Source/Python/Common/caching.py", line 34, in __get__
Value = obj.__dict__[self._function.__name__] = self._function(obj)
File "BaseTools/Source/Python/AutoGen/AutoGen.py", line 2987, in
BuildOption
RetVal, self.BuildRuleOrder =
self.PlatformInfo.ApplyBuildOption(self.Module)
File "BaseTools/Source/Python/AutoGen/AutoGen.py", line 2477, in
ApplyBuildOption
PlatformModuleOptions.keys() + ModuleTypeOptions.keys() +
TypeError: unsupported operand type(s) for +: 'dict_keys' and 'dict_keys'
I could narrow down to last good commit: 9c2d68c0a299,
first commit reporting Leif symptoms: d943b0c339fe.
Regards,
Phil.
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: BaseTools: build failure when specifying multiple BUILDTARGET
2019-02-04 11:54 ` Philippe Mathieu-Daudé
@ 2019-02-04 12:22 ` Philippe Mathieu-Daudé
2019-02-04 16:48 ` Leif Lindholm
0 siblings, 1 reply; 4+ messages in thread
From: Philippe Mathieu-Daudé @ 2019-02-04 12:22 UTC (permalink / raw)
To: Liming Gao, Bob Feng, Leif Lindholm; +Cc: edk2-devel
[-- Attachment #1.1: Type: text/plain, Size: 2832 bytes --]
On 2/4/19 12:54 PM, Philippe Mathieu-Daudé wrote:
> On 2/4/19 10:58 AM, Leif Lindholm wrote:
>> Hi Bob, Liming,
>>
>> With the latest BaseTools (current HEAD, 6c61ec4c62), building
>> multiple targets from a single command line crashes.
>>
>> To reproduce:
>> build -a IA32 -t GCC5 -b RELEASE -b NOOPT -p OvmfPkg/OvmfPkgIa32.dsc
>> (I first built with -n32, but dropped that to see if it would make a
>> difference - it does not.)
>>
>> The first target specified builds successfully. When starting on the
>> second, the output is as below, and build exits.
>>
>> /
>> Leif
>>
>> Architecture(s) = IA32
>> Build target = NOOPT
>> Toolchain = GCC5
>>
>> Active Platform = /work/git/edk2/OvmfPkg/OvmfPkgIa32.dsc
>> Flash Image Definition = /work/git/edk2/OvmfPkg/OvmfPkgIa32.fdf
>>
>> Processing meta-data ...
>>
>>
>> build.py...
>> : error C0DE: Unknown fatal error when processing [/work/git/edk2/OvmfPkg/OvmfPkgIa32.dsc]
>>
>> (Please send email to edk2-devel@lists.01.org for help, attaching following call stack trace!)
>>
>> (Python 3.5.3 on linux) Traceback (most recent call last):
>> File "/work/git/edk2/BaseTools/BinWrappers/PosixLike/../../Source/Python/build/build.py", line 2387, in Main
>> MyBuild.Launch()
>> File "/work/git/edk2/BaseTools/BinWrappers/PosixLike/../../Source/Python/build/build.py", line 2141, in Launch
>> self._MultiThreadBuildPlatform()
>> File "/work/git/edk2/BaseTools/BinWrappers/PosixLike/../../Source/Python/build/build.py", line 1921, in _MultiThreadBuildPlatform
>> self.Progress
>> File "/work/git/edk2/BaseTools/Source/Python/AutoGen/AutoGen.py", line 304, in __init__
>> self._InitWorker(Workspace, MetaFile, Target, Toolchain, Arch, *args, **kwargs)
>> File "/work/git/edk2/BaseTools/Source/Python/AutoGen/AutoGen.py", line 477, in _InitWorker
>> for BuildData in PGen.BuildDatabase._CACHE_.values():
>> RuntimeError: dictionary changed size during iteration
>
> I believe the culprit is f8d11e5a4aaa.
With Python3 the dict.value() method returns an iterator.
Using list() to force a copy works for me.
Since I'm not a Python expert (and less with joys of py2/py3
conversion), I'll start sharing a snippet rather than a formal patch :)
-- >8 --
--- a/BaseTools/Source/Python/AutoGen/AutoGen.py
+++ b/BaseTools/Source/Python/AutoGen/AutoGen.py
@@ -475,5 +475,5 @@ class WorkspaceAutoGen(AutoGen):
# generate the SourcePcdDict and BinaryPcdDict
PGen = PlatformAutoGen(self, self.MetaFile, Target,
Toolchain, Arch)
- for BuildData in PGen.BuildDatabase._CACHE_.values():
+ for BuildData in list(PGen.BuildDatabase._CACHE_.values()):
if BuildData.Arch != Arch:
continue
---
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: BaseTools: build failure when specifying multiple BUILDTARGET
2019-02-04 12:22 ` Philippe Mathieu-Daudé
@ 2019-02-04 16:48 ` Leif Lindholm
0 siblings, 0 replies; 4+ messages in thread
From: Leif Lindholm @ 2019-02-04 16:48 UTC (permalink / raw)
To: Philippe Mathieu-Daudé; +Cc: Liming Gao, Bob Feng, edk2-devel
On Mon, Feb 04, 2019 at 01:22:03PM +0100, Philippe Mathieu-Daudé wrote:
> On 2/4/19 12:54 PM, Philippe Mathieu-Daudé wrote:
> > On 2/4/19 10:58 AM, Leif Lindholm wrote:
> >> Hi Bob, Liming,
> >>
> >> With the latest BaseTools (current HEAD, 6c61ec4c62), building
> >> multiple targets from a single command line crashes.
> >>
> >> To reproduce:
> >> build -a IA32 -t GCC5 -b RELEASE -b NOOPT -p OvmfPkg/OvmfPkgIa32.dsc
> >> (I first built with -n32, but dropped that to see if it would make a
> >> difference - it does not.)
> >>
> >> The first target specified builds successfully. When starting on the
> >> second, the output is as below, and build exits.
> >>
> >> /
> >> Leif
> >>
> >> Architecture(s) = IA32
> >> Build target = NOOPT
> >> Toolchain = GCC5
> >>
> >> Active Platform = /work/git/edk2/OvmfPkg/OvmfPkgIa32.dsc
> >> Flash Image Definition = /work/git/edk2/OvmfPkg/OvmfPkgIa32.fdf
> >>
> >> Processing meta-data ...
> >>
> >>
> >> build.py...
> >> : error C0DE: Unknown fatal error when processing [/work/git/edk2/OvmfPkg/OvmfPkgIa32.dsc]
> >>
> >> (Please send email to edk2-devel@lists.01.org for help, attaching following call stack trace!)
> >>
> >> (Python 3.5.3 on linux) Traceback (most recent call last):
> >> File "/work/git/edk2/BaseTools/BinWrappers/PosixLike/../../Source/Python/build/build.py", line 2387, in Main
> >> MyBuild.Launch()
> >> File "/work/git/edk2/BaseTools/BinWrappers/PosixLike/../../Source/Python/build/build.py", line 2141, in Launch
> >> self._MultiThreadBuildPlatform()
> >> File "/work/git/edk2/BaseTools/BinWrappers/PosixLike/../../Source/Python/build/build.py", line 1921, in _MultiThreadBuildPlatform
> >> self.Progress
> >> File "/work/git/edk2/BaseTools/Source/Python/AutoGen/AutoGen.py", line 304, in __init__
> >> self._InitWorker(Workspace, MetaFile, Target, Toolchain, Arch, *args, **kwargs)
> >> File "/work/git/edk2/BaseTools/Source/Python/AutoGen/AutoGen.py", line 477, in _InitWorker
> >> for BuildData in PGen.BuildDatabase._CACHE_.values():
> >> RuntimeError: dictionary changed size during iteration
> >
> > I believe the culprit is f8d11e5a4aaa.
>
> With Python3 the dict.value() method returns an iterator.
> Using list() to force a copy works for me.
>
> Since I'm not a Python expert (and less with joys of py2/py3
> conversion), I'll start sharing a snippet rather than a formal patch :)
I can confirm this change *appears* to fix the problem with both
python3.5 and python2.7. So from my end:
Tested-by: Leif Lindholm <leif.lindholm@linaro.org>
Thanks!
>
> -- >8 --
> --- a/BaseTools/Source/Python/AutoGen/AutoGen.py
> +++ b/BaseTools/Source/Python/AutoGen/AutoGen.py
> @@ -475,5 +475,5 @@ class WorkspaceAutoGen(AutoGen):
> # generate the SourcePcdDict and BinaryPcdDict
> PGen = PlatformAutoGen(self, self.MetaFile, Target,
> Toolchain, Arch)
> - for BuildData in PGen.BuildDatabase._CACHE_.values():
> + for BuildData in list(PGen.BuildDatabase._CACHE_.values()):
> if BuildData.Arch != Arch:
> continue
> ---
>
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2019-02-04 16:48 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2019-02-04 9:58 BaseTools: build failure when specifying multiple BUILDTARGET Leif Lindholm
2019-02-04 11:54 ` Philippe Mathieu-Daudé
2019-02-04 12:22 ` Philippe Mathieu-Daudé
2019-02-04 16:48 ` Leif Lindholm
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox