* How to debug a BaseTools build Python hang?
@ 2021-06-25 20:05 Andrew Fish
0 siblings, 0 replies; only message in thread
From: Andrew Fish @ 2021-06-25 20:05 UTC (permalink / raw)
To: edk2-devel-groups-io
I’m hitting an issue with deadlocks on our fork of the edk2 build system. It only happens on our builders (30+ cores), and I can’t repo locally. We realized we had fixed the number of threads, and when we removed that from our build config we started seeing this issue.
The build is deadlocking during the `…` so that implies during the autogen phase. I hooked up a signal handler so I could catch `kill != 9` and dump the Python threads.
I see these threads:
MainThread: seems stuck at the self.AutoGenMgr.join() in StartAutoGen().
Thread-1: looks like a logging thread.
Thread-2: looks like the thread printing …
2 x QueueFeederThread: I’m not sure if these are workers for Python or kicked off by the build?
I’m looking for some advice on how to debug the deadlock. I’ve tried sampling via kill -2 etc. and my exception handler but I don’t seem to be able to see the work happening on Python threads….
Thread: QueueFeederThread(123145563672576)
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py:774 function __bootstrap
self.__bootstrap_inner()
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py:801 function __bootstrap_inner
self.run()
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py:754 function run
self.__target(*self.__args, **self.__kwargs)
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/queues.py:252 function _feed
nwait()
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py:340 function wait
waiter.acquire()
Thread: Thread-2(123145555259392)
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py:774 function __bootstrap
self.__bootstrap_inner()
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py:801 function __bootstrap_inner
self.run()
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py:754 function run
self.__target(*self.__args, **self.__kwargs)
./BaseTools/Source/Python/Common/Misc.py:907 function _ProgressThreadEntry
time.sleep(self._CheckInterval)
Thread: MainThread(4614635008)
./BaseTools/Source/Python/build/build.py:2754 function <module>
r = Main()
./BaseTools/Source/Python/build/build.py:2643 function Main
MyBuild.Launch()
./BaseTools/Source/Python/build/build.py:2438 function Launch
self._MultiThreadBuildPlatform()
./BaseTools/Source/Python/build/build.py:2246 function _MultiThreadBuildPlatform
Wa, self.BuildModules = self.PerformAutoGen(BuildTarget,ToolChain)
./BaseTools/Source/Python/build/build.py:2185 function PerformAutoGen
autogen_rt, errorcode = self.StartAutoGen(mqueue, Pa.DataPipe, self.SkipAutoGen, PcdMaList, cqueue)
./BaseTools/Source/Python/build/build.py:889 function StartAutoGen
self.AutoGenMgr.join()
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py:940 function join
self.__block.wait()
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py:340 function wait
waiter.acquire()
./BaseTools/Source/Python/build/build.py:2737 function AppleDumpAllThreadStacks
for filename, lineno, name, line in traceback.extract_stack(stack):
Thread: Thread-1(123145546846208)
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py:774 function __bootstrap
self.__bootstrap_inner()
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py:801 function __bootstrap_inner
self.run()
./BaseTools/Source/Python/AutoGen/AutoGenWorker.py:85 function run
log_message = self.log_q.get()
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/queues.py:117 function get
res = self._recv()
Thread: QueueFeederThread(123145551052800)
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py:774 function __bootstrap
self.__bootstrap_inner()
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py:801 function __bootstrap_inner
self.run()
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py:754 function run
self.__target(*self.__args, **self.__kwargs)
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/queues.py:252 function _feed
nwait()
/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py:340 function wait
waiter.acquire()
Thanks,
Andrew Fish
^ permalink raw reply [flat|nested] only message in thread
only message in thread, other threads:[~2021-06-25 20:05 UTC | newest]
Thread overview: (only message) (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2021-06-25 20:05 How to debug a BaseTools build Python hang? Andrew Fish
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox