How to debug a BaseTools build Python hang?

["Andrew Fish" <afish@apple.com>]

I’m hitting an issue with deadlocks on our fork of the edk2 build system. It only happens on our builders (30+ cores), and I can’t repo locally. We realized we had fixed the number of threads, and when we removed that from our build config we started seeing this issue. 

The build is deadlocking during the `…` so that implies during the autogen phase. I hooked up a signal handler so I could catch `kill != 9` and dump the Python threads. 
I see these threads:
MainThread:  seems stuck at the  self.AutoGenMgr.join() in StartAutoGen(). 
Thread-1: looks like a logging thread. 
Thread-2: looks like the thread printing …
2 x QueueFeederThread: I’m not sure if these are workers for Python or kicked off by the build? 

I’m looking for some advice on how to debug the deadlock. I’ve tried sampling via kill -2 etc. and my exception handler but I don’t seem to be able to see the work happening on Python threads….

Thread: QueueFeederThread(123145563672576)
  /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py:774 function __bootstrap
    self.__bootstrap_inner()
  /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py:801 function __bootstrap_inner
    self.run()
  /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py:754 function run
    self.__target(*self.__args, **self.__kwargs)
  /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/queues.py:252 function _feed
    nwait()
  /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py:340 function wait
    waiter.acquire()

Thread: Thread-2(123145555259392)
  /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py:774 function __bootstrap
    self.__bootstrap_inner()
  /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py:801 function __bootstrap_inner
    self.run()
  /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py:754 function run
    self.__target(*self.__args, **self.__kwargs)
 ./BaseTools/Source/Python/Common/Misc.py:907 function _ProgressThreadEntry
    time.sleep(self._CheckInterval)

Thread: MainThread(4614635008)
 ./BaseTools/Source/Python/build/build.py:2754 function <module>
    r = Main()
 ./BaseTools/Source/Python/build/build.py:2643 function Main
    MyBuild.Launch()
 ./BaseTools/Source/Python/build/build.py:2438 function Launch
    self._MultiThreadBuildPlatform()
 ./BaseTools/Source/Python/build/build.py:2246 function _MultiThreadBuildPlatform
    Wa, self.BuildModules = self.PerformAutoGen(BuildTarget,ToolChain)
 ./BaseTools/Source/Python/build/build.py:2185 function PerformAutoGen
    autogen_rt, errorcode = self.StartAutoGen(mqueue, Pa.DataPipe, self.SkipAutoGen, PcdMaList, cqueue)
 ./BaseTools/Source/Python/build/build.py:889 function StartAutoGen
    self.AutoGenMgr.join()
  /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py:940 function join
    self.__block.wait()
  /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py:340 function wait
    waiter.acquire()
 ./BaseTools/Source/Python/build/build.py:2737 function AppleDumpAllThreadStacks
    for filename, lineno, name, line in traceback.extract_stack(stack):

Thread: Thread-1(123145546846208)
  /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py:774 function __bootstrap
    self.__bootstrap_inner()
  /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py:801 function __bootstrap_inner
    self.run()
 ./BaseTools/Source/Python/AutoGen/AutoGenWorker.py:85 function run
    log_message = self.log_q.get()
  /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/queues.py:117 function get
    res = self._recv()

Thread: QueueFeederThread(123145551052800)
  /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py:774 function __bootstrap
    self.__bootstrap_inner()
  /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py:801 function __bootstrap_inner
    self.run()
  /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py:754 function run
    self.__target(*self.__args, **self.__kwargs)
  /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/queues.py:252 function _feed
    nwait()
  /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/threading.py:340 function wait
    waiter.acquire()

Thanks,

Andrew Fish