From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received-SPF: Pass (sender SPF authorized) identity=mailfrom; client-ip=17.151.62.26; helo=mail-in4.apple.com; envelope-from=afish@apple.com; receiver=edk2-devel@lists.01.org Received: from mail-in4.apple.com (mail-out4.apple.com [17.151.62.26]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ml01.01.org (Postfix) with ESMTPS id 1E9292034C5DD for ; Wed, 22 Nov 2017 15:57:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; d=apple.com; s=mailout2048s; c=relaxed/simple; q=dns/txt; i=@apple.com; t=1511395310; h=From:Sender:Reply-To:Subject:Date:Message-id:To:Cc:MIME-version:Content-type: Content-transfer-encoding:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:In-reply-to:References:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=84BxfXWBI/msNotn2j+jUZhnU+6sLuwzxChQas2BEj4=; b=Zt40Bb0Y8yIWwYWP7iZbnIiRu28Z2oYeCkQdaVDjwcbwcvi4MgwTV1Ogy3WsNbbr x35icAuOcUAt7bAcxWa0+Qp9APSeDWHDXdbgOIuzzfxvOGBMlD41e+Mn889gyCS1 CjgvDMmFfcWDKSMktzy2GPCvuwMtRWFUSTg8OZC1mVlmdoULZ4XpUi/YOm8G3tda kwDYKARe2HC9zGjOdlpe9O+muCCgh2/1eSP9kHs39cfin4vq7/IHV4ebovsDSeG3 QeU7lQes397ndSGAw6sqFw4uiZdhC/B8kShPGXUWFKr9qFKAviAyYC+Bgl137Xaa 40sD2yzTMGqDjDVOmxjV6w==; Received: from relay5.apple.com (relay5.apple.com [17.128.113.88]) (using TLS with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mail-in4.apple.com (Apple Secure Mail Relay) with SMTP id 51.7B.16042.EEF061A5; Wed, 22 Nov 2017 16:01:50 -0800 (PST) X-AuditID: 11973e12-801fd9c000003eaa-16-5a160fee9a26 Received: from nwk-mmpp-sz12.apple.com (nwk-mmpp-sz12.apple.com [17.128.115.204]) by relay5.apple.com (Apple SCV relay) with SMTP id 60.3F.18983.EEF061A5; Wed, 22 Nov 2017 16:01:50 -0800 (PST) MIME-version: 1.0 Received: from [17.235.1.113] by nwk-mmpp-sz12.apple.com (Oracle Communications Messaging Server 8.0.2.1.20171102 64bit (built Nov 2 2017)) with ESMTPSA id <0OZU00BSYG30EZ00@nwk-mmpp-sz12.apple.com>; Wed, 22 Nov 2017 16:01:50 -0800 (PST) Sender: afish@apple.com From: Andrew Fish In-reply-to: <02A34F284D1DA44BB705E61F7180EF0A97FB2D01@ORSMSX114.amr.corp.intel.com> Date: Wed, 22 Nov 2017 16:01:49 -0800 Cc: "Gao, Liming" , Aaron Durbin , "edk2-devel@lists.01.org" Message-id: <10CEC444-4BE6-4A13-BF3E-D5ABE4C6971E@apple.com> References: <4A89E2EF3DFEDB4C8BFDE51014F606A14E17CC73@SHSMSX104.ccr.corp.intel.com> <4A89E2EF3DFEDB4C8BFDE51014F606A14E17D6C3@SHSMSX104.ccr.corp.intel.com> <02A34F284D1DA44BB705E61F7180EF0A97FB2D01@ORSMSX114.amr.corp.intel.com> To: "Desimone, Nathaniel L" X-Mailer: Apple Mail (2.3273) X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrOLMWRmVeSWpSXmKPExsUi2FAYofuOXyzK4P43DYtzl7Mt9hw6ymyx 4t4Gdovjvz4wO7B4LNhU6rF4z0smj+7Z/1gCmKO4bFJSczLLUov07RK4Mo6emMBesMO+Ys1G uQbGeSZdjBwcEgImEu/6Q7sYuTiEBFYzSczqaGTqYuQEi19bdoYdInGIUeL6rmtgCV4BQYkf k++xgDQzC8hLHDwvCxJmFtCS+P6olQWi/gujRNe1Q8wgCWEBcYl3ZzZB2aoSm79/BpvDJqAs sWL+B3YQm1MgTOLukYmMIDYLUM2NxQ2sIIOYBZoYJd59fAW12Ebi5f97jBAbvjNJND7/CTZV BCix48EtNoizZSVuzb7EDGE/Z5PYvsZrAqPwLCSHz0I4fBaSwxcwMq9iFMpNzMzRzcwz0Uss KMhJ1UvOz93ECAr56XZCOxhPrbI6xCjAwajEw7tijmiUEGtiWXFl7iFGaQ4WJXFeUWDECAmk J5akZqemFqQWxReV5qQWH2Jk4uCUamCM0+1d9GZ2VGjp5aY0XwZXlSfrHrWL2vEHaz8JjOH8 JHY/2s1y1hlm53t31uziDbj5lKWwc8qjb4XFk+dptRR94PO4GGH5uifJd/pEt8LgvPK4ki3G PxSn7DHzEGNQCGr7rsEjmT0xb1nn1BAxsc1rehuclX+s6M2ssHZk8Hhi8yij+2HNcSWW4oxE Qy3mouJEAEOUhepaAgAA X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFprCIsWRmVeSWpSXmKPExsUi2FB8Rvcdv1iUwac2NYtzl7Mt9hw6ymyx 4t4Gdovjvz4wO7B4LNhU6rF4z0smj+7Z/1gCmKMMbdLyi8oTi1IUipILSmyVijMSU/LL4y2N jUwdEgsKclL1kvNzlfTtbFJSczLLUov07RIMM46emMBesMO+Ys1GuQbGeSZdjJwcEgImEteW nWHvYuTiEBI4xChxfdc1JpAEr4CgxI/J91i6GDk4mAXkJQ6elwUJMwtoSXx/1MoCUf+FUaLr 2iFmkISwgLjEuzOboGxVic3fP4PNYRNQllgx/wM7iM0pECZx98hERhCbBajmxuIGVpBBzAJN jBLvPr6CWmwj8fL/PUaIDd+ZJBqf/wSbKgKU2PHgFhvE2bISt2ZfYp7AKDALybGzEI6dheTY BYzMqxgFilJzEitN9eChs4kRHPSFETsY/y+zOsQowMGoxMO7Yo5olBBrYllxZS4wNDiYlUR4 gxcAhXhTEiurUovy44tKc1KLDzH6AH0wkVlKNDkfGJF5JfGGxhbGliYWBgYmlmYmOISVxHmP e/JHCQmkJ5akZqemFqQWwYxj4uCUamDsnr7k7J9tDd8l7i+sT3aZ6Bu68Zdv2l+RZ1s5ZYJU vWb/bb+S4LX1Sdc7psuvfVusstw5dZxnzIwwmptxf/eGxflPNY6+8S74o/Jrwu2ma4K1UU8M N3zceJJp36x5rXd6FH4unnRBwUbO+uF1R0fDhqRnRqFtB/T2V+zva/327MkLm42C0xUtlViA 6chQi7moOBEAOoXG66cCAAA= Subject: Re: EDK2 build issues X-BeenThere: edk2-devel@lists.01.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: EDK II Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Nov 2017 23:57:34 -0000 Content-transfer-encoding: 7BIT Content-type: text/plain; CHARSET=US-ASCII > On Nov 16, 2017, at 1:37 PM, Desimone, Nathaniel L wrote: > > Hi Liming, > > I chatted with Aaron on the phone today. The VfrCompiler race condition was discovered using "make -j " (where n > 1). I have filed the bug in Bugzilla: > I think there are a few issues like that in BaseTools. We have a parallel makefile and we had to turn off parallelism when calling the BaseTools. In general the edk2 build system fights against "make -j " since the Python build command is trying to do the threading and invoking make in parallel. Given we ported EDK to parallel build, the Python parallel edk2 build is a lot slower than the EDK make parallel build, even when adjusting for the extra work the edk2 does. I even noticed a 1,000,000 calls to regex in Python during the ... phase of the build. I think the correct short term fix may be to put .NOTPARALLEL: in the BaseTools GNUmakefile given it is constructed in a way that it does not support parallelism. Thanks, Andrew Fish > https://bugzilla.tianocore.org/show_bug.cgi?id=786 > > For the ARCH environment variable, we brainstormed two possible new names HOST_ARCH or BASETOOLS_ARCH. Should I file the ARCH variable request in Bugzilla as well? > > Thanks, > > Nate > > -----Original Message----- > From: edk2-devel [mailto:edk2-devel-bounces@lists.01.org] On Behalf Of Gao, Liming > Sent: Monday, November 13, 2017 5:46 PM > To: Aaron Durbin > Cc: edk2-devel@lists.01.org > Subject: Re: [edk2] EDK2 build issues > > I add my comments. > >> -----Original Message----- >> From: edk2-devel [mailto:edk2-devel-bounces@lists.01.org] On Behalf Of >> Aaron Durbin >> Sent: Tuesday, November 14, 2017 1:38 AM >> To: Gao, Liming >> Cc: edk2-devel@lists.01.org >> Subject: Re: [edk2] EDK2 build issues >> >> On Sun, Nov 12, 2017 at 7:15 PM, Gao, Liming wrote: >>> Yes. BaseTools needs some environments. Before build BaseTools, we need to call edkset.sh to setup them. >>> >>> 1. BaseTools Soure tools are compiled in serial instead of parallel. >>> And, VfrCompiler depends on Anltr tool. So, Anltr is required to be >> built first. I agree that this way takes some time to compile >> BaseTools source. I have one proposal to compile C source tools with multiple thread. I can share my draft patch. >> >> If the depedencies are correctly met in the makefiles then why are >> they built serially? It sounds like you understand the dependencies so >> why aren't those explicitly noted so one can build in parallel? >> > Seemly, VfrCompiler Makefile doesn't list those dependency clearly. Could you submit this issue in bugzillar (https://bugzilla.tianocore.org/)? > Besides, do you use make -j option to enable Parallel Execution? I want to know how to verify it. > >>> 2. BaseTools C source are compiled to the executable files. They run >>> in host machine. Here ARCH is for the executable files those can >> run in host machine. edk2\BaseTools\Source\C\GNUmakefile auto detects ARCH based on 'uname -m' command. >> >> ARCH != host is my point. Why are those being conflated? Or are you >> saying edk2 tools define ARCH to be what the host is? >> > Yes. Edk2 BaseTools C source uses it as host arch. I agree ARCH name is too generic. In your environment, ARCH may be defined for the different purpose. > >>> 3. There are more GCC compiler distribution. We try to use the >>> generic options in GCC tool chain. If you find the option doesn't >>> work >> on your GCC compiler, please report it. And, we also notice >> tools_def.txt is too big to be hard to be maintain. We have one proposal to simplify it. I will send this RFC to edk2 community. >>> >>>> -----Original Message----- >>>> From: edk2-devel [mailto:edk2-devel-bounces@lists.01.org] On Behalf >>>> Of Aaron Durbin >>>> Sent: Saturday, November 11, 2017 6:27 AM >>>> To: edk2-devel@lists.01.org >>>> Subject: [edk2] EDK2 build issues >>>> >>>> Hello, >>>> >>>> Here are some observations I've encountered trying to utilize edk2 >>>> for certain builds. Part of the problem seems to be with implicit >>>> assumptions in how edk2 is used. I'm trying to build things using >>>> edk2 from a clean enviroment on an automated builder. i.e. there >>>> isn't a workspace that exists on one persons computer for the >>>> lifetime of development. >>>> >>>> 1. BaseTools can't build in parallel. The tests are racey which >>>> result in test failures. Because of this one has to build these in >>>> serial instead of in parallel. >>>> >>>> =========================================================== >>>> =========== >>>> FAIL: testHelp (TianoCompress.Tests) >>>> -------------------------------------------------------------------- >>>> -- >>>> Traceback (most recent call last): >>>> File >>>> "/build/zoombini/tmp/portage/sys-boot/fsp-cnl-9999/work/fsp-cnl- >>>> 9999/Edk2/BaseTools/Tests/TianoCompress.py", >>>> line 34, in testHelp >>>> self.assertTrue(result == 0) >>>> AssertionError: False is not true >>>> >>>> =========================================================== >>>> =========== >>>> FAIL: testRandomDataCycles (TianoCompress.Tests) >>>> -------------------------------------------------------------------- >>>> -- >>>> Traceback (most recent call last): >>>> File >>>> "/build/zoombini/tmp/portage/sys-boot/fsp-cnl-9999/work/fsp-cnl- >>>> 9999/Edk2/BaseTools/Tests/TianoCompress.py", >>>> line 65, in testRandomDataCycles >>>> self.compressionTestCycle(data) >>>> File >>>> "/build/zoombini/tmp/portage/sys-boot/fsp-cnl-9999/work/fsp-cnl- >>>> 9999/Edk2/BaseTools/Tests/TianoCompress.py", >>>> line 44, in compressionTestCycle >>>> self.assertTrue(result == 0) >>>> AssertionError: False is not true >>>> >>>> In addition, it seems compilation even breaks trying to build a parser: >>>> >>>> VfrSyntax.cpp:53:1: error: expected class-name before '{' token {^M >>>> ^ >>>> VfrSyntax.cpp: In constructor 'CVfrDLGLexer::CVfrDLGLexer(DLGFileInput*)': >>>> VfrSyntax.cpp:55:36: error: class 'CVfrDLGLexer' does not have any >>>> field named 'VfrLexer' >>>> CVfrDLGLexer (DLGFileInput *F) : VfrLexer (F) {};^M >>>> ^ >>>> >>>> This just slows down builds needing to things serially. >>>> >>>> 2. It appears the BaseTools uses the ARCH environment variable. I'm >>>> not sure of the origins, but ARCH seems like a complete misnomer for >>>> the *host* you are trying to build tools on -- not the target. >>>> Trying to incorporate edk2 builds into a portage environment >>>> effectively breaks because of this as ARCH refers to target >>>> architecture -- not host builder's ARCH. >>>> >>>> 3. This more of an observation, but the tools definition seems to >>>> make quite the leap on how consistent compilers are of a certain version. >>>> e.g. GCC 4.9 can be built with all kinds of default options that >>>> edk2 implicitly assumes are set based on some distribution's default flags? >>>> And in order to extend toolchain support one needs to create a >>>> series of entries associated with a certain family. Barrier to entry >>>> is pretty high in teasing out where to put things in the ~8000 line >>>> tools_def.template. >>>> >>>> Thanks for the consideration. >>>> >>>> -Aaron >>>> _______________________________________________ >>>> edk2-devel mailing list >>>> edk2-devel@lists.01.org >>>> https://lists.01.org/mailman/listinfo/edk2-devel >> _______________________________________________ >> edk2-devel mailing list >> edk2-devel@lists.01.org >> https://lists.01.org/mailman/listinfo/edk2-devel > _______________________________________________ > edk2-devel mailing list > edk2-devel@lists.01.org > https://lists.01.org/mailman/listinfo/edk2-devel > _______________________________________________ > edk2-devel mailing list > edk2-devel@lists.01.org > https://lists.01.org/mailman/listinfo/edk2-devel