From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received-SPF: Pass (sender SPF authorized) identity=mailfrom; client-ip=192.55.52.88; helo=mga01.intel.com; envelope-from=liming.gao@intel.com; receiver=edk2-devel@lists.01.org Received: from mga01.intel.com (mga01.intel.com [192.55.52.88]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ml01.01.org (Postfix) with ESMTPS id 732142035BB19 for ; Wed, 22 Nov 2017 21:24:41 -0800 (PST) Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga101.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 22 Nov 2017 21:28:58 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.44,440,1505804400"; d="scan'208";a="1247526237" Received: from fmsmsx107.amr.corp.intel.com ([10.18.124.205]) by fmsmga002.fm.intel.com with ESMTP; 22 Nov 2017 21:28:58 -0800 Received: from shsmsx103.ccr.corp.intel.com (10.239.4.69) by fmsmsx107.amr.corp.intel.com (10.18.124.205) with Microsoft SMTP Server (TLS) id 14.3.319.2; Wed, 22 Nov 2017 21:28:56 -0800 Received: from shsmsx104.ccr.corp.intel.com ([169.254.5.152]) by SHSMSX103.ccr.corp.intel.com ([169.254.4.213]) with mapi id 14.03.0319.002; Thu, 23 Nov 2017 13:28:55 +0800 From: "Gao, Liming" To: "afish@apple.com" , "Desimone, Nathaniel L" CC: Aaron Durbin , "edk2-devel@lists.01.org" Thread-Topic: [edk2] EDK2 build issues Thread-Index: AQHTWnMIgiNl8WcTQ0u+gy17uoE/wKMQH5LwgAHxlYCAAQU14IAD9OcAgAmWP4CAAOCD4A== Date: Thu, 23 Nov 2017 05:28:54 +0000 Message-ID: <4A89E2EF3DFEDB4C8BFDE51014F606A14E1822EE@SHSMSX104.ccr.corp.intel.com> References: <4A89E2EF3DFEDB4C8BFDE51014F606A14E17CC73@SHSMSX104.ccr.corp.intel.com> <4A89E2EF3DFEDB4C8BFDE51014F606A14E17D6C3@SHSMSX104.ccr.corp.intel.com> <02A34F284D1DA44BB705E61F7180EF0A97FB2D01@ORSMSX114.amr.corp.intel.com> <10CEC444-4BE6-4A13-BF3E-D5ABE4C6971E@apple.com> In-Reply-To: <10CEC444-4BE6-4A13-BF3E-D5ABE4C6971E@apple.com> Accept-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.239.127.40] MIME-Version: 1.0 Subject: Re: EDK2 build issues X-BeenThere: edk2-devel@lists.01.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: EDK II Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 23 Nov 2017 05:24:41 -0000 Content-Language: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Andrew: To compile BaseTools C tools is a separate step. It happens before build.= I review BaseTools C tool top GNUmakefile. It can be enhanced to support m= ake -j N that has the better performance. I will provide my patch for code = review.=20 Thanks Liming >-----Original Message----- >From: afish@apple.com [mailto:afish@apple.com] >Sent: Thursday, November 23, 2017 8:02 AM >To: Desimone, Nathaniel L >Cc: Gao, Liming ; Aaron Durbin >; edk2-devel@lists.01.org >Subject: Re: [edk2] EDK2 build issues > > >> On Nov 16, 2017, at 1:37 PM, Desimone, Nathaniel L > wrote: >> >> Hi Liming, >> >> I chatted with Aaron on the phone today. The VfrCompiler race condition >was discovered using "make -j " (where n > 1). I have filed the bug in >Bugzilla: >> > >I think there are a few issues like that in BaseTools. We have a parallel >makefile and we had to turn off parallelism when calling the BaseTools. > >In general the edk2 build system fights against "make -j " since the Py= thon >build command is trying to do the threading and invoking make in parallel. >Given we ported EDK to parallel build, the Python parallel edk2 build is a= lot >slower than the EDK make parallel build, even when adjusting for the extra >work the edk2 does. > >I even noticed a 1,000,000 calls to regex in Python during the ... phase o= f the >build. > >I think the correct short term fix may be to put .NOTPARALLEL: in the >BaseTools GNUmakefile given it is constructed in a way that it does not >support parallelism. > >Thanks, > >Andrew Fish > > > >> https://bugzilla.tianocore.org/show_bug.cgi?id=3D786 >> >> For the ARCH environment variable, we brainstormed two possible new >names HOST_ARCH or BASETOOLS_ARCH. Should I file the ARCH variable >request in Bugzilla as well? >> >> Thanks, >> >> Nate >> >> -----Original Message----- >> From: edk2-devel [mailto:edk2-devel-bounces@lists.01.org] On Behalf Of >Gao, Liming >> Sent: Monday, November 13, 2017 5:46 PM >> To: Aaron Durbin >> Cc: edk2-devel@lists.01.org >> Subject: Re: [edk2] EDK2 build issues >> >> I add my comments. >> >>> -----Original Message----- >>> From: edk2-devel [mailto:edk2-devel-bounces@lists.01.org] On Behalf Of >>> Aaron Durbin >>> Sent: Tuesday, November 14, 2017 1:38 AM >>> To: Gao, Liming >>> Cc: edk2-devel@lists.01.org >>> Subject: Re: [edk2] EDK2 build issues >>> >>> On Sun, Nov 12, 2017 at 7:15 PM, Gao, Liming >wrote: >>>> Yes. BaseTools needs some environments. Before build BaseTools, we >need to call edkset.sh to setup them. >>>> >>>> 1. BaseTools Soure tools are compiled in serial instead of parallel. >>>> And, VfrCompiler depends on Anltr tool. So, Anltr is required to be >>> built first. I agree that this way takes some time to compile >>> BaseTools source. I have one proposal to compile C source tools with >multiple thread. I can share my draft patch. >>> >>> If the depedencies are correctly met in the makefiles then why are >>> they built serially? It sounds like you understand the dependencies so >>> why aren't those explicitly noted so one can build in parallel? >>> >> Seemly, VfrCompiler Makefile doesn't list those dependency clearly. Coul= d >you submit this issue in bugzillar (https://bugzilla.tianocore.org/)? >> Besides, do you use make -j option to enable Parallel Execution? I want = to >know how to verify it. >> >>>> 2. BaseTools C source are compiled to the executable files. They run >>>> in host machine. Here ARCH is for the executable files those can >>> run in host machine. edk2\BaseTools\Source\C\GNUmakefile auto detects >ARCH based on 'uname -m' command. >>> >>> ARCH !=3D host is my point. Why are those being conflated? Or are you >>> saying edk2 tools define ARCH to be what the host is? >>> >> Yes. Edk2 BaseTools C source uses it as host arch. I agree ARCH name is = too >generic. In your environment, ARCH may be defined for the different purpos= e. >> >>>> 3. There are more GCC compiler distribution. We try to use the >>>> generic options in GCC tool chain. If you find the option doesn't >>>> work >>> on your GCC compiler, please report it. And, we also notice >>> tools_def.txt is too big to be hard to be maintain. We have one proposa= l to >simplify it. I will send this RFC to edk2 community. >>>> >>>>> -----Original Message----- >>>>> From: edk2-devel [mailto:edk2-devel-bounces@lists.01.org] On Behalf >>>>> Of Aaron Durbin >>>>> Sent: Saturday, November 11, 2017 6:27 AM >>>>> To: edk2-devel@lists.01.org >>>>> Subject: [edk2] EDK2 build issues >>>>> >>>>> Hello, >>>>> >>>>> Here are some observations I've encountered trying to utilize edk2 >>>>> for certain builds. Part of the problem seems to be with implicit >>>>> assumptions in how edk2 is used. I'm trying to build things using >>>>> edk2 from a clean enviroment on an automated builder. i.e. there >>>>> isn't a workspace that exists on one persons computer for the >>>>> lifetime of development. >>>>> >>>>> 1. BaseTools can't build in parallel. The tests are racey which >>>>> result in test failures. Because of this one has to build these in >>>>> serial instead of in parallel. >>>>> >>>>> >=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >>>>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >>>>> FAIL: testHelp (TianoCompress.Tests) >>>>> -------------------------------------------------------------------- >>>>> -- >>>>> Traceback (most recent call last): >>>>> File >>>>> "/build/zoombini/tmp/portage/sys-boot/fsp-cnl-9999/work/fsp-cnl- >>>>> 9999/Edk2/BaseTools/Tests/TianoCompress.py", >>>>> line 34, in testHelp >>>>> self.assertTrue(result =3D=3D 0) >>>>> AssertionError: False is not true >>>>> >>>>> >=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >>>>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >>>>> FAIL: testRandomDataCycles (TianoCompress.Tests) >>>>> -------------------------------------------------------------------- >>>>> -- >>>>> Traceback (most recent call last): >>>>> File >>>>> "/build/zoombini/tmp/portage/sys-boot/fsp-cnl-9999/work/fsp-cnl- >>>>> 9999/Edk2/BaseTools/Tests/TianoCompress.py", >>>>> line 65, in testRandomDataCycles >>>>> self.compressionTestCycle(data) >>>>> File >>>>> "/build/zoombini/tmp/portage/sys-boot/fsp-cnl-9999/work/fsp-cnl- >>>>> 9999/Edk2/BaseTools/Tests/TianoCompress.py", >>>>> line 44, in compressionTestCycle >>>>> self.assertTrue(result =3D=3D 0) >>>>> AssertionError: False is not true >>>>> >>>>> In addition, it seems compilation even breaks trying to build a parse= r: >>>>> >>>>> VfrSyntax.cpp:53:1: error: expected class-name before '{' token {^M >>>>> ^ >>>>> VfrSyntax.cpp: In constructor >'CVfrDLGLexer::CVfrDLGLexer(DLGFileInput*)': >>>>> VfrSyntax.cpp:55:36: error: class 'CVfrDLGLexer' does not have any >>>>> field named 'VfrLexer' >>>>> CVfrDLGLexer (DLGFileInput *F) : VfrLexer (F) {};^M >>>>> ^ >>>>> >>>>> This just slows down builds needing to things serially. >>>>> >>>>> 2. It appears the BaseTools uses the ARCH environment variable. I'm >>>>> not sure of the origins, but ARCH seems like a complete misnomer for >>>>> the *host* you are trying to build tools on -- not the target. >>>>> Trying to incorporate edk2 builds into a portage environment >>>>> effectively breaks because of this as ARCH refers to target >>>>> architecture -- not host builder's ARCH. >>>>> >>>>> 3. This more of an observation, but the tools definition seems to >>>>> make quite the leap on how consistent compilers are of a certain vers= ion. >>>>> e.g. GCC 4.9 can be built with all kinds of default options that >>>>> edk2 implicitly assumes are set based on some distribution's default >flags? >>>>> And in order to extend toolchain support one needs to create a >>>>> series of entries associated with a certain family. Barrier to entry >>>>> is pretty high in teasing out where to put things in the ~8000 line >>>>> tools_def.template. >>>>> >>>>> Thanks for the consideration. >>>>> >>>>> -Aaron >>>>> _______________________________________________ >>>>> edk2-devel mailing list >>>>> edk2-devel@lists.01.org >>>>> https://lists.01.org/mailman/listinfo/edk2-devel >>> _______________________________________________ >>> edk2-devel mailing list >>> edk2-devel@lists.01.org >>> https://lists.01.org/mailman/listinfo/edk2-devel >> _______________________________________________ >> edk2-devel mailing list >> edk2-devel@lists.01.org >> https://lists.01.org/mailman/listinfo/edk2-devel >> _______________________________________________ >> edk2-devel mailing list >> edk2-devel@lists.01.org >> https://lists.01.org/mailman/listinfo/edk2-devel