From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received-SPF: Pass (sender SPF authorized) identity=mailfrom; client-ip=17.171.2.33; helo=mail-in23.apple.com; envelope-from=afish@apple.com; receiver=edk2-devel@lists.01.org Received: from mail-in23.apple.com (mail-out23.apple.com [17.171.2.33]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ml01.01.org (Postfix) with ESMTPS id E774E2211B43C for ; Mon, 27 Nov 2017 08:14:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; d=apple.com; s=mailout2048s; c=relaxed/simple; q=dns/txt; i=@apple.com; t=1511799524; h=From:Sender:Reply-To:Subject:Date:Message-id:To:Cc:MIME-version:Content-type: Content-transfer-encoding:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:In-reply-to:References:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=ghQY+NxFgWflTheBaapTdkF9C3cJjntCL6LH/1c2IQc=; b=FVKu9KEq8iyYj4KwFU1DqHiDQ4j5/5B+8gq8cfqHFxrasbMj5fVd854nkWmumty7 ngUcmCO3OGMt+khhaaf9Hah9U6ZOea3eDjv9lqkgF+ieMMthBBNF9WcOXBe+SdXE wtrVv33mTeA+zwgjTPc5eI6kN4BEuMIBLsN5wpgdfwB7Uq64GQev+ayNpj9bAX0R C2jRCuKEOMJodHFJNOkvnUJ//8/f4LwZhBwvWuOzhPlq0eKH/Xm+uD1HDDAh4/fe MgDNE/X9NPFy/ZYn4Z6L1Fk/FvIu4yIKIfxr7uCT3LcB8gzlUm5MLPTcaKkVUPiK 48LCm6kN1eh2ycEoWdPcGw==; Received: from relay22.apple.com (relay22.apple.com [17.171.128.103]) (using TLS with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mail-in23.apple.com (Apple Secure Mail Relay) with SMTP id 10.F1.29340.4EA3C1A5; Mon, 27 Nov 2017 08:18:44 -0800 (PST) X-AuditID: 11ab0217-949ff7000000729c-d5-5a1c3ae4ffcc Received: from ma1-mmpp-sz10.apple.com (ma1-mmpp-sz10.apple.com [17.171.128.150]) by relay22.apple.com (Apple SCV relay) with SMTP id AE.7F.23356.4EA3C1A5; Mon, 27 Nov 2017 08:18:44 -0800 (PST) MIME-version: 1.0 Received: from [17.234.162.130] by ma1-mmpp-sz10.apple.com (Oracle Communications Messaging Server 8.0.2.1.20171102 64bit (built Nov 2 2017)) with ESMTPSA id <0P0300MMX3Z1ZN10@ma1-mmpp-sz10.apple.com>; Mon, 27 Nov 2017 08:18:44 -0800 (PST) Sender: afish@apple.com From: Andrew Fish In-reply-to: <4A89E2EF3DFEDB4C8BFDE51014F606A14E1822EE@SHSMSX104.ccr.corp.intel.com> Date: Mon, 27 Nov 2017 08:18:48 -0800 Cc: "Desimone, Nathaniel L" , Aaron Durbin , "edk2-devel@lists.01.org" Message-id: <1BBD2474-00CD-4CCC-B30A-A3141EDE38CA@apple.com> References: <4A89E2EF3DFEDB4C8BFDE51014F606A14E17CC73@SHSMSX104.ccr.corp.intel.com> <4A89E2EF3DFEDB4C8BFDE51014F606A14E17D6C3@SHSMSX104.ccr.corp.intel.com> <02A34F284D1DA44BB705E61F7180EF0A97FB2D01@ORSMSX114.amr.corp.intel.com> <10CEC444-4BE6-4A13-BF3E-D5ABE4C6971E@apple.com> <4A89E2EF3DFEDB4C8BFDE51014F606A14E1822EE@SHSMSX104.ccr.corp.intel.com> To: "Gao, Liming" X-Mailer: Apple Mail (2.3273) X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFjrJLMWRmVeSWpSXmKPExsUiuLohXfeJlUyUwYQWUYtzl7Mt9hw6ymyx 4t4Gdovjvz4wO7B4LNhU6rF4z0smj+7Z/1gCmKO4bFJSczLLUov07RK4MiatbmQvmO9V8Wqz RwPjNtsuRk4OCQETiY2ftrB1MXJxCAmsZ5J4M+EUC0zixIu9rBCJw4wSr89uZwJJ8AoISvyY fA+oiIODWUBe4uB5WZAws4CWxPdHrWC9QgLfGCVmv9QAsYUFxCXendnEDGGrSmz+/hlsDJuA ssSK+R/YQWxOgTCJFedusoHYLEA123sPMIHsZRaYwSix79wORoi9NhI9pw8zQhz0hVli9adN YNtEBDQkHt77zQxxtazErdmXmEGKJARes0nsXXiZaQKj8Cwkh89COHwWksMXMDKvYhTOTczM 0c3MMzLWSywoyEnVS87P3cQIDnsm8R2Mn18bHmIU4GBU4uFV8JGOEmJNLCuuzD3EKM3BoiTO u/ajVJSQQHpiSWp2ampBalF8UWlOavEhRiYOTqkGxtapaxL5Vxyx2ed9LkLja2P6At7DV51W SYke/de6cfN0x9X/F9rMrQ++yTTl4IKVJ9k7zMSmXpytzH8+3sDmyYs9L9341+b1HT7/y29K lsKyNJGcV7U//ydKshj3tGq87f3TPkdqp5D8nhsMXx/ynXkjaxTo2W1/3WfGqplsn+eEMiUU JiwJ8VNiKc5INNRiLipOBACsfAsAXAIAAA== X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFprCIsWRmVeSWpSXmKPExsUiuLphmu4TK5kog1vrBS3OXc622HPoKLPF insb2C2O//rA7MDisWBTqcfiPS+ZPLpn/2MJYI4ytEnLLypPLEpRKEouKLFVKs5ITMkvj7c0 NjJ1SCwoyEnVS87PVdK3s0lJzcksSy3St0swzJi0upG9YL5XxavNHg2M22y7GDk5JARMJE68 2MvaxcjFISRwmFHi9dntTCAJXgFBiR+T77F0MXJwMAvISxw8LwsSZhbQkvj+qJUFxBYS+MYo MfulBogtLCAu8e7MJmYIW1Vi8/fPYGPYBJQlVsz/wA5icwqESaw4d5MNxGYBqtnee4AJZC+z wAxGiX3ndjBC7LWR6Dl9mBHioC/MEqs/bQLbJiKgIfHw3m9miKtlJW7NvsQ8gVFgFpJbZyHc OgvJrQsYmVcxChal5iRWGhnpwUNnEyMk6NN3MB65aXaIUYCDUYmHV8FHOkqINbGsuDL3EKME B7OSCK/sQ6AQb0piZVVqUX58UWlOavEhRh+gFyYyS4km5wMjMq8k3tDYwtjSxMLAwMLCyBSH sJI4b/Eq8SghgfTEktTs1NSC1CKYcUwcnFINjDzJKiqdP3s1Ntasmeax1H/P0eI+gZmR0iIW EerubUe/ZSglbXw6dckvg9Zec9uudiGdtdtnz5PLYq2ZtC9Dc0XjPNl57ZcLNmtu/mH7VThP f/vDHVlhXEWyliKFbVPbF/kJHKvjYu+5HFfmrmKavuPgAbVtkkcymJXED0bcN7p8Yhrvy6mn lViA6chQi7moOBEASekCx6cCAAA= Subject: Re: EDK2 build issues X-BeenThere: edk2-devel@lists.01.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: EDK II Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Nov 2017 16:14:24 -0000 Content-transfer-encoding: 7BIT Content-type: text/plain; CHARSET=US-ASCII Liming, We have a top level GNUmakefile vs. having script files to build targets. Thus building BaseTools, or using build are subtasks of this higher level makefile. Looks like you have added some fixes to the BaseTools! Thanks, Andrew Fish > On Nov 22, 2017, at 9:28 PM, Gao, Liming wrote: > > Andrew: > To compile BaseTools C tools is a separate step. It happens before build. I review BaseTools C tool top GNUmakefile. It can be enhanced to support make -j N that has the better performance. I will provide my patch for code review. > > Thanks > Liming >> -----Original Message----- >> From: afish@apple.com [mailto:afish@apple.com] >> Sent: Thursday, November 23, 2017 8:02 AM >> To: Desimone, Nathaniel L >> Cc: Gao, Liming ; Aaron Durbin >> ; edk2-devel@lists.01.org >> Subject: Re: [edk2] EDK2 build issues >> >> >>> On Nov 16, 2017, at 1:37 PM, Desimone, Nathaniel L >> wrote: >>> >>> Hi Liming, >>> >>> I chatted with Aaron on the phone today. The VfrCompiler race condition >> was discovered using "make -j " (where n > 1). I have filed the bug in >> Bugzilla: >>> >> >> I think there are a few issues like that in BaseTools. We have a parallel >> makefile and we had to turn off parallelism when calling the BaseTools. >> >> In general the edk2 build system fights against "make -j " since the Python >> build command is trying to do the threading and invoking make in parallel. >> Given we ported EDK to parallel build, the Python parallel edk2 build is a lot >> slower than the EDK make parallel build, even when adjusting for the extra >> work the edk2 does. >> >> I even noticed a 1,000,000 calls to regex in Python during the ... phase of the >> build. >> >> I think the correct short term fix may be to put .NOTPARALLEL: in the >> BaseTools GNUmakefile given it is constructed in a way that it does not >> support parallelism. >> >> Thanks, >> >> Andrew Fish >> >> >> >>> https://bugzilla.tianocore.org/show_bug.cgi?id=786 >>> >>> For the ARCH environment variable, we brainstormed two possible new >> names HOST_ARCH or BASETOOLS_ARCH. Should I file the ARCH variable >> request in Bugzilla as well? >>> >>> Thanks, >>> >>> Nate >>> >>> -----Original Message----- >>> From: edk2-devel [mailto:edk2-devel-bounces@lists.01.org] On Behalf Of >> Gao, Liming >>> Sent: Monday, November 13, 2017 5:46 PM >>> To: Aaron Durbin >>> Cc: edk2-devel@lists.01.org >>> Subject: Re: [edk2] EDK2 build issues >>> >>> I add my comments. >>> >>>> -----Original Message----- >>>> From: edk2-devel [mailto:edk2-devel-bounces@lists.01.org] On Behalf Of >>>> Aaron Durbin >>>> Sent: Tuesday, November 14, 2017 1:38 AM >>>> To: Gao, Liming >>>> Cc: edk2-devel@lists.01.org >>>> Subject: Re: [edk2] EDK2 build issues >>>> >>>> On Sun, Nov 12, 2017 at 7:15 PM, Gao, Liming >> wrote: >>>>> Yes. BaseTools needs some environments. Before build BaseTools, we >> need to call edkset.sh to setup them. >>>>> >>>>> 1. BaseTools Soure tools are compiled in serial instead of parallel. >>>>> And, VfrCompiler depends on Anltr tool. So, Anltr is required to be >>>> built first. I agree that this way takes some time to compile >>>> BaseTools source. I have one proposal to compile C source tools with >> multiple thread. I can share my draft patch. >>>> >>>> If the depedencies are correctly met in the makefiles then why are >>>> they built serially? It sounds like you understand the dependencies so >>>> why aren't those explicitly noted so one can build in parallel? >>>> >>> Seemly, VfrCompiler Makefile doesn't list those dependency clearly. Could >> you submit this issue in bugzillar (https://bugzilla.tianocore.org/)? >>> Besides, do you use make -j option to enable Parallel Execution? I want to >> know how to verify it. >>> >>>>> 2. BaseTools C source are compiled to the executable files. They run >>>>> in host machine. Here ARCH is for the executable files those can >>>> run in host machine. edk2\BaseTools\Source\C\GNUmakefile auto detects >> ARCH based on 'uname -m' command. >>>> >>>> ARCH != host is my point. Why are those being conflated? Or are you >>>> saying edk2 tools define ARCH to be what the host is? >>>> >>> Yes. Edk2 BaseTools C source uses it as host arch. I agree ARCH name is too >> generic. In your environment, ARCH may be defined for the different purpose. >>> >>>>> 3. There are more GCC compiler distribution. We try to use the >>>>> generic options in GCC tool chain. If you find the option doesn't >>>>> work >>>> on your GCC compiler, please report it. And, we also notice >>>> tools_def.txt is too big to be hard to be maintain. We have one proposal to >> simplify it. I will send this RFC to edk2 community. >>>>> >>>>>> -----Original Message----- >>>>>> From: edk2-devel [mailto:edk2-devel-bounces@lists.01.org] On Behalf >>>>>> Of Aaron Durbin >>>>>> Sent: Saturday, November 11, 2017 6:27 AM >>>>>> To: edk2-devel@lists.01.org >>>>>> Subject: [edk2] EDK2 build issues >>>>>> >>>>>> Hello, >>>>>> >>>>>> Here are some observations I've encountered trying to utilize edk2 >>>>>> for certain builds. Part of the problem seems to be with implicit >>>>>> assumptions in how edk2 is used. I'm trying to build things using >>>>>> edk2 from a clean enviroment on an automated builder. i.e. there >>>>>> isn't a workspace that exists on one persons computer for the >>>>>> lifetime of development. >>>>>> >>>>>> 1. BaseTools can't build in parallel. The tests are racey which >>>>>> result in test failures. Because of this one has to build these in >>>>>> serial instead of in parallel. >>>>>> >>>>>> >> =========================================================== >>>>>> =========== >>>>>> FAIL: testHelp (TianoCompress.Tests) >>>>>> -------------------------------------------------------------------- >>>>>> -- >>>>>> Traceback (most recent call last): >>>>>> File >>>>>> "/build/zoombini/tmp/portage/sys-boot/fsp-cnl-9999/work/fsp-cnl- >>>>>> 9999/Edk2/BaseTools/Tests/TianoCompress.py", >>>>>> line 34, in testHelp >>>>>> self.assertTrue(result == 0) >>>>>> AssertionError: False is not true >>>>>> >>>>>> >> =========================================================== >>>>>> =========== >>>>>> FAIL: testRandomDataCycles (TianoCompress.Tests) >>>>>> -------------------------------------------------------------------- >>>>>> -- >>>>>> Traceback (most recent call last): >>>>>> File >>>>>> "/build/zoombini/tmp/portage/sys-boot/fsp-cnl-9999/work/fsp-cnl- >>>>>> 9999/Edk2/BaseTools/Tests/TianoCompress.py", >>>>>> line 65, in testRandomDataCycles >>>>>> self.compressionTestCycle(data) >>>>>> File >>>>>> "/build/zoombini/tmp/portage/sys-boot/fsp-cnl-9999/work/fsp-cnl- >>>>>> 9999/Edk2/BaseTools/Tests/TianoCompress.py", >>>>>> line 44, in compressionTestCycle >>>>>> self.assertTrue(result == 0) >>>>>> AssertionError: False is not true >>>>>> >>>>>> In addition, it seems compilation even breaks trying to build a parser: >>>>>> >>>>>> VfrSyntax.cpp:53:1: error: expected class-name before '{' token {^M >>>>>> ^ >>>>>> VfrSyntax.cpp: In constructor >> 'CVfrDLGLexer::CVfrDLGLexer(DLGFileInput*)': >>>>>> VfrSyntax.cpp:55:36: error: class 'CVfrDLGLexer' does not have any >>>>>> field named 'VfrLexer' >>>>>> CVfrDLGLexer (DLGFileInput *F) : VfrLexer (F) {};^M >>>>>> ^ >>>>>> >>>>>> This just slows down builds needing to things serially. >>>>>> >>>>>> 2. It appears the BaseTools uses the ARCH environment variable. I'm >>>>>> not sure of the origins, but ARCH seems like a complete misnomer for >>>>>> the *host* you are trying to build tools on -- not the target. >>>>>> Trying to incorporate edk2 builds into a portage environment >>>>>> effectively breaks because of this as ARCH refers to target >>>>>> architecture -- not host builder's ARCH. >>>>>> >>>>>> 3. This more of an observation, but the tools definition seems to >>>>>> make quite the leap on how consistent compilers are of a certain version. >>>>>> e.g. GCC 4.9 can be built with all kinds of default options that >>>>>> edk2 implicitly assumes are set based on some distribution's default >> flags? >>>>>> And in order to extend toolchain support one needs to create a >>>>>> series of entries associated with a certain family. Barrier to entry >>>>>> is pretty high in teasing out where to put things in the ~8000 line >>>>>> tools_def.template. >>>>>> >>>>>> Thanks for the consideration. >>>>>> >>>>>> -Aaron >>>>>> _______________________________________________ >>>>>> edk2-devel mailing list >>>>>> edk2-devel@lists.01.org >>>>>> https://lists.01.org/mailman/listinfo/edk2-devel >>>> _______________________________________________ >>>> edk2-devel mailing list >>>> edk2-devel@lists.01.org >>>> https://lists.01.org/mailman/listinfo/edk2-devel >>> _______________________________________________ >>> edk2-devel mailing list >>> edk2-devel@lists.01.org >>> https://lists.01.org/mailman/listinfo/edk2-devel >>> _______________________________________________ >>> edk2-devel mailing list >>> edk2-devel@lists.01.org >>> https://lists.01.org/mailman/listinfo/edk2-devel >