From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received-SPF: Pass (sender SPF authorized) identity=mailfrom; client-ip=17.151.62.66; helo=nwk-aaemail-lapp01.apple.com; envelope-from=afish@apple.com; receiver=edk2-devel@lists.01.org Received: from nwk-aaemail-lapp01.apple.com (nwk-aaemail-lapp01.apple.com [17.151.62.66]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ml01.01.org (Postfix) with ESMTPS id 919712116DF9B for ; Fri, 12 Oct 2018 13:29:23 -0700 (PDT) Received: from pps.filterd (nwk-aaemail-lapp01.apple.com [127.0.0.1]) by nwk-aaemail-lapp01.apple.com (8.16.0.22/8.16.0.22) with SMTP id w9CKR0tl003981 for ; Fri, 12 Oct 2018 13:29:23 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=apple.com; h=mime-version : content-transfer-encoding : content-type : sender : from : subject : date : references : to : in-reply-to : message-id; s=20180706; bh=Oqlu1FrNO7uadPF0G2PN7REJXauD7eEcYCKXKc7/aHU=; b=W8uEIqKfLPZEfFuJopXzyTTXQYtFs6ihKwFyrhY2/tsrq56Cyo8IygPl8YGf2eOGOfQ0 sAy27wT2J6Z1XTS+VURz1KG9KYCLoeudRHO5OfwJ/pVeDZXOvS4oeOvZ9A2Tg2Efqp8L dcXAzbMX8TvAwb21Ip2IMc6X6dbt4DlI5kJ/wgzqTOY5v3PVlsuIc7k29DIAGA9vN/0E Z1nF4M7AdDKFMiT4OchRfSM66RVcT0Y6IPMGaz2PVoRRgwhRGPvZs4BJ4UYxYdaJDfWa yv4c0hS71h38dYc403Jswk0UsJvmBJMoT8K/8ec4WK2VJDglDGOg08o1fm4hjDmzNkzg 3A== Received: from mr2-mtap-s02.rno.apple.com (mr2-mtap-s02.rno.apple.com [17.179.226.134]) by nwk-aaemail-lapp01.apple.com with ESMTP id 2mxv09rap2-3 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Fri, 12 Oct 2018 13:29:22 -0700 MIME-version: 1.0 Received: from ma1-mmpp-sz09.apple.com (ma1-mmpp-sz09.apple.com [17.171.128.183]) by mr2-mtap-s02.rno.apple.com (Oracle Communications Messaging Server 8.0.2.3.20180614 64bit (built Jun 14 2018)) with ESMTPS id <0PGI00AN667YK3E0@mr2-mtap-s02.rno.apple.com> for edk2-devel@lists.01.org; Fri, 12 Oct 2018 13:28:46 -0700 (PDT) Received: from process_viserion-daemon.ma1-mmpp-sz09.apple.com by ma1-mmpp-sz09.apple.com (Oracle Communications Messaging Server 8.0.2.3.20180614 64bit (built Jun 14 2018)) id <0PGI00G005Q6DX00@ma1-mmpp-sz09.apple.com> for edk2-devel@lists.01.org; Fri, 12 Oct 2018 13:28:46 -0700 (PDT) X-Va-A: X-Va-T-CD: 242eef930ced4d4f7f74b6d5f343603a X-Va-E-CD: bfb2777e4756679b1ba5aaadc995d351 X-Va-R-CD: 6a64947708146bea7411e6a6f5086601 X-Va-CD: 0 X-Va-ID: d07311e1-5626-4920-b8b6-d4cef6b1a402 X-V-A: X-V-T-CD: 839b4f5fe00976b87887c466bcf0a930 X-V-E-CD: bfb2777e4756679b1ba5aaadc995d351 X-V-R-CD: 6a64947708146bea7411e6a6f5086601 X-V-CD: 0 X-V-ID: 7282e112-dc81-4620-8e44-ce0b6e26e513 Received: from process_milters-daemon.ma1-mmpp-sz09.apple.com by ma1-mmpp-sz09.apple.com (Oracle Communications Messaging Server 8.0.2.3.20180614 64bit (built Jun 14 2018)) id <0PGI00D005I39Z00@ma1-mmpp-sz09.apple.com> for edk2-devel@lists.01.org; Fri, 12 Oct 2018 13:28:46 -0700 (PDT) X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2018-10-12_13:,, signatures=0 X-Proofpoint-Scanner-Instance: ma-grpmailp-qapp25.corp.apple.com-10000_instance1 Received: from [17.234.241.37] (unknown [17.234.241.37]) by ma1-mmpp-sz09.apple.com (Oracle Communications Messaging Server 8.0.2.3.20180614 64bit (built Jun 14 2018)) with ESMTPSA id <0PGI000NZ67VS420@ma1-mmpp-sz09.apple.com> for edk2-devel@lists.01.org; Fri, 12 Oct 2018 13:28:46 -0700 (PDT) Sender: afish@apple.com From: Andrew Fish Date: Fri, 12 Oct 2018 13:28:28 -0700 References: To: edk2-devel-01 In-reply-to: Message-id: <23198583-75D4-4BF1-AAA2-6F271EBE691B@apple.com> X-Mailer: Apple Mail (2.3445.6.18) X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:, , definitions=2018-10-12_13:, , signatures=0 Subject: Re: TianoCore Community Meeting Minutes X-BeenThere: edk2-devel@lists.01.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: EDK II Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 12 Oct 2018 20:29:23 -0000 Content-transfer-encoding: 7BIT Content-type: text/plain; CHARSET=US-ASCII > On Oct 11, 2018, at 10:43 AM, stephano wrote: > > Thank you all for a great set of meetings! > > This is an overview of the topics discussed and the tasks that were assigned. Please feel free to send me any questions or comments. > ... > Improvements to the Build Process > -------------------------------- > We would like to gather a list of concrete specific proposals. Nate mentioned that there will always be some specialized tooling because of the nature of BIOS. Shawn mentioned that we need to keep in mind that making development easier for current developers should be a priority over making the processes easier for newcomers. I will send out emails asking for specific feedback on some of these topics: > > -Which toolchains are being used, which are validated, and which are known to create reproducible builds. > -When toolchains are known to be orphaned, should they be archived or simply removed. > -Could we add some kconfig-like tool that allows introspection into what type of builds are available. > -How can we better track the code quality of BaseTools and the current build system on the whole. Should we add a "requires documentation change" flag to BZ so that it will be easier to compile a list of required doc changes. > Stephano, I'll help start the conversation on the build. 1) Making the process easier for newcomers. a) Getting it to build the 1st time I think we could aspire to a more Unix like flow: git clone ./configure make ovmf make ovmf.run The ./config step would automate resolving dependencies. Things like figuring out what compiler is installed, what extra tools need to be installed etc. It could be interactive? make would build all the build tools, and do what ever local config is required to get the build working. Then build the requested target. make .run is more of an optional step. It could do things like launch Ovmf with your new ROM you just constructed. b) Debugging failures. Better quality error messages. Making the build.log from the build command a default setting(or have an abstracted defined database that can be queried), and add documentation on how to use it. Maybe make the output easier to consume? c) Making changes. That can be a phase 2. 2) Build performance, complexity, and maintainability. I'm about to complain about the build system, but I was around when it was first invented so I'm to blame as much as anyone.... We have way too much custom Python that is very complex, hard to maintain, and slow. One thing we did better on the EDK was break up the build tools into smaller chunks (maybe too small in some cases) and I always found I could figure things out, but then again all the code was C. Another interesting data point is we added parallel build, via makefiles, to the EDK. When we moved from the EDK to the edk2 we noticed the builds slowed down a lot. I did some profiling on the edk2 build and I noticed 1,000,000 calls to regex (part of that long ..... sequence on a build of a complex platform). It turns out we have custom Python code that is generating the makefile dependencies so it was scanning all the C code. The other issue I see is build parallelism is controlled from the Python code. It also seems the edk2 build system is not a complete solution and a lot of full platform implementations have shell scripts or batch files, that wrap the calls to build. So I'd like to see a build system: 1) Start with a top level makefile in the root of the tree. 2) Replaces custom batch or shell scripts with makefiles to handle pre and post build tasks. 3) Use a tool to construct a parser for the ekd2 build files (INF, DSC, DEC). Use this data to construct the makefiles and build a database for other build components to use. Chris Roberts has messed around and built an Earley parser using the EBNF in the build documents. There are some problematic issues in the grammar, but some grammar streamlining might make it possible to use tooling to construct a parser vs. writing one from scratch in Python. 4) Move to using the compiler to generate dependencies. gcc/clang support this via -M*. I think it may be a little more complex for NMAKE but there is a /showIncludes compiler flag. 5) Have the build tool terminate after parallel makefile generation 6) Top level makefile can invoke parallel build on generated makefiles. 7) Invoke build tool to generate FVs and FDs from makefile. 8) Invoke custom post processing steps in the makefile. Thanks, Andrew Fish