Re: [PATCH v2 1/1] BaseTools: Build ASL files before C files

["PierreGondois" <pierre.gondois@arm.com>]

Hello everyone,
Do you have any input on the patch?

Regards,
Pierre

-----Original Message-----
From: PierreGondois <pierre.gondois@arm.com> 
Sent: 14 January 2020 18:46
To: devel@edk2.groups.io
Cc: Pierre Gondois <Pierre.Gondois@arm.com>; ard.biesheuvel@linaro.org; bob.c.feng@intel.com; liming.gao@intel.com; Sami Mujawar <Sami.Mujawar@arm.com>; nd <nd@arm.com>
Subject: [PATCH v2 1/1] BaseTools: Build ASL files before C files

From: Pierre Gondois <pierre.gondois@arm.com>

The dependencies for C files are satisfied by the build system.
However, there are use cases where source files with different languages are inter-dependent. The EDKII build framework currently doesn't have options to specify such dependencies.
E.g. It may be necessary to specify the build order between
     ASL files and C files. The use case being that the AML
     blob generated by compiling the ASL files is loaded and
     parsed by some C code.

This patch allows to describe dependencies between files listed in the '[Sources]' section of '.inf' files. The list of source files to build prior are colon separated (':'), e.g.:
[Sources]
  FileName1.X
  FileName2.Y : FileName1.X
  FileName3.Z : FileName1.X : FileName3.Z

In the example above:
  * FileName1.X will be built prior to FileName2.Y.
  * FileName1.X and FileName2.Y will be built prior to
    FileName3.Z.

This does not affect the file specific build options, which are pipe separated, e.g.:
  FileName2.Y | GCC : FileName1.X

The list of colon separated files must be part of the module, i.e. they must be present in the [Sources] section.

Their file extension must be described in the BaseTools/Conf/build_rule.template file, and generate an output file, i.e. have a non-empty '<OutputFile>' section.

Declaring a dependency in the [Sources] sections leads to the creation of makefile rules between these files in the build. The makefile rule is between the generated files.
E.g.:
  FileName1.X generates FileName1.objx
  FileName2.Y generates FileName2.objy
  Then
    FileName2.Y : FileName1.X
  will create the following makefile rule:
    FileName2.objy : FileName1.objx

INF Specification updates:
s3.2.1, "Common Definitions":
  <SrcDependencySeperator> ::= ":"
  <DepS>                   ::= <TS> <SrcDependencySeperator> <TS>

s2.5, "[Sources] Section":
  To describe a build order dependency between source
  files, specify the source files to build prior by
  listing them, colon separated.
  <opt3> ::= <FS> [<FeatureFlagExpress>] [<opt4>]*
  <opt4> ::= <DepS> [FileNameDependency]

Signed-off-by: Pierre Gondois <pierre.gondois@arm.com>
---

The changes can be seen at https://github.com/PierreARM/edk2/tree/676_build_asl_first_v2

Notes:
    v2:
      - Enable file dependencies in .if files [Pierre]

 BaseTools/Source/Python/AutoGen/BuildEngine.py      |  6 ++++
 BaseTools/Source/Python/AutoGen/GenMake.py          |  7 +++++
 BaseTools/Source/Python/AutoGen/ModuleAutoGen.py    | 17 +++++++++++
 BaseTools/Source/Python/Common/DataType.py          |  3 +-
 BaseTools/Source/Python/Common/Misc.py              |  3 ++
 BaseTools/Source/Python/Workspace/InfBuildData.py   | 32 ++++++++++++++++++--
 BaseTools/Source/Python/Workspace/MetaFileParser.py | 18 +++++++++--
 7 files changed, 81 insertions(+), 5 deletions(-)

diff --git a/BaseTools/Source/Python/AutoGen/BuildEngine.py b/BaseTools/Source/Python/AutoGen/BuildEngine.py
index d602414ca41f37155c9c6d00eec54ea3918840c3..687617756619a5b547f18c99c4773842a8328ae8 100644
--- a/BaseTools/Source/Python/AutoGen/BuildEngine.py
+++ b/BaseTools/Source/Python/AutoGen/BuildEngine.py
@@ -2,6 +2,8 @@
 # The engine for building files
 #
 # Copyright (c) 2007 - 2018, Intel Corporation. All rights reserved.<BR>
+# Copyright (c) 2020, ARM Limited. All rights reserved.<BR> #
 # SPDX-License-Identifier: BSD-2-Clause-Patent  #
 
@@ -53,6 +55,7 @@ class TargetDescBlock(object):
         self.Outputs = Outputs
         self.Commands = Commands
         self.Dependencies = Dependencies
+        self.SourceFileDependencies = None
         if self.Outputs:
             self.Target = self.Outputs[0]
         else:
@@ -277,6 +280,9 @@ class FileBuildRule:
             TargetDesc.GenListFile = self.GenListFile
             TargetDesc.GenIncListFile = self.GenIncListFile
             self.BuildTargets[DstFile[0]] = TargetDesc
+
+        TargetDesc.SourceFileDependencies = 
+ SourceFile.SourceFileDependencies
+
         return TargetDesc
 
     def _BuildCommand(self, Macros):
diff --git a/BaseTools/Source/Python/AutoGen/GenMake.py b/BaseTools/Source/Python/AutoGen/GenMake.py
index fe94f9a4c232bb599a59563444c3985700c78ec6..5d932b66fb68baa06570a2eb421d717f0dab8187 100755
--- a/BaseTools/Source/Python/AutoGen/GenMake.py
+++ b/BaseTools/Source/Python/AutoGen/GenMake.py
@@ -2,6 +2,8 @@
 # Create makefile for MS nmake and GNU make  #  # Copyright (c) 2007 - 2018, Intel Corporation. All rights reserved.<BR>
+# Copyright (c) 2020, ARM Limited. All rights reserved.<BR> #
 # SPDX-License-Identifier: BSD-2-Clause-Patent  #
 
@@ -997,6 +999,11 @@ cleanlib:
 
                 Deps = []
                 CCodeDeps = []
+
+                if T.SourceFileDependencies:
+                    for File in T.SourceFileDependencies:
+                        Deps.append(self.PlaceMacro(str(File), 
+ self.Macros))
+
                 # Add force-dependencies
                 for Dep in T.Dependencies:
                     Deps.append(self.PlaceMacro(str(Dep), self.Macros)) diff --git a/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py b/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py
index aad591de65f086043d55aeea5661f59c53792e7c..b1cba5178e88db0855f5d0979bb7aa355093eade 100755
--- a/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py
+++ b/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py
@@ -2,6 +2,8 @@
 # Create makefile for MS nmake and GNU make  #  # Copyright (c) 2019, Intel Corporation. All rights reserved.<BR>
+# Copyright (c) 2020, ARM Limited. All rights reserved.<BR> #
 # SPDX-License-Identifier: BSD-2-Clause-Patent  #  from __future__ import absolute_import @@ -876,6 +878,21 @@ class ModuleAutoGen(AutoGen):
             if Source != File:
                 CreateDirectory(Source.Dir)
 
+            # The Makefile rule created from with source file dependency must
+            # have an object file as prerequisite.
+            # This gets the ouput format of sources.
+            OutPutSourceFileDependencies = []
+            if Source.SourceFileDependencies:
+                for File in Source.SourceFileDependencies:
+                    if File.Type in self.BuildRules:
+                        DepRuleObject = self.BuildRules[File.Type]
+                        DepTarget = DepRuleObject.Apply(File, self.BuildRuleOrder)
+                        # Files not producing outputs like '.h files don't create a target.
+                        if DepTarget:
+                            
+ OutPutSourceFileDependencies.extend(DepTarget.Outputs)
+
+                Source.SourceFileDependencies = 
+ OutPutSourceFileDependencies
+
             if File.IsBinary and File == Source and File in BinaryFileList:
                 # Skip all files that are not binary libraries
                 if not self.IsLibrary:
diff --git a/BaseTools/Source/Python/Common/DataType.py b/BaseTools/Source/Python/Common/DataType.py
index 8d80b410892946c8b862e060b0bf5f630b409825..8f21e1060446982f866c653aaea70cd662e307a0 100644
--- a/BaseTools/Source/Python/Common/DataType.py
+++ b/BaseTools/Source/Python/Common/DataType.py
@@ -2,7 +2,7 @@
 # This file is used to define common static strings used by INF/DEC/DSC files  #  # Copyright (c) 2007 - 2018, Intel Corporation. All rights reserved.<BR> -# Portions copyright (c) 2011 - 2013, ARM Ltd. All rights reserved.<BR>
+# Portions copyright (c) 2011 - 2020, ARM Ltd. All rights reserved.<BR>
 # SPDX-License-Identifier: BSD-2-Clause-Patent
 
 ##
@@ -19,6 +19,7 @@ TAB_VALUE_SPLIT = '|'
 TAB_COMMA_SPLIT = ','
 TAB_SPACE_SPLIT = ' '
 TAB_SEMI_COLON_SPLIT = ';'
+TAB_COLON_SPLIT = ':'
 TAB_SECTION_START = '['
 TAB_SECTION_END = ']'
 TAB_OPTION_START = '<'
diff --git a/BaseTools/Source/Python/Common/Misc.py b/BaseTools/Source/Python/Common/Misc.py
index da5fb380f0354d6e885aa716d2c9b2fead249d63..fb7a8f62087cfe7e0e007a9864832dc1eefc3068 100755
--- a/BaseTools/Source/Python/Common/Misc.py
+++ b/BaseTools/Source/Python/Common/Misc.py
@@ -2,6 +2,8 @@
 # Common routines used by all tools
 #
 # Copyright (c) 2007 - 2019, Intel Corporation. All rights reserved.<BR>
+# Copyright (c) 2020, ARM Limited. All rights reserved.<BR> #
 # SPDX-License-Identifier: BSD-2-Clause-Patent  #
 
@@ -1438,6 +1440,7 @@ class PathClass(object):
                  Arch='COMMON', ToolChainFamily='', Target='', TagName='', ToolCode=''):
         self.Arch = Arch
         self.File = str(File)
+        self.SourceFileDependencies = []
         if os.path.isabs(self.File):
             self.Root = ''
             self.AlterRoot = ''
diff --git a/BaseTools/Source/Python/Workspace/InfBuildData.py b/BaseTools/Source/Python/Workspace/InfBuildData.py
index 7675b0ea00ebd6a5fc3e823c965e32066f66f650..02e5c578b426e66b343df937a56ccea319b9f646 100644
--- a/BaseTools/Source/Python/Workspace/InfBuildData.py
+++ b/BaseTools/Source/Python/Workspace/InfBuildData.py
@@ -3,6 +3,8 @@
 #
 # Copyright (c) 2008 - 2018, Intel Corporation. All rights reserved.<BR>  # (C) Copyright 2016 Hewlett Packard Enterprise Development LP<BR>
+# Copyright (c) 2020, ARM Limited. All rights reserved.<BR> #
 # SPDX-License-Identifier: BSD-2-Clause-Patent  #
 
@@ -536,12 +538,19 @@ class InfBuildData(ModuleBuildClassObject):
             return []
 
         RetVal = []
+        PendingSourceFileDependencies = dict()
         RecordList = self._RawData[MODEL_EFI_SOURCE_FILE, self._Arch, self._Platform]
         Macros = self._Macros
         for Record in RecordList:
             LineNo = Record[-1]
-            ToolChainFamily = Record[1]
-            TagName = Record[2]
+
+            BuildOptions = ['', '']
+            SplittedValueList = GetSplitValueList(Record[1], TAB_VALUE_SPLIT, 1)
+            BuildOptions[0:len(SplittedValueList)] = SplittedValueList
+            SourceFileDependencies = 
+ list(set(GetSplitValueList(Record[2], TAB_COLON_SPLIT)))
+
+            ToolChainFamily = BuildOptions[0]
+            TagName = BuildOptions[1]
             ToolCode = Record[3]
 
             File = PathClass(NormPath(Record[0], Macros), self._ModuleDir, '', @@ -552,9 +561,28 @@ class InfBuildData(ModuleBuildClassObject):
                 EdkLogger.error('build', ErrorCode, ExtraData=ErrorInfo, File=self.MetaFile, Line=LineNo)
 
             RetVal.append(File)
+
+            # If there are source file dependencies, resolve them after all
+            # the PathClass instances of the source files have been created.
+            if SourceFileDependencies[0] != '':
+                PendingSourceFileDependencies[File] = 
+ SourceFileDependencies
+
         # add any previously found dependency files to the source list
         if self._DependencyFileList:
             RetVal.extend(self._DependencyFileList)
+
+        # Resolve the dependencies between the PathClass instances.
+        for SourceFile, SourceFileDepList in PendingSourceFileDependencies.items():
+            for SourceFileDepFile in SourceFileDepList:
+                Found = False
+                for Val in RetVal:
+                    if SourceFileDepFile == Val.File:
+                        SourceFile.SourceFileDependencies.append(Val)
+                        Found = True
+                        break
+                if not Found:
+                    EdkLogger.error("build", FILE_NOT_FOUND, 
+ ExtraData=SourceFileDepFile)
+
         return RetVal
 
     ## Retrieve library classes employed by this module diff --git a/BaseTools/Source/Python/Workspace/MetaFileParser.py b/BaseTools/Source/Python/Workspace/MetaFileParser.py
index a3b6edbd15ee5bf79cef0ac1fc5e53db30356c91..04a656b8ba1e55addc5f37ec8ee2b0439cef0ef1 100644
--- a/BaseTools/Source/Python/Workspace/MetaFileParser.py
+++ b/BaseTools/Source/Python/Workspace/MetaFileParser.py
@@ -3,6 +3,8 @@
 #
 # Copyright (c) 2008 - 2018, Intel Corporation. All rights reserved.<BR>  # (C) Copyright 2015-2018 Hewlett Packard Enterprise Development LP<BR>
+# Copyright (c) 2020, ARM Limited. All rights reserved.<BR> #
 # SPDX-License-Identifier: BSD-2-Clause-Patent  #
 
@@ -735,8 +737,20 @@ class InfParser(MetaFileParser):
     #
     @ParseMacro
     def _SourceFileParser(self):
-        TokenList = GetSplitValueList(self._CurrentLine, TAB_VALUE_SPLIT)
-        self._ValueList[0:len(TokenList)] = TokenList
+        SourceFileDependencySplit = GetSplitValueList(self._CurrentLine, TAB_COLON_SPLIT, 1)
+        BuildOptionSplit = 
+ GetSplitValueList(SourceFileDependencySplit[0], TAB_VALUE_SPLIT, 1)
+
+        # Get the filename.
+        self._ValueList[0] = BuildOptionSplit[0]
+
+        # Get the build options.
+        if len(BuildOptionSplit) > 1:
+            self._ValueList[1] = BuildOptionSplit[1]
+
+        # Get the dependencies.
+        if len(SourceFileDependencySplit) > 1:
+            self._ValueList[2] = SourceFileDependencySplit[1]
+
         Macros = self._Macros
         # For Acpi tables, remove macro like ' TABLE_NAME=Sata1'
         if 'COMPONENT_TYPE' in Macros:
--
'Guid(CE165669-3EF3-493F-B85D-6190EE5B9759)'