Re: [edk2-devel] [PATCH v2 1/1] BaseTools: Build ASL files before C files

["Liming Gao" <liming.gao@intel.com>]

Pierre:
  I see you use ':' to describe the dependency between the different source file. This is makefile syntax for the dependency. If one file depends on more than one files, other files will list after this file with the separator space ' '. So, I think the example should be like below.

> [Sources]
>   FileName1.X
>   FileName2.Y : FileName1.X
>   FileName3.Z : FileName1.X  FileName3.Y

Thanks
Liming
> -----Original Message-----
> From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of PierreGondois
> Sent: Thursday, January 23, 2020 1:33 AM
> To: Pierre Gondois <Pierre.Gondois@arm.com>; devel@edk2.groups.io
> Cc: ard.biesheuvel@linaro.org; Feng, Bob C <bob.c.feng@intel.com>; Gao, Liming <liming.gao@intel.com>; Sami Mujawar
> <Sami.Mujawar@arm.com>; nd <nd@arm.com>
> Subject: Re: [edk2-devel] [PATCH v2 1/1] BaseTools: Build ASL files before C files
> 
> Hello everyone,
> Do you have any input on the patch?
> 
> Regards,
> Pierre
> 
> -----Original Message-----
> From: PierreGondois <pierre.gondois@arm.com>
> Sent: 14 January 2020 18:46
> To: devel@edk2.groups.io
> Cc: Pierre Gondois <Pierre.Gondois@arm.com>; ard.biesheuvel@linaro.org; bob.c.feng@intel.com; liming.gao@intel.com; Sami
> Mujawar <Sami.Mujawar@arm.com>; nd <nd@arm.com>
> Subject: [PATCH v2 1/1] BaseTools: Build ASL files before C files
> 
> From: Pierre Gondois <pierre.gondois@arm.com>
> 
> The dependencies for C files are satisfied by the build system.
> However, there are use cases where source files with different languages are inter-dependent. The EDKII build framework currently
> doesn't have options to specify such dependencies.
> E.g. It may be necessary to specify the build order between
>      ASL files and C files. The use case being that the AML
>      blob generated by compiling the ASL files is loaded and
>      parsed by some C code.
> 
> This patch allows to describe dependencies between files listed in the '[Sources]' section of '.inf' files. The list of source files to build
> prior are colon separated (':'), e.g.:
> [Sources]
>   FileName1.X
>   FileName2.Y : FileName1.X
>   FileName3.Z : FileName1.X : FileName3.Z
> 
> In the example above:
>   * FileName1.X will be built prior to FileName2.Y.
>   * FileName1.X and FileName2.Y will be built prior to
>     FileName3.Z.
> 
> This does not affect the file specific build options, which are pipe separated, e.g.:
>   FileName2.Y | GCC : FileName1.X
> 
> The list of colon separated files must be part of the module, i.e. they must be present in the [Sources] section.
> 
> Their file extension must be described in the BaseTools/Conf/build_rule.template file, and generate an output file, i.e. have a non-
> empty '<OutputFile>' section.
> 
> Declaring a dependency in the [Sources] sections leads to the creation of makefile rules between these files in the build. The makefile
> rule is between the generated files.
> E.g.:
>   FileName1.X generates FileName1.objx
>   FileName2.Y generates FileName2.objy
>   Then
>     FileName2.Y : FileName1.X
>   will create the following makefile rule:
>     FileName2.objy : FileName1.objx
> 
> INF Specification updates:
> s3.2.1, "Common Definitions":
>   <SrcDependencySeperator> ::= ":"
>   <DepS>                   ::= <TS> <SrcDependencySeperator> <TS>
> 
> s2.5, "[Sources] Section":
>   To describe a build order dependency between source
>   files, specify the source files to build prior by
>   listing them, colon separated.
>   <opt3> ::= <FS> [<FeatureFlagExpress>] [<opt4>]*
>   <opt4> ::= <DepS> [FileNameDependency]
> 
> Signed-off-by: Pierre Gondois <pierre.gondois@arm.com>
> ---
> 
> The changes can be seen at https://github.com/PierreARM/edk2/tree/676_build_asl_first_v2
> 
> Notes:
>     v2:
>       - Enable file dependencies in .if files [Pierre]
> 
>  BaseTools/Source/Python/AutoGen/BuildEngine.py      |  6 ++++
>  BaseTools/Source/Python/AutoGen/GenMake.py          |  7 +++++
>  BaseTools/Source/Python/AutoGen/ModuleAutoGen.py    | 17 +++++++++++
>  BaseTools/Source/Python/Common/DataType.py          |  3 +-
>  BaseTools/Source/Python/Common/Misc.py              |  3 ++
>  BaseTools/Source/Python/Workspace/InfBuildData.py   | 32 ++++++++++++++++++--
>  BaseTools/Source/Python/Workspace/MetaFileParser.py | 18 +++++++++--
>  7 files changed, 81 insertions(+), 5 deletions(-)
> 
> diff --git a/BaseTools/Source/Python/AutoGen/BuildEngine.py b/BaseTools/Source/Python/AutoGen/BuildEngine.py
> index d602414ca41f37155c9c6d00eec54ea3918840c3..687617756619a5b547f18c99c4773842a8328ae8 100644
> --- a/BaseTools/Source/Python/AutoGen/BuildEngine.py
> +++ b/BaseTools/Source/Python/AutoGen/BuildEngine.py
> @@ -2,6 +2,8 @@
>  # The engine for building files
>  #
>  # Copyright (c) 2007 - 2018, Intel Corporation. All rights reserved.<BR>
> +# Copyright (c) 2020, ARM Limited. All rights reserved.<BR> #
>  # SPDX-License-Identifier: BSD-2-Clause-Patent  #
> 
> @@ -53,6 +55,7 @@ class TargetDescBlock(object):
>          self.Outputs = Outputs
>          self.Commands = Commands
>          self.Dependencies = Dependencies
> +        self.SourceFileDependencies = None
>          if self.Outputs:
>              self.Target = self.Outputs[0]
>          else:
> @@ -277,6 +280,9 @@ class FileBuildRule:
>              TargetDesc.GenListFile = self.GenListFile
>              TargetDesc.GenIncListFile = self.GenIncListFile
>              self.BuildTargets[DstFile[0]] = TargetDesc
> +
> +        TargetDesc.SourceFileDependencies =
> + SourceFile.SourceFileDependencies
> +
>          return TargetDesc
> 
>      def _BuildCommand(self, Macros):
> diff --git a/BaseTools/Source/Python/AutoGen/GenMake.py b/BaseTools/Source/Python/AutoGen/GenMake.py
> index fe94f9a4c232bb599a59563444c3985700c78ec6..5d932b66fb68baa06570a2eb421d717f0dab8187 100755
> --- a/BaseTools/Source/Python/AutoGen/GenMake.py
> +++ b/BaseTools/Source/Python/AutoGen/GenMake.py
> @@ -2,6 +2,8 @@
>  # Create makefile for MS nmake and GNU make  #  # Copyright (c) 2007 - 2018, Intel Corporation. All rights reserved.<BR>
> +# Copyright (c) 2020, ARM Limited. All rights reserved.<BR> #
>  # SPDX-License-Identifier: BSD-2-Clause-Patent  #
> 
> @@ -997,6 +999,11 @@ cleanlib:
> 
>                  Deps = []
>                  CCodeDeps = []
> +
> +                if T.SourceFileDependencies:
> +                    for File in T.SourceFileDependencies:
> +                        Deps.append(self.PlaceMacro(str(File),
> + self.Macros))
> +
>                  # Add force-dependencies
>                  for Dep in T.Dependencies:
>                      Deps.append(self.PlaceMacro(str(Dep), self.Macros)) diff --git a/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py
> b/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py
> index aad591de65f086043d55aeea5661f59c53792e7c..b1cba5178e88db0855f5d0979bb7aa355093eade 100755
> --- a/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py
> +++ b/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py
> @@ -2,6 +2,8 @@
>  # Create makefile for MS nmake and GNU make  #  # Copyright (c) 2019, Intel Corporation. All rights reserved.<BR>
> +# Copyright (c) 2020, ARM Limited. All rights reserved.<BR> #
>  # SPDX-License-Identifier: BSD-2-Clause-Patent  #  from __future__ import absolute_import @@ -876,6 +878,21 @@ class
> ModuleAutoGen(AutoGen):
>              if Source != File:
>                  CreateDirectory(Source.Dir)
> 
> +            # The Makefile rule created from with source file dependency must
> +            # have an object file as prerequisite.
> +            # This gets the ouput format of sources.
> +            OutPutSourceFileDependencies = []
> +            if Source.SourceFileDependencies:
> +                for File in Source.SourceFileDependencies:
> +                    if File.Type in self.BuildRules:
> +                        DepRuleObject = self.BuildRules[File.Type]
> +                        DepTarget = DepRuleObject.Apply(File, self.BuildRuleOrder)
> +                        # Files not producing outputs like '.h files don't create a target.
> +                        if DepTarget:
> +
> + OutPutSourceFileDependencies.extend(DepTarget.Outputs)
> +
> +                Source.SourceFileDependencies =
> + OutPutSourceFileDependencies
> +
>              if File.IsBinary and File == Source and File in BinaryFileList:
>                  # Skip all files that are not binary libraries
>                  if not self.IsLibrary:
> diff --git a/BaseTools/Source/Python/Common/DataType.py b/BaseTools/Source/Python/Common/DataType.py
> index 8d80b410892946c8b862e060b0bf5f630b409825..8f21e1060446982f866c653aaea70cd662e307a0 100644
> --- a/BaseTools/Source/Python/Common/DataType.py
> +++ b/BaseTools/Source/Python/Common/DataType.py
> @@ -2,7 +2,7 @@
>  # This file is used to define common static strings used by INF/DEC/DSC files  #  # Copyright (c) 2007 - 2018, Intel Corporation. All
> rights reserved.<BR> -# Portions copyright (c) 2011 - 2013, ARM Ltd. All rights reserved.<BR>
> +# Portions copyright (c) 2011 - 2020, ARM Ltd. All rights reserved.<BR>
>  # SPDX-License-Identifier: BSD-2-Clause-Patent
> 
>  ##
> @@ -19,6 +19,7 @@ TAB_VALUE_SPLIT = '|'
>  TAB_COMMA_SPLIT = ','
>  TAB_SPACE_SPLIT = ' '
>  TAB_SEMI_COLON_SPLIT = ';'
> +TAB_COLON_SPLIT = ':'
>  TAB_SECTION_START = '['
>  TAB_SECTION_END = ']'
>  TAB_OPTION_START = '<'
> diff --git a/BaseTools/Source/Python/Common/Misc.py b/BaseTools/Source/Python/Common/Misc.py
> index da5fb380f0354d6e885aa716d2c9b2fead249d63..fb7a8f62087cfe7e0e007a9864832dc1eefc3068 100755
> --- a/BaseTools/Source/Python/Common/Misc.py
> +++ b/BaseTools/Source/Python/Common/Misc.py
> @@ -2,6 +2,8 @@
>  # Common routines used by all tools
>  #
>  # Copyright (c) 2007 - 2019, Intel Corporation. All rights reserved.<BR>
> +# Copyright (c) 2020, ARM Limited. All rights reserved.<BR> #
>  # SPDX-License-Identifier: BSD-2-Clause-Patent  #
> 
> @@ -1438,6 +1440,7 @@ class PathClass(object):
>                   Arch='COMMON', ToolChainFamily='', Target='', TagName='', ToolCode=''):
>          self.Arch = Arch
>          self.File = str(File)
> +        self.SourceFileDependencies = []
>          if os.path.isabs(self.File):
>              self.Root = ''
>              self.AlterRoot = ''
> diff --git a/BaseTools/Source/Python/Workspace/InfBuildData.py b/BaseTools/Source/Python/Workspace/InfBuildData.py
> index 7675b0ea00ebd6a5fc3e823c965e32066f66f650..02e5c578b426e66b343df937a56ccea319b9f646 100644
> --- a/BaseTools/Source/Python/Workspace/InfBuildData.py
> +++ b/BaseTools/Source/Python/Workspace/InfBuildData.py
> @@ -3,6 +3,8 @@
>  #
>  # Copyright (c) 2008 - 2018, Intel Corporation. All rights reserved.<BR>  # (C) Copyright 2016 Hewlett Packard Enterprise Development
> LP<BR>
> +# Copyright (c) 2020, ARM Limited. All rights reserved.<BR> #
>  # SPDX-License-Identifier: BSD-2-Clause-Patent  #
> 
> @@ -536,12 +538,19 @@ class InfBuildData(ModuleBuildClassObject):
>              return []
> 
>          RetVal = []
> +        PendingSourceFileDependencies = dict()
>          RecordList = self._RawData[MODEL_EFI_SOURCE_FILE, self._Arch, self._Platform]
>          Macros = self._Macros
>          for Record in RecordList:
>              LineNo = Record[-1]
> -            ToolChainFamily = Record[1]
> -            TagName = Record[2]
> +
> +            BuildOptions = ['', '']
> +            SplittedValueList = GetSplitValueList(Record[1], TAB_VALUE_SPLIT, 1)
> +            BuildOptions[0:len(SplittedValueList)] = SplittedValueList
> +            SourceFileDependencies =
> + list(set(GetSplitValueList(Record[2], TAB_COLON_SPLIT)))
> +
> +            ToolChainFamily = BuildOptions[0]
> +            TagName = BuildOptions[1]
>              ToolCode = Record[3]
> 
>              File = PathClass(NormPath(Record[0], Macros), self._ModuleDir, '', @@ -552,9 +561,28 @@ class
> InfBuildData(ModuleBuildClassObject):
>                  EdkLogger.error('build', ErrorCode, ExtraData=ErrorInfo, File=self.MetaFile, Line=LineNo)
> 
>              RetVal.append(File)
> +
> +            # If there are source file dependencies, resolve them after all
> +            # the PathClass instances of the source files have been created.
> +            if SourceFileDependencies[0] != '':
> +                PendingSourceFileDependencies[File] =
> + SourceFileDependencies
> +
>          # add any previously found dependency files to the source list
>          if self._DependencyFileList:
>              RetVal.extend(self._DependencyFileList)
> +
> +        # Resolve the dependencies between the PathClass instances.
> +        for SourceFile, SourceFileDepList in PendingSourceFileDependencies.items():
> +            for SourceFileDepFile in SourceFileDepList:
> +                Found = False
> +                for Val in RetVal:
> +                    if SourceFileDepFile == Val.File:
> +                        SourceFile.SourceFileDependencies.append(Val)
> +                        Found = True
> +                        break
> +                if not Found:
> +                    EdkLogger.error("build", FILE_NOT_FOUND,
> + ExtraData=SourceFileDepFile)
> +
>          return RetVal
> 
>      ## Retrieve library classes employed by this module diff --git a/BaseTools/Source/Python/Workspace/MetaFileParser.py
> b/BaseTools/Source/Python/Workspace/MetaFileParser.py
> index a3b6edbd15ee5bf79cef0ac1fc5e53db30356c91..04a656b8ba1e55addc5f37ec8ee2b0439cef0ef1 100644
> --- a/BaseTools/Source/Python/Workspace/MetaFileParser.py
> +++ b/BaseTools/Source/Python/Workspace/MetaFileParser.py
> @@ -3,6 +3,8 @@
>  #
>  # Copyright (c) 2008 - 2018, Intel Corporation. All rights reserved.<BR>  # (C) Copyright 2015-2018 Hewlett Packard Enterprise
> Development LP<BR>
> +# Copyright (c) 2020, ARM Limited. All rights reserved.<BR> #
>  # SPDX-License-Identifier: BSD-2-Clause-Patent  #
> 
> @@ -735,8 +737,20 @@ class InfParser(MetaFileParser):
>      #
>      @ParseMacro
>      def _SourceFileParser(self):
> -        TokenList = GetSplitValueList(self._CurrentLine, TAB_VALUE_SPLIT)
> -        self._ValueList[0:len(TokenList)] = TokenList
> +        SourceFileDependencySplit = GetSplitValueList(self._CurrentLine, TAB_COLON_SPLIT, 1)
> +        BuildOptionSplit =
> + GetSplitValueList(SourceFileDependencySplit[0], TAB_VALUE_SPLIT, 1)
> +
> +        # Get the filename.
> +        self._ValueList[0] = BuildOptionSplit[0]
> +
> +        # Get the build options.
> +        if len(BuildOptionSplit) > 1:
> +            self._ValueList[1] = BuildOptionSplit[1]
> +
> +        # Get the dependencies.
> +        if len(SourceFileDependencySplit) > 1:
> +            self._ValueList[2] = SourceFileDependencySplit[1]
> +
>          Macros = self._Macros
>          # For Acpi tables, remove macro like ' TABLE_NAME=Sata1'
>          if 'COMPONENT_TYPE' in Macros:
> --
> 'Guid(CE165669-3EF3-493F-B85D-6190EE5B9759)'
> 
> 
> 
>