* [PATCH v1 0/3] Compile AML bytecode array into OBJ file
@ 2020-05-18 14:11 PierreGondois
2020-05-18 14:11 ` [PATCH v1 1/3] BaseTools: Generate multiple rules when multiple output files PierreGondois
` (5 more replies)
0 siblings, 6 replies; 17+ messages in thread
From: PierreGondois @ 2020-05-18 14:11 UTC (permalink / raw)
To: devel; +Cc: Pierre Gondois, bob.c.feng, liming.gao, Sami.Mujawar, Tomas.Pilar,
nd
Following the BZ at https://bugzilla.tianocore.org/show_bug.cgi?id=2425
This patch serie is a another way to solve the dependency
of C files over ASL files. With this new method, the
dependency is resolved at the linking stage.
The last method to solve this dependency was to add
the possibility to modify INF files to depict such a
dependency. This method was not accepted. The discussion
is available at https://edk2.groups.io/g/devel/topic/72655342#56658
The last patch modifying the INF specification and INF
parsing are available at:
https://edk2.groups.io/g/devel/topic/72655342#56658
https://edk2.groups.io/g/devel/topic/72656060#56662
Pierre Gondois (3):
BaseTools: Generate multiple rules when multiple output files
BaseTools: Rename AmlToHex script to AmlToC
BaseTools: Compile AML bytecode arrays into .obj file
BaseTools/BinWrappers/PosixLike/{AmlToHex => AmlToC} | 28 +++----
BaseTools/BinWrappers/WindowsLike/{AmlToHex.bat => AmlToC.bat} | 0
BaseTools/Conf/build_rule.template | 15 +++-
BaseTools/Source/Python/{AmlToHex/AmlToHex.py => AmlToC/AmlToC.py} | 82 ++++++++------------
BaseTools/Source/Python/AutoGen/GenMake.py | 6 ++
BaseTools/Source/Python/AutoGen/ModuleAutoGen.py | 40 +++++-----
6 files changed, 86 insertions(+), 85 deletions(-)
rename BaseTools/BinWrappers/PosixLike/{AmlToHex => AmlToC} (97%)
rename BaseTools/BinWrappers/WindowsLike/{AmlToHex.bat => AmlToC.bat} (100%)
rename BaseTools/Source/Python/{AmlToHex/AmlToHex.py => AmlToC/AmlToC.py} (52%)
--
'Guid(CE165669-3EF3-493F-B85D-6190EE5B9759)'
^ permalink raw reply [flat|nested] 17+ messages in thread* [PATCH v1 1/3] BaseTools: Generate multiple rules when multiple output files 2020-05-18 14:11 [PATCH v1 0/3] Compile AML bytecode array into OBJ file PierreGondois @ 2020-05-18 14:11 ` PierreGondois 2020-06-02 11:15 ` [edk2-devel] " Bob Feng 2020-05-18 14:11 ` [PATCH v1 2/3] BaseTools: Rename AmlToHex script to AmlToC PierreGondois ` (4 subsequent siblings) 5 siblings, 1 reply; 17+ messages in thread From: PierreGondois @ 2020-05-18 14:11 UTC (permalink / raw) To: devel; +Cc: Pierre Gondois, bob.c.feng, liming.gao, Sami.Mujawar, Tomas.Pilar, nd From: Pierre Gondois <pierre.gondois@arm.com> BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=2425 This patch modifies the Makefile generation not to stop adding Makfile rules when the first final target is found. E.g.: If the following rules are described in build_rule.txt: -[Rule1]: .X files generate .Y and .Z files; -[Rule2]: .Z files generate .Z1 files. Currently, if a File1.X file was part of the sources of a module, only [Rule1] would be generated in the Makefile. Indeed, there are no rules to apply to .Y files: .Y files are a final target. However, there is still [Rule2] to apply to .Z files. This patch also adds a dependency between the first ouput file of a rule and the other output files. For instance, with the same example as above, File1.Y and File1.Z are generated by the following rule: File1.Y: File1.X <Generate File1.Y> <Generate File1.Z> and the new dependency is: File1.Z: File1.Y This is necessary to keep a dependency order during the execution of the Makefile. Indeed, .Y and .Z files are generated by the execution of a common set of commands, and without this rule, there is no explicit dependency relation between them. Signed-off-by: Pierre Gondois <Pierre.Gondois@arm.com> --- The changes can be seen at: https://github.com/PierreARM/edk2/tree/803_Compile_AML_bytecode_array_into_OBJ_file_v1 Notes: v1: - Generate multiple rules when multiple output files are specified in the build_rule.txt file. [Pierre] BaseTools/Source/Python/AutoGen/GenMake.py | 6 +++ BaseTools/Source/Python/AutoGen/ModuleAutoGen.py | 40 ++++++++++---------- 2 files changed, 25 insertions(+), 21 deletions(-) diff --git a/BaseTools/Source/Python/AutoGen/GenMake.py b/BaseTools/Source/Python/AutoGen/GenMake.py index bbb3c29446f53fa7f2cb61a216a5b119f72c3fbc..0314d0ea34d99a014379e8d30c46ac0f0a7068ce 100755 --- a/BaseTools/Source/Python/AutoGen/GenMake.py +++ b/BaseTools/Source/Python/AutoGen/GenMake.py @@ -1054,6 +1054,12 @@ cleanlib: TargetDict = {"target": self.PlaceMacro(T.Target.Path, self.Macros), "cmd": "\n\t".join(T.Commands),"deps": Deps} self.BuildTargetList.append(self._BUILD_TARGET_TEMPLATE.Replace(TargetDict)) + # Add a Makefile rule for targets generating multiple files. + # The main output is a prerequisite for the other output files. + for i in T.Outputs[1:]: + AnnexeTargetDict = {"target": self.PlaceMacro(i.Path, self.Macros), "cmd": "", "deps": self.PlaceMacro(T.Target.Path, self.Macros)} + self.BuildTargetList.append(self._BUILD_TARGET_TEMPLATE.Replace(AnnexeTargetDict)) + def ParserCCodeFile(self, T, Type, CmdSumDict, CmdTargetDict, CmdCppDict, DependencyDict): if not CmdSumDict: for item in self._AutoGenObject.Targets[Type]: diff --git a/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py b/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py index aad591de65f086043d55aeea5661f59c53792e7c..82facdb2774c56833718317f4414e0c1861de47f 100755 --- a/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py +++ b/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py @@ -860,7 +860,8 @@ class ModuleAutoGen(AutoGen): SubDirectory = os.path.join(self.OutputDir, File.SubDir) if not os.path.exists(SubDirectory): CreateDirectory(SubDirectory) - LastTarget = None + TargetList = set() + FinalTargetName = set() RuleChain = set() SourceList = [File] Index = 0 @@ -872,7 +873,7 @@ class ModuleAutoGen(AutoGen): while Index < len(SourceList): Source = SourceList[Index] Index = Index + 1 - + FileType = TAB_UNKNOWN_FILE if Source != File: CreateDirectory(Source.Dir) @@ -881,34 +882,27 @@ class ModuleAutoGen(AutoGen): if not self.IsLibrary: continue RuleObject = self.BuildRules[TAB_DEFAULT_BINARY_FILE] - elif FileType in self.BuildRules: - RuleObject = self.BuildRules[FileType] elif Source.Ext in self.BuildRules: RuleObject = self.BuildRules[Source.Ext] else: - # stop at no more rules - if LastTarget: - self._FinalBuildTargetList.add(LastTarget) - break - + # No more rule to apply: Source is a final target. + FinalTargetName.add(Source) + continue FileType = RuleObject.SourceFileType self._FileTypes[FileType].add(Source) # stop at STATIC_LIBRARY for library if self.IsLibrary and FileType == TAB_STATIC_LIBRARY: - if LastTarget: - self._FinalBuildTargetList.add(LastTarget) - break + FinalTargetName.add(Source) + continue Target = RuleObject.Apply(Source, self.BuildRuleOrder) if not Target: - if LastTarget: - self._FinalBuildTargetList.add(LastTarget) - break - elif not Target.Outputs: - # Only do build for target with outputs - self._FinalBuildTargetList.add(Target) + # No Target: Source is a final target. + FinalTargetName.add(Source) + continue + TargetList.add(Target) self._BuildTargets[FileType].add(Target) if not Source.IsBinary and Source == File: @@ -916,12 +910,16 @@ class ModuleAutoGen(AutoGen): # to avoid cyclic rule if FileType in RuleChain: - break + EdkLogger.error("build", ERROR_STATEMENT, "Cyclic dependency detected while generating rule for %s" % str(Source)) RuleChain.add(FileType) SourceList.extend(Target.Outputs) - LastTarget = Target - FileType = TAB_UNKNOWN_FILE + + # For each final target, retrieve the TargetDescBlock instance. + for FTargetName in FinalTargetName: + for Target in TargetList: + if FTargetName == Target.Target: + self._FinalBuildTargetList.add(Target) @cached_property def Targets(self): -- 'Guid(CE165669-3EF3-493F-B85D-6190EE5B9759)' ^ permalink raw reply related [flat|nested] 17+ messages in thread
* Re: [edk2-devel] [PATCH v1 1/3] BaseTools: Generate multiple rules when multiple output files 2020-05-18 14:11 ` [PATCH v1 1/3] BaseTools: Generate multiple rules when multiple output files PierreGondois @ 2020-06-02 11:15 ` Bob Feng 2020-06-02 13:03 ` PierreGondois 0 siblings, 1 reply; 17+ messages in thread From: Bob Feng @ 2020-06-02 11:15 UTC (permalink / raw) To: devel@edk2.groups.io, pierre.gondois@arm.com Cc: Gao, Liming, Sami.Mujawar@arm.com, Tomas.Pilar@arm.com, nd@arm.com My comments are inline marked as [Bob]. Thanks, Bob -----Original Message----- From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of PierreGondois Sent: Monday, May 18, 2020 10:11 PM To: devel@edk2.groups.io Cc: Pierre Gondois <Pierre.Gondois@arm.com>; Feng, Bob C <bob.c.feng@intel.com>; Gao, Liming <liming.gao@intel.com>; Sami.Mujawar@arm.com; Tomas.Pilar@arm.com; nd@arm.com Subject: [edk2-devel] [PATCH v1 1/3] BaseTools: Generate multiple rules when multiple output files From: Pierre Gondois <pierre.gondois@arm.com> BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=2425 This patch modifies the Makefile generation not to stop adding Makfile rules when the first final target is found. E.g.: If the following rules are described in build_rule.txt: -[Rule1]: .X files generate .Y and .Z files; -[Rule2]: .Z files generate .Z1 files. Currently, if a File1.X file was part of the sources of a module, only [Rule1] would be generated in the Makefile. Indeed, there are no rules to apply to .Y files: .Y files are a final target. However, there is still [Rule2] to apply to .Z files. [Bob] I think currently a rule's output file will be added back to source file list, and in the later loop, that output file will be handled by another rule. Doesn't that algorithm handle your case above? This patch also adds a dependency between the first ouput file of a rule and the other output files. For instance, with the same example as above, File1.Y and File1.Z are generated by the following rule: File1.Y: File1.X <Generate File1.Y> <Generate File1.Z> and the new dependency is: File1.Z: File1.Y This is necessary to keep a dependency order during the execution of the Makefile. Indeed, .Y and .Z files are generated by the execution of a common set of commands, and without this rule, there is no explicit dependency relation between them. [Bob] If there are 3 output files, for example: <OutputFile> A B C Will the dependency relationship be B: A C: B Signed-off-by: Pierre Gondois <Pierre.Gondois@arm.com> --- The changes can be seen at: https://github.com/PierreARM/edk2/tree/803_Compile_AML_bytecode_array_into_OBJ_file_v1 Notes: v1: - Generate multiple rules when multiple output files are specified in the build_rule.txt file. [Pierre] BaseTools/Source/Python/AutoGen/GenMake.py | 6 +++ BaseTools/Source/Python/AutoGen/ModuleAutoGen.py | 40 ++++++++++---------- 2 files changed, 25 insertions(+), 21 deletions(-) diff --git a/BaseTools/Source/Python/AutoGen/GenMake.py b/BaseTools/Source/Python/AutoGen/GenMake.py index bbb3c29446f53fa7f2cb61a216a5b119f72c3fbc..0314d0ea34d99a014379e8d30c46ac0f0a7068ce 100755 --- a/BaseTools/Source/Python/AutoGen/GenMake.py +++ b/BaseTools/Source/Python/AutoGen/GenMake.py @@ -1054,6 +1054,12 @@ cleanlib: TargetDict = {"target": self.PlaceMacro(T.Target.Path, self.Macros), "cmd": "\n\t".join(T.Commands),"deps": Deps} self.BuildTargetList.append(self._BUILD_TARGET_TEMPLATE.Replace(TargetDict)) + # Add a Makefile rule for targets generating multiple files. + # The main output is a prerequisite for the other output files. + for i in T.Outputs[1:]: + AnnexeTargetDict = {"target": self.PlaceMacro(i.Path, self.Macros), "cmd": "", "deps": self.PlaceMacro(T.Target.Path, self.Macros)} + + self.BuildTargetList.append(self._BUILD_TARGET_TEMPLATE.Replace(Annexe + TargetDict)) + def ParserCCodeFile(self, T, Type, CmdSumDict, CmdTargetDict, CmdCppDict, DependencyDict): if not CmdSumDict: for item in self._AutoGenObject.Targets[Type]: diff --git a/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py b/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py index aad591de65f086043d55aeea5661f59c53792e7c..82facdb2774c56833718317f4414e0c1861de47f 100755 --- a/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py +++ b/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py @@ -860,7 +860,8 @@ class ModuleAutoGen(AutoGen): SubDirectory = os.path.join(self.OutputDir, File.SubDir) if not os.path.exists(SubDirectory): CreateDirectory(SubDirectory) - LastTarget = None + TargetList = set() + FinalTargetName = set() RuleChain = set() SourceList = [File] Index = 0 @@ -872,7 +873,7 @@ class ModuleAutoGen(AutoGen): while Index < len(SourceList): Source = SourceList[Index] Index = Index + 1 - + FileType = TAB_UNKNOWN_FILE if Source != File: CreateDirectory(Source.Dir) @@ -881,34 +882,27 @@ class ModuleAutoGen(AutoGen): if not self.IsLibrary: continue RuleObject = self.BuildRules[TAB_DEFAULT_BINARY_FILE] - elif FileType in self.BuildRules: - RuleObject = self.BuildRules[FileType] elif Source.Ext in self.BuildRules: RuleObject = self.BuildRules[Source.Ext] else: - # stop at no more rules - if LastTarget: - self._FinalBuildTargetList.add(LastTarget) - break - + # No more rule to apply: Source is a final target. + FinalTargetName.add(Source) + continue FileType = RuleObject.SourceFileType self._FileTypes[FileType].add(Source) # stop at STATIC_LIBRARY for library if self.IsLibrary and FileType == TAB_STATIC_LIBRARY: - if LastTarget: - self._FinalBuildTargetList.add(LastTarget) - break + FinalTargetName.add(Source) + continue Target = RuleObject.Apply(Source, self.BuildRuleOrder) if not Target: - if LastTarget: - self._FinalBuildTargetList.add(LastTarget) - break - elif not Target.Outputs: - # Only do build for target with outputs - self._FinalBuildTargetList.add(Target) + # No Target: Source is a final target. + FinalTargetName.add(Source) + continue + TargetList.add(Target) self._BuildTargets[FileType].add(Target) if not Source.IsBinary and Source == File: @@ -916,12 +910,16 @@ class ModuleAutoGen(AutoGen): # to avoid cyclic rule if FileType in RuleChain: - break + EdkLogger.error("build", ERROR_STATEMENT, "Cyclic + dependency detected while generating rule for %s" % str(Source)) RuleChain.add(FileType) SourceList.extend(Target.Outputs) - LastTarget = Target - FileType = TAB_UNKNOWN_FILE + + # For each final target, retrieve the TargetDescBlock instance. + for FTargetName in FinalTargetName: + for Target in TargetList: + if FTargetName == Target.Target: + self._FinalBuildTargetList.add(Target) @cached_property def Targets(self): -- 'Guid(CE165669-3EF3-493F-B85D-6190EE5B9759)' ^ permalink raw reply related [flat|nested] 17+ messages in thread
* Re: [edk2-devel] [PATCH v1 1/3] BaseTools: Generate multiple rules when multiple output files 2020-06-02 11:15 ` [edk2-devel] " Bob Feng @ 2020-06-02 13:03 ` PierreGondois 2020-06-08 14:00 ` PierreGondois 0 siblings, 1 reply; 17+ messages in thread From: PierreGondois @ 2020-06-02 13:03 UTC (permalink / raw) To: Feng, Bob C, devel@edk2.groups.io Cc: Gao, Liming, Sami Mujawar, Tomas Pilar, nd Hello Bob, Thank you for your answer, I put my comments as [Pierre], Regards, Pierre -----Original Message----- From: Feng, Bob C <bob.c.feng@intel.com> Sent: 02 June 2020 12:16 To: devel@edk2.groups.io; Pierre Gondois <Pierre.Gondois@arm.com> Cc: Gao, Liming <liming.gao@intel.com>; Sami Mujawar <Sami.Mujawar@arm.com>; Tomas Pilar <Tomas.Pilar@arm.com>; nd <nd@arm.com> Subject: RE: [edk2-devel] [PATCH v1 1/3] BaseTools: Generate multiple rules when multiple output files My comments are inline marked as [Bob]. Thanks, Bob -----Original Message----- From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of PierreGondois Sent: Monday, May 18, 2020 10:11 PM To: devel@edk2.groups.io Cc: Pierre Gondois <Pierre.Gondois@arm.com>; Feng, Bob C <bob.c.feng@intel.com>; Gao, Liming <liming.gao@intel.com>; Sami.Mujawar@arm.com; Tomas.Pilar@arm.com; nd@arm.com Subject: [edk2-devel] [PATCH v1 1/3] BaseTools: Generate multiple rules when multiple output files From: Pierre Gondois <pierre.gondois@arm.com> BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=2425 This patch modifies the Makefile generation not to stop adding Makfile rules when the first final target is found. E.g.: If the following rules are described in build_rule.txt: -[Rule1]: .X files generate .Y and .Z files; -[Rule2]: .Z files generate .Z1 files. Currently, if a File1.X file was part of the sources of a module, only [Rule1] would be generated in the Makefile. Indeed, there are no rules to apply to .Y files: .Y files are a final target. However, there is still [Rule2] to apply to .Z files. [Bob] I think currently a rule's output file will be added back to source file list, and in the later loop, that output file will be handled by another rule. Doesn't that algorithm handle your case above? [Pierre] The rule's output file was effectively added to the list of source files, and a rule was searched for this output file. However, the loop stopped when the first final target was found. By final target I mean "a file that isn't the input of a rule". For the asl/aml/amli case, this meant that: -(first loop iteration: treating the ASL file) The rule for ASL files was found and applied to the input ASL file. The AML and ".amli" files were added to the list of source files to look for a rule to apply to them. The loop continues. -(second loop iteration: treating the AML file) There is no rule for AML files. This means that the AML file is a final target. The loop ends without having applied the rule on the ".amli" file. This is why all the "break" instructions of the loop have been replaced with "continue". This prevents the loop from exiting without having treated all the files in the "SourceList". This patch also adds a dependency between the first ouput file of a rule and the other output files. For instance, with the same example as above, File1.Y and File1.Z are generated by the following rule: File1.Y: File1.X <Generate File1.Y> <Generate File1.Z> and the new dependency is: File1.Z: File1.Y This is necessary to keep a dependency order during the execution of the Makefile. Indeed, .Y and .Z files are generated by the execution of a common set of commands, and without this rule, there is no explicit dependency relation between them. [Bob] If there are 3 output files, for example: <OutputFile> A B C Will the dependency relationship be B: A C: B [Pierre] Currently, the dependency relation that would be created is: B: A C: A This can be changed to the dependency you described. Signed-off-by: Pierre Gondois <Pierre.Gondois@arm.com> --- The changes can be seen at: https://github.com/PierreARM/edk2/tree/803_Compile_AML_bytecode_array_into_OBJ_file_v1 Notes: v1: - Generate multiple rules when multiple output files are specified in the build_rule.txt file. [Pierre] BaseTools/Source/Python/AutoGen/GenMake.py | 6 +++ BaseTools/Source/Python/AutoGen/ModuleAutoGen.py | 40 ++++++++++---------- 2 files changed, 25 insertions(+), 21 deletions(-) diff --git a/BaseTools/Source/Python/AutoGen/GenMake.py b/BaseTools/Source/Python/AutoGen/GenMake.py index bbb3c29446f53fa7f2cb61a216a5b119f72c3fbc..0314d0ea34d99a014379e8d30c46ac0f0a7068ce 100755 --- a/BaseTools/Source/Python/AutoGen/GenMake.py +++ b/BaseTools/Source/Python/AutoGen/GenMake.py @@ -1054,6 +1054,12 @@ cleanlib: TargetDict = {"target": self.PlaceMacro(T.Target.Path, self.Macros), "cmd": "\n\t".join(T.Commands),"deps": Deps} self.BuildTargetList.append(self._BUILD_TARGET_TEMPLATE.Replace(TargetDict)) + # Add a Makefile rule for targets generating multiple files. + # The main output is a prerequisite for the other output files. + for i in T.Outputs[1:]: + AnnexeTargetDict = {"target": + self.PlaceMacro(i.Path, self.Macros), "cmd": "", "deps": + self.PlaceMacro(T.Target.Path, self.Macros)} + + self.BuildTargetList.append(self._BUILD_TARGET_TEMPLATE.Replace(Annexe + TargetDict)) + def ParserCCodeFile(self, T, Type, CmdSumDict, CmdTargetDict, CmdCppDict, DependencyDict): if not CmdSumDict: for item in self._AutoGenObject.Targets[Type]: diff --git a/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py b/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py index aad591de65f086043d55aeea5661f59c53792e7c..82facdb2774c56833718317f4414e0c1861de47f 100755 --- a/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py +++ b/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py @@ -860,7 +860,8 @@ class ModuleAutoGen(AutoGen): SubDirectory = os.path.join(self.OutputDir, File.SubDir) if not os.path.exists(SubDirectory): CreateDirectory(SubDirectory) - LastTarget = None + TargetList = set() + FinalTargetName = set() RuleChain = set() SourceList = [File] Index = 0 @@ -872,7 +873,7 @@ class ModuleAutoGen(AutoGen): while Index < len(SourceList): Source = SourceList[Index] Index = Index + 1 - + FileType = TAB_UNKNOWN_FILE if Source != File: CreateDirectory(Source.Dir) @@ -881,34 +882,27 @@ class ModuleAutoGen(AutoGen): if not self.IsLibrary: continue RuleObject = self.BuildRules[TAB_DEFAULT_BINARY_FILE] - elif FileType in self.BuildRules: - RuleObject = self.BuildRules[FileType] elif Source.Ext in self.BuildRules: RuleObject = self.BuildRules[Source.Ext] else: - # stop at no more rules - if LastTarget: - self._FinalBuildTargetList.add(LastTarget) - break - + # No more rule to apply: Source is a final target. + FinalTargetName.add(Source) + continue FileType = RuleObject.SourceFileType self._FileTypes[FileType].add(Source) # stop at STATIC_LIBRARY for library if self.IsLibrary and FileType == TAB_STATIC_LIBRARY: - if LastTarget: - self._FinalBuildTargetList.add(LastTarget) - break + FinalTargetName.add(Source) + continue Target = RuleObject.Apply(Source, self.BuildRuleOrder) if not Target: - if LastTarget: - self._FinalBuildTargetList.add(LastTarget) - break - elif not Target.Outputs: - # Only do build for target with outputs - self._FinalBuildTargetList.add(Target) + # No Target: Source is a final target. + FinalTargetName.add(Source) + continue + TargetList.add(Target) self._BuildTargets[FileType].add(Target) if not Source.IsBinary and Source == File: @@ -916,12 +910,16 @@ class ModuleAutoGen(AutoGen): # to avoid cyclic rule if FileType in RuleChain: - break + EdkLogger.error("build", ERROR_STATEMENT, "Cyclic + dependency detected while generating rule for %s" % str(Source)) RuleChain.add(FileType) SourceList.extend(Target.Outputs) - LastTarget = Target - FileType = TAB_UNKNOWN_FILE + + # For each final target, retrieve the TargetDescBlock instance. + for FTargetName in FinalTargetName: + for Target in TargetList: + if FTargetName == Target.Target: + self._FinalBuildTargetList.add(Target) @cached_property def Targets(self): -- 'Guid(CE165669-3EF3-493F-B85D-6190EE5B9759)' ^ permalink raw reply related [flat|nested] 17+ messages in thread
* Re: [edk2-devel] [PATCH v1 1/3] BaseTools: Generate multiple rules when multiple output files 2020-06-02 13:03 ` PierreGondois @ 2020-06-08 14:00 ` PierreGondois 2020-06-10 3:58 ` Bob Feng 0 siblings, 1 reply; 17+ messages in thread From: PierreGondois @ 2020-06-08 14:00 UTC (permalink / raw) To: Feng, Bob C, devel@edk2.groups.io Cc: Gao, Liming, Sami Mujawar, Tomas Pilar, nd Hello Bob, Should I modify the patch ? Regards, Pierre -----Original Message----- From: Pierre Gondois Sent: Tuesday, June 2, 2020 2:04 PM To: Feng, Bob C <bob.c.feng@intel.com>; devel@edk2.groups.io Cc: Gao, Liming <liming.gao@intel.com>; Sami Mujawar <Sami.Mujawar@arm.com>; Tomas Pilar <Tomas.Pilar@arm.com>; nd <nd@arm.com> Subject: RE: [edk2-devel] [PATCH v1 1/3] BaseTools: Generate multiple rules when multiple output files Hello Bob, Thank you for your answer, I put my comments as [Pierre], Regards, Pierre -----Original Message----- From: Feng, Bob C <bob.c.feng@intel.com> Sent: 02 June 2020 12:16 To: devel@edk2.groups.io; Pierre Gondois <Pierre.Gondois@arm.com> Cc: Gao, Liming <liming.gao@intel.com>; Sami Mujawar <Sami.Mujawar@arm.com>; Tomas Pilar <Tomas.Pilar@arm.com>; nd <nd@arm.com> Subject: RE: [edk2-devel] [PATCH v1 1/3] BaseTools: Generate multiple rules when multiple output files My comments are inline marked as [Bob]. Thanks, Bob -----Original Message----- From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of PierreGondois Sent: Monday, May 18, 2020 10:11 PM To: devel@edk2.groups.io Cc: Pierre Gondois <Pierre.Gondois@arm.com>; Feng, Bob C <bob.c.feng@intel.com>; Gao, Liming <liming.gao@intel.com>; Sami.Mujawar@arm.com; Tomas.Pilar@arm.com; nd@arm.com Subject: [edk2-devel] [PATCH v1 1/3] BaseTools: Generate multiple rules when multiple output files From: Pierre Gondois <pierre.gondois@arm.com> BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=2425 This patch modifies the Makefile generation not to stop adding Makfile rules when the first final target is found. E.g.: If the following rules are described in build_rule.txt: -[Rule1]: .X files generate .Y and .Z files; -[Rule2]: .Z files generate .Z1 files. Currently, if a File1.X file was part of the sources of a module, only [Rule1] would be generated in the Makefile. Indeed, there are no rules to apply to .Y files: .Y files are a final target. However, there is still [Rule2] to apply to .Z files. [Bob] I think currently a rule's output file will be added back to source file list, and in the later loop, that output file will be handled by another rule. Doesn't that algorithm handle your case above? [Pierre] The rule's output file was effectively added to the list of source files, and a rule was searched for this output file. However, the loop stopped when the first final target was found. By final target I mean "a file that isn't the input of a rule". For the asl/aml/amli case, this meant that: -(first loop iteration: treating the ASL file) The rule for ASL files was found and applied to the input ASL file. The AML and ".amli" files were added to the list of source files to look for a rule to apply to them. The loop continues. -(second loop iteration: treating the AML file) There is no rule for AML files. This means that the AML file is a final target. The loop ends without having applied the rule on the ".amli" file. This is why all the "break" instructions of the loop have been replaced with "continue". This prevents the loop from exiting without having treated all the files in the "SourceList". This patch also adds a dependency between the first ouput file of a rule and the other output files. For instance, with the same example as above, File1.Y and File1.Z are generated by the following rule: File1.Y: File1.X <Generate File1.Y> <Generate File1.Z> and the new dependency is: File1.Z: File1.Y This is necessary to keep a dependency order during the execution of the Makefile. Indeed, .Y and .Z files are generated by the execution of a common set of commands, and without this rule, there is no explicit dependency relation between them. [Bob] If there are 3 output files, for example: <OutputFile> A B C Will the dependency relationship be B: A C: B [Pierre] Currently, the dependency relation that would be created is: B: A C: A This can be changed to the dependency you described. Signed-off-by: Pierre Gondois <Pierre.Gondois@arm.com> --- The changes can be seen at: https://github.com/PierreARM/edk2/tree/803_Compile_AML_bytecode_array_into_OBJ_file_v1 Notes: v1: - Generate multiple rules when multiple output files are specified in the build_rule.txt file. [Pierre] BaseTools/Source/Python/AutoGen/GenMake.py | 6 +++ BaseTools/Source/Python/AutoGen/ModuleAutoGen.py | 40 ++++++++++---------- 2 files changed, 25 insertions(+), 21 deletions(-) diff --git a/BaseTools/Source/Python/AutoGen/GenMake.py b/BaseTools/Source/Python/AutoGen/GenMake.py index bbb3c29446f53fa7f2cb61a216a5b119f72c3fbc..0314d0ea34d99a014379e8d30c46ac0f0a7068ce 100755 --- a/BaseTools/Source/Python/AutoGen/GenMake.py +++ b/BaseTools/Source/Python/AutoGen/GenMake.py @@ -1054,6 +1054,12 @@ cleanlib: TargetDict = {"target": self.PlaceMacro(T.Target.Path, self.Macros), "cmd": "\n\t".join(T.Commands),"deps": Deps} self.BuildTargetList.append(self._BUILD_TARGET_TEMPLATE.Replace(TargetDict)) + # Add a Makefile rule for targets generating multiple files. + # The main output is a prerequisite for the other output files. + for i in T.Outputs[1:]: + AnnexeTargetDict = {"target": + self.PlaceMacro(i.Path, self.Macros), "cmd": "", "deps": + self.PlaceMacro(T.Target.Path, self.Macros)} + + self.BuildTargetList.append(self._BUILD_TARGET_TEMPLATE.Replace(Annexe + TargetDict)) + def ParserCCodeFile(self, T, Type, CmdSumDict, CmdTargetDict, CmdCppDict, DependencyDict): if not CmdSumDict: for item in self._AutoGenObject.Targets[Type]: diff --git a/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py b/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py index aad591de65f086043d55aeea5661f59c53792e7c..82facdb2774c56833718317f4414e0c1861de47f 100755 --- a/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py +++ b/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py @@ -860,7 +860,8 @@ class ModuleAutoGen(AutoGen): SubDirectory = os.path.join(self.OutputDir, File.SubDir) if not os.path.exists(SubDirectory): CreateDirectory(SubDirectory) - LastTarget = None + TargetList = set() + FinalTargetName = set() RuleChain = set() SourceList = [File] Index = 0 @@ -872,7 +873,7 @@ class ModuleAutoGen(AutoGen): while Index < len(SourceList): Source = SourceList[Index] Index = Index + 1 - + FileType = TAB_UNKNOWN_FILE if Source != File: CreateDirectory(Source.Dir) @@ -881,34 +882,27 @@ class ModuleAutoGen(AutoGen): if not self.IsLibrary: continue RuleObject = self.BuildRules[TAB_DEFAULT_BINARY_FILE] - elif FileType in self.BuildRules: - RuleObject = self.BuildRules[FileType] elif Source.Ext in self.BuildRules: RuleObject = self.BuildRules[Source.Ext] else: - # stop at no more rules - if LastTarget: - self._FinalBuildTargetList.add(LastTarget) - break - + # No more rule to apply: Source is a final target. + FinalTargetName.add(Source) + continue FileType = RuleObject.SourceFileType self._FileTypes[FileType].add(Source) # stop at STATIC_LIBRARY for library if self.IsLibrary and FileType == TAB_STATIC_LIBRARY: - if LastTarget: - self._FinalBuildTargetList.add(LastTarget) - break + FinalTargetName.add(Source) + continue Target = RuleObject.Apply(Source, self.BuildRuleOrder) if not Target: - if LastTarget: - self._FinalBuildTargetList.add(LastTarget) - break - elif not Target.Outputs: - # Only do build for target with outputs - self._FinalBuildTargetList.add(Target) + # No Target: Source is a final target. + FinalTargetName.add(Source) + continue + TargetList.add(Target) self._BuildTargets[FileType].add(Target) if not Source.IsBinary and Source == File: @@ -916,12 +910,16 @@ class ModuleAutoGen(AutoGen): # to avoid cyclic rule if FileType in RuleChain: - break + EdkLogger.error("build", ERROR_STATEMENT, "Cyclic + dependency detected while generating rule for %s" % str(Source)) RuleChain.add(FileType) SourceList.extend(Target.Outputs) - LastTarget = Target - FileType = TAB_UNKNOWN_FILE + + # For each final target, retrieve the TargetDescBlock instance. + for FTargetName in FinalTargetName: + for Target in TargetList: + if FTargetName == Target.Target: + self._FinalBuildTargetList.add(Target) @cached_property def Targets(self): -- 'Guid(CE165669-3EF3-493F-B85D-6190EE5B9759)' ^ permalink raw reply related [flat|nested] 17+ messages in thread
* Re: [edk2-devel] [PATCH v1 1/3] BaseTools: Generate multiple rules when multiple output files 2020-06-08 14:00 ` PierreGondois @ 2020-06-10 3:58 ` Bob Feng 2020-06-15 15:57 ` PierreGondois 0 siblings, 1 reply; 17+ messages in thread From: Bob Feng @ 2020-06-10 3:58 UTC (permalink / raw) To: Pierre Gondois, devel@edk2.groups.io Cc: Gao, Liming, Sami Mujawar, Tomas Pilar, nd Hi Pierre, I found there is an incorrect target generated in the OvmfPkg/AcpiTables Makefile when I tried to build Ovmf. That incorrect target causes ovmf to build failed. $(DEBUG_DIR)\PlatformAcpiTables : $(MAKE_FILE) $(DEBUG_DIR)\PlatformAcpiTables : $(STATIC_LIBRARY_FILES) $(DEBUG_DIR)\PlatformAcpiTables : $(STATIC_LIBRARY_FILES_LIST) "$(DLINK)" $(DLINK_FLAGS) $(DLINK_SPATH) @$(STATIC_LIBRARY_FILES_LIST) Regarding this patch, I think it changes the logic of the _ApplyBuildRule, replacing "break" with "continue" and removing some if and elseif blocks, so it would be hard for me to make sure your current logic can cover all the original cases. Would you show me how many testing you have done? Thanks, Bob -----Original Message----- From: Pierre Gondois <Pierre.Gondois@arm.com> Sent: Monday, June 8, 2020 10:01 PM To: Feng, Bob C <bob.c.feng@intel.com>; devel@edk2.groups.io Cc: Gao, Liming <liming.gao@intel.com>; Sami Mujawar <Sami.Mujawar@arm.com>; Tomas Pilar <Tomas.Pilar@arm.com>; nd <nd@arm.com> Subject: RE: [edk2-devel] [PATCH v1 1/3] BaseTools: Generate multiple rules when multiple output files Hello Bob, Should I modify the patch ? Regards, Pierre -----Original Message----- From: Pierre Gondois Sent: Tuesday, June 2, 2020 2:04 PM To: Feng, Bob C <bob.c.feng@intel.com>; devel@edk2.groups.io Cc: Gao, Liming <liming.gao@intel.com>; Sami Mujawar <Sami.Mujawar@arm.com>; Tomas Pilar <Tomas.Pilar@arm.com>; nd <nd@arm.com> Subject: RE: [edk2-devel] [PATCH v1 1/3] BaseTools: Generate multiple rules when multiple output files Hello Bob, Thank you for your answer, I put my comments as [Pierre], Regards, Pierre -----Original Message----- From: Feng, Bob C <bob.c.feng@intel.com> Sent: 02 June 2020 12:16 To: devel@edk2.groups.io; Pierre Gondois <Pierre.Gondois@arm.com> Cc: Gao, Liming <liming.gao@intel.com>; Sami Mujawar <Sami.Mujawar@arm.com>; Tomas Pilar <Tomas.Pilar@arm.com>; nd <nd@arm.com> Subject: RE: [edk2-devel] [PATCH v1 1/3] BaseTools: Generate multiple rules when multiple output files My comments are inline marked as [Bob]. Thanks, Bob -----Original Message----- From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of PierreGondois Sent: Monday, May 18, 2020 10:11 PM To: devel@edk2.groups.io Cc: Pierre Gondois <Pierre.Gondois@arm.com>; Feng, Bob C <bob.c.feng@intel.com>; Gao, Liming <liming.gao@intel.com>; Sami.Mujawar@arm.com; Tomas.Pilar@arm.com; nd@arm.com Subject: [edk2-devel] [PATCH v1 1/3] BaseTools: Generate multiple rules when multiple output files From: Pierre Gondois <pierre.gondois@arm.com> BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=2425 This patch modifies the Makefile generation not to stop adding Makfile rules when the first final target is found. E.g.: If the following rules are described in build_rule.txt: -[Rule1]: .X files generate .Y and .Z files; -[Rule2]: .Z files generate .Z1 files. Currently, if a File1.X file was part of the sources of a module, only [Rule1] would be generated in the Makefile. Indeed, there are no rules to apply to .Y files: .Y files are a final target. However, there is still [Rule2] to apply to .Z files. [Bob] I think currently a rule's output file will be added back to source file list, and in the later loop, that output file will be handled by another rule. Doesn't that algorithm handle your case above? [Pierre] The rule's output file was effectively added to the list of source files, and a rule was searched for this output file. However, the loop stopped when the first final target was found. By final target I mean "a file that isn't the input of a rule". For the asl/aml/amli case, this meant that: -(first loop iteration: treating the ASL file) The rule for ASL files was found and applied to the input ASL file. The AML and ".amli" files were added to the list of source files to look for a rule to apply to them. The loop continues. -(second loop iteration: treating the AML file) There is no rule for AML files. This means that the AML file is a final target. The loop ends without having applied the rule on the ".amli" file. This is why all the "break" instructions of the loop have been replaced with "continue". This prevents the loop from exiting without having treated all the files in the "SourceList". This patch also adds a dependency between the first ouput file of a rule and the other output files. For instance, with the same example as above, File1.Y and File1.Z are generated by the following rule: File1.Y: File1.X <Generate File1.Y> <Generate File1.Z> and the new dependency is: File1.Z: File1.Y This is necessary to keep a dependency order during the execution of the Makefile. Indeed, .Y and .Z files are generated by the execution of a common set of commands, and without this rule, there is no explicit dependency relation between them. [Bob] If there are 3 output files, for example: <OutputFile> A B C Will the dependency relationship be B: A C: B [Pierre] Currently, the dependency relation that would be created is: B: A C: A This can be changed to the dependency you described. Signed-off-by: Pierre Gondois <Pierre.Gondois@arm.com> --- The changes can be seen at: https://github.com/PierreARM/edk2/tree/803_Compile_AML_bytecode_array_into_OBJ_file_v1 Notes: v1: - Generate multiple rules when multiple output files are specified in the build_rule.txt file. [Pierre] BaseTools/Source/Python/AutoGen/GenMake.py | 6 +++ BaseTools/Source/Python/AutoGen/ModuleAutoGen.py | 40 ++++++++++---------- 2 files changed, 25 insertions(+), 21 deletions(-) diff --git a/BaseTools/Source/Python/AutoGen/GenMake.py b/BaseTools/Source/Python/AutoGen/GenMake.py index bbb3c29446f53fa7f2cb61a216a5b119f72c3fbc..0314d0ea34d99a014379e8d30c46ac0f0a7068ce 100755 --- a/BaseTools/Source/Python/AutoGen/GenMake.py +++ b/BaseTools/Source/Python/AutoGen/GenMake.py @@ -1054,6 +1054,12 @@ cleanlib: TargetDict = {"target": self.PlaceMacro(T.Target.Path, self.Macros), "cmd": "\n\t".join(T.Commands),"deps": Deps} self.BuildTargetList.append(self._BUILD_TARGET_TEMPLATE.Replace(TargetDict)) + # Add a Makefile rule for targets generating multiple files. + # The main output is a prerequisite for the other output files. + for i in T.Outputs[1:]: + AnnexeTargetDict = {"target": + self.PlaceMacro(i.Path, self.Macros), "cmd": "", "deps": + self.PlaceMacro(T.Target.Path, self.Macros)} + + self.BuildTargetList.append(self._BUILD_TARGET_TEMPLATE.Replace(Annexe + TargetDict)) + def ParserCCodeFile(self, T, Type, CmdSumDict, CmdTargetDict, CmdCppDict, DependencyDict): if not CmdSumDict: for item in self._AutoGenObject.Targets[Type]: diff --git a/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py b/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py index aad591de65f086043d55aeea5661f59c53792e7c..82facdb2774c56833718317f4414e0c1861de47f 100755 --- a/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py +++ b/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py @@ -860,7 +860,8 @@ class ModuleAutoGen(AutoGen): SubDirectory = os.path.join(self.OutputDir, File.SubDir) if not os.path.exists(SubDirectory): CreateDirectory(SubDirectory) - LastTarget = None + TargetList = set() + FinalTargetName = set() RuleChain = set() SourceList = [File] Index = 0 @@ -872,7 +873,7 @@ class ModuleAutoGen(AutoGen): while Index < len(SourceList): Source = SourceList[Index] Index = Index + 1 - + FileType = TAB_UNKNOWN_FILE if Source != File: CreateDirectory(Source.Dir) @@ -881,34 +882,27 @@ class ModuleAutoGen(AutoGen): if not self.IsLibrary: continue RuleObject = self.BuildRules[TAB_DEFAULT_BINARY_FILE] - elif FileType in self.BuildRules: - RuleObject = self.BuildRules[FileType] elif Source.Ext in self.BuildRules: RuleObject = self.BuildRules[Source.Ext] else: - # stop at no more rules - if LastTarget: - self._FinalBuildTargetList.add(LastTarget) - break - + # No more rule to apply: Source is a final target. + FinalTargetName.add(Source) + continue FileType = RuleObject.SourceFileType self._FileTypes[FileType].add(Source) # stop at STATIC_LIBRARY for library if self.IsLibrary and FileType == TAB_STATIC_LIBRARY: - if LastTarget: - self._FinalBuildTargetList.add(LastTarget) - break + FinalTargetName.add(Source) + continue Target = RuleObject.Apply(Source, self.BuildRuleOrder) if not Target: - if LastTarget: - self._FinalBuildTargetList.add(LastTarget) - break - elif not Target.Outputs: - # Only do build for target with outputs - self._FinalBuildTargetList.add(Target) + # No Target: Source is a final target. + FinalTargetName.add(Source) + continue + TargetList.add(Target) self._BuildTargets[FileType].add(Target) if not Source.IsBinary and Source == File: @@ -916,12 +910,16 @@ class ModuleAutoGen(AutoGen): # to avoid cyclic rule if FileType in RuleChain: - break + EdkLogger.error("build", ERROR_STATEMENT, "Cyclic + dependency detected while generating rule for %s" % str(Source)) RuleChain.add(FileType) SourceList.extend(Target.Outputs) - LastTarget = Target - FileType = TAB_UNKNOWN_FILE + + # For each final target, retrieve the TargetDescBlock instance. + for FTargetName in FinalTargetName: + for Target in TargetList: + if FTargetName == Target.Target: + self._FinalBuildTargetList.add(Target) @cached_property def Targets(self): -- 'Guid(CE165669-3EF3-493F-B85D-6190EE5B9759)' ^ permalink raw reply related [flat|nested] 17+ messages in thread
* Re: [edk2-devel] [PATCH v1 1/3] BaseTools: Generate multiple rules when multiple output files 2020-06-10 3:58 ` Bob Feng @ 2020-06-15 15:57 ` PierreGondois 2020-06-16 8:48 ` Bob Feng 0 siblings, 1 reply; 17+ messages in thread From: PierreGondois @ 2020-06-15 15:57 UTC (permalink / raw) To: Feng, Bob C, devel@edk2.groups.io Cc: Gao, Liming, Sami Mujawar, Tomas Pilar, nd Hello Bob, I have locally corrected the patch. On non-arm architecture, I had previously tested it for the following configurations, but this wasn't enough apparently. I am currently testing it on more platforms and I will send you the list of the tested platforms along with the V2. Plarform="Platform/Intel/MinPlatformPkg/MinPlatformPkg.dsc"|"Platform/Intel/BoardModulePkg/BoardModulePkg.dsc" Host=Windows|Linux BuildType=DEBUG|RELEASE Toolchain=VS2017|GCC5 Arch=X64|IA32 Before submitting a V2, I am planning to create a pull request on the edk2 github repository, allowing me to check whether it passes the CI tests. Would it be fine for you? About your comment about changing the logic of ApplyBuildRule, the logic of _ApplyBuildRule currently breaks out of the loop whenever a final target is found. The new rule on ".amli" files needs to continue looking for rules to apply, otherwise the processing on the initial ASL file stops when the AML file is generated (thus, not continuing the path ".amli"->".c"->".obj"). I agree this is hard to check, but I believe the edk2 CI tests should allow to put more trust in the patch serie. Regards, Pierre -----Original Message----- From: Feng, Bob C <bob.c.feng@intel.com> Sent: 10 June 2020 04:58 To: Pierre Gondois <Pierre.Gondois@arm.com>; devel@edk2.groups.io Cc: Gao, Liming <liming.gao@intel.com>; Sami Mujawar <Sami.Mujawar@arm.com>; Tomas Pilar <Tomas.Pilar@arm.com>; nd <nd@arm.com> Subject: RE: [edk2-devel] [PATCH v1 1/3] BaseTools: Generate multiple rules when multiple output files Hi Pierre, I found there is an incorrect target generated in the OvmfPkg/AcpiTables Makefile when I tried to build Ovmf. That incorrect target causes ovmf to build failed. $(DEBUG_DIR)\PlatformAcpiTables : $(MAKE_FILE) $(DEBUG_DIR)\PlatformAcpiTables : $(STATIC_LIBRARY_FILES) $(DEBUG_DIR)\PlatformAcpiTables : $(STATIC_LIBRARY_FILES_LIST) "$(DLINK)" $(DLINK_FLAGS) $(DLINK_SPATH) @$(STATIC_LIBRARY_FILES_LIST) Regarding this patch, I think it changes the logic of the _ApplyBuildRule, replacing "break" with "continue" and removing some if and elseif blocks, so it would be hard for me to make sure your current logic can cover all the original cases. Would you show me how many testing you have done? Thanks, Bob -----Original Message----- From: Pierre Gondois <Pierre.Gondois@arm.com> Sent: Monday, June 8, 2020 10:01 PM To: Feng, Bob C <bob.c.feng@intel.com>; devel@edk2.groups.io Cc: Gao, Liming <liming.gao@intel.com>; Sami Mujawar <Sami.Mujawar@arm.com>; Tomas Pilar <Tomas.Pilar@arm.com>; nd <nd@arm.com> Subject: RE: [edk2-devel] [PATCH v1 1/3] BaseTools: Generate multiple rules when multiple output files Hello Bob, Should I modify the patch ? Regards, Pierre -----Original Message----- From: Pierre Gondois Sent: Tuesday, June 2, 2020 2:04 PM To: Feng, Bob C <bob.c.feng@intel.com>; devel@edk2.groups.io Cc: Gao, Liming <liming.gao@intel.com>; Sami Mujawar <Sami.Mujawar@arm.com>; Tomas Pilar <Tomas.Pilar@arm.com>; nd <nd@arm.com> Subject: RE: [edk2-devel] [PATCH v1 1/3] BaseTools: Generate multiple rules when multiple output files Hello Bob, Thank you for your answer, I put my comments as [Pierre], Regards, Pierre -----Original Message----- From: Feng, Bob C <bob.c.feng@intel.com> Sent: 02 June 2020 12:16 To: devel@edk2.groups.io; Pierre Gondois <Pierre.Gondois@arm.com> Cc: Gao, Liming <liming.gao@intel.com>; Sami Mujawar <Sami.Mujawar@arm.com>; Tomas Pilar <Tomas.Pilar@arm.com>; nd <nd@arm.com> Subject: RE: [edk2-devel] [PATCH v1 1/3] BaseTools: Generate multiple rules when multiple output files My comments are inline marked as [Bob]. Thanks, Bob -----Original Message----- From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of PierreGondois Sent: Monday, May 18, 2020 10:11 PM To: devel@edk2.groups.io Cc: Pierre Gondois <Pierre.Gondois@arm.com>; Feng, Bob C <bob.c.feng@intel.com>; Gao, Liming <liming.gao@intel.com>; Sami.Mujawar@arm.com; Tomas.Pilar@arm.com; nd@arm.com Subject: [edk2-devel] [PATCH v1 1/3] BaseTools: Generate multiple rules when multiple output files From: Pierre Gondois <pierre.gondois@arm.com> BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=2425 This patch modifies the Makefile generation not to stop adding Makfile rules when the first final target is found. E.g.: If the following rules are described in build_rule.txt: -[Rule1]: .X files generate .Y and .Z files; -[Rule2]: .Z files generate .Z1 files. Currently, if a File1.X file was part of the sources of a module, only [Rule1] would be generated in the Makefile. Indeed, there are no rules to apply to .Y files: .Y files are a final target. However, there is still [Rule2] to apply to .Z files. [Bob] I think currently a rule's output file will be added back to source file list, and in the later loop, that output file will be handled by another rule. Doesn't that algorithm handle your case above? [Pierre] The rule's output file was effectively added to the list of source files, and a rule was searched for this output file. However, the loop stopped when the first final target was found. By final target I mean "a file that isn't the input of a rule". For the asl/aml/amli case, this meant that: -(first loop iteration: treating the ASL file) The rule for ASL files was found and applied to the input ASL file. The AML and ".amli" files were added to the list of source files to look for a rule to apply to them. The loop continues. -(second loop iteration: treating the AML file) There is no rule for AML files. This means that the AML file is a final target. The loop ends without having applied the rule on the ".amli" file. This is why all the "break" instructions of the loop have been replaced with "continue". This prevents the loop from exiting without having treated all the files in the "SourceList". This patch also adds a dependency between the first ouput file of a rule and the other output files. For instance, with the same example as above, File1.Y and File1.Z are generated by the following rule: File1.Y: File1.X <Generate File1.Y> <Generate File1.Z> and the new dependency is: File1.Z: File1.Y This is necessary to keep a dependency order during the execution of the Makefile. Indeed, .Y and .Z files are generated by the execution of a common set of commands, and without this rule, there is no explicit dependency relation between them. [Bob] If there are 3 output files, for example: <OutputFile> A B C Will the dependency relationship be B: A C: B [Pierre] Currently, the dependency relation that would be created is: B: A C: A This can be changed to the dependency you described. Signed-off-by: Pierre Gondois <Pierre.Gondois@arm.com> --- The changes can be seen at: https://github.com/PierreARM/edk2/tree/803_Compile_AML_bytecode_array_into_OBJ_file_v1 Notes: v1: - Generate multiple rules when multiple output files are specified in the build_rule.txt file. [Pierre] BaseTools/Source/Python/AutoGen/GenMake.py | 6 +++ BaseTools/Source/Python/AutoGen/ModuleAutoGen.py | 40 ++++++++++---------- 2 files changed, 25 insertions(+), 21 deletions(-) diff --git a/BaseTools/Source/Python/AutoGen/GenMake.py b/BaseTools/Source/Python/AutoGen/GenMake.py index bbb3c29446f53fa7f2cb61a216a5b119f72c3fbc..0314d0ea34d99a014379e8d30c46ac0f0a7068ce 100755 --- a/BaseTools/Source/Python/AutoGen/GenMake.py +++ b/BaseTools/Source/Python/AutoGen/GenMake.py @@ -1054,6 +1054,12 @@ cleanlib: TargetDict = {"target": self.PlaceMacro(T.Target.Path, self.Macros), "cmd": "\n\t".join(T.Commands),"deps": Deps} self.BuildTargetList.append(self._BUILD_TARGET_TEMPLATE.Replace(TargetDict)) + # Add a Makefile rule for targets generating multiple files. + # The main output is a prerequisite for the other output files. + for i in T.Outputs[1:]: + AnnexeTargetDict = {"target": + self.PlaceMacro(i.Path, self.Macros), "cmd": "", "deps": + self.PlaceMacro(T.Target.Path, self.Macros)} + + self.BuildTargetList.append(self._BUILD_TARGET_TEMPLATE.Replace(Annexe + TargetDict)) + def ParserCCodeFile(self, T, Type, CmdSumDict, CmdTargetDict, CmdCppDict, DependencyDict): if not CmdSumDict: for item in self._AutoGenObject.Targets[Type]: diff --git a/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py b/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py index aad591de65f086043d55aeea5661f59c53792e7c..82facdb2774c56833718317f4414e0c1861de47f 100755 --- a/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py +++ b/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py @@ -860,7 +860,8 @@ class ModuleAutoGen(AutoGen): SubDirectory = os.path.join(self.OutputDir, File.SubDir) if not os.path.exists(SubDirectory): CreateDirectory(SubDirectory) - LastTarget = None + TargetList = set() + FinalTargetName = set() RuleChain = set() SourceList = [File] Index = 0 @@ -872,7 +873,7 @@ class ModuleAutoGen(AutoGen): while Index < len(SourceList): Source = SourceList[Index] Index = Index + 1 - + FileType = TAB_UNKNOWN_FILE if Source != File: CreateDirectory(Source.Dir) @@ -881,34 +882,27 @@ class ModuleAutoGen(AutoGen): if not self.IsLibrary: continue RuleObject = self.BuildRules[TAB_DEFAULT_BINARY_FILE] - elif FileType in self.BuildRules: - RuleObject = self.BuildRules[FileType] elif Source.Ext in self.BuildRules: RuleObject = self.BuildRules[Source.Ext] else: - # stop at no more rules - if LastTarget: - self._FinalBuildTargetList.add(LastTarget) - break - + # No more rule to apply: Source is a final target. + FinalTargetName.add(Source) + continue FileType = RuleObject.SourceFileType self._FileTypes[FileType].add(Source) # stop at STATIC_LIBRARY for library if self.IsLibrary and FileType == TAB_STATIC_LIBRARY: - if LastTarget: - self._FinalBuildTargetList.add(LastTarget) - break + FinalTargetName.add(Source) + continue Target = RuleObject.Apply(Source, self.BuildRuleOrder) if not Target: - if LastTarget: - self._FinalBuildTargetList.add(LastTarget) - break - elif not Target.Outputs: - # Only do build for target with outputs - self._FinalBuildTargetList.add(Target) + # No Target: Source is a final target. + FinalTargetName.add(Source) + continue + TargetList.add(Target) self._BuildTargets[FileType].add(Target) if not Source.IsBinary and Source == File: @@ -916,12 +910,16 @@ class ModuleAutoGen(AutoGen): # to avoid cyclic rule if FileType in RuleChain: - break + EdkLogger.error("build", ERROR_STATEMENT, "Cyclic + dependency detected while generating rule for %s" % str(Source)) RuleChain.add(FileType) SourceList.extend(Target.Outputs) - LastTarget = Target - FileType = TAB_UNKNOWN_FILE + + # For each final target, retrieve the TargetDescBlock instance. + for FTargetName in FinalTargetName: + for Target in TargetList: + if FTargetName == Target.Target: + self._FinalBuildTargetList.add(Target) @cached_property def Targets(self): -- 'Guid(CE165669-3EF3-493F-B85D-6190EE5B9759)' ^ permalink raw reply related [flat|nested] 17+ messages in thread
* Re: [edk2-devel] [PATCH v1 1/3] BaseTools: Generate multiple rules when multiple output files 2020-06-15 15:57 ` PierreGondois @ 2020-06-16 8:48 ` Bob Feng 0 siblings, 0 replies; 17+ messages in thread From: Bob Feng @ 2020-06-16 8:48 UTC (permalink / raw) To: Pierre Gondois, devel@edk2.groups.io Cc: Gao, Liming, Sami Mujawar, Tomas Pilar, nd Hi Pierre, Yes. It's fine for me. Thanks, Bob -----Original Message----- From: Pierre Gondois <Pierre.Gondois@arm.com> Sent: Monday, June 15, 2020 11:58 PM To: Feng, Bob C <bob.c.feng@intel.com>; devel@edk2.groups.io Cc: Gao, Liming <liming.gao@intel.com>; Sami Mujawar <Sami.Mujawar@arm.com>; Tomas Pilar <Tomas.Pilar@arm.com>; nd <nd@arm.com> Subject: RE: [edk2-devel] [PATCH v1 1/3] BaseTools: Generate multiple rules when multiple output files Hello Bob, I have locally corrected the patch. On non-arm architecture, I had previously tested it for the following configurations, but this wasn't enough apparently. I am currently testing it on more platforms and I will send you the list of the tested platforms along with the V2. Plarform="Platform/Intel/MinPlatformPkg/MinPlatformPkg.dsc"|"Platform/Intel/BoardModulePkg/BoardModulePkg.dsc" Host=Windows|Linux BuildType=DEBUG|RELEASE Toolchain=VS2017|GCC5 Arch=X64|IA32 Before submitting a V2, I am planning to create a pull request on the edk2 github repository, allowing me to check whether it passes the CI tests. Would it be fine for you? About your comment about changing the logic of ApplyBuildRule, the logic of _ApplyBuildRule currently breaks out of the loop whenever a final target is found. The new rule on ".amli" files needs to continue looking for rules to apply, otherwise the processing on the initial ASL file stops when the AML file is generated (thus, not continuing the path ".amli"->".c"->".obj"). I agree this is hard to check, but I believe the edk2 CI tests should allow to put more trust in the patch serie. Regards, Pierre -----Original Message----- From: Feng, Bob C <bob.c.feng@intel.com> Sent: 10 June 2020 04:58 To: Pierre Gondois <Pierre.Gondois@arm.com>; devel@edk2.groups.io Cc: Gao, Liming <liming.gao@intel.com>; Sami Mujawar <Sami.Mujawar@arm.com>; Tomas Pilar <Tomas.Pilar@arm.com>; nd <nd@arm.com> Subject: RE: [edk2-devel] [PATCH v1 1/3] BaseTools: Generate multiple rules when multiple output files Hi Pierre, I found there is an incorrect target generated in the OvmfPkg/AcpiTables Makefile when I tried to build Ovmf. That incorrect target causes ovmf to build failed. $(DEBUG_DIR)\PlatformAcpiTables : $(MAKE_FILE) $(DEBUG_DIR)\PlatformAcpiTables : $(STATIC_LIBRARY_FILES) $(DEBUG_DIR)\PlatformAcpiTables : $(STATIC_LIBRARY_FILES_LIST) "$(DLINK)" $(DLINK_FLAGS) $(DLINK_SPATH) @$(STATIC_LIBRARY_FILES_LIST) Regarding this patch, I think it changes the logic of the _ApplyBuildRule, replacing "break" with "continue" and removing some if and elseif blocks, so it would be hard for me to make sure your current logic can cover all the original cases. Would you show me how many testing you have done? Thanks, Bob -----Original Message----- From: Pierre Gondois <Pierre.Gondois@arm.com> Sent: Monday, June 8, 2020 10:01 PM To: Feng, Bob C <bob.c.feng@intel.com>; devel@edk2.groups.io Cc: Gao, Liming <liming.gao@intel.com>; Sami Mujawar <Sami.Mujawar@arm.com>; Tomas Pilar <Tomas.Pilar@arm.com>; nd <nd@arm.com> Subject: RE: [edk2-devel] [PATCH v1 1/3] BaseTools: Generate multiple rules when multiple output files Hello Bob, Should I modify the patch ? Regards, Pierre -----Original Message----- From: Pierre Gondois Sent: Tuesday, June 2, 2020 2:04 PM To: Feng, Bob C <bob.c.feng@intel.com>; devel@edk2.groups.io Cc: Gao, Liming <liming.gao@intel.com>; Sami Mujawar <Sami.Mujawar@arm.com>; Tomas Pilar <Tomas.Pilar@arm.com>; nd <nd@arm.com> Subject: RE: [edk2-devel] [PATCH v1 1/3] BaseTools: Generate multiple rules when multiple output files Hello Bob, Thank you for your answer, I put my comments as [Pierre], Regards, Pierre -----Original Message----- From: Feng, Bob C <bob.c.feng@intel.com> Sent: 02 June 2020 12:16 To: devel@edk2.groups.io; Pierre Gondois <Pierre.Gondois@arm.com> Cc: Gao, Liming <liming.gao@intel.com>; Sami Mujawar <Sami.Mujawar@arm.com>; Tomas Pilar <Tomas.Pilar@arm.com>; nd <nd@arm.com> Subject: RE: [edk2-devel] [PATCH v1 1/3] BaseTools: Generate multiple rules when multiple output files My comments are inline marked as [Bob]. Thanks, Bob -----Original Message----- From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of PierreGondois Sent: Monday, May 18, 2020 10:11 PM To: devel@edk2.groups.io Cc: Pierre Gondois <Pierre.Gondois@arm.com>; Feng, Bob C <bob.c.feng@intel.com>; Gao, Liming <liming.gao@intel.com>; Sami.Mujawar@arm.com; Tomas.Pilar@arm.com; nd@arm.com Subject: [edk2-devel] [PATCH v1 1/3] BaseTools: Generate multiple rules when multiple output files From: Pierre Gondois <pierre.gondois@arm.com> BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=2425 This patch modifies the Makefile generation not to stop adding Makfile rules when the first final target is found. E.g.: If the following rules are described in build_rule.txt: -[Rule1]: .X files generate .Y and .Z files; -[Rule2]: .Z files generate .Z1 files. Currently, if a File1.X file was part of the sources of a module, only [Rule1] would be generated in the Makefile. Indeed, there are no rules to apply to .Y files: .Y files are a final target. However, there is still [Rule2] to apply to .Z files. [Bob] I think currently a rule's output file will be added back to source file list, and in the later loop, that output file will be handled by another rule. Doesn't that algorithm handle your case above? [Pierre] The rule's output file was effectively added to the list of source files, and a rule was searched for this output file. However, the loop stopped when the first final target was found. By final target I mean "a file that isn't the input of a rule". For the asl/aml/amli case, this meant that: -(first loop iteration: treating the ASL file) The rule for ASL files was found and applied to the input ASL file. The AML and ".amli" files were added to the list of source files to look for a rule to apply to them. The loop continues. -(second loop iteration: treating the AML file) There is no rule for AML files. This means that the AML file is a final target. The loop ends without having applied the rule on the ".amli" file. This is why all the "break" instructions of the loop have been replaced with "continue". This prevents the loop from exiting without having treated all the files in the "SourceList". This patch also adds a dependency between the first ouput file of a rule and the other output files. For instance, with the same example as above, File1.Y and File1.Z are generated by the following rule: File1.Y: File1.X <Generate File1.Y> <Generate File1.Z> and the new dependency is: File1.Z: File1.Y This is necessary to keep a dependency order during the execution of the Makefile. Indeed, .Y and .Z files are generated by the execution of a common set of commands, and without this rule, there is no explicit dependency relation between them. [Bob] If there are 3 output files, for example: <OutputFile> A B C Will the dependency relationship be B: A C: B [Pierre] Currently, the dependency relation that would be created is: B: A C: A This can be changed to the dependency you described. Signed-off-by: Pierre Gondois <Pierre.Gondois@arm.com> --- The changes can be seen at: https://github.com/PierreARM/edk2/tree/803_Compile_AML_bytecode_array_into_OBJ_file_v1 Notes: v1: - Generate multiple rules when multiple output files are specified in the build_rule.txt file. [Pierre] BaseTools/Source/Python/AutoGen/GenMake.py | 6 +++ BaseTools/Source/Python/AutoGen/ModuleAutoGen.py | 40 ++++++++++---------- 2 files changed, 25 insertions(+), 21 deletions(-) diff --git a/BaseTools/Source/Python/AutoGen/GenMake.py b/BaseTools/Source/Python/AutoGen/GenMake.py index bbb3c29446f53fa7f2cb61a216a5b119f72c3fbc..0314d0ea34d99a014379e8d30c46ac0f0a7068ce 100755 --- a/BaseTools/Source/Python/AutoGen/GenMake.py +++ b/BaseTools/Source/Python/AutoGen/GenMake.py @@ -1054,6 +1054,12 @@ cleanlib: TargetDict = {"target": self.PlaceMacro(T.Target.Path, self.Macros), "cmd": "\n\t".join(T.Commands),"deps": Deps} self.BuildTargetList.append(self._BUILD_TARGET_TEMPLATE.Replace(TargetDict)) + # Add a Makefile rule for targets generating multiple files. + # The main output is a prerequisite for the other output files. + for i in T.Outputs[1:]: + AnnexeTargetDict = {"target": + self.PlaceMacro(i.Path, self.Macros), "cmd": "", "deps": + self.PlaceMacro(T.Target.Path, self.Macros)} + + self.BuildTargetList.append(self._BUILD_TARGET_TEMPLATE.Replace(Annexe + TargetDict)) + def ParserCCodeFile(self, T, Type, CmdSumDict, CmdTargetDict, CmdCppDict, DependencyDict): if not CmdSumDict: for item in self._AutoGenObject.Targets[Type]: diff --git a/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py b/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py index aad591de65f086043d55aeea5661f59c53792e7c..82facdb2774c56833718317f4414e0c1861de47f 100755 --- a/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py +++ b/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py @@ -860,7 +860,8 @@ class ModuleAutoGen(AutoGen): SubDirectory = os.path.join(self.OutputDir, File.SubDir) if not os.path.exists(SubDirectory): CreateDirectory(SubDirectory) - LastTarget = None + TargetList = set() + FinalTargetName = set() RuleChain = set() SourceList = [File] Index = 0 @@ -872,7 +873,7 @@ class ModuleAutoGen(AutoGen): while Index < len(SourceList): Source = SourceList[Index] Index = Index + 1 - + FileType = TAB_UNKNOWN_FILE if Source != File: CreateDirectory(Source.Dir) @@ -881,34 +882,27 @@ class ModuleAutoGen(AutoGen): if not self.IsLibrary: continue RuleObject = self.BuildRules[TAB_DEFAULT_BINARY_FILE] - elif FileType in self.BuildRules: - RuleObject = self.BuildRules[FileType] elif Source.Ext in self.BuildRules: RuleObject = self.BuildRules[Source.Ext] else: - # stop at no more rules - if LastTarget: - self._FinalBuildTargetList.add(LastTarget) - break - + # No more rule to apply: Source is a final target. + FinalTargetName.add(Source) + continue FileType = RuleObject.SourceFileType self._FileTypes[FileType].add(Source) # stop at STATIC_LIBRARY for library if self.IsLibrary and FileType == TAB_STATIC_LIBRARY: - if LastTarget: - self._FinalBuildTargetList.add(LastTarget) - break + FinalTargetName.add(Source) + continue Target = RuleObject.Apply(Source, self.BuildRuleOrder) if not Target: - if LastTarget: - self._FinalBuildTargetList.add(LastTarget) - break - elif not Target.Outputs: - # Only do build for target with outputs - self._FinalBuildTargetList.add(Target) + # No Target: Source is a final target. + FinalTargetName.add(Source) + continue + TargetList.add(Target) self._BuildTargets[FileType].add(Target) if not Source.IsBinary and Source == File: @@ -916,12 +910,16 @@ class ModuleAutoGen(AutoGen): # to avoid cyclic rule if FileType in RuleChain: - break + EdkLogger.error("build", ERROR_STATEMENT, "Cyclic + dependency detected while generating rule for %s" % str(Source)) RuleChain.add(FileType) SourceList.extend(Target.Outputs) - LastTarget = Target - FileType = TAB_UNKNOWN_FILE + + # For each final target, retrieve the TargetDescBlock instance. + for FTargetName in FinalTargetName: + for Target in TargetList: + if FTargetName == Target.Target: + self._FinalBuildTargetList.add(Target) @cached_property def Targets(self): -- 'Guid(CE165669-3EF3-493F-B85D-6190EE5B9759)' ^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH v1 2/3] BaseTools: Rename AmlToHex script to AmlToC 2020-05-18 14:11 [PATCH v1 0/3] Compile AML bytecode array into OBJ file PierreGondois 2020-05-18 14:11 ` [PATCH v1 1/3] BaseTools: Generate multiple rules when multiple output files PierreGondois @ 2020-05-18 14:11 ` PierreGondois 2020-05-18 14:11 ` [PATCH v1 3/3] BaseTools: Compile AML bytecode arrays into .obj file PierreGondois ` (3 subsequent siblings) 5 siblings, 0 replies; 17+ messages in thread From: PierreGondois @ 2020-05-18 14:11 UTC (permalink / raw) To: devel; +Cc: Pierre Gondois, bob.c.feng, liming.gao, Sami.Mujawar, Tomas.Pilar, nd From: Pierre Gondois <pierre.gondois@arm.com> BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=2425 The AmlToHex script and Posix/WindowsLike wrappers convert an AML file to a .hex file, containing a C array storing AML bytecode. This ".hex" file can then be included in a C file, allowing to access the AML bytecode from this C file. The EDK2 build system doesn't allow to a depict dependency orders between files of different languages. For instance, in a module containing a ".c" file and a ".asl", the ".c" file may or may not be built prior to the ".asl" file. This prevents any inclusion of a generated ".hex" in a ".c" file since this later ".hex" file may or may not have been created yet. This patch renames the script as AmlToC. It is posted as a separate patch to prevent git from seeing the renaming as a deletion plus addition of a new file. The ending line of the posix-like bin-wrapper script has also been corrected. This is a first step toward generating a C file containing the AML bytecode from an ASL file. This C file will then be handled by the EDK2 build system to generate an object file. Thus, no file inclusion will be required anymore. The C file requiring the AML bytecode as a C array, and the ASL file, will be compiled independently. The C array must be defined as an external symbol. The linker is resolving the reference to the C array symbol. To summarize, the flow goes as: -1. ASL file is compiled to AML; -2. AML file is copied to a ".amli" intermediate file; -3. EDK2 build system applies the rule relevant to ".amli" files. This is, calling the "AmlToC" script, generating a C file from the ".amli" file; -4. EDK2 build system applies the rule relevant to C files. This is creating an object file. -5. EDK2 build system links the object file containing the AML bytecode with the object file requiring it. Signed-off-by: Pierre Gondois <Pierre.Gondois@arm.com> Signed-off-by: Tomas Pilar <Tomas.Pilar@arm.com> --- The changes can be seen at: https://github.com/PierreARM/edk2/tree/803_Compile_AML_bytecode_array_into_OBJ_file_v1 Notes: v1: - Rename AmlToHex scripts to AmlToC, and change line endings of the PosixLike bin-wrapper. [Pierre] BaseTools/BinWrappers/PosixLike/{AmlToHex => AmlToC} | 28 ++++++++++---------- BaseTools/BinWrappers/WindowsLike/{AmlToHex.bat => AmlToC.bat} | 0 BaseTools/Source/Python/{AmlToHex/AmlToHex.py => AmlToC/AmlToC.py} | 0 3 files changed, 14 insertions(+), 14 deletions(-) diff --git a/BaseTools/BinWrappers/PosixLike/AmlToHex b/BaseTools/BinWrappers/PosixLike/AmlToC similarity index 97% rename from BaseTools/BinWrappers/PosixLike/AmlToHex rename to BaseTools/BinWrappers/PosixLike/AmlToC index 9fb68299e4c67d1f332cd883fd348a896f1bdc50..1dd28e966288f6ea4fc52d42e2dc7b1f74226c23 100755 --- a/BaseTools/BinWrappers/PosixLike/AmlToHex +++ b/BaseTools/BinWrappers/PosixLike/AmlToC @@ -1,14 +1,14 @@ -#!/usr/bin/env bash -#python `dirname $0`/RunToolFromSource.py `basename $0` $* - -# If a ${PYTHON_COMMAND} command is available, use it in preference to python -if command -v ${PYTHON_COMMAND} >/dev/null 2>&1; then - python_exe=${PYTHON_COMMAND} -fi - -full_cmd=${BASH_SOURCE:-$0} # see http://mywiki.wooledge.org/BashFAQ/028 for a discussion of why $0 is not a good choice here -dir=$(dirname "$full_cmd") -exe=$(basename "$full_cmd") - -export PYTHONPATH="$dir/../../Source/Python${PYTHONPATH:+:"$PYTHONPATH"}" -exec "${python_exe:-python}" "$dir/../../Source/Python/$exe/$exe.py" "$@" +#!/usr/bin/env bash +#python `dirname $0`/RunToolFromSource.py `basename $0` $* + +# If a ${PYTHON_COMMAND} command is available, use it in preference to python +if command -v ${PYTHON_COMMAND} >/dev/null 2>&1; then + python_exe=${PYTHON_COMMAND} +fi + +full_cmd=${BASH_SOURCE:-$0} # see http://mywiki.wooledge.org/BashFAQ/028 for a discussion of why $0 is not a good choice here +dir=$(dirname "$full_cmd") +exe=$(basename "$full_cmd") + +export PYTHONPATH="$dir/../../Source/Python${PYTHONPATH:+:"$PYTHONPATH"}" +exec "${python_exe:-python}" "$dir/../../Source/Python/$exe/$exe.py" "$@" diff --git a/BaseTools/BinWrappers/WindowsLike/AmlToHex.bat b/BaseTools/BinWrappers/WindowsLike/AmlToC.bat similarity index 100% rename from BaseTools/BinWrappers/WindowsLike/AmlToHex.bat rename to BaseTools/BinWrappers/WindowsLike/AmlToC.bat diff --git a/BaseTools/Source/Python/AmlToHex/AmlToHex.py b/BaseTools/Source/Python/AmlToC/AmlToC.py similarity index 100% rename from BaseTools/Source/Python/AmlToHex/AmlToHex.py rename to BaseTools/Source/Python/AmlToC/AmlToC.py -- 'Guid(CE165669-3EF3-493F-B85D-6190EE5B9759)' ^ permalink raw reply related [flat|nested] 17+ messages in thread
* [PATCH v1 3/3] BaseTools: Compile AML bytecode arrays into .obj file 2020-05-18 14:11 [PATCH v1 0/3] Compile AML bytecode array into OBJ file PierreGondois 2020-05-18 14:11 ` [PATCH v1 1/3] BaseTools: Generate multiple rules when multiple output files PierreGondois 2020-05-18 14:11 ` [PATCH v1 2/3] BaseTools: Rename AmlToHex script to AmlToC PierreGondois @ 2020-05-18 14:11 ` PierreGondois 2020-05-18 14:31 ` [PATCH v1 0/3] Compile AML bytecode array into OBJ file PierreGondois ` (2 subsequent siblings) 5 siblings, 0 replies; 17+ messages in thread From: PierreGondois @ 2020-05-18 14:11 UTC (permalink / raw) To: devel; +Cc: Pierre Gondois, bob.c.feng, liming.gao, Sami.Mujawar, Tomas.Pilar, nd From: Pierre Gondois <pierre.gondois@arm.com> BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=2425 The AmlToHex script and Posix/WindowsLike wrappers convert an AML file to a .hex file, containing a C array storing AML bytecode. This ".hex" file can then be included in a C file, allowing to access the AML bytecode from this C file. The EDK2 build system doesn't allow to a depict dependency orders between files of different languages. For instance, in a module containing a ".c" file and a ".asl", the ".c" file may or may not be built prior to the ".asl" file. This prevents any inclusion of a generated ".hex" in a ".c" file since this later ".hex" file may or may not have been created yet. This patch modifies the AmlToC script to generate a C file instead of a ".hex" file. It also adds the generation of an intermediate ".amli" file when compiling an ASL file, and adds a rule to convert this ".amli" to a C file. This allows to generate a C file containing the AML bytecode from an ASL file. This C file will then be handled by the EDK2 build system to generate an object file. Thus, no file inclusion will be required anymore. The C file requiring the AML bytecode as a C array, and the ASL file, will be compiled independently. The C array must be defined as an external symbol. The linker is resolving the reference to the C array symbol. To summarize, the flow goes as: -1. ASL file is compiled to AML; -2. AML file is copied to a ".amli" intermediate file; -3. EDK2 build system applies the rule relevant to ".amli" files. This is, calling the "AmlToC" script, generating a C file from the ".amli" file; -4. EDK2 build system applies the rule relevant to C files. This is creating an object file. -5. EDK2 build system links the object file containing the AML bytecode with the object file requiring it. Signed-off-by: Pierre Gondois <Pierre.Gondois@arm.com> Signed-off-by: Tomas Pilar <Tomas.Pilar@arm.com> --- The changes can be seen at: https://github.com/PierreARM/edk2/tree/803_Compile_AML_bytecode_array_into_OBJ_file_v1 Notes: v1: - Add a new rule to the build_rule.template file to generate ".obj" files from .asl files, and modify the AmlToC script accordingly. [Pierre] BaseTools/Conf/build_rule.template | 15 +++- BaseTools/Source/Python/AmlToC/AmlToC.py | 82 ++++++++------------ 2 files changed, 47 insertions(+), 50 deletions(-) diff --git a/BaseTools/Conf/build_rule.template b/BaseTools/Conf/build_rule.template index 0822b681fcd9f61c6508e6f93ffc31fa70fd7059..a467d9097650435dab9b6d0b3c575be31f0c84e0 100755 --- a/BaseTools/Conf/build_rule.template +++ b/BaseTools/Conf/build_rule.template @@ -419,6 +419,7 @@ <OutputFile> $(OUTPUT_DIR)(+)${s_dir}(+)${s_base}.aml + $(OUTPUT_DIR)(+)${s_dir}(+)${s_base}.amli <ExtraDependency> $(MAKE_FILE) @@ -428,14 +429,24 @@ "$(ASLPP)" $(DEPS_FLAGS) $(ASLPP_FLAGS) $(INC) /I${s_path} $(OUTPUT_DIR)(+)${s_dir}(+)${s_base}.i > $(OUTPUT_DIR)(+)${s_dir}(+)${s_base}.iii Trim --source-code -l -o $(OUTPUT_DIR)(+)${s_dir}(+)${s_base}.iiii $(OUTPUT_DIR)(+)${s_dir}(+)${s_base}.iii "$(ASL)" $(ASL_FLAGS) $(ASL_OUTFLAGS)${dst} $(OUTPUT_DIR)(+)${s_dir}(+)${s_base}.iiii - -AmlToHex $(OUTPUT_DIR)(+)${s_dir}(+)${s_base}.aml + $(CP) $(OUTPUT_DIR)(+)${s_dir}(+)${s_base}.aml $(OUTPUT_DIR)(+)${s_dir}(+)${s_base}.amli <Command.GCC> Trim --asl-file --asl-deps -o $(OUTPUT_DIR)(+)${s_dir}(+)${s_base}.i -i $(INC_LIST) ${src} "$(ASLPP)" $(DEPS_FLAGS) $(ASLPP_FLAGS) $(INC) -I${s_path} $(OUTPUT_DIR)(+)${s_dir}(+)${s_base}.i > $(OUTPUT_DIR)(+)${s_dir}(+)${s_base}.iii Trim --source-code -l -o $(OUTPUT_DIR)(+)${s_dir}(+)${s_base}.iiii $(OUTPUT_DIR)(+)${s_dir}(+)${s_base}.iii "$(ASL)" $(ASL_FLAGS) $(ASL_OUTFLAGS)${dst} $(OUTPUT_DIR)(+)${s_dir}(+)${s_base}.iiii - -AmlToHex $(OUTPUT_DIR)(+)${s_dir}(+)${s_base}.aml + $(CP) $(OUTPUT_DIR)(+)${s_dir}(+)${s_base}.aml $(OUTPUT_DIR)(+)${s_dir}(+)${s_base}.amli + +[Acpi-Machine-Language-File-to-C] + <InputFile> + ?.amli + + <OutputFile> + ${s_path}(+)${s_base}.c + + <Command> + -AmlToC ${src} [C-Code-File.AcpiTable] <InputFile> diff --git a/BaseTools/Source/Python/AmlToC/AmlToC.py b/BaseTools/Source/Python/AmlToC/AmlToC.py index 643db2910e37acfdd80ac18d288c921320a79ce1..346de7159de702d860bbd809ddbe8175f1493cfb 100644 --- a/BaseTools/Source/Python/AmlToC/AmlToC.py +++ b/BaseTools/Source/Python/AmlToC/AmlToC.py @@ -1,9 +1,9 @@ ## @file # -# Convert an AML file to a .hex file containing the AML bytecode stored in a +# Convert an AML file to a .c file containing the AML bytecode stored in a # C array. -# By default, "Tables\Dsdt.aml" will generate "Tables\Dsdt.hex". -# "Tables\Dsdt.hex" will contain a C array named "dsdt_aml_code" that contains +# By default, "Tables\Dsdt.aml" will generate "Tables\Dsdt.c". +# "Tables\Dsdt.c" will contain a C array named "dsdt_aml_code" that contains # the AML bytecode. # # Copyright (c) 2020, ARM Limited. All rights reserved.<BR> @@ -17,31 +17,26 @@ from Common.BuildToolError import * import sys import os +__description__ = """ +Convert an AML file to a .c file containing the AML bytecode stored in a C +array. By default, Tables\Dsdt.aml will generate Tables\Dsdt.c. +Tables\Dsdt.c will contain a C array named "dsdt_aml_code" that contains +the AML bytecode. +""" + ## Parse the command line arguments. # # @retval A argparse.NameSpace instance, containing parsed values. # def ParseArgs(): # Initialize the parser. - Parser = argparse.ArgumentParser( - description="Convert an AML file to a .hex file containing the AML " + \ - "bytecode stored in a C array. By default, " + \ - "\"Tables\\Dsdt.aml\" will generate" + \ - "\"Tables\\Dsdt.hex\". \"Tables\\Dsdt.hex\" will " + \ - "contain a C array named \"dsdt_aml_code\" that " + \ - "contains the AML bytecode." - ) + Parser = argparse.ArgumentParser(description=__description__) # Define the possible arguments. - Parser.add_argument( - dest="InputFile", - help="Path to an input AML file to generate a .hex file from." - ) - Parser.add_argument( - "-o", "--out-dir", dest="OutDir", - help="Output directory where the .hex file will be generated. " + \ - "Default is the input file's directory." - ) + Parser.add_argument(dest="InputFile", + help="Path to an input AML file to generate a .c file from.") + Parser.add_argument("-o", "--out-dir", dest="OutDir", + help="Output directory where the .c file will be generated. Default is the input file's directory.") # Parse the input arguments. Args = Parser.parse_args() @@ -55,9 +50,7 @@ def ParseArgs(): with open(Args.InputFile, "rb") as fIn: Signature = str(fIn.read(4)) if ("DSDT" not in Signature) and ("SSDT" not in Signature): - EdkLogger.info("Invalid file type. " + \ - "File does not have a valid " + \ - "DSDT or SSDT signature: %s" % Args.InputFile) + EdkLogger.info("Invalid file type. File does not have a valid DSDT or SSDT signature: {}".format(Args.InputFile)) return None # Get the basename of the input file. @@ -66,42 +59,39 @@ def ParseArgs(): # If no output directory is specified, output to the input directory. if not Args.OutDir: - Args.OutputFile = os.path.join( - os.path.dirname(Args.InputFile), - BaseName + ".hex" - ) + Args.OutputFile = os.path.join(os.path.dirname(Args.InputFile), + BaseName + ".c") else: if not os.path.exists(Args.OutDir): os.mkdir(Args.OutDir) - Args.OutputFile = os.path.join(Args.OutDir, BaseName + ".hex") + Args.OutputFile = os.path.join(Args.OutDir, BaseName + ".c") Args.BaseName = BaseName return Args -## Convert an AML file to a .hex file containing the AML bytecode stored +## Convert an AML file to a .c file containing the AML bytecode stored # in a C array. # # @param InputFile Path to the input AML file. -# @param OutputFile Path to the output .hex file to generate. +# @param OutputFile Path to the output .c file to generate. # @param BaseName Base name of the input file. -# This is also the name of the generated .hex file. +# This is also the name of the generated .c file. # -def AmlToHex(InputFile, OutputFile, BaseName): +def AmlToC(InputFile, OutputFile, BaseName): - MacroName = "__{}_HEX__".format(BaseName.upper()) ArrayName = BaseName.lower() + "_aml_code" + FileHeader =\ +""" +// This file has been generated from: +// -Python script: {} +// -Input AML file: {} + +""" with open(InputFile, "rb") as fIn, open(OutputFile, "w") as fOut: # Write header. - fOut.write("// This file has been generated from:\n" + \ - "// \tPython script: " + \ - os.path.abspath(__file__) + "\n" + \ - "// \tInput AML file: " + \ - os.path.abspath(InputFile) + "\n\n" + \ - "#ifndef {}\n".format(MacroName) + \ - "#define {}\n\n".format(MacroName) - ) + fOut.write(FileHeader.format(os.path.abspath(InputFile), os.path.abspath(__file__))) # Write the array and its content. fOut.write("unsigned char {}[] = {{\n ".format(ArrayName)) @@ -115,15 +105,12 @@ def AmlToHex(InputFile, OutputFile, BaseName): byte = fIn.read(1) fOut.write("\n};\n") - # Write footer. - fOut.write("#endif // {}\n".format(MacroName)) - ## Main method # # This method: # 1- Initialize an EdkLogger instance. # 2- Parses the input arguments. -# 3- Converts an AML file to a .hex file containing the AML bytecode stored +# 3- Converts an AML file to a .c file containing the AML bytecode stored # in a C array. # # @retval 0 Success. @@ -139,10 +126,9 @@ def Main(): if not CommandArguments: return 1 - # Convert an AML file to a .hex file containing the AML bytecode stored + # Convert an AML file to a .c file containing the AML bytecode stored # in a C array. - AmlToHex(CommandArguments.InputFile, CommandArguments.OutputFile, - CommandArguments.BaseName) + AmlToC(CommandArguments.InputFile, CommandArguments.OutputFile, CommandArguments.BaseName) except Exception as e: print(e) return 1 -- 'Guid(CE165669-3EF3-493F-B85D-6190EE5B9759)' ^ permalink raw reply related [flat|nested] 17+ messages in thread
* Re: [PATCH v1 0/3] Compile AML bytecode array into OBJ file 2020-05-18 14:11 [PATCH v1 0/3] Compile AML bytecode array into OBJ file PierreGondois ` (2 preceding siblings ...) 2020-05-18 14:11 ` [PATCH v1 3/3] BaseTools: Compile AML bytecode arrays into .obj file PierreGondois @ 2020-05-18 14:31 ` PierreGondois [not found] ` <161025E2E32EBB17.23434@groups.io> 2020-06-13 9:37 ` Ard Biesheuvel 5 siblings, 0 replies; 17+ messages in thread From: PierreGondois @ 2020-05-18 14:31 UTC (permalink / raw) To: Pierre Gondois, devel@edk2.groups.io Cc: bob.c.feng@intel.com, liming.gao@intel.com, Sami Mujawar, Tomas Pilar, nd [Bob] https://edk2.groups.io/g/devel/message/58782 I'd prefer the option 2#. I think that is the easiest way to meet the requirement. Since the aml file is the definite output of asl file, we can use it to descript the dependency relationship between c source file and asl source code. [Pierre] We have found what might be a better way to address the issue. If not convenient, we can still follow the option 2# that we discussed. Regards, Pierre -----Original Message----- From: PierreGondois <pierre.gondois@arm.com> Sent: Monday, May 18, 2020 3:11 PM To: devel@edk2.groups.io Cc: Pierre Gondois <Pierre.Gondois@arm.com>; bob.c.feng@intel.com; liming.gao@intel.com; Sami Mujawar <Sami.Mujawar@arm.com>; Tomas Pilar <Tomas.Pilar@arm.com>; nd <nd@arm.com> Subject: [PATCH v1 0/3] Compile AML bytecode array into OBJ file Following the BZ at https://bugzilla.tianocore.org/show_bug.cgi?id=2425 This patch serie is a another way to solve the dependency of C files over ASL files. With this new method, the dependency is resolved at the linking stage. The last method to solve this dependency was to add the possibility to modify INF files to depict such a dependency. This method was not accepted. The discussion is available at https://edk2.groups.io/g/devel/topic/72655342#56658 The last patch modifying the INF specification and INF parsing are available at: https://edk2.groups.io/g/devel/topic/72655342#56658 https://edk2.groups.io/g/devel/topic/72656060#56662 Pierre Gondois (3): BaseTools: Generate multiple rules when multiple output files BaseTools: Rename AmlToHex script to AmlToC BaseTools: Compile AML bytecode arrays into .obj file BaseTools/BinWrappers/PosixLike/{AmlToHex => AmlToC} | 28 +++---- BaseTools/BinWrappers/WindowsLike/{AmlToHex.bat => AmlToC.bat} | 0 BaseTools/Conf/build_rule.template | 15 +++- BaseTools/Source/Python/{AmlToHex/AmlToHex.py => AmlToC/AmlToC.py} | 82 ++++++++------------ BaseTools/Source/Python/AutoGen/GenMake.py | 6 ++ BaseTools/Source/Python/AutoGen/ModuleAutoGen.py | 40 +++++----- 6 files changed, 86 insertions(+), 85 deletions(-) rename BaseTools/BinWrappers/PosixLike/{AmlToHex => AmlToC} (97%) rename BaseTools/BinWrappers/WindowsLike/{AmlToHex.bat => AmlToC.bat} (100%) rename BaseTools/Source/Python/{AmlToHex/AmlToHex.py => AmlToC/AmlToC.py} (52%) -- 'Guid(CE165669-3EF3-493F-B85D-6190EE5B9759)' ^ permalink raw reply [flat|nested] 17+ messages in thread
[parent not found: <161025E2E32EBB17.23434@groups.io>]
* Re: [edk2-devel] [PATCH v1 0/3] Compile AML bytecode array into OBJ file [not found] ` <161025E2E32EBB17.23434@groups.io> @ 2020-05-22 12:01 ` PierreGondois [not found] ` <161157FAC70B55F8.21259@groups.io> 1 sibling, 0 replies; 17+ messages in thread From: PierreGondois @ 2020-05-22 12:01 UTC (permalink / raw) To: devel@edk2.groups.io, Pierre Gondois Cc: bob.c.feng@intel.com, liming.gao@intel.com, Sami Mujawar, Tomas Pilar, nd Hello Bob and Liming, What do you think about the patch? Do you think it goes in the right direction? Regards, Pierre -----Original Message----- From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of PierreGondois via groups.io Sent: Monday, May 18, 2020 3:32 PM To: Pierre Gondois <Pierre.Gondois@arm.com>; devel@edk2.groups.io Cc: bob.c.feng@intel.com; liming.gao@intel.com; Sami Mujawar <Sami.Mujawar@arm.com>; Tomas Pilar <Tomas.Pilar@arm.com>; nd <nd@arm.com> Subject: Re: [edk2-devel] [PATCH v1 0/3] Compile AML bytecode array into OBJ file [Bob] https://edk2.groups.io/g/devel/message/58782 I'd prefer the option 2#. I think that is the easiest way to meet the requirement. Since the aml file is the definite output of asl file, we can use it to descript the dependency relationship between c source file and asl source code. [Pierre] We have found what might be a better way to address the issue. If not convenient, we can still follow the option 2# that we discussed. Regards, Pierre -----Original Message----- From: PierreGondois <pierre.gondois@arm.com> Sent: Monday, May 18, 2020 3:11 PM To: devel@edk2.groups.io Cc: Pierre Gondois <Pierre.Gondois@arm.com>; bob.c.feng@intel.com; liming.gao@intel.com; Sami Mujawar <Sami.Mujawar@arm.com>; Tomas Pilar <Tomas.Pilar@arm.com>; nd <nd@arm.com> Subject: [PATCH v1 0/3] Compile AML bytecode array into OBJ file Following the BZ at https://bugzilla.tianocore.org/show_bug.cgi?id=2425 This patch serie is a another way to solve the dependency of C files over ASL files. With this new method, the dependency is resolved at the linking stage. The last method to solve this dependency was to add the possibility to modify INF files to depict such a dependency. This method was not accepted. The discussion is available at https://edk2.groups.io/g/devel/topic/72655342#56658 The last patch modifying the INF specification and INF parsing are available at: https://edk2.groups.io/g/devel/topic/72655342#56658 https://edk2.groups.io/g/devel/topic/72656060#56662 Pierre Gondois (3): BaseTools: Generate multiple rules when multiple output files BaseTools: Rename AmlToHex script to AmlToC BaseTools: Compile AML bytecode arrays into .obj file BaseTools/BinWrappers/PosixLike/{AmlToHex => AmlToC} | 28 +++---- BaseTools/BinWrappers/WindowsLike/{AmlToHex.bat => AmlToC.bat} | 0 BaseTools/Conf/build_rule.template | 15 +++- BaseTools/Source/Python/{AmlToHex/AmlToHex.py => AmlToC/AmlToC.py} | 82 ++++++++------------ BaseTools/Source/Python/AutoGen/GenMake.py | 6 ++ BaseTools/Source/Python/AutoGen/ModuleAutoGen.py | 40 +++++----- 6 files changed, 86 insertions(+), 85 deletions(-) rename BaseTools/BinWrappers/PosixLike/{AmlToHex => AmlToC} (97%) rename BaseTools/BinWrappers/WindowsLike/{AmlToHex.bat => AmlToC.bat} (100%) rename BaseTools/Source/Python/{AmlToHex/AmlToHex.py => AmlToC/AmlToC.py} (52%) -- 'Guid(CE165669-3EF3-493F-B85D-6190EE5B9759)' ^ permalink raw reply [flat|nested] 17+ messages in thread
[parent not found: <161157FAC70B55F8.21259@groups.io>]
* Re: [edk2-devel] [PATCH v1 0/3] Compile AML bytecode array into OBJ file [not found] ` <161157FAC70B55F8.21259@groups.io> @ 2020-05-28 11:03 ` PierreGondois 2020-05-28 11:10 ` Bob Feng 2020-06-02 11:15 ` Bob Feng 0 siblings, 2 replies; 17+ messages in thread From: PierreGondois @ 2020-05-28 11:03 UTC (permalink / raw) To: devel@edk2.groups.io, Pierre Gondois Cc: bob.c.feng@intel.com, liming.gao@intel.com, Sami Mujawar, Tomas Pilar, nd Hello Bob and Liming, Just a message in case you missed the patch set, Regards, Pierre -----Original Message----- From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of PierreGondois via groups.io Sent: Friday, May 22, 2020 1:01 PM To: devel@edk2.groups.io; Pierre Gondois <Pierre.Gondois@arm.com> Cc: bob.c.feng@intel.com; liming.gao@intel.com; Sami Mujawar <Sami.Mujawar@arm.com>; Tomas Pilar <Tomas.Pilar@arm.com>; nd <nd@arm.com> Subject: Re: [edk2-devel] [PATCH v1 0/3] Compile AML bytecode array into OBJ file Hello Bob and Liming, What do you think about the patch? Do you think it goes in the right direction? Regards, Pierre -----Original Message----- From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of PierreGondois via groups.io Sent: Monday, May 18, 2020 3:32 PM To: Pierre Gondois <Pierre.Gondois@arm.com>; devel@edk2.groups.io Cc: bob.c.feng@intel.com; liming.gao@intel.com; Sami Mujawar <Sami.Mujawar@arm.com>; Tomas Pilar <Tomas.Pilar@arm.com>; nd <nd@arm.com> Subject: Re: [edk2-devel] [PATCH v1 0/3] Compile AML bytecode array into OBJ file [Bob] https://edk2.groups.io/g/devel/message/58782 I'd prefer the option 2#. I think that is the easiest way to meet the requirement. Since the aml file is the definite output of asl file, we can use it to descript the dependency relationship between c source file and asl source code. [Pierre] We have found what might be a better way to address the issue. If not convenient, we can still follow the option 2# that we discussed. Regards, Pierre -----Original Message----- From: PierreGondois <pierre.gondois@arm.com> Sent: Monday, May 18, 2020 3:11 PM To: devel@edk2.groups.io Cc: Pierre Gondois <Pierre.Gondois@arm.com>; bob.c.feng@intel.com; liming.gao@intel.com; Sami Mujawar <Sami.Mujawar@arm.com>; Tomas Pilar <Tomas.Pilar@arm.com>; nd <nd@arm.com> Subject: [PATCH v1 0/3] Compile AML bytecode array into OBJ file Following the BZ at https://bugzilla.tianocore.org/show_bug.cgi?id=2425 This patch serie is a another way to solve the dependency of C files over ASL files. With this new method, the dependency is resolved at the linking stage. The last method to solve this dependency was to add the possibility to modify INF files to depict such a dependency. This method was not accepted. The discussion is available at https://edk2.groups.io/g/devel/topic/72655342#56658 The last patch modifying the INF specification and INF parsing are available at: https://edk2.groups.io/g/devel/topic/72655342#56658 https://edk2.groups.io/g/devel/topic/72656060#56662 Pierre Gondois (3): BaseTools: Generate multiple rules when multiple output files BaseTools: Rename AmlToHex script to AmlToC BaseTools: Compile AML bytecode arrays into .obj file BaseTools/BinWrappers/PosixLike/{AmlToHex => AmlToC} | 28 +++---- BaseTools/BinWrappers/WindowsLike/{AmlToHex.bat => AmlToC.bat} | 0 BaseTools/Conf/build_rule.template | 15 +++- BaseTools/Source/Python/{AmlToHex/AmlToHex.py => AmlToC/AmlToC.py} | 82 ++++++++------------ BaseTools/Source/Python/AutoGen/GenMake.py | 6 ++ BaseTools/Source/Python/AutoGen/ModuleAutoGen.py | 40 +++++----- 6 files changed, 86 insertions(+), 85 deletions(-) rename BaseTools/BinWrappers/PosixLike/{AmlToHex => AmlToC} (97%) rename BaseTools/BinWrappers/WindowsLike/{AmlToHex.bat => AmlToC.bat} (100%) rename BaseTools/Source/Python/{AmlToHex/AmlToHex.py => AmlToC/AmlToC.py} (52%) -- 'Guid(CE165669-3EF3-493F-B85D-6190EE5B9759)' ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [edk2-devel] [PATCH v1 0/3] Compile AML bytecode array into OBJ file 2020-05-28 11:03 ` PierreGondois @ 2020-05-28 11:10 ` Bob Feng 2020-06-02 11:15 ` Bob Feng 1 sibling, 0 replies; 17+ messages in thread From: Bob Feng @ 2020-05-28 11:10 UTC (permalink / raw) To: devel@edk2.groups.io, pierre.gondois@arm.com Cc: Gao, Liming, Sami Mujawar, Tomas Pilar, nd Hi Pierre, Sorry for late response. I'll review the patch in this week and response early in the next week. Thanks, Bob -----Original Message----- From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of PierreGondois Sent: Thursday, May 28, 2020 7:04 PM To: devel@edk2.groups.io; Pierre Gondois <Pierre.Gondois@arm.com> Cc: Feng, Bob C <bob.c.feng@intel.com>; Gao, Liming <liming.gao@intel.com>; Sami Mujawar <Sami.Mujawar@arm.com>; Tomas Pilar <Tomas.Pilar@arm.com>; nd <nd@arm.com> Subject: Re: [edk2-devel] [PATCH v1 0/3] Compile AML bytecode array into OBJ file Hello Bob and Liming, Just a message in case you missed the patch set, Regards, Pierre -----Original Message----- From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of PierreGondois via groups.io Sent: Friday, May 22, 2020 1:01 PM To: devel@edk2.groups.io; Pierre Gondois <Pierre.Gondois@arm.com> Cc: bob.c.feng@intel.com; liming.gao@intel.com; Sami Mujawar <Sami.Mujawar@arm.com>; Tomas Pilar <Tomas.Pilar@arm.com>; nd <nd@arm.com> Subject: Re: [edk2-devel] [PATCH v1 0/3] Compile AML bytecode array into OBJ file Hello Bob and Liming, What do you think about the patch? Do you think it goes in the right direction? Regards, Pierre -----Original Message----- From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of PierreGondois via groups.io Sent: Monday, May 18, 2020 3:32 PM To: Pierre Gondois <Pierre.Gondois@arm.com>; devel@edk2.groups.io Cc: bob.c.feng@intel.com; liming.gao@intel.com; Sami Mujawar <Sami.Mujawar@arm.com>; Tomas Pilar <Tomas.Pilar@arm.com>; nd <nd@arm.com> Subject: Re: [edk2-devel] [PATCH v1 0/3] Compile AML bytecode array into OBJ file [Bob] https://edk2.groups.io/g/devel/message/58782 I'd prefer the option 2#. I think that is the easiest way to meet the requirement. Since the aml file is the definite output of asl file, we can use it to descript the dependency relationship between c source file and asl source code. [Pierre] We have found what might be a better way to address the issue. If not convenient, we can still follow the option 2# that we discussed. Regards, Pierre -----Original Message----- From: PierreGondois <pierre.gondois@arm.com> Sent: Monday, May 18, 2020 3:11 PM To: devel@edk2.groups.io Cc: Pierre Gondois <Pierre.Gondois@arm.com>; bob.c.feng@intel.com; liming.gao@intel.com; Sami Mujawar <Sami.Mujawar@arm.com>; Tomas Pilar <Tomas.Pilar@arm.com>; nd <nd@arm.com> Subject: [PATCH v1 0/3] Compile AML bytecode array into OBJ file Following the BZ at https://bugzilla.tianocore.org/show_bug.cgi?id=2425 This patch serie is a another way to solve the dependency of C files over ASL files. With this new method, the dependency is resolved at the linking stage. The last method to solve this dependency was to add the possibility to modify INF files to depict such a dependency. This method was not accepted. The discussion is available at https://edk2.groups.io/g/devel/topic/72655342#56658 The last patch modifying the INF specification and INF parsing are available at: https://edk2.groups.io/g/devel/topic/72655342#56658 https://edk2.groups.io/g/devel/topic/72656060#56662 Pierre Gondois (3): BaseTools: Generate multiple rules when multiple output files BaseTools: Rename AmlToHex script to AmlToC BaseTools: Compile AML bytecode arrays into .obj file BaseTools/BinWrappers/PosixLike/{AmlToHex => AmlToC} | 28 +++---- BaseTools/BinWrappers/WindowsLike/{AmlToHex.bat => AmlToC.bat} | 0 BaseTools/Conf/build_rule.template | 15 +++- BaseTools/Source/Python/{AmlToHex/AmlToHex.py => AmlToC/AmlToC.py} | 82 ++++++++------------ BaseTools/Source/Python/AutoGen/GenMake.py | 6 ++ BaseTools/Source/Python/AutoGen/ModuleAutoGen.py | 40 +++++----- 6 files changed, 86 insertions(+), 85 deletions(-) rename BaseTools/BinWrappers/PosixLike/{AmlToHex => AmlToC} (97%) rename BaseTools/BinWrappers/WindowsLike/{AmlToHex.bat => AmlToC.bat} (100%) rename BaseTools/Source/Python/{AmlToHex/AmlToHex.py => AmlToC/AmlToC.py} (52%) -- 'Guid(CE165669-3EF3-493F-B85D-6190EE5B9759)' ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [edk2-devel] [PATCH v1 0/3] Compile AML bytecode array into OBJ file 2020-05-28 11:03 ` PierreGondois 2020-05-28 11:10 ` Bob Feng @ 2020-06-02 11:15 ` Bob Feng 1 sibling, 0 replies; 17+ messages in thread From: Bob Feng @ 2020-06-02 11:15 UTC (permalink / raw) To: devel@edk2.groups.io, pierre.gondois@arm.com Cc: Gao, Liming, Sami Mujawar, Tomas Pilar, nd Hi Pierre, I agree with your solution. My comments are in the 1/3 patch mail. Thanks, Bob -----Original Message----- From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of PierreGondois Sent: Thursday, May 28, 2020 7:04 PM To: devel@edk2.groups.io; Pierre Gondois <Pierre.Gondois@arm.com> Cc: Feng, Bob C <bob.c.feng@intel.com>; Gao, Liming <liming.gao@intel.com>; Sami Mujawar <Sami.Mujawar@arm.com>; Tomas Pilar <Tomas.Pilar@arm.com>; nd <nd@arm.com> Subject: Re: [edk2-devel] [PATCH v1 0/3] Compile AML bytecode array into OBJ file Hello Bob and Liming, Just a message in case you missed the patch set, Regards, Pierre -----Original Message----- From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of PierreGondois via groups.io Sent: Friday, May 22, 2020 1:01 PM To: devel@edk2.groups.io; Pierre Gondois <Pierre.Gondois@arm.com> Cc: bob.c.feng@intel.com; liming.gao@intel.com; Sami Mujawar <Sami.Mujawar@arm.com>; Tomas Pilar <Tomas.Pilar@arm.com>; nd <nd@arm.com> Subject: Re: [edk2-devel] [PATCH v1 0/3] Compile AML bytecode array into OBJ file Hello Bob and Liming, What do you think about the patch? Do you think it goes in the right direction? Regards, Pierre -----Original Message----- From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of PierreGondois via groups.io Sent: Monday, May 18, 2020 3:32 PM To: Pierre Gondois <Pierre.Gondois@arm.com>; devel@edk2.groups.io Cc: bob.c.feng@intel.com; liming.gao@intel.com; Sami Mujawar <Sami.Mujawar@arm.com>; Tomas Pilar <Tomas.Pilar@arm.com>; nd <nd@arm.com> Subject: Re: [edk2-devel] [PATCH v1 0/3] Compile AML bytecode array into OBJ file [Bob] https://edk2.groups.io/g/devel/message/58782 I'd prefer the option 2#. I think that is the easiest way to meet the requirement. Since the aml file is the definite output of asl file, we can use it to descript the dependency relationship between c source file and asl source code. [Pierre] We have found what might be a better way to address the issue. If not convenient, we can still follow the option 2# that we discussed. Regards, Pierre -----Original Message----- From: PierreGondois <pierre.gondois@arm.com> Sent: Monday, May 18, 2020 3:11 PM To: devel@edk2.groups.io Cc: Pierre Gondois <Pierre.Gondois@arm.com>; bob.c.feng@intel.com; liming.gao@intel.com; Sami Mujawar <Sami.Mujawar@arm.com>; Tomas Pilar <Tomas.Pilar@arm.com>; nd <nd@arm.com> Subject: [PATCH v1 0/3] Compile AML bytecode array into OBJ file Following the BZ at https://bugzilla.tianocore.org/show_bug.cgi?id=2425 This patch serie is a another way to solve the dependency of C files over ASL files. With this new method, the dependency is resolved at the linking stage. The last method to solve this dependency was to add the possibility to modify INF files to depict such a dependency. This method was not accepted. The discussion is available at https://edk2.groups.io/g/devel/topic/72655342#56658 The last patch modifying the INF specification and INF parsing are available at: https://edk2.groups.io/g/devel/topic/72655342#56658 https://edk2.groups.io/g/devel/topic/72656060#56662 Pierre Gondois (3): BaseTools: Generate multiple rules when multiple output files BaseTools: Rename AmlToHex script to AmlToC BaseTools: Compile AML bytecode arrays into .obj file BaseTools/BinWrappers/PosixLike/{AmlToHex => AmlToC} | 28 +++---- BaseTools/BinWrappers/WindowsLike/{AmlToHex.bat => AmlToC.bat} | 0 BaseTools/Conf/build_rule.template | 15 +++- BaseTools/Source/Python/{AmlToHex/AmlToHex.py => AmlToC/AmlToC.py} | 82 ++++++++------------ BaseTools/Source/Python/AutoGen/GenMake.py | 6 ++ BaseTools/Source/Python/AutoGen/ModuleAutoGen.py | 40 +++++----- 6 files changed, 86 insertions(+), 85 deletions(-) rename BaseTools/BinWrappers/PosixLike/{AmlToHex => AmlToC} (97%) rename BaseTools/BinWrappers/WindowsLike/{AmlToHex.bat => AmlToC.bat} (100%) rename BaseTools/Source/Python/{AmlToHex/AmlToHex.py => AmlToC/AmlToC.py} (52%) -- 'Guid(CE165669-3EF3-493F-B85D-6190EE5B9759)' ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [edk2-devel] [PATCH v1 0/3] Compile AML bytecode array into OBJ file 2020-05-18 14:11 [PATCH v1 0/3] Compile AML bytecode array into OBJ file PierreGondois ` (4 preceding siblings ...) [not found] ` <161025E2E32EBB17.23434@groups.io> @ 2020-06-13 9:37 ` Ard Biesheuvel 2020-06-15 16:08 ` PierreGondois 5 siblings, 1 reply; 17+ messages in thread From: Ard Biesheuvel @ 2020-06-13 9:37 UTC (permalink / raw) To: devel, pierre.gondois Cc: bob.c.feng, liming.gao, Sami.Mujawar, Tomas.Pilar, nd On 5/18/20 4:11 PM, PierreGondois via groups.io wrote: > Following the BZ at https://bugzilla.tianocore.org/show_bug.cgi?id=2425 > This patch serie is a another way to solve the dependency > of C files over ASL files. With this new method, the > dependency is resolved at the linking stage. > > The last method to solve this dependency was to add > the possibility to modify INF files to depict such a > dependency. This method was not accepted. The discussion > is available at https://edk2.groups.io/g/devel/topic/72655342#56658 > > The last patch modifying the INF specification and INF > parsing are available at: > https://edk2.groups.io/g/devel/topic/72655342#56658 > https://edk2.groups.io/g/devel/topic/72656060#56662 > > Pierre Gondois (3): > BaseTools: Generate multiple rules when multiple output files > BaseTools: Rename AmlToHex script to AmlToC > BaseTools: Compile AML bytecode arrays into .obj file > Hello Pierre, I have no comments on this series, but I noticed that AmlToHex was checked into the repository with CRLF line endings: $ file BaseTools/BinWrappers/PosixLike/AmlToHex Bourne-Again shell script, ASCII text executable, with CRLF line terminators resulting in Build output like Intel ACPI Component Architecture ASL+ Optimizing Compiler/Disassembler version 20181213 Copyright (c) 2000 - 2018 Intel Corporation ASL Input: /home/ard/build/edk2-workspace/Build/ArmVirtQemuKernel-ARM/RELEASE_CLANG38/ARM/MdeModulePkg/Universal/Disk/RamDiskDxe/RamDiskDxe/OUTPUT/./RamDisk.iiii - 42 lines, 305 bytes, 6 keywords AML Output: /home/ard/build/edk2-workspace/Build/ArmVirtQemuKernel-ARM/RELEASE_CLANG38/ARM/MdeModulePkg/Universal/Disk/RamDiskDxe/RamDiskDxe/OUTPUT/./RamDisk.aml - 124 bytes, 5 named objects, 1 executable opcodes Compilation complete. 0 Errors, 0 Warnings, 0 Remarks, 0 Optimizations /usr/bin/env: ‘bash\r’: No such file or directory Note that this platform has no use for the hex-dumped output of the AML, so it would be better if we only call AmlToHex when necessary. Thanks, Ard. ^ permalink raw reply [flat|nested] 17+ messages in thread
* Re: [edk2-devel] [PATCH v1 0/3] Compile AML bytecode array into OBJ file 2020-06-13 9:37 ` Ard Biesheuvel @ 2020-06-15 16:08 ` PierreGondois 0 siblings, 0 replies; 17+ messages in thread From: PierreGondois @ 2020-06-15 16:08 UTC (permalink / raw) To: Ard Biesheuvel, devel@edk2.groups.io Cc: bob.c.feng@intel.com, liming.gao@intel.com, Sami Mujawar, Tomas Pilar, nd Hello Ard and Bob, I also noticed the line ending is incorrect for BaseTools/BinWrappers/PosixLike/AmlToHex. The patch "[PATCH v1 2/3] BaseTools: Rename AmlToHex script to AmlToC" is correcting the line ending. This script is generating an optional output file and doesn't prevent edk2 from building, due to the dash present at the beginning of the command line in Conf/build_rule.txt. Do you want the line ending to be fixed as soon as possible, or can it wait a bit to go with this patch serie? About the fact that the file generated by AmlToHex is not need by all platforms: A solution would be to add an "enable" flag for this script. The script would not generate any output file unless this "enable" flag is provided to the script. The platforms willing to have the file generated by AmlToHex would provide it. However this would add one more flag to the build system, in the Conf/tools_def.txt file. Regards, Pierre -----Original Message----- From: Ard Biesheuvel <ard.biesheuvel@arm.com> Sent: 13 June 2020 10:38 To: devel@edk2.groups.io; Pierre Gondois <Pierre.Gondois@arm.com> Cc: bob.c.feng@intel.com; liming.gao@intel.com; Sami Mujawar <Sami.Mujawar@arm.com>; Tomas Pilar <Tomas.Pilar@arm.com>; nd <nd@arm.com> Subject: Re: [edk2-devel] [PATCH v1 0/3] Compile AML bytecode array into OBJ file On 5/18/20 4:11 PM, PierreGondois via groups.io wrote: > Following the BZ at > https://bugzilla.tianocore.org/show_bug.cgi?id=2425 > This patch serie is a another way to solve the dependency of C files > over ASL files. With this new method, the dependency is resolved at > the linking stage. > > The last method to solve this dependency was to add the possibility to > modify INF files to depict such a dependency. This method was not > accepted. The discussion is available at > https://edk2.groups.io/g/devel/topic/72655342#56658 > > The last patch modifying the INF specification and INF parsing are > available at: > https://edk2.groups.io/g/devel/topic/72655342#56658 > https://edk2.groups.io/g/devel/topic/72656060#56662 > > Pierre Gondois (3): > BaseTools: Generate multiple rules when multiple output files > BaseTools: Rename AmlToHex script to AmlToC > BaseTools: Compile AML bytecode arrays into .obj file > Hello Pierre, I have no comments on this series, but I noticed that AmlToHex was checked into the repository with CRLF line endings: $ file BaseTools/BinWrappers/PosixLike/AmlToHex Bourne-Again shell script, ASCII text executable, with CRLF line terminators resulting in Build output like Intel ACPI Component Architecture ASL+ Optimizing Compiler/Disassembler version 20181213 Copyright (c) 2000 - 2018 Intel Corporation ASL Input: /home/ard/build/edk2-workspace/Build/ArmVirtQemuKernel-ARM/RELEASE_CLANG38/ARM/MdeModulePkg/Universal/Disk/RamDiskDxe/RamDiskDxe/OUTPUT/./RamDisk.iiii - 42 lines, 305 bytes, 6 keywords AML Output: /home/ard/build/edk2-workspace/Build/ArmVirtQemuKernel-ARM/RELEASE_CLANG38/ARM/MdeModulePkg/Universal/Disk/RamDiskDxe/RamDiskDxe/OUTPUT/./RamDisk.aml - 124 bytes, 5 named objects, 1 executable opcodes Compilation complete. 0 Errors, 0 Warnings, 0 Remarks, 0 Optimizations /usr/bin/env: ‘bash\r’: No such file or directory Note that this platform has no use for the hex-dumped output of the AML, so it would be better if we only call AmlToHex when necessary. Thanks, Ard. ^ permalink raw reply [flat|nested] 17+ messages in thread
end of thread, other threads:[~2020-06-16 8:48 UTC | newest]
Thread overview: 17+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2020-05-18 14:11 [PATCH v1 0/3] Compile AML bytecode array into OBJ file PierreGondois
2020-05-18 14:11 ` [PATCH v1 1/3] BaseTools: Generate multiple rules when multiple output files PierreGondois
2020-06-02 11:15 ` [edk2-devel] " Bob Feng
2020-06-02 13:03 ` PierreGondois
2020-06-08 14:00 ` PierreGondois
2020-06-10 3:58 ` Bob Feng
2020-06-15 15:57 ` PierreGondois
2020-06-16 8:48 ` Bob Feng
2020-05-18 14:11 ` [PATCH v1 2/3] BaseTools: Rename AmlToHex script to AmlToC PierreGondois
2020-05-18 14:11 ` [PATCH v1 3/3] BaseTools: Compile AML bytecode arrays into .obj file PierreGondois
2020-05-18 14:31 ` [PATCH v1 0/3] Compile AML bytecode array into OBJ file PierreGondois
[not found] ` <161025E2E32EBB17.23434@groups.io>
2020-05-22 12:01 ` [edk2-devel] " PierreGondois
[not found] ` <161157FAC70B55F8.21259@groups.io>
2020-05-28 11:03 ` PierreGondois
2020-05-28 11:10 ` Bob Feng
2020-06-02 11:15 ` Bob Feng
2020-06-13 9:37 ` Ard Biesheuvel
2020-06-15 16:08 ` PierreGondois
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox