* Re: [edk2-devel] [PATCH v2 1/1] BaseTools: Rationalise makefile generation [not found] <15F0315032D95F8E.2867@groups.io> @ 2020-02-04 12:02 ` PierreGondois 2020-02-04 13:50 ` Liming Gao 2020-02-05 12:01 ` Bob Feng 0 siblings, 2 replies; 4+ messages in thread From: PierreGondois @ 2020-02-04 12:02 UTC (permalink / raw) To: devel@edk2.groups.io, Pierre Gondois Cc: bob.c.feng@intel.com, liming.gao@intel.com, Sami Mujawar, nd Hello Liming and Bob, To answer Liming's questions, I am building the DynamicTablesPkg with the patch with the AARCH64-DEBUG and the following configurations: Windows - GCC - GNUmake Windows - GCC - nmake Windows - VS2017 - GNUmake Windows - VS2017 - nmake Linux - GCC5 - GNUmake For the record, when building with the Windows-VS2017-GNUmake configuration, the *_*_*_MAKE_FLAG(S) variable is set to "/nologo". As this is a nmake specific flag, it has to be removed when using GNUmake. Some warning flags also need to be slightly modified when building with VS2017. To modify the type of make tool to use (GNUmake or nmake), I modify the following variable in Conf/tools_def.txt: *_[GCC5|VS2017]_*_MAKE_PATH = [nmake|GNUmake] I am building the ShellPkg with the patch and a small modification on how nasm files are added for IA32 and X64 for the following configurations: Windows - VS2017 - GNUmake Windows - VS2017 - nmake Linux - GCC - GNUmake Unfortunately, the gcc compiler available on windows doesn't provide all the options of the one available on linux. For instances: * When building ShellPkg-DEBUG build-X64 -Windows-GCC-[nmake], the "-mcmodel=small" option provided is not supported. * When building ShellPkg-DEBUG build-IA32-Windows-GCC-[nmake], the "-m,elf_i386" option provided is not supported. These issues are related to the compiler and not to the patch. The Windows-GCC-GNUmake configuration is very important for Arm. This configuration was working fine until patch "0c3e8e9947a6c13b4327dd11b20acb95441701cf BaseTools: Enhance Basetool for incremental build" was merged in edk2. Regards, Pierre -----Original Message----- From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of PierreGondois via Groups.Io Sent: 04 February 2020 12:01 To: devel@edk2.groups.io Cc: Pierre Gondois <Pierre.Gondois@arm.com>; bob.c.feng@intel.com; liming.gao@intel.com; Sami Mujawar <Sami.Mujawar@arm.com>; Pierre Gondois <Pierre.Gondois@arm.com>; nd <nd@arm.com> Subject: [edk2-devel] [PATCH v2 1/1] BaseTools: Rationalise makefile generation From: Pierre Gondois <pierre.gondois@arm.com> The GenMake.py script tests the platform environment to determine the type of makefile that needs to be generated. If a Windows build host is detected, the makefile generated is of Nmake type. Otherwise a GNUmake type is generated. Furthermore, the <TARGET>_<TAGNAME>_<ARCH>_MAKE_PATH option in tools_def.template defines the make tool to use. E.g.: for VS2017 this is configured to use Nmake, cf. *_VS2017_*_MAKE_PATH = DEF(VS2017_BIN_HOST)\nmake.exe while for GCC5 it is setup to use GNU make. *_GCC5_*_MAKE_PATH = DEF(GCC_HOST_PREFIX)make This prevents using the GCC compiler toolchain on a Windows build host. To address this issue this patch introduces 2 factors to determine the generated makefile output. 1. Platform -> to determine shell commands used in makefile. 2. MakeTool -> to determine the type of makefile that needs to be generated. Signed-off-by: Pierre Gondois <pierre.gondois@arm.com> Signed-off-by: Sami Mujawar <sami.mujawar@arm.com> --- The changes can be seen at https://github.com/PierreARM/edk2/tree/720_BaseTools_Rationalise_makefile_generation_v2 Notes: v2: - Rebase on latest master [Pierre] BaseTools/Source/Python/AutoGen/GenMake.py | 122 ++++++++++---------- BaseTools/Source/Python/AutoGen/IncludesAutoGen.py | 34 +++--- BaseTools/Source/Python/build/build.py | 7 +- 3 files changed, 88 insertions(+), 75 deletions(-) diff --git a/BaseTools/Source/Python/AutoGen/GenMake.py b/BaseTools/Source/Python/AutoGen/GenMake.py index ba199c1aa73dc46856b41c13e07e3d9770081acd..f49c765c791d57c06fcccf7059b37a018dc68ae6 100755 --- a/BaseTools/Source/Python/AutoGen/GenMake.py +++ b/BaseTools/Source/Python/AutoGen/GenMake.py @@ -2,6 +2,7 @@ # Create makefile for MS nmake and GNU make # # Copyright (c) 2007 - 2020, Intel Corporation. All rights reserved.<BR> +# Copyright (c) 2020, ARM Limited. All rights reserved.<BR> # SPDX-License-Identifier: BSD-2-Clause-Patent # @@ -52,13 +53,6 @@ gIncludeMacroConversion = { "EFI_PPI_DEPENDENCY" : gPpiDefinition, } -## default makefile type -gMakeType = "" -if sys.platform == "win32": - gMakeType = "nmake" -else: - gMakeType = "gmake" - ## BuildFile class # @@ -77,6 +71,13 @@ class BuildFile(object): "gmake" : "GNUmakefile" } + # Get Makefile name. + def getMakefileName(self): + if not self._FileType: + return _DEFAULT_FILE_NAME_ + else: + return self._FILE_NAME_[self._FileType] + ## Fixed header string for makefile _MAKEFILE_HEADER = '''# # DO NOT EDIT @@ -106,7 +107,7 @@ class BuildFile(object): # $(RD) remove dir command # _SHELL_CMD_ = { - "nmake" : { + "win32" : { "CP" : "copy /y", "MV" : "move /y", "RM" : "del /f /q", @@ -114,7 +115,7 @@ class BuildFile(object): "RD" : "rmdir /s /q", }, - "gmake" : { + "posix" : { "CP" : "cp -f", "MV" : "mv -f", "RM" : "rm -f", @@ -125,35 +126,35 @@ class BuildFile(object): ## directory separator _SEP_ = { - "nmake" : "\\", - "gmake" : "/" + "win32" : "\\", + "posix" : "/" } ## directory creation template _MD_TEMPLATE_ = { - "nmake" : 'if not exist %(dir)s $(MD) %(dir)s', - "gmake" : "$(MD) %(dir)s" + "win32" : 'if not exist %(dir)s $(MD) %(dir)s', + "posix" : "$(MD) %(dir)s" } ## directory removal template _RD_TEMPLATE_ = { - "nmake" : 'if exist %(dir)s $(RD) %(dir)s', - "gmake" : "$(RD) %(dir)s" + "win32" : 'if exist %(dir)s $(RD) %(dir)s', + "posix" : "$(RD) %(dir)s" } ## cp if exist _CP_TEMPLATE_ = { - "nmake" : 'if exist %(Src)s $(CP) %(Src)s %(Dst)s', - "gmake" : "test -f %(Src)s && $(CP) %(Src)s %(Dst)s" + "win32" : 'if exist %(Src)s $(CP) %(Src)s %(Dst)s', + "posix" : "test -f %(Src)s && $(CP) %(Src)s %(Dst)s" } _CD_TEMPLATE_ = { - "nmake" : 'if exist %(dir)s cd %(dir)s', - "gmake" : "test -e %(dir)s && cd %(dir)s" + "win32" : 'if exist %(dir)s cd %(dir)s', + "posix" : "test -e %(dir)s && cd %(dir)s" } _MAKE_TEMPLATE_ = { - "nmake" : 'if exist %(file)s "$(MAKE)" $(MAKE_FLAGS) -f %(file)s', - "gmake" : 'test -e %(file)s && "$(MAKE)" $(MAKE_FLAGS) -f %(file)s' + "win32" : 'if exist %(file)s "$(MAKE)" $(MAKE_FLAGS) -f %(file)s', + "posix" : 'test -e %(file)s && "$(MAKE)" $(MAKE_FLAGS) -f %(file)s' } _INCLUDE_CMD_ = { @@ -169,22 +170,30 @@ class BuildFile(object): # def __init__(self, AutoGenObject): self._AutoGenObject = AutoGenObject - self._FileType = gMakeType - ## Create build file + MakePath = AutoGenObject.BuildOption.get('MAKE', {}).get('PATH') + if not MakePath: + self._FileType = "" + elif "nmake" in MakePath: + self._FileType = "nmake" + else: + self._FileType = "gmake" + + if sys.platform == "win32": + self._Platform = "win32" + else: + self._Platform = "posix" + + ## Create build file. # - # @param FileType Type of build file. Only nmake and gmake are supported now. + # Only nmake and gmake are supported. # - # @retval TRUE The build file is created or re-created successfully - # @retval FALSE The build file exists and is the same as the one to be generated + # @retval TRUE The build file is created or re-created successfully + # @retval FALSE The build file exists and is the same as the one to be generated # - def Generate(self, FileType=gMakeType): - if FileType not in self._FILE_NAME_: - EdkLogger.error("build", PARAMETER_INVALID, "Invalid build type [%s]" % FileType, - ExtraData="[%s]" % str(self._AutoGenObject)) - self._FileType = FileType + def Generate(self): FileContent = self._TEMPLATE_.Replace(self._TemplateDict) - FileName = self._FILE_NAME_[FileType] + FileName = self.getMakefileName() if not os.path.exists(os.path.join(self._AutoGenObject.MakeFileDir, "deps.txt")): with open(os.path.join(self._AutoGenObject.MakeFileDir, "deps.txt"),"w+") as fd: fd.write("") @@ -203,7 +212,7 @@ class BuildFile(object): # @retval list The directory creation command list # def GetCreateDirectoryCommand(self, DirList): - return [self._MD_TEMPLATE_[self._FileType] % {'dir':Dir} for Dir in DirList] + return [self._MD_TEMPLATE_[self._Platform] % {'dir':Dir} for + Dir in DirList] ## Return a list of directory removal command string # @@ -212,7 +221,7 @@ class BuildFile(object): # @retval list The directory removal command list # def GetRemoveDirectoryCommand(self, DirList): - return [self._RD_TEMPLATE_[self._FileType] % {'dir':Dir} for Dir in DirList] + return [self._RD_TEMPLATE_[self._Platform] % {'dir':Dir} for + Dir in DirList] def PlaceMacro(self, Path, MacroDefinitions=None): if Path.startswith("$("): @@ -462,11 +471,8 @@ cleanlib: # Compose a dict object containing information used to do replacement in template @property def _TemplateDict(self): - if self._FileType not in self._SEP_: - EdkLogger.error("build", PARAMETER_INVALID, "Invalid Makefile type [%s]" % self._FileType, - ExtraData="[%s]" % str(self._AutoGenObject)) MyAgo = self._AutoGenObject - Separator = self._SEP_[self._FileType] + Separator = self._SEP_[self._Platform] # break build if no source files and binary files are found if len(MyAgo.SourceFileList) == 0 and len(MyAgo.BinaryFileList) == 0: @@ -628,10 +634,10 @@ cleanlib: BcTargetList = [] - MakefileName = self._FILE_NAME_[self._FileType] + MakefileName = self.getMakefileName() LibraryMakeCommandList = [] for D in self.LibraryBuildDirectoryList: - Command = self._MAKE_TEMPLATE_[self._FileType] % {"file":os.path.join(D, MakefileName)} + Command = self._MAKE_TEMPLATE_[self._Platform] % + {"file":os.path.join(D, MakefileName)} LibraryMakeCommandList.append(Command) package_rel_dir = MyAgo.SourceDir @@ -683,8 +689,8 @@ cleanlib: "separator" : Separator, "module_tool_definitions" : ToolsDef, - "shell_command_code" : list(self._SHELL_CMD_[self._FileType].keys()), - "shell_command" : list(self._SHELL_CMD_[self._FileType].values()), + "shell_command_code" : list(self._SHELL_CMD_[self._Platform].keys()), + "shell_command" : list(self._SHELL_CMD_[self._Platform].values()), "module_entry_point" : ModuleEntryPoint, "image_entry_point" : ImageEntryPoint, @@ -721,7 +727,7 @@ cleanlib: self.ResultFileList.append(Dst) if '%s :' %(Dst) not in self.BuildTargetList: self.BuildTargetList.append("%s :" %(Dst)) - self.BuildTargetList.append('\t' + self._CP_TEMPLATE_[self._FileType] %{'Src': Src, 'Dst': Dst}) + self.BuildTargetList.append('\t' + + self._CP_TEMPLATE_[self._Platform] %{'Src': Src, 'Dst': Dst}) FfsCmdList = Cmd[0] for index, Str in enumerate(FfsCmdList): @@ -1222,7 +1228,7 @@ ${BEGIN}\t-@${create_directory_command}\n${END}\ # Compose a dict object containing information used to do replacement in template @property def _TemplateDict(self): - Separator = self._SEP_[self._FileType] + Separator = self._SEP_[self._Platform] MyAgo = self._AutoGenObject if self._FileType not in MyAgo.CustomMakefile: EdkLogger.error('build', OPTION_NOT_SUPPORTED, "No custom makefile for %s" % self._FileType, @@ -1252,7 +1258,7 @@ ${BEGIN}\t-@${create_directory_command}\n${END}\ ToolsDef.append("%s_%s = %s" % (Tool, Attr, MyAgo.BuildOption[Tool][Attr])) ToolsDef.append("") - MakefileName = self._FILE_NAME_[self._FileType] + MakefileName = self.getMakefileName() MakefileTemplateDict = { "makefile_header" : self._FILE_HEADER_[self._FileType], "makefile_path" : os.path.join("$(MODULE_BUILD_DIR)", MakefileName), @@ -1285,8 +1291,8 @@ ${BEGIN}\t-@${create_directory_command}\n${END}\ "separator" : Separator, "module_tool_definitions" : ToolsDef, - "shell_command_code" : list(self._SHELL_CMD_[self._FileType].keys()), - "shell_command" : list(self._SHELL_CMD_[self._FileType].values()), + "shell_command_code" : list(self._SHELL_CMD_[self._Platform].keys()), + "shell_command" : list(self._SHELL_CMD_[self._Platform].values()), "create_directory_command" : self.GetCreateDirectoryCommand(self.IntermediateDirectoryList), "custom_makefile_content" : CustomMakefile @@ -1413,7 +1419,7 @@ cleanlib: # Compose a dict object containing information used to do replacement in template @property def _TemplateDict(self): - Separator = self._SEP_[self._FileType] + Separator = self._SEP_[self._Platform] MyAgo = self._AutoGenObject if "MAKE" not in MyAgo.ToolDefinition or "PATH" not in MyAgo.ToolDefinition["MAKE"]: @@ -1424,13 +1430,13 @@ cleanlib: self.ModuleBuildDirectoryList = self.GetModuleBuildDirectoryList() self.LibraryBuildDirectoryList = self.GetLibraryBuildDirectoryList() - MakefileName = self._FILE_NAME_[self._FileType] + MakefileName = self.getMakefileName() LibraryMakefileList = [] LibraryMakeCommandList = [] for D in self.LibraryBuildDirectoryList: D = self.PlaceMacro(D, {"BUILD_DIR":MyAgo.BuildDir}) Makefile = os.path.join(D, MakefileName) - Command = self._MAKE_TEMPLATE_[self._FileType] % {"file":Makefile} + Command = self._MAKE_TEMPLATE_[self._Platform] % + {"file":Makefile} LibraryMakefileList.append(Makefile) LibraryMakeCommandList.append(Command) self.LibraryMakeCommandList = LibraryMakeCommandList @@ -1440,7 +1446,7 @@ cleanlib: for D in self.ModuleBuildDirectoryList: D = self.PlaceMacro(D, {"BUILD_DIR":MyAgo.BuildDir}) Makefile = os.path.join(D, MakefileName) - Command = self._MAKE_TEMPLATE_[self._FileType] % {"file":Makefile} + Command = self._MAKE_TEMPLATE_[self._Platform] % + {"file":Makefile} ModuleMakefileList.append(Makefile) ModuleMakeCommandList.append(Command) @@ -1460,8 +1466,8 @@ cleanlib: "toolchain_tag" : MyAgo.ToolChain, "build_target" : MyAgo.BuildTarget, - "shell_command_code" : list(self._SHELL_CMD_[self._FileType].keys()), - "shell_command" : list(self._SHELL_CMD_[self._FileType].values()), + "shell_command_code" : list(self._SHELL_CMD_[self._Platform].keys()), + "shell_command" : list(self._SHELL_CMD_[self._Platform].values()), "build_architecture_list" : MyAgo.Arch, "architecture" : MyAgo.Arch, "separator" : Separator, @@ -1519,7 +1525,7 @@ class TopLevelMakefile(BuildFile): # Compose a dict object containing information used to do replacement in template @property def _TemplateDict(self): - Separator = self._SEP_[self._FileType] + Separator = self._SEP_[self._Platform] # any platform autogen object is ok because we just need common information MyAgo = self._AutoGenObject @@ -1575,10 +1581,10 @@ class TopLevelMakefile(BuildFile): else: ExtraOption += " --pcd " + pcdname + '=' + pcd[3] - MakefileName = self._FILE_NAME_[self._FileType] + MakefileName = self.getMakefileName() SubBuildCommandList = [] for A in MyAgo.ArchList: - Command = self._MAKE_TEMPLATE_[self._FileType] % {"file":os.path.join("$(BUILD_DIR)", A, MakefileName)} + Command = self._MAKE_TEMPLATE_[self._Platform] % + {"file":os.path.join("$(BUILD_DIR)", A, MakefileName)} SubBuildCommandList.append(Command) MakefileTemplateDict = { @@ -1593,8 +1599,8 @@ class TopLevelMakefile(BuildFile): "toolchain_tag" : MyAgo.ToolChain, "build_target" : MyAgo.BuildTarget, - "shell_command_code" : list(self._SHELL_CMD_[self._FileType].keys()), - "shell_command" : list(self._SHELL_CMD_[self._FileType].values()), + "shell_command_code" : list(self._SHELL_CMD_[self._Platform].keys()), + "shell_command" : list(self._SHELL_CMD_[self._Platform].values()), 'arch' : list(MyAgo.ArchList), "build_architecture_list" : ','.join(MyAgo.ArchList), "separator" : Separator, diff --git a/BaseTools/Source/Python/AutoGen/IncludesAutoGen.py b/BaseTools/Source/Python/AutoGen/IncludesAutoGen.py index 1ca1798907ef9b46a17d8938bb2f673d5ade662f..ca9e02d19b4afad82241abb8e15845731cabd22f 100644 --- a/BaseTools/Source/Python/AutoGen/IncludesAutoGen.py +++ b/BaseTools/Source/Python/AutoGen/IncludesAutoGen.py @@ -2,6 +2,7 @@ # Build cache intermediate result and state # # Copyright (c) 2019 - 2020, Intel Corporation. All rights reserved.<BR> +# Copyright (c) 2020, ARM Limited. All rights reserved.<BR> # SPDX-License-Identifier: BSD-2-Clause-Patent # from Common.caching import cached_property @@ -12,20 +13,6 @@ from Common.Misc import SaveFileOnChange, PathClass from Common.Misc import TemplateString import sys gIsFileMap = {} -if sys.platform == "win32": - _INCLUDE_DEPS_TEMPLATE = TemplateString(''' -${BEGIN} -!IF EXIST(${deps_file}) -!INCLUDE ${deps_file} -!ENDIF -${END} - ''') -else: - _INCLUDE_DEPS_TEMPLATE = TemplateString(''' -${BEGIN} --include ${deps_file} -${END} - ''') DEP_FILE_TAIL = "# Updated \n" @@ -59,6 +46,25 @@ class IncludesAutoGen(): def CreateDepsInclude(self): deps_file = {'deps_file':self.deps_files} + + MakePath = self.module_autogen.BuildOption.get('MAKE', {}).get('PATH') + if not MakePath: + EdkLogger.error("build", PARAMETER_MISSING, Message="No Make path available.") + elif "nmake" in MakePath: + _INCLUDE_DEPS_TEMPLATE = TemplateString(''' + ${BEGIN} + !IF EXIST(${deps_file}) + !INCLUDE ${deps_file} + !ENDIF + ${END} + ''') + else: + _INCLUDE_DEPS_TEMPLATE = TemplateString(''' + ${BEGIN} + -include ${deps_file} + ${END} + ''') + try: deps_include_str = _INCLUDE_DEPS_TEMPLATE.Replace(deps_file) except Exception as e: diff --git a/BaseTools/Source/Python/build/build.py b/BaseTools/Source/Python/build/build.py index 1e47e382cba94dc6fba7d8ecad2bc924b02cea1d..8e6625ae42afe5ed5fddbdb72ef5e8ca044609b4 100755 --- a/BaseTools/Source/Python/build/build.py +++ b/BaseTools/Source/Python/build/build.py @@ -4,6 +4,7 @@ # Copyright (c) 2014, Hewlett-Packard Development Company, L.P.<BR> # Copyright (c) 2007 - 2020, Intel Corporation. All rights reserved.<BR> # Copyright (c) 2018, Hewlett Packard Enterprise Development, L.P.<BR> +# Copyright (c) 2020, ARM Limited. All rights reserved.<BR> # # SPDX-License-Identifier: BSD-2-Clause-Patent # @@ -1251,7 +1252,7 @@ class Build(): (AutoGenObject.BuildTarget, AutoGenObject.ToolChain, AutoGenObject.Arch), ExtraData=str(AutoGenObject)) - makefile = GenMake.BuildFile(AutoGenObject)._FILE_NAME_[GenMake.gMakeType] + makefile = GenMake.BuildFile(AutoGenObject).getMakefileName() # run if Target == 'run': @@ -2040,10 +2041,10 @@ class Build(): ModuleBuildDirectoryList = data_pipe.Get("ModuleBuildDirectoryList") for m_build_dir in LibraryBuildDirectoryList: - if not os.path.exists(os.path.join(m_build_dir,GenMake.BuildFile._FILE_NAME_[GenMake.gMakeType])): + if not os.path.exists(os.path.join(m_build_dir,GenMake.BuildFile.getMakefileName())): return None for m_build_dir in ModuleBuildDirectoryList: - if not os.path.exists(os.path.join(m_build_dir,GenMake.BuildFile._FILE_NAME_[GenMake.gMakeType])): + if not os.path.exists(os.path.join(m_build_dir,GenMake.BuildFile.getMakefileName())): return None Wa = WorkSpaceInfo( workspacedir,active_p,target,toolchain,archlist -- 'Guid(CE165669-3EF3-493F-B85D-6190EE5B9759)' ^ permalink raw reply related [flat|nested] 4+ messages in thread
* Re: [edk2-devel] [PATCH v2 1/1] BaseTools: Rationalise makefile generation 2020-02-04 12:02 ` [edk2-devel] [PATCH v2 1/1] BaseTools: Rationalise makefile generation PierreGondois @ 2020-02-04 13:50 ` Liming Gao 2020-02-04 14:42 ` PierreGondois 2020-02-05 12:01 ` Bob Feng 1 sibling, 1 reply; 4+ messages in thread From: Liming Gao @ 2020-02-04 13:50 UTC (permalink / raw) To: devel@edk2.groups.io, pierre.gondois@arm.com Cc: Feng, Bob C, Sami Mujawar, nd Pierre: I think below three configurations are common. With this patch, they can work fine, right? And, do you mean Windows - GCC - GNUmake is still blocked by the change "0c3e8e9947a6c13b4327dd11b20acb95441701cf BaseTools: Enhance Basetool for incremental build"? > Windows - GCC - GNUmake > Windows - VS2017 - nmake > Linux - GCC5 - GNUmake Thanks Liming > -----Original Message----- > From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of PierreGondois > Sent: Tuesday, February 4, 2020 8:03 PM > To: devel@edk2.groups.io; Pierre Gondois <Pierre.Gondois@arm.com> > Cc: Feng, Bob C <bob.c.feng@intel.com>; Gao, Liming <liming.gao@intel.com>; Sami Mujawar <Sami.Mujawar@arm.com>; nd > <nd@arm.com> > Subject: Re: [edk2-devel] [PATCH v2 1/1] BaseTools: Rationalise makefile generation > > Hello Liming and Bob, > To answer Liming's questions, I am building the DynamicTablesPkg with the patch with the AARCH64-DEBUG and the following > configurations: > Windows - GCC - GNUmake > Windows - GCC - nmake > Windows - VS2017 - GNUmake > Windows - VS2017 - nmake > Linux - GCC5 - GNUmake > For the record, when building with the Windows-VS2017-GNUmake configuration, the *_*_*_MAKE_FLAG(S) variable is set to "/nologo". > As this is a nmake specific flag, it has to be removed when using GNUmake. Some warning flags also need to be slightly modified when > building with VS2017. > To modify the type of make tool to use (GNUmake or nmake), I modify the following variable in Conf/tools_def.txt: > *_[GCC5|VS2017]_*_MAKE_PATH = [nmake|GNUmake] > > I am building the ShellPkg with the patch and a small modification on how nasm files are added for IA32 and X64 for the following > configurations: > Windows - VS2017 - GNUmake > Windows - VS2017 - nmake > Linux - GCC - GNUmake > > Unfortunately, the gcc compiler available on windows doesn't provide all the options of the one available on linux. For instances: > * When building ShellPkg-DEBUG build-X64 -Windows-GCC-[nmake], the "-mcmodel=small" option provided is not supported. > * When building ShellPkg-DEBUG build-IA32-Windows-GCC-[nmake], the "-m,elf_i386" option provided is not supported. > These issues are related to the compiler and not to the patch. > > The Windows-GCC-GNUmake configuration is very important for Arm. This configuration was working fine until patch > "0c3e8e9947a6c13b4327dd11b20acb95441701cf BaseTools: Enhance Basetool for incremental build" was merged in edk2. > > Regards, > Pierre > > -----Original Message----- > From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of PierreGondois via Groups.Io > Sent: 04 February 2020 12:01 > To: devel@edk2.groups.io > Cc: Pierre Gondois <Pierre.Gondois@arm.com>; bob.c.feng@intel.com; liming.gao@intel.com; Sami Mujawar > <Sami.Mujawar@arm.com>; Pierre Gondois <Pierre.Gondois@arm.com>; nd <nd@arm.com> > Subject: [edk2-devel] [PATCH v2 1/1] BaseTools: Rationalise makefile generation > > From: Pierre Gondois <pierre.gondois@arm.com> > > The GenMake.py script tests the platform environment to determine the type of makefile that needs to be generated. If a Windows build > host is detected, the makefile generated is of Nmake type. Otherwise a GNUmake type is generated. > > Furthermore, the <TARGET>_<TAGNAME>_<ARCH>_MAKE_PATH > option in tools_def.template defines the make tool to use. > E.g.: for VS2017 this is configured to use Nmake, cf. > *_VS2017_*_MAKE_PATH = DEF(VS2017_BIN_HOST)\nmake.exe while for GCC5 it is setup to use GNU make. > *_GCC5_*_MAKE_PATH = DEF(GCC_HOST_PREFIX)make > > This prevents using the GCC compiler toolchain on a Windows build host. > > To address this issue this patch introduces 2 factors to determine the generated makefile output. > 1. Platform -> to determine shell commands used > in makefile. > 2. MakeTool -> to determine the type of makefile > that needs to be generated. > > Signed-off-by: Pierre Gondois <pierre.gondois@arm.com> > Signed-off-by: Sami Mujawar <sami.mujawar@arm.com> > --- > > The changes can be seen at https://github.com/PierreARM/edk2/tree/720_BaseTools_Rationalise_makefile_generation_v2 > > Notes: > v2: > - Rebase on latest master [Pierre] > > BaseTools/Source/Python/AutoGen/GenMake.py | 122 ++++++++++---------- > BaseTools/Source/Python/AutoGen/IncludesAutoGen.py | 34 +++--- > BaseTools/Source/Python/build/build.py | 7 +- > 3 files changed, 88 insertions(+), 75 deletions(-) > > diff --git a/BaseTools/Source/Python/AutoGen/GenMake.py b/BaseTools/Source/Python/AutoGen/GenMake.py > index ba199c1aa73dc46856b41c13e07e3d9770081acd..f49c765c791d57c06fcccf7059b37a018dc68ae6 100755 > --- a/BaseTools/Source/Python/AutoGen/GenMake.py > +++ b/BaseTools/Source/Python/AutoGen/GenMake.py > @@ -2,6 +2,7 @@ > # Create makefile for MS nmake and GNU make # # Copyright (c) 2007 - 2020, Intel Corporation. All rights reserved.<BR> > +# Copyright (c) 2020, ARM Limited. All rights reserved.<BR> > # SPDX-License-Identifier: BSD-2-Clause-Patent # > > @@ -52,13 +53,6 @@ gIncludeMacroConversion = { > "EFI_PPI_DEPENDENCY" : gPpiDefinition, > } > > -## default makefile type > -gMakeType = "" > -if sys.platform == "win32": > - gMakeType = "nmake" > -else: > - gMakeType = "gmake" > - > > ## BuildFile class > # > @@ -77,6 +71,13 @@ class BuildFile(object): > "gmake" : "GNUmakefile" > } > > + # Get Makefile name. > + def getMakefileName(self): > + if not self._FileType: > + return _DEFAULT_FILE_NAME_ > + else: > + return self._FILE_NAME_[self._FileType] > + > ## Fixed header string for makefile > _MAKEFILE_HEADER = '''# > # DO NOT EDIT > @@ -106,7 +107,7 @@ class BuildFile(object): > # $(RD) remove dir command > # > _SHELL_CMD_ = { > - "nmake" : { > + "win32" : { > "CP" : "copy /y", > "MV" : "move /y", > "RM" : "del /f /q", > @@ -114,7 +115,7 @@ class BuildFile(object): > "RD" : "rmdir /s /q", > }, > > - "gmake" : { > + "posix" : { > "CP" : "cp -f", > "MV" : "mv -f", > "RM" : "rm -f", > @@ -125,35 +126,35 @@ class BuildFile(object): > > ## directory separator > _SEP_ = { > - "nmake" : "\\", > - "gmake" : "/" > + "win32" : "\\", > + "posix" : "/" > } > > ## directory creation template > _MD_TEMPLATE_ = { > - "nmake" : 'if not exist %(dir)s $(MD) %(dir)s', > - "gmake" : "$(MD) %(dir)s" > + "win32" : 'if not exist %(dir)s $(MD) %(dir)s', > + "posix" : "$(MD) %(dir)s" > } > > ## directory removal template > _RD_TEMPLATE_ = { > - "nmake" : 'if exist %(dir)s $(RD) %(dir)s', > - "gmake" : "$(RD) %(dir)s" > + "win32" : 'if exist %(dir)s $(RD) %(dir)s', > + "posix" : "$(RD) %(dir)s" > } > ## cp if exist > _CP_TEMPLATE_ = { > - "nmake" : 'if exist %(Src)s $(CP) %(Src)s %(Dst)s', > - "gmake" : "test -f %(Src)s && $(CP) %(Src)s %(Dst)s" > + "win32" : 'if exist %(Src)s $(CP) %(Src)s %(Dst)s', > + "posix" : "test -f %(Src)s && $(CP) %(Src)s %(Dst)s" > } > > _CD_TEMPLATE_ = { > - "nmake" : 'if exist %(dir)s cd %(dir)s', > - "gmake" : "test -e %(dir)s && cd %(dir)s" > + "win32" : 'if exist %(dir)s cd %(dir)s', > + "posix" : "test -e %(dir)s && cd %(dir)s" > } > > _MAKE_TEMPLATE_ = { > - "nmake" : 'if exist %(file)s "$(MAKE)" $(MAKE_FLAGS) -f %(file)s', > - "gmake" : 'test -e %(file)s && "$(MAKE)" $(MAKE_FLAGS) -f %(file)s' > + "win32" : 'if exist %(file)s "$(MAKE)" $(MAKE_FLAGS) -f %(file)s', > + "posix" : 'test -e %(file)s && "$(MAKE)" $(MAKE_FLAGS) -f %(file)s' > } > > _INCLUDE_CMD_ = { > @@ -169,22 +170,30 @@ class BuildFile(object): > # > def __init__(self, AutoGenObject): > self._AutoGenObject = AutoGenObject > - self._FileType = gMakeType > > - ## Create build file > + MakePath = AutoGenObject.BuildOption.get('MAKE', {}).get('PATH') > + if not MakePath: > + self._FileType = "" > + elif "nmake" in MakePath: > + self._FileType = "nmake" > + else: > + self._FileType = "gmake" > + > + if sys.platform == "win32": > + self._Platform = "win32" > + else: > + self._Platform = "posix" > + > + ## Create build file. > # > - # @param FileType Type of build file. Only nmake and gmake are supported now. > + # Only nmake and gmake are supported. > # > - # @retval TRUE The build file is created or re-created successfully > - # @retval FALSE The build file exists and is the same as the one to be generated > + # @retval TRUE The build file is created or re-created successfully > + # @retval FALSE The build file exists and is the same as the one to be generated > # > - def Generate(self, FileType=gMakeType): > - if FileType not in self._FILE_NAME_: > - EdkLogger.error("build", PARAMETER_INVALID, "Invalid build type [%s]" % FileType, > - ExtraData="[%s]" % str(self._AutoGenObject)) > - self._FileType = FileType > + def Generate(self): > FileContent = self._TEMPLATE_.Replace(self._TemplateDict) > - FileName = self._FILE_NAME_[FileType] > + FileName = self.getMakefileName() > if not os.path.exists(os.path.join(self._AutoGenObject.MakeFileDir, "deps.txt")): > with open(os.path.join(self._AutoGenObject.MakeFileDir, "deps.txt"),"w+") as fd: > fd.write("") > @@ -203,7 +212,7 @@ class BuildFile(object): > # @retval list The directory creation command list > # > def GetCreateDirectoryCommand(self, DirList): > - return [self._MD_TEMPLATE_[self._FileType] % {'dir':Dir} for Dir in DirList] > + return [self._MD_TEMPLATE_[self._Platform] % {'dir':Dir} for > + Dir in DirList] > > ## Return a list of directory removal command string > # > @@ -212,7 +221,7 @@ class BuildFile(object): > # @retval list The directory removal command list > # > def GetRemoveDirectoryCommand(self, DirList): > - return [self._RD_TEMPLATE_[self._FileType] % {'dir':Dir} for Dir in DirList] > + return [self._RD_TEMPLATE_[self._Platform] % {'dir':Dir} for > + Dir in DirList] > > def PlaceMacro(self, Path, MacroDefinitions=None): > if Path.startswith("$("): > @@ -462,11 +471,8 @@ cleanlib: > # Compose a dict object containing information used to do replacement in template > @property > def _TemplateDict(self): > - if self._FileType not in self._SEP_: > - EdkLogger.error("build", PARAMETER_INVALID, "Invalid Makefile type [%s]" % self._FileType, > - ExtraData="[%s]" % str(self._AutoGenObject)) > MyAgo = self._AutoGenObject > - Separator = self._SEP_[self._FileType] > + Separator = self._SEP_[self._Platform] > > # break build if no source files and binary files are found > if len(MyAgo.SourceFileList) == 0 and len(MyAgo.BinaryFileList) == 0: > @@ -628,10 +634,10 @@ cleanlib: > > BcTargetList = [] > > - MakefileName = self._FILE_NAME_[self._FileType] > + MakefileName = self.getMakefileName() > LibraryMakeCommandList = [] > for D in self.LibraryBuildDirectoryList: > - Command = self._MAKE_TEMPLATE_[self._FileType] % {"file":os.path.join(D, MakefileName)} > + Command = self._MAKE_TEMPLATE_[self._Platform] % > + {"file":os.path.join(D, MakefileName)} > LibraryMakeCommandList.append(Command) > > package_rel_dir = MyAgo.SourceDir @@ -683,8 +689,8 @@ cleanlib: > "separator" : Separator, > "module_tool_definitions" : ToolsDef, > > - "shell_command_code" : list(self._SHELL_CMD_[self._FileType].keys()), > - "shell_command" : list(self._SHELL_CMD_[self._FileType].values()), > + "shell_command_code" : list(self._SHELL_CMD_[self._Platform].keys()), > + "shell_command" : list(self._SHELL_CMD_[self._Platform].values()), > > "module_entry_point" : ModuleEntryPoint, > "image_entry_point" : ImageEntryPoint, > @@ -721,7 +727,7 @@ cleanlib: > self.ResultFileList.append(Dst) > if '%s :' %(Dst) not in self.BuildTargetList: > self.BuildTargetList.append("%s :" %(Dst)) > - self.BuildTargetList.append('\t' + self._CP_TEMPLATE_[self._FileType] %{'Src': Src, 'Dst': Dst}) > + self.BuildTargetList.append('\t' + > + self._CP_TEMPLATE_[self._Platform] %{'Src': Src, 'Dst': Dst}) > > FfsCmdList = Cmd[0] > for index, Str in enumerate(FfsCmdList): > @@ -1222,7 +1228,7 @@ ${BEGIN}\t-@${create_directory_command}\n${END}\ > # Compose a dict object containing information used to do replacement in template > @property > def _TemplateDict(self): > - Separator = self._SEP_[self._FileType] > + Separator = self._SEP_[self._Platform] > MyAgo = self._AutoGenObject > if self._FileType not in MyAgo.CustomMakefile: > EdkLogger.error('build', OPTION_NOT_SUPPORTED, "No custom makefile for %s" % self._FileType, @@ -1252,7 +1258,7 @@ > ${BEGIN}\t-@${create_directory_command}\n${END}\ > ToolsDef.append("%s_%s = %s" % (Tool, Attr, MyAgo.BuildOption[Tool][Attr])) > ToolsDef.append("") > > - MakefileName = self._FILE_NAME_[self._FileType] > + MakefileName = self.getMakefileName() > MakefileTemplateDict = { > "makefile_header" : self._FILE_HEADER_[self._FileType], > "makefile_path" : os.path.join("$(MODULE_BUILD_DIR)", MakefileName), > @@ -1285,8 +1291,8 @@ ${BEGIN}\t-@${create_directory_command}\n${END}\ > "separator" : Separator, > "module_tool_definitions" : ToolsDef, > > - "shell_command_code" : list(self._SHELL_CMD_[self._FileType].keys()), > - "shell_command" : list(self._SHELL_CMD_[self._FileType].values()), > + "shell_command_code" : list(self._SHELL_CMD_[self._Platform].keys()), > + "shell_command" : list(self._SHELL_CMD_[self._Platform].values()), > > "create_directory_command" : self.GetCreateDirectoryCommand(self.IntermediateDirectoryList), > "custom_makefile_content" : CustomMakefile > @@ -1413,7 +1419,7 @@ cleanlib: > # Compose a dict object containing information used to do replacement in template > @property > def _TemplateDict(self): > - Separator = self._SEP_[self._FileType] > + Separator = self._SEP_[self._Platform] > > MyAgo = self._AutoGenObject > if "MAKE" not in MyAgo.ToolDefinition or "PATH" not in MyAgo.ToolDefinition["MAKE"]: > @@ -1424,13 +1430,13 @@ cleanlib: > self.ModuleBuildDirectoryList = self.GetModuleBuildDirectoryList() > self.LibraryBuildDirectoryList = self.GetLibraryBuildDirectoryList() > > - MakefileName = self._FILE_NAME_[self._FileType] > + MakefileName = self.getMakefileName() > LibraryMakefileList = [] > LibraryMakeCommandList = [] > for D in self.LibraryBuildDirectoryList: > D = self.PlaceMacro(D, {"BUILD_DIR":MyAgo.BuildDir}) > Makefile = os.path.join(D, MakefileName) > - Command = self._MAKE_TEMPLATE_[self._FileType] % {"file":Makefile} > + Command = self._MAKE_TEMPLATE_[self._Platform] % > + {"file":Makefile} > LibraryMakefileList.append(Makefile) > LibraryMakeCommandList.append(Command) > self.LibraryMakeCommandList = LibraryMakeCommandList @@ -1440,7 +1446,7 @@ cleanlib: > for D in self.ModuleBuildDirectoryList: > D = self.PlaceMacro(D, {"BUILD_DIR":MyAgo.BuildDir}) > Makefile = os.path.join(D, MakefileName) > - Command = self._MAKE_TEMPLATE_[self._FileType] % {"file":Makefile} > + Command = self._MAKE_TEMPLATE_[self._Platform] % > + {"file":Makefile} > ModuleMakefileList.append(Makefile) > ModuleMakeCommandList.append(Command) > > @@ -1460,8 +1466,8 @@ cleanlib: > > "toolchain_tag" : MyAgo.ToolChain, > "build_target" : MyAgo.BuildTarget, > - "shell_command_code" : list(self._SHELL_CMD_[self._FileType].keys()), > - "shell_command" : list(self._SHELL_CMD_[self._FileType].values()), > + "shell_command_code" : list(self._SHELL_CMD_[self._Platform].keys()), > + "shell_command" : list(self._SHELL_CMD_[self._Platform].values()), > "build_architecture_list" : MyAgo.Arch, > "architecture" : MyAgo.Arch, > "separator" : Separator, > @@ -1519,7 +1525,7 @@ class TopLevelMakefile(BuildFile): > # Compose a dict object containing information used to do replacement in template > @property > def _TemplateDict(self): > - Separator = self._SEP_[self._FileType] > + Separator = self._SEP_[self._Platform] > > # any platform autogen object is ok because we just need common information > MyAgo = self._AutoGenObject > @@ -1575,10 +1581,10 @@ class TopLevelMakefile(BuildFile): > else: > ExtraOption += " --pcd " + pcdname + '=' + pcd[3] > > - MakefileName = self._FILE_NAME_[self._FileType] > + MakefileName = self.getMakefileName() > SubBuildCommandList = [] > for A in MyAgo.ArchList: > - Command = self._MAKE_TEMPLATE_[self._FileType] % {"file":os.path.join("$(BUILD_DIR)", A, MakefileName)} > + Command = self._MAKE_TEMPLATE_[self._Platform] % > + {"file":os.path.join("$(BUILD_DIR)", A, MakefileName)} > SubBuildCommandList.append(Command) > > MakefileTemplateDict = { > @@ -1593,8 +1599,8 @@ class TopLevelMakefile(BuildFile): > > "toolchain_tag" : MyAgo.ToolChain, > "build_target" : MyAgo.BuildTarget, > - "shell_command_code" : list(self._SHELL_CMD_[self._FileType].keys()), > - "shell_command" : list(self._SHELL_CMD_[self._FileType].values()), > + "shell_command_code" : list(self._SHELL_CMD_[self._Platform].keys()), > + "shell_command" : list(self._SHELL_CMD_[self._Platform].values()), > 'arch' : list(MyAgo.ArchList), > "build_architecture_list" : ','.join(MyAgo.ArchList), > "separator" : Separator, > diff --git a/BaseTools/Source/Python/AutoGen/IncludesAutoGen.py b/BaseTools/Source/Python/AutoGen/IncludesAutoGen.py > index 1ca1798907ef9b46a17d8938bb2f673d5ade662f..ca9e02d19b4afad82241abb8e15845731cabd22f 100644 > --- a/BaseTools/Source/Python/AutoGen/IncludesAutoGen.py > +++ b/BaseTools/Source/Python/AutoGen/IncludesAutoGen.py > @@ -2,6 +2,7 @@ > # Build cache intermediate result and state # # Copyright (c) 2019 - 2020, Intel Corporation. All rights reserved.<BR> > +# Copyright (c) 2020, ARM Limited. All rights reserved.<BR> > # SPDX-License-Identifier: BSD-2-Clause-Patent # from Common.caching import cached_property @@ -12,20 +13,6 @@ from > Common.Misc import SaveFileOnChange, PathClass from Common.Misc import TemplateString import sys gIsFileMap = {} -if > sys.platform == "win32": > - _INCLUDE_DEPS_TEMPLATE = TemplateString(''' > -${BEGIN} > -!IF EXIST(${deps_file}) > -!INCLUDE ${deps_file} > -!ENDIF > -${END} > - ''') > -else: > - _INCLUDE_DEPS_TEMPLATE = TemplateString(''' > -${BEGIN} > --include ${deps_file} > -${END} > - ''') > > DEP_FILE_TAIL = "# Updated \n" > > @@ -59,6 +46,25 @@ class IncludesAutoGen(): > > def CreateDepsInclude(self): > deps_file = {'deps_file':self.deps_files} > + > + MakePath = self.module_autogen.BuildOption.get('MAKE', {}).get('PATH') > + if not MakePath: > + EdkLogger.error("build", PARAMETER_MISSING, Message="No Make path available.") > + elif "nmake" in MakePath: > + _INCLUDE_DEPS_TEMPLATE = TemplateString(''' > + ${BEGIN} > + !IF EXIST(${deps_file}) > + !INCLUDE ${deps_file} > + !ENDIF > + ${END} > + ''') > + else: > + _INCLUDE_DEPS_TEMPLATE = TemplateString(''' > + ${BEGIN} > + -include ${deps_file} > + ${END} > + ''') > + > try: > deps_include_str = _INCLUDE_DEPS_TEMPLATE.Replace(deps_file) > except Exception as e: > diff --git a/BaseTools/Source/Python/build/build.py b/BaseTools/Source/Python/build/build.py > index 1e47e382cba94dc6fba7d8ecad2bc924b02cea1d..8e6625ae42afe5ed5fddbdb72ef5e8ca044609b4 100755 > --- a/BaseTools/Source/Python/build/build.py > +++ b/BaseTools/Source/Python/build/build.py > @@ -4,6 +4,7 @@ > # Copyright (c) 2014, Hewlett-Packard Development Company, L.P.<BR> # Copyright (c) 2007 - 2020, Intel Corporation. All rights > reserved.<BR> # Copyright (c) 2018, Hewlett Packard Enterprise Development, L.P.<BR> > +# Copyright (c) 2020, ARM Limited. All rights reserved.<BR> > # > # SPDX-License-Identifier: BSD-2-Clause-Patent # @@ -1251,7 +1252,7 @@ class Build(): > (AutoGenObject.BuildTarget, AutoGenObject.ToolChain, AutoGenObject.Arch), > ExtraData=str(AutoGenObject)) > > - makefile = GenMake.BuildFile(AutoGenObject)._FILE_NAME_[GenMake.gMakeType] > + makefile = GenMake.BuildFile(AutoGenObject).getMakefileName() > > # run > if Target == 'run': > @@ -2040,10 +2041,10 @@ class Build(): > ModuleBuildDirectoryList = data_pipe.Get("ModuleBuildDirectoryList") > > for m_build_dir in LibraryBuildDirectoryList: > - if not os.path.exists(os.path.join(m_build_dir,GenMake.BuildFile._FILE_NAME_[GenMake.gMakeType])): > + if not os.path.exists(os.path.join(m_build_dir,GenMake.BuildFile.getMakefileName())): > return None > for m_build_dir in ModuleBuildDirectoryList: > - if not os.path.exists(os.path.join(m_build_dir,GenMake.BuildFile._FILE_NAME_[GenMake.gMakeType])): > + if not os.path.exists(os.path.join(m_build_dir,GenMake.BuildFile.getMakefileName())): > return None > Wa = WorkSpaceInfo( > workspacedir,active_p,target,toolchain,archlist > -- > 'Guid(CE165669-3EF3-493F-B85D-6190EE5B9759)' > > > > > > > ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [edk2-devel] [PATCH v2 1/1] BaseTools: Rationalise makefile generation 2020-02-04 13:50 ` Liming Gao @ 2020-02-04 14:42 ` PierreGondois 0 siblings, 0 replies; 4+ messages in thread From: PierreGondois @ 2020-02-04 14:42 UTC (permalink / raw) To: devel@edk2.groups.io, liming.gao@intel.com; +Cc: Feng, Bob C, Sami Mujawar, nd Hello Liming, > I think below three configurations are common. With this patch, they can work fine, right? [1] Windows - GCC - GNUmake [2] Windows - VS2017 - nmake [3] Linux - GCC5 - GNUmake I tested the following setups: * On AARCH64 - DEBUG build - [ShellPkg], configurations [1], [2] and [3] are working well. * On [IA32|X64] - DEBUG build - [ShellPkg], configurations [2] and [3] are working well. * On [IA32|X64] - DEBUG build - [ShellPkg], configuration [1] fails because of unsupported compiler options from mingw-gcc. These compiler issues are not related with the patch. This configuration could not work before this patch and making this configuration works is beyond the scope of this patch. > And, do you mean Windows - GCC - GNUmake is still blocked by the change "0c3e8e9947a6c13b4327dd11b20acb95441701cf BaseTools: Enhance Basetool for incremental build"? We are mainly using the configuration [1] to build edk2 for AARCH64. We believe the change introduced by 0c3e8e9947a6c13b4327dd11b20acb95441701cf made our build stop working. This patch would allow it to work again. Regards, Pierre -----Original Message----- From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Liming Gao via Groups.Io Sent: 04 February 2020 13:50 To: devel@edk2.groups.io; Pierre Gondois <Pierre.Gondois@arm.com> Cc: Feng, Bob C <bob.c.feng@intel.com>; Sami Mujawar <Sami.Mujawar@arm.com>; nd <nd@arm.com> Subject: Re: [edk2-devel] [PATCH v2 1/1] BaseTools: Rationalise makefile generation Pierre: I think below three configurations are common. With this patch, they can work fine, right? And, do you mean Windows - GCC - GNUmake is still blocked by the change "0c3e8e9947a6c13b4327dd11b20acb95441701cf BaseTools: Enhance Basetool for incremental build"? > Windows - GCC - GNUmake > Windows - VS2017 - nmake > Linux - GCC5 - GNUmake Thanks Liming > -----Original Message----- > From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of > PierreGondois > Sent: Tuesday, February 4, 2020 8:03 PM > To: devel@edk2.groups.io; Pierre Gondois <Pierre.Gondois@arm.com> > Cc: Feng, Bob C <bob.c.feng@intel.com>; Gao, Liming > <liming.gao@intel.com>; Sami Mujawar <Sami.Mujawar@arm.com>; nd > <nd@arm.com> > Subject: Re: [edk2-devel] [PATCH v2 1/1] BaseTools: Rationalise > makefile generation > > Hello Liming and Bob, > To answer Liming's questions, I am building the DynamicTablesPkg with > the patch with the AARCH64-DEBUG and the following > configurations: > Windows - GCC - GNUmake > Windows - GCC - nmake > Windows - VS2017 - GNUmake > Windows - VS2017 - nmake > Linux - GCC5 - GNUmake > For the record, when building with the Windows-VS2017-GNUmake configuration, the *_*_*_MAKE_FLAG(S) variable is set to "/nologo". > As this is a nmake specific flag, it has to be removed when using > GNUmake. Some warning flags also need to be slightly modified when building with VS2017. > To modify the type of make tool to use (GNUmake or nmake), I modify the following variable in Conf/tools_def.txt: > *_[GCC5|VS2017]_*_MAKE_PATH = [nmake|GNUmake] > > I am building the ShellPkg with the patch and a small modification on > how nasm files are added for IA32 and X64 for the following > configurations: > Windows - VS2017 - GNUmake > Windows - VS2017 - nmake > Linux - GCC - GNUmake > > Unfortunately, the gcc compiler available on windows doesn't provide all the options of the one available on linux. For instances: > * When building ShellPkg-DEBUG build-X64 -Windows-GCC-[nmake], the "-mcmodel=small" option provided is not supported. > * When building ShellPkg-DEBUG build-IA32-Windows-GCC-[nmake], the "-m,elf_i386" option provided is not supported. > These issues are related to the compiler and not to the patch. > > The Windows-GCC-GNUmake configuration is very important for Arm. This > configuration was working fine until patch "0c3e8e9947a6c13b4327dd11b20acb95441701cf BaseTools: Enhance Basetool for incremental build" was merged in edk2. > > Regards, > Pierre > > -----Original Message----- > From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of > PierreGondois via Groups.Io > Sent: 04 February 2020 12:01 > To: devel@edk2.groups.io > Cc: Pierre Gondois <Pierre.Gondois@arm.com>; bob.c.feng@intel.com; > liming.gao@intel.com; Sami Mujawar <Sami.Mujawar@arm.com>; Pierre > Gondois <Pierre.Gondois@arm.com>; nd <nd@arm.com> > Subject: [edk2-devel] [PATCH v2 1/1] BaseTools: Rationalise makefile > generation > > From: Pierre Gondois <pierre.gondois@arm.com> > > The GenMake.py script tests the platform environment to determine the > type of makefile that needs to be generated. If a Windows build host is detected, the makefile generated is of Nmake type. Otherwise a GNUmake type is generated. > > Furthermore, the <TARGET>_<TAGNAME>_<ARCH>_MAKE_PATH > option in tools_def.template defines the make tool to use. > E.g.: for VS2017 this is configured to use Nmake, cf. > *_VS2017_*_MAKE_PATH = DEF(VS2017_BIN_HOST)\nmake.exe while for GCC5 it is setup to use GNU make. > *_GCC5_*_MAKE_PATH = DEF(GCC_HOST_PREFIX)make > > This prevents using the GCC compiler toolchain on a Windows build host. > > To address this issue this patch introduces 2 factors to determine the generated makefile output. > 1. Platform -> to determine shell commands used > in makefile. > 2. MakeTool -> to determine the type of makefile > that needs to be generated. > > Signed-off-by: Pierre Gondois <pierre.gondois@arm.com> > Signed-off-by: Sami Mujawar <sami.mujawar@arm.com> > --- > > The changes can be seen at > https://github.com/PierreARM/edk2/tree/720_BaseTools_Rationalise_makef > ile_generation_v2 > > Notes: > v2: > - Rebase on latest master [Pierre] > > BaseTools/Source/Python/AutoGen/GenMake.py | 122 ++++++++++---------- > BaseTools/Source/Python/AutoGen/IncludesAutoGen.py | 34 +++--- > BaseTools/Source/Python/build/build.py | 7 +- > 3 files changed, 88 insertions(+), 75 deletions(-) > > diff --git a/BaseTools/Source/Python/AutoGen/GenMake.py > b/BaseTools/Source/Python/AutoGen/GenMake.py > index > ba199c1aa73dc46856b41c13e07e3d9770081acd..f49c765c791d57c06fcccf7059b3 > 7a018dc68ae6 100755 > --- a/BaseTools/Source/Python/AutoGen/GenMake.py > +++ b/BaseTools/Source/Python/AutoGen/GenMake.py > @@ -2,6 +2,7 @@ > # Create makefile for MS nmake and GNU make # # Copyright (c) 2007 > - 2020, Intel Corporation. All rights reserved.<BR> > +# Copyright (c) 2020, ARM Limited. All rights reserved.<BR> > # SPDX-License-Identifier: BSD-2-Clause-Patent # > > @@ -52,13 +53,6 @@ gIncludeMacroConversion = { > "EFI_PPI_DEPENDENCY" : gPpiDefinition, > } > > -## default makefile type > -gMakeType = "" > -if sys.platform == "win32": > - gMakeType = "nmake" > -else: > - gMakeType = "gmake" > - > > ## BuildFile class > # > @@ -77,6 +71,13 @@ class BuildFile(object): > "gmake" : "GNUmakefile" > } > > + # Get Makefile name. > + def getMakefileName(self): > + if not self._FileType: > + return _DEFAULT_FILE_NAME_ > + else: > + return self._FILE_NAME_[self._FileType] > + > ## Fixed header string for makefile > _MAKEFILE_HEADER = '''# > # DO NOT EDIT > @@ -106,7 +107,7 @@ class BuildFile(object): > # $(RD) remove dir command > # > _SHELL_CMD_ = { > - "nmake" : { > + "win32" : { > "CP" : "copy /y", > "MV" : "move /y", > "RM" : "del /f /q", > @@ -114,7 +115,7 @@ class BuildFile(object): > "RD" : "rmdir /s /q", > }, > > - "gmake" : { > + "posix" : { > "CP" : "cp -f", > "MV" : "mv -f", > "RM" : "rm -f", > @@ -125,35 +126,35 @@ class BuildFile(object): > > ## directory separator > _SEP_ = { > - "nmake" : "\\", > - "gmake" : "/" > + "win32" : "\\", > + "posix" : "/" > } > > ## directory creation template > _MD_TEMPLATE_ = { > - "nmake" : 'if not exist %(dir)s $(MD) %(dir)s', > - "gmake" : "$(MD) %(dir)s" > + "win32" : 'if not exist %(dir)s $(MD) %(dir)s', > + "posix" : "$(MD) %(dir)s" > } > > ## directory removal template > _RD_TEMPLATE_ = { > - "nmake" : 'if exist %(dir)s $(RD) %(dir)s', > - "gmake" : "$(RD) %(dir)s" > + "win32" : 'if exist %(dir)s $(RD) %(dir)s', > + "posix" : "$(RD) %(dir)s" > } > ## cp if exist > _CP_TEMPLATE_ = { > - "nmake" : 'if exist %(Src)s $(CP) %(Src)s %(Dst)s', > - "gmake" : "test -f %(Src)s && $(CP) %(Src)s %(Dst)s" > + "win32" : 'if exist %(Src)s $(CP) %(Src)s %(Dst)s', > + "posix" : "test -f %(Src)s && $(CP) %(Src)s %(Dst)s" > } > > _CD_TEMPLATE_ = { > - "nmake" : 'if exist %(dir)s cd %(dir)s', > - "gmake" : "test -e %(dir)s && cd %(dir)s" > + "win32" : 'if exist %(dir)s cd %(dir)s', > + "posix" : "test -e %(dir)s && cd %(dir)s" > } > > _MAKE_TEMPLATE_ = { > - "nmake" : 'if exist %(file)s "$(MAKE)" $(MAKE_FLAGS) -f %(file)s', > - "gmake" : 'test -e %(file)s && "$(MAKE)" $(MAKE_FLAGS) -f %(file)s' > + "win32" : 'if exist %(file)s "$(MAKE)" $(MAKE_FLAGS) -f %(file)s', > + "posix" : 'test -e %(file)s && "$(MAKE)" $(MAKE_FLAGS) -f %(file)s' > } > > _INCLUDE_CMD_ = { > @@ -169,22 +170,30 @@ class BuildFile(object): > # > def __init__(self, AutoGenObject): > self._AutoGenObject = AutoGenObject > - self._FileType = gMakeType > > - ## Create build file > + MakePath = AutoGenObject.BuildOption.get('MAKE', {}).get('PATH') > + if not MakePath: > + self._FileType = "" > + elif "nmake" in MakePath: > + self._FileType = "nmake" > + else: > + self._FileType = "gmake" > + > + if sys.platform == "win32": > + self._Platform = "win32" > + else: > + self._Platform = "posix" > + > + ## Create build file. > # > - # @param FileType Type of build file. Only nmake and gmake are supported now. > + # Only nmake and gmake are supported. > # > - # @retval TRUE The build file is created or re-created successfully > - # @retval FALSE The build file exists and is the same as the one to be generated > + # @retval TRUE The build file is created or re-created successfully > + # @retval FALSE The build file exists and is the same as the one to be generated > # > - def Generate(self, FileType=gMakeType): > - if FileType not in self._FILE_NAME_: > - EdkLogger.error("build", PARAMETER_INVALID, "Invalid build type [%s]" % FileType, > - ExtraData="[%s]" % str(self._AutoGenObject)) > - self._FileType = FileType > + def Generate(self): > FileContent = self._TEMPLATE_.Replace(self._TemplateDict) > - FileName = self._FILE_NAME_[FileType] > + FileName = self.getMakefileName() > if not os.path.exists(os.path.join(self._AutoGenObject.MakeFileDir, "deps.txt")): > with open(os.path.join(self._AutoGenObject.MakeFileDir, "deps.txt"),"w+") as fd: > fd.write("") > @@ -203,7 +212,7 @@ class BuildFile(object): > # @retval list The directory creation command list > # > def GetCreateDirectoryCommand(self, DirList): > - return [self._MD_TEMPLATE_[self._FileType] % {'dir':Dir} for Dir in DirList] > + return [self._MD_TEMPLATE_[self._Platform] % {'dir':Dir} for > + Dir in DirList] > > ## Return a list of directory removal command string > # > @@ -212,7 +221,7 @@ class BuildFile(object): > # @retval list The directory removal command list > # > def GetRemoveDirectoryCommand(self, DirList): > - return [self._RD_TEMPLATE_[self._FileType] % {'dir':Dir} for Dir in DirList] > + return [self._RD_TEMPLATE_[self._Platform] % {'dir':Dir} for > + Dir in DirList] > > def PlaceMacro(self, Path, MacroDefinitions=None): > if Path.startswith("$("): > @@ -462,11 +471,8 @@ cleanlib: > # Compose a dict object containing information used to do replacement in template > @property > def _TemplateDict(self): > - if self._FileType not in self._SEP_: > - EdkLogger.error("build", PARAMETER_INVALID, "Invalid Makefile type [%s]" % self._FileType, > - ExtraData="[%s]" % str(self._AutoGenObject)) > MyAgo = self._AutoGenObject > - Separator = self._SEP_[self._FileType] > + Separator = self._SEP_[self._Platform] > > # break build if no source files and binary files are found > if len(MyAgo.SourceFileList) == 0 and len(MyAgo.BinaryFileList) == 0: > @@ -628,10 +634,10 @@ cleanlib: > > BcTargetList = [] > > - MakefileName = self._FILE_NAME_[self._FileType] > + MakefileName = self.getMakefileName() > LibraryMakeCommandList = [] > for D in self.LibraryBuildDirectoryList: > - Command = self._MAKE_TEMPLATE_[self._FileType] % {"file":os.path.join(D, MakefileName)} > + Command = self._MAKE_TEMPLATE_[self._Platform] % > + {"file":os.path.join(D, MakefileName)} > LibraryMakeCommandList.append(Command) > > package_rel_dir = MyAgo.SourceDir @@ -683,8 +689,8 @@ cleanlib: > "separator" : Separator, > "module_tool_definitions" : ToolsDef, > > - "shell_command_code" : list(self._SHELL_CMD_[self._FileType].keys()), > - "shell_command" : list(self._SHELL_CMD_[self._FileType].values()), > + "shell_command_code" : list(self._SHELL_CMD_[self._Platform].keys()), > + "shell_command" : list(self._SHELL_CMD_[self._Platform].values()), > > "module_entry_point" : ModuleEntryPoint, > "image_entry_point" : ImageEntryPoint, > @@ -721,7 +727,7 @@ cleanlib: > self.ResultFileList.append(Dst) > if '%s :' %(Dst) not in self.BuildTargetList: > self.BuildTargetList.append("%s :" %(Dst)) > - self.BuildTargetList.append('\t' + self._CP_TEMPLATE_[self._FileType] %{'Src': Src, 'Dst': Dst}) > + self.BuildTargetList.append('\t' + > + self._CP_TEMPLATE_[self._Platform] %{'Src': Src, 'Dst': Dst}) > > FfsCmdList = Cmd[0] > for index, Str in enumerate(FfsCmdList): > @@ -1222,7 +1228,7 @@ ${BEGIN}\t-@${create_directory_command}\n${END}\ > # Compose a dict object containing information used to do replacement in template > @property > def _TemplateDict(self): > - Separator = self._SEP_[self._FileType] > + Separator = self._SEP_[self._Platform] > MyAgo = self._AutoGenObject > if self._FileType not in MyAgo.CustomMakefile: > EdkLogger.error('build', OPTION_NOT_SUPPORTED, "No custom > makefile for %s" % self._FileType, @@ -1252,7 +1258,7 @@ ${BEGIN}\t-@${create_directory_command}\n${END}\ > ToolsDef.append("%s_%s = %s" % (Tool, Attr, MyAgo.BuildOption[Tool][Attr])) > ToolsDef.append("") > > - MakefileName = self._FILE_NAME_[self._FileType] > + MakefileName = self.getMakefileName() > MakefileTemplateDict = { > "makefile_header" : self._FILE_HEADER_[self._FileType], > "makefile_path" : os.path.join("$(MODULE_BUILD_DIR)", MakefileName), > @@ -1285,8 +1291,8 @@ ${BEGIN}\t-@${create_directory_command}\n${END}\ > "separator" : Separator, > "module_tool_definitions" : ToolsDef, > > - "shell_command_code" : list(self._SHELL_CMD_[self._FileType].keys()), > - "shell_command" : list(self._SHELL_CMD_[self._FileType].values()), > + "shell_command_code" : list(self._SHELL_CMD_[self._Platform].keys()), > + "shell_command" : list(self._SHELL_CMD_[self._Platform].values()), > > "create_directory_command" : self.GetCreateDirectoryCommand(self.IntermediateDirectoryList), > "custom_makefile_content" : CustomMakefile > @@ -1413,7 +1419,7 @@ cleanlib: > # Compose a dict object containing information used to do replacement in template > @property > def _TemplateDict(self): > - Separator = self._SEP_[self._FileType] > + Separator = self._SEP_[self._Platform] > > MyAgo = self._AutoGenObject > if "MAKE" not in MyAgo.ToolDefinition or "PATH" not in MyAgo.ToolDefinition["MAKE"]: > @@ -1424,13 +1430,13 @@ cleanlib: > self.ModuleBuildDirectoryList = self.GetModuleBuildDirectoryList() > self.LibraryBuildDirectoryList = > self.GetLibraryBuildDirectoryList() > > - MakefileName = self._FILE_NAME_[self._FileType] > + MakefileName = self.getMakefileName() > LibraryMakefileList = [] > LibraryMakeCommandList = [] > for D in self.LibraryBuildDirectoryList: > D = self.PlaceMacro(D, {"BUILD_DIR":MyAgo.BuildDir}) > Makefile = os.path.join(D, MakefileName) > - Command = self._MAKE_TEMPLATE_[self._FileType] % {"file":Makefile} > + Command = self._MAKE_TEMPLATE_[self._Platform] % > + {"file":Makefile} > LibraryMakefileList.append(Makefile) > LibraryMakeCommandList.append(Command) > self.LibraryMakeCommandList = LibraryMakeCommandList @@ -1440,7 +1446,7 @@ cleanlib: > for D in self.ModuleBuildDirectoryList: > D = self.PlaceMacro(D, {"BUILD_DIR":MyAgo.BuildDir}) > Makefile = os.path.join(D, MakefileName) > - Command = self._MAKE_TEMPLATE_[self._FileType] % {"file":Makefile} > + Command = self._MAKE_TEMPLATE_[self._Platform] % > + {"file":Makefile} > ModuleMakefileList.append(Makefile) > ModuleMakeCommandList.append(Command) > > @@ -1460,8 +1466,8 @@ cleanlib: > > "toolchain_tag" : MyAgo.ToolChain, > "build_target" : MyAgo.BuildTarget, > - "shell_command_code" : list(self._SHELL_CMD_[self._FileType].keys()), > - "shell_command" : list(self._SHELL_CMD_[self._FileType].values()), > + "shell_command_code" : list(self._SHELL_CMD_[self._Platform].keys()), > + "shell_command" : list(self._SHELL_CMD_[self._Platform].values()), > "build_architecture_list" : MyAgo.Arch, > "architecture" : MyAgo.Arch, > "separator" : Separator, > @@ -1519,7 +1525,7 @@ class TopLevelMakefile(BuildFile): > # Compose a dict object containing information used to do replacement in template > @property > def _TemplateDict(self): > - Separator = self._SEP_[self._FileType] > + Separator = self._SEP_[self._Platform] > > # any platform autogen object is ok because we just need common information > MyAgo = self._AutoGenObject > @@ -1575,10 +1581,10 @@ class TopLevelMakefile(BuildFile): > else: > ExtraOption += " --pcd " + pcdname + '=' + pcd[3] > > - MakefileName = self._FILE_NAME_[self._FileType] > + MakefileName = self.getMakefileName() > SubBuildCommandList = [] > for A in MyAgo.ArchList: > - Command = self._MAKE_TEMPLATE_[self._FileType] % {"file":os.path.join("$(BUILD_DIR)", A, MakefileName)} > + Command = self._MAKE_TEMPLATE_[self._Platform] % > + {"file":os.path.join("$(BUILD_DIR)", A, MakefileName)} > SubBuildCommandList.append(Command) > > MakefileTemplateDict = { > @@ -1593,8 +1599,8 @@ class TopLevelMakefile(BuildFile): > > "toolchain_tag" : MyAgo.ToolChain, > "build_target" : MyAgo.BuildTarget, > - "shell_command_code" : list(self._SHELL_CMD_[self._FileType].keys()), > - "shell_command" : list(self._SHELL_CMD_[self._FileType].values()), > + "shell_command_code" : list(self._SHELL_CMD_[self._Platform].keys()), > + "shell_command" : list(self._SHELL_CMD_[self._Platform].values()), > 'arch' : list(MyAgo.ArchList), > "build_architecture_list" : ','.join(MyAgo.ArchList), > "separator" : Separator, > diff --git a/BaseTools/Source/Python/AutoGen/IncludesAutoGen.py > b/BaseTools/Source/Python/AutoGen/IncludesAutoGen.py > index > 1ca1798907ef9b46a17d8938bb2f673d5ade662f..ca9e02d19b4afad82241abb8e158 > 45731cabd22f 100644 > --- a/BaseTools/Source/Python/AutoGen/IncludesAutoGen.py > +++ b/BaseTools/Source/Python/AutoGen/IncludesAutoGen.py > @@ -2,6 +2,7 @@ > # Build cache intermediate result and state # # Copyright (c) 2019 > - 2020, Intel Corporation. All rights reserved.<BR> > +# Copyright (c) 2020, ARM Limited. All rights reserved.<BR> > # SPDX-License-Identifier: BSD-2-Clause-Patent # from > Common.caching import cached_property @@ -12,20 +13,6 @@ from > Common.Misc import SaveFileOnChange, PathClass from Common.Misc import TemplateString import sys gIsFileMap = {} -if sys.platform == "win32": > - _INCLUDE_DEPS_TEMPLATE = TemplateString(''' > -${BEGIN} > -!IF EXIST(${deps_file}) > -!INCLUDE ${deps_file} > -!ENDIF > -${END} > - ''') > -else: > - _INCLUDE_DEPS_TEMPLATE = TemplateString(''' > -${BEGIN} > --include ${deps_file} > -${END} > - ''') > > DEP_FILE_TAIL = "# Updated \n" > > @@ -59,6 +46,25 @@ class IncludesAutoGen(): > > def CreateDepsInclude(self): > deps_file = {'deps_file':self.deps_files} > + > + MakePath = self.module_autogen.BuildOption.get('MAKE', {}).get('PATH') > + if not MakePath: > + EdkLogger.error("build", PARAMETER_MISSING, Message="No Make path available.") > + elif "nmake" in MakePath: > + _INCLUDE_DEPS_TEMPLATE = TemplateString(''' > + ${BEGIN} > + !IF EXIST(${deps_file}) > + !INCLUDE ${deps_file} > + !ENDIF > + ${END} > + ''') > + else: > + _INCLUDE_DEPS_TEMPLATE = TemplateString(''' > + ${BEGIN} > + -include ${deps_file} > + ${END} > + ''') > + > try: > deps_include_str = _INCLUDE_DEPS_TEMPLATE.Replace(deps_file) > except Exception as e: > diff --git a/BaseTools/Source/Python/build/build.py > b/BaseTools/Source/Python/build/build.py > index > 1e47e382cba94dc6fba7d8ecad2bc924b02cea1d..8e6625ae42afe5ed5fddbdb72ef5 > e8ca044609b4 100755 > --- a/BaseTools/Source/Python/build/build.py > +++ b/BaseTools/Source/Python/build/build.py > @@ -4,6 +4,7 @@ > # Copyright (c) 2014, Hewlett-Packard Development Company, L.P.<BR> > # Copyright (c) 2007 - 2020, Intel Corporation. All rights > reserved.<BR> # Copyright (c) 2018, Hewlett Packard Enterprise > Development, L.P.<BR> > +# Copyright (c) 2020, ARM Limited. All rights reserved.<BR> > # > # SPDX-License-Identifier: BSD-2-Clause-Patent # @@ -1251,7 +1252,7 @@ class Build(): > (AutoGenObject.BuildTarget, AutoGenObject.ToolChain, AutoGenObject.Arch), > ExtraData=str(AutoGenObject)) > > - makefile = GenMake.BuildFile(AutoGenObject)._FILE_NAME_[GenMake.gMakeType] > + makefile = GenMake.BuildFile(AutoGenObject).getMakefileName() > > # run > if Target == 'run': > @@ -2040,10 +2041,10 @@ class Build(): > ModuleBuildDirectoryList = > data_pipe.Get("ModuleBuildDirectoryList") > > for m_build_dir in LibraryBuildDirectoryList: > - if not os.path.exists(os.path.join(m_build_dir,GenMake.BuildFile._FILE_NAME_[GenMake.gMakeType])): > + if not os.path.exists(os.path.join(m_build_dir,GenMake.BuildFile.getMakefileName())): > return None > for m_build_dir in ModuleBuildDirectoryList: > - if not os.path.exists(os.path.join(m_build_dir,GenMake.BuildFile._FILE_NAME_[GenMake.gMakeType])): > + if not os.path.exists(os.path.join(m_build_dir,GenMake.BuildFile.getMakefileName())): > return None > Wa = WorkSpaceInfo( > workspacedir,active_p,target,toolchain,archlist > -- > 'Guid(CE165669-3EF3-493F-B85D-6190EE5B9759)' > > > > > > > ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [edk2-devel] [PATCH v2 1/1] BaseTools: Rationalise makefile generation 2020-02-04 12:02 ` [edk2-devel] [PATCH v2 1/1] BaseTools: Rationalise makefile generation PierreGondois 2020-02-04 13:50 ` Liming Gao @ 2020-02-05 12:01 ` Bob Feng 1 sibling, 0 replies; 4+ messages in thread From: Bob Feng @ 2020-02-05 12:01 UTC (permalink / raw) To: Pierre Gondois, devel@edk2.groups.io; +Cc: Gao, Liming, Sami Mujawar, nd Hi Pierre, I agree with this change. But for this patch, I found there are two bugs. 1. + # Get Makefile name. + def getMakefileName(self): + if not self._FileType: + return _DEFAULT_FILE_NAME_ Should be self. _DEFAULT_FILE_NAME_ + else: + return self._FILE_NAME_[self._FileType] + 2. - if not os.path.exists(os.path.join(m_build_dir,GenMake.BuildFile._FILE_NAME_[GenMake.gMakeType])): + if not os.path.exists(os.path.join(m_build_dir,GenMake.BuildFile.getMakefileName())): return None for m_build_dir in ModuleBuildDirectoryList: - if not os.path.exists(os.path.join(m_build_dir,GenMake.BuildFile._FILE_NAME_[GenMake.gMakeType])): + if not os.path.exists(os.path.join(m_build_dir,GenMake.BuildFile.getMakefileName())): return None getMakefileName() can't be called from its class. This bug can be triggered by build genmake -p OvmfPkg\OvmfPkgIa32.dsc -a IA32 -t VS2015x86 build -u -p OvmfPkg\OvmfPkgIa32.dsc -a IA32 -t VS2015x86 The second build will fail. Thanks, Bob -----Original Message----- From: Pierre Gondois <Pierre.Gondois@arm.com> Sent: Tuesday, February 4, 2020 8:03 PM To: devel@edk2.groups.io; Pierre Gondois <Pierre.Gondois@arm.com> Cc: Feng, Bob C <bob.c.feng@intel.com>; Gao, Liming <liming.gao@intel.com>; Sami Mujawar <Sami.Mujawar@arm.com>; nd <nd@arm.com> Subject: RE: [edk2-devel] [PATCH v2 1/1] BaseTools: Rationalise makefile generation Hello Liming and Bob, To answer Liming's questions, I am building the DynamicTablesPkg with the patch with the AARCH64-DEBUG and the following configurations: Windows - GCC - GNUmake Windows - GCC - nmake Windows - VS2017 - GNUmake Windows - VS2017 - nmake Linux - GCC5 - GNUmake For the record, when building with the Windows-VS2017-GNUmake configuration, the *_*_*_MAKE_FLAG(S) variable is set to "/nologo". As this is a nmake specific flag, it has to be removed when using GNUmake. Some warning flags also need to be slightly modified when building with VS2017. To modify the type of make tool to use (GNUmake or nmake), I modify the following variable in Conf/tools_def.txt: *_[GCC5|VS2017]_*_MAKE_PATH = [nmake|GNUmake] I am building the ShellPkg with the patch and a small modification on how nasm files are added for IA32 and X64 for the following configurations: Windows - VS2017 - GNUmake Windows - VS2017 - nmake Linux - GCC - GNUmake Unfortunately, the gcc compiler available on windows doesn't provide all the options of the one available on linux. For instances: * When building ShellPkg-DEBUG build-X64 -Windows-GCC-[nmake], the "-mcmodel=small" option provided is not supported. * When building ShellPkg-DEBUG build-IA32-Windows-GCC-[nmake], the "-m,elf_i386" option provided is not supported. These issues are related to the compiler and not to the patch. The Windows-GCC-GNUmake configuration is very important for Arm. This configuration was working fine until patch "0c3e8e9947a6c13b4327dd11b20acb95441701cf BaseTools: Enhance Basetool for incremental build" was merged in edk2. Regards, Pierre -----Original Message----- From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of PierreGondois via Groups.Io Sent: 04 February 2020 12:01 To: devel@edk2.groups.io Cc: Pierre Gondois <Pierre.Gondois@arm.com>; bob.c.feng@intel.com; liming.gao@intel.com; Sami Mujawar <Sami.Mujawar@arm.com>; Pierre Gondois <Pierre.Gondois@arm.com>; nd <nd@arm.com> Subject: [edk2-devel] [PATCH v2 1/1] BaseTools: Rationalise makefile generation From: Pierre Gondois <pierre.gondois@arm.com> The GenMake.py script tests the platform environment to determine the type of makefile that needs to be generated. If a Windows build host is detected, the makefile generated is of Nmake type. Otherwise a GNUmake type is generated. Furthermore, the <TARGET>_<TAGNAME>_<ARCH>_MAKE_PATH option in tools_def.template defines the make tool to use. E.g.: for VS2017 this is configured to use Nmake, cf. *_VS2017_*_MAKE_PATH = DEF(VS2017_BIN_HOST)\nmake.exe while for GCC5 it is setup to use GNU make. *_GCC5_*_MAKE_PATH = DEF(GCC_HOST_PREFIX)make This prevents using the GCC compiler toolchain on a Windows build host. To address this issue this patch introduces 2 factors to determine the generated makefile output. 1. Platform -> to determine shell commands used in makefile. 2. MakeTool -> to determine the type of makefile that needs to be generated. Signed-off-by: Pierre Gondois <pierre.gondois@arm.com> Signed-off-by: Sami Mujawar <sami.mujawar@arm.com> --- The changes can be seen at https://github.com/PierreARM/edk2/tree/720_BaseTools_Rationalise_makefile_generation_v2 Notes: v2: - Rebase on latest master [Pierre] BaseTools/Source/Python/AutoGen/GenMake.py | 122 ++++++++++---------- BaseTools/Source/Python/AutoGen/IncludesAutoGen.py | 34 +++--- BaseTools/Source/Python/build/build.py | 7 +- 3 files changed, 88 insertions(+), 75 deletions(-) diff --git a/BaseTools/Source/Python/AutoGen/GenMake.py b/BaseTools/Source/Python/AutoGen/GenMake.py index ba199c1aa73dc46856b41c13e07e3d9770081acd..f49c765c791d57c06fcccf7059b37a018dc68ae6 100755 --- a/BaseTools/Source/Python/AutoGen/GenMake.py +++ b/BaseTools/Source/Python/AutoGen/GenMake.py @@ -2,6 +2,7 @@ # Create makefile for MS nmake and GNU make # # Copyright (c) 2007 - 2020, Intel Corporation. All rights reserved.<BR> +# Copyright (c) 2020, ARM Limited. All rights reserved.<BR> # SPDX-License-Identifier: BSD-2-Clause-Patent # @@ -52,13 +53,6 @@ gIncludeMacroConversion = { "EFI_PPI_DEPENDENCY" : gPpiDefinition, } -## default makefile type -gMakeType = "" -if sys.platform == "win32": - gMakeType = "nmake" -else: - gMakeType = "gmake" - ## BuildFile class # @@ -77,6 +71,13 @@ class BuildFile(object): "gmake" : "GNUmakefile" } + # Get Makefile name. + def getMakefileName(self): + if not self._FileType: + return _DEFAULT_FILE_NAME_ + else: + return self._FILE_NAME_[self._FileType] + ## Fixed header string for makefile _MAKEFILE_HEADER = '''# # DO NOT EDIT @@ -106,7 +107,7 @@ class BuildFile(object): # $(RD) remove dir command # _SHELL_CMD_ = { - "nmake" : { + "win32" : { "CP" : "copy /y", "MV" : "move /y", "RM" : "del /f /q", @@ -114,7 +115,7 @@ class BuildFile(object): "RD" : "rmdir /s /q", }, - "gmake" : { + "posix" : { "CP" : "cp -f", "MV" : "mv -f", "RM" : "rm -f", @@ -125,35 +126,35 @@ class BuildFile(object): ## directory separator _SEP_ = { - "nmake" : "\\", - "gmake" : "/" + "win32" : "\\", + "posix" : "/" } ## directory creation template _MD_TEMPLATE_ = { - "nmake" : 'if not exist %(dir)s $(MD) %(dir)s', - "gmake" : "$(MD) %(dir)s" + "win32" : 'if not exist %(dir)s $(MD) %(dir)s', + "posix" : "$(MD) %(dir)s" } ## directory removal template _RD_TEMPLATE_ = { - "nmake" : 'if exist %(dir)s $(RD) %(dir)s', - "gmake" : "$(RD) %(dir)s" + "win32" : 'if exist %(dir)s $(RD) %(dir)s', + "posix" : "$(RD) %(dir)s" } ## cp if exist _CP_TEMPLATE_ = { - "nmake" : 'if exist %(Src)s $(CP) %(Src)s %(Dst)s', - "gmake" : "test -f %(Src)s && $(CP) %(Src)s %(Dst)s" + "win32" : 'if exist %(Src)s $(CP) %(Src)s %(Dst)s', + "posix" : "test -f %(Src)s && $(CP) %(Src)s %(Dst)s" } _CD_TEMPLATE_ = { - "nmake" : 'if exist %(dir)s cd %(dir)s', - "gmake" : "test -e %(dir)s && cd %(dir)s" + "win32" : 'if exist %(dir)s cd %(dir)s', + "posix" : "test -e %(dir)s && cd %(dir)s" } _MAKE_TEMPLATE_ = { - "nmake" : 'if exist %(file)s "$(MAKE)" $(MAKE_FLAGS) -f %(file)s', - "gmake" : 'test -e %(file)s && "$(MAKE)" $(MAKE_FLAGS) -f %(file)s' + "win32" : 'if exist %(file)s "$(MAKE)" $(MAKE_FLAGS) -f %(file)s', + "posix" : 'test -e %(file)s && "$(MAKE)" $(MAKE_FLAGS) -f %(file)s' } _INCLUDE_CMD_ = { @@ -169,22 +170,30 @@ class BuildFile(object): # def __init__(self, AutoGenObject): self._AutoGenObject = AutoGenObject - self._FileType = gMakeType - ## Create build file + MakePath = AutoGenObject.BuildOption.get('MAKE', {}).get('PATH') + if not MakePath: + self._FileType = "" + elif "nmake" in MakePath: + self._FileType = "nmake" + else: + self._FileType = "gmake" + + if sys.platform == "win32": + self._Platform = "win32" + else: + self._Platform = "posix" + + ## Create build file. # - # @param FileType Type of build file. Only nmake and gmake are supported now. + # Only nmake and gmake are supported. # - # @retval TRUE The build file is created or re-created successfully - # @retval FALSE The build file exists and is the same as the one to be generated + # @retval TRUE The build file is created or re-created successfully + # @retval FALSE The build file exists and is the same as the one to be generated # - def Generate(self, FileType=gMakeType): - if FileType not in self._FILE_NAME_: - EdkLogger.error("build", PARAMETER_INVALID, "Invalid build type [%s]" % FileType, - ExtraData="[%s]" % str(self._AutoGenObject)) - self._FileType = FileType + def Generate(self): FileContent = self._TEMPLATE_.Replace(self._TemplateDict) - FileName = self._FILE_NAME_[FileType] + FileName = self.getMakefileName() if not os.path.exists(os.path.join(self._AutoGenObject.MakeFileDir, "deps.txt")): with open(os.path.join(self._AutoGenObject.MakeFileDir, "deps.txt"),"w+") as fd: fd.write("") @@ -203,7 +212,7 @@ class BuildFile(object): # @retval list The directory creation command list # def GetCreateDirectoryCommand(self, DirList): - return [self._MD_TEMPLATE_[self._FileType] % {'dir':Dir} for Dir in DirList] + return [self._MD_TEMPLATE_[self._Platform] % {'dir':Dir} for + Dir in DirList] ## Return a list of directory removal command string # @@ -212,7 +221,7 @@ class BuildFile(object): # @retval list The directory removal command list # def GetRemoveDirectoryCommand(self, DirList): - return [self._RD_TEMPLATE_[self._FileType] % {'dir':Dir} for Dir in DirList] + return [self._RD_TEMPLATE_[self._Platform] % {'dir':Dir} for + Dir in DirList] def PlaceMacro(self, Path, MacroDefinitions=None): if Path.startswith("$("): @@ -462,11 +471,8 @@ cleanlib: # Compose a dict object containing information used to do replacement in template @property def _TemplateDict(self): - if self._FileType not in self._SEP_: - EdkLogger.error("build", PARAMETER_INVALID, "Invalid Makefile type [%s]" % self._FileType, - ExtraData="[%s]" % str(self._AutoGenObject)) MyAgo = self._AutoGenObject - Separator = self._SEP_[self._FileType] + Separator = self._SEP_[self._Platform] # break build if no source files and binary files are found if len(MyAgo.SourceFileList) == 0 and len(MyAgo.BinaryFileList) == 0: @@ -628,10 +634,10 @@ cleanlib: BcTargetList = [] - MakefileName = self._FILE_NAME_[self._FileType] + MakefileName = self.getMakefileName() LibraryMakeCommandList = [] for D in self.LibraryBuildDirectoryList: - Command = self._MAKE_TEMPLATE_[self._FileType] % {"file":os.path.join(D, MakefileName)} + Command = self._MAKE_TEMPLATE_[self._Platform] % + {"file":os.path.join(D, MakefileName)} LibraryMakeCommandList.append(Command) package_rel_dir = MyAgo.SourceDir @@ -683,8 +689,8 @@ cleanlib: "separator" : Separator, "module_tool_definitions" : ToolsDef, - "shell_command_code" : list(self._SHELL_CMD_[self._FileType].keys()), - "shell_command" : list(self._SHELL_CMD_[self._FileType].values()), + "shell_command_code" : list(self._SHELL_CMD_[self._Platform].keys()), + "shell_command" : list(self._SHELL_CMD_[self._Platform].values()), "module_entry_point" : ModuleEntryPoint, "image_entry_point" : ImageEntryPoint, @@ -721,7 +727,7 @@ cleanlib: self.ResultFileList.append(Dst) if '%s :' %(Dst) not in self.BuildTargetList: self.BuildTargetList.append("%s :" %(Dst)) - self.BuildTargetList.append('\t' + self._CP_TEMPLATE_[self._FileType] %{'Src': Src, 'Dst': Dst}) + self.BuildTargetList.append('\t' + + self._CP_TEMPLATE_[self._Platform] %{'Src': Src, 'Dst': Dst}) FfsCmdList = Cmd[0] for index, Str in enumerate(FfsCmdList): @@ -1222,7 +1228,7 @@ ${BEGIN}\t-@${create_directory_command}\n${END}\ # Compose a dict object containing information used to do replacement in template @property def _TemplateDict(self): - Separator = self._SEP_[self._FileType] + Separator = self._SEP_[self._Platform] MyAgo = self._AutoGenObject if self._FileType not in MyAgo.CustomMakefile: EdkLogger.error('build', OPTION_NOT_SUPPORTED, "No custom makefile for %s" % self._FileType, @@ -1252,7 +1258,7 @@ ${BEGIN}\t-@${create_directory_command}\n${END}\ ToolsDef.append("%s_%s = %s" % (Tool, Attr, MyAgo.BuildOption[Tool][Attr])) ToolsDef.append("") - MakefileName = self._FILE_NAME_[self._FileType] + MakefileName = self.getMakefileName() MakefileTemplateDict = { "makefile_header" : self._FILE_HEADER_[self._FileType], "makefile_path" : os.path.join("$(MODULE_BUILD_DIR)", MakefileName), @@ -1285,8 +1291,8 @@ ${BEGIN}\t-@${create_directory_command}\n${END}\ "separator" : Separator, "module_tool_definitions" : ToolsDef, - "shell_command_code" : list(self._SHELL_CMD_[self._FileType].keys()), - "shell_command" : list(self._SHELL_CMD_[self._FileType].values()), + "shell_command_code" : list(self._SHELL_CMD_[self._Platform].keys()), + "shell_command" : list(self._SHELL_CMD_[self._Platform].values()), "create_directory_command" : self.GetCreateDirectoryCommand(self.IntermediateDirectoryList), "custom_makefile_content" : CustomMakefile @@ -1413,7 +1419,7 @@ cleanlib: # Compose a dict object containing information used to do replacement in template @property def _TemplateDict(self): - Separator = self._SEP_[self._FileType] + Separator = self._SEP_[self._Platform] MyAgo = self._AutoGenObject if "MAKE" not in MyAgo.ToolDefinition or "PATH" not in MyAgo.ToolDefinition["MAKE"]: @@ -1424,13 +1430,13 @@ cleanlib: self.ModuleBuildDirectoryList = self.GetModuleBuildDirectoryList() self.LibraryBuildDirectoryList = self.GetLibraryBuildDirectoryList() - MakefileName = self._FILE_NAME_[self._FileType] + MakefileName = self.getMakefileName() LibraryMakefileList = [] LibraryMakeCommandList = [] for D in self.LibraryBuildDirectoryList: D = self.PlaceMacro(D, {"BUILD_DIR":MyAgo.BuildDir}) Makefile = os.path.join(D, MakefileName) - Command = self._MAKE_TEMPLATE_[self._FileType] % {"file":Makefile} + Command = self._MAKE_TEMPLATE_[self._Platform] % + {"file":Makefile} LibraryMakefileList.append(Makefile) LibraryMakeCommandList.append(Command) self.LibraryMakeCommandList = LibraryMakeCommandList @@ -1440,7 +1446,7 @@ cleanlib: for D in self.ModuleBuildDirectoryList: D = self.PlaceMacro(D, {"BUILD_DIR":MyAgo.BuildDir}) Makefile = os.path.join(D, MakefileName) - Command = self._MAKE_TEMPLATE_[self._FileType] % {"file":Makefile} + Command = self._MAKE_TEMPLATE_[self._Platform] % + {"file":Makefile} ModuleMakefileList.append(Makefile) ModuleMakeCommandList.append(Command) @@ -1460,8 +1466,8 @@ cleanlib: "toolchain_tag" : MyAgo.ToolChain, "build_target" : MyAgo.BuildTarget, - "shell_command_code" : list(self._SHELL_CMD_[self._FileType].keys()), - "shell_command" : list(self._SHELL_CMD_[self._FileType].values()), + "shell_command_code" : list(self._SHELL_CMD_[self._Platform].keys()), + "shell_command" : list(self._SHELL_CMD_[self._Platform].values()), "build_architecture_list" : MyAgo.Arch, "architecture" : MyAgo.Arch, "separator" : Separator, @@ -1519,7 +1525,7 @@ class TopLevelMakefile(BuildFile): # Compose a dict object containing information used to do replacement in template @property def _TemplateDict(self): - Separator = self._SEP_[self._FileType] + Separator = self._SEP_[self._Platform] # any platform autogen object is ok because we just need common information MyAgo = self._AutoGenObject @@ -1575,10 +1581,10 @@ class TopLevelMakefile(BuildFile): else: ExtraOption += " --pcd " + pcdname + '=' + pcd[3] - MakefileName = self._FILE_NAME_[self._FileType] + MakefileName = self.getMakefileName() SubBuildCommandList = [] for A in MyAgo.ArchList: - Command = self._MAKE_TEMPLATE_[self._FileType] % {"file":os.path.join("$(BUILD_DIR)", A, MakefileName)} + Command = self._MAKE_TEMPLATE_[self._Platform] % + {"file":os.path.join("$(BUILD_DIR)", A, MakefileName)} SubBuildCommandList.append(Command) MakefileTemplateDict = { @@ -1593,8 +1599,8 @@ class TopLevelMakefile(BuildFile): "toolchain_tag" : MyAgo.ToolChain, "build_target" : MyAgo.BuildTarget, - "shell_command_code" : list(self._SHELL_CMD_[self._FileType].keys()), - "shell_command" : list(self._SHELL_CMD_[self._FileType].values()), + "shell_command_code" : list(self._SHELL_CMD_[self._Platform].keys()), + "shell_command" : list(self._SHELL_CMD_[self._Platform].values()), 'arch' : list(MyAgo.ArchList), "build_architecture_list" : ','.join(MyAgo.ArchList), "separator" : Separator, diff --git a/BaseTools/Source/Python/AutoGen/IncludesAutoGen.py b/BaseTools/Source/Python/AutoGen/IncludesAutoGen.py index 1ca1798907ef9b46a17d8938bb2f673d5ade662f..ca9e02d19b4afad82241abb8e15845731cabd22f 100644 --- a/BaseTools/Source/Python/AutoGen/IncludesAutoGen.py +++ b/BaseTools/Source/Python/AutoGen/IncludesAutoGen.py @@ -2,6 +2,7 @@ # Build cache intermediate result and state # # Copyright (c) 2019 - 2020, Intel Corporation. All rights reserved.<BR> +# Copyright (c) 2020, ARM Limited. All rights reserved.<BR> # SPDX-License-Identifier: BSD-2-Clause-Patent # from Common.caching import cached_property @@ -12,20 +13,6 @@ from Common.Misc import SaveFileOnChange, PathClass from Common.Misc import TemplateString import sys gIsFileMap = {} -if sys.platform == "win32": - _INCLUDE_DEPS_TEMPLATE = TemplateString(''' -${BEGIN} -!IF EXIST(${deps_file}) -!INCLUDE ${deps_file} -!ENDIF -${END} - ''') -else: - _INCLUDE_DEPS_TEMPLATE = TemplateString(''' -${BEGIN} --include ${deps_file} -${END} - ''') DEP_FILE_TAIL = "# Updated \n" @@ -59,6 +46,25 @@ class IncludesAutoGen(): def CreateDepsInclude(self): deps_file = {'deps_file':self.deps_files} + + MakePath = self.module_autogen.BuildOption.get('MAKE', {}).get('PATH') + if not MakePath: + EdkLogger.error("build", PARAMETER_MISSING, Message="No Make path available.") + elif "nmake" in MakePath: + _INCLUDE_DEPS_TEMPLATE = TemplateString(''' + ${BEGIN} + !IF EXIST(${deps_file}) + !INCLUDE ${deps_file} + !ENDIF + ${END} + ''') + else: + _INCLUDE_DEPS_TEMPLATE = TemplateString(''' + ${BEGIN} + -include ${deps_file} + ${END} + ''') + try: deps_include_str = _INCLUDE_DEPS_TEMPLATE.Replace(deps_file) except Exception as e: diff --git a/BaseTools/Source/Python/build/build.py b/BaseTools/Source/Python/build/build.py index 1e47e382cba94dc6fba7d8ecad2bc924b02cea1d..8e6625ae42afe5ed5fddbdb72ef5e8ca044609b4 100755 --- a/BaseTools/Source/Python/build/build.py +++ b/BaseTools/Source/Python/build/build.py @@ -4,6 +4,7 @@ # Copyright (c) 2014, Hewlett-Packard Development Company, L.P.<BR> # Copyright (c) 2007 - 2020, Intel Corporation. All rights reserved.<BR> # Copyright (c) 2018, Hewlett Packard Enterprise Development, L.P.<BR> +# Copyright (c) 2020, ARM Limited. All rights reserved.<BR> # # SPDX-License-Identifier: BSD-2-Clause-Patent # @@ -1251,7 +1252,7 @@ class Build(): (AutoGenObject.BuildTarget, AutoGenObject.ToolChain, AutoGenObject.Arch), ExtraData=str(AutoGenObject)) - makefile = GenMake.BuildFile(AutoGenObject)._FILE_NAME_[GenMake.gMakeType] + makefile = GenMake.BuildFile(AutoGenObject).getMakefileName() # run if Target == 'run': @@ -2040,10 +2041,10 @@ class Build(): ModuleBuildDirectoryList = data_pipe.Get("ModuleBuildDirectoryList") for m_build_dir in LibraryBuildDirectoryList: - if not os.path.exists(os.path.join(m_build_dir,GenMake.BuildFile._FILE_NAME_[GenMake.gMakeType])): + if not os.path.exists(os.path.join(m_build_dir,GenMake.BuildFile.getMakefileName())): return None for m_build_dir in ModuleBuildDirectoryList: - if not os.path.exists(os.path.join(m_build_dir,GenMake.BuildFile._FILE_NAME_[GenMake.gMakeType])): + if not os.path.exists(os.path.join(m_build_dir,GenMake.BuildFile.getMakefileName())): return None Wa = WorkSpaceInfo( workspacedir,active_p,target,toolchain,archlist -- 'Guid(CE165669-3EF3-493F-B85D-6190EE5B9759)' ^ permalink raw reply related [flat|nested] 4+ messages in thread
end of thread, other threads:[~2020-02-05 12:01 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <15F0315032D95F8E.2867@groups.io>
2020-02-04 12:02 ` [edk2-devel] [PATCH v2 1/1] BaseTools: Rationalise makefile generation PierreGondois
2020-02-04 13:50 ` Liming Gao
2020-02-04 14:42 ` PierreGondois
2020-02-05 12:01 ` Bob Feng
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox