Re: [edk2-devel][PATCH V1 1/1] BaseTools: Generate compile information in build report

["Ni, Ray" <ray.ni@intel.com>]

Christine,
If BaseTools related changes is implemented in edk2-basetools repo,
does that mean if I only checkout edk2 repo, I am using an older version of BaseTools?

Thanks,
Ray

> -----Original Message-----
> From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Yuwei
> Chen
> Sent: Wednesday, February 15, 2023 1:38 PM
> To: Kinney, Michael D <michael.d.kinney@intel.com>; devel@edk2.groups.io;
> Palomino Sosa, Guillermo A <guillermo.a.palomino.sosa@intel.com>
> Cc: Feng, Bob C <bob.c.feng@intel.com>; Gao, Liming
> <gaoliming@byosoft.com.cn>
> Subject: Re: [edk2-devel][PATCH V1 1/1] BaseTools: Generate compile
> information in build report
> 
> Hi Mike, thanks for reminder.
> 
> Hi Willy, currently, BaseTools related changes will be implemented on the
> edk2-basetools repo. Please send the patch based on the edk2-basetools
> repo~
> 
> Thanks,
> Christine
> 
> > -----Original Message-----
> > From: Kinney, Michael D <michael.d.kinney@intel.com>
> > Sent: Wednesday, February 15, 2023 11:43 AM
> > To: devel@edk2.groups.io; Chen, Christine <yuwei.chen@intel.com>;
> > Palomino Sosa, Guillermo A <guillermo.a.palomino.sosa@intel.com>
> > Cc: Feng, Bob C <bob.c.feng@intel.com>; Gao, Liming
> > <gaoliming@byosoft.com.cn>; Kinney, Michael D
> > <michael.d.kinney@intel.com>
> > Subject: RE: [edk2-devel][PATCH V1 1/1] BaseTools: Generate compile
> > information in build report
> >
> > Has this been reviewed for edk2-basetools repo?
> >
> > Mike
> >
> > > -----Original Message-----
> > > From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of
> Yuwei
> > > Chen
> > > Sent: Tuesday, February 14, 2023 6:44 PM
> > > To: Palomino Sosa, Guillermo A <guillermo.a.palomino.sosa@intel.com>;
> > > devel@edk2.groups.io
> > > Cc: Feng, Bob C <bob.c.feng@intel.com>; Gao, Liming
> > > <gaoliming@byosoft.com.cn>
> > > Subject: Re: [edk2-devel][PATCH V1 1/1] BaseTools: Generate compile
> > > information in build report
> > >
> > > Reviewed-by: Yuwei Chen <yuwei.chen@intel.com>
> > >
> > > > -----Original Message-----
> > > > From: Palomino Sosa, Guillermo A
> > > > <guillermo.a.palomino.sosa@intel.com>
> > > > Sent: Tuesday, February 7, 2023 11:07 AM
> > > > To: devel@edk2.groups.io
> > > > Cc: Feng, Bob C <bob.c.feng@intel.com>; Gao, Liming
> > > > <gaoliming@byosoft.com.cn>; Chen, Christine <yuwei.chen@intel.com>
> > > > Subject: [edk2-devel][PATCH V1 1/1] BaseTools: Generate compile
> > > > information in build report
> > > >
> > > > REF:https://bugzilla.tianocore.org/show_bug.cgi?id=2850
> > > >
> > > > Add "-Y REPORT_INFO" option to build command to generate compile
> > > > information as part of BuildReport.
> > > > This option generates files to be used by external tools as IDE's to
> > > > enhance functionality.
> > > > Files are created inside build folder:
> > > > <Build>/<BuildTarget>/<ToolChain>/CompileInfo
> > > >
> > > > Files created:
> > > > * compile_commands.json - Compilation Database. To be used by IDE's
> > > >   to enable advance features
> > > > * cscope.files - List of files used in compilation. Used by Cscope to parse
> > > >   C code and provide browse functionality.
> > > > * module_report.json - Module data form buildReport in Json format.
> > > >
> > > > Signed-off-by: Guillermo Antonio Palomino Sosa
> > > > <guillermo.a.palomino.sosa@intel.com>
> > > > ---
> > > >  BaseTools/Source/Python/build/BuildReport.py  | 139
> > > > +++++++++++++++++++-
> > > >  BaseTools/Source/Python/build/buildoptions.py |   4 +-
> > > >  2 files changed, 140 insertions(+), 3 deletions(-)
> > > >
> > > > diff --git a/BaseTools/Source/Python/build/BuildReport.py
> > > > b/BaseTools/Source/Python/build/BuildReport.py
> > > > index 468772930c..33b43d471f 100644
> > > > --- a/BaseTools/Source/Python/build/BuildReport.py
> > > > +++ b/BaseTools/Source/Python/build/BuildReport.py
> > > > @@ -10,6 +10,8 @@
> > > >
> > > >  ## Import Modules
> > > >  #
> > > > +import json
> > > > +from pathlib import Path
> > > >  import Common.LongFilePathOs as os
> > > >  import re
> > > >  import platform
> > > > @@ -41,6 +43,7 @@ from Common.DataType import *  import
> > collections
> > > > from Common.Expression import *  from GenFds.AprioriSection import
> > > > DXE_APRIORI_GUID, PEI_APRIORI_GUID
> > > > +from AutoGen.IncludesAutoGen import IncludesAutoGen
> > > >
> > > >  ## Pattern to extract contents in EDK DXS files
> > > > gDxsDependencyPattern =
> > > > re.compile(r"DEPENDENCY_START(.+)DEPENDENCY_END", re.DOTALL)
> > @@ -
> > > > 2298,6 +2301,10 @@ class BuildReport(object):
> > > >      def GenerateReport(self, BuildDuration, AutoGenTime, MakeTime,
> > > > GenFdsTime):
> > > >          if self.ReportFile:
> > > >              try:
> > > > +
> > > > +                if "COMPILE_INFO" in self.ReportType:
> > > > +                    self.GenerateCompileInfo()
> > > > +
> > > >                  File = []
> > > >                  for (Wa, MaList) in self.ReportList:
> > > >                      PlatformReport(Wa, MaList,
> > > > self.ReportType).GenerateReport(File, BuildDuration, AutoGenTime,
> > > > MakeTime, GenFdsTime, self.ReportType) @@ -2310,7 +2317,137 @@
> > class
> > > > BuildReport(object):
> > > >                  EdkLogger.error("BuildReport", CODE_ERROR, "Unknown
> > > > fatal error when generating build report",
> > > > ExtraData=self.ReportFile,
> > > > RaiseError=False)
> > > >                  EdkLogger.quiet("(Python %s on %s\n%s)" %
> > > > (platform.python_version(), sys.platform, traceback.format_exc()))
> > > >
> > > > +
> > > > +    ##
> > > > +    # Generates compile data files to be used by external tools.
> > > > +    # Compile information will be generated in
> > > > <Build>/<BuildTarget>/<ToolChain>/CompileInfo
> > > > +    # Files generated: compile_commands.json, cscope.files,
> > > > modules_report.json
> > > > +    #
> > > > +    # @param self            The object pointer
> > > > +    #
> > > > +    def GenerateCompileInfo(self):
> > > > +        try:
> > > > +            # Lists for the output elements
> > > > +            compile_commands = []
> > > > +            used_files = set()
> > > > +            module_report = []
> > > > +
> > > > +            for (Wa, MaList) in self.ReportList:
> > > > +                # Obtain list of all processed Workspace files
> > > > +                for file_path in Wa._GetMetaFiles(Wa.BuildTarget,
> > Wa.ToolChain):
> > > > +                    used_files.add(file_path)
> > > > +
> > > > +                for autoGen in Wa.AutoGenObjectList:
> > > > +
> > > > +                    # Loop through all modules
> > > > +                    for module in (autoGen.LibraryAutoGenList +
> > > > autoGen.ModuleAutoGenList):
> > > > +
> > > > +                        used_files.add(module.MetaFile.Path)
> > > > +
> > > > +                        # Main elements of module report
> > > > +                        module_report_data = {}
> > > > +                        module_report_data["Name"] = module.Name
> > > > +                        module_report_data["Arch"] = module.Arch
> > > > +                        module_report_data["Path"] = module.MetaFile.Path
> > > > +                        module_report_data["Guid"] = module.Guid
> > > > +                        module_report_data["BuildType"] = module.BuildType
> > > > +                        module_report_data["IsLibrary"] = module.IsLibrary
> > > > +                        module_report_data["SourceDir"] = module.SourceDir
> > > > +                        module_report_data["Files"] = []
> > > > +
> > > > +                        # Files used by module
> > > > +                        for data_file in module.SourceFileList:
> > > > +                            module_report_data["Files"].append({"Name":
> > > > + data_file.Name, "Path": data_file.Path})
> > > > +
> > > > +                        # Libraries used by module
> > > > +                        module_report_data["Libraries"] = []
> > > > +                        for data_library in module.LibraryAutoGenList:
> > > > +
> > > > + module_report_data["Libraries"].append({"Path":
> > > > + data_library.MetaFile.Path})
> > > > +
> > > > +                        # Packages used by module
> > > > +                        module_report_data["Packages"] = []
> > > > +                        for data_package in module.PackageList:
> > > > +                            module_report_data["Packages"].append({"Path":
> > > > data_package.MetaFile.Path, "Includes": []})
> > > > +                            # Includes path used in package
> > > > +                            for data_package_include in data_package.Includes:
> > > > +
> > > > + module_report_data["Packages"][-
> > > > 1]["Includes"].append(data_package_inc
> > > > + lude.Path)
> > > > +
> > > > +                        # PPI's in module
> > > > +                        module_report_data["PPI"] = []
> > > > +                        for data_ppi in module.PpiList.keys():
> > > > +                            module_report_data["PPI"].append({"Name":
> > > > + data_ppi, "Guid": module.PpiList[data_ppi]})
> > > > +
> > > > +                        # Protocol's in module
> > > > +                        module_report_data["Protocol"] = []
> > > > +                        for data_protocol in module.ProtocolList.keys():
> > > > +
> > > > + module_report_data["Protocol"].append({"Name": data_protocol,
> > "Guid":
> > > > + module.ProtocolList[data_protocol]})
> > > > +
> > > > +                        # PCD's in module
> > > > +                        module_report_data["Pcd"] = []
> > > > +                        for data_pcd in module.LibraryPcdList:
> > > > +                            module_report_data["Pcd"].append({"Space":
> > > > data_pcd.TokenSpaceGuidCName,
> > > > +                                                              "Name": data_pcd.TokenCName,
> > > > +                                                              "Value": data_pcd.TokenValue,
> > > > +                                                              "Guid":
> > data_pcd.TokenSpaceGuidValue,
> > > > +                                                              "DatumType": data_pcd.DatumType,
> > > > +                                                              "Type": data_pcd.Type,
> > > > +                                                              "DefaultValue":
> > data_pcd.DefaultValue})
> > > > +                        # Add module to report
> > > > +                        module_report.append(module_report_data)
> > > > +
> > > > +                        # Include file dependencies to used files
> > > > +                        includes_autogen =
> > > > + IncludesAutoGen(module.MakeFileDir,
> > > > module)
> > > > +                        for dep in includes_autogen.DepsCollection:
> > > > +                            used_files.add(dep)
> > > > +
> > > > +                        inc_flag = "-I" # Default include flag
> > > > +                        if module.BuildRuleFamily == TAB_COMPILER_MSFT:
> > > > +                            inc_flag = "/I"
> > > > +
> > > > +                        for source in module.SourceFileList:
> > > > +                            used_files.add(source.Path)
> > > > +                            compile_command = {}
> > > > +                            if source.Ext in [".c", ".cc", ".cpp"]:
> > > > +                                #
> > > > +                                # Generate compile command for each c file
> > > > +                                #
> > > > +                                compile_command["file"] = source.Path
> > > > +                                compile_command["directory"] = source.Dir
> > > > +                                build_command =
> > > > module.BuildRules[source.Ext].CommandList[0]
> > > > +                                build_command_variables =
> > > > + re.findall(r"\$\((.*?)\)",
> > > > build_command)
> > > > +                                for var in build_command_variables:
> > > > +                                    var_tokens = var.split("_")
> > > > +                                    var_main = var_tokens[0]
> > > > +                                    if len(var_tokens) == 1:
> > > > +                                        var_value =
> > module.BuildOption[var_main]["PATH"]
> > > > +                                    else:
> > > > +                                        var_value =
> > > > module.BuildOption[var_main][var_tokens[1]]
> > > > +                                    build_command =
> > > > + build_command.replace(f"$({var})",
> > > > var_value)
> > > > +                                    include_files = f"
> > {inc_flag}".join(module.IncludePathList)
> > > > +                                    build_command =
> > > > + build_command.replace("${src}",
> > > > include_files)
> > > > +                                    build_command =
> > > > + build_command.replace("${dst}", module.OutputDir)
> > > > +
> > > > +                                # Remove un defined macros
> > > > +                                compile_command["command"] =
> > > > + re.sub(r"\$\(.*?\)", "",
> > > > build_command)
> > > > +
> > > > + compile_commands.append(compile_command)
> > > > +
> > > > +                # Create output folder if doesn't exist
> > > > +                compile_info_folder =
> > Path(Wa.BuildDir).joinpath("CompileInfo")
> > > > +                compile_info_folder.mkdir(exist_ok=True)
> > > > +
> > > > +                # Sort and save files
> > > > +                compile_commands.sort(key=lambda x: x["file"])
> > > > +
> > > > +
> > > >
> >
> SaveFileOnChange(compile_info_folder.joinpath(f"compile_commands.json
> "
> > > > + ),json.dumps(compile_commands, indent=2), False)
> > > > +
> > > > +
> > > > + SaveFileOnChange(compile_info_folder.joinpath(f"cscope.files"),
> > > > + "\n".join(sorted(used_files)), False)
> > > > +
> > > > +                module_report.sort(key=lambda x: x["Path"])
> > > > +
> > > > +
> SaveFileOnChange(compile_info_folder.joinpath(f"module_report.json
> > > > + "), json.dumps(module_report, indent=2), False)
> > > > +
> > > > +        except:
> > > > +            EdkLogger.error("BuildReport", CODE_ERROR, "Unknown
> > > > + fatal error
> > > > when generating build report compile information",
> > > > ExtraData=self.ReportFile, RaiseError=False)
> > > > +            EdkLogger.quiet("(Python %s on %s\n%s)" %
> > > > + (platform.python_version(), sys.platform, traceback.format_exc()))
> > > > +
> > > >  # This acts like the main() function for the script, unless it is
> > > > 'import'ed into another script.
> > > >  if __name__ == '__main__':
> > > >      pass
> > > > -
> > > > diff --git a/BaseTools/Source/Python/build/buildoptions.py
> > > > b/BaseTools/Source/Python/build/buildoptions.py
> > > > index 8334604b46..5ec561f7ec 100644
> > > > --- a/BaseTools/Source/Python/build/buildoptions.py
> > > > +++ b/BaseTools/Source/Python/build/buildoptions.py
> > > > @@ -84,8 +84,8 @@ class MyOptionParser():
> > > >          Parser.add_option("-D", "--define", action="append",
> > > > type="string", dest="Macros", help="Macro: \"Name [= Value]\".")
> > > >
> > > >          Parser.add_option("-y", "--report-file", action="store",
> > > > dest="ReportFile", help="Create/overwrite the report to the
> > > > specified
> > > > filename.")
> > > > -        Parser.add_option("-Y", "--report-type", action="append",
> > > > type="choice", choices=['PCD', 'LIBRARY', 'FLASH', 'DEPEX',
> > > > 'BUILD_FLAGS', 'FIXED_ADDRESS', 'HASH', 'EXECUTION_ORDER'],
> > > > dest="ReportType", default=[],
> > > > -            help="Flags that control the type of build report to generate.
> > Must be
> > > > one of: [PCD, LIBRARY, FLASH, DEPEX, BUILD_FLAGS, FIXED_ADDRESS,
> > > > HASH, EXECUTION_ORDER].  "\
> > > > +        Parser.add_option("-Y", "--report-type", action="append",
> > > > type="choice", choices=['PCD', 'LIBRARY', 'FLASH', 'DEPEX',
> > > > 'BUILD_FLAGS', 'FIXED_ADDRESS', 'HASH', 'EXECUTION_ORDER',
> > > > 'COMPILE_INFO'], dest="ReportType", default=[],
> > > > +            help="Flags that control the type of build report to
> > > > + generate.  Must be one of: [PCD, LIBRARY, FLASH, DEPEX,
> > > > + BUILD_FLAGS, FIXED_ADDRESS, HASH, EXECUTION_ORDER,
> > COMPILE_INFO].
> > > > + "\
> > > >                   "To specify more than one flag, repeat this option
> > > > on the command line and the default flag set is [PCD, LIBRARY,
> > > > FLASH, DEPEX, HASH, BUILD_FLAGS, FIXED_ADDRESS]")
> > > >          Parser.add_option("-F", "--flag", action="store",
> > > > type="string", dest="Flag",
> > > >              help="Specify the specific option to parse EDK UNI file. Must be
> > one of:
> > > > [-c, -s]. -c is for EDK framework UNI file, and -s is for EDK UEFI
> > > > UNI file. "\
> > > > --
> > > > 2.28.0.windows.1
> > >
> > >
> > >
> > >
> > >
> 
> 
> 
> 
>