From: "Michael D Kinney" <michael.d.kinney@intel.com>
To: "devel@edk2.groups.io" <devel@edk2.groups.io>,
"Chen, Christine" <yuwei.chen@intel.com>,
"Palomino Sosa,
Guillermo A" <guillermo.a.palomino.sosa@intel.com>
Cc: "Feng, Bob C" <bob.c.feng@intel.com>,
"Gao, Liming" <gaoliming@byosoft.com.cn>,
"Kinney, Michael D" <michael.d.kinney@intel.com>
Subject: Re: [edk2-devel][PATCH V1 1/1] BaseTools: Generate compile information in build report
Date: Wed, 15 Feb 2023 03:42:43 +0000 [thread overview]
Message-ID: <CO1PR11MB49295F1C4CAC44C268DB0D00D2A39@CO1PR11MB4929.namprd11.prod.outlook.com> (raw)
In-Reply-To: <MW5PR11MB59066FFAE3DD5EE09D7C6ABF96A39@MW5PR11MB5906.namprd11.prod.outlook.com>
Has this been reviewed for edk2-basetools repo?
Mike
> -----Original Message-----
> From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Yuwei Chen
> Sent: Tuesday, February 14, 2023 6:44 PM
> To: Palomino Sosa, Guillermo A <guillermo.a.palomino.sosa@intel.com>; devel@edk2.groups.io
> Cc: Feng, Bob C <bob.c.feng@intel.com>; Gao, Liming <gaoliming@byosoft.com.cn>
> Subject: Re: [edk2-devel][PATCH V1 1/1] BaseTools: Generate compile information in build report
>
> Reviewed-by: Yuwei Chen <yuwei.chen@intel.com>
>
> > -----Original Message-----
> > From: Palomino Sosa, Guillermo A <guillermo.a.palomino.sosa@intel.com>
> > Sent: Tuesday, February 7, 2023 11:07 AM
> > To: devel@edk2.groups.io
> > Cc: Feng, Bob C <bob.c.feng@intel.com>; Gao, Liming
> > <gaoliming@byosoft.com.cn>; Chen, Christine <yuwei.chen@intel.com>
> > Subject: [edk2-devel][PATCH V1 1/1] BaseTools: Generate compile
> > information in build report
> >
> > REF:https://bugzilla.tianocore.org/show_bug.cgi?id=2850
> >
> > Add "-Y REPORT_INFO" option to build command to generate compile
> > information as part of BuildReport.
> > This option generates files to be used by external tools as IDE's to enhance
> > functionality.
> > Files are created inside build folder:
> > <Build>/<BuildTarget>/<ToolChain>/CompileInfo
> >
> > Files created:
> > * compile_commands.json - Compilation Database. To be used by IDE's
> > to enable advance features
> > * cscope.files - List of files used in compilation. Used by Cscope to parse
> > C code and provide browse functionality.
> > * module_report.json - Module data form buildReport in Json format.
> >
> > Signed-off-by: Guillermo Antonio Palomino Sosa
> > <guillermo.a.palomino.sosa@intel.com>
> > ---
> > BaseTools/Source/Python/build/BuildReport.py | 139
> > +++++++++++++++++++-
> > BaseTools/Source/Python/build/buildoptions.py | 4 +-
> > 2 files changed, 140 insertions(+), 3 deletions(-)
> >
> > diff --git a/BaseTools/Source/Python/build/BuildReport.py
> > b/BaseTools/Source/Python/build/BuildReport.py
> > index 468772930c..33b43d471f 100644
> > --- a/BaseTools/Source/Python/build/BuildReport.py
> > +++ b/BaseTools/Source/Python/build/BuildReport.py
> > @@ -10,6 +10,8 @@
> >
> > ## Import Modules
> > #
> > +import json
> > +from pathlib import Path
> > import Common.LongFilePathOs as os
> > import re
> > import platform
> > @@ -41,6 +43,7 @@ from Common.DataType import * import collections
> > from Common.Expression import * from GenFds.AprioriSection import
> > DXE_APRIORI_GUID, PEI_APRIORI_GUID
> > +from AutoGen.IncludesAutoGen import IncludesAutoGen
> >
> > ## Pattern to extract contents in EDK DXS files gDxsDependencyPattern =
> > re.compile(r"DEPENDENCY_START(.+)DEPENDENCY_END", re.DOTALL) @@ -
> > 2298,6 +2301,10 @@ class BuildReport(object):
> > def GenerateReport(self, BuildDuration, AutoGenTime, MakeTime,
> > GenFdsTime):
> > if self.ReportFile:
> > try:
> > +
> > + if "COMPILE_INFO" in self.ReportType:
> > + self.GenerateCompileInfo()
> > +
> > File = []
> > for (Wa, MaList) in self.ReportList:
> > PlatformReport(Wa, MaList,
> > self.ReportType).GenerateReport(File, BuildDuration, AutoGenTime,
> > MakeTime, GenFdsTime, self.ReportType) @@ -2310,7 +2317,137 @@ class
> > BuildReport(object):
> > EdkLogger.error("BuildReport", CODE_ERROR, "Unknown fatal
> > error when generating build report", ExtraData=self.ReportFile,
> > RaiseError=False)
> > EdkLogger.quiet("(Python %s on %s\n%s)" %
> > (platform.python_version(), sys.platform, traceback.format_exc()))
> >
> > +
> > + ##
> > + # Generates compile data files to be used by external tools.
> > + # Compile information will be generated in
> > <Build>/<BuildTarget>/<ToolChain>/CompileInfo
> > + # Files generated: compile_commands.json, cscope.files,
> > modules_report.json
> > + #
> > + # @param self The object pointer
> > + #
> > + def GenerateCompileInfo(self):
> > + try:
> > + # Lists for the output elements
> > + compile_commands = []
> > + used_files = set()
> > + module_report = []
> > +
> > + for (Wa, MaList) in self.ReportList:
> > + # Obtain list of all processed Workspace files
> > + for file_path in Wa._GetMetaFiles(Wa.BuildTarget, Wa.ToolChain):
> > + used_files.add(file_path)
> > +
> > + for autoGen in Wa.AutoGenObjectList:
> > +
> > + # Loop through all modules
> > + for module in (autoGen.LibraryAutoGenList +
> > autoGen.ModuleAutoGenList):
> > +
> > + used_files.add(module.MetaFile.Path)
> > +
> > + # Main elements of module report
> > + module_report_data = {}
> > + module_report_data["Name"] = module.Name
> > + module_report_data["Arch"] = module.Arch
> > + module_report_data["Path"] = module.MetaFile.Path
> > + module_report_data["Guid"] = module.Guid
> > + module_report_data["BuildType"] = module.BuildType
> > + module_report_data["IsLibrary"] = module.IsLibrary
> > + module_report_data["SourceDir"] = module.SourceDir
> > + module_report_data["Files"] = []
> > +
> > + # Files used by module
> > + for data_file in module.SourceFileList:
> > + module_report_data["Files"].append({"Name":
> > + data_file.Name, "Path": data_file.Path})
> > +
> > + # Libraries used by module
> > + module_report_data["Libraries"] = []
> > + for data_library in module.LibraryAutoGenList:
> > +
> > + module_report_data["Libraries"].append({"Path":
> > + data_library.MetaFile.Path})
> > +
> > + # Packages used by module
> > + module_report_data["Packages"] = []
> > + for data_package in module.PackageList:
> > + module_report_data["Packages"].append({"Path":
> > data_package.MetaFile.Path, "Includes": []})
> > + # Includes path used in package
> > + for data_package_include in data_package.Includes:
> > +
> > + module_report_data["Packages"][-
> > 1]["Includes"].append(data_package_inc
> > + lude.Path)
> > +
> > + # PPI's in module
> > + module_report_data["PPI"] = []
> > + for data_ppi in module.PpiList.keys():
> > + module_report_data["PPI"].append({"Name":
> > + data_ppi, "Guid": module.PpiList[data_ppi]})
> > +
> > + # Protocol's in module
> > + module_report_data["Protocol"] = []
> > + for data_protocol in module.ProtocolList.keys():
> > +
> > + module_report_data["Protocol"].append({"Name": data_protocol, "Guid":
> > + module.ProtocolList[data_protocol]})
> > +
> > + # PCD's in module
> > + module_report_data["Pcd"] = []
> > + for data_pcd in module.LibraryPcdList:
> > + module_report_data["Pcd"].append({"Space":
> > data_pcd.TokenSpaceGuidCName,
> > + "Name": data_pcd.TokenCName,
> > + "Value": data_pcd.TokenValue,
> > + "Guid": data_pcd.TokenSpaceGuidValue,
> > + "DatumType": data_pcd.DatumType,
> > + "Type": data_pcd.Type,
> > + "DefaultValue": data_pcd.DefaultValue})
> > + # Add module to report
> > + module_report.append(module_report_data)
> > +
> > + # Include file dependencies to used files
> > + includes_autogen = IncludesAutoGen(module.MakeFileDir,
> > module)
> > + for dep in includes_autogen.DepsCollection:
> > + used_files.add(dep)
> > +
> > + inc_flag = "-I" # Default include flag
> > + if module.BuildRuleFamily == TAB_COMPILER_MSFT:
> > + inc_flag = "/I"
> > +
> > + for source in module.SourceFileList:
> > + used_files.add(source.Path)
> > + compile_command = {}
> > + if source.Ext in [".c", ".cc", ".cpp"]:
> > + #
> > + # Generate compile command for each c file
> > + #
> > + compile_command["file"] = source.Path
> > + compile_command["directory"] = source.Dir
> > + build_command =
> > module.BuildRules[source.Ext].CommandList[0]
> > + build_command_variables = re.findall(r"\$\((.*?)\)",
> > build_command)
> > + for var in build_command_variables:
> > + var_tokens = var.split("_")
> > + var_main = var_tokens[0]
> > + if len(var_tokens) == 1:
> > + var_value = module.BuildOption[var_main]["PATH"]
> > + else:
> > + var_value =
> > module.BuildOption[var_main][var_tokens[1]]
> > + build_command = build_command.replace(f"$({var})",
> > var_value)
> > + include_files = f" {inc_flag}".join(module.IncludePathList)
> > + build_command = build_command.replace("${src}",
> > include_files)
> > + build_command =
> > + build_command.replace("${dst}", module.OutputDir)
> > +
> > + # Remove un defined macros
> > + compile_command["command"] = re.sub(r"\$\(.*?\)", "",
> > build_command)
> > +
> > + compile_commands.append(compile_command)
> > +
> > + # Create output folder if doesn't exist
> > + compile_info_folder = Path(Wa.BuildDir).joinpath("CompileInfo")
> > + compile_info_folder.mkdir(exist_ok=True)
> > +
> > + # Sort and save files
> > + compile_commands.sort(key=lambda x: x["file"])
> > +
> > +
> > SaveFileOnChange(compile_info_folder.joinpath(f"compile_commands.json"
> > + ),json.dumps(compile_commands, indent=2), False)
> > +
> > +
> > + SaveFileOnChange(compile_info_folder.joinpath(f"cscope.files"),
> > + "\n".join(sorted(used_files)), False)
> > +
> > + module_report.sort(key=lambda x: x["Path"])
> > +
> > + SaveFileOnChange(compile_info_folder.joinpath(f"module_report.json"),
> > + json.dumps(module_report, indent=2), False)
> > +
> > + except:
> > + EdkLogger.error("BuildReport", CODE_ERROR, "Unknown fatal error
> > when generating build report compile information",
> > ExtraData=self.ReportFile, RaiseError=False)
> > + EdkLogger.quiet("(Python %s on %s\n%s)" %
> > + (platform.python_version(), sys.platform, traceback.format_exc()))
> > +
> > # This acts like the main() function for the script, unless it is 'import'ed into
> > another script.
> > if __name__ == '__main__':
> > pass
> > -
> > diff --git a/BaseTools/Source/Python/build/buildoptions.py
> > b/BaseTools/Source/Python/build/buildoptions.py
> > index 8334604b46..5ec561f7ec 100644
> > --- a/BaseTools/Source/Python/build/buildoptions.py
> > +++ b/BaseTools/Source/Python/build/buildoptions.py
> > @@ -84,8 +84,8 @@ class MyOptionParser():
> > Parser.add_option("-D", "--define", action="append", type="string",
> > dest="Macros", help="Macro: \"Name [= Value]\".")
> >
> > Parser.add_option("-y", "--report-file", action="store",
> > dest="ReportFile", help="Create/overwrite the report to the specified
> > filename.")
> > - Parser.add_option("-Y", "--report-type", action="append",
> > type="choice", choices=['PCD', 'LIBRARY', 'FLASH', 'DEPEX', 'BUILD_FLAGS',
> > 'FIXED_ADDRESS', 'HASH', 'EXECUTION_ORDER'], dest="ReportType",
> > default=[],
> > - help="Flags that control the type of build report to generate. Must be
> > one of: [PCD, LIBRARY, FLASH, DEPEX, BUILD_FLAGS, FIXED_ADDRESS, HASH,
> > EXECUTION_ORDER]. "\
> > + Parser.add_option("-Y", "--report-type", action="append",
> > type="choice", choices=['PCD', 'LIBRARY', 'FLASH', 'DEPEX', 'BUILD_FLAGS',
> > 'FIXED_ADDRESS', 'HASH', 'EXECUTION_ORDER', 'COMPILE_INFO'],
> > dest="ReportType", default=[],
> > + help="Flags that control the type of build report to
> > + generate. Must be one of: [PCD, LIBRARY, FLASH, DEPEX, BUILD_FLAGS,
> > + FIXED_ADDRESS, HASH, EXECUTION_ORDER, COMPILE_INFO]. "\
> > "To specify more than one flag, repeat this option on the command
> > line and the default flag set is [PCD, LIBRARY, FLASH, DEPEX, HASH,
> > BUILD_FLAGS, FIXED_ADDRESS]")
> > Parser.add_option("-F", "--flag", action="store", type="string",
> > dest="Flag",
> > help="Specify the specific option to parse EDK UNI file. Must be one of:
> > [-c, -s]. -c is for EDK framework UNI file, and -s is for EDK UEFI UNI file. "\
> > --
> > 2.28.0.windows.1
>
>
>
>
>
next prev parent reply other threads:[~2023-02-15 3:42 UTC|newest]
Thread overview: 17+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-02-07 3:07 [edk2-devel][PATCH V1 1/1] BaseTools: Generate compile information in build report Guillermo Antonio Palomino Sosa
2023-02-15 2:43 ` Yuwei Chen
2023-02-15 3:42 ` Michael D Kinney [this message]
2023-02-15 5:38 ` Yuwei Chen
2023-02-15 8:17 ` Ni, Ray
2023-02-15 23:12 ` Michael D Kinney
2023-02-16 0:50 ` Ni, Ray
2023-02-18 4:42 ` [edk2-devel] [PATCH " Guillermo Antonio Palomino Sosa
2023-02-27 17:40 ` [edk2-devel][PATCH " Guillermo Antonio Palomino Sosa
2023-02-27 18:57 ` Ard Biesheuvel
2023-02-27 22:35 ` Michael D Kinney
2023-02-27 23:42 ` Guillermo Antonio Palomino Sosa
2023-03-01 0:52 ` Ni, Ray
2023-03-01 4:00 ` Guillermo Antonio Palomino Sosa
2023-03-06 15:40 ` Guillermo Antonio Palomino Sosa
2023-03-15 1:12 ` Ni, Ray
2023-03-15 10:28 ` Bob Feng
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-list from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=CO1PR11MB49295F1C4CAC44C268DB0D00D2A39@CO1PR11MB4929.namprd11.prod.outlook.com \
--to=devel@edk2.groups.io \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox