* [Patch 0/1] BaseTools/Scripts: Add package dependency graphing tool @ 2019-12-13 19:44 Michael D Kinney 2019-12-13 19:44 ` [Patch 1/1] " Michael D Kinney 0 siblings, 1 reply; 12+ messages in thread From: Michael D Kinney @ 2019-12-13 19:44 UTC (permalink / raw) To: devel; +Cc: Ray Ni, Bob Feng, Liming Gao, Sean Brogan, Bret Barkelew https://bugzilla.tianocore.org/show_bug.cgi?id=2161 Add python script that recursively scans a directory for EDK II packages and generates GraphViz dot input that is used to render a graph of package dependencies in SVG format. Detects following error/warning conditions: * Ambiguous dependencies (multiple matches) * Unresolved dependencies * Circular dependencies * Nested packages Links to sample graphs https://github.com/mdkinney/edk2/wiki/images/edk2.svg?sanitize=true https://github.com/mdkinney/edk2/wiki/images/edk2-SkipPlatforms.svg?sanitize=true https://github.com/mdkinney/edk2/wiki/images/edk2-SkipPlatformsAndArm.svg?sanitize=true https://github.com/mdkinney/edk2/wiki/images/edk2-platforms.svg?sanitize=true https://github.com/mdkinney/edk2/wiki/images/edk2-platforms-HideUnresolved.svg?sanitize=true https://github.com/mdkinney/edk2/wiki/images/edk2-platforms-AddBinariesHideUnresolved.svg?sanitize=true https://github.com/mdkinney/edk2/wiki/images/edk2-non-osi.svg?sanitize=true https://github.com/mdkinney/edk2/wiki/images/FSP.svg?sanitize=true https://github.com/mdkinney/edk2/wiki/images/all.svg?sanitize=true This patch is available for review at the following branch: https://github.com/mdkinney/edk2/tree/Bug_2161_PackageDependencyGraph A proposed update to the EDK II Packages wiki page that adds details on package dependency rules that this tool can help to visualize and resolve issues. https://github.com/mdkinney/edk2/wiki/EDKII-Packages A new wiki page with graphs from the current EDK II repos and instructions on how to use this new script to generate the graphs. Complex graphs can be saved locally and viewed with an SVG viewer. https://github.com/mdkinney/edk2/wiki/EDK-II-Package-Dependency-Graphs Cc: Ray Ni <ray.ni@intel.com> Cc: Bob Feng <bob.c.feng@intel.com> Cc: Liming Gao <liming.gao@intel.com> Cc: Sean Brogan <sean.brogan@microsoft.com> Cc: Bret Barkelew <Bret.Barkelew@microsoft.com> Signed-off-by: Michael D Kinney <michael.d.kinney@intel.com> Michael D Kinney (1): BaseTools/Scripts: Add package dependency graphing tool BaseTools/Scripts/PackageDependencyGraph.py | 296 ++++++++++++++++++++ 1 file changed, 296 insertions(+) create mode 100644 BaseTools/Scripts/PackageDependencyGraph.py -- 2.21.0.windows.1 ^ permalink raw reply [flat|nested] 12+ messages in thread
* [Patch 1/1] BaseTools/Scripts: Add package dependency graphing tool 2019-12-13 19:44 [Patch 0/1] BaseTools/Scripts: Add package dependency graphing tool Michael D Kinney @ 2019-12-13 19:44 ` Michael D Kinney 2019-12-16 5:56 ` Ni, Ray [not found] ` <15E0C4623803B81C.31060@groups.io> 0 siblings, 2 replies; 12+ messages in thread From: Michael D Kinney @ 2019-12-13 19:44 UTC (permalink / raw) To: devel; +Cc: Ray Ni, Bob Feng, Liming Gao, Sean Brogan, Bret Barkelew https://bugzilla.tianocore.org/show_bug.cgi?id=2161 Add python script that recursively scans a directory for EDK II packages and generates GraphViz dot input that is used to render a graph of package dependencies in SVG format. Detects following error/warning conditions: * Ambiguous dependencies (multiple matches) * Unresolved dependencies * Circular dependencies * Nested packages usage: PackageDependencyGraph [-h] [-w WORKSPACE] [-p PACKAGESPATH] [-d DOTOUTPUTFILE] [-o SVGOUTPUTFILE] [-g IGNOREDIRECTORY] [-k SKIPPACKAGE] [-s] [-u] [-l] [-f] [-b] [-v] [-q] [--debug [0-9]] Recursively scan a directory for EDK II packages and generate GraphViz dot input that is used to render a graph of package dependencies in SVG format. Copyright (c) 2019, Intel Corporation. All rights reserved. optional arguments: -h, --help show this help message and exit -w WORKSPACE, --workspace WORKSPACE Directory to recursively scan for EDK II packages. Default is current directory. -p PACKAGESPATH, --packages-path PACKAGESPATH List of directories to recursively scan for EDK II packages. -d DOTOUTPUTFILE, --dot-output DOTOUTPUTFILE DOT output filename. -o OUTPUTFILE, --output OUTPUTFILE SVG output filename. -g IGNOREDIRECTORY, --ignore-directory IGNOREDIRECTORY Name of directory to ignore. Option can be repeated to ignore multiple directories. -k SKIPPACKAGE, --skip-package SKIPPACKAGE Name of EDK II Package DEC file to skip. Option can be repeated to skip multiple EDK II packages. -s, --self-dependency Include self links in dependency graph. Default is disabled. -u, --unresolved Include unresolved EDK II packages in dependency graph. Default is disabled. -l, --label Label links with the number of EDK II package dependencies. Default is disabled. -f, --full-paths Label package nodes with full path to EDK II package. Default is disabled. -b, --web-browser Display SVG output file in default web browser. Default is disabled. -v, --verbose Increase output messages -q, --quiet Reduce output messages --debug [0-9] Set debug level Cc: Ray Ni <ray.ni@intel.com> Cc: Bob Feng <bob.c.feng@intel.com> Cc: Liming Gao <liming.gao@intel.com> Cc: Sean Brogan <sean.brogan@microsoft.com> Cc: Bret Barkelew <Bret.Barkelew@microsoft.com> Signed-off-by: Michael D Kinney <michael.d.kinney@intel.com> --- BaseTools/Scripts/PackageDependencyGraph.py | 296 ++++++++++++++++++++ 1 file changed, 296 insertions(+) create mode 100644 BaseTools/Scripts/PackageDependencyGraph.py diff --git a/BaseTools/Scripts/PackageDependencyGraph.py b/BaseTools/Scripts/PackageDependencyGraph.py new file mode 100644 index 0000000000..b3c8e41774 --- /dev/null +++ b/BaseTools/Scripts/PackageDependencyGraph.py @@ -0,0 +1,296 @@ +# @file +# Recursively scan a directory for EDK II packages and generate GraphViz dot +# input that is used to render a graph of package dependencies in SVG format. +# +# Copyright (c) 2019, Intel Corporation. All rights reserved.<BR> +# SPDX-License-Identifier: BSD-2-Clause-Patent +# +## + +import os +import sys +import argparse +import subprocess +import webbrowser +import networkx as nx +from edk2toollib.uefi.edk2.parsers.inf_parser import InfParser + +# +# Globals for help information +# +__prog__ = 'PackageDependencyGraph' +__copyright__ = 'Copyright (c) 2019, Intel Corporation. All rights reserved.' +__description__ = '''Recursively scan a directory for EDK II packages and +generate GraphViz dot input that is used to render a graph of package +dependencies in SVG format.''' + +if __name__ == '__main__': + + # + # Create command line argument parser object + # + parser = argparse.ArgumentParser (prog = __prog__, + description = __description__ + '\n' + __copyright__, + conflict_handler = 'resolve') + parser.add_argument ("-w", "--workspace", dest = 'Workspace', default = os.curdir, + help = "Directory to recursively scan for EDK II packages. Default is current directory.") + parser.add_argument ("-p", "--packages-path", dest = 'PackagesPath', default = None, + help = "List of directories to recursively scan for EDK II packages.") + parser.add_argument ("-d", "--dot-output", dest = 'DotOutputFile', + help = "DOT output filename.") + parser.add_argument ("-o", "--output", dest = 'SvgOutputFile', + help = "SVG output filename.") + parser.add_argument ("-g", "--ignore-directory", dest = 'IgnoreDirectory', action='append', default=[], + help = "Name of directory to ignore. Option can be repeated to ignore multiple directories.") + parser.add_argument ("-k", "--skip-package", dest = 'SkipPackage', action='append', default=[], + help = "Name of EDK II Package DEC file to skip. Option can be repeated to skip multiple EDK II packages.") + parser.add_argument ("-s", "--self-dependency", dest = 'SelfDependency', action = "store_true", default = False, + help = "Include self links in dependency graph. Default is disabled.") + parser.add_argument ("-u", "--unresolved", dest = 'Unresolved', action = "store_true", default=False, + help = "Include unresolved EDK II packages in dependency graph. Default is disabled.") + parser.add_argument ("-l", "--label", dest = 'Label', action = "store_true", default=False, + help = "Label links with the number of EDK II package dependencies. Default is disabled.") + parser.add_argument ("-f", "--full-paths", dest = 'FullPaths', action = "store_true", default=False, + help = "Label package nodes with full path to EDK II package. Default is disabled.") + parser.add_argument ("-b", "--web-browser", dest = 'WebBrowser', action = "store_true", default=False, + help = "Display SVG output file in default web browser. Default is disabled.") + parser.add_argument ("-v", "--verbose", dest = 'Verbose', action = "store_true", + help = "Increase output messages") + parser.add_argument ("-q", "--quiet", dest = 'Quiet', action = "store_true", + help = "Reduce output messages") + parser.add_argument ("--debug", dest = 'Debug', type = int, metavar = '[0-9]', choices = range (0, 10), default = 0, + help = "Set debug level") + + # + # Parse command line arguments + # + args = parser.parse_args () + + # + # Find all EDK II package DEC files + # + Components = {} + SearchPaths = [args.Workspace] + if args.PackagesPath: + SearchPaths += args.PackagesPath.split(os.pathsep) + SearchPaths = [os.path.realpath(x) for x in SearchPaths] + for SearchPath in SearchPaths: + for root, dirs, files in os.walk (SearchPath): + for name in files: + FilePath = os.path.join (root, name) + if set(FilePath.split(os.sep)).intersection(set(args.IgnoreDirectory)) != set(): + if args.Verbose: + print ('IGNORE:' + FilePath) + continue + if os.path.splitext(FilePath)[1].lower() in ['.dec']: + DecFile = os.path.realpath (FilePath) + if os.path.split(DecFile)[1] in args.SkipPackage: + if args.Verbose: + print ('SKIP:' + DecFile) + continue + if DecFile not in Components: + if args.Verbose: + print ('PACKAGE:' + DecFile) + Components[DecFile] = {} + + # + # Find EDK II component INF files in each EDK II package + # + PackageLabels = {} + UnresolvedPackages = [] + AmbiguousDependencies = [] + for DecFile in Components: + DecPath = os.path.split (DecFile)[0] + if DecFile not in PackageLabels: + PackageLabels[DecFile] = (DecFile.replace(os.path.sep,'\\n'), 'white') + if not args.FullPaths: + PackagePath = os.path.relpath(DecFile, os.path.split(DecPath)[0]) + PackageLabels[DecFile] = (PackagePath.replace(os.path.sep,'\\n'), 'white') + for root, dirs, files in os.walk (DecPath): + for name in files: + FilePath = os.path.join(root, name) + if set(FilePath.split(os.sep)).intersection(set(args.IgnoreDirectory)) != set(): + if args.Verbose: + print ('IGNORE:' + FilePath) + continue + if os.path.splitext(FilePath)[1].lower() in ['.inf']: + InfFile = os.path.realpath (FilePath) + Inf = InfParser () + Inf.ParseFile (InfFile) + DependentPackages = [] + for Dependency in Inf.PackagesUsed: + Dependency = os.path.normpath(Dependency) + if os.path.split(Dependency)[1] in args.SkipPackage: + if args.Verbose: + print ('SKIP:' + Dependency) + continue + Found = False + for SearchPath in SearchPaths: + PackagePath = os.path.realpath(os.path.join(SearchPath, Dependency)) + if os.path.exists(PackagePath): + DependentPackages.append(PackagePath) + if not args.FullPaths: + PackageLabels[PackagePath] = (Dependency.replace(os.path.sep,'\\n'), 'white') + Found = True + break + if not Found: + Count = 0 + Match = '' + for DecFile2 in Components: + if DecFile2.endswith(Dependency): + if Count == 0: + Match = DecFile2 + Count = Count + 1 + if Count > 1: + AmbiguousDependencies.append (Dependency) + if Count == 1: + DependentPackages.append(Match) + if not args.FullPaths: + PackageLabels[Match] = (Dependency.replace(os.path.sep,'\\n'), 'white') + Found = True + if not Found and args.Unresolved: + if args.Verbose: + print ('WARNING: Dependent package not found ' + Dependency) + DependentPackages.append(Dependency) + UnresolvedPackages.append(Dependency) + PackageLabels[Dependency] = (Dependency.replace(os.path.sep,'\\n'), 'white') + Components[DecFile][InfFile] = DependentPackages + if AmbiguousDependencies: + for Dependency in set(AmbiguousDependencies): + print ('ERROR: MULTIPLE: ' + Dependency) + print ('Use --packages-path to provide search priority.') + sys.exit(1) + + # + # Generate GraphViz dot input file contents. + # Use networkx to detect circular dependencies. + # + MaxWeight = 0 + MaxWeightDecFile = list(Components.keys())[0] + Graph = nx.DiGraph() + Edges = [] + for DecFile in Components: + if args.Verbose: + print ('PACKAGE DEPENDENCIES:' + DecFile) + AllDependencies = [] + Dependencies = set() + for InfFile in Components[DecFile]: + AllDependencies += Components[DecFile][InfFile] + Dependencies = Dependencies.union(Components[DecFile][InfFile]) + if not args.SelfDependency: + Dependencies = Dependencies.difference(set([DecFile])) + for Dependency in Dependencies: + Weight = AllDependencies.count(Dependency) + if Weight > MaxWeight: + MaxWeight = Weight + MaxWeightDecFile = DecFile + if args.Verbose: + print (' DEPENDENCY: Weight(' + str(Weight) + ') ' + Dependency) + Edges.append(' "{Package}" -> "{Dependency}" [label = "{Weight}"];'.format( + Package = DecFile, + Dependency = Dependency, + Weight = str(Weight) if args.Label else '' + )) + Graph.add_edge(DecFile, Dependency) + Edges.sort() + + # + # Set fill color to yellow if a packages is part of a circular dependency + # + for Node in set([x for y in list(nx.simple_cycles(Graph)) for x in y]): + PackageLabels[Node] = (PackageLabels[Node][0], 'yellow') + print ('ERROR: CIRCULAR: ' + Node) + + # + # Set fill color to pink if a package that is nested inside another package. + # Set fill color to orange if a package is nested inside another package and + # is part of a circular dependency. + # + for DecFile in Components: + DecPath = os.path.split (DecFile)[0] + for DecFile2 in Components: + if DecFile2 == DecFile: + continue + if os.path.commonpath ([DecPath, DecFile2]) != DecPath: + continue + if len(DecFile2) < len(DecPath): + DecFile2 = DecFile + print ('ERROR: NESTED: ' + DecFile2) + if PackageLabels[DecFile2][1] == 'yellow': + PackageLabels[DecFile2] = (PackageLabels[DecFile2][0], 'orange') + else: + PackageLabels[DecFile2] = (PackageLabels[DecFile2][0], 'pink') + + # + # Set fill color to red for nodes that are unresolved + # + for Node in set(UnresolvedPackages): + PackageLabels[Node] = (PackageLabels[Node][0], 'red') + print ('ERROR: UNRESOLVED: ' + Node) + + # + # Add node statements to set node label and fill color + # + Nodes = [] + for Package in PackageLabels: + Nodes.append(' "{Package}" [label="{Label}",fillcolor={Color}];'.format( + Package = Package, + Label = PackageLabels[Package][0], + Color = PackageLabels[Package][1] + )) + Nodes.sort() + + # + # Generate dot file from Nodes and Edges and add a Legend at top of graph + # + Dot = [] + Dot.append('digraph {') + Dot.append(' rankdir=BT;') + Dot.append(' node [shape=Mrecord,style=filled];') + Dot.append('') + Dot = Dot + Nodes + Dot.append('') + Dot = Dot + Edges + Dot.append('') + Dot.append(' subgraph legend {') + Dot.append(' rank=sink;') + Dot.append(' Unresolved [label="Unresolved Dependency", fillcolor=red];') + Dot.append(' Circular [label="Circular Dependency", fillcolor=yellow];') + Dot.append(' Nested [label="Nested Package", fillcolor=pink];') + Dot.append(' NestedCircular [label="Nested Package with Circular Dependency",fillcolor=orange];') + Dot.append(' }') + if MaxWeightDecFile: + Dot.append(' Unresolved->"' + MaxWeightDecFile + '" [style=invis];') + Dot.append('}') + + if args.DotOutputFile: + # + # Write GraphViz dot file contents to DotOutputFile + # + with open(os.path.realpath(args.DotOutputFile), 'w') as File: + File.write('\n'.join(Dot)) + if args.SvgOutputFile: + # + # Use GraphViz 'dot' command to generate SVG output file + # + args.SvgOutputFile = os.path.realpath(args.SvgOutputFile) + try: + Process = subprocess.Popen('dot -Tsvg', + stdin=subprocess.PIPE, + stdout=open(args.SvgOutputFile, 'w'), + stderr=subprocess.PIPE, + shell=True + ) + Process.stdin.write ('\n'.join(Dot).encode()) + Process.communicate() + if Process.returncode != 0: + print ("ERROR: Can not run GraphViz 'dot' command. Check install and path.") + sys.exit(Process.returncode) + except: + print ("ERROR: Can not run GraphViz 'dot' command. Check install and path.") + sys.exit(1) + # + # Display SVG file in default web browser + # + if args.WebBrowser: + webbrowser.open(args.SvgOutputFile) -- 2.21.0.windows.1 ^ permalink raw reply related [flat|nested] 12+ messages in thread
* Re: [Patch 1/1] BaseTools/Scripts: Add package dependency graphing tool 2019-12-13 19:44 ` [Patch 1/1] " Michael D Kinney @ 2019-12-16 5:56 ` Ni, Ray [not found] ` <15E0C4623803B81C.31060@groups.io> 1 sibling, 0 replies; 12+ messages in thread From: Ni, Ray @ 2019-12-16 5:56 UTC (permalink / raw) To: Kinney, Michael D, devel@edk2.groups.io Cc: Feng, Bob C, Gao, Liming, Sean Brogan, Bret Barkelew Mike, 2 minor comments regarding the help string in below. > -----Original Message----- > From: Kinney, Michael D <michael.d.kinney@intel.com> > Sent: Saturday, December 14, 2019 3:45 AM > To: devel@edk2.groups.io > Cc: Ni, Ray <ray.ni@intel.com>; Feng, Bob C <bob.c.feng@intel.com>; Gao, > Liming <liming.gao@intel.com>; Sean Brogan <sean.brogan@microsoft.com>; > Bret Barkelew <Bret.Barkelew@microsoft.com> > Subject: [Patch 1/1] BaseTools/Scripts: Add package dependency graphing > tool > > https://bugzilla.tianocore.org/show_bug.cgi?id=2161 > > Add python script that recursively scans a directory for EDK II > packages and generates GraphViz dot input that is used to render > a graph of package dependencies in SVG format. > > Detects following error/warning conditions: > * Ambiguous dependencies (multiple matches) > * Unresolved dependencies > * Circular dependencies > * Nested packages > > usage: PackageDependencyGraph [-h] [-w WORKSPACE] [-p PACKAGESPATH] > [-d DOTOUTPUTFILE] [-o SVGOUTPUTFILE] > [-g IGNOREDIRECTORY] [-k SKIPPACKAGE] > [-s] [-u] [-l] [-f] [-b] [-v] [-q] > [--debug [0-9]] > > Recursively scan a directory for EDK II packages and generate GraphViz dot > input that is used to render a graph of package dependencies in SVG format. > Copyright (c) 2019, Intel Corporation. All rights reserved. > > optional arguments: > -h, --help show this help message and exit > -w WORKSPACE, --workspace WORKSPACE > Directory to recursively scan for EDK II packages. > Default is current directory. > -p PACKAGESPATH, --packages-path PACKAGESPATH > List of directories to recursively scan for EDK II > packages. > -d DOTOUTPUTFILE, --dot-output DOTOUTPUTFILE > DOT output filename. > -o OUTPUTFILE, --output OUTPUTFILE > SVG output filename. > -g IGNOREDIRECTORY, --ignore-directory IGNOREDIRECTORY > Name of directory to ignore. Option can be repeated > to ignore multiple directories. 1. Name of directory to ignore when scanning for EDKII Module INFs. > -k SKIPPACKAGE, --skip-package SKIPPACKAGE > Name of EDK II Package DEC file to skip. Option can > be repeated to skip multiple EDK II packages. 2. Name of EDK II Package DEC file (including .DEC suffix) to skip. Option can Be repeated to skip multiple EDK II packages. > -s, --self-dependency > Include self links in dependency graph. Default is > disabled. > -u, --unresolved Include unresolved EDK II packages in dependency > graph. Default is disabled. > -l, --label Label links with the number of EDK II package > dependencies. Default is disabled. > -f, --full-paths Label package nodes with full path to EDK II > package. Default is disabled. > -b, --web-browser Display SVG output file in default web browser. > Default is disabled. > -v, --verbose Increase output messages > -q, --quiet Reduce output messages > --debug [0-9] Set debug level > > Cc: Ray Ni <ray.ni@intel.com> > Cc: Bob Feng <bob.c.feng@intel.com> > Cc: Liming Gao <liming.gao@intel.com> > Cc: Sean Brogan <sean.brogan@microsoft.com> > Cc: Bret Barkelew <Bret.Barkelew@microsoft.com> > Signed-off-by: Michael D Kinney <michael.d.kinney@intel.com> > --- > BaseTools/Scripts/PackageDependencyGraph.py | 296 > ++++++++++++++++++++ > 1 file changed, 296 insertions(+) > create mode 100644 BaseTools/Scripts/PackageDependencyGraph.py > > diff --git a/BaseTools/Scripts/PackageDependencyGraph.py > b/BaseTools/Scripts/PackageDependencyGraph.py > new file mode 100644 > index 0000000000..b3c8e41774 > --- /dev/null > +++ b/BaseTools/Scripts/PackageDependencyGraph.py > @@ -0,0 +1,296 @@ > +# @file > +# Recursively scan a directory for EDK II packages and generate GraphViz > dot > +# input that is used to render a graph of package dependencies in SVG > format. > +# > +# Copyright (c) 2019, Intel Corporation. All rights reserved.<BR> > +# SPDX-License-Identifier: BSD-2-Clause-Patent > +# > +## > + > +import os > +import sys > +import argparse > +import subprocess > +import webbrowser > +import networkx as nx > +from edk2toollib.uefi.edk2.parsers.inf_parser import InfParser > + > +# > +# Globals for help information > +# > +__prog__ = 'PackageDependencyGraph' > +__copyright__ = 'Copyright (c) 2019, Intel Corporation. All rights reserved.' > +__description__ = '''Recursively scan a directory for EDK II packages and > +generate GraphViz dot input that is used to render a graph of package > +dependencies in SVG format.''' > + > +if __name__ == '__main__': > + > + # > + # Create command line argument parser object > + # > + parser = argparse.ArgumentParser (prog = __prog__, > + description = __description__ + '\n' + __copyright__, > + conflict_handler = 'resolve') > + parser.add_argument ("-w", "--workspace", dest = 'Workspace', default = > os.curdir, > + help = "Directory to recursively scan for EDK II packages. > Default is current directory.") > + parser.add_argument ("-p", "--packages-path", dest = 'PackagesPath', > default = None, > + help = "List of directories to recursively scan for EDK II > packages.") > + parser.add_argument ("-d", "--dot-output", dest = 'DotOutputFile', > + help = "DOT output filename.") > + parser.add_argument ("-o", "--output", dest = 'SvgOutputFile', > + help = "SVG output filename.") > + parser.add_argument ("-g", "--ignore-directory", dest = 'IgnoreDirectory', > action='append', default=[], > + help = "Name of directory to ignore. Option can be repeated to > ignore multiple directories.") > + parser.add_argument ("-k", "--skip-package", dest = 'SkipPackage', > action='append', default=[], > + help = "Name of EDK II Package DEC file to skip. Option can be > repeated to skip multiple EDK II packages.") > + parser.add_argument ("-s", "--self-dependency", dest = > 'SelfDependency', action = "store_true", default = False, > + help = "Include self links in dependency graph. Default is > disabled.") > + parser.add_argument ("-u", "--unresolved", dest = 'Unresolved', action = > "store_true", default=False, > + help = "Include unresolved EDK II packages in dependency > graph. Default is disabled.") > + parser.add_argument ("-l", "--label", dest = 'Label', action = "store_true", > default=False, > + help = "Label links with the number of EDK II package > dependencies. Default is disabled.") > + parser.add_argument ("-f", "--full-paths", dest = 'FullPaths', action = > "store_true", default=False, > + help = "Label package nodes with full path to EDK II package. > Default is disabled.") > + parser.add_argument ("-b", "--web-browser", dest = 'WebBrowser', > action = "store_true", default=False, > + help = "Display SVG output file in default web browser. Default > is disabled.") > + parser.add_argument ("-v", "--verbose", dest = 'Verbose', action = > "store_true", > + help = "Increase output messages") > + parser.add_argument ("-q", "--quiet", dest = 'Quiet', action = > "store_true", > + help = "Reduce output messages") > + parser.add_argument ("--debug", dest = 'Debug', type = int, metavar = > '[0-9]', choices = range (0, 10), default = 0, > + help = "Set debug level") > + > + # > + # Parse command line arguments > + # > + args = parser.parse_args () > + > + # > + # Find all EDK II package DEC files > + # > + Components = {} > + SearchPaths = [args.Workspace] > + if args.PackagesPath: > + SearchPaths += args.PackagesPath.split(os.pathsep) > + SearchPaths = [os.path.realpath(x) for x in SearchPaths] > + for SearchPath in SearchPaths: > + for root, dirs, files in os.walk (SearchPath): > + for name in files: > + FilePath = os.path.join (root, name) > + if > set(FilePath.split(os.sep)).intersection(set(args.IgnoreDirectory)) != set(): > + if args.Verbose: > + print ('IGNORE:' + FilePath) > + continue > + if os.path.splitext(FilePath)[1].lower() in ['.dec']: > + DecFile = os.path.realpath (FilePath) > + if os.path.split(DecFile)[1] in args.SkipPackage: > + if args.Verbose: > + print ('SKIP:' + DecFile) > + continue > + if DecFile not in Components: > + if args.Verbose: > + print ('PACKAGE:' + DecFile) > + Components[DecFile] = {} > + > + # > + # Find EDK II component INF files in each EDK II package > + # > + PackageLabels = {} > + UnresolvedPackages = [] > + AmbiguousDependencies = [] > + for DecFile in Components: > + DecPath = os.path.split (DecFile)[0] > + if DecFile not in PackageLabels: > + PackageLabels[DecFile] = (DecFile.replace(os.path.sep,'\\n'), 'white') > + if not args.FullPaths: > + PackagePath = os.path.relpath(DecFile, os.path.split(DecPath)[0]) > + PackageLabels[DecFile] = (PackagePath.replace(os.path.sep,'\\n'), > 'white') > + for root, dirs, files in os.walk (DecPath): > + for name in files: > + FilePath = os.path.join(root, name) > + if > set(FilePath.split(os.sep)).intersection(set(args.IgnoreDirectory)) != set(): > + if args.Verbose: > + print ('IGNORE:' + FilePath) > + continue > + if os.path.splitext(FilePath)[1].lower() in ['.inf']: > + InfFile = os.path.realpath (FilePath) > + Inf = InfParser () > + Inf.ParseFile (InfFile) > + DependentPackages = [] > + for Dependency in Inf.PackagesUsed: > + Dependency = os.path.normpath(Dependency) > + if os.path.split(Dependency)[1] in args.SkipPackage: > + if args.Verbose: > + print ('SKIP:' + Dependency) > + continue > + Found = False > + for SearchPath in SearchPaths: > + PackagePath = os.path.realpath(os.path.join(SearchPath, > Dependency)) > + if os.path.exists(PackagePath): > + DependentPackages.append(PackagePath) > + if not args.FullPaths: > + PackageLabels[PackagePath] = > (Dependency.replace(os.path.sep,'\\n'), 'white') > + Found = True > + break > + if not Found: > + Count = 0 > + Match = '' > + for DecFile2 in Components: > + if DecFile2.endswith(Dependency): > + if Count == 0: > + Match = DecFile2 > + Count = Count + 1 > + if Count > 1: > + AmbiguousDependencies.append (Dependency) > + if Count == 1: > + DependentPackages.append(Match) > + if not args.FullPaths: > + PackageLabels[Match] = > (Dependency.replace(os.path.sep,'\\n'), 'white') > + Found = True > + if not Found and args.Unresolved: > + if args.Verbose: > + print ('WARNING: Dependent package not found ' + > Dependency) > + DependentPackages.append(Dependency) > + UnresolvedPackages.append(Dependency) > + PackageLabels[Dependency] = > (Dependency.replace(os.path.sep,'\\n'), 'white') > + Components[DecFile][InfFile] = DependentPackages > + if AmbiguousDependencies: > + for Dependency in set(AmbiguousDependencies): > + print ('ERROR: MULTIPLE: ' + Dependency) > + print ('Use --packages-path to provide search priority.') > + sys.exit(1) > + > + # > + # Generate GraphViz dot input file contents. > + # Use networkx to detect circular dependencies. > + # > + MaxWeight = 0 > + MaxWeightDecFile = list(Components.keys())[0] > + Graph = nx.DiGraph() > + Edges = [] > + for DecFile in Components: > + if args.Verbose: > + print ('PACKAGE DEPENDENCIES:' + DecFile) > + AllDependencies = [] > + Dependencies = set() > + for InfFile in Components[DecFile]: > + AllDependencies += Components[DecFile][InfFile] > + Dependencies = Dependencies.union(Components[DecFile][InfFile]) > + if not args.SelfDependency: > + Dependencies = Dependencies.difference(set([DecFile])) > + for Dependency in Dependencies: > + Weight = AllDependencies.count(Dependency) > + if Weight > MaxWeight: > + MaxWeight = Weight > + MaxWeightDecFile = DecFile > + if args.Verbose: > + print (' DEPENDENCY: Weight(' + str(Weight) + ') ' + Dependency) > + Edges.append(' "{Package}" -> "{Dependency}" [label = > "{Weight}"];'.format( > + Package = DecFile, > + Dependency = Dependency, > + Weight = str(Weight) if args.Label else '' > + )) > + Graph.add_edge(DecFile, Dependency) > + Edges.sort() > + > + # > + # Set fill color to yellow if a packages is part of a circular dependency > + # > + for Node in set([x for y in list(nx.simple_cycles(Graph)) for x in y]): > + PackageLabels[Node] = (PackageLabels[Node][0], 'yellow') > + print ('ERROR: CIRCULAR: ' + Node) > + > + # > + # Set fill color to pink if a package that is nested inside another package. > + # Set fill color to orange if a package is nested inside another package and > + # is part of a circular dependency. > + # > + for DecFile in Components: > + DecPath = os.path.split (DecFile)[0] > + for DecFile2 in Components: > + if DecFile2 == DecFile: > + continue > + if os.path.commonpath ([DecPath, DecFile2]) != DecPath: > + continue > + if len(DecFile2) < len(DecPath): > + DecFile2 = DecFile > + print ('ERROR: NESTED: ' + DecFile2) > + if PackageLabels[DecFile2][1] == 'yellow': > + PackageLabels[DecFile2] = (PackageLabels[DecFile2][0], 'orange') > + else: > + PackageLabels[DecFile2] = (PackageLabels[DecFile2][0], 'pink') > + > + # > + # Set fill color to red for nodes that are unresolved > + # > + for Node in set(UnresolvedPackages): > + PackageLabels[Node] = (PackageLabels[Node][0], 'red') > + print ('ERROR: UNRESOLVED: ' + Node) > + > + # > + # Add node statements to set node label and fill color > + # > + Nodes = [] > + for Package in PackageLabels: > + Nodes.append(' "{Package}" [label="{Label}",fillcolor={Color}];'.format( > + Package = Package, > + Label = PackageLabels[Package][0], > + Color = PackageLabels[Package][1] > + )) > + Nodes.sort() > + > + # > + # Generate dot file from Nodes and Edges and add a Legend at top of > graph > + # > + Dot = [] > + Dot.append('digraph {') > + Dot.append(' rankdir=BT;') > + Dot.append(' node [shape=Mrecord,style=filled];') > + Dot.append('') > + Dot = Dot + Nodes > + Dot.append('') > + Dot = Dot + Edges > + Dot.append('') > + Dot.append(' subgraph legend {') > + Dot.append(' rank=sink;') > + Dot.append(' Unresolved [label="Unresolved Dependency", > fillcolor=red];') > + Dot.append(' Circular [label="Circular Dependency", > fillcolor=yellow];') > + Dot.append(' Nested [label="Nested Package", > fillcolor=pink];') > + Dot.append(' NestedCircular [label="Nested Package with Circular > Dependency",fillcolor=orange];') > + Dot.append(' }') > + if MaxWeightDecFile: > + Dot.append(' Unresolved->"' + MaxWeightDecFile + '" [style=invis];') > + Dot.append('}') > + > + if args.DotOutputFile: > + # > + # Write GraphViz dot file contents to DotOutputFile > + # > + with open(os.path.realpath(args.DotOutputFile), 'w') as File: > + File.write('\n'.join(Dot)) > + if args.SvgOutputFile: > + # > + # Use GraphViz 'dot' command to generate SVG output file > + # > + args.SvgOutputFile = os.path.realpath(args.SvgOutputFile) > + try: > + Process = subprocess.Popen('dot -Tsvg', > + stdin=subprocess.PIPE, > + stdout=open(args.SvgOutputFile, 'w'), > + stderr=subprocess.PIPE, > + shell=True > + ) > + Process.stdin.write ('\n'.join(Dot).encode()) > + Process.communicate() > + if Process.returncode != 0: > + print ("ERROR: Can not run GraphViz 'dot' command. Check install > and path.") > + sys.exit(Process.returncode) > + except: > + print ("ERROR: Can not run GraphViz 'dot' command. Check install and > path.") > + sys.exit(1) > + # > + # Display SVG file in default web browser > + # > + if args.WebBrowser: > + webbrowser.open(args.SvgOutputFile) > -- > 2.21.0.windows.1 ^ permalink raw reply [flat|nested] 12+ messages in thread
[parent not found: <15E0C4623803B81C.31060@groups.io>]
* Re: [edk2-devel] [Patch 1/1] BaseTools/Scripts: Add package dependency graphing tool [not found] ` <15E0C4623803B81C.31060@groups.io> @ 2019-12-16 8:40 ` Ni, Ray 2019-12-16 8:48 ` Steven Shi 2019-12-16 17:14 ` Michael D Kinney 0 siblings, 2 replies; 12+ messages in thread From: Ni, Ray @ 2019-12-16 8:40 UTC (permalink / raw) To: devel@edk2.groups.io, Ni, Ray, Kinney, Michael D Cc: Feng, Bob C, Gao, Liming, Sean Brogan, Bret Barkelew Mike, This pkg dep tool can tell through weight when a module depends on a new pkg. But it cannot tell when a module depends on another PCD/Protocol/Guid/Ppi/Library which belongs to an already-depended pkg. Failure to tell such information may make a bad module (that violates the dependency rules) worse (wrongly depend on more interfaces). Do you think that the tool can be enhanced in future to detect the case? Thanks, Ray > -----Original Message----- > From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Ni, Ray > Sent: Monday, December 16, 2019 1:57 PM > To: Kinney, Michael D <michael.d.kinney@intel.com>; devel@edk2.groups.io > Cc: Feng, Bob C <bob.c.feng@intel.com>; Gao, Liming > <liming.gao@intel.com>; Sean Brogan <sean.brogan@microsoft.com>; Bret > Barkelew <Bret.Barkelew@microsoft.com> > Subject: Re: [edk2-devel] [Patch 1/1] BaseTools/Scripts: Add package > dependency graphing tool > > Mike, > 2 minor comments regarding the help string in below. > > > -----Original Message----- > > From: Kinney, Michael D <michael.d.kinney@intel.com> > > Sent: Saturday, December 14, 2019 3:45 AM > > To: devel@edk2.groups.io > > Cc: Ni, Ray <ray.ni@intel.com>; Feng, Bob C <bob.c.feng@intel.com>; Gao, > > Liming <liming.gao@intel.com>; Sean Brogan > <sean.brogan@microsoft.com>; > > Bret Barkelew <Bret.Barkelew@microsoft.com> > > Subject: [Patch 1/1] BaseTools/Scripts: Add package dependency graphing > > tool > > > > https://bugzilla.tianocore.org/show_bug.cgi?id=2161 > > > > Add python script that recursively scans a directory for EDK II > > packages and generates GraphViz dot input that is used to render > > a graph of package dependencies in SVG format. > > > > Detects following error/warning conditions: > > * Ambiguous dependencies (multiple matches) > > * Unresolved dependencies > > * Circular dependencies > > * Nested packages > > > > usage: PackageDependencyGraph [-h] [-w WORKSPACE] [-p > PACKAGESPATH] > > [-d DOTOUTPUTFILE] [-o SVGOUTPUTFILE] > > [-g IGNOREDIRECTORY] [-k SKIPPACKAGE] > > [-s] [-u] [-l] [-f] [-b] [-v] [-q] > > [--debug [0-9]] > > > > Recursively scan a directory for EDK II packages and generate GraphViz dot > > input that is used to render a graph of package dependencies in SVG > format. > > Copyright (c) 2019, Intel Corporation. All rights reserved. > > > > optional arguments: > > -h, --help show this help message and exit > > -w WORKSPACE, --workspace WORKSPACE > > Directory to recursively scan for EDK II packages. > > Default is current directory. > > -p PACKAGESPATH, --packages-path PACKAGESPATH > > List of directories to recursively scan for EDK II > > packages. > > -d DOTOUTPUTFILE, --dot-output DOTOUTPUTFILE > > DOT output filename. > > -o OUTPUTFILE, --output OUTPUTFILE > > SVG output filename. > > -g IGNOREDIRECTORY, --ignore-directory IGNOREDIRECTORY > > Name of directory to ignore. Option can be repeated > > to ignore multiple directories. > > 1. Name of directory to ignore when scanning for EDKII Module INFs. > > > -k SKIPPACKAGE, --skip-package SKIPPACKAGE > > Name of EDK II Package DEC file to skip. Option can > > be repeated to skip multiple EDK II packages. > > 2. Name of EDK II Package DEC file (including .DEC suffix) to skip. Option can > Be repeated to skip multiple EDK II packages. > > > -s, --self-dependency > > Include self links in dependency graph. Default is > > disabled. > > -u, --unresolved Include unresolved EDK II packages in dependency > > graph. Default is disabled. > > -l, --label Label links with the number of EDK II package > > dependencies. Default is disabled. > > -f, --full-paths Label package nodes with full path to EDK II > > package. Default is disabled. > > -b, --web-browser Display SVG output file in default web browser. > > Default is disabled. > > -v, --verbose Increase output messages > > -q, --quiet Reduce output messages > > --debug [0-9] Set debug level > > > > Cc: Ray Ni <ray.ni@intel.com> > > Cc: Bob Feng <bob.c.feng@intel.com> > > Cc: Liming Gao <liming.gao@intel.com> > > Cc: Sean Brogan <sean.brogan@microsoft.com> > > Cc: Bret Barkelew <Bret.Barkelew@microsoft.com> > > Signed-off-by: Michael D Kinney <michael.d.kinney@intel.com> > > --- > > BaseTools/Scripts/PackageDependencyGraph.py | 296 > > ++++++++++++++++++++ > > 1 file changed, 296 insertions(+) > > create mode 100644 BaseTools/Scripts/PackageDependencyGraph.py > > > > diff --git a/BaseTools/Scripts/PackageDependencyGraph.py > > b/BaseTools/Scripts/PackageDependencyGraph.py > > new file mode 100644 > > index 0000000000..b3c8e41774 > > --- /dev/null > > +++ b/BaseTools/Scripts/PackageDependencyGraph.py > > @@ -0,0 +1,296 @@ > > +# @file > > +# Recursively scan a directory for EDK II packages and generate GraphViz > > dot > > +# input that is used to render a graph of package dependencies in SVG > > format. > > +# > > +# Copyright (c) 2019, Intel Corporation. All rights reserved.<BR> > > +# SPDX-License-Identifier: BSD-2-Clause-Patent > > +# > > +## > > + > > +import os > > +import sys > > +import argparse > > +import subprocess > > +import webbrowser > > +import networkx as nx > > +from edk2toollib.uefi.edk2.parsers.inf_parser import InfParser > > + > > +# > > +# Globals for help information > > +# > > +__prog__ = 'PackageDependencyGraph' > > +__copyright__ = 'Copyright (c) 2019, Intel Corporation. All rights > reserved.' > > +__description__ = '''Recursively scan a directory for EDK II packages and > > +generate GraphViz dot input that is used to render a graph of package > > +dependencies in SVG format.''' > > + > > +if __name__ == '__main__': > > + > > + # > > + # Create command line argument parser object > > + # > > + parser = argparse.ArgumentParser (prog = __prog__, > > + description = __description__ + '\n' + __copyright__, > > + conflict_handler = 'resolve') > > + parser.add_argument ("-w", "--workspace", dest = 'Workspace', default > = > > os.curdir, > > + help = "Directory to recursively scan for EDK II packages. > > Default is current directory.") > > + parser.add_argument ("-p", "--packages-path", dest = 'PackagesPath', > > default = None, > > + help = "List of directories to recursively scan for EDK II > > packages.") > > + parser.add_argument ("-d", "--dot-output", dest = 'DotOutputFile', > > + help = "DOT output filename.") > > + parser.add_argument ("-o", "--output", dest = 'SvgOutputFile', > > + help = "SVG output filename.") > > + parser.add_argument ("-g", "--ignore-directory", dest = > 'IgnoreDirectory', > > action='append', default=[], > > + help = "Name of directory to ignore. Option can be repeated > to > > ignore multiple directories.") > > + parser.add_argument ("-k", "--skip-package", dest = 'SkipPackage', > > action='append', default=[], > > + help = "Name of EDK II Package DEC file to skip. Option can > be > > repeated to skip multiple EDK II packages.") > > + parser.add_argument ("-s", "--self-dependency", dest = > > 'SelfDependency', action = "store_true", default = False, > > + help = "Include self links in dependency graph. Default is > > disabled.") > > + parser.add_argument ("-u", "--unresolved", dest = 'Unresolved', action > = > > "store_true", default=False, > > + help = "Include unresolved EDK II packages in dependency > > graph. Default is disabled.") > > + parser.add_argument ("-l", "--label", dest = 'Label', action = > "store_true", > > default=False, > > + help = "Label links with the number of EDK II package > > dependencies. Default is disabled.") > > + parser.add_argument ("-f", "--full-paths", dest = 'FullPaths', action = > > "store_true", default=False, > > + help = "Label package nodes with full path to EDK II package. > > Default is disabled.") > > + parser.add_argument ("-b", "--web-browser", dest = 'WebBrowser', > > action = "store_true", default=False, > > + help = "Display SVG output file in default web browser. > Default > > is disabled.") > > + parser.add_argument ("-v", "--verbose", dest = 'Verbose', action = > > "store_true", > > + help = "Increase output messages") > > + parser.add_argument ("-q", "--quiet", dest = 'Quiet', action = > > "store_true", > > + help = "Reduce output messages") > > + parser.add_argument ("--debug", dest = 'Debug', type = int, metavar = > > '[0-9]', choices = range (0, 10), default = 0, > > + help = "Set debug level") > > + > > + # > > + # Parse command line arguments > > + # > > + args = parser.parse_args () > > + > > + # > > + # Find all EDK II package DEC files > > + # > > + Components = {} > > + SearchPaths = [args.Workspace] > > + if args.PackagesPath: > > + SearchPaths += args.PackagesPath.split(os.pathsep) > > + SearchPaths = [os.path.realpath(x) for x in SearchPaths] > > + for SearchPath in SearchPaths: > > + for root, dirs, files in os.walk (SearchPath): > > + for name in files: > > + FilePath = os.path.join (root, name) > > + if > > set(FilePath.split(os.sep)).intersection(set(args.IgnoreDirectory)) != set(): > > + if args.Verbose: > > + print ('IGNORE:' + FilePath) > > + continue > > + if os.path.splitext(FilePath)[1].lower() in ['.dec']: > > + DecFile = os.path.realpath (FilePath) > > + if os.path.split(DecFile)[1] in args.SkipPackage: > > + if args.Verbose: > > + print ('SKIP:' + DecFile) > > + continue > > + if DecFile not in Components: > > + if args.Verbose: > > + print ('PACKAGE:' + DecFile) > > + Components[DecFile] = {} > > + > > + # > > + # Find EDK II component INF files in each EDK II package > > + # > > + PackageLabels = {} > > + UnresolvedPackages = [] > > + AmbiguousDependencies = [] > > + for DecFile in Components: > > + DecPath = os.path.split (DecFile)[0] > > + if DecFile not in PackageLabels: > > + PackageLabels[DecFile] = (DecFile.replace(os.path.sep,'\\n'), 'white') > > + if not args.FullPaths: > > + PackagePath = os.path.relpath(DecFile, os.path.split(DecPath)[0]) > > + PackageLabels[DecFile] = (PackagePath.replace(os.path.sep,'\\n'), > > 'white') > > + for root, dirs, files in os.walk (DecPath): > > + for name in files: > > + FilePath = os.path.join(root, name) > > + if > > set(FilePath.split(os.sep)).intersection(set(args.IgnoreDirectory)) != set(): > > + if args.Verbose: > > + print ('IGNORE:' + FilePath) > > + continue > > + if os.path.splitext(FilePath)[1].lower() in ['.inf']: > > + InfFile = os.path.realpath (FilePath) > > + Inf = InfParser () > > + Inf.ParseFile (InfFile) > > + DependentPackages = [] > > + for Dependency in Inf.PackagesUsed: > > + Dependency = os.path.normpath(Dependency) > > + if os.path.split(Dependency)[1] in args.SkipPackage: > > + if args.Verbose: > > + print ('SKIP:' + Dependency) > > + continue > > + Found = False > > + for SearchPath in SearchPaths: > > + PackagePath = os.path.realpath(os.path.join(SearchPath, > > Dependency)) > > + if os.path.exists(PackagePath): > > + DependentPackages.append(PackagePath) > > + if not args.FullPaths: > > + PackageLabels[PackagePath] = > > (Dependency.replace(os.path.sep,'\\n'), 'white') > > + Found = True > > + break > > + if not Found: > > + Count = 0 > > + Match = '' > > + for DecFile2 in Components: > > + if DecFile2.endswith(Dependency): > > + if Count == 0: > > + Match = DecFile2 > > + Count = Count + 1 > > + if Count > 1: > > + AmbiguousDependencies.append (Dependency) > > + if Count == 1: > > + DependentPackages.append(Match) > > + if not args.FullPaths: > > + PackageLabels[Match] = > > (Dependency.replace(os.path.sep,'\\n'), 'white') > > + Found = True > > + if not Found and args.Unresolved: > > + if args.Verbose: > > + print ('WARNING: Dependent package not found ' + > > Dependency) > > + DependentPackages.append(Dependency) > > + UnresolvedPackages.append(Dependency) > > + PackageLabels[Dependency] = > > (Dependency.replace(os.path.sep,'\\n'), 'white') > > + Components[DecFile][InfFile] = DependentPackages > > + if AmbiguousDependencies: > > + for Dependency in set(AmbiguousDependencies): > > + print ('ERROR: MULTIPLE: ' + Dependency) > > + print ('Use --packages-path to provide search priority.') > > + sys.exit(1) > > + > > + # > > + # Generate GraphViz dot input file contents. > > + # Use networkx to detect circular dependencies. > > + # > > + MaxWeight = 0 > > + MaxWeightDecFile = list(Components.keys())[0] > > + Graph = nx.DiGraph() > > + Edges = [] > > + for DecFile in Components: > > + if args.Verbose: > > + print ('PACKAGE DEPENDENCIES:' + DecFile) > > + AllDependencies = [] > > + Dependencies = set() > > + for InfFile in Components[DecFile]: > > + AllDependencies += Components[DecFile][InfFile] > > + Dependencies = Dependencies.union(Components[DecFile][InfFile]) > > + if not args.SelfDependency: > > + Dependencies = Dependencies.difference(set([DecFile])) > > + for Dependency in Dependencies: > > + Weight = AllDependencies.count(Dependency) > > + if Weight > MaxWeight: > > + MaxWeight = Weight > > + MaxWeightDecFile = DecFile > > + if args.Verbose: > > + print (' DEPENDENCY: Weight(' + str(Weight) + ') ' + Dependency) > > + Edges.append(' "{Package}" -> "{Dependency}" [label = > > "{Weight}"];'.format( > > + Package = DecFile, > > + Dependency = Dependency, > > + Weight = str(Weight) if args.Label else '' > > + )) > > + Graph.add_edge(DecFile, Dependency) > > + Edges.sort() > > + > > + # > > + # Set fill color to yellow if a packages is part of a circular dependency > > + # > > + for Node in set([x for y in list(nx.simple_cycles(Graph)) for x in y]): > > + PackageLabels[Node] = (PackageLabels[Node][0], 'yellow') > > + print ('ERROR: CIRCULAR: ' + Node) > > + > > + # > > + # Set fill color to pink if a package that is nested inside another package. > > + # Set fill color to orange if a package is nested inside another package > and > > + # is part of a circular dependency. > > + # > > + for DecFile in Components: > > + DecPath = os.path.split (DecFile)[0] > > + for DecFile2 in Components: > > + if DecFile2 == DecFile: > > + continue > > + if os.path.commonpath ([DecPath, DecFile2]) != DecPath: > > + continue > > + if len(DecFile2) < len(DecPath): > > + DecFile2 = DecFile > > + print ('ERROR: NESTED: ' + DecFile2) > > + if PackageLabels[DecFile2][1] == 'yellow': > > + PackageLabels[DecFile2] = (PackageLabels[DecFile2][0], 'orange') > > + else: > > + PackageLabels[DecFile2] = (PackageLabels[DecFile2][0], 'pink') > > + > > + # > > + # Set fill color to red for nodes that are unresolved > > + # > > + for Node in set(UnresolvedPackages): > > + PackageLabels[Node] = (PackageLabels[Node][0], 'red') > > + print ('ERROR: UNRESOLVED: ' + Node) > > + > > + # > > + # Add node statements to set node label and fill color > > + # > > + Nodes = [] > > + for Package in PackageLabels: > > + Nodes.append(' "{Package}" > [label="{Label}",fillcolor={Color}];'.format( > > + Package = Package, > > + Label = PackageLabels[Package][0], > > + Color = PackageLabels[Package][1] > > + )) > > + Nodes.sort() > > + > > + # > > + # Generate dot file from Nodes and Edges and add a Legend at top of > > graph > > + # > > + Dot = [] > > + Dot.append('digraph {') > > + Dot.append(' rankdir=BT;') > > + Dot.append(' node [shape=Mrecord,style=filled];') > > + Dot.append('') > > + Dot = Dot + Nodes > > + Dot.append('') > > + Dot = Dot + Edges > > + Dot.append('') > > + Dot.append(' subgraph legend {') > > + Dot.append(' rank=sink;') > > + Dot.append(' Unresolved [label="Unresolved Dependency", > > fillcolor=red];') > > + Dot.append(' Circular [label="Circular Dependency", > > fillcolor=yellow];') > > + Dot.append(' Nested [label="Nested Package", > > fillcolor=pink];') > > + Dot.append(' NestedCircular [label="Nested Package with Circular > > Dependency",fillcolor=orange];') > > + Dot.append(' }') > > + if MaxWeightDecFile: > > + Dot.append(' Unresolved->"' + MaxWeightDecFile + '" [style=invis];') > > + Dot.append('}') > > + > > + if args.DotOutputFile: > > + # > > + # Write GraphViz dot file contents to DotOutputFile > > + # > > + with open(os.path.realpath(args.DotOutputFile), 'w') as File: > > + File.write('\n'.join(Dot)) > > + if args.SvgOutputFile: > > + # > > + # Use GraphViz 'dot' command to generate SVG output file > > + # > > + args.SvgOutputFile = os.path.realpath(args.SvgOutputFile) > > + try: > > + Process = subprocess.Popen('dot -Tsvg', > > + stdin=subprocess.PIPE, > > + stdout=open(args.SvgOutputFile, 'w'), > > + stderr=subprocess.PIPE, > > + shell=True > > + ) > > + Process.stdin.write ('\n'.join(Dot).encode()) > > + Process.communicate() > > + if Process.returncode != 0: > > + print ("ERROR: Can not run GraphViz 'dot' command. Check install > > and path.") > > + sys.exit(Process.returncode) > > + except: > > + print ("ERROR: Can not run GraphViz 'dot' command. Check install > and > > path.") > > + sys.exit(1) > > + # > > + # Display SVG file in default web browser > > + # > > + if args.WebBrowser: > > + webbrowser.open(args.SvgOutputFile) > > -- > > 2.21.0.windows.1 > > > ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [edk2-devel] [Patch 1/1] BaseTools/Scripts: Add package dependency graphing tool 2019-12-16 8:40 ` [edk2-devel] " Ni, Ray @ 2019-12-16 8:48 ` Steven Shi 2019-12-16 17:16 ` Michael D Kinney 2019-12-16 17:14 ` Michael D Kinney 1 sibling, 1 reply; 12+ messages in thread From: Steven Shi @ 2019-12-16 8:48 UTC (permalink / raw) To: devel@edk2.groups.io, Ni, Ray, Kinney, Michael D Cc: Feng, Bob C, Gao, Liming, Sean Brogan, Bret Barkelew [-- Attachment #1: Type: text/plain, Size: 24778 bytes --] Ray, I think you are asking for a module-level dependency tool. 😊 Mike, This tool is cool. Besides the package level, do you think we can go further to figure out a module level dependency tool? With module level dependency info, we can define more interesting and comprehensive Modularity Metrics to measure the edk2 architecture quality (Architectural Technical Debt). Thanks Steven Shi Intel\SSG\SFE\FIE Firmware Infrastructure > -----Original Message----- > From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Ni, Ray > Sent: Monday, December 16, 2019 4:41 PM > To: devel@edk2.groups.io; Ni, Ray <ray.ni@intel.com>; Kinney, Michael D > <michael.d.kinney@intel.com> > Cc: Feng, Bob C <bob.c.feng@intel.com>; Gao, Liming > <liming.gao@intel.com>; Sean Brogan <sean.brogan@microsoft.com>; Bret > Barkelew <Bret.Barkelew@microsoft.com> > Subject: Re: [edk2-devel] [Patch 1/1] BaseTools/Scripts: Add package > dependency graphing tool > > Mike, > This pkg dep tool can tell through weight when a module depends on a new > pkg. > But it cannot tell when a module depends on another > PCD/Protocol/Guid/Ppi/Library which belongs to an already-depended pkg. > > Failure to tell such information may make a bad module (that violates the > dependency rules) worse (wrongly depend on more interfaces). > > Do you think that the tool can be enhanced in future to detect the case? > > Thanks, > Ray > > > -----Original Message----- > > From: devel@edk2.groups.io<mailto:devel@edk2.groups.io> <devel@edk2.groups.io<mailto:devel@edk2.groups.io>> On Behalf Of Ni, > Ray > > Sent: Monday, December 16, 2019 1:57 PM > > To: Kinney, Michael D <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>>; > devel@edk2.groups.io<mailto:devel@edk2.groups.io> > > Cc: Feng, Bob C <bob.c.feng@intel.com<mailto:bob.c.feng@intel.com>>; Gao, Liming > > <liming.gao@intel.com<mailto:liming.gao@intel.com>>; Sean Brogan <sean.brogan@microsoft.com<mailto:sean.brogan@microsoft.com>>; > Bret > > Barkelew <Bret.Barkelew@microsoft.com<mailto:Bret.Barkelew@microsoft.com>> > > Subject: Re: [edk2-devel] [Patch 1/1] BaseTools/Scripts: Add package > > dependency graphing tool > > > > Mike, > > 2 minor comments regarding the help string in below. > > > > > -----Original Message----- > > > From: Kinney, Michael D <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>> > > > Sent: Saturday, December 14, 2019 3:45 AM > > > To: devel@edk2.groups.io<mailto:devel@edk2.groups.io> > > > Cc: Ni, Ray <ray.ni@intel.com<mailto:ray.ni@intel.com>>; Feng, Bob C <bob.c.feng@intel.com<mailto:bob.c.feng@intel.com>>; > Gao, > > > Liming <liming.gao@intel.com<mailto:liming.gao@intel.com>>; Sean Brogan > > <sean.brogan@microsoft.com<mailto:sean.brogan@microsoft.com>>; > > > Bret Barkelew <Bret.Barkelew@microsoft.com<mailto:Bret.Barkelew@microsoft.com>> > > > Subject: [Patch 1/1] BaseTools/Scripts: Add package dependency > graphing > > > tool > > > > > > https://bugzilla.tianocore.org/show_bug.cgi?id=2161 > > > > > > Add python script that recursively scans a directory for EDK II > > > packages and generates GraphViz dot input that is used to render > > > a graph of package dependencies in SVG format. > > > > > > Detects following error/warning conditions: > > > * Ambiguous dependencies (multiple matches) > > > * Unresolved dependencies > > > * Circular dependencies > > > * Nested packages > > > > > > usage: PackageDependencyGraph [-h] [-w WORKSPACE] [-p > > PACKAGESPATH] > > > [-d DOTOUTPUTFILE] [-o SVGOUTPUTFILE] > > > [-g IGNOREDIRECTORY] [-k SKIPPACKAGE] > > > [-s] [-u] [-l] [-f] [-b] [-v] [-q] > > > [--debug [0-9]] > > > > > > Recursively scan a directory for EDK II packages and generate GraphViz > dot > > > input that is used to render a graph of package dependencies in SVG > > format. > > > Copyright (c) 2019, Intel Corporation. All rights reserved. > > > > > > optional arguments: > > > -h, --help show this help message and exit > > > -w WORKSPACE, --workspace WORKSPACE > > > Directory to recursively scan for EDK II packages. > > > Default is current directory. > > > -p PACKAGESPATH, --packages-path PACKAGESPATH > > > List of directories to recursively scan for EDK II > > > packages. > > > -d DOTOUTPUTFILE, --dot-output DOTOUTPUTFILE > > > DOT output filename. > > > -o OUTPUTFILE, --output OUTPUTFILE > > > SVG output filename. > > > -g IGNOREDIRECTORY, --ignore-directory IGNOREDIRECTORY > > > Name of directory to ignore. Option can be repeated > > > to ignore multiple directories. > > > > 1. Name of directory to ignore when scanning for EDKII Module INFs. > > > > > -k SKIPPACKAGE, --skip-package SKIPPACKAGE > > > Name of EDK II Package DEC file to skip. Option can > > > be repeated to skip multiple EDK II packages. > > > > 2. Name of EDK II Package DEC file (including .DEC suffix) to skip. Option > can > > Be repeated to skip multiple EDK II packages. > > > > > -s, --self-dependency > > > Include self links in dependency graph. Default is > > > disabled. > > > -u, --unresolved Include unresolved EDK II packages in dependency > > > graph. Default is disabled. > > > -l, --label Label links with the number of EDK II package > > > dependencies. Default is disabled. > > > -f, --full-paths Label package nodes with full path to EDK II > > > package. Default is disabled. > > > -b, --web-browser Display SVG output file in default web browser. > > > Default is disabled. > > > -v, --verbose Increase output messages > > > -q, --quiet Reduce output messages > > > --debug [0-9] Set debug level > > > > > > Cc: Ray Ni <ray.ni@intel.com<mailto:ray.ni@intel.com>> > > > Cc: Bob Feng <bob.c.feng@intel.com<mailto:bob.c.feng@intel.com>> > > > Cc: Liming Gao <liming.gao@intel.com<mailto:liming.gao@intel.com>> > > > Cc: Sean Brogan <sean.brogan@microsoft.com<mailto:sean.brogan@microsoft.com>> > > > Cc: Bret Barkelew <Bret.Barkelew@microsoft.com<mailto:Bret.Barkelew@microsoft.com>> > > > Signed-off-by: Michael D Kinney <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>> > > > --- > > > BaseTools/Scripts/PackageDependencyGraph.py | 296 > > > ++++++++++++++++++++ > > > 1 file changed, 296 insertions(+) > > > create mode 100644 BaseTools/Scripts/PackageDependencyGraph.py > > > > > > diff --git a/BaseTools/Scripts/PackageDependencyGraph.py > > > b/BaseTools/Scripts/PackageDependencyGraph.py > > > new file mode 100644 > > > index 0000000000..b3c8e41774 > > > --- /dev/null > > > +++ b/BaseTools/Scripts/PackageDependencyGraph.py > > > @@ -0,0 +1,296 @@ > > > +# @file > > > +# Recursively scan a directory for EDK II packages and generate > GraphViz > > > dot > > > +# input that is used to render a graph of package dependencies in SVG > > > format. > > > +# > > > +# Copyright (c) 2019, Intel Corporation. All rights reserved.<BR> > > > +# SPDX-License-Identifier: BSD-2-Clause-Patent > > > +# > > > +## > > > + > > > +import os > > > +import sys > > > +import argparse > > > +import subprocess > > > +import webbrowser > > > +import networkx as nx > > > +from edk2toollib.uefi.edk2.parsers.inf_parser import InfParser > > > + > > > +# > > > +# Globals for help information > > > +# > > > +__prog__ = 'PackageDependencyGraph' > > > +__copyright__ = 'Copyright (c) 2019, Intel Corporation. All rights > > reserved.' > > > +__description__ = '''Recursively scan a directory for EDK II packages and > > > +generate GraphViz dot input that is used to render a graph of package > > > +dependencies in SVG format.''' > > > + > > > +if __name__ == '__main__': > > > + > > > + # > > > + # Create command line argument parser object > > > + # > > > + parser = argparse.ArgumentParser (prog = __prog__, > > > + description = __description__ + '\n' + > __copyright__, > > > + conflict_handler = 'resolve') > > > + parser.add_argument ("-w", "--workspace", dest = 'Workspace', > default > > = > > > os.curdir, > > > + help = "Directory to recursively scan for EDK II packages. > > > Default is current directory.") > > > + parser.add_argument ("-p", "--packages-path", dest = 'PackagesPath', > > > default = None, > > > + help = "List of directories to recursively scan for EDK II > > > packages.") > > > + parser.add_argument ("-d", "--dot-output", dest = 'DotOutputFile', > > > + help = "DOT output filename.") > > > + parser.add_argument ("-o", "--output", dest = 'SvgOutputFile', > > > + help = "SVG output filename.") > > > + parser.add_argument ("-g", "--ignore-directory", dest = > > 'IgnoreDirectory', > > > action='append', default=[], > > > + help = "Name of directory to ignore. Option can be > repeated > > to > > > ignore multiple directories.") > > > + parser.add_argument ("-k", "--skip-package", dest = 'SkipPackage', > > > action='append', default=[], > > > + help = "Name of EDK II Package DEC file to skip. Option > can > > be > > > repeated to skip multiple EDK II packages.") > > > + parser.add_argument ("-s", "--self-dependency", dest = > > > 'SelfDependency', action = "store_true", default = False, > > > + help = "Include self links in dependency graph. Default is > > > disabled.") > > > + parser.add_argument ("-u", "--unresolved", dest = 'Unresolved', > action > > = > > > "store_true", default=False, > > > + help = "Include unresolved EDK II packages in dependency > > > graph. Default is disabled.") > > > + parser.add_argument ("-l", "--label", dest = 'Label', action = > > "store_true", > > > default=False, > > > + help = "Label links with the number of EDK II package > > > dependencies. Default is disabled.") > > > + parser.add_argument ("-f", "--full-paths", dest = 'FullPaths', action = > > > "store_true", default=False, > > > + help = "Label package nodes with full path to EDK II > package. > > > Default is disabled.") > > > + parser.add_argument ("-b", "--web-browser", dest = 'WebBrowser', > > > action = "store_true", default=False, > > > + help = "Display SVG output file in default web browser. > > Default > > > is disabled.") > > > + parser.add_argument ("-v", "--verbose", dest = 'Verbose', action = > > > "store_true", > > > + help = "Increase output messages") > > > + parser.add_argument ("-q", "--quiet", dest = 'Quiet', action = > > > "store_true", > > > + help = "Reduce output messages") > > > + parser.add_argument ("--debug", dest = 'Debug', type = int, metavar = > > > '[0-9]', choices = range (0, 10), default = 0, > > > + help = "Set debug level") > > > + > > > + # > > > + # Parse command line arguments > > > + # > > > + args = parser.parse_args () > > > + > > > + # > > > + # Find all EDK II package DEC files > > > + # > > > + Components = {} > > > + SearchPaths = [args.Workspace] > > > + if args.PackagesPath: > > > + SearchPaths += args.PackagesPath.split(os.pathsep) > > > + SearchPaths = [os.path.realpath(x) for x in SearchPaths] > > > + for SearchPath in SearchPaths: > > > + for root, dirs, files in os.walk (SearchPath): > > > + for name in files: > > > + FilePath = os.path.join (root, name) > > > + if > > > set(FilePath.split(os.sep)).intersection(set(args.IgnoreDirectory)) != set(): > > > + if args.Verbose: > > > + print ('IGNORE:' + FilePath) > > > + continue > > > + if os.path.splitext(FilePath)[1].lower() in ['.dec']: > > > + DecFile = os.path.realpath (FilePath) > > > + if os.path.split(DecFile)[1] in args.SkipPackage: > > > + if args.Verbose: > > > + print ('SKIP:' + DecFile) > > > + continue > > > + if DecFile not in Components: > > > + if args.Verbose: > > > + print ('PACKAGE:' + DecFile) > > > + Components[DecFile] = {} > > > + > > > + # > > > + # Find EDK II component INF files in each EDK II package > > > + # > > > + PackageLabels = {} > > > + UnresolvedPackages = [] > > > + AmbiguousDependencies = [] > > > + for DecFile in Components: > > > + DecPath = os.path.split (DecFile)[0] > > > + if DecFile not in PackageLabels: > > > + PackageLabels[DecFile] = (DecFile.replace(os.path.sep,'\\n'), > 'white') > > > + if not args.FullPaths: > > > + PackagePath = os.path.relpath(DecFile, > os.path.split(DecPath)[0]) > > > + PackageLabels[DecFile] = > (PackagePath.replace(os.path.sep,'\\n'), > > > 'white') > > > + for root, dirs, files in os.walk (DecPath): > > > + for name in files: > > > + FilePath = os.path.join(root, name) > > > + if > > > set(FilePath.split(os.sep)).intersection(set(args.IgnoreDirectory)) != set(): > > > + if args.Verbose: > > > + print ('IGNORE:' + FilePath) > > > + continue > > > + if os.path.splitext(FilePath)[1].lower() in ['.inf']: > > > + InfFile = os.path.realpath (FilePath) > > > + Inf = InfParser () > > > + Inf.ParseFile (InfFile) > > > + DependentPackages = [] > > > + for Dependency in Inf.PackagesUsed: > > > + Dependency = os.path.normpath(Dependency) > > > + if os.path.split(Dependency)[1] in args.SkipPackage: > > > + if args.Verbose: > > > + print ('SKIP:' + Dependency) > > > + continue > > > + Found = False > > > + for SearchPath in SearchPaths: > > > + PackagePath = os.path.realpath(os.path.join(SearchPath, > > > Dependency)) > > > + if os.path.exists(PackagePath): > > > + DependentPackages.append(PackagePath) > > > + if not args.FullPaths: > > > + PackageLabels[PackagePath] = > > > (Dependency.replace(os.path.sep,'\\n'), 'white') > > > + Found = True > > > + break > > > + if not Found: > > > + Count = 0 > > > + Match = '' > > > + for DecFile2 in Components: > > > + if DecFile2.endswith(Dependency): > > > + if Count == 0: > > > + Match = DecFile2 > > > + Count = Count + 1 > > > + if Count > 1: > > > + AmbiguousDependencies.append (Dependency) > > > + if Count == 1: > > > + DependentPackages.append(Match) > > > + if not args.FullPaths: > > > + PackageLabels[Match] = > > > (Dependency.replace(os.path.sep,'\\n'), 'white') > > > + Found = True > > > + if not Found and args.Unresolved: > > > + if args.Verbose: > > > + print ('WARNING: Dependent package not found ' + > > > Dependency) > > > + DependentPackages.append(Dependency) > > > + UnresolvedPackages.append(Dependency) > > > + PackageLabels[Dependency] = > > > (Dependency.replace(os.path.sep,'\\n'), 'white') > > > + Components[DecFile][InfFile] = DependentPackages > > > + if AmbiguousDependencies: > > > + for Dependency in set(AmbiguousDependencies): > > > + print ('ERROR: MULTIPLE: ' + Dependency) > > > + print ('Use --packages-path to provide search priority.') > > > + sys.exit(1) > > > + > > > + # > > > + # Generate GraphViz dot input file contents. > > > + # Use networkx to detect circular dependencies. > > > + # > > > + MaxWeight = 0 > > > + MaxWeightDecFile = list(Components.keys())[0] > > > + Graph = nx.DiGraph() > > > + Edges = [] > > > + for DecFile in Components: > > > + if args.Verbose: > > > + print ('PACKAGE DEPENDENCIES:' + DecFile) > > > + AllDependencies = [] > > > + Dependencies = set() > > > + for InfFile in Components[DecFile]: > > > + AllDependencies += Components[DecFile][InfFile] > > > + Dependencies = > Dependencies.union(Components[DecFile][InfFile]) > > > + if not args.SelfDependency: > > > + Dependencies = Dependencies.difference(set([DecFile])) > > > + for Dependency in Dependencies: > > > + Weight = AllDependencies.count(Dependency) > > > + if Weight > MaxWeight: > > > + MaxWeight = Weight > > > + MaxWeightDecFile = DecFile > > > + if args.Verbose: > > > + print (' DEPENDENCY: Weight(' + str(Weight) + ') ' + > Dependency) > > > + Edges.append(' "{Package}" -> "{Dependency}" [label = > > > "{Weight}"];'.format( > > > + Package = DecFile, > > > + Dependency = Dependency, > > > + Weight = str(Weight) if args.Label else '' > > > + )) > > > + Graph.add_edge(DecFile, Dependency) > > > + Edges.sort() > > > + > > > + # > > > + # Set fill color to yellow if a packages is part of a circular dependency > > > + # > > > + for Node in set([x for y in list(nx.simple_cycles(Graph)) for x in y]): > > > + PackageLabels[Node] = (PackageLabels[Node][0], 'yellow') > > > + print ('ERROR: CIRCULAR: ' + Node) > > > + > > > + # > > > + # Set fill color to pink if a package that is nested inside another > package. > > > + # Set fill color to orange if a package is nested inside another package > > and > > > + # is part of a circular dependency. > > > + # > > > + for DecFile in Components: > > > + DecPath = os.path.split (DecFile)[0] > > > + for DecFile2 in Components: > > > + if DecFile2 == DecFile: > > > + continue > > > + if os.path.commonpath ([DecPath, DecFile2]) != DecPath: > > > + continue > > > + if len(DecFile2) < len(DecPath): > > > + DecFile2 = DecFile > > > + print ('ERROR: NESTED: ' + DecFile2) > > > + if PackageLabels[DecFile2][1] == 'yellow': > > > + PackageLabels[DecFile2] = (PackageLabels[DecFile2][0], > 'orange') > > > + else: > > > + PackageLabels[DecFile2] = (PackageLabels[DecFile2][0], 'pink') > > > + > > > + # > > > + # Set fill color to red for nodes that are unresolved > > > + # > > > + for Node in set(UnresolvedPackages): > > > + PackageLabels[Node] = (PackageLabels[Node][0], 'red') > > > + print ('ERROR: UNRESOLVED: ' + Node) > > > + > > > + # > > > + # Add node statements to set node label and fill color > > > + # > > > + Nodes = [] > > > + for Package in PackageLabels: > > > + Nodes.append(' "{Package}" > > [label="{Label}",fillcolor={Color}];'.format( > > > + Package = Package, > > > + Label = PackageLabels[Package][0], > > > + Color = PackageLabels[Package][1] > > > + )) > > > + Nodes.sort() > > > + > > > + # > > > + # Generate dot file from Nodes and Edges and add a Legend at top of > > > graph > > > + # > > > + Dot = [] > > > + Dot.append('digraph {') > > > + Dot.append(' rankdir=BT;') > > > + Dot.append(' node [shape=Mrecord,style=filled];') > > > + Dot.append('') > > > + Dot = Dot + Nodes > > > + Dot.append('') > > > + Dot = Dot + Edges > > > + Dot.append('') > > > + Dot.append(' subgraph legend {') > > > + Dot.append(' rank=sink;') > > > + Dot.append(' Unresolved [label="Unresolved Dependency", > > > fillcolor=red];') > > > + Dot.append(' Circular [label="Circular Dependency", > > > fillcolor=yellow];') > > > + Dot.append(' Nested [label="Nested Package", > > > fillcolor=pink];') > > > + Dot.append(' NestedCircular [label="Nested Package with Circular > > > Dependency",fillcolor=orange];') > > > + Dot.append(' }') > > > + if MaxWeightDecFile: > > > + Dot.append(' Unresolved->"' + MaxWeightDecFile + '" [style=invis];') > > > + Dot.append('}') > > > + > > > + if args.DotOutputFile: > > > + # > > > + # Write GraphViz dot file contents to DotOutputFile > > > + # > > > + with open(os.path.realpath(args.DotOutputFile), 'w') as File: > > > + File.write('\n'.join(Dot)) > > > + if args.SvgOutputFile: > > > + # > > > + # Use GraphViz 'dot' command to generate SVG output file > > > + # > > > + args.SvgOutputFile = os.path.realpath(args.SvgOutputFile) > > > + try: > > > + Process = subprocess.Popen('dot -Tsvg', > > > + stdin=subprocess.PIPE, > > > + stdout=open(args.SvgOutputFile, 'w'), > > > + stderr=subprocess.PIPE, > > > + shell=True > > > + ) > > > + Process.stdin.write ('\n'.join(Dot).encode()) > > > + Process.communicate() > > > + if Process.returncode != 0: > > > + print ("ERROR: Can not run GraphViz 'dot' command. Check > install > > > and path.") > > > + sys.exit(Process.returncode) > > > + except: > > > + print ("ERROR: Can not run GraphViz 'dot' command. Check > install > > and > > > path.") > > > + sys.exit(1) > > > + # > > > + # Display SVG file in default web browser > > > + # > > > + if args.WebBrowser: > > > + webbrowser.open(args.SvgOutputFile) > > > -- > > > 2.21.0.windows.1 > > > > > > > > > [-- Attachment #2: Type: text/html, Size: 67697 bytes --] ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [edk2-devel] [Patch 1/1] BaseTools/Scripts: Add package dependency graphing tool 2019-12-16 8:48 ` Steven Shi @ 2019-12-16 17:16 ` Michael D Kinney 2019-12-16 19:45 ` Sean 2019-12-19 1:11 ` Steven Shi 0 siblings, 2 replies; 12+ messages in thread From: Michael D Kinney @ 2019-12-16 17:16 UTC (permalink / raw) To: Shi, Steven, devel@edk2.groups.io, Ni, Ray, Kinney, Michael D Cc: Feng, Bob C, Gao, Liming, Sean Brogan, Bret Barkelew [-- Attachment #1: Type: text/plain, Size: 25742 bytes --] Steve, Yes. I think a Module scoped view and a Platform scoped view would be good additions. Can you please enter new BZ feature requests for these and add a summary of the features you would like to see in each? Thanks, Mike From: Shi, Steven <steven.shi@intel.com> Sent: Monday, December 16, 2019 12:48 AM To: devel@edk2.groups.io; Ni, Ray <ray.ni@intel.com>; Kinney, Michael D <michael.d.kinney@intel.com> Cc: Feng, Bob C <bob.c.feng@intel.com>; Gao, Liming <liming.gao@intel.com>; Sean Brogan <sean.brogan@microsoft.com>; Bret Barkelew <Bret.Barkelew@microsoft.com> Subject: RE: [edk2-devel] [Patch 1/1] BaseTools/Scripts: Add package dependency graphing tool Ray, I think you are asking for a module-level dependency tool. 😊 Mike, This tool is cool. Besides the package level, do you think we can go further to figure out a module level dependency tool? With module level dependency info, we can define more interesting and comprehensive Modularity Metrics to measure the edk2 architecture quality (Architectural Technical Debt). Thanks Steven Shi Intel\SSG\SFE\FIE Firmware Infrastructure > -----Original Message----- > From: devel@edk2.groups.io<mailto:devel@edk2.groups.io> <devel@edk2.groups.io<mailto:devel@edk2.groups.io>> On Behalf Of Ni, Ray > Sent: Monday, December 16, 2019 4:41 PM > To: devel@edk2.groups.io<mailto:devel@edk2.groups.io>; Ni, Ray <ray.ni@intel.com<mailto:ray.ni@intel.com>>; Kinney, Michael D > <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>> > Cc: Feng, Bob C <bob.c.feng@intel.com<mailto:bob.c.feng@intel.com>>; Gao, Liming > <liming.gao@intel.com<mailto:liming.gao@intel.com>>; Sean Brogan <sean.brogan@microsoft.com<mailto:sean.brogan@microsoft.com>>; Bret > Barkelew <Bret.Barkelew@microsoft.com<mailto:Bret.Barkelew@microsoft.com>> > Subject: Re: [edk2-devel] [Patch 1/1] BaseTools/Scripts: Add package > dependency graphing tool > > Mike, > This pkg dep tool can tell through weight when a module depends on a new > pkg. > But it cannot tell when a module depends on another > PCD/Protocol/Guid/Ppi/Library which belongs to an already-depended pkg. > > Failure to tell such information may make a bad module (that violates the > dependency rules) worse (wrongly depend on more interfaces). > > Do you think that the tool can be enhanced in future to detect the case? > > Thanks, > Ray > > > -----Original Message----- > > From: devel@edk2.groups.io<mailto:devel@edk2.groups.io> <devel@edk2.groups.io<mailto:devel@edk2.groups.io>> On Behalf Of Ni, > Ray > > Sent: Monday, December 16, 2019 1:57 PM > > To: Kinney, Michael D <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>>; > devel@edk2.groups.io<mailto:devel@edk2.groups.io> > > Cc: Feng, Bob C <bob.c.feng@intel.com<mailto:bob.c.feng@intel.com>>; Gao, Liming > > <liming.gao@intel.com<mailto:liming.gao@intel.com>>; Sean Brogan <sean.brogan@microsoft.com<mailto:sean.brogan@microsoft.com>>; > Bret > > Barkelew <Bret.Barkelew@microsoft.com<mailto:Bret.Barkelew@microsoft.com>> > > Subject: Re: [edk2-devel] [Patch 1/1] BaseTools/Scripts: Add package > > dependency graphing tool > > > > Mike, > > 2 minor comments regarding the help string in below. > > > > > -----Original Message----- > > > From: Kinney, Michael D <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>> > > > Sent: Saturday, December 14, 2019 3:45 AM > > > To: devel@edk2.groups.io<mailto:devel@edk2.groups.io> > > > Cc: Ni, Ray <ray.ni@intel.com<mailto:ray.ni@intel.com>>; Feng, Bob C <bob.c.feng@intel.com<mailto:bob.c.feng@intel.com>>; > Gao, > > > Liming <liming.gao@intel.com<mailto:liming.gao@intel.com>>; Sean Brogan > > <sean.brogan@microsoft.com<mailto:sean.brogan@microsoft.com>>; > > > Bret Barkelew <Bret.Barkelew@microsoft.com<mailto:Bret.Barkelew@microsoft.com>> > > > Subject: [Patch 1/1] BaseTools/Scripts: Add package dependency > graphing > > > tool > > > > > > https://bugzilla.tianocore.org/show_bug.cgi?id=2161 > > > > > > Add python script that recursively scans a directory for EDK II > > > packages and generates GraphViz dot input that is used to render > > > a graph of package dependencies in SVG format. > > > > > > Detects following error/warning conditions: > > > * Ambiguous dependencies (multiple matches) > > > * Unresolved dependencies > > > * Circular dependencies > > > * Nested packages > > > > > > usage: PackageDependencyGraph [-h] [-w WORKSPACE] [-p > > PACKAGESPATH] > > > [-d DOTOUTPUTFILE] [-o SVGOUTPUTFILE] > > > [-g IGNOREDIRECTORY] [-k SKIPPACKAGE] > > > [-s] [-u] [-l] [-f] [-b] [-v] [-q] > > > [--debug [0-9]] > > > > > > Recursively scan a directory for EDK II packages and generate GraphViz > dot > > > input that is used to render a graph of package dependencies in SVG > > format. > > > Copyright (c) 2019, Intel Corporation. All rights reserved. > > > > > > optional arguments: > > > -h, --help show this help message and exit > > > -w WORKSPACE, --workspace WORKSPACE > > > Directory to recursively scan for EDK II packages. > > > Default is current directory. > > > -p PACKAGESPATH, --packages-path PACKAGESPATH > > > List of directories to recursively scan for EDK II > > > packages. > > > -d DOTOUTPUTFILE, --dot-output DOTOUTPUTFILE > > > DOT output filename. > > > -o OUTPUTFILE, --output OUTPUTFILE > > > SVG output filename. > > > -g IGNOREDIRECTORY, --ignore-directory IGNOREDIRECTORY > > > Name of directory to ignore. Option can be repeated > > > to ignore multiple directories. > > > > 1. Name of directory to ignore when scanning for EDKII Module INFs. > > > > > -k SKIPPACKAGE, --skip-package SKIPPACKAGE > > > Name of EDK II Package DEC file to skip. Option can > > > be repeated to skip multiple EDK II packages. > > > > 2. Name of EDK II Package DEC file (including .DEC suffix) to skip. Option > can > > Be repeated to skip multiple EDK II packages. > > > > > -s, --self-dependency > > > Include self links in dependency graph. Default is > > > disabled. > > > -u, --unresolved Include unresolved EDK II packages in dependency > > > graph. Default is disabled. > > > -l, --label Label links with the number of EDK II package > > > dependencies. Default is disabled. > > > -f, --full-paths Label package nodes with full path to EDK II > > > package. Default is disabled. > > > -b, --web-browser Display SVG output file in default web browser. > > > Default is disabled. > > > -v, --verbose Increase output messages > > > -q, --quiet Reduce output messages > > > --debug [0-9] Set debug level > > > > > > Cc: Ray Ni <ray.ni@intel.com<mailto:ray.ni@intel.com>> > > > Cc: Bob Feng <bob.c.feng@intel.com<mailto:bob.c.feng@intel.com>> > > > Cc: Liming Gao <liming.gao@intel.com<mailto:liming.gao@intel.com>> > > > Cc: Sean Brogan <sean.brogan@microsoft.com<mailto:sean.brogan@microsoft.com>> > > > Cc: Bret Barkelew <Bret.Barkelew@microsoft.com<mailto:Bret.Barkelew@microsoft.com>> > > > Signed-off-by: Michael D Kinney <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>> > > > --- > > > BaseTools/Scripts/PackageDependencyGraph.py | 296 > > > ++++++++++++++++++++ > > > 1 file changed, 296 insertions(+) > > > create mode 100644 BaseTools/Scripts/PackageDependencyGraph.py > > > > > > diff --git a/BaseTools/Scripts/PackageDependencyGraph.py > > > b/BaseTools/Scripts/PackageDependencyGraph.py > > > new file mode 100644 > > > index 0000000000..b3c8e41774 > > > --- /dev/null > > > +++ b/BaseTools/Scripts/PackageDependencyGraph.py > > > @@ -0,0 +1,296 @@ > > > +# @file > > > +# Recursively scan a directory for EDK II packages and generate > GraphViz > > > dot > > > +# input that is used to render a graph of package dependencies in SVG > > > format. > > > +# > > > +# Copyright (c) 2019, Intel Corporation. All rights reserved.<BR> > > > +# SPDX-License-Identifier: BSD-2-Clause-Patent > > > +# > > > +## > > > + > > > +import os > > > +import sys > > > +import argparse > > > +import subprocess > > > +import webbrowser > > > +import networkx as nx > > > +from edk2toollib.uefi.edk2.parsers.inf_parser import InfParser > > > + > > > +# > > > +# Globals for help information > > > +# > > > +__prog__ = 'PackageDependencyGraph' > > > +__copyright__ = 'Copyright (c) 2019, Intel Corporation. All rights > > reserved.' > > > +__description__ = '''Recursively scan a directory for EDK II packages and > > > +generate GraphViz dot input that is used to render a graph of package > > > +dependencies in SVG format.''' > > > + > > > +if __name__ == '__main__': > > > + > > > + # > > > + # Create command line argument parser object > > > + # > > > + parser = argparse.ArgumentParser (prog = __prog__, > > > + description = __description__ + '\n' + > __copyright__, > > > + conflict_handler = 'resolve') > > > + parser.add_argument ("-w", "--workspace", dest = 'Workspace', > default > > = > > > os.curdir, > > > + help = "Directory to recursively scan for EDK II packages. > > > Default is current directory.") > > > + parser.add_argument ("-p", "--packages-path", dest = 'PackagesPath', > > > default = None, > > > + help = "List of directories to recursively scan for EDK II > > > packages.") > > > + parser.add_argument ("-d", "--dot-output", dest = 'DotOutputFile', > > > + help = "DOT output filename.") > > > + parser.add_argument ("-o", "--output", dest = 'SvgOutputFile', > > > + help = "SVG output filename.") > > > + parser.add_argument ("-g", "--ignore-directory", dest = > > 'IgnoreDirectory', > > > action='append', default=[], > > > + help = "Name of directory to ignore. Option can be > repeated > > to > > > ignore multiple directories.") > > > + parser.add_argument ("-k", "--skip-package", dest = 'SkipPackage', > > > action='append', default=[], > > > + help = "Name of EDK II Package DEC file to skip. Option > can > > be > > > repeated to skip multiple EDK II packages.") > > > + parser.add_argument ("-s", "--self-dependency", dest = > > > 'SelfDependency', action = "store_true", default = False, > > > + help = "Include self links in dependency graph. Default is > > > disabled.") > > > + parser.add_argument ("-u", "--unresolved", dest = 'Unresolved', > action > > = > > > "store_true", default=False, > > > + help = "Include unresolved EDK II packages in dependency > > > graph. Default is disabled.") > > > + parser.add_argument ("-l", "--label", dest = 'Label', action = > > "store_true", > > > default=False, > > > + help = "Label links with the number of EDK II package > > > dependencies. Default is disabled.") > > > + parser.add_argument ("-f", "--full-paths", dest = 'FullPaths', action = > > > "store_true", default=False, > > > + help = "Label package nodes with full path to EDK II > package. > > > Default is disabled.") > > > + parser.add_argument ("-b", "--web-browser", dest = 'WebBrowser', > > > action = "store_true", default=False, > > > + help = "Display SVG output file in default web browser. > > Default > > > is disabled.") > > > + parser.add_argument ("-v", "--verbose", dest = 'Verbose', action = > > > "store_true", > > > + help = "Increase output messages") > > > + parser.add_argument ("-q", "--quiet", dest = 'Quiet', action = > > > "store_true", > > > + help = "Reduce output messages") > > > + parser.add_argument ("--debug", dest = 'Debug', type = int, metavar = > > > '[0-9]', choices = range (0, 10), default = 0, > > > + help = "Set debug level") > > > + > > > + # > > > + # Parse command line arguments > > > + # > > > + args = parser.parse_args () > > > + > > > + # > > > + # Find all EDK II package DEC files > > > + # > > > + Components = {} > > > + SearchPaths = [args.Workspace] > > > + if args.PackagesPath: > > > + SearchPaths += args.PackagesPath.split(os.pathsep) > > > + SearchPaths = [os.path.realpath(x) for x in SearchPaths] > > > + for SearchPath in SearchPaths: > > > + for root, dirs, files in os.walk (SearchPath): > > > + for name in files: > > > + FilePath = os.path.join (root, name) > > > + if > > > set(FilePath.split(os.sep)).intersection(set(args.IgnoreDirectory)) != set(): > > > + if args.Verbose: > > > + print ('IGNORE:' + FilePath) > > > + continue > > > + if os.path.splitext(FilePath)[1].lower() in ['.dec']: > > > + DecFile = os.path.realpath (FilePath) > > > + if os.path.split(DecFile)[1] in args.SkipPackage: > > > + if args.Verbose: > > > + print ('SKIP:' + DecFile) > > > + continue > > > + if DecFile not in Components: > > > + if args.Verbose: > > > + print ('PACKAGE:' + DecFile) > > > + Components[DecFile] = {} > > > + > > > + # > > > + # Find EDK II component INF files in each EDK II package > > > + # > > > + PackageLabels = {} > > > + UnresolvedPackages = [] > > > + AmbiguousDependencies = [] > > > + for DecFile in Components: > > > + DecPath = os.path.split (DecFile)[0] > > > + if DecFile not in PackageLabels: > > > + PackageLabels[DecFile] = (DecFile.replace(os.path.sep,'\\n'), > 'white') > > > + if not args.FullPaths: > > > + PackagePath = os.path.relpath(DecFile, > os.path.split(DecPath)[0]) > > > + PackageLabels[DecFile] = > (PackagePath.replace(os.path.sep,'\\n'), > > > 'white') > > > + for root, dirs, files in os.walk (DecPath): > > > + for name in files: > > > + FilePath = os.path.join(root, name) > > > + if > > > set(FilePath.split(os.sep)).intersection(set(args.IgnoreDirectory)) != set(): > > > + if args.Verbose: > > > + print ('IGNORE:' + FilePath) > > > + continue > > > + if os.path.splitext(FilePath)[1].lower() in ['.inf']: > > > + InfFile = os.path.realpath (FilePath) > > > + Inf = InfParser () > > > + Inf.ParseFile (InfFile) > > > + DependentPackages = [] > > > + for Dependency in Inf.PackagesUsed: > > > + Dependency = os.path.normpath(Dependency) > > > + if os.path.split(Dependency)[1] in args.SkipPackage: > > > + if args.Verbose: > > > + print ('SKIP:' + Dependency) > > > + continue > > > + Found = False > > > + for SearchPath in SearchPaths: > > > + PackagePath = os.path.realpath(os.path.join(SearchPath, > > > Dependency)) > > > + if os.path.exists(PackagePath): > > > + DependentPackages.append(PackagePath) > > > + if not args.FullPaths: > > > + PackageLabels[PackagePath] = > > > (Dependency.replace(os.path.sep,'\\n'), 'white') > > > + Found = True > > > + break > > > + if not Found: > > > + Count = 0 > > > + Match = '' > > > + for DecFile2 in Components: > > > + if DecFile2.endswith(Dependency): > > > + if Count == 0: > > > + Match = DecFile2 > > > + Count = Count + 1 > > > + if Count > 1: > > > + AmbiguousDependencies.append (Dependency) > > > + if Count == 1: > > > + DependentPackages.append(Match) > > > + if not args.FullPaths: > > > + PackageLabels[Match] = > > > (Dependency.replace(os.path.sep,'\\n'), 'white') > > > + Found = True > > > + if not Found and args.Unresolved: > > > + if args.Verbose: > > > + print ('WARNING: Dependent package not found ' + > > > Dependency) > > > + DependentPackages.append(Dependency) > > > + UnresolvedPackages.append(Dependency) > > > + PackageLabels[Dependency] = > > > (Dependency.replace(os.path.sep,'\\n'), 'white') > > > + Components[DecFile][InfFile] = DependentPackages > > > + if AmbiguousDependencies: > > > + for Dependency in set(AmbiguousDependencies): > > > + print ('ERROR: MULTIPLE: ' + Dependency) > > > + print ('Use --packages-path to provide search priority.') > > > + sys.exit(1) > > > + > > > + # > > > + # Generate GraphViz dot input file contents. > > > + # Use networkx to detect circular dependencies. > > > + # > > > + MaxWeight = 0 > > > + MaxWeightDecFile = list(Components.keys())[0] > > > + Graph = nx.DiGraph() > > > + Edges = [] > > > + for DecFile in Components: > > > + if args.Verbose: > > > + print ('PACKAGE DEPENDENCIES:' + DecFile) > > > + AllDependencies = [] > > > + Dependencies = set() > > > + for InfFile in Components[DecFile]: > > > + AllDependencies += Components[DecFile][InfFile] > > > + Dependencies = > Dependencies.union(Components[DecFile][InfFile]) > > > + if not args.SelfDependency: > > > + Dependencies = Dependencies.difference(set([DecFile])) > > > + for Dependency in Dependencies: > > > + Weight = AllDependencies.count(Dependency) > > > + if Weight > MaxWeight: > > > + MaxWeight = Weight > > > + MaxWeightDecFile = DecFile > > > + if args.Verbose: > > > + print (' DEPENDENCY: Weight(' + str(Weight) + ') ' + > Dependency) > > > + Edges.append(' "{Package}" -> "{Dependency}" [label = > > > "{Weight}"];'.format( > > > + Package = DecFile, > > > + Dependency = Dependency, > > > + Weight = str(Weight) if args.Label else '' > > > + )) > > > + Graph.add_edge(DecFile, Dependency) > > > + Edges.sort() > > > + > > > + # > > > + # Set fill color to yellow if a packages is part of a circular dependency > > > + # > > > + for Node in set([x for y in list(nx.simple_cycles(Graph)) for x in y]): > > > + PackageLabels[Node] = (PackageLabels[Node][0], 'yellow') > > > + print ('ERROR: CIRCULAR: ' + Node) > > > + > > > + # > > > + # Set fill color to pink if a package that is nested inside another > package. > > > + # Set fill color to orange if a package is nested inside another package > > and > > > + # is part of a circular dependency. > > > + # > > > + for DecFile in Components: > > > + DecPath = os.path.split (DecFile)[0] > > > + for DecFile2 in Components: > > > + if DecFile2 == DecFile: > > > + continue > > > + if os.path.commonpath ([DecPath, DecFile2]) != DecPath: > > > + continue > > > + if len(DecFile2) < len(DecPath): > > > + DecFile2 = DecFile > > > + print ('ERROR: NESTED: ' + DecFile2) > > > + if PackageLabels[DecFile2][1] == 'yellow': > > > + PackageLabels[DecFile2] = (PackageLabels[DecFile2][0], > 'orange') > > > + else: > > > + PackageLabels[DecFile2] = (PackageLabels[DecFile2][0], 'pink') > > > + > > > + # > > > + # Set fill color to red for nodes that are unresolved > > > + # > > > + for Node in set(UnresolvedPackages): > > > + PackageLabels[Node] = (PackageLabels[Node][0], 'red') > > > + print ('ERROR: UNRESOLVED: ' + Node) > > > + > > > + # > > > + # Add node statements to set node label and fill color > > > + # > > > + Nodes = [] > > > + for Package in PackageLabels: > > > + Nodes.append(' "{Package}" > > [label="{Label}",fillcolor={Color}];'.format( > > > + Package = Package, > > > + Label = PackageLabels[Package][0], > > > + Color = PackageLabels[Package][1] > > > + )) > > > + Nodes.sort() > > > + > > > + # > > > + # Generate dot file from Nodes and Edges and add a Legend at top of > > > graph > > > + # > > > + Dot = [] > > > + Dot.append('digraph {') > > > + Dot.append(' rankdir=BT;') > > > + Dot.append(' node [shape=Mrecord,style=filled];') > > > + Dot.append('') > > > + Dot = Dot + Nodes > > > + Dot.append('') > > > + Dot = Dot + Edges > > > + Dot.append('') > > > + Dot.append(' subgraph legend {') > > > + Dot.append(' rank=sink;') > > > + Dot.append(' Unresolved [label="Unresolved Dependency", > > > fillcolor=red];') > > > + Dot.append(' Circular [label="Circular Dependency", > > > fillcolor=yellow];') > > > + Dot.append(' Nested [label="Nested Package", > > > fillcolor=pink];') > > > + Dot.append(' NestedCircular [label="Nested Package with Circular > > > Dependency",fillcolor=orange];') > > > + Dot.append(' }') > > > + if MaxWeightDecFile: > > > + Dot.append(' Unresolved->"' + MaxWeightDecFile + '" [style=invis];') > > > + Dot.append('}') > > > + > > > + if args.DotOutputFile: > > > + # > > > + # Write GraphViz dot file contents to DotOutputFile > > > + # > > > + with open(os.path.realpath(args.DotOutputFile), 'w') as File: > > > + File.write('\n'.join(Dot)) > > > + if args.SvgOutputFile: > > > + # > > > + # Use GraphViz 'dot' command to generate SVG output file > > > + # > > > + args.SvgOutputFile = os.path.realpath(args.SvgOutputFile) > > > + try: > > > + Process = subprocess.Popen('dot -Tsvg', > > > + stdin=subprocess.PIPE, > > > + stdout=open(args.SvgOutputFile, 'w'), > > > + stderr=subprocess.PIPE, > > > + shell=True > > > + ) > > > + Process.stdin.write ('\n'.join(Dot).encode()) > > > + Process.communicate() > > > + if Process.returncode != 0: > > > + print ("ERROR: Can not run GraphViz 'dot' command. Check > install > > > and path.") > > > + sys.exit(Process.returncode) > > > + except: > > > + print ("ERROR: Can not run GraphViz 'dot' command. Check > install > > and > > > path.") > > > + sys.exit(1) > > > + # > > > + # Display SVG file in default web browser > > > + # > > > + if args.WebBrowser: > > > + webbrowser.open(args.SvgOutputFile) > > > -- > > > 2.21.0.windows.1 > > > > > > > > > [-- Attachment #2: Type: text/html, Size: 113125 bytes --] ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [edk2-devel] [Patch 1/1] BaseTools/Scripts: Add package dependency graphing tool 2019-12-16 17:16 ` Michael D Kinney @ 2019-12-16 19:45 ` Sean 2019-12-17 2:48 ` Ni, Ray 2019-12-19 1:11 ` Steven Shi 1 sibling, 1 reply; 12+ messages in thread From: Sean @ 2019-12-16 19:45 UTC (permalink / raw) To: Kinney, Michael D, Shi, Steven, devel@edk2.groups.io, Ni, Ray, Kinney, Michael D Cc: Feng, Bob C, Gao, Liming, Bret Barkelew [-- Attachment #1: Type: text/plain, Size: 27283 bytes --] Mike/Steven/Ray, We have had a tool for a couple of years with similar functionality. It is a python tool you run on your code base and it creates a single (local) webpage that you can then use to build charts. Please give it a try and see if you think it is useful. Here is the tool. https://github.com/spbrogan/edk2_dep And here is a html report run on edk2 https://github.com/spbrogan/edk2_dep/blob/sample/edk2/Edk2_core.html Thanks Sean From: Kinney, Michael D <michael.d.kinney@intel.com> Sent: Monday, December 16, 2019 9:16 AM To: Shi, Steven <steven.shi@intel.com>; devel@edk2.groups.io; Ni, Ray <ray.ni@intel.com>; Kinney, Michael D <michael.d.kinney@intel.com> Cc: Feng, Bob C <bob.c.feng@intel.com>; Gao, Liming <liming.gao@intel.com>; Sean Brogan <sean.brogan@microsoft.com>; Bret Barkelew <Bret.Barkelew@microsoft.com> Subject: [EXTERNAL] RE: [edk2-devel] [Patch 1/1] BaseTools/Scripts: Add package dependency graphing tool Steve, Yes. I think a Module scoped view and a Platform scoped view would be good additions. Can you please enter new BZ feature requests for these and add a summary of the features you would like to see in each? Thanks, Mike From: Shi, Steven <steven.shi@intel.com<mailto:steven.shi@intel.com>> Sent: Monday, December 16, 2019 12:48 AM To: devel@edk2.groups.io<mailto:devel@edk2.groups.io>; Ni, Ray <ray.ni@intel.com<mailto:ray.ni@intel.com>>; Kinney, Michael D <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>> Cc: Feng, Bob C <bob.c.feng@intel.com<mailto:bob.c.feng@intel.com>>; Gao, Liming <liming.gao@intel.com<mailto:liming.gao@intel.com>>; Sean Brogan <sean.brogan@microsoft.com<mailto:sean.brogan@microsoft.com>>; Bret Barkelew <Bret.Barkelew@microsoft.com<mailto:Bret.Barkelew@microsoft.com>> Subject: RE: [edk2-devel] [Patch 1/1] BaseTools/Scripts: Add package dependency graphing tool Ray, I think you are asking for a module-level dependency tool. 😊 Mike, This tool is cool. Besides the package level, do you think we can go further to figure out a module level dependency tool? With module level dependency info, we can define more interesting and comprehensive Modularity Metrics to measure the edk2 architecture quality (Architectural Technical Debt). Thanks Steven Shi Intel\SSG\SFE\FIE Firmware Infrastructure > -----Original Message----- > From: devel@edk2.groups.io<mailto:devel@edk2.groups.io> <devel@edk2.groups.io<mailto:devel@edk2.groups.io>> On Behalf Of Ni, Ray > Sent: Monday, December 16, 2019 4:41 PM > To: devel@edk2.groups.io<mailto:devel@edk2.groups.io>; Ni, Ray <ray.ni@intel.com<mailto:ray.ni@intel.com>>; Kinney, Michael D > <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>> > Cc: Feng, Bob C <bob.c.feng@intel.com<mailto:bob.c.feng@intel.com>>; Gao, Liming > <liming.gao@intel.com<mailto:liming.gao@intel.com>>; Sean Brogan <sean.brogan@microsoft.com<mailto:sean.brogan@microsoft.com>>; Bret > Barkelew <Bret.Barkelew@microsoft.com<mailto:Bret.Barkelew@microsoft.com>> > Subject: Re: [edk2-devel] [Patch 1/1] BaseTools/Scripts: Add package > dependency graphing tool > > Mike, > This pkg dep tool can tell through weight when a module depends on a new > pkg. > But it cannot tell when a module depends on another > PCD/Protocol/Guid/Ppi/Library which belongs to an already-depended pkg. > > Failure to tell such information may make a bad module (that violates the > dependency rules) worse (wrongly depend on more interfaces). > > Do you think that the tool can be enhanced in future to detect the case? > > Thanks, > Ray > > > -----Original Message----- > > From: devel@edk2.groups.io<mailto:devel@edk2.groups.io> <devel@edk2.groups.io<mailto:devel@edk2.groups.io>> On Behalf Of Ni, > Ray > > Sent: Monday, December 16, 2019 1:57 PM > > To: Kinney, Michael D <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>>; > devel@edk2.groups.io<mailto:devel@edk2.groups.io> > > Cc: Feng, Bob C <bob.c.feng@intel.com<mailto:bob.c.feng@intel.com>>; Gao, Liming > > <liming.gao@intel.com<mailto:liming.gao@intel.com>>; Sean Brogan <sean.brogan@microsoft.com<mailto:sean.brogan@microsoft.com>>; > Bret > > Barkelew <Bret.Barkelew@microsoft.com<mailto:Bret.Barkelew@microsoft.com>> > > Subject: Re: [edk2-devel] [Patch 1/1] BaseTools/Scripts: Add package > > dependency graphing tool > > > > Mike, > > 2 minor comments regarding the help string in below. > > > > > -----Original Message----- > > > From: Kinney, Michael D <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>> > > > Sent: Saturday, December 14, 2019 3:45 AM > > > To: devel@edk2.groups.io<mailto:devel@edk2.groups.io> > > > Cc: Ni, Ray <ray.ni@intel.com<mailto:ray.ni@intel.com>>; Feng, Bob C <bob.c.feng@intel.com<mailto:bob.c.feng@intel.com>>; > Gao, > > > Liming <liming.gao@intel.com<mailto:liming.gao@intel.com>>; Sean Brogan > > <sean.brogan@microsoft.com<mailto:sean.brogan@microsoft.com>>; > > > Bret Barkelew <Bret.Barkelew@microsoft.com<mailto:Bret.Barkelew@microsoft.com>> > > > Subject: [Patch 1/1] BaseTools/Scripts: Add package dependency > graphing > > > tool > > > > > > https://bugzilla.tianocore.org/show_bug.cgi?id=2161<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugzilla.tianocore.org%2Fshow_bug.cgi%3Fid%3D2161&data=02%7C01%7Csean.brogan%40microsoft.com%7Cf7fb1953185b4a17f27408d7824ba488%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637121133718397648&sdata=NE82lFZmNfIcNjUfo49D%2Bksn94atxzdx7F2j%2FWSME0E%3D&reserved=0> > > > > > > Add python script that recursively scans a directory for EDK II > > > packages and generates GraphViz dot input that is used to render > > > a graph of package dependencies in SVG format. > > > > > > Detects following error/warning conditions: > > > * Ambiguous dependencies (multiple matches) > > > * Unresolved dependencies > > > * Circular dependencies > > > * Nested packages > > > > > > usage: PackageDependencyGraph [-h] [-w WORKSPACE] [-p > > PACKAGESPATH] > > > [-d DOTOUTPUTFILE] [-o SVGOUTPUTFILE] > > > [-g IGNOREDIRECTORY] [-k SKIPPACKAGE] > > > [-s] [-u] [-l] [-f] [-b] [-v] [-q] > > > [--debug [0-9]] > > > > > > Recursively scan a directory for EDK II packages and generate GraphViz > dot > > > input that is used to render a graph of package dependencies in SVG > > format. > > > Copyright (c) 2019, Intel Corporation. All rights reserved. > > > > > > optional arguments: > > > -h, --help show this help message and exit > > > -w WORKSPACE, --workspace WORKSPACE > > > Directory to recursively scan for EDK II packages. > > > Default is current directory. > > > -p PACKAGESPATH, --packages-path PACKAGESPATH > > > List of directories to recursively scan for EDK II > > > packages. > > > -d DOTOUTPUTFILE, --dot-output DOTOUTPUTFILE > > > DOT output filename. > > > -o OUTPUTFILE, --output OUTPUTFILE > > > SVG output filename. > > > -g IGNOREDIRECTORY, --ignore-directory IGNOREDIRECTORY > > > Name of directory to ignore. Option can be repeated > > > to ignore multiple directories. > > > > 1. Name of directory to ignore when scanning for EDKII Module INFs. > > > > > -k SKIPPACKAGE, --skip-package SKIPPACKAGE > > > Name of EDK II Package DEC file to skip. Option can > > > be repeated to skip multiple EDK II packages. > > > > 2. Name of EDK II Package DEC file (including .DEC suffix) to skip. Option > can > > Be repeated to skip multiple EDK II packages. > > > > > -s, --self-dependency > > > Include self links in dependency graph. Default is > > > disabled. > > > -u, --unresolved Include unresolved EDK II packages in dependency > > > graph. Default is disabled. > > > -l, --label Label links with the number of EDK II package > > > dependencies. Default is disabled. > > > -f, --full-paths Label package nodes with full path to EDK II > > > package. Default is disabled. > > > -b, --web-browser Display SVG output file in default web browser. > > > Default is disabled. > > > -v, --verbose Increase output messages > > > -q, --quiet Reduce output messages > > > --debug [0-9] Set debug level > > > > > > Cc: Ray Ni <ray.ni@intel.com<mailto:ray.ni@intel.com>> > > > Cc: Bob Feng <bob.c.feng@intel.com<mailto:bob.c.feng@intel.com>> > > > Cc: Liming Gao <liming.gao@intel.com<mailto:liming.gao@intel.com>> > > > Cc: Sean Brogan <sean.brogan@microsoft.com<mailto:sean.brogan@microsoft.com>> > > > Cc: Bret Barkelew <Bret.Barkelew@microsoft.com<mailto:Bret.Barkelew@microsoft.com>> > > > Signed-off-by: Michael D Kinney <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>> > > > --- > > > BaseTools/Scripts/PackageDependencyGraph.py | 296 > > > ++++++++++++++++++++ > > > 1 file changed, 296 insertions(+) > > > create mode 100644 BaseTools/Scripts/PackageDependencyGraph.py > > > > > > diff --git a/BaseTools/Scripts/PackageDependencyGraph.py > > > b/BaseTools/Scripts/PackageDependencyGraph.py > > > new file mode 100644 > > > index 0000000000..b3c8e41774 > > > --- /dev/null > > > +++ b/BaseTools/Scripts/PackageDependencyGraph.py > > > @@ -0,0 +1,296 @@ > > > +# @file > > > +# Recursively scan a directory for EDK II packages and generate > GraphViz > > > dot > > > +# input that is used to render a graph of package dependencies in SVG > > > format. > > > +# > > > +# Copyright (c) 2019, Intel Corporation. All rights reserved.<BR> > > > +# SPDX-License-Identifier: BSD-2-Clause-Patent > > > +# > > > +## > > > + > > > +import os > > > +import sys > > > +import argparse > > > +import subprocess > > > +import webbrowser > > > +import networkx as nx > > > +from edk2toollib.uefi.edk2.parsers.inf_parser import InfParser > > > + > > > +# > > > +# Globals for help information > > > +# > > > +__prog__ = 'PackageDependencyGraph' > > > +__copyright__ = 'Copyright (c) 2019, Intel Corporation. All rights > > reserved.' > > > +__description__ = '''Recursively scan a directory for EDK II packages and > > > +generate GraphViz dot input that is used to render a graph of package > > > +dependencies in SVG format.''' > > > + > > > +if __name__ == '__main__': > > > + > > > + # > > > + # Create command line argument parser object > > > + # > > > + parser = argparse.ArgumentParser (prog = __prog__, > > > + description = __description__ + '\n' + > __copyright__, > > > + conflict_handler = 'resolve') > > > + parser.add_argument ("-w", "--workspace", dest = 'Workspace', > default > > = > > > os.curdir, > > > + help = "Directory to recursively scan for EDK II packages. > > > Default is current directory.") > > > + parser.add_argument ("-p", "--packages-path", dest = 'PackagesPath', > > > default = None, > > > + help = "List of directories to recursively scan for EDK II > > > packages.") > > > + parser.add_argument ("-d", "--dot-output", dest = 'DotOutputFile', > > > + help = "DOT output filename.") > > > + parser.add_argument ("-o", "--output", dest = 'SvgOutputFile', > > > + help = "SVG output filename.") > > > + parser.add_argument ("-g", "--ignore-directory", dest = > > 'IgnoreDirectory', > > > action='append', default=[], > > > + help = "Name of directory to ignore. Option can be > repeated > > to > > > ignore multiple directories.") > > > + parser.add_argument ("-k", "--skip-package", dest = 'SkipPackage', > > > action='append', default=[], > > > + help = "Name of EDK II Package DEC file to skip. Option > can > > be > > > repeated to skip multiple EDK II packages.") > > > + parser.add_argument ("-s", "--self-dependency", dest = > > > 'SelfDependency', action = "store_true", default = False, > > > + help = "Include self links in dependency graph. Default is > > > disabled.") > > > + parser.add_argument ("-u", "--unresolved", dest = 'Unresolved', > action > > = > > > "store_true", default=False, > > > + help = "Include unresolved EDK II packages in dependency > > > graph. Default is disabled.") > > > + parser.add_argument ("-l", "--label", dest = 'Label', action = > > "store_true", > > > default=False, > > > + help = "Label links with the number of EDK II package > > > dependencies. Default is disabled.") > > > + parser.add_argument ("-f", "--full-paths", dest = 'FullPaths', action = > > > "store_true", default=False, > > > + help = "Label package nodes with full path to EDK II > package. > > > Default is disabled.") > > > + parser.add_argument ("-b", "--web-browser", dest = 'WebBrowser', > > > action = "store_true", default=False, > > > + help = "Display SVG output file in default web browser. > > Default > > > is disabled.") > > > + parser.add_argument ("-v", "--verbose", dest = 'Verbose', action = > > > "store_true", > > > + help = "Increase output messages") > > > + parser.add_argument ("-q", "--quiet", dest = 'Quiet', action = > > > "store_true", > > > + help = "Reduce output messages") > > > + parser.add_argument ("--debug", dest = 'Debug', type = int, metavar = > > > '[0-9]', choices = range (0, 10), default = 0, > > > + help = "Set debug level") > > > + > > > + # > > > + # Parse command line arguments > > > + # > > > + args = parser.parse_args () > > > + > > > + # > > > + # Find all EDK II package DEC files > > > + # > > > + Components = {} > > > + SearchPaths = [args.Workspace] > > > + if args.PackagesPath: > > > + SearchPaths += args.PackagesPath.split(os.pathsep) > > > + SearchPaths = [os.path.realpath(x) for x in SearchPaths] > > > + for SearchPath in SearchPaths: > > > + for root, dirs, files in os.walk (SearchPath): > > > + for name in files: > > > + FilePath = os.path.join (root, name) > > > + if > > > set(FilePath.split(os.sep)).intersection(set(args.IgnoreDirectory)) != set(): > > > + if args.Verbose: > > > + print ('IGNORE:' + FilePath) > > > + continue > > > + if os.path.splitext(FilePath)[1].lower() in ['.dec']: > > > + DecFile = os.path.realpath (FilePath) > > > + if os.path.split(DecFile)[1] in args.SkipPackage: > > > + if args.Verbose: > > > + print ('SKIP:' + DecFile) > > > + continue > > > + if DecFile not in Components: > > > + if args.Verbose: > > > + print ('PACKAGE:' + DecFile) > > > + Components[DecFile] = {} > > > + > > > + # > > > + # Find EDK II component INF files in each EDK II package > > > + # > > > + PackageLabels = {} > > > + UnresolvedPackages = [] > > > + AmbiguousDependencies = [] > > > + for DecFile in Components: > > > + DecPath = os.path.split (DecFile)[0] > > > + if DecFile not in PackageLabels: > > > + PackageLabels[DecFile] = (DecFile.replace(os.path.sep,'\\n'), > 'white') > > > + if not args.FullPaths: > > > + PackagePath = os.path.relpath(DecFile, > os.path.split(DecPath)[0]) > > > + PackageLabels[DecFile] = > (PackagePath.replace(os.path.sep,'\\n'), > > > 'white') > > > + for root, dirs, files in os.walk (DecPath): > > > + for name in files: > > > + FilePath = os.path.join(root, name) > > > + if > > > set(FilePath.split(os.sep)).intersection(set(args.IgnoreDirectory)) != set(): > > > + if args.Verbose: > > > + print ('IGNORE:' + FilePath) > > > + continue > > > + if os.path.splitext(FilePath)[1].lower() in ['.inf']: > > > + InfFile = os.path.realpath (FilePath) > > > + Inf = InfParser () > > > + Inf.ParseFile (InfFile) > > > + DependentPackages = [] > > > + for Dependency in Inf.PackagesUsed: > > > + Dependency = os.path.normpath(Dependency) > > > + if os.path.split(Dependency)[1] in args.SkipPackage: > > > + if args.Verbose: > > > + print ('SKIP:' + Dependency) > > > + continue > > > + Found = False > > > + for SearchPath in SearchPaths: > > > + PackagePath = os.path.realpath(os.path.join(SearchPath, > > > Dependency)) > > > + if os.path.exists(PackagePath): > > > + DependentPackages.append(PackagePath) > > > + if not args.FullPaths: > > > + PackageLabels[PackagePath] = > > > (Dependency.replace(os.path.sep,'\\n'), 'white') > > > + Found = True > > > + break > > > + if not Found: > > > + Count = 0 > > > + Match = '' > > > + for DecFile2 in Components: > > > + if DecFile2.endswith(Dependency): > > > + if Count == 0: > > > + Match = DecFile2 > > > + Count = Count + 1 > > > + if Count > 1: > > > + AmbiguousDependencies.append (Dependency) > > > + if Count == 1: > > > + DependentPackages.append(Match) > > > + if not args.FullPaths: > > > + PackageLabels[Match] = > > > (Dependency.replace(os.path.sep,'\\n'), 'white') > > > + Found = True > > > + if not Found and args.Unresolved: > > > + if args.Verbose: > > > + print ('WARNING: Dependent package not found ' + > > > Dependency) > > > + DependentPackages.append(Dependency) > > > + UnresolvedPackages.append(Dependency) > > > + PackageLabels[Dependency] = > > > (Dependency.replace(os.path.sep,'\\n'), 'white') > > > + Components[DecFile][InfFile] = DependentPackages > > > + if AmbiguousDependencies: > > > + for Dependency in set(AmbiguousDependencies): > > > + print ('ERROR: MULTIPLE: ' + Dependency) > > > + print ('Use --packages-path to provide search priority.') > > > + sys.exit(1) > > > + > > > + # > > > + # Generate GraphViz dot input file contents. > > > + # Use networkx to detect circular dependencies. > > > + # > > > + MaxWeight = 0 > > > + MaxWeightDecFile = list(Components.keys())[0] > > > + Graph = nx.DiGraph() > > > + Edges = [] > > > + for DecFile in Components: > > > + if args.Verbose: > > > + print ('PACKAGE DEPENDENCIES:' + DecFile) > > > + AllDependencies = [] > > > + Dependencies = set() > > > + for InfFile in Components[DecFile]: > > > + AllDependencies += Components[DecFile][InfFile] > > > + Dependencies = > Dependencies.union(Components[DecFile][InfFile]) > > > + if not args.SelfDependency: > > > + Dependencies = Dependencies.difference(set([DecFile])) > > > + for Dependency in Dependencies: > > > + Weight = AllDependencies.count(Dependency) > > > + if Weight > MaxWeight: > > > + MaxWeight = Weight > > > + MaxWeightDecFile = DecFile > > > + if args.Verbose: > > > + print (' DEPENDENCY: Weight(' + str(Weight) + ') ' + > Dependency) > > > + Edges.append(' "{Package}" -> "{Dependency}" [label = > > > "{Weight}"];'.format( > > > + Package = DecFile, > > > + Dependency = Dependency, > > > + Weight = str(Weight) if args.Label else '' > > > + )) > > > + Graph.add_edge(DecFile, Dependency) > > > + Edges.sort() > > > + > > > + # > > > + # Set fill color to yellow if a packages is part of a circular dependency > > > + # > > > + for Node in set([x for y in list(nx.simple_cycles(Graph)) for x in y]): > > > + PackageLabels[Node] = (PackageLabels[Node][0], 'yellow') > > > + print ('ERROR: CIRCULAR: ' + Node) > > > + > > > + # > > > + # Set fill color to pink if a package that is nested inside another > package. > > > + # Set fill color to orange if a package is nested inside another package > > and > > > + # is part of a circular dependency. > > > + # > > > + for DecFile in Components: > > > + DecPath = os.path.split (DecFile)[0] > > > + for DecFile2 in Components: > > > + if DecFile2 == DecFile: > > > + continue > > > + if os.path.commonpath ([DecPath, DecFile2]) != DecPath: > > > + continue > > > + if len(DecFile2) < len(DecPath): > > > + DecFile2 = DecFile > > > + print ('ERROR: NESTED: ' + DecFile2) > > > + if PackageLabels[DecFile2][1] == 'yellow': > > > + PackageLabels[DecFile2] = (PackageLabels[DecFile2][0], > 'orange') > > > + else: > > > + PackageLabels[DecFile2] = (PackageLabels[DecFile2][0], 'pink') > > > + > > > + # > > > + # Set fill color to red for nodes that are unresolved > > > + # > > > + for Node in set(UnresolvedPackages): > > > + PackageLabels[Node] = (PackageLabels[Node][0], 'red') > > > + print ('ERROR: UNRESOLVED: ' + Node) > > > + > > > + # > > > + # Add node statements to set node label and fill color > > > + # > > > + Nodes = [] > > > + for Package in PackageLabels: > > > + Nodes.append(' "{Package}" > > [label="{Label}",fillcolor={Color}];'.format( > > > + Package = Package, > > > + Label = PackageLabels[Package][0], > > > + Color = PackageLabels[Package][1] > > > + )) > > > + Nodes.sort() > > > + > > > + # > > > + # Generate dot file from Nodes and Edges and add a Legend at top of > > > graph > > > + # > > > + Dot = [] > > > + Dot.append('digraph {') > > > + Dot.append(' rankdir=BT;') > > > + Dot.append(' node [shape=Mrecord,style=filled];') > > > + Dot.append('') > > > + Dot = Dot + Nodes > > > + Dot.append('') > > > + Dot = Dot + Edges > > > + Dot.append('') > > > + Dot.append(' subgraph legend {') > > > + Dot.append(' rank=sink;') > > > + Dot.append(' Unresolved [label="Unresolved Dependency", > > > fillcolor=red];') > > > + Dot.append(' Circular [label="Circular Dependency", > > > fillcolor=yellow];') > > > + Dot.append(' Nested [label="Nested Package", > > > fillcolor=pink];') > > > + Dot.append(' NestedCircular [label="Nested Package with Circular > > > Dependency",fillcolor=orange];') > > > + Dot.append(' }') > > > + if MaxWeightDecFile: > > > + Dot.append(' Unresolved->"' + MaxWeightDecFile + '" [style=invis];') > > > + Dot.append('}') > > > + > > > + if args.DotOutputFile: > > > + # > > > + # Write GraphViz dot file contents to DotOutputFile > > > + # > > > + with open(os.path.realpath(args.DotOutputFile), 'w') as File: > > > + File.write('\n'.join(Dot)) > > > + if args.SvgOutputFile: > > > + # > > > + # Use GraphViz 'dot' command to generate SVG output file > > > + # > > > + args.SvgOutputFile = os.path.realpath(args.SvgOutputFile) > > > + try: > > > + Process = subprocess.Popen('dot -Tsvg', > > > + stdin=subprocess.PIPE, > > > + stdout=open(args.SvgOutputFile, 'w'), > > > + stderr=subprocess.PIPE, > > > + shell=True > > > + ) > > > + Process.stdin.write ('\n'.join(Dot).encode()) > > > + Process.communicate() > > > + if Process.returncode != 0: > > > + print ("ERROR: Can not run GraphViz 'dot' command. Check > install > > > and path.") > > > + sys.exit(Process.returncode) > > > + except: > > > + print ("ERROR: Can not run GraphViz 'dot' command. Check > install > > and > > > path.") > > > + sys.exit(1) > > > + # > > > + # Display SVG file in default web browser > > > + # > > > + if args.WebBrowser: > > > + webbrowser.open(args.SvgOutputFile) > > > -- > > > 2.21.0.windows.1 > > > > > > > > > [-- Attachment #2: Type: text/html, Size: 78170 bytes --] ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [edk2-devel] [Patch 1/1] BaseTools/Scripts: Add package dependency graphing tool 2019-12-16 19:45 ` Sean @ 2019-12-17 2:48 ` Ni, Ray 2019-12-17 16:10 ` Steven Shi 0 siblings, 1 reply; 12+ messages in thread From: Ni, Ray @ 2019-12-17 2:48 UTC (permalink / raw) To: devel@edk2.groups.io, sean.brogan@microsoft.com, Kinney, Michael D, Shi, Steven Cc: Feng, Bob C, Gao, Liming, Bret Barkelew [-- Attachment #1: Type: text/plain, Size: 28348 bytes --] Sean, This is a very cool tool! We could learn from your tool source code to know how to use DecParser to get more detailed information, like which interfaces (PPI/Protocol/Guid/PCD/Lib) from which package a module is depending on. Thanks, Ray From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Sean via Groups.Io Sent: Tuesday, December 17, 2019 3:46 AM To: Kinney, Michael D <michael.d.kinney@intel.com>; Shi, Steven <steven.shi@intel.com>; devel@edk2.groups.io; Ni, Ray <ray.ni@intel.com>; Kinney, Michael D <michael.d.kinney@intel.com> Cc: Feng, Bob C <bob.c.feng@intel.com>; Gao, Liming <liming.gao@intel.com>; Bret Barkelew <Bret.Barkelew@microsoft.com> Subject: Re: [edk2-devel] [Patch 1/1] BaseTools/Scripts: Add package dependency graphing tool Mike/Steven/Ray, We have had a tool for a couple of years with similar functionality. It is a python tool you run on your code base and it creates a single (local) webpage that you can then use to build charts. Please give it a try and see if you think it is useful. Here is the tool. https://github.com/spbrogan/edk2_dep And here is a html report run on edk2 https://github.com/spbrogan/edk2_dep/blob/sample/edk2/Edk2_core.html Thanks Sean From: Kinney, Michael D <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>> Sent: Monday, December 16, 2019 9:16 AM To: Shi, Steven <steven.shi@intel.com<mailto:steven.shi@intel.com>>; devel@edk2.groups.io<mailto:devel@edk2.groups.io>; Ni, Ray <ray.ni@intel.com<mailto:ray.ni@intel.com>>; Kinney, Michael D <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>> Cc: Feng, Bob C <bob.c.feng@intel.com<mailto:bob.c.feng@intel.com>>; Gao, Liming <liming.gao@intel.com<mailto:liming.gao@intel.com>>; Sean Brogan <sean.brogan@microsoft.com<mailto:sean.brogan@microsoft.com>>; Bret Barkelew <Bret.Barkelew@microsoft.com<mailto:Bret.Barkelew@microsoft.com>> Subject: [EXTERNAL] RE: [edk2-devel] [Patch 1/1] BaseTools/Scripts: Add package dependency graphing tool Steve, Yes. I think a Module scoped view and a Platform scoped view would be good additions. Can you please enter new BZ feature requests for these and add a summary of the features you would like to see in each? Thanks, Mike From: Shi, Steven <steven.shi@intel.com<mailto:steven.shi@intel.com>> Sent: Monday, December 16, 2019 12:48 AM To: devel@edk2.groups.io<mailto:devel@edk2.groups.io>; Ni, Ray <ray.ni@intel.com<mailto:ray.ni@intel.com>>; Kinney, Michael D <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>> Cc: Feng, Bob C <bob.c.feng@intel.com<mailto:bob.c.feng@intel.com>>; Gao, Liming <liming.gao@intel.com<mailto:liming.gao@intel.com>>; Sean Brogan <sean.brogan@microsoft.com<mailto:sean.brogan@microsoft.com>>; Bret Barkelew <Bret.Barkelew@microsoft.com<mailto:Bret.Barkelew@microsoft.com>> Subject: RE: [edk2-devel] [Patch 1/1] BaseTools/Scripts: Add package dependency graphing tool Ray, I think you are asking for a module-level dependency tool. 😊 Mike, This tool is cool. Besides the package level, do you think we can go further to figure out a module level dependency tool? With module level dependency info, we can define more interesting and comprehensive Modularity Metrics to measure the edk2 architecture quality (Architectural Technical Debt). Thanks Steven Shi Intel\SSG\SFE\FIE Firmware Infrastructure > -----Original Message----- > From: devel@edk2.groups.io<mailto:devel@edk2.groups.io> <devel@edk2.groups.io<mailto:devel@edk2.groups.io>> On Behalf Of Ni, Ray > Sent: Monday, December 16, 2019 4:41 PM > To: devel@edk2.groups.io<mailto:devel@edk2.groups.io>; Ni, Ray <ray.ni@intel.com<mailto:ray.ni@intel.com>>; Kinney, Michael D > <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>> > Cc: Feng, Bob C <bob.c.feng@intel.com<mailto:bob.c.feng@intel.com>>; Gao, Liming > <liming.gao@intel.com<mailto:liming.gao@intel.com>>; Sean Brogan <sean.brogan@microsoft.com<mailto:sean.brogan@microsoft.com>>; Bret > Barkelew <Bret.Barkelew@microsoft.com<mailto:Bret.Barkelew@microsoft.com>> > Subject: Re: [edk2-devel] [Patch 1/1] BaseTools/Scripts: Add package > dependency graphing tool > > Mike, > This pkg dep tool can tell through weight when a module depends on a new > pkg. > But it cannot tell when a module depends on another > PCD/Protocol/Guid/Ppi/Library which belongs to an already-depended pkg. > > Failure to tell such information may make a bad module (that violates the > dependency rules) worse (wrongly depend on more interfaces). > > Do you think that the tool can be enhanced in future to detect the case? > > Thanks, > Ray > > > -----Original Message----- > > From: devel@edk2.groups.io<mailto:devel@edk2.groups.io> <devel@edk2.groups.io<mailto:devel@edk2.groups.io>> On Behalf Of Ni, > Ray > > Sent: Monday, December 16, 2019 1:57 PM > > To: Kinney, Michael D <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>>; > devel@edk2.groups.io<mailto:devel@edk2.groups.io> > > Cc: Feng, Bob C <bob.c.feng@intel.com<mailto:bob.c.feng@intel.com>>; Gao, Liming > > <liming.gao@intel.com<mailto:liming.gao@intel.com>>; Sean Brogan <sean.brogan@microsoft.com<mailto:sean.brogan@microsoft.com>>; > Bret > > Barkelew <Bret.Barkelew@microsoft.com<mailto:Bret.Barkelew@microsoft.com>> > > Subject: Re: [edk2-devel] [Patch 1/1] BaseTools/Scripts: Add package > > dependency graphing tool > > > > Mike, > > 2 minor comments regarding the help string in below. > > > > > -----Original Message----- > > > From: Kinney, Michael D <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>> > > > Sent: Saturday, December 14, 2019 3:45 AM > > > To: devel@edk2.groups.io<mailto:devel@edk2.groups.io> > > > Cc: Ni, Ray <ray.ni@intel.com<mailto:ray.ni@intel.com>>; Feng, Bob C <bob.c.feng@intel.com<mailto:bob.c.feng@intel.com>>; > Gao, > > > Liming <liming.gao@intel.com<mailto:liming.gao@intel.com>>; Sean Brogan > > <sean.brogan@microsoft.com<mailto:sean.brogan@microsoft.com>>; > > > Bret Barkelew <Bret.Barkelew@microsoft.com<mailto:Bret.Barkelew@microsoft.com>> > > > Subject: [Patch 1/1] BaseTools/Scripts: Add package dependency > graphing > > > tool > > > > > > https://bugzilla.tianocore.org/show_bug.cgi?id=2161<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugzilla.tianocore.org%2Fshow_bug.cgi%3Fid%3D2161&data=02%7C01%7Csean.brogan%40microsoft.com%7Cf7fb1953185b4a17f27408d7824ba488%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637121133718397648&sdata=NE82lFZmNfIcNjUfo49D%2Bksn94atxzdx7F2j%2FWSME0E%3D&reserved=0> > > > > > > Add python script that recursively scans a directory for EDK II > > > packages and generates GraphViz dot input that is used to render > > > a graph of package dependencies in SVG format. > > > > > > Detects following error/warning conditions: > > > * Ambiguous dependencies (multiple matches) > > > * Unresolved dependencies > > > * Circular dependencies > > > * Nested packages > > > > > > usage: PackageDependencyGraph [-h] [-w WORKSPACE] [-p > > PACKAGESPATH] > > > [-d DOTOUTPUTFILE] [-o SVGOUTPUTFILE] > > > [-g IGNOREDIRECTORY] [-k SKIPPACKAGE] > > > [-s] [-u] [-l] [-f] [-b] [-v] [-q] > > > [--debug [0-9]] > > > > > > Recursively scan a directory for EDK II packages and generate GraphViz > dot > > > input that is used to render a graph of package dependencies in SVG > > format. > > > Copyright (c) 2019, Intel Corporation. All rights reserved. > > > > > > optional arguments: > > > -h, --help show this help message and exit > > > -w WORKSPACE, --workspace WORKSPACE > > > Directory to recursively scan for EDK II packages. > > > Default is current directory. > > > -p PACKAGESPATH, --packages-path PACKAGESPATH > > > List of directories to recursively scan for EDK II > > > packages. > > > -d DOTOUTPUTFILE, --dot-output DOTOUTPUTFILE > > > DOT output filename. > > > -o OUTPUTFILE, --output OUTPUTFILE > > > SVG output filename. > > > -g IGNOREDIRECTORY, --ignore-directory IGNOREDIRECTORY > > > Name of directory to ignore. Option can be repeated > > > to ignore multiple directories. > > > > 1. Name of directory to ignore when scanning for EDKII Module INFs. > > > > > -k SKIPPACKAGE, --skip-package SKIPPACKAGE > > > Name of EDK II Package DEC file to skip. Option can > > > be repeated to skip multiple EDK II packages. > > > > 2. Name of EDK II Package DEC file (including .DEC suffix) to skip. Option > can > > Be repeated to skip multiple EDK II packages. > > > > > -s, --self-dependency > > > Include self links in dependency graph. Default is > > > disabled. > > > -u, --unresolved Include unresolved EDK II packages in dependency > > > graph. Default is disabled. > > > -l, --label Label links with the number of EDK II package > > > dependencies. Default is disabled. > > > -f, --full-paths Label package nodes with full path to EDK II > > > package. Default is disabled. > > > -b, --web-browser Display SVG output file in default web browser. > > > Default is disabled. > > > -v, --verbose Increase output messages > > > -q, --quiet Reduce output messages > > > --debug [0-9] Set debug level > > > > > > Cc: Ray Ni <ray.ni@intel.com<mailto:ray.ni@intel.com>> > > > Cc: Bob Feng <bob.c.feng@intel.com<mailto:bob.c.feng@intel.com>> > > > Cc: Liming Gao <liming.gao@intel.com<mailto:liming.gao@intel.com>> > > > Cc: Sean Brogan <sean.brogan@microsoft.com<mailto:sean.brogan@microsoft.com>> > > > Cc: Bret Barkelew <Bret.Barkelew@microsoft.com<mailto:Bret.Barkelew@microsoft.com>> > > > Signed-off-by: Michael D Kinney <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>> > > > --- > > > BaseTools/Scripts/PackageDependencyGraph.py | 296 > > > ++++++++++++++++++++ > > > 1 file changed, 296 insertions(+) > > > create mode 100644 BaseTools/Scripts/PackageDependencyGraph.py > > > > > > diff --git a/BaseTools/Scripts/PackageDependencyGraph.py > > > b/BaseTools/Scripts/PackageDependencyGraph.py > > > new file mode 100644 > > > index 0000000000..b3c8e41774 > > > --- /dev/null > > > +++ b/BaseTools/Scripts/PackageDependencyGraph.py > > > @@ -0,0 +1,296 @@ > > > +# @file > > > +# Recursively scan a directory for EDK II packages and generate > GraphViz > > > dot > > > +# input that is used to render a graph of package dependencies in SVG > > > format. > > > +# > > > +# Copyright (c) 2019, Intel Corporation. All rights reserved.<BR> > > > +# SPDX-License-Identifier: BSD-2-Clause-Patent > > > +# > > > +## > > > + > > > +import os > > > +import sys > > > +import argparse > > > +import subprocess > > > +import webbrowser > > > +import networkx as nx > > > +from edk2toollib.uefi.edk2.parsers.inf_parser import InfParser > > > + > > > +# > > > +# Globals for help information > > > +# > > > +__prog__ = 'PackageDependencyGraph' > > > +__copyright__ = 'Copyright (c) 2019, Intel Corporation. All rights > > reserved.' > > > +__description__ = '''Recursively scan a directory for EDK II packages and > > > +generate GraphViz dot input that is used to render a graph of package > > > +dependencies in SVG format.''' > > > + > > > +if __name__ == '__main__': > > > + > > > + # > > > + # Create command line argument parser object > > > + # > > > + parser = argparse.ArgumentParser (prog = __prog__, > > > + description = __description__ + '\n' + > __copyright__, > > > + conflict_handler = 'resolve') > > > + parser.add_argument ("-w", "--workspace", dest = 'Workspace', > default > > = > > > os.curdir, > > > + help = "Directory to recursively scan for EDK II packages. > > > Default is current directory.") > > > + parser.add_argument ("-p", "--packages-path", dest = 'PackagesPath', > > > default = None, > > > + help = "List of directories to recursively scan for EDK II > > > packages.") > > > + parser.add_argument ("-d", "--dot-output", dest = 'DotOutputFile', > > > + help = "DOT output filename.") > > > + parser.add_argument ("-o", "--output", dest = 'SvgOutputFile', > > > + help = "SVG output filename.") > > > + parser.add_argument ("-g", "--ignore-directory", dest = > > 'IgnoreDirectory', > > > action='append', default=[], > > > + help = "Name of directory to ignore. Option can be > repeated > > to > > > ignore multiple directories.") > > > + parser.add_argument ("-k", "--skip-package", dest = 'SkipPackage', > > > action='append', default=[], > > > + help = "Name of EDK II Package DEC file to skip. Option > can > > be > > > repeated to skip multiple EDK II packages.") > > > + parser.add_argument ("-s", "--self-dependency", dest = > > > 'SelfDependency', action = "store_true", default = False, > > > + help = "Include self links in dependency graph. Default is > > > disabled.") > > > + parser.add_argument ("-u", "--unresolved", dest = 'Unresolved', > action > > = > > > "store_true", default=False, > > > + help = "Include unresolved EDK II packages in dependency > > > graph. Default is disabled.") > > > + parser.add_argument ("-l", "--label", dest = 'Label', action = > > "store_true", > > > default=False, > > > + help = "Label links with the number of EDK II package > > > dependencies. Default is disabled.") > > > + parser.add_argument ("-f", "--full-paths", dest = 'FullPaths', action = > > > "store_true", default=False, > > > + help = "Label package nodes with full path to EDK II > package. > > > Default is disabled.") > > > + parser.add_argument ("-b", "--web-browser", dest = 'WebBrowser', > > > action = "store_true", default=False, > > > + help = "Display SVG output file in default web browser. > > Default > > > is disabled.") > > > + parser.add_argument ("-v", "--verbose", dest = 'Verbose', action = > > > "store_true", > > > + help = "Increase output messages") > > > + parser.add_argument ("-q", "--quiet", dest = 'Quiet', action = > > > "store_true", > > > + help = "Reduce output messages") > > > + parser.add_argument ("--debug", dest = 'Debug', type = int, metavar = > > > '[0-9]', choices = range (0, 10), default = 0, > > > + help = "Set debug level") > > > + > > > + # > > > + # Parse command line arguments > > > + # > > > + args = parser.parse_args () > > > + > > > + # > > > + # Find all EDK II package DEC files > > > + # > > > + Components = {} > > > + SearchPaths = [args.Workspace] > > > + if args.PackagesPath: > > > + SearchPaths += args.PackagesPath.split(os.pathsep) > > > + SearchPaths = [os.path.realpath(x) for x in SearchPaths] > > > + for SearchPath in SearchPaths: > > > + for root, dirs, files in os.walk (SearchPath): > > > + for name in files: > > > + FilePath = os.path.join (root, name) > > > + if > > > set(FilePath.split(os.sep)).intersection(set(args.IgnoreDirectory)) != set(): > > > + if args.Verbose: > > > + print ('IGNORE:' + FilePath) > > > + continue > > > + if os.path.splitext(FilePath)[1].lower() in ['.dec']: > > > + DecFile = os.path.realpath (FilePath) > > > + if os.path.split(DecFile)[1] in args.SkipPackage: > > > + if args.Verbose: > > > + print ('SKIP:' + DecFile) > > > + continue > > > + if DecFile not in Components: > > > + if args.Verbose: > > > + print ('PACKAGE:' + DecFile) > > > + Components[DecFile] = {} > > > + > > > + # > > > + # Find EDK II component INF files in each EDK II package > > > + # > > > + PackageLabels = {} > > > + UnresolvedPackages = [] > > > + AmbiguousDependencies = [] > > > + for DecFile in Components: > > > + DecPath = os.path.split (DecFile)[0] > > > + if DecFile not in PackageLabels: > > > + PackageLabels[DecFile] = (DecFile.replace(os.path.sep,'\\n'), > 'white') > > > + if not args.FullPaths: > > > + PackagePath = os.path.relpath(DecFile, > os.path.split(DecPath)[0]) > > > + PackageLabels[DecFile] = > (PackagePath.replace(os.path.sep,'\\n'), > > > 'white') > > > + for root, dirs, files in os.walk (DecPath): > > > + for name in files: > > > + FilePath = os.path.join(root, name) > > > + if > > > set(FilePath.split(os.sep)).intersection(set(args.IgnoreDirectory)) != set(): > > > + if args.Verbose: > > > + print ('IGNORE:' + FilePath) > > > + continue > > > + if os.path.splitext(FilePath)[1].lower() in ['.inf']: > > > + InfFile = os.path.realpath (FilePath) > > > + Inf = InfParser () > > > + Inf.ParseFile (InfFile) > > > + DependentPackages = [] > > > + for Dependency in Inf.PackagesUsed: > > > + Dependency = os.path.normpath(Dependency) > > > + if os.path.split(Dependency)[1] in args.SkipPackage: > > > + if args.Verbose: > > > + print ('SKIP:' + Dependency) > > > + continue > > > + Found = False > > > + for SearchPath in SearchPaths: > > > + PackagePath = os.path.realpath(os.path.join(SearchPath, > > > Dependency)) > > > + if os.path.exists(PackagePath): > > > + DependentPackages.append(PackagePath) > > > + if not args.FullPaths: > > > + PackageLabels[PackagePath] = > > > (Dependency.replace(os.path.sep,'\\n'), 'white') > > > + Found = True > > > + break > > > + if not Found: > > > + Count = 0 > > > + Match = '' > > > + for DecFile2 in Components: > > > + if DecFile2.endswith(Dependency): > > > + if Count == 0: > > > + Match = DecFile2 > > > + Count = Count + 1 > > > + if Count > 1: > > > + AmbiguousDependencies.append (Dependency) > > > + if Count == 1: > > > + DependentPackages.append(Match) > > > + if not args.FullPaths: > > > + PackageLabels[Match] = > > > (Dependency.replace(os.path.sep,'\\n'), 'white') > > > + Found = True > > > + if not Found and args.Unresolved: > > > + if args.Verbose: > > > + print ('WARNING: Dependent package not found ' + > > > Dependency) > > > + DependentPackages.append(Dependency) > > > + UnresolvedPackages.append(Dependency) > > > + PackageLabels[Dependency] = > > > (Dependency.replace(os.path.sep,'\\n'), 'white') > > > + Components[DecFile][InfFile] = DependentPackages > > > + if AmbiguousDependencies: > > > + for Dependency in set(AmbiguousDependencies): > > > + print ('ERROR: MULTIPLE: ' + Dependency) > > > + print ('Use --packages-path to provide search priority.') > > > + sys.exit(1) > > > + > > > + # > > > + # Generate GraphViz dot input file contents. > > > + # Use networkx to detect circular dependencies. > > > + # > > > + MaxWeight = 0 > > > + MaxWeightDecFile = list(Components.keys())[0] > > > + Graph = nx.DiGraph() > > > + Edges = [] > > > + for DecFile in Components: > > > + if args.Verbose: > > > + print ('PACKAGE DEPENDENCIES:' + DecFile) > > > + AllDependencies = [] > > > + Dependencies = set() > > > + for InfFile in Components[DecFile]: > > > + AllDependencies += Components[DecFile][InfFile] > > > + Dependencies = > Dependencies.union(Components[DecFile][InfFile]) > > > + if not args.SelfDependency: > > > + Dependencies = Dependencies.difference(set([DecFile])) > > > + for Dependency in Dependencies: > > > + Weight = AllDependencies.count(Dependency) > > > + if Weight > MaxWeight: > > > + MaxWeight = Weight > > > + MaxWeightDecFile = DecFile > > > + if args.Verbose: > > > + print (' DEPENDENCY: Weight(' + str(Weight) + ') ' + > Dependency) > > > + Edges.append(' "{Package}" -> "{Dependency}" [label = > > > "{Weight}"];'.format( > > > + Package = DecFile, > > > + Dependency = Dependency, > > > + Weight = str(Weight) if args.Label else '' > > > + )) > > > + Graph.add_edge(DecFile, Dependency) > > > + Edges.sort() > > > + > > > + # > > > + # Set fill color to yellow if a packages is part of a circular dependency > > > + # > > > + for Node in set([x for y in list(nx.simple_cycles(Graph)) for x in y]): > > > + PackageLabels[Node] = (PackageLabels[Node][0], 'yellow') > > > + print ('ERROR: CIRCULAR: ' + Node) > > > + > > > + # > > > + # Set fill color to pink if a package that is nested inside another > package. > > > + # Set fill color to orange if a package is nested inside another package > > and > > > + # is part of a circular dependency. > > > + # > > > + for DecFile in Components: > > > + DecPath = os.path.split (DecFile)[0] > > > + for DecFile2 in Components: > > > + if DecFile2 == DecFile: > > > + continue > > > + if os.path.commonpath ([DecPath, DecFile2]) != DecPath: > > > + continue > > > + if len(DecFile2) < len(DecPath): > > > + DecFile2 = DecFile > > > + print ('ERROR: NESTED: ' + DecFile2) > > > + if PackageLabels[DecFile2][1] == 'yellow': > > > + PackageLabels[DecFile2] = (PackageLabels[DecFile2][0], > 'orange') > > > + else: > > > + PackageLabels[DecFile2] = (PackageLabels[DecFile2][0], 'pink') > > > + > > > + # > > > + # Set fill color to red for nodes that are unresolved > > > + # > > > + for Node in set(UnresolvedPackages): > > > + PackageLabels[Node] = (PackageLabels[Node][0], 'red') > > > + print ('ERROR: UNRESOLVED: ' + Node) > > > + > > > + # > > > + # Add node statements to set node label and fill color > > > + # > > > + Nodes = [] > > > + for Package in PackageLabels: > > > + Nodes.append(' "{Package}" > > [label="{Label}",fillcolor={Color}];'.format( > > > + Package = Package, > > > + Label = PackageLabels[Package][0], > > > + Color = PackageLabels[Package][1] > > > + )) > > > + Nodes.sort() > > > + > > > + # > > > + # Generate dot file from Nodes and Edges and add a Legend at top of > > > graph > > > + # > > > + Dot = [] > > > + Dot.append('digraph {') > > > + Dot.append(' rankdir=BT;') > > > + Dot.append(' node [shape=Mrecord,style=filled];') > > > + Dot.append('') > > > + Dot = Dot + Nodes > > > + Dot.append('') > > > + Dot = Dot + Edges > > > + Dot.append('') > > > + Dot.append(' subgraph legend {') > > > + Dot.append(' rank=sink;') > > > + Dot.append(' Unresolved [label="Unresolved Dependency", > > > fillcolor=red];') > > > + Dot.append(' Circular [label="Circular Dependency", > > > fillcolor=yellow];') > > > + Dot.append(' Nested [label="Nested Package", > > > fillcolor=pink];') > > > + Dot.append(' NestedCircular [label="Nested Package with Circular > > > Dependency",fillcolor=orange];') > > > + Dot.append(' }') > > > + if MaxWeightDecFile: > > > + Dot.append(' Unresolved->"' + MaxWeightDecFile + '" [style=invis];') > > > + Dot.append('}') > > > + > > > + if args.DotOutputFile: > > > + # > > > + # Write GraphViz dot file contents to DotOutputFile > > > + # > > > + with open(os.path.realpath(args.DotOutputFile), 'w') as File: > > > + File.write('\n'.join(Dot)) > > > + if args.SvgOutputFile: > > > + # > > > + # Use GraphViz 'dot' command to generate SVG output file > > > + # > > > + args.SvgOutputFile = os.path.realpath(args.SvgOutputFile) > > > + try: > > > + Process = subprocess.Popen('dot -Tsvg', > > > + stdin=subprocess.PIPE, > > > + stdout=open(args.SvgOutputFile, 'w'), > > > + stderr=subprocess.PIPE, > > > + shell=True > > > + ) > > > + Process.stdin.write ('\n'.join(Dot).encode()) > > > + Process.communicate() > > > + if Process.returncode != 0: > > > + print ("ERROR: Can not run GraphViz 'dot' command. Check > install > > > and path.") > > > + sys.exit(Process.returncode) > > > + except: > > > + print ("ERROR: Can not run GraphViz 'dot' command. Check > install > > and > > > path.") > > > + sys.exit(1) > > > + # > > > + # Display SVG file in default web browser > > > + # > > > + if args.WebBrowser: > > > + webbrowser.open(args.SvgOutputFile) > > > -- > > > 2.21.0.windows.1 > > > > > > > > > [-- Attachment #2: Type: text/html, Size: 82059 bytes --] ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [edk2-devel] [Patch 1/1] BaseTools/Scripts: Add package dependency graphing tool 2019-12-17 2:48 ` Ni, Ray @ 2019-12-17 16:10 ` Steven Shi 0 siblings, 0 replies; 12+ messages in thread From: Steven Shi @ 2019-12-17 16:10 UTC (permalink / raw) To: Ni, Ray, devel@edk2.groups.io, sean.brogan@microsoft.com, Kinney, Michael D Cc: Feng, Bob C, Gao, Liming, Bret Barkelew [-- Attachment #1.1: Type: text/plain, Size: 30371 bytes --] Hi Sean, Your tool is cool. I wish we could merge you and Mike’s tool together. I create a dependency html report with your tool in attachment. I have below feedback: 1. The filters response in Dependency Explorer tab are slow. 2. Only have fanout views, but no fanin views. I have a Stability Metric to measure the stability of a module, and I need the module fanin data. For a given module m, let fanin(m) be the set of modules that depend on m and let fanout(m) be the set of modules that m depends on. Following Martin [1][2], the Instability I(m) of m is defined as fanout(m)/( fanin(m) + fanout(m)). I’m OK that the fanin(m) data only make sense for a real platform build, and only need to generate it if input a platform build report. 3. If possible, please combine and output all the dependency data like below as a single Json file, which will be easy for other metrics tool to consume. ModuleDeps = [] # hold all deps ModuleList = [] AsBuiltMappings = [] LibraryClassInstanceDict = {} [1] https://fi.ort.edu.uy/innovaportal/file/2032/1/design_principles.pdf, page 23, Stability Metrics [2] https://ieeexplore.ieee.org/abstract/document/4027146, chapter 6.3, Module Interaction Stability Index Thanks Steven From: Ni, Ray <ray.ni@intel.com> Sent: Tuesday, December 17, 2019 10:49 AM To: devel@edk2.groups.io; sean.brogan@microsoft.com; Kinney, Michael D <michael.d.kinney@intel.com>; Shi, Steven <steven.shi@intel.com> Cc: Feng, Bob C <bob.c.feng@intel.com>; Gao, Liming <liming.gao@intel.com>; Bret Barkelew <Bret.Barkelew@microsoft.com> Subject: RE: [edk2-devel] [Patch 1/1] BaseTools/Scripts: Add package dependency graphing tool Sean, This is a very cool tool! We could learn from your tool source code to know how to use DecParser to get more detailed information, like which interfaces (PPI/Protocol/Guid/PCD/Lib) from which package a module is depending on. Thanks, Ray From: devel@edk2.groups.io<mailto:devel@edk2.groups.io> <devel@edk2.groups.io<mailto:devel@edk2.groups.io>> On Behalf Of Sean via Groups.Io Sent: Tuesday, December 17, 2019 3:46 AM To: Kinney, Michael D <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>>; Shi, Steven <steven.shi@intel.com<mailto:steven.shi@intel.com>>; devel@edk2.groups.io<mailto:devel@edk2.groups.io>; Ni, Ray <ray.ni@intel.com<mailto:ray.ni@intel.com>>; Kinney, Michael D <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>> Cc: Feng, Bob C <bob.c.feng@intel.com<mailto:bob.c.feng@intel.com>>; Gao, Liming <liming.gao@intel.com<mailto:liming.gao@intel.com>>; Bret Barkelew <Bret.Barkelew@microsoft.com<mailto:Bret.Barkelew@microsoft.com>> Subject: Re: [edk2-devel] [Patch 1/1] BaseTools/Scripts: Add package dependency graphing tool Mike/Steven/Ray, We have had a tool for a couple of years with similar functionality. It is a python tool you run on your code base and it creates a single (local) webpage that you can then use to build charts. Please give it a try and see if you think it is useful. Here is the tool. https://github.com/spbrogan/edk2_dep And here is a html report run on edk2 https://github.com/spbrogan/edk2_dep/blob/sample/edk2/Edk2_core.html Thanks Sean From: Kinney, Michael D <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>> Sent: Monday, December 16, 2019 9:16 AM To: Shi, Steven <steven.shi@intel.com<mailto:steven.shi@intel.com>>; devel@edk2.groups.io<mailto:devel@edk2.groups.io>; Ni, Ray <ray.ni@intel.com<mailto:ray.ni@intel.com>>; Kinney, Michael D <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>> Cc: Feng, Bob C <bob.c.feng@intel.com<mailto:bob.c.feng@intel.com>>; Gao, Liming <liming.gao@intel.com<mailto:liming.gao@intel.com>>; Sean Brogan <sean.brogan@microsoft.com<mailto:sean.brogan@microsoft.com>>; Bret Barkelew <Bret.Barkelew@microsoft.com<mailto:Bret.Barkelew@microsoft.com>> Subject: [EXTERNAL] RE: [edk2-devel] [Patch 1/1] BaseTools/Scripts: Add package dependency graphing tool Steve, Yes. I think a Module scoped view and a Platform scoped view would be good additions. Can you please enter new BZ feature requests for these and add a summary of the features you would like to see in each? Thanks, Mike From: Shi, Steven <steven.shi@intel.com<mailto:steven.shi@intel.com>> Sent: Monday, December 16, 2019 12:48 AM To: devel@edk2.groups.io<mailto:devel@edk2.groups.io>; Ni, Ray <ray.ni@intel.com<mailto:ray.ni@intel.com>>; Kinney, Michael D <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>> Cc: Feng, Bob C <bob.c.feng@intel.com<mailto:bob.c.feng@intel.com>>; Gao, Liming <liming.gao@intel.com<mailto:liming.gao@intel.com>>; Sean Brogan <sean.brogan@microsoft.com<mailto:sean.brogan@microsoft.com>>; Bret Barkelew <Bret.Barkelew@microsoft.com<mailto:Bret.Barkelew@microsoft.com>> Subject: RE: [edk2-devel] [Patch 1/1] BaseTools/Scripts: Add package dependency graphing tool Ray, I think you are asking for a module-level dependency tool. 😊 Mike, This tool is cool. Besides the package level, do you think we can go further to figure out a module level dependency tool? With module level dependency info, we can define more interesting and comprehensive Modularity Metrics to measure the edk2 architecture quality (Architectural Technical Debt). Thanks Steven Shi Intel\SSG\SFE\FIE Firmware Infrastructure > -----Original Message----- > From: devel@edk2.groups.io<mailto:devel@edk2.groups.io> <devel@edk2.groups.io<mailto:devel@edk2.groups.io>> On Behalf Of Ni, Ray > Sent: Monday, December 16, 2019 4:41 PM > To: devel@edk2.groups.io<mailto:devel@edk2.groups.io>; Ni, Ray <ray.ni@intel.com<mailto:ray.ni@intel.com>>; Kinney, Michael D > <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>> > Cc: Feng, Bob C <bob.c.feng@intel.com<mailto:bob.c.feng@intel.com>>; Gao, Liming > <liming.gao@intel.com<mailto:liming.gao@intel.com>>; Sean Brogan <sean.brogan@microsoft.com<mailto:sean.brogan@microsoft.com>>; Bret > Barkelew <Bret.Barkelew@microsoft.com<mailto:Bret.Barkelew@microsoft.com>> > Subject: Re: [edk2-devel] [Patch 1/1] BaseTools/Scripts: Add package > dependency graphing tool > > Mike, > This pkg dep tool can tell through weight when a module depends on a new > pkg. > But it cannot tell when a module depends on another > PCD/Protocol/Guid/Ppi/Library which belongs to an already-depended pkg. > > Failure to tell such information may make a bad module (that violates the > dependency rules) worse (wrongly depend on more interfaces). > > Do you think that the tool can be enhanced in future to detect the case? > > Thanks, > Ray > > > -----Original Message----- > > From: devel@edk2.groups.io<mailto:devel@edk2.groups.io> <devel@edk2.groups.io<mailto:devel@edk2.groups.io>> On Behalf Of Ni, > Ray > > Sent: Monday, December 16, 2019 1:57 PM > > To: Kinney, Michael D <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>>; > devel@edk2.groups.io<mailto:devel@edk2.groups.io> > > Cc: Feng, Bob C <bob.c.feng@intel.com<mailto:bob.c.feng@intel.com>>; Gao, Liming > > <liming.gao@intel.com<mailto:liming.gao@intel.com>>; Sean Brogan <sean.brogan@microsoft.com<mailto:sean.brogan@microsoft.com>>; > Bret > > Barkelew <Bret.Barkelew@microsoft.com<mailto:Bret.Barkelew@microsoft.com>> > > Subject: Re: [edk2-devel] [Patch 1/1] BaseTools/Scripts: Add package > > dependency graphing tool > > > > Mike, > > 2 minor comments regarding the help string in below. > > > > > -----Original Message----- > > > From: Kinney, Michael D <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>> > > > Sent: Saturday, December 14, 2019 3:45 AM > > > To: devel@edk2.groups.io<mailto:devel@edk2.groups.io> > > > Cc: Ni, Ray <ray.ni@intel.com<mailto:ray.ni@intel.com>>; Feng, Bob C <bob.c.feng@intel.com<mailto:bob.c.feng@intel.com>>; > Gao, > > > Liming <liming.gao@intel.com<mailto:liming.gao@intel.com>>; Sean Brogan > > <sean.brogan@microsoft.com<mailto:sean.brogan@microsoft.com>>; > > > Bret Barkelew <Bret.Barkelew@microsoft.com<mailto:Bret.Barkelew@microsoft.com>> > > > Subject: [Patch 1/1] BaseTools/Scripts: Add package dependency > graphing > > > tool > > > > > > https://bugzilla.tianocore.org/show_bug.cgi?id=2161<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugzilla.tianocore.org%2Fshow_bug.cgi%3Fid%3D2161&data=02%7C01%7Csean.brogan%40microsoft.com%7Cf7fb1953185b4a17f27408d7824ba488%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637121133718397648&sdata=NE82lFZmNfIcNjUfo49D%2Bksn94atxzdx7F2j%2FWSME0E%3D&reserved=0> > > > > > > Add python script that recursively scans a directory for EDK II > > > packages and generates GraphViz dot input that is used to render > > > a graph of package dependencies in SVG format. > > > > > > Detects following error/warning conditions: > > > * Ambiguous dependencies (multiple matches) > > > * Unresolved dependencies > > > * Circular dependencies > > > * Nested packages > > > > > > usage: PackageDependencyGraph [-h] [-w WORKSPACE] [-p > > PACKAGESPATH] > > > [-d DOTOUTPUTFILE] [-o SVGOUTPUTFILE] > > > [-g IGNOREDIRECTORY] [-k SKIPPACKAGE] > > > [-s] [-u] [-l] [-f] [-b] [-v] [-q] > > > [--debug [0-9]] > > > > > > Recursively scan a directory for EDK II packages and generate GraphViz > dot > > > input that is used to render a graph of package dependencies in SVG > > format. > > > Copyright (c) 2019, Intel Corporation. All rights reserved. > > > > > > optional arguments: > > > -h, --help show this help message and exit > > > -w WORKSPACE, --workspace WORKSPACE > > > Directory to recursively scan for EDK II packages. > > > Default is current directory. > > > -p PACKAGESPATH, --packages-path PACKAGESPATH > > > List of directories to recursively scan for EDK II > > > packages. > > > -d DOTOUTPUTFILE, --dot-output DOTOUTPUTFILE > > > DOT output filename. > > > -o OUTPUTFILE, --output OUTPUTFILE > > > SVG output filename. > > > -g IGNOREDIRECTORY, --ignore-directory IGNOREDIRECTORY > > > Name of directory to ignore. Option can be repeated > > > to ignore multiple directories. > > > > 1. Name of directory to ignore when scanning for EDKII Module INFs. > > > > > -k SKIPPACKAGE, --skip-package SKIPPACKAGE > > > Name of EDK II Package DEC file to skip. Option can > > > be repeated to skip multiple EDK II packages. > > > > 2. Name of EDK II Package DEC file (including .DEC suffix) to skip. Option > can > > Be repeated to skip multiple EDK II packages. > > > > > -s, --self-dependency > > > Include self links in dependency graph. Default is > > > disabled. > > > -u, --unresolved Include unresolved EDK II packages in dependency > > > graph. Default is disabled. > > > -l, --label Label links with the number of EDK II package > > > dependencies. Default is disabled. > > > -f, --full-paths Label package nodes with full path to EDK II > > > package. Default is disabled. > > > -b, --web-browser Display SVG output file in default web browser. > > > Default is disabled. > > > -v, --verbose Increase output messages > > > -q, --quiet Reduce output messages > > > --debug [0-9] Set debug level > > > > > > Cc: Ray Ni <ray.ni@intel.com<mailto:ray.ni@intel.com>> > > > Cc: Bob Feng <bob.c.feng@intel.com<mailto:bob.c.feng@intel.com>> > > > Cc: Liming Gao <liming.gao@intel.com<mailto:liming.gao@intel.com>> > > > Cc: Sean Brogan <sean.brogan@microsoft.com<mailto:sean.brogan@microsoft.com>> > > > Cc: Bret Barkelew <Bret.Barkelew@microsoft.com<mailto:Bret.Barkelew@microsoft.com>> > > > Signed-off-by: Michael D Kinney <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>> > > > --- > > > BaseTools/Scripts/PackageDependencyGraph.py | 296 > > > ++++++++++++++++++++ > > > 1 file changed, 296 insertions(+) > > > create mode 100644 BaseTools/Scripts/PackageDependencyGraph.py > > > > > > diff --git a/BaseTools/Scripts/PackageDependencyGraph.py > > > b/BaseTools/Scripts/PackageDependencyGraph.py > > > new file mode 100644 > > > index 0000000000..b3c8e41774 > > > --- /dev/null > > > +++ b/BaseTools/Scripts/PackageDependencyGraph.py > > > @@ -0,0 +1,296 @@ > > > +# @file > > > +# Recursively scan a directory for EDK II packages and generate > GraphViz > > > dot > > > +# input that is used to render a graph of package dependencies in SVG > > > format. > > > +# > > > +# Copyright (c) 2019, Intel Corporation. All rights reserved.<BR> > > > +# SPDX-License-Identifier: BSD-2-Clause-Patent > > > +# > > > +## > > > + > > > +import os > > > +import sys > > > +import argparse > > > +import subprocess > > > +import webbrowser > > > +import networkx as nx > > > +from edk2toollib.uefi.edk2.parsers.inf_parser import InfParser > > > + > > > +# > > > +# Globals for help information > > > +# > > > +__prog__ = 'PackageDependencyGraph' > > > +__copyright__ = 'Copyright (c) 2019, Intel Corporation. All rights > > reserved.' > > > +__description__ = '''Recursively scan a directory for EDK II packages and > > > +generate GraphViz dot input that is used to render a graph of package > > > +dependencies in SVG format.''' > > > + > > > +if __name__ == '__main__': > > > + > > > + # > > > + # Create command line argument parser object > > > + # > > > + parser = argparse.ArgumentParser (prog = __prog__, > > > + description = __description__ + '\n' + > __copyright__, > > > + conflict_handler = 'resolve') > > > + parser.add_argument ("-w", "--workspace", dest = 'Workspace', > default > > = > > > os.curdir, > > > + help = "Directory to recursively scan for EDK II packages. > > > Default is current directory.") > > > + parser.add_argument ("-p", "--packages-path", dest = 'PackagesPath', > > > default = None, > > > + help = "List of directories to recursively scan for EDK II > > > packages.") > > > + parser.add_argument ("-d", "--dot-output", dest = 'DotOutputFile', > > > + help = "DOT output filename.") > > > + parser.add_argument ("-o", "--output", dest = 'SvgOutputFile', > > > + help = "SVG output filename.") > > > + parser.add_argument ("-g", "--ignore-directory", dest = > > 'IgnoreDirectory', > > > action='append', default=[], > > > + help = "Name of directory to ignore. Option can be > repeated > > to > > > ignore multiple directories.") > > > + parser.add_argument ("-k", "--skip-package", dest = 'SkipPackage', > > > action='append', default=[], > > > + help = "Name of EDK II Package DEC file to skip. Option > can > > be > > > repeated to skip multiple EDK II packages.") > > > + parser.add_argument ("-s", "--self-dependency", dest = > > > 'SelfDependency', action = "store_true", default = False, > > > + help = "Include self links in dependency graph. Default is > > > disabled.") > > > + parser.add_argument ("-u", "--unresolved", dest = 'Unresolved', > action > > = > > > "store_true", default=False, > > > + help = "Include unresolved EDK II packages in dependency > > > graph. Default is disabled.") > > > + parser.add_argument ("-l", "--label", dest = 'Label', action = > > "store_true", > > > default=False, > > > + help = "Label links with the number of EDK II package > > > dependencies. Default is disabled.") > > > + parser.add_argument ("-f", "--full-paths", dest = 'FullPaths', action = > > > "store_true", default=False, > > > + help = "Label package nodes with full path to EDK II > package. > > > Default is disabled.") > > > + parser.add_argument ("-b", "--web-browser", dest = 'WebBrowser', > > > action = "store_true", default=False, > > > + help = "Display SVG output file in default web browser. > > Default > > > is disabled.") > > > + parser.add_argument ("-v", "--verbose", dest = 'Verbose', action = > > > "store_true", > > > + help = "Increase output messages") > > > + parser.add_argument ("-q", "--quiet", dest = 'Quiet', action = > > > "store_true", > > > + help = "Reduce output messages") > > > + parser.add_argument ("--debug", dest = 'Debug', type = int, metavar = > > > '[0-9]', choices = range (0, 10), default = 0, > > > + help = "Set debug level") > > > + > > > + # > > > + # Parse command line arguments > > > + # > > > + args = parser.parse_args () > > > + > > > + # > > > + # Find all EDK II package DEC files > > > + # > > > + Components = {} > > > + SearchPaths = [args.Workspace] > > > + if args.PackagesPath: > > > + SearchPaths += args.PackagesPath.split(os.pathsep) > > > + SearchPaths = [os.path.realpath(x) for x in SearchPaths] > > > + for SearchPath in SearchPaths: > > > + for root, dirs, files in os.walk (SearchPath): > > > + for name in files: > > > + FilePath = os.path.join (root, name) > > > + if > > > set(FilePath.split(os.sep)).intersection(set(args.IgnoreDirectory)) != set(): > > > + if args.Verbose: > > > + print ('IGNORE:' + FilePath) > > > + continue > > > + if os.path.splitext(FilePath)[1].lower() in ['.dec']: > > > + DecFile = os.path.realpath (FilePath) > > > + if os.path.split(DecFile)[1] in args.SkipPackage: > > > + if args.Verbose: > > > + print ('SKIP:' + DecFile) > > > + continue > > > + if DecFile not in Components: > > > + if args.Verbose: > > > + print ('PACKAGE:' + DecFile) > > > + Components[DecFile] = {} > > > + > > > + # > > > + # Find EDK II component INF files in each EDK II package > > > + # > > > + PackageLabels = {} > > > + UnresolvedPackages = [] > > > + AmbiguousDependencies = [] > > > + for DecFile in Components: > > > + DecPath = os.path.split (DecFile)[0] > > > + if DecFile not in PackageLabels: > > > + PackageLabels[DecFile] = (DecFile.replace(os.path.sep,'\\n'), > 'white') > > > + if not args.FullPaths: > > > + PackagePath = os.path.relpath(DecFile, > os.path.split(DecPath)[0]) > > > + PackageLabels[DecFile] = > (PackagePath.replace(os.path.sep,'\\n'), > > > 'white') > > > + for root, dirs, files in os.walk (DecPath): > > > + for name in files: > > > + FilePath = os.path.join(root, name) > > > + if > > > set(FilePath.split(os.sep)).intersection(set(args.IgnoreDirectory)) != set(): > > > + if args.Verbose: > > > + print ('IGNORE:' + FilePath) > > > + continue > > > + if os.path.splitext(FilePath)[1].lower() in ['.inf']: > > > + InfFile = os.path.realpath (FilePath) > > > + Inf = InfParser () > > > + Inf.ParseFile (InfFile) > > > + DependentPackages = [] > > > + for Dependency in Inf.PackagesUsed: > > > + Dependency = os.path.normpath(Dependency) > > > + if os.path.split(Dependency)[1] in args.SkipPackage: > > > + if args.Verbose: > > > + print ('SKIP:' + Dependency) > > > + continue > > > + Found = False > > > + for SearchPath in SearchPaths: > > > + PackagePath = os.path.realpath(os.path.join(SearchPath, > > > Dependency)) > > > + if os.path.exists(PackagePath): > > > + DependentPackages.append(PackagePath) > > > + if not args.FullPaths: > > > + PackageLabels[PackagePath] = > > > (Dependency.replace(os.path.sep,'\\n'), 'white') > > > + Found = True > > > + break > > > + if not Found: > > > + Count = 0 > > > + Match = '' > > > + for DecFile2 in Components: > > > + if DecFile2.endswith(Dependency): > > > + if Count == 0: > > > + Match = DecFile2 > > > + Count = Count + 1 > > > + if Count > 1: > > > + AmbiguousDependencies.append (Dependency) > > > + if Count == 1: > > > + DependentPackages.append(Match) > > > + if not args.FullPaths: > > > + PackageLabels[Match] = > > > (Dependency.replace(os.path.sep,'\\n'), 'white') > > > + Found = True > > > + if not Found and args.Unresolved: > > > + if args.Verbose: > > > + print ('WARNING: Dependent package not found ' + > > > Dependency) > > > + DependentPackages.append(Dependency) > > > + UnresolvedPackages.append(Dependency) > > > + PackageLabels[Dependency] = > > > (Dependency.replace(os.path.sep,'\\n'), 'white') > > > + Components[DecFile][InfFile] = DependentPackages > > > + if AmbiguousDependencies: > > > + for Dependency in set(AmbiguousDependencies): > > > + print ('ERROR: MULTIPLE: ' + Dependency) > > > + print ('Use --packages-path to provide search priority.') > > > + sys.exit(1) > > > + > > > + # > > > + # Generate GraphViz dot input file contents. > > > + # Use networkx to detect circular dependencies. > > > + # > > > + MaxWeight = 0 > > > + MaxWeightDecFile = list(Components.keys())[0] > > > + Graph = nx.DiGraph() > > > + Edges = [] > > > + for DecFile in Components: > > > + if args.Verbose: > > > + print ('PACKAGE DEPENDENCIES:' + DecFile) > > > + AllDependencies = [] > > > + Dependencies = set() > > > + for InfFile in Components[DecFile]: > > > + AllDependencies += Components[DecFile][InfFile] > > > + Dependencies = > Dependencies.union(Components[DecFile][InfFile]) > > > + if not args.SelfDependency: > > > + Dependencies = Dependencies.difference(set([DecFile])) > > > + for Dependency in Dependencies: > > > + Weight = AllDependencies.count(Dependency) > > > + if Weight > MaxWeight: > > > + MaxWeight = Weight > > > + MaxWeightDecFile = DecFile > > > + if args.Verbose: > > > + print (' DEPENDENCY: Weight(' + str(Weight) + ') ' + > Dependency) > > > + Edges.append(' "{Package}" -> "{Dependency}" [label = > > > "{Weight}"];'.format( > > > + Package = DecFile, > > > + Dependency = Dependency, > > > + Weight = str(Weight) if args.Label else '' > > > + )) > > > + Graph.add_edge(DecFile, Dependency) > > > + Edges.sort() > > > + > > > + # > > > + # Set fill color to yellow if a packages is part of a circular dependency > > > + # > > > + for Node in set([x for y in list(nx.simple_cycles(Graph)) for x in y]): > > > + PackageLabels[Node] = (PackageLabels[Node][0], 'yellow') > > > + print ('ERROR: CIRCULAR: ' + Node) > > > + > > > + # > > > + # Set fill color to pink if a package that is nested inside another > package. > > > + # Set fill color to orange if a package is nested inside another package > > and > > > + # is part of a circular dependency. > > > + # > > > + for DecFile in Components: > > > + DecPath = os.path.split (DecFile)[0] > > > + for DecFile2 in Components: > > > + if DecFile2 == DecFile: > > > + continue > > > + if os.path.commonpath ([DecPath, DecFile2]) != DecPath: > > > + continue > > > + if len(DecFile2) < len(DecPath): > > > + DecFile2 = DecFile > > > + print ('ERROR: NESTED: ' + DecFile2) > > > + if PackageLabels[DecFile2][1] == 'yellow': > > > + PackageLabels[DecFile2] = (PackageLabels[DecFile2][0], > 'orange') > > > + else: > > > + PackageLabels[DecFile2] = (PackageLabels[DecFile2][0], 'pink') > > > + > > > + # > > > + # Set fill color to red for nodes that are unresolved > > > + # > > > + for Node in set(UnresolvedPackages): > > > + PackageLabels[Node] = (PackageLabels[Node][0], 'red') > > > + print ('ERROR: UNRESOLVED: ' + Node) > > > + > > > + # > > > + # Add node statements to set node label and fill color > > > + # > > > + Nodes = [] > > > + for Package in PackageLabels: > > > + Nodes.append(' "{Package}" > > [label="{Label}",fillcolor={Color}];'.format( > > > + Package = Package, > > > + Label = PackageLabels[Package][0], > > > + Color = PackageLabels[Package][1] > > > + )) > > > + Nodes.sort() > > > + > > > + # > > > + # Generate dot file from Nodes and Edges and add a Legend at top of > > > graph > > > + # > > > + Dot = [] > > > + Dot.append('digraph {') > > > + Dot.append(' rankdir=BT;') > > > + Dot.append(' node [shape=Mrecord,style=filled];') > > > + Dot.append('') > > > + Dot = Dot + Nodes > > > + Dot.append('') > > > + Dot = Dot + Edges > > > + Dot.append('') > > > + Dot.append(' subgraph legend {') > > > + Dot.append(' rank=sink;') > > > + Dot.append(' Unresolved [label="Unresolved Dependency", > > > fillcolor=red];') > > > + Dot.append(' Circular [label="Circular Dependency", > > > fillcolor=yellow];') > > > + Dot.append(' Nested [label="Nested Package", > > > fillcolor=pink];') > > > + Dot.append(' NestedCircular [label="Nested Package with Circular > > > Dependency",fillcolor=orange];') > > > + Dot.append(' }') > > > + if MaxWeightDecFile: > > > + Dot.append(' Unresolved->"' + MaxWeightDecFile + '" [style=invis];') > > > + Dot.append('}') > > > + > > > + if args.DotOutputFile: > > > + # > > > + # Write GraphViz dot file contents to DotOutputFile > > > + # > > > + with open(os.path.realpath(args.DotOutputFile), 'w') as File: > > > + File.write('\n'.join(Dot)) > > > + if args.SvgOutputFile: > > > + # > > > + # Use GraphViz 'dot' command to generate SVG output file > > > + # > > > + args.SvgOutputFile = os.path.realpath(args.SvgOutputFile) > > > + try: > > > + Process = subprocess.Popen('dot -Tsvg', > > > + stdin=subprocess.PIPE, > > > + stdout=open(args.SvgOutputFile, 'w'), > > > + stderr=subprocess.PIPE, > > > + shell=True > > > + ) > > > + Process.stdin.write ('\n'.join(Dot).encode()) > > > + Process.communicate() > > > + if Process.returncode != 0: > > > + print ("ERROR: Can not run GraphViz 'dot' command. Check > install > > > and path.") > > > + sys.exit(Process.returncode) > > > + except: > > > + print ("ERROR: Can not run GraphViz 'dot' command. Check > install > > and > > > path.") > > > + sys.exit(1) > > > + # > > > + # Display SVG file in default web browser > > > + # > > > + if args.WebBrowser: > > > + webbrowser.open(args.SvgOutputFile) > > > -- > > > 2.21.0.windows.1 > > > > > > > > > [-- Attachment #1.2: Type: text/html, Size: 86215 bytes --] [-- Attachment #2: dep_tool.zip --] [-- Type: application/x-zip-compressed, Size: 301758 bytes --] ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [edk2-devel] [Patch 1/1] BaseTools/Scripts: Add package dependency graphing tool 2019-12-16 17:16 ` Michael D Kinney 2019-12-16 19:45 ` Sean @ 2019-12-19 1:11 ` Steven Shi 1 sibling, 0 replies; 12+ messages in thread From: Steven Shi @ 2019-12-19 1:11 UTC (permalink / raw) To: Kinney, Michael D, devel@edk2.groups.io, Ni, Ray Cc: Feng, Bob C, Gao, Liming, Sean Brogan, Bret Barkelew [-- Attachment #1: Type: text/plain, Size: 28031 bytes --] I reuse the BZ 2161 and add my requirement there as below. https://bugzilla.tianocore.org/show_bug.cgi?id=2161 1. If without platform build log or DSC info, the tool only need to output module fanout build dependency info which includes what APIs (PPI/Protocol/Guid/PCD/Lib) a module depends on. No need to tell what these API's implementation instance are (e.g. you don't know which library instance implement the library class/API without platform build info). 2. If with platform build log or DSC info, the tool need to output both module fanin and fanout info. And for the fanin and fanout info, the tool need output both APIs and their implementation instances paths (only need library instance module path and Static PCDs which impact a module build). 3. Please be aware the above dependency is only for the build dependency, not for the functionality dependency. Because the Uefi PPI/Protocol/Guid API's instances are always dynamic resolved/binding, and are not really needed in other module's static build, so only the PPI/Protocol/Guid API's definitions, not their instances, are the dependency of other module. But the library and static PCD are different, and their instances are necessary for other depending module pass build, so the library and static PCD instances are the dependency of other module. 4. Besides graph views, please combine and output all the dependency data as a single Json file, which will be easy for other metrics tool to consume. Thanks Steven Shi Intel\SSG\SFE\FIE Firmware Infrastructure From: Kinney, Michael D <michael.d.kinney@intel.com> Sent: Tuesday, December 17, 2019 1:16 AM To: Shi, Steven <steven.shi@intel.com>; devel@edk2.groups.io; Ni, Ray <ray.ni@intel.com>; Kinney, Michael D <michael.d.kinney@intel.com> Cc: Feng, Bob C <bob.c.feng@intel.com>; Gao, Liming <liming.gao@intel.com>; Sean Brogan <sean.brogan@microsoft.com>; Bret Barkelew <Bret.Barkelew@microsoft.com> Subject: RE: [edk2-devel] [Patch 1/1] BaseTools/Scripts: Add package dependency graphing tool Steve, Yes. I think a Module scoped view and a Platform scoped view would be good additions. Can you please enter new BZ feature requests for these and add a summary of the features you would like to see in each? Thanks, Mike From: Shi, Steven <steven.shi@intel.com<mailto:steven.shi@intel.com>> Sent: Monday, December 16, 2019 12:48 AM To: devel@edk2.groups.io<mailto:devel@edk2.groups.io>; Ni, Ray <ray.ni@intel.com<mailto:ray.ni@intel.com>>; Kinney, Michael D <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>> Cc: Feng, Bob C <bob.c.feng@intel.com<mailto:bob.c.feng@intel.com>>; Gao, Liming <liming.gao@intel.com<mailto:liming.gao@intel.com>>; Sean Brogan <sean.brogan@microsoft.com<mailto:sean.brogan@microsoft.com>>; Bret Barkelew <Bret.Barkelew@microsoft.com<mailto:Bret.Barkelew@microsoft.com>> Subject: RE: [edk2-devel] [Patch 1/1] BaseTools/Scripts: Add package dependency graphing tool Ray, I think you are asking for a module-level dependency tool. 😊 Mike, This tool is cool. Besides the package level, do you think we can go further to figure out a module level dependency tool? With module level dependency info, we can define more interesting and comprehensive Modularity Metrics to measure the edk2 architecture quality (Architectural Technical Debt). Thanks Steven Shi Intel\SSG\SFE\FIE Firmware Infrastructure > -----Original Message----- > From: devel@edk2.groups.io<mailto:devel@edk2.groups.io> <devel@edk2.groups.io<mailto:devel@edk2.groups.io>> On Behalf Of Ni, Ray > Sent: Monday, December 16, 2019 4:41 PM > To: devel@edk2.groups.io<mailto:devel@edk2.groups.io>; Ni, Ray <ray.ni@intel.com<mailto:ray.ni@intel.com>>; Kinney, Michael D > <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>> > Cc: Feng, Bob C <bob.c.feng@intel.com<mailto:bob.c.feng@intel.com>>; Gao, Liming > <liming.gao@intel.com<mailto:liming.gao@intel.com>>; Sean Brogan <sean.brogan@microsoft.com<mailto:sean.brogan@microsoft.com>>; Bret > Barkelew <Bret.Barkelew@microsoft.com<mailto:Bret.Barkelew@microsoft.com>> > Subject: Re: [edk2-devel] [Patch 1/1] BaseTools/Scripts: Add package > dependency graphing tool > > Mike, > This pkg dep tool can tell through weight when a module depends on a new > pkg. > But it cannot tell when a module depends on another > PCD/Protocol/Guid/Ppi/Library which belongs to an already-depended pkg. > > Failure to tell such information may make a bad module (that violates the > dependency rules) worse (wrongly depend on more interfaces). > > Do you think that the tool can be enhanced in future to detect the case? > > Thanks, > Ray > > > -----Original Message----- > > From: devel@edk2.groups.io<mailto:devel@edk2.groups.io> <devel@edk2.groups.io<mailto:devel@edk2.groups.io>> On Behalf Of Ni, > Ray > > Sent: Monday, December 16, 2019 1:57 PM > > To: Kinney, Michael D <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>>; > devel@edk2.groups.io<mailto:devel@edk2.groups.io> > > Cc: Feng, Bob C <bob.c.feng@intel.com<mailto:bob.c.feng@intel.com>>; Gao, Liming > > <liming.gao@intel.com<mailto:liming.gao@intel.com>>; Sean Brogan <sean.brogan@microsoft.com<mailto:sean.brogan@microsoft.com>>; > Bret > > Barkelew <Bret.Barkelew@microsoft.com<mailto:Bret.Barkelew@microsoft.com>> > > Subject: Re: [edk2-devel] [Patch 1/1] BaseTools/Scripts: Add package > > dependency graphing tool > > > > Mike, > > 2 minor comments regarding the help string in below. > > > > > -----Original Message----- > > > From: Kinney, Michael D <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>> > > > Sent: Saturday, December 14, 2019 3:45 AM > > > To: devel@edk2.groups.io<mailto:devel@edk2.groups.io> > > > Cc: Ni, Ray <ray.ni@intel.com<mailto:ray.ni@intel.com>>; Feng, Bob C <bob.c.feng@intel.com<mailto:bob.c.feng@intel.com>>; > Gao, > > > Liming <liming.gao@intel.com<mailto:liming.gao@intel.com>>; Sean Brogan > > <sean.brogan@microsoft.com<mailto:sean.brogan@microsoft.com>>; > > > Bret Barkelew <Bret.Barkelew@microsoft.com<mailto:Bret.Barkelew@microsoft.com>> > > > Subject: [Patch 1/1] BaseTools/Scripts: Add package dependency > graphing > > > tool > > > > > > https://bugzilla.tianocore.org/show_bug.cgi?id=2161 > > > > > > Add python script that recursively scans a directory for EDK II > > > packages and generates GraphViz dot input that is used to render > > > a graph of package dependencies in SVG format. > > > > > > Detects following error/warning conditions: > > > * Ambiguous dependencies (multiple matches) > > > * Unresolved dependencies > > > * Circular dependencies > > > * Nested packages > > > > > > usage: PackageDependencyGraph [-h] [-w WORKSPACE] [-p > > PACKAGESPATH] > > > [-d DOTOUTPUTFILE] [-o SVGOUTPUTFILE] > > > [-g IGNOREDIRECTORY] [-k SKIPPACKAGE] > > > [-s] [-u] [-l] [-f] [-b] [-v] [-q] > > > [--debug [0-9]] > > > > > > Recursively scan a directory for EDK II packages and generate GraphViz > dot > > > input that is used to render a graph of package dependencies in SVG > > format. > > > Copyright (c) 2019, Intel Corporation. All rights reserved. > > > > > > optional arguments: > > > -h, --help show this help message and exit > > > -w WORKSPACE, --workspace WORKSPACE > > > Directory to recursively scan for EDK II packages. > > > Default is current directory. > > > -p PACKAGESPATH, --packages-path PACKAGESPATH > > > List of directories to recursively scan for EDK II > > > packages. > > > -d DOTOUTPUTFILE, --dot-output DOTOUTPUTFILE > > > DOT output filename. > > > -o OUTPUTFILE, --output OUTPUTFILE > > > SVG output filename. > > > -g IGNOREDIRECTORY, --ignore-directory IGNOREDIRECTORY > > > Name of directory to ignore. Option can be repeated > > > to ignore multiple directories. > > > > 1. Name of directory to ignore when scanning for EDKII Module INFs. > > > > > -k SKIPPACKAGE, --skip-package SKIPPACKAGE > > > Name of EDK II Package DEC file to skip. Option can > > > be repeated to skip multiple EDK II packages. > > > > 2. Name of EDK II Package DEC file (including .DEC suffix) to skip. Option > can > > Be repeated to skip multiple EDK II packages. > > > > > -s, --self-dependency > > > Include self links in dependency graph. Default is > > > disabled. > > > -u, --unresolved Include unresolved EDK II packages in dependency > > > graph. Default is disabled. > > > -l, --label Label links with the number of EDK II package > > > dependencies. Default is disabled. > > > -f, --full-paths Label package nodes with full path to EDK II > > > package. Default is disabled. > > > -b, --web-browser Display SVG output file in default web browser. > > > Default is disabled. > > > -v, --verbose Increase output messages > > > -q, --quiet Reduce output messages > > > --debug [0-9] Set debug level > > > > > > Cc: Ray Ni <ray.ni@intel.com<mailto:ray.ni@intel.com>> > > > Cc: Bob Feng <bob.c.feng@intel.com<mailto:bob.c.feng@intel.com>> > > > Cc: Liming Gao <liming.gao@intel.com<mailto:liming.gao@intel.com>> > > > Cc: Sean Brogan <sean.brogan@microsoft.com<mailto:sean.brogan@microsoft.com>> > > > Cc: Bret Barkelew <Bret.Barkelew@microsoft.com<mailto:Bret.Barkelew@microsoft.com>> > > > Signed-off-by: Michael D Kinney <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>> > > > --- > > > BaseTools/Scripts/PackageDependencyGraph.py | 296 > > > ++++++++++++++++++++ > > > 1 file changed, 296 insertions(+) > > > create mode 100644 BaseTools/Scripts/PackageDependencyGraph.py > > > > > > diff --git a/BaseTools/Scripts/PackageDependencyGraph.py > > > b/BaseTools/Scripts/PackageDependencyGraph.py > > > new file mode 100644 > > > index 0000000000..b3c8e41774 > > > --- /dev/null > > > +++ b/BaseTools/Scripts/PackageDependencyGraph.py > > > @@ -0,0 +1,296 @@ > > > +# @file > > > +# Recursively scan a directory for EDK II packages and generate > GraphViz > > > dot > > > +# input that is used to render a graph of package dependencies in SVG > > > format. > > > +# > > > +# Copyright (c) 2019, Intel Corporation. All rights reserved.<BR> > > > +# SPDX-License-Identifier: BSD-2-Clause-Patent > > > +# > > > +## > > > + > > > +import os > > > +import sys > > > +import argparse > > > +import subprocess > > > +import webbrowser > > > +import networkx as nx > > > +from edk2toollib.uefi.edk2.parsers.inf_parser import InfParser > > > + > > > +# > > > +# Globals for help information > > > +# > > > +__prog__ = 'PackageDependencyGraph' > > > +__copyright__ = 'Copyright (c) 2019, Intel Corporation. All rights > > reserved.' > > > +__description__ = '''Recursively scan a directory for EDK II packages and > > > +generate GraphViz dot input that is used to render a graph of package > > > +dependencies in SVG format.''' > > > + > > > +if __name__ == '__main__': > > > + > > > + # > > > + # Create command line argument parser object > > > + # > > > + parser = argparse.ArgumentParser (prog = __prog__, > > > + description = __description__ + '\n' + > __copyright__, > > > + conflict_handler = 'resolve') > > > + parser.add_argument ("-w", "--workspace", dest = 'Workspace', > default > > = > > > os.curdir, > > > + help = "Directory to recursively scan for EDK II packages. > > > Default is current directory.") > > > + parser.add_argument ("-p", "--packages-path", dest = 'PackagesPath', > > > default = None, > > > + help = "List of directories to recursively scan for EDK II > > > packages.") > > > + parser.add_argument ("-d", "--dot-output", dest = 'DotOutputFile', > > > + help = "DOT output filename.") > > > + parser.add_argument ("-o", "--output", dest = 'SvgOutputFile', > > > + help = "SVG output filename.") > > > + parser.add_argument ("-g", "--ignore-directory", dest = > > 'IgnoreDirectory', > > > action='append', default=[], > > > + help = "Name of directory to ignore. Option can be > repeated > > to > > > ignore multiple directories.") > > > + parser.add_argument ("-k", "--skip-package", dest = 'SkipPackage', > > > action='append', default=[], > > > + help = "Name of EDK II Package DEC file to skip. Option > can > > be > > > repeated to skip multiple EDK II packages.") > > > + parser.add_argument ("-s", "--self-dependency", dest = > > > 'SelfDependency', action = "store_true", default = False, > > > + help = "Include self links in dependency graph. Default is > > > disabled.") > > > + parser.add_argument ("-u", "--unresolved", dest = 'Unresolved', > action > > = > > > "store_true", default=False, > > > + help = "Include unresolved EDK II packages in dependency > > > graph. Default is disabled.") > > > + parser.add_argument ("-l", "--label", dest = 'Label', action = > > "store_true", > > > default=False, > > > + help = "Label links with the number of EDK II package > > > dependencies. Default is disabled.") > > > + parser.add_argument ("-f", "--full-paths", dest = 'FullPaths', action = > > > "store_true", default=False, > > > + help = "Label package nodes with full path to EDK II > package. > > > Default is disabled.") > > > + parser.add_argument ("-b", "--web-browser", dest = 'WebBrowser', > > > action = "store_true", default=False, > > > + help = "Display SVG output file in default web browser. > > Default > > > is disabled.") > > > + parser.add_argument ("-v", "--verbose", dest = 'Verbose', action = > > > "store_true", > > > + help = "Increase output messages") > > > + parser.add_argument ("-q", "--quiet", dest = 'Quiet', action = > > > "store_true", > > > + help = "Reduce output messages") > > > + parser.add_argument ("--debug", dest = 'Debug', type = int, metavar = > > > '[0-9]', choices = range (0, 10), default = 0, > > > + help = "Set debug level") > > > + > > > + # > > > + # Parse command line arguments > > > + # > > > + args = parser.parse_args () > > > + > > > + # > > > + # Find all EDK II package DEC files > > > + # > > > + Components = {} > > > + SearchPaths = [args.Workspace] > > > + if args.PackagesPath: > > > + SearchPaths += args.PackagesPath.split(os.pathsep) > > > + SearchPaths = [os.path.realpath(x) for x in SearchPaths] > > > + for SearchPath in SearchPaths: > > > + for root, dirs, files in os.walk (SearchPath): > > > + for name in files: > > > + FilePath = os.path.join (root, name) > > > + if > > > set(FilePath.split(os.sep)).intersection(set(args.IgnoreDirectory)) != set(): > > > + if args.Verbose: > > > + print ('IGNORE:' + FilePath) > > > + continue > > > + if os.path.splitext(FilePath)[1].lower() in ['.dec']: > > > + DecFile = os.path.realpath (FilePath) > > > + if os.path.split(DecFile)[1] in args.SkipPackage: > > > + if args.Verbose: > > > + print ('SKIP:' + DecFile) > > > + continue > > > + if DecFile not in Components: > > > + if args.Verbose: > > > + print ('PACKAGE:' + DecFile) > > > + Components[DecFile] = {} > > > + > > > + # > > > + # Find EDK II component INF files in each EDK II package > > > + # > > > + PackageLabels = {} > > > + UnresolvedPackages = [] > > > + AmbiguousDependencies = [] > > > + for DecFile in Components: > > > + DecPath = os.path.split (DecFile)[0] > > > + if DecFile not in PackageLabels: > > > + PackageLabels[DecFile] = (DecFile.replace(os.path.sep,'\\n'), > 'white') > > > + if not args.FullPaths: > > > + PackagePath = os.path.relpath(DecFile, > os.path.split(DecPath)[0]) > > > + PackageLabels[DecFile] = > (PackagePath.replace(os.path.sep,'\\n'), > > > 'white') > > > + for root, dirs, files in os.walk (DecPath): > > > + for name in files: > > > + FilePath = os.path.join(root, name) > > > + if > > > set(FilePath.split(os.sep)).intersection(set(args.IgnoreDirectory)) != set(): > > > + if args.Verbose: > > > + print ('IGNORE:' + FilePath) > > > + continue > > > + if os.path.splitext(FilePath)[1].lower() in ['.inf']: > > > + InfFile = os.path.realpath (FilePath) > > > + Inf = InfParser () > > > + Inf.ParseFile (InfFile) > > > + DependentPackages = [] > > > + for Dependency in Inf.PackagesUsed: > > > + Dependency = os.path.normpath(Dependency) > > > + if os.path.split(Dependency)[1] in args.SkipPackage: > > > + if args.Verbose: > > > + print ('SKIP:' + Dependency) > > > + continue > > > + Found = False > > > + for SearchPath in SearchPaths: > > > + PackagePath = os.path.realpath(os.path.join(SearchPath, > > > Dependency)) > > > + if os.path.exists(PackagePath): > > > + DependentPackages.append(PackagePath) > > > + if not args.FullPaths: > > > + PackageLabels[PackagePath] = > > > (Dependency.replace(os.path.sep,'\\n'), 'white') > > > + Found = True > > > + break > > > + if not Found: > > > + Count = 0 > > > + Match = '' > > > + for DecFile2 in Components: > > > + if DecFile2.endswith(Dependency): > > > + if Count == 0: > > > + Match = DecFile2 > > > + Count = Count + 1 > > > + if Count > 1: > > > + AmbiguousDependencies.append (Dependency) > > > + if Count == 1: > > > + DependentPackages.append(Match) > > > + if not args.FullPaths: > > > + PackageLabels[Match] = > > > (Dependency.replace(os.path.sep,'\\n'), 'white') > > > + Found = True > > > + if not Found and args.Unresolved: > > > + if args.Verbose: > > > + print ('WARNING: Dependent package not found ' + > > > Dependency) > > > + DependentPackages.append(Dependency) > > > + UnresolvedPackages.append(Dependency) > > > + PackageLabels[Dependency] = > > > (Dependency.replace(os.path.sep,'\\n'), 'white') > > > + Components[DecFile][InfFile] = DependentPackages > > > + if AmbiguousDependencies: > > > + for Dependency in set(AmbiguousDependencies): > > > + print ('ERROR: MULTIPLE: ' + Dependency) > > > + print ('Use --packages-path to provide search priority.') > > > + sys.exit(1) > > > + > > > + # > > > + # Generate GraphViz dot input file contents. > > > + # Use networkx to detect circular dependencies. > > > + # > > > + MaxWeight = 0 > > > + MaxWeightDecFile = list(Components.keys())[0] > > > + Graph = nx.DiGraph() > > > + Edges = [] > > > + for DecFile in Components: > > > + if args.Verbose: > > > + print ('PACKAGE DEPENDENCIES:' + DecFile) > > > + AllDependencies = [] > > > + Dependencies = set() > > > + for InfFile in Components[DecFile]: > > > + AllDependencies += Components[DecFile][InfFile] > > > + Dependencies = > Dependencies.union(Components[DecFile][InfFile]) > > > + if not args.SelfDependency: > > > + Dependencies = Dependencies.difference(set([DecFile])) > > > + for Dependency in Dependencies: > > > + Weight = AllDependencies.count(Dependency) > > > + if Weight > MaxWeight: > > > + MaxWeight = Weight > > > + MaxWeightDecFile = DecFile > > > + if args.Verbose: > > > + print (' DEPENDENCY: Weight(' + str(Weight) + ') ' + > Dependency) > > > + Edges.append(' "{Package}" -> "{Dependency}" [label = > > > "{Weight}"];'.format( > > > + Package = DecFile, > > > + Dependency = Dependency, > > > + Weight = str(Weight) if args.Label else '' > > > + )) > > > + Graph.add_edge(DecFile, Dependency) > > > + Edges.sort() > > > + > > > + # > > > + # Set fill color to yellow if a packages is part of a circular dependency > > > + # > > > + for Node in set([x for y in list(nx.simple_cycles(Graph)) for x in y]): > > > + PackageLabels[Node] = (PackageLabels[Node][0], 'yellow') > > > + print ('ERROR: CIRCULAR: ' + Node) > > > + > > > + # > > > + # Set fill color to pink if a package that is nested inside another > package. > > > + # Set fill color to orange if a package is nested inside another package > > and > > > + # is part of a circular dependency. > > > + # > > > + for DecFile in Components: > > > + DecPath = os.path.split (DecFile)[0] > > > + for DecFile2 in Components: > > > + if DecFile2 == DecFile: > > > + continue > > > + if os.path.commonpath ([DecPath, DecFile2]) != DecPath: > > > + continue > > > + if len(DecFile2) < len(DecPath): > > > + DecFile2 = DecFile > > > + print ('ERROR: NESTED: ' + DecFile2) > > > + if PackageLabels[DecFile2][1] == 'yellow': > > > + PackageLabels[DecFile2] = (PackageLabels[DecFile2][0], > 'orange') > > > + else: > > > + PackageLabels[DecFile2] = (PackageLabels[DecFile2][0], 'pink') > > > + > > > + # > > > + # Set fill color to red for nodes that are unresolved > > > + # > > > + for Node in set(UnresolvedPackages): > > > + PackageLabels[Node] = (PackageLabels[Node][0], 'red') > > > + print ('ERROR: UNRESOLVED: ' + Node) > > > + > > > + # > > > + # Add node statements to set node label and fill color > > > + # > > > + Nodes = [] > > > + for Package in PackageLabels: > > > + Nodes.append(' "{Package}" > > [label="{Label}",fillcolor={Color}];'.format( > > > + Package = Package, > > > + Label = PackageLabels[Package][0], > > > + Color = PackageLabels[Package][1] > > > + )) > > > + Nodes.sort() > > > + > > > + # > > > + # Generate dot file from Nodes and Edges and add a Legend at top of > > > graph > > > + # > > > + Dot = [] > > > + Dot.append('digraph {') > > > + Dot.append(' rankdir=BT;') > > > + Dot.append(' node [shape=Mrecord,style=filled];') > > > + Dot.append('') > > > + Dot = Dot + Nodes > > > + Dot.append('') > > > + Dot = Dot + Edges > > > + Dot.append('') > > > + Dot.append(' subgraph legend {') > > > + Dot.append(' rank=sink;') > > > + Dot.append(' Unresolved [label="Unresolved Dependency", > > > fillcolor=red];') > > > + Dot.append(' Circular [label="Circular Dependency", > > > fillcolor=yellow];') > > > + Dot.append(' Nested [label="Nested Package", > > > fillcolor=pink];') > > > + Dot.append(' NestedCircular [label="Nested Package with Circular > > > Dependency",fillcolor=orange];') > > > + Dot.append(' }') > > > + if MaxWeightDecFile: > > > + Dot.append(' Unresolved->"' + MaxWeightDecFile + '" [style=invis];') > > > + Dot.append('}') > > > + > > > + if args.DotOutputFile: > > > + # > > > + # Write GraphViz dot file contents to DotOutputFile > > > + # > > > + with open(os.path.realpath(args.DotOutputFile), 'w') as File: > > > + File.write('\n'.join(Dot)) > > > + if args.SvgOutputFile: > > > + # > > > + # Use GraphViz 'dot' command to generate SVG output file > > > + # > > > + args.SvgOutputFile = os.path.realpath(args.SvgOutputFile) > > > + try: > > > + Process = subprocess.Popen('dot -Tsvg', > > > + stdin=subprocess.PIPE, > > > + stdout=open(args.SvgOutputFile, 'w'), > > > + stderr=subprocess.PIPE, > > > + shell=True > > > + ) > > > + Process.stdin.write ('\n'.join(Dot).encode()) > > > + Process.communicate() > > > + if Process.returncode != 0: > > > + print ("ERROR: Can not run GraphViz 'dot' command. Check > install > > > and path.") > > > + sys.exit(Process.returncode) > > > + except: > > > + print ("ERROR: Can not run GraphViz 'dot' command. Check > install > > and > > > path.") > > > + sys.exit(1) > > > + # > > > + # Display SVG file in default web browser > > > + # > > > + if args.WebBrowser: > > > + webbrowser.open(args.SvgOutputFile) > > > -- > > > 2.21.0.windows.1 > > > > > > > > > [-- Attachment #2: Type: text/html, Size: 80142 bytes --] ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [edk2-devel] [Patch 1/1] BaseTools/Scripts: Add package dependency graphing tool 2019-12-16 8:40 ` [edk2-devel] " Ni, Ray 2019-12-16 8:48 ` Steven Shi @ 2019-12-16 17:14 ` Michael D Kinney 2019-12-17 2:53 ` Ni, Ray 1 sibling, 1 reply; 12+ messages in thread From: Michael D Kinney @ 2019-12-16 17:14 UTC (permalink / raw) To: Ni, Ray, devel@edk2.groups.io, Kinney, Michael D Cc: Feng, Bob C, Gao, Liming, Sean Brogan, Bret Barkelew Ray, Yes. More features can be added. Can you add to the BZ some examples of the output you would like to see. https://bugzilla.tianocore.org/show_bug.cgi?id=2161 Thanks, Mike > -----Original Message----- > From: Ni, Ray <ray.ni@intel.com> > Sent: Monday, December 16, 2019 12:41 AM > To: devel@edk2.groups.io; Ni, Ray <ray.ni@intel.com>; > Kinney, Michael D <michael.d.kinney@intel.com> > Cc: Feng, Bob C <bob.c.feng@intel.com>; Gao, Liming > <liming.gao@intel.com>; Sean Brogan > <sean.brogan@microsoft.com>; Bret Barkelew > <Bret.Barkelew@microsoft.com> > Subject: RE: [edk2-devel] [Patch 1/1] > BaseTools/Scripts: Add package dependency graphing tool > > Mike, > This pkg dep tool can tell through weight when a module > depends on a new pkg. > But it cannot tell when a module depends on another > PCD/Protocol/Guid/Ppi/Library which belongs to an > already-depended pkg. > > Failure to tell such information may make a bad module > (that violates the dependency rules) worse (wrongly > depend on more interfaces). > > Do you think that the tool can be enhanced in future to > detect the case? > > Thanks, > Ray > > > -----Original Message----- > > From: devel@edk2.groups.io <devel@edk2.groups.io> On > Behalf Of Ni, Ray > > Sent: Monday, December 16, 2019 1:57 PM > > To: Kinney, Michael D <michael.d.kinney@intel.com>; > devel@edk2.groups.io > > Cc: Feng, Bob C <bob.c.feng@intel.com>; Gao, Liming > > <liming.gao@intel.com>; Sean Brogan > <sean.brogan@microsoft.com>; Bret > > Barkelew <Bret.Barkelew@microsoft.com> > > Subject: Re: [edk2-devel] [Patch 1/1] > BaseTools/Scripts: Add package > > dependency graphing tool > > > > Mike, > > 2 minor comments regarding the help string in below. > > > > > -----Original Message----- > > > From: Kinney, Michael D > <michael.d.kinney@intel.com> > > > Sent: Saturday, December 14, 2019 3:45 AM > > > To: devel@edk2.groups.io > > > Cc: Ni, Ray <ray.ni@intel.com>; Feng, Bob C > <bob.c.feng@intel.com>; Gao, > > > Liming <liming.gao@intel.com>; Sean Brogan > > <sean.brogan@microsoft.com>; > > > Bret Barkelew <Bret.Barkelew@microsoft.com> > > > Subject: [Patch 1/1] BaseTools/Scripts: Add package > dependency graphing > > > tool > > > > > > https://bugzilla.tianocore.org/show_bug.cgi?id=2161 > > > > > > Add python script that recursively scans a > directory for EDK II > > > packages and generates GraphViz dot input that is > used to render > > > a graph of package dependencies in SVG format. > > > > > > Detects following error/warning conditions: > > > * Ambiguous dependencies (multiple matches) > > > * Unresolved dependencies > > > * Circular dependencies > > > * Nested packages > > > > > > usage: PackageDependencyGraph [-h] [-w WORKSPACE] > [-p > > PACKAGESPATH] > > > [-d DOTOUTPUTFILE] [- > o SVGOUTPUTFILE] > > > [-g IGNOREDIRECTORY] > [-k SKIPPACKAGE] > > > [-s] [-u] [-l] [-f] > [-b] [-v] [-q] > > > [--debug [0-9]] > > > > > > Recursively scan a directory for EDK II packages > and generate GraphViz dot > > > input that is used to render a graph of package > dependencies in SVG > > format. > > > Copyright (c) 2019, Intel Corporation. All rights > reserved. > > > > > > optional arguments: > > > -h, --help show this help message and > exit > > > -w WORKSPACE, --workspace WORKSPACE > > > Directory to recursively > scan for EDK II packages. > > > Default is current > directory. > > > -p PACKAGESPATH, --packages-path PACKAGESPATH > > > List of directories to > recursively scan for EDK II > > > packages. > > > -d DOTOUTPUTFILE, --dot-output DOTOUTPUTFILE > > > DOT output filename. > > > -o OUTPUTFILE, --output OUTPUTFILE > > > SVG output filename. > > > -g IGNOREDIRECTORY, --ignore-directory > IGNOREDIRECTORY > > > Name of directory to > ignore. Option can be repeated > > > to ignore multiple > directories. > > > > 1. Name of directory to ignore when scanning for > EDKII Module INFs. > > > > > -k SKIPPACKAGE, --skip-package SKIPPACKAGE > > > Name of EDK II Package DEC > file to skip. Option can > > > be repeated to skip > multiple EDK II packages. > > > > 2. Name of EDK II Package DEC file (including .DEC > suffix) to skip. Option can > > Be repeated to skip multiple EDK II packages. > > > > > -s, --self-dependency > > > Include self links in > dependency graph. Default is > > > disabled. > > > -u, --unresolved Include unresolved EDK II > packages in dependency > > > graph. Default is disabled. > > > -l, --label Label links with the number > of EDK II package > > > dependencies. Default is > disabled. > > > -f, --full-paths Label package nodes with > full path to EDK II > > > package. Default is > disabled. > > > -b, --web-browser Display SVG output file in > default web browser. > > > Default is disabled. > > > -v, --verbose Increase output messages > > > -q, --quiet Reduce output messages > > > --debug [0-9] Set debug level > > > > > > Cc: Ray Ni <ray.ni@intel.com> > > > Cc: Bob Feng <bob.c.feng@intel.com> > > > Cc: Liming Gao <liming.gao@intel.com> > > > Cc: Sean Brogan <sean.brogan@microsoft.com> > > > Cc: Bret Barkelew <Bret.Barkelew@microsoft.com> > > > Signed-off-by: Michael D Kinney > <michael.d.kinney@intel.com> > > > --- > > > BaseTools/Scripts/PackageDependencyGraph.py | 296 > > > ++++++++++++++++++++ > > > 1 file changed, 296 insertions(+) > > > create mode 100644 > BaseTools/Scripts/PackageDependencyGraph.py > > > > > > diff --git > a/BaseTools/Scripts/PackageDependencyGraph.py > > > b/BaseTools/Scripts/PackageDependencyGraph.py > > > new file mode 100644 > > > index 0000000000..b3c8e41774 > > > --- /dev/null > > > +++ b/BaseTools/Scripts/PackageDependencyGraph.py > > > @@ -0,0 +1,296 @@ > > > +# @file > > > +# Recursively scan a directory for EDK II packages > and generate GraphViz > > > dot > > > +# input that is used to render a graph of package > dependencies in SVG > > > format. > > > +# > > > +# Copyright (c) 2019, Intel Corporation. All > rights reserved.<BR> > > > +# SPDX-License-Identifier: BSD-2-Clause-Patent > > > +# > > > +## > > > + > > > +import os > > > +import sys > > > +import argparse > > > +import subprocess > > > +import webbrowser > > > +import networkx as nx > > > +from edk2toollib.uefi.edk2.parsers.inf_parser > import InfParser > > > + > > > +# > > > +# Globals for help information > > > +# > > > +__prog__ = 'PackageDependencyGraph' > > > +__copyright__ = 'Copyright (c) 2019, Intel > Corporation. All rights > > reserved.' > > > +__description__ = '''Recursively scan a directory > for EDK II packages and > > > +generate GraphViz dot input that is used to render > a graph of package > > > +dependencies in SVG format.''' > > > + > > > +if __name__ == '__main__': > > > + > > > + # > > > + # Create command line argument parser object > > > + # > > > + parser = argparse.ArgumentParser (prog = > __prog__, > > > + description > = __description__ + '\n' + __copyright__, > > > + > conflict_handler = 'resolve') > > > + parser.add_argument ("-w", "--workspace", dest > = 'Workspace', default > > = > > > os.curdir, > > > + help = "Directory to > recursively scan for EDK II packages. > > > Default is current directory.") > > > + parser.add_argument ("-p", "--packages-path", > dest = 'PackagesPath', > > > default = None, > > > + help = "List of > directories to recursively scan for EDK II > > > packages.") > > > + parser.add_argument ("-d", "--dot-output", > dest = 'DotOutputFile', > > > + help = "DOT output > filename.") > > > + parser.add_argument ("-o", "--output", dest = > 'SvgOutputFile', > > > + help = "SVG output > filename.") > > > + parser.add_argument ("-g", "--ignore- > directory", dest = > > 'IgnoreDirectory', > > > action='append', default=[], > > > + help = "Name of directory > to ignore. Option can be repeated > > to > > > ignore multiple directories.") > > > + parser.add_argument ("-k", "--skip-package", > dest = 'SkipPackage', > > > action='append', default=[], > > > + help = "Name of EDK II > Package DEC file to skip. Option can > > be > > > repeated to skip multiple EDK II packages.") > > > + parser.add_argument ("-s", "--self- > dependency", dest = > > > 'SelfDependency', action = "store_true", default = > False, > > > + help = "Include self > links in dependency graph. Default is > > > disabled.") > > > + parser.add_argument ("-u", "--unresolved", > dest = 'Unresolved', action > > = > > > "store_true", default=False, > > > + help = "Include > unresolved EDK II packages in dependency > > > graph. Default is disabled.") > > > + parser.add_argument ("-l", "--label", dest = > 'Label', action = > > "store_true", > > > default=False, > > > + help = "Label links with > the number of EDK II package > > > dependencies. Default is disabled.") > > > + parser.add_argument ("-f", "--full-paths", > dest = 'FullPaths', action = > > > "store_true", default=False, > > > + help = "Label package > nodes with full path to EDK II package. > > > Default is disabled.") > > > + parser.add_argument ("-b", "--web-browser", > dest = 'WebBrowser', > > > action = "store_true", default=False, > > > + help = "Display SVG > output file in default web browser. > > Default > > > is disabled.") > > > + parser.add_argument ("-v", "--verbose", dest = > 'Verbose', action = > > > "store_true", > > > + help = "Increase output > messages") > > > + parser.add_argument ("-q", "--quiet", dest = > 'Quiet', action = > > > "store_true", > > > + help = "Reduce output > messages") > > > + parser.add_argument ("--debug", dest = > 'Debug', type = int, metavar = > > > '[0-9]', choices = range (0, 10), default = 0, > > > + help = "Set debug level") > > > + > > > + # > > > + # Parse command line arguments > > > + # > > > + args = parser.parse_args () > > > + > > > + # > > > + # Find all EDK II package DEC files > > > + # > > > + Components = {} > > > + SearchPaths = [args.Workspace] > > > + if args.PackagesPath: > > > + SearchPaths += > args.PackagesPath.split(os.pathsep) > > > + SearchPaths = [os.path.realpath(x) for x in > SearchPaths] > > > + for SearchPath in SearchPaths: > > > + for root, dirs, files in os.walk > (SearchPath): > > > + for name in files: > > > + FilePath = os.path.join (root, > name) > > > + if > > > > set(FilePath.split(os.sep)).intersection(set(args.Ignor > eDirectory)) != set(): > > > + if args.Verbose: > > > + print ('IGNORE:' + > FilePath) > > > + continue > > > + if > os.path.splitext(FilePath)[1].lower() in ['.dec']: > > > + DecFile = os.path.realpath > (FilePath) > > > + if os.path.split(DecFile)[1] > in args.SkipPackage: > > > + if args.Verbose: > > > + print ('SKIP:' + > DecFile) > > > + continue > > > + if DecFile not in Components: > > > + if args.Verbose: > > > + print ('PACKAGE:' + > DecFile) > > > + Components[DecFile] = {} > > > + > > > + # > > > + # Find EDK II component INF files in each EDK > II package > > > + # > > > + PackageLabels = {} > > > + UnresolvedPackages = [] > > > + AmbiguousDependencies = [] > > > + for DecFile in Components: > > > + DecPath = os.path.split (DecFile)[0] > > > + if DecFile not in PackageLabels: > > > + PackageLabels[DecFile] = > (DecFile.replace(os.path.sep,'\\n'), 'white') > > > + if not args.FullPaths: > > > + PackagePath = > os.path.relpath(DecFile, os.path.split(DecPath)[0]) > > > + PackageLabels[DecFile] = > (PackagePath.replace(os.path.sep,'\\n'), > > > 'white') > > > + for root, dirs, files in os.walk > (DecPath): > > > + for name in files: > > > + FilePath = os.path.join(root, > name) > > > + if > > > > set(FilePath.split(os.sep)).intersection(set(args.Ignor > eDirectory)) != set(): > > > + if args.Verbose: > > > + print ('IGNORE:' + > FilePath) > > > + continue > > > + if > os.path.splitext(FilePath)[1].lower() in ['.inf']: > > > + InfFile = os.path.realpath > (FilePath) > > > + Inf = InfParser () > > > + Inf.ParseFile (InfFile) > > > + DependentPackages = [] > > > + for Dependency in > Inf.PackagesUsed: > > > + Dependency = > os.path.normpath(Dependency) > > > + if > os.path.split(Dependency)[1] in args.SkipPackage: > > > + if args.Verbose: > > > + print ('SKIP:' + > Dependency) > > > + continue > > > + Found = False > > > + for SearchPath in > SearchPaths: > > > + PackagePath = > os.path.realpath(os.path.join(SearchPath, > > > Dependency)) > > > + if > os.path.exists(PackagePath): > > > + > DependentPackages.append(PackagePath) > > > + if not > args.FullPaths: > > > + > PackageLabels[PackagePath] = > > > (Dependency.replace(os.path.sep,'\\n'), 'white') > > > + Found = True > > > + break > > > + if not Found: > > > + Count = 0 > > > + Match = '' > > > + for DecFile2 in > Components: > > > + if > DecFile2.endswith(Dependency): > > > + if Count == 0: > > > + Match = > DecFile2 > > > + Count = Count > + 1 > > > + if Count > 1: > > > + > AmbiguousDependencies.append (Dependency) > > > + if Count == 1: > > > + > DependentPackages.append(Match) > > > + if not > args.FullPaths: > > > + > PackageLabels[Match] = > > > (Dependency.replace(os.path.sep,'\\n'), 'white') > > > + Found = True > > > + if not Found and > args.Unresolved: > > > + if args.Verbose: > > > + print ('WARNING: > Dependent package not found ' + > > > Dependency) > > > + > DependentPackages.append(Dependency) > > > + > UnresolvedPackages.append(Dependency) > > > + > PackageLabels[Dependency] = > > > (Dependency.replace(os.path.sep,'\\n'), 'white') > > > + Components[DecFile][InfFile] = > DependentPackages > > > + if AmbiguousDependencies: > > > + for Dependency in > set(AmbiguousDependencies): > > > + print ('ERROR: MULTIPLE: ' + > Dependency) > > > + print ('Use --packages-path to provide > search priority.') > > > + sys.exit(1) > > > + > > > + # > > > + # Generate GraphViz dot input file contents. > > > + # Use networkx to detect circular > dependencies. > > > + # > > > + MaxWeight = 0 > > > + MaxWeightDecFile = list(Components.keys())[0] > > > + Graph = nx.DiGraph() > > > + Edges = [] > > > + for DecFile in Components: > > > + if args.Verbose: > > > + print ('PACKAGE DEPENDENCIES:' + > DecFile) > > > + AllDependencies = [] > > > + Dependencies = set() > > > + for InfFile in Components[DecFile]: > > > + AllDependencies += > Components[DecFile][InfFile] > > > + Dependencies = > Dependencies.union(Components[DecFile][InfFile]) > > > + if not args.SelfDependency: > > > + Dependencies = > Dependencies.difference(set([DecFile])) > > > + for Dependency in Dependencies: > > > + Weight = > AllDependencies.count(Dependency) > > > + if Weight > MaxWeight: > > > + MaxWeight = Weight > > > + MaxWeightDecFile = DecFile > > > + if args.Verbose: > > > + print (' DEPENDENCY: Weight(' + > str(Weight) + ') ' + Dependency) > > > + Edges.append(' "{Package}" -> > "{Dependency}" [label = > > > "{Weight}"];'.format( > > > + Package = DecFile, > > > + Dependency = Dependency, > > > + Weight = str(Weight) if > args.Label else '' > > > + )) > > > + Graph.add_edge(DecFile, Dependency) > > > + Edges.sort() > > > + > > > + # > > > + # Set fill color to yellow if a packages is > part of a circular dependency > > > + # > > > + for Node in set([x for y in > list(nx.simple_cycles(Graph)) for x in y]): > > > + PackageLabels[Node] = > (PackageLabels[Node][0], 'yellow') > > > + print ('ERROR: CIRCULAR: ' + Node) > > > + > > > + # > > > + # Set fill color to pink if a package that is > nested inside another package. > > > + # Set fill color to orange if a package is > nested inside another package > > and > > > + # is part of a circular dependency. > > > + # > > > + for DecFile in Components: > > > + DecPath = os.path.split (DecFile)[0] > > > + for DecFile2 in Components: > > > + if DecFile2 == DecFile: > > > + continue > > > + if os.path.commonpath ([DecPath, > DecFile2]) != DecPath: > > > + continue > > > + if len(DecFile2) < len(DecPath): > > > + DecFile2 = DecFile > > > + print ('ERROR: NESTED: ' + > DecFile2) > > > + if PackageLabels[DecFile2][1] == > 'yellow': > > > + PackageLabels[DecFile2] = > (PackageLabels[DecFile2][0], 'orange') > > > + else: > > > + PackageLabels[DecFile2] = > (PackageLabels[DecFile2][0], 'pink') > > > + > > > + # > > > + # Set fill color to red for nodes that are > unresolved > > > + # > > > + for Node in set(UnresolvedPackages): > > > + PackageLabels[Node] = > (PackageLabels[Node][0], 'red') > > > + print ('ERROR: UNRESOLVED: ' + Node) > > > + > > > + # > > > + # Add node statements to set node label and > fill color > > > + # > > > + Nodes = [] > > > + for Package in PackageLabels: > > > + Nodes.append(' "{Package}" > > [label="{Label}",fillcolor={Color}];'.format( > > > + Package = Package, > > > + Label = PackageLabels[Package][0], > > > + Color = PackageLabels[Package][1] > > > + )) > > > + Nodes.sort() > > > + > > > + # > > > + # Generate dot file from Nodes and Edges and > add a Legend at top of > > > graph > > > + # > > > + Dot = [] > > > + Dot.append('digraph {') > > > + Dot.append(' rankdir=BT;') > > > + Dot.append(' node > [shape=Mrecord,style=filled];') > > > + Dot.append('') > > > + Dot = Dot + Nodes > > > + Dot.append('') > > > + Dot = Dot + Edges > > > + Dot.append('') > > > + Dot.append(' subgraph legend {') > > > + Dot.append(' rank=sink;') > > > + Dot.append(' Unresolved > [label="Unresolved Dependency", > > > fillcolor=red];') > > > + Dot.append(' Circular > [label="Circular Dependency", > > > fillcolor=yellow];') > > > + Dot.append(' Nested [label="Nested > Package", > > > fillcolor=pink];') > > > + Dot.append(' NestedCircular [label="Nested > Package with Circular > > > Dependency",fillcolor=orange];') > > > + Dot.append(' }') > > > + if MaxWeightDecFile: > > > + Dot.append(' Unresolved->"' + > MaxWeightDecFile + '" [style=invis];') > > > + Dot.append('}') > > > + > > > + if args.DotOutputFile: > > > + # > > > + # Write GraphViz dot file contents to > DotOutputFile > > > + # > > > + with > open(os.path.realpath(args.DotOutputFile), 'w') as > File: > > > + File.write('\n'.join(Dot)) > > > + if args.SvgOutputFile: > > > + # > > > + # Use GraphViz 'dot' command to generate > SVG output file > > > + # > > > + args.SvgOutputFile = > os.path.realpath(args.SvgOutputFile) > > > + try: > > > + Process = subprocess.Popen('dot - > Tsvg', > > > + stdin=subprocess.PIPE, > > > + stdout=open(args.SvgOutputFile, > 'w'), > > > + stderr=subprocess.PIPE, > > > + shell=True > > > + ) > > > + Process.stdin.write > ('\n'.join(Dot).encode()) > > > + Process.communicate() > > > + if Process.returncode != 0: > > > + print ("ERROR: Can not run > GraphViz 'dot' command. Check install > > > and path.") > > > + sys.exit(Process.returncode) > > > + except: > > > + print ("ERROR: Can not run GraphViz > 'dot' command. Check install > > and > > > path.") > > > + sys.exit(1) > > > + # > > > + # Display SVG file in default web browser > > > + # > > > + if args.WebBrowser: > > > + webbrowser.open(args.SvgOutputFile) > > > -- > > > 2.21.0.windows.1 > > > > > > ^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [edk2-devel] [Patch 1/1] BaseTools/Scripts: Add package dependency graphing tool 2019-12-16 17:14 ` Michael D Kinney @ 2019-12-17 2:53 ` Ni, Ray 0 siblings, 0 replies; 12+ messages in thread From: Ni, Ray @ 2019-12-17 2:53 UTC (permalink / raw) To: Kinney, Michael D, devel@edk2.groups.io Cc: Feng, Bob C, Gao, Liming, Sean Brogan, Bret Barkelew Done. > -----Original Message----- > From: Kinney, Michael D <michael.d.kinney@intel.com> > Sent: Tuesday, December 17, 2019 1:15 AM > To: Ni, Ray <ray.ni@intel.com>; devel@edk2.groups.io; Kinney, Michael D > <michael.d.kinney@intel.com> > Cc: Feng, Bob C <bob.c.feng@intel.com>; Gao, Liming > <liming.gao@intel.com>; Sean Brogan <sean.brogan@microsoft.com>; Bret > Barkelew <Bret.Barkelew@microsoft.com> > Subject: RE: [edk2-devel] [Patch 1/1] BaseTools/Scripts: Add package > dependency graphing tool > > Ray, > > Yes. More features can be added. Can you add to the BZ some > examples of the output you would like to see. > > https://bugzilla.tianocore.org/show_bug.cgi?id=2161 > > Thanks, > > Mike > > > -----Original Message----- > > From: Ni, Ray <ray.ni@intel.com> > > Sent: Monday, December 16, 2019 12:41 AM > > To: devel@edk2.groups.io; Ni, Ray <ray.ni@intel.com>; > > Kinney, Michael D <michael.d.kinney@intel.com> > > Cc: Feng, Bob C <bob.c.feng@intel.com>; Gao, Liming > > <liming.gao@intel.com>; Sean Brogan > > <sean.brogan@microsoft.com>; Bret Barkelew > > <Bret.Barkelew@microsoft.com> > > Subject: RE: [edk2-devel] [Patch 1/1] > > BaseTools/Scripts: Add package dependency graphing tool > > > > Mike, > > This pkg dep tool can tell through weight when a module > > depends on a new pkg. > > But it cannot tell when a module depends on another > > PCD/Protocol/Guid/Ppi/Library which belongs to an > > already-depended pkg. > > > > Failure to tell such information may make a bad module > > (that violates the dependency rules) worse (wrongly > > depend on more interfaces). > > > > Do you think that the tool can be enhanced in future to > > detect the case? > > > > Thanks, > > Ray > > > > > -----Original Message----- > > > From: devel@edk2.groups.io <devel@edk2.groups.io> On > > Behalf Of Ni, Ray > > > Sent: Monday, December 16, 2019 1:57 PM > > > To: Kinney, Michael D <michael.d.kinney@intel.com>; > > devel@edk2.groups.io > > > Cc: Feng, Bob C <bob.c.feng@intel.com>; Gao, Liming > > > <liming.gao@intel.com>; Sean Brogan > > <sean.brogan@microsoft.com>; Bret > > > Barkelew <Bret.Barkelew@microsoft.com> > > > Subject: Re: [edk2-devel] [Patch 1/1] > > BaseTools/Scripts: Add package > > > dependency graphing tool > > > > > > Mike, > > > 2 minor comments regarding the help string in below. > > > > > > > -----Original Message----- > > > > From: Kinney, Michael D > > <michael.d.kinney@intel.com> > > > > Sent: Saturday, December 14, 2019 3:45 AM > > > > To: devel@edk2.groups.io > > > > Cc: Ni, Ray <ray.ni@intel.com>; Feng, Bob C > > <bob.c.feng@intel.com>; Gao, > > > > Liming <liming.gao@intel.com>; Sean Brogan > > > <sean.brogan@microsoft.com>; > > > > Bret Barkelew <Bret.Barkelew@microsoft.com> > > > > Subject: [Patch 1/1] BaseTools/Scripts: Add package > > dependency graphing > > > > tool > > > > > > > > https://bugzilla.tianocore.org/show_bug.cgi?id=2161 > > > > > > > > Add python script that recursively scans a > > directory for EDK II > > > > packages and generates GraphViz dot input that is > > used to render > > > > a graph of package dependencies in SVG format. > > > > > > > > Detects following error/warning conditions: > > > > * Ambiguous dependencies (multiple matches) > > > > * Unresolved dependencies > > > > * Circular dependencies > > > > * Nested packages > > > > > > > > usage: PackageDependencyGraph [-h] [-w WORKSPACE] > > [-p > > > PACKAGESPATH] > > > > [-d DOTOUTPUTFILE] [- > > o SVGOUTPUTFILE] > > > > [-g IGNOREDIRECTORY] > > [-k SKIPPACKAGE] > > > > [-s] [-u] [-l] [-f] > > [-b] [-v] [-q] > > > > [--debug [0-9]] > > > > > > > > Recursively scan a directory for EDK II packages > > and generate GraphViz dot > > > > input that is used to render a graph of package > > dependencies in SVG > > > format. > > > > Copyright (c) 2019, Intel Corporation. All rights > > reserved. > > > > > > > > optional arguments: > > > > -h, --help show this help message and > > exit > > > > -w WORKSPACE, --workspace WORKSPACE > > > > Directory to recursively > > scan for EDK II packages. > > > > Default is current > > directory. > > > > -p PACKAGESPATH, --packages-path PACKAGESPATH > > > > List of directories to > > recursively scan for EDK II > > > > packages. > > > > -d DOTOUTPUTFILE, --dot-output DOTOUTPUTFILE > > > > DOT output filename. > > > > -o OUTPUTFILE, --output OUTPUTFILE > > > > SVG output filename. > > > > -g IGNOREDIRECTORY, --ignore-directory > > IGNOREDIRECTORY > > > > Name of directory to > > ignore. Option can be repeated > > > > to ignore multiple > > directories. > > > > > > 1. Name of directory to ignore when scanning for > > EDKII Module INFs. > > > > > > > -k SKIPPACKAGE, --skip-package SKIPPACKAGE > > > > Name of EDK II Package DEC > > file to skip. Option can > > > > be repeated to skip > > multiple EDK II packages. > > > > > > 2. Name of EDK II Package DEC file (including .DEC > > suffix) to skip. Option can > > > Be repeated to skip multiple EDK II packages. > > > > > > > -s, --self-dependency > > > > Include self links in > > dependency graph. Default is > > > > disabled. > > > > -u, --unresolved Include unresolved EDK II > > packages in dependency > > > > graph. Default is disabled. > > > > -l, --label Label links with the number > > of EDK II package > > > > dependencies. Default is > > disabled. > > > > -f, --full-paths Label package nodes with > > full path to EDK II > > > > package. Default is > > disabled. > > > > -b, --web-browser Display SVG output file in > > default web browser. > > > > Default is disabled. > > > > -v, --verbose Increase output messages > > > > -q, --quiet Reduce output messages > > > > --debug [0-9] Set debug level > > > > > > > > Cc: Ray Ni <ray.ni@intel.com> > > > > Cc: Bob Feng <bob.c.feng@intel.com> > > > > Cc: Liming Gao <liming.gao@intel.com> > > > > Cc: Sean Brogan <sean.brogan@microsoft.com> > > > > Cc: Bret Barkelew <Bret.Barkelew@microsoft.com> > > > > Signed-off-by: Michael D Kinney > > <michael.d.kinney@intel.com> > > > > --- > > > > BaseTools/Scripts/PackageDependencyGraph.py | 296 > > > > ++++++++++++++++++++ > > > > 1 file changed, 296 insertions(+) > > > > create mode 100644 > > BaseTools/Scripts/PackageDependencyGraph.py > > > > > > > > diff --git > > a/BaseTools/Scripts/PackageDependencyGraph.py > > > > b/BaseTools/Scripts/PackageDependencyGraph.py > > > > new file mode 100644 > > > > index 0000000000..b3c8e41774 > > > > --- /dev/null > > > > +++ b/BaseTools/Scripts/PackageDependencyGraph.py > > > > @@ -0,0 +1,296 @@ > > > > +# @file > > > > +# Recursively scan a directory for EDK II packages > > and generate GraphViz > > > > dot > > > > +# input that is used to render a graph of package > > dependencies in SVG > > > > format. > > > > +# > > > > +# Copyright (c) 2019, Intel Corporation. All > > rights reserved.<BR> > > > > +# SPDX-License-Identifier: BSD-2-Clause-Patent > > > > +# > > > > +## > > > > + > > > > +import os > > > > +import sys > > > > +import argparse > > > > +import subprocess > > > > +import webbrowser > > > > +import networkx as nx > > > > +from edk2toollib.uefi.edk2.parsers.inf_parser > > import InfParser > > > > + > > > > +# > > > > +# Globals for help information > > > > +# > > > > +__prog__ = 'PackageDependencyGraph' > > > > +__copyright__ = 'Copyright (c) 2019, Intel > > Corporation. All rights > > > reserved.' > > > > +__description__ = '''Recursively scan a directory > > for EDK II packages and > > > > +generate GraphViz dot input that is used to render > > a graph of package > > > > +dependencies in SVG format.''' > > > > + > > > > +if __name__ == '__main__': > > > > + > > > > + # > > > > + # Create command line argument parser object > > > > + # > > > > + parser = argparse.ArgumentParser (prog = > > __prog__, > > > > + description > > = __description__ + '\n' + __copyright__, > > > > + > > conflict_handler = 'resolve') > > > > + parser.add_argument ("-w", "--workspace", dest > > = 'Workspace', default > > > = > > > > os.curdir, > > > > + help = "Directory to > > recursively scan for EDK II packages. > > > > Default is current directory.") > > > > + parser.add_argument ("-p", "--packages-path", > > dest = 'PackagesPath', > > > > default = None, > > > > + help = "List of > > directories to recursively scan for EDK II > > > > packages.") > > > > + parser.add_argument ("-d", "--dot-output", > > dest = 'DotOutputFile', > > > > + help = "DOT output > > filename.") > > > > + parser.add_argument ("-o", "--output", dest = > > 'SvgOutputFile', > > > > + help = "SVG output > > filename.") > > > > + parser.add_argument ("-g", "--ignore- > > directory", dest = > > > 'IgnoreDirectory', > > > > action='append', default=[], > > > > + help = "Name of directory > > to ignore. Option can be repeated > > > to > > > > ignore multiple directories.") > > > > + parser.add_argument ("-k", "--skip-package", > > dest = 'SkipPackage', > > > > action='append', default=[], > > > > + help = "Name of EDK II > > Package DEC file to skip. Option can > > > be > > > > repeated to skip multiple EDK II packages.") > > > > + parser.add_argument ("-s", "--self- > > dependency", dest = > > > > 'SelfDependency', action = "store_true", default = > > False, > > > > + help = "Include self > > links in dependency graph. Default is > > > > disabled.") > > > > + parser.add_argument ("-u", "--unresolved", > > dest = 'Unresolved', action > > > = > > > > "store_true", default=False, > > > > + help = "Include > > unresolved EDK II packages in dependency > > > > graph. Default is disabled.") > > > > + parser.add_argument ("-l", "--label", dest = > > 'Label', action = > > > "store_true", > > > > default=False, > > > > + help = "Label links with > > the number of EDK II package > > > > dependencies. Default is disabled.") > > > > + parser.add_argument ("-f", "--full-paths", > > dest = 'FullPaths', action = > > > > "store_true", default=False, > > > > + help = "Label package > > nodes with full path to EDK II package. > > > > Default is disabled.") > > > > + parser.add_argument ("-b", "--web-browser", > > dest = 'WebBrowser', > > > > action = "store_true", default=False, > > > > + help = "Display SVG > > output file in default web browser. > > > Default > > > > is disabled.") > > > > + parser.add_argument ("-v", "--verbose", dest = > > 'Verbose', action = > > > > "store_true", > > > > + help = "Increase output > > messages") > > > > + parser.add_argument ("-q", "--quiet", dest = > > 'Quiet', action = > > > > "store_true", > > > > + help = "Reduce output > > messages") > > > > + parser.add_argument ("--debug", dest = > > 'Debug', type = int, metavar = > > > > '[0-9]', choices = range (0, 10), default = 0, > > > > + help = "Set debug level") > > > > + > > > > + # > > > > + # Parse command line arguments > > > > + # > > > > + args = parser.parse_args () > > > > + > > > > + # > > > > + # Find all EDK II package DEC files > > > > + # > > > > + Components = {} > > > > + SearchPaths = [args.Workspace] > > > > + if args.PackagesPath: > > > > + SearchPaths += > > args.PackagesPath.split(os.pathsep) > > > > + SearchPaths = [os.path.realpath(x) for x in > > SearchPaths] > > > > + for SearchPath in SearchPaths: > > > > + for root, dirs, files in os.walk > > (SearchPath): > > > > + for name in files: > > > > + FilePath = os.path.join (root, > > name) > > > > + if > > > > > > set(FilePath.split(os.sep)).intersection(set(args.Ignor > > eDirectory)) != set(): > > > > + if args.Verbose: > > > > + print ('IGNORE:' + > > FilePath) > > > > + continue > > > > + if > > os.path.splitext(FilePath)[1].lower() in ['.dec']: > > > > + DecFile = os.path.realpath > > (FilePath) > > > > + if os.path.split(DecFile)[1] > > in args.SkipPackage: > > > > + if args.Verbose: > > > > + print ('SKIP:' + > > DecFile) > > > > + continue > > > > + if DecFile not in Components: > > > > + if args.Verbose: > > > > + print ('PACKAGE:' + > > DecFile) > > > > + Components[DecFile] = {} > > > > + > > > > + # > > > > + # Find EDK II component INF files in each EDK > > II package > > > > + # > > > > + PackageLabels = {} > > > > + UnresolvedPackages = [] > > > > + AmbiguousDependencies = [] > > > > + for DecFile in Components: > > > > + DecPath = os.path.split (DecFile)[0] > > > > + if DecFile not in PackageLabels: > > > > + PackageLabels[DecFile] = > > (DecFile.replace(os.path.sep,'\\n'), 'white') > > > > + if not args.FullPaths: > > > > + PackagePath = > > os.path.relpath(DecFile, os.path.split(DecPath)[0]) > > > > + PackageLabels[DecFile] = > > (PackagePath.replace(os.path.sep,'\\n'), > > > > 'white') > > > > + for root, dirs, files in os.walk > > (DecPath): > > > > + for name in files: > > > > + FilePath = os.path.join(root, > > name) > > > > + if > > > > > > set(FilePath.split(os.sep)).intersection(set(args.Ignor > > eDirectory)) != set(): > > > > + if args.Verbose: > > > > + print ('IGNORE:' + > > FilePath) > > > > + continue > > > > + if > > os.path.splitext(FilePath)[1].lower() in ['.inf']: > > > > + InfFile = os.path.realpath > > (FilePath) > > > > + Inf = InfParser () > > > > + Inf.ParseFile (InfFile) > > > > + DependentPackages = [] > > > > + for Dependency in > > Inf.PackagesUsed: > > > > + Dependency = > > os.path.normpath(Dependency) > > > > + if > > os.path.split(Dependency)[1] in args.SkipPackage: > > > > + if args.Verbose: > > > > + print ('SKIP:' + > > Dependency) > > > > + continue > > > > + Found = False > > > > + for SearchPath in > > SearchPaths: > > > > + PackagePath = > > os.path.realpath(os.path.join(SearchPath, > > > > Dependency)) > > > > + if > > os.path.exists(PackagePath): > > > > + > > DependentPackages.append(PackagePath) > > > > + if not > > args.FullPaths: > > > > + > > PackageLabels[PackagePath] = > > > > (Dependency.replace(os.path.sep,'\\n'), 'white') > > > > + Found = True > > > > + break > > > > + if not Found: > > > > + Count = 0 > > > > + Match = '' > > > > + for DecFile2 in > > Components: > > > > + if > > DecFile2.endswith(Dependency): > > > > + if Count == 0: > > > > + Match = > > DecFile2 > > > > + Count = Count > > + 1 > > > > + if Count > 1: > > > > + > > AmbiguousDependencies.append (Dependency) > > > > + if Count == 1: > > > > + > > DependentPackages.append(Match) > > > > + if not > > args.FullPaths: > > > > + > > PackageLabels[Match] = > > > > (Dependency.replace(os.path.sep,'\\n'), 'white') > > > > + Found = True > > > > + if not Found and > > args.Unresolved: > > > > + if args.Verbose: > > > > + print ('WARNING: > > Dependent package not found ' + > > > > Dependency) > > > > + > > DependentPackages.append(Dependency) > > > > + > > UnresolvedPackages.append(Dependency) > > > > + > > PackageLabels[Dependency] = > > > > (Dependency.replace(os.path.sep,'\\n'), 'white') > > > > + Components[DecFile][InfFile] = > > DependentPackages > > > > + if AmbiguousDependencies: > > > > + for Dependency in > > set(AmbiguousDependencies): > > > > + print ('ERROR: MULTIPLE: ' + > > Dependency) > > > > + print ('Use --packages-path to provide > > search priority.') > > > > + sys.exit(1) > > > > + > > > > + # > > > > + # Generate GraphViz dot input file contents. > > > > + # Use networkx to detect circular > > dependencies. > > > > + # > > > > + MaxWeight = 0 > > > > + MaxWeightDecFile = list(Components.keys())[0] > > > > + Graph = nx.DiGraph() > > > > + Edges = [] > > > > + for DecFile in Components: > > > > + if args.Verbose: > > > > + print ('PACKAGE DEPENDENCIES:' + > > DecFile) > > > > + AllDependencies = [] > > > > + Dependencies = set() > > > > + for InfFile in Components[DecFile]: > > > > + AllDependencies += > > Components[DecFile][InfFile] > > > > + Dependencies = > > Dependencies.union(Components[DecFile][InfFile]) > > > > + if not args.SelfDependency: > > > > + Dependencies = > > Dependencies.difference(set([DecFile])) > > > > + for Dependency in Dependencies: > > > > + Weight = > > AllDependencies.count(Dependency) > > > > + if Weight > MaxWeight: > > > > + MaxWeight = Weight > > > > + MaxWeightDecFile = DecFile > > > > + if args.Verbose: > > > > + print (' DEPENDENCY: Weight(' + > > str(Weight) + ') ' + Dependency) > > > > + Edges.append(' "{Package}" -> > > "{Dependency}" [label = > > > > "{Weight}"];'.format( > > > > + Package = DecFile, > > > > + Dependency = Dependency, > > > > + Weight = str(Weight) if > > args.Label else '' > > > > + )) > > > > + Graph.add_edge(DecFile, Dependency) > > > > + Edges.sort() > > > > + > > > > + # > > > > + # Set fill color to yellow if a packages is > > part of a circular dependency > > > > + # > > > > + for Node in set([x for y in > > list(nx.simple_cycles(Graph)) for x in y]): > > > > + PackageLabels[Node] = > > (PackageLabels[Node][0], 'yellow') > > > > + print ('ERROR: CIRCULAR: ' + Node) > > > > + > > > > + # > > > > + # Set fill color to pink if a package that is > > nested inside another package. > > > > + # Set fill color to orange if a package is > > nested inside another package > > > and > > > > + # is part of a circular dependency. > > > > + # > > > > + for DecFile in Components: > > > > + DecPath = os.path.split (DecFile)[0] > > > > + for DecFile2 in Components: > > > > + if DecFile2 == DecFile: > > > > + continue > > > > + if os.path.commonpath ([DecPath, > > DecFile2]) != DecPath: > > > > + continue > > > > + if len(DecFile2) < len(DecPath): > > > > + DecFile2 = DecFile > > > > + print ('ERROR: NESTED: ' + > > DecFile2) > > > > + if PackageLabels[DecFile2][1] == > > 'yellow': > > > > + PackageLabels[DecFile2] = > > (PackageLabels[DecFile2][0], 'orange') > > > > + else: > > > > + PackageLabels[DecFile2] = > > (PackageLabels[DecFile2][0], 'pink') > > > > + > > > > + # > > > > + # Set fill color to red for nodes that are > > unresolved > > > > + # > > > > + for Node in set(UnresolvedPackages): > > > > + PackageLabels[Node] = > > (PackageLabels[Node][0], 'red') > > > > + print ('ERROR: UNRESOLVED: ' + Node) > > > > + > > > > + # > > > > + # Add node statements to set node label and > > fill color > > > > + # > > > > + Nodes = [] > > > > + for Package in PackageLabels: > > > > + Nodes.append(' "{Package}" > > > [label="{Label}",fillcolor={Color}];'.format( > > > > + Package = Package, > > > > + Label = PackageLabels[Package][0], > > > > + Color = PackageLabels[Package][1] > > > > + )) > > > > + Nodes.sort() > > > > + > > > > + # > > > > + # Generate dot file from Nodes and Edges and > > add a Legend at top of > > > > graph > > > > + # > > > > + Dot = [] > > > > + Dot.append('digraph {') > > > > + Dot.append(' rankdir=BT;') > > > > + Dot.append(' node > > [shape=Mrecord,style=filled];') > > > > + Dot.append('') > > > > + Dot = Dot + Nodes > > > > + Dot.append('') > > > > + Dot = Dot + Edges > > > > + Dot.append('') > > > > + Dot.append(' subgraph legend {') > > > > + Dot.append(' rank=sink;') > > > > + Dot.append(' Unresolved > > [label="Unresolved Dependency", > > > > fillcolor=red];') > > > > + Dot.append(' Circular > > [label="Circular Dependency", > > > > fillcolor=yellow];') > > > > + Dot.append(' Nested [label="Nested > > Package", > > > > fillcolor=pink];') > > > > + Dot.append(' NestedCircular [label="Nested > > Package with Circular > > > > Dependency",fillcolor=orange];') > > > > + Dot.append(' }') > > > > + if MaxWeightDecFile: > > > > + Dot.append(' Unresolved->"' + > > MaxWeightDecFile + '" [style=invis];') > > > > + Dot.append('}') > > > > + > > > > + if args.DotOutputFile: > > > > + # > > > > + # Write GraphViz dot file contents to > > DotOutputFile > > > > + # > > > > + with > > open(os.path.realpath(args.DotOutputFile), 'w') as > > File: > > > > + File.write('\n'.join(Dot)) > > > > + if args.SvgOutputFile: > > > > + # > > > > + # Use GraphViz 'dot' command to generate > > SVG output file > > > > + # > > > > + args.SvgOutputFile = > > os.path.realpath(args.SvgOutputFile) > > > > + try: > > > > + Process = subprocess.Popen('dot - > > Tsvg', > > > > + stdin=subprocess.PIPE, > > > > + stdout=open(args.SvgOutputFile, > > 'w'), > > > > + stderr=subprocess.PIPE, > > > > + shell=True > > > > + ) > > > > + Process.stdin.write > > ('\n'.join(Dot).encode()) > > > > + Process.communicate() > > > > + if Process.returncode != 0: > > > > + print ("ERROR: Can not run > > GraphViz 'dot' command. Check install > > > > and path.") > > > > + sys.exit(Process.returncode) > > > > + except: > > > > + print ("ERROR: Can not run GraphViz > > 'dot' command. Check install > > > and > > > > path.") > > > > + sys.exit(1) > > > > + # > > > > + # Display SVG file in default web browser > > > > + # > > > > + if args.WebBrowser: > > > > + webbrowser.open(args.SvgOutputFile) > > > > -- > > > > 2.21.0.windows.1 > > > > > > > > > ^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2019-12-19 1:11 UTC | newest]
Thread overview: 12+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2019-12-13 19:44 [Patch 0/1] BaseTools/Scripts: Add package dependency graphing tool Michael D Kinney
2019-12-13 19:44 ` [Patch 1/1] " Michael D Kinney
2019-12-16 5:56 ` Ni, Ray
[not found] ` <15E0C4623803B81C.31060@groups.io>
2019-12-16 8:40 ` [edk2-devel] " Ni, Ray
2019-12-16 8:48 ` Steven Shi
2019-12-16 17:16 ` Michael D Kinney
2019-12-16 19:45 ` Sean
2019-12-17 2:48 ` Ni, Ray
2019-12-17 16:10 ` Steven Shi
2019-12-19 1:11 ` Steven Shi
2019-12-16 17:14 ` Michael D Kinney
2019-12-17 2:53 ` Ni, Ray
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox