[Patch 0/1] BaseTools/Scripts: Add package dependency graphing tool

[Patch 0/1] BaseTools/Scripts: Add package dependency graphing tool

[Michael D Kinney]

https://bugzilla.tianocore.org/show_bug.cgi?id=2161

Add python script that recursively scans a directory for EDK II
packages and generates GraphViz dot input that is used to render
a graph of package dependencies in SVG format.

Detects following error/warning conditions:
* Ambiguous dependencies (multiple matches)
* Unresolved dependencies
* Circular dependencies
* Nested packages

Links to sample graphs

https://github.com/mdkinney/edk2/wiki/images/edk2.svg?sanitize=true
https://github.com/mdkinney/edk2/wiki/images/edk2-SkipPlatforms.svg?sanitize=true
https://github.com/mdkinney/edk2/wiki/images/edk2-SkipPlatformsAndArm.svg?sanitize=true
https://github.com/mdkinney/edk2/wiki/images/edk2-platforms.svg?sanitize=true
https://github.com/mdkinney/edk2/wiki/images/edk2-platforms-HideUnresolved.svg?sanitize=true
https://github.com/mdkinney/edk2/wiki/images/edk2-platforms-AddBinariesHideUnresolved.svg?sanitize=true
https://github.com/mdkinney/edk2/wiki/images/edk2-non-osi.svg?sanitize=true
https://github.com/mdkinney/edk2/wiki/images/FSP.svg?sanitize=true
https://github.com/mdkinney/edk2/wiki/images/all.svg?sanitize=true

This patch is available for review at the following branch:

https://github.com/mdkinney/edk2/tree/Bug_2161_PackageDependencyGraph

A proposed update to the EDK II Packages wiki page that adds details on 
package dependency rules that this tool can help to visualize and resolve
issues.

https://github.com/mdkinney/edk2/wiki/EDKII-Packages

A new wiki page with graphs from the current EDK II repos and instructions
on how to use this new script to generate the graphs.  Complex graphs can be
saved locally and viewed with an SVG viewer.

https://github.com/mdkinney/edk2/wiki/EDK-II-Package-Dependency-Graphs


Cc: Ray Ni <ray.ni@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Sean Brogan <sean.brogan@microsoft.com>
Cc: Bret Barkelew <Bret.Barkelew@microsoft.com>
Signed-off-by: Michael D Kinney <michael.d.kinney@intel.com>

Michael D Kinney (1):
  BaseTools/Scripts: Add package dependency graphing tool

 BaseTools/Scripts/PackageDependencyGraph.py | 296 ++++++++++++++++++++
 1 file changed, 296 insertions(+)
 create mode 100644 BaseTools/Scripts/PackageDependencyGraph.py

-- 
2.21.0.windows.1

[Patch 1/1] BaseTools/Scripts: Add package dependency graphing tool

[Michael D Kinney]

https://bugzilla.tianocore.org/show_bug.cgi?id=2161

Add python script that recursively scans a directory for EDK II
packages and generates GraphViz dot input that is used to render
a graph of package dependencies in SVG format.

Detects following error/warning conditions:
* Ambiguous dependencies (multiple matches)
* Unresolved dependencies
* Circular dependencies
* Nested packages

usage: PackageDependencyGraph [-h] [-w WORKSPACE] [-p PACKAGESPATH]
                              [-d DOTOUTPUTFILE] [-o SVGOUTPUTFILE]
                              [-g IGNOREDIRECTORY] [-k SKIPPACKAGE]
                              [-s] [-u] [-l] [-f] [-b] [-v] [-q]
                              [--debug [0-9]]

Recursively scan a directory for EDK II packages and generate GraphViz dot
input that is used to render a graph of package dependencies in SVG format.
Copyright (c) 2019, Intel Corporation. All rights reserved.

optional arguments:
  -h, --help            show this help message and exit
  -w WORKSPACE, --workspace WORKSPACE
                        Directory to recursively scan for EDK II packages.
                        Default is current directory.
  -p PACKAGESPATH, --packages-path PACKAGESPATH
                        List of directories to recursively scan for EDK II
                        packages.
  -d DOTOUTPUTFILE, --dot-output DOTOUTPUTFILE
                        DOT output filename.
  -o OUTPUTFILE, --output OUTPUTFILE
                        SVG output filename.
  -g IGNOREDIRECTORY, --ignore-directory IGNOREDIRECTORY
                        Name of directory to ignore. Option can be repeated
                        to ignore multiple directories.
  -k SKIPPACKAGE, --skip-package SKIPPACKAGE
                        Name of EDK II Package DEC file to skip. Option can
                        be repeated to skip multiple EDK II packages.
  -s, --self-dependency
                        Include self links in dependency graph. Default is
                        disabled.
  -u, --unresolved      Include unresolved EDK II packages in dependency
                        graph. Default is disabled.
  -l, --label           Label links with the number of EDK II package
                        dependencies. Default is disabled.
  -f, --full-paths      Label package nodes with full path to EDK II
                        package.  Default is disabled.
  -b, --web-browser     Display SVG output file in default web browser.
                        Default is disabled.
  -v, --verbose         Increase output messages
  -q, --quiet           Reduce output messages
  --debug [0-9]         Set debug level

Cc: Ray Ni <ray.ni@intel.com>
Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Sean Brogan <sean.brogan@microsoft.com>
Cc: Bret Barkelew <Bret.Barkelew@microsoft.com>
Signed-off-by: Michael D Kinney <michael.d.kinney@intel.com>
---
 BaseTools/Scripts/PackageDependencyGraph.py | 296 ++++++++++++++++++++
 1 file changed, 296 insertions(+)
 create mode 100644 BaseTools/Scripts/PackageDependencyGraph.py

diff --git a/BaseTools/Scripts/PackageDependencyGraph.py b/BaseTools/Scripts/PackageDependencyGraph.py
new file mode 100644
index 0000000000..b3c8e41774
--- /dev/null
+++ b/BaseTools/Scripts/PackageDependencyGraph.py
@@ -0,0 +1,296 @@
+# @file
+# Recursively scan a directory for EDK II packages and generate GraphViz dot
+# input that is used to render a graph of package dependencies in SVG format.
+#
+# Copyright (c) 2019, Intel Corporation. All rights reserved.<BR>
+# SPDX-License-Identifier: BSD-2-Clause-Patent
+#
+##
+
+import os
+import sys
+import argparse
+import subprocess
+import webbrowser
+import networkx as nx
+from edk2toollib.uefi.edk2.parsers.inf_parser import InfParser
+
+#
+# Globals for help information
+#
+__prog__        = 'PackageDependencyGraph'
+__copyright__   = 'Copyright (c) 2019, Intel Corporation. All rights reserved.'
+__description__ = '''Recursively scan a directory for EDK II packages and
+generate GraphViz dot input that is used to render a graph of package
+dependencies in SVG format.'''
+
+if __name__ == '__main__':
+
+    #
+    # Create command line argument parser object
+    #
+    parser = argparse.ArgumentParser (prog = __prog__,
+                                      description = __description__ + '\n' + __copyright__,
+                                      conflict_handler = 'resolve')
+    parser.add_argument ("-w", "--workspace", dest = 'Workspace', default = os.curdir,
+                         help = "Directory to recursively scan for EDK II packages.  Default is current directory.")
+    parser.add_argument ("-p", "--packages-path", dest = 'PackagesPath', default = None,
+                         help = "List of directories to recursively scan for EDK II packages.")
+    parser.add_argument ("-d", "--dot-output", dest = 'DotOutputFile',
+                         help = "DOT output filename.")
+    parser.add_argument ("-o", "--output", dest = 'SvgOutputFile',
+                         help = "SVG output filename.")
+    parser.add_argument ("-g", "--ignore-directory", dest = 'IgnoreDirectory', action='append', default=[],
+                         help = "Name of directory to ignore.  Option can be repeated to ignore multiple directories.")
+    parser.add_argument ("-k", "--skip-package", dest = 'SkipPackage', action='append', default=[],
+                         help = "Name of EDK II Package DEC file to skip.  Option can be repeated to skip multiple EDK II packages.")
+    parser.add_argument ("-s", "--self-dependency", dest = 'SelfDependency', action = "store_true", default = False,
+                         help = "Include self links in dependency graph.  Default is disabled.")
+    parser.add_argument ("-u", "--unresolved", dest = 'Unresolved', action = "store_true", default=False,
+                         help = "Include unresolved EDK II packages in dependency graph.  Default is disabled.")
+    parser.add_argument ("-l", "--label", dest = 'Label', action = "store_true", default=False,
+                         help = "Label links with the number of EDK II package dependencies.  Default is disabled.")
+    parser.add_argument ("-f", "--full-paths", dest = 'FullPaths', action = "store_true", default=False,
+                         help = "Label package nodes with full path to EDK II package.  Default is disabled.")
+    parser.add_argument ("-b", "--web-browser", dest = 'WebBrowser', action = "store_true", default=False,
+                         help = "Display SVG output file in default web browser.  Default is disabled.")
+    parser.add_argument ("-v", "--verbose", dest = 'Verbose', action = "store_true",
+                         help = "Increase output messages")
+    parser.add_argument ("-q", "--quiet", dest = 'Quiet', action = "store_true",
+                         help = "Reduce output messages")
+    parser.add_argument ("--debug", dest = 'Debug', type = int, metavar = '[0-9]', choices = range (0, 10), default = 0,
+                         help = "Set debug level")
+
+    #
+    # Parse command line arguments
+    #
+    args = parser.parse_args ()
+
+    #
+    # Find all EDK II package DEC files
+    #
+    Components = {}
+    SearchPaths = [args.Workspace]
+    if args.PackagesPath:
+        SearchPaths += args.PackagesPath.split(os.pathsep)
+    SearchPaths = [os.path.realpath(x) for x in SearchPaths]
+    for SearchPath in SearchPaths:
+        for root, dirs, files in os.walk (SearchPath):
+            for name in files:
+                FilePath = os.path.join (root, name)
+                if set(FilePath.split(os.sep)).intersection(set(args.IgnoreDirectory)) != set():
+                    if args.Verbose:
+                        print ('IGNORE:' + FilePath)
+                    continue
+                if os.path.splitext(FilePath)[1].lower() in ['.dec']:
+                    DecFile = os.path.realpath (FilePath)
+                    if os.path.split(DecFile)[1] in args.SkipPackage:
+                        if args.Verbose:
+                            print ('SKIP:' + DecFile)
+                        continue
+                    if DecFile not in Components:
+                        if args.Verbose:
+                            print ('PACKAGE:' + DecFile)
+                        Components[DecFile] = {}
+
+    #
+    # Find EDK II component INF files in each EDK II package
+    #
+    PackageLabels = {}
+    UnresolvedPackages = []
+    AmbiguousDependencies = []
+    for DecFile in Components:
+        DecPath = os.path.split (DecFile)[0]
+        if DecFile not in PackageLabels:
+            PackageLabels[DecFile] = (DecFile.replace(os.path.sep,'\\n'), 'white')
+            if not args.FullPaths:
+                PackagePath = os.path.relpath(DecFile, os.path.split(DecPath)[0])
+                PackageLabels[DecFile] = (PackagePath.replace(os.path.sep,'\\n'), 'white')
+        for root, dirs, files in os.walk (DecPath):
+            for name in files:
+                FilePath = os.path.join(root, name)
+                if set(FilePath.split(os.sep)).intersection(set(args.IgnoreDirectory)) != set():
+                    if args.Verbose:
+                        print ('IGNORE:' + FilePath)
+                    continue
+                if os.path.splitext(FilePath)[1].lower() in ['.inf']:
+                    InfFile = os.path.realpath (FilePath)
+                    Inf = InfParser ()
+                    Inf.ParseFile (InfFile)
+                    DependentPackages = []
+                    for Dependency in Inf.PackagesUsed:
+                        Dependency = os.path.normpath(Dependency)
+                        if os.path.split(Dependency)[1] in args.SkipPackage:
+                            if args.Verbose:
+                                print ('SKIP:' + Dependency)
+                            continue
+                        Found = False
+                        for SearchPath in SearchPaths:
+                            PackagePath = os.path.realpath(os.path.join(SearchPath, Dependency))
+                            if os.path.exists(PackagePath):
+                                DependentPackages.append(PackagePath)
+                                if not args.FullPaths:
+                                    PackageLabels[PackagePath] = (Dependency.replace(os.path.sep,'\\n'), 'white')
+                                Found = True
+                                break
+                        if not Found:
+                            Count = 0
+                            Match = ''
+                            for DecFile2 in Components:
+                                if DecFile2.endswith(Dependency):
+                                    if Count == 0:
+                                        Match = DecFile2
+                                    Count = Count + 1
+                            if Count > 1:
+                                AmbiguousDependencies.append (Dependency)
+                            if Count == 1:
+                                DependentPackages.append(Match)
+                                if not args.FullPaths:
+                                    PackageLabels[Match] = (Dependency.replace(os.path.sep,'\\n'), 'white')
+                                Found = True
+                        if not Found and args.Unresolved:
+                            if args.Verbose:
+                                print ('WARNING: Dependent package not found ' + Dependency)
+                            DependentPackages.append(Dependency)
+                            UnresolvedPackages.append(Dependency)
+                            PackageLabels[Dependency] = (Dependency.replace(os.path.sep,'\\n'), 'white')
+                    Components[DecFile][InfFile] = DependentPackages
+    if AmbiguousDependencies:
+        for Dependency in set(AmbiguousDependencies):
+            print ('ERROR: MULTIPLE:   ' + Dependency)
+        print ('Use --packages-path to provide search priority.')
+        sys.exit(1)
+
+    #
+    # Generate GraphViz dot input file contents.
+    # Use networkx to detect circular dependencies.
+    #
+    MaxWeight = 0
+    MaxWeightDecFile = list(Components.keys())[0]
+    Graph = nx.DiGraph()
+    Edges = []
+    for DecFile in Components:
+        if args.Verbose:
+            print ('PACKAGE DEPENDENCIES:' + DecFile)
+        AllDependencies = []
+        Dependencies = set()
+        for InfFile in Components[DecFile]:
+            AllDependencies += Components[DecFile][InfFile]
+            Dependencies = Dependencies.union(Components[DecFile][InfFile])
+        if not args.SelfDependency:
+            Dependencies = Dependencies.difference(set([DecFile]))
+        for Dependency in Dependencies:
+            Weight = AllDependencies.count(Dependency)
+            if Weight > MaxWeight:
+                MaxWeight = Weight
+                MaxWeightDecFile = DecFile
+            if args.Verbose:
+                print ('  DEPENDENCY: Weight(' + str(Weight) + ') ' + Dependency)
+            Edges.append('  "{Package}" -> "{Dependency}" [label = "{Weight}"];'.format(
+                Package    = DecFile,
+                Dependency = Dependency,
+                Weight     = str(Weight) if args.Label else ''
+                ))
+            Graph.add_edge(DecFile, Dependency)
+    Edges.sort()
+
+    #
+    # Set fill color to yellow if a packages is part of a circular dependency
+    #
+    for Node in set([x for y in list(nx.simple_cycles(Graph)) for x in y]):
+        PackageLabels[Node] = (PackageLabels[Node][0], 'yellow')
+        print ('ERROR: CIRCULAR:   ' + Node)
+
+    #
+    # Set fill color to pink if a package that is nested inside another package.
+    # Set fill color to orange if a package is nested inside another package and
+    # is part of a circular dependency.
+    #
+    for DecFile in Components:
+        DecPath = os.path.split (DecFile)[0]
+        for DecFile2 in Components:
+            if DecFile2 == DecFile:
+                continue
+            if os.path.commonpath ([DecPath, DecFile2]) != DecPath:
+                continue
+            if len(DecFile2) < len(DecPath):
+                DecFile2 = DecFile
+            print ('ERROR: NESTED:     ' + DecFile2)
+            if PackageLabels[DecFile2][1] == 'yellow':
+                PackageLabels[DecFile2] = (PackageLabels[DecFile2][0], 'orange')
+            else:
+                PackageLabels[DecFile2] = (PackageLabels[DecFile2][0], 'pink')
+
+    #
+    # Set fill color to red for nodes that are unresolved
+    #
+    for Node in set(UnresolvedPackages):
+        PackageLabels[Node] = (PackageLabels[Node][0], 'red')
+        print ('ERROR: UNRESOLVED: ' + Node)
+
+    #
+    # Add node statements to set node label and fill color
+    #
+    Nodes = []
+    for Package in PackageLabels:
+        Nodes.append('  "{Package}" [label="{Label}",fillcolor={Color}];'.format(
+            Package = Package,
+            Label = PackageLabels[Package][0],
+            Color = PackageLabels[Package][1]
+            ))
+    Nodes.sort()
+
+    #
+    # Generate dot file from Nodes and Edges and add a Legend at top of graph
+    #
+    Dot = []
+    Dot.append('digraph {')
+    Dot.append('  rankdir=BT;')
+    Dot.append('  node [shape=Mrecord,style=filled];')
+    Dot.append('')
+    Dot = Dot + Nodes
+    Dot.append('')
+    Dot = Dot + Edges
+    Dot.append('')
+    Dot.append('  subgraph legend {')
+    Dot.append('    rank=sink;')
+    Dot.append('    Unresolved     [label="Unresolved Dependency",                  fillcolor=red];')
+    Dot.append('    Circular       [label="Circular Dependency",                    fillcolor=yellow];')
+    Dot.append('    Nested         [label="Nested Package",                         fillcolor=pink];')
+    Dot.append('    NestedCircular [label="Nested Package with Circular Dependency",fillcolor=orange];')
+    Dot.append('  }')
+    if MaxWeightDecFile:
+        Dot.append('  Unresolved->"' + MaxWeightDecFile + '" [style=invis];')
+    Dot.append('}')
+
+    if args.DotOutputFile:
+        #
+        # Write GraphViz dot file contents to DotOutputFile
+        #
+        with open(os.path.realpath(args.DotOutputFile), 'w') as File:
+            File.write('\n'.join(Dot))
+    if args.SvgOutputFile:
+        #
+        # Use GraphViz 'dot' command to generate SVG output file
+        #
+        args.SvgOutputFile = os.path.realpath(args.SvgOutputFile)
+        try:
+            Process = subprocess.Popen('dot -Tsvg',
+              stdin=subprocess.PIPE,
+              stdout=open(args.SvgOutputFile, 'w'),
+              stderr=subprocess.PIPE,
+              shell=True
+              )
+            Process.stdin.write ('\n'.join(Dot).encode())
+            Process.communicate()
+            if Process.returncode != 0:
+                print ("ERROR: Can not run GraphViz 'dot' command.  Check install and path.")
+                sys.exit(Process.returncode)
+        except:
+            print ("ERROR: Can not run GraphViz 'dot' command.  Check install and path.")
+            sys.exit(1)
+        #
+        # Display SVG file in default web browser
+        #
+        if args.WebBrowser:
+            webbrowser.open(args.SvgOutputFile)
-- 
2.21.0.windows.1

Re: [Patch 1/1] BaseTools/Scripts: Add package dependency graphing tool

[Ni, Ray]

Mike,
2 minor comments regarding the help string in below.

> -----Original Message-----
> From: Kinney, Michael D <michael.d.kinney@intel.com>
> Sent: Saturday, December 14, 2019 3:45 AM
> To: devel@edk2.groups.io
> Cc: Ni, Ray <ray.ni@intel.com>; Feng, Bob C <bob.c.feng@intel.com>; Gao,
> Liming <liming.gao@intel.com>; Sean Brogan <sean.brogan@microsoft.com>;
> Bret Barkelew <Bret.Barkelew@microsoft.com>
> Subject: [Patch 1/1] BaseTools/Scripts: Add package dependency graphing
> tool
> 
> https://bugzilla.tianocore.org/show_bug.cgi?id=2161
> 
> Add python script that recursively scans a directory for EDK II
> packages and generates GraphViz dot input that is used to render
> a graph of package dependencies in SVG format.
> 
> Detects following error/warning conditions:
> * Ambiguous dependencies (multiple matches)
> * Unresolved dependencies
> * Circular dependencies
> * Nested packages
> 
> usage: PackageDependencyGraph [-h] [-w WORKSPACE] [-p PACKAGESPATH]
>                               [-d DOTOUTPUTFILE] [-o SVGOUTPUTFILE]
>                               [-g IGNOREDIRECTORY] [-k SKIPPACKAGE]
>                               [-s] [-u] [-l] [-f] [-b] [-v] [-q]
>                               [--debug [0-9]]
> 
> Recursively scan a directory for EDK II packages and generate GraphViz dot
> input that is used to render a graph of package dependencies in SVG format.
> Copyright (c) 2019, Intel Corporation. All rights reserved.
> 
> optional arguments:
>   -h, --help            show this help message and exit
>   -w WORKSPACE, --workspace WORKSPACE
>                         Directory to recursively scan for EDK II packages.
>                         Default is current directory.
>   -p PACKAGESPATH, --packages-path PACKAGESPATH
>                         List of directories to recursively scan for EDK II
>                         packages.
>   -d DOTOUTPUTFILE, --dot-output DOTOUTPUTFILE
>                         DOT output filename.
>   -o OUTPUTFILE, --output OUTPUTFILE
>                         SVG output filename.
>   -g IGNOREDIRECTORY, --ignore-directory IGNOREDIRECTORY
>                         Name of directory to ignore. Option can be repeated
>                         to ignore multiple directories.

1. Name of directory to ignore when scanning for EDKII Module INFs.

>   -k SKIPPACKAGE, --skip-package SKIPPACKAGE
>                         Name of EDK II Package DEC file to skip. Option can
>                         be repeated to skip multiple EDK II packages.

2. Name of EDK II Package DEC file (including .DEC suffix) to skip. Option can
Be repeated to skip multiple EDK II packages.

>   -s, --self-dependency
>                         Include self links in dependency graph. Default is
>                         disabled.
>   -u, --unresolved      Include unresolved EDK II packages in dependency
>                         graph. Default is disabled.
>   -l, --label           Label links with the number of EDK II package
>                         dependencies. Default is disabled.
>   -f, --full-paths      Label package nodes with full path to EDK II
>                         package.  Default is disabled.
>   -b, --web-browser     Display SVG output file in default web browser.
>                         Default is disabled.
>   -v, --verbose         Increase output messages
>   -q, --quiet           Reduce output messages
>   --debug [0-9]         Set debug level
> 
> Cc: Ray Ni <ray.ni@intel.com>
> Cc: Bob Feng <bob.c.feng@intel.com>
> Cc: Liming Gao <liming.gao@intel.com>
> Cc: Sean Brogan <sean.brogan@microsoft.com>
> Cc: Bret Barkelew <Bret.Barkelew@microsoft.com>
> Signed-off-by: Michael D Kinney <michael.d.kinney@intel.com>
> ---
>  BaseTools/Scripts/PackageDependencyGraph.py | 296
> ++++++++++++++++++++
>  1 file changed, 296 insertions(+)
>  create mode 100644 BaseTools/Scripts/PackageDependencyGraph.py
> 
> diff --git a/BaseTools/Scripts/PackageDependencyGraph.py
> b/BaseTools/Scripts/PackageDependencyGraph.py
> new file mode 100644
> index 0000000000..b3c8e41774
> --- /dev/null
> +++ b/BaseTools/Scripts/PackageDependencyGraph.py
> @@ -0,0 +1,296 @@
> +# @file
> +# Recursively scan a directory for EDK II packages and generate GraphViz
> dot
> +# input that is used to render a graph of package dependencies in SVG
> format.
> +#
> +# Copyright (c) 2019, Intel Corporation. All rights reserved.<BR>
> +# SPDX-License-Identifier: BSD-2-Clause-Patent
> +#
> +##
> +
> +import os
> +import sys
> +import argparse
> +import subprocess
> +import webbrowser
> +import networkx as nx
> +from edk2toollib.uefi.edk2.parsers.inf_parser import InfParser
> +
> +#
> +# Globals for help information
> +#
> +__prog__        = 'PackageDependencyGraph'
> +__copyright__   = 'Copyright (c) 2019, Intel Corporation. All rights reserved.'
> +__description__ = '''Recursively scan a directory for EDK II packages and
> +generate GraphViz dot input that is used to render a graph of package
> +dependencies in SVG format.'''
> +
> +if __name__ == '__main__':
> +
> +    #
> +    # Create command line argument parser object
> +    #
> +    parser = argparse.ArgumentParser (prog = __prog__,
> +                                      description = __description__ + '\n' + __copyright__,
> +                                      conflict_handler = 'resolve')
> +    parser.add_argument ("-w", "--workspace", dest = 'Workspace', default =
> os.curdir,
> +                         help = "Directory to recursively scan for EDK II packages.
> Default is current directory.")
> +    parser.add_argument ("-p", "--packages-path", dest = 'PackagesPath',
> default = None,
> +                         help = "List of directories to recursively scan for EDK II
> packages.")
> +    parser.add_argument ("-d", "--dot-output", dest = 'DotOutputFile',
> +                         help = "DOT output filename.")
> +    parser.add_argument ("-o", "--output", dest = 'SvgOutputFile',
> +                         help = "SVG output filename.")
> +    parser.add_argument ("-g", "--ignore-directory", dest = 'IgnoreDirectory',
> action='append', default=[],
> +                         help = "Name of directory to ignore.  Option can be repeated to
> ignore multiple directories.")
> +    parser.add_argument ("-k", "--skip-package", dest = 'SkipPackage',
> action='append', default=[],
> +                         help = "Name of EDK II Package DEC file to skip.  Option can be
> repeated to skip multiple EDK II packages.")
> +    parser.add_argument ("-s", "--self-dependency", dest =
> 'SelfDependency', action = "store_true", default = False,
> +                         help = "Include self links in dependency graph.  Default is
> disabled.")
> +    parser.add_argument ("-u", "--unresolved", dest = 'Unresolved', action =
> "store_true", default=False,
> +                         help = "Include unresolved EDK II packages in dependency
> graph.  Default is disabled.")
> +    parser.add_argument ("-l", "--label", dest = 'Label', action = "store_true",
> default=False,
> +                         help = "Label links with the number of EDK II package
> dependencies.  Default is disabled.")
> +    parser.add_argument ("-f", "--full-paths", dest = 'FullPaths', action =
> "store_true", default=False,
> +                         help = "Label package nodes with full path to EDK II package.
> Default is disabled.")
> +    parser.add_argument ("-b", "--web-browser", dest = 'WebBrowser',
> action = "store_true", default=False,
> +                         help = "Display SVG output file in default web browser.  Default
> is disabled.")
> +    parser.add_argument ("-v", "--verbose", dest = 'Verbose', action =
> "store_true",
> +                         help = "Increase output messages")
> +    parser.add_argument ("-q", "--quiet", dest = 'Quiet', action =
> "store_true",
> +                         help = "Reduce output messages")
> +    parser.add_argument ("--debug", dest = 'Debug', type = int, metavar =
> '[0-9]', choices = range (0, 10), default = 0,
> +                         help = "Set debug level")
> +
> +    #
> +    # Parse command line arguments
> +    #
> +    args = parser.parse_args ()
> +
> +    #
> +    # Find all EDK II package DEC files
> +    #
> +    Components = {}
> +    SearchPaths = [args.Workspace]
> +    if args.PackagesPath:
> +        SearchPaths += args.PackagesPath.split(os.pathsep)
> +    SearchPaths = [os.path.realpath(x) for x in SearchPaths]
> +    for SearchPath in SearchPaths:
> +        for root, dirs, files in os.walk (SearchPath):
> +            for name in files:
> +                FilePath = os.path.join (root, name)
> +                if
> set(FilePath.split(os.sep)).intersection(set(args.IgnoreDirectory)) != set():
> +                    if args.Verbose:
> +                        print ('IGNORE:' + FilePath)
> +                    continue
> +                if os.path.splitext(FilePath)[1].lower() in ['.dec']:
> +                    DecFile = os.path.realpath (FilePath)
> +                    if os.path.split(DecFile)[1] in args.SkipPackage:
> +                        if args.Verbose:
> +                            print ('SKIP:' + DecFile)
> +                        continue
> +                    if DecFile not in Components:
> +                        if args.Verbose:
> +                            print ('PACKAGE:' + DecFile)
> +                        Components[DecFile] = {}
> +
> +    #
> +    # Find EDK II component INF files in each EDK II package
> +    #
> +    PackageLabels = {}
> +    UnresolvedPackages = []
> +    AmbiguousDependencies = []
> +    for DecFile in Components:
> +        DecPath = os.path.split (DecFile)[0]
> +        if DecFile not in PackageLabels:
> +            PackageLabels[DecFile] = (DecFile.replace(os.path.sep,'\\n'), 'white')
> +            if not args.FullPaths:
> +                PackagePath = os.path.relpath(DecFile, os.path.split(DecPath)[0])
> +                PackageLabels[DecFile] = (PackagePath.replace(os.path.sep,'\\n'),
> 'white')
> +        for root, dirs, files in os.walk (DecPath):
> +            for name in files:
> +                FilePath = os.path.join(root, name)
> +                if
> set(FilePath.split(os.sep)).intersection(set(args.IgnoreDirectory)) != set():
> +                    if args.Verbose:
> +                        print ('IGNORE:' + FilePath)
> +                    continue
> +                if os.path.splitext(FilePath)[1].lower() in ['.inf']:
> +                    InfFile = os.path.realpath (FilePath)
> +                    Inf = InfParser ()
> +                    Inf.ParseFile (InfFile)
> +                    DependentPackages = []
> +                    for Dependency in Inf.PackagesUsed:
> +                        Dependency = os.path.normpath(Dependency)
> +                        if os.path.split(Dependency)[1] in args.SkipPackage:
> +                            if args.Verbose:
> +                                print ('SKIP:' + Dependency)
> +                            continue
> +                        Found = False
> +                        for SearchPath in SearchPaths:
> +                            PackagePath = os.path.realpath(os.path.join(SearchPath,
> Dependency))
> +                            if os.path.exists(PackagePath):
> +                                DependentPackages.append(PackagePath)
> +                                if not args.FullPaths:
> +                                    PackageLabels[PackagePath] =
> (Dependency.replace(os.path.sep,'\\n'), 'white')
> +                                Found = True
> +                                break
> +                        if not Found:
> +                            Count = 0
> +                            Match = ''
> +                            for DecFile2 in Components:
> +                                if DecFile2.endswith(Dependency):
> +                                    if Count == 0:
> +                                        Match = DecFile2
> +                                    Count = Count + 1
> +                            if Count > 1:
> +                                AmbiguousDependencies.append (Dependency)
> +                            if Count == 1:
> +                                DependentPackages.append(Match)
> +                                if not args.FullPaths:
> +                                    PackageLabels[Match] =
> (Dependency.replace(os.path.sep,'\\n'), 'white')
> +                                Found = True
> +                        if not Found and args.Unresolved:
> +                            if args.Verbose:
> +                                print ('WARNING: Dependent package not found ' +
> Dependency)
> +                            DependentPackages.append(Dependency)
> +                            UnresolvedPackages.append(Dependency)
> +                            PackageLabels[Dependency] =
> (Dependency.replace(os.path.sep,'\\n'), 'white')
> +                    Components[DecFile][InfFile] = DependentPackages
> +    if AmbiguousDependencies:
> +        for Dependency in set(AmbiguousDependencies):
> +            print ('ERROR: MULTIPLE:   ' + Dependency)
> +        print ('Use --packages-path to provide search priority.')
> +        sys.exit(1)
> +
> +    #
> +    # Generate GraphViz dot input file contents.
> +    # Use networkx to detect circular dependencies.
> +    #
> +    MaxWeight = 0
> +    MaxWeightDecFile = list(Components.keys())[0]
> +    Graph = nx.DiGraph()
> +    Edges = []
> +    for DecFile in Components:
> +        if args.Verbose:
> +            print ('PACKAGE DEPENDENCIES:' + DecFile)
> +        AllDependencies = []
> +        Dependencies = set()
> +        for InfFile in Components[DecFile]:
> +            AllDependencies += Components[DecFile][InfFile]
> +            Dependencies = Dependencies.union(Components[DecFile][InfFile])
> +        if not args.SelfDependency:
> +            Dependencies = Dependencies.difference(set([DecFile]))
> +        for Dependency in Dependencies:
> +            Weight = AllDependencies.count(Dependency)
> +            if Weight > MaxWeight:
> +                MaxWeight = Weight
> +                MaxWeightDecFile = DecFile
> +            if args.Verbose:
> +                print ('  DEPENDENCY: Weight(' + str(Weight) + ') ' + Dependency)
> +            Edges.append('  "{Package}" -> "{Dependency}" [label =
> "{Weight}"];'.format(
> +                Package    = DecFile,
> +                Dependency = Dependency,
> +                Weight     = str(Weight) if args.Label else ''
> +                ))
> +            Graph.add_edge(DecFile, Dependency)
> +    Edges.sort()
> +
> +    #
> +    # Set fill color to yellow if a packages is part of a circular dependency
> +    #
> +    for Node in set([x for y in list(nx.simple_cycles(Graph)) for x in y]):
> +        PackageLabels[Node] = (PackageLabels[Node][0], 'yellow')
> +        print ('ERROR: CIRCULAR:   ' + Node)
> +
> +    #
> +    # Set fill color to pink if a package that is nested inside another package.
> +    # Set fill color to orange if a package is nested inside another package and
> +    # is part of a circular dependency.
> +    #
> +    for DecFile in Components:
> +        DecPath = os.path.split (DecFile)[0]
> +        for DecFile2 in Components:
> +            if DecFile2 == DecFile:
> +                continue
> +            if os.path.commonpath ([DecPath, DecFile2]) != DecPath:
> +                continue
> +            if len(DecFile2) < len(DecPath):
> +                DecFile2 = DecFile
> +            print ('ERROR: NESTED:     ' + DecFile2)
> +            if PackageLabels[DecFile2][1] == 'yellow':
> +                PackageLabels[DecFile2] = (PackageLabels[DecFile2][0], 'orange')
> +            else:
> +                PackageLabels[DecFile2] = (PackageLabels[DecFile2][0], 'pink')
> +
> +    #
> +    # Set fill color to red for nodes that are unresolved
> +    #
> +    for Node in set(UnresolvedPackages):
> +        PackageLabels[Node] = (PackageLabels[Node][0], 'red')
> +        print ('ERROR: UNRESOLVED: ' + Node)
> +
> +    #
> +    # Add node statements to set node label and fill color
> +    #
> +    Nodes = []
> +    for Package in PackageLabels:
> +        Nodes.append('  "{Package}" [label="{Label}",fillcolor={Color}];'.format(
> +            Package = Package,
> +            Label = PackageLabels[Package][0],
> +            Color = PackageLabels[Package][1]
> +            ))
> +    Nodes.sort()
> +
> +    #
> +    # Generate dot file from Nodes and Edges and add a Legend at top of
> graph
> +    #
> +    Dot = []
> +    Dot.append('digraph {')
> +    Dot.append('  rankdir=BT;')
> +    Dot.append('  node [shape=Mrecord,style=filled];')
> +    Dot.append('')
> +    Dot = Dot + Nodes
> +    Dot.append('')
> +    Dot = Dot + Edges
> +    Dot.append('')
> +    Dot.append('  subgraph legend {')
> +    Dot.append('    rank=sink;')
> +    Dot.append('    Unresolved     [label="Unresolved Dependency",
> fillcolor=red];')
> +    Dot.append('    Circular       [label="Circular Dependency",
> fillcolor=yellow];')
> +    Dot.append('    Nested         [label="Nested Package",
> fillcolor=pink];')
> +    Dot.append('    NestedCircular [label="Nested Package with Circular
> Dependency",fillcolor=orange];')
> +    Dot.append('  }')
> +    if MaxWeightDecFile:
> +        Dot.append('  Unresolved->"' + MaxWeightDecFile + '" [style=invis];')
> +    Dot.append('}')
> +
> +    if args.DotOutputFile:
> +        #
> +        # Write GraphViz dot file contents to DotOutputFile
> +        #
> +        with open(os.path.realpath(args.DotOutputFile), 'w') as File:
> +            File.write('\n'.join(Dot))
> +    if args.SvgOutputFile:
> +        #
> +        # Use GraphViz 'dot' command to generate SVG output file
> +        #
> +        args.SvgOutputFile = os.path.realpath(args.SvgOutputFile)
> +        try:
> +            Process = subprocess.Popen('dot -Tsvg',
> +              stdin=subprocess.PIPE,
> +              stdout=open(args.SvgOutputFile, 'w'),
> +              stderr=subprocess.PIPE,
> +              shell=True
> +              )
> +            Process.stdin.write ('\n'.join(Dot).encode())
> +            Process.communicate()
> +            if Process.returncode != 0:
> +                print ("ERROR: Can not run GraphViz 'dot' command.  Check install
> and path.")
> +                sys.exit(Process.returncode)
> +        except:
> +            print ("ERROR: Can not run GraphViz 'dot' command.  Check install and
> path.")
> +            sys.exit(1)
> +        #
> +        # Display SVG file in default web browser
> +        #
> +        if args.WebBrowser:
> +            webbrowser.open(args.SvgOutputFile)
> --
> 2.21.0.windows.1

Re: [edk2-devel] [Patch 1/1] BaseTools/Scripts: Add package dependency graphing tool

[Ni, Ray]

Mike,
This pkg dep tool can tell through weight when a module depends on a new pkg.
But it cannot tell when a module depends on another PCD/Protocol/Guid/Ppi/Library which belongs to an already-depended pkg.

Failure to tell such information may make a bad module (that violates the dependency rules) worse (wrongly depend on more interfaces).

Do you think that the tool can be enhanced in future to detect the case?

Thanks,
Ray

> -----Original Message-----
> From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Ni, Ray
> Sent: Monday, December 16, 2019 1:57 PM
> To: Kinney, Michael D <michael.d.kinney@intel.com>; devel@edk2.groups.io
> Cc: Feng, Bob C <bob.c.feng@intel.com>; Gao, Liming
> <liming.gao@intel.com>; Sean Brogan <sean.brogan@microsoft.com>; Bret
> Barkelew <Bret.Barkelew@microsoft.com>
> Subject: Re: [edk2-devel] [Patch 1/1] BaseTools/Scripts: Add package
> dependency graphing tool
> 
> Mike,
> 2 minor comments regarding the help string in below.
> 
> > -----Original Message-----
> > From: Kinney, Michael D <michael.d.kinney@intel.com>
> > Sent: Saturday, December 14, 2019 3:45 AM
> > To: devel@edk2.groups.io
> > Cc: Ni, Ray <ray.ni@intel.com>; Feng, Bob C <bob.c.feng@intel.com>; Gao,
> > Liming <liming.gao@intel.com>; Sean Brogan
> <sean.brogan@microsoft.com>;
> > Bret Barkelew <Bret.Barkelew@microsoft.com>
> > Subject: [Patch 1/1] BaseTools/Scripts: Add package dependency graphing
> > tool
> >
> > https://bugzilla.tianocore.org/show_bug.cgi?id=2161
> >
> > Add python script that recursively scans a directory for EDK II
> > packages and generates GraphViz dot input that is used to render
> > a graph of package dependencies in SVG format.
> >
> > Detects following error/warning conditions:
> > * Ambiguous dependencies (multiple matches)
> > * Unresolved dependencies
> > * Circular dependencies
> > * Nested packages
> >
> > usage: PackageDependencyGraph [-h] [-w WORKSPACE] [-p
> PACKAGESPATH]
> >                               [-d DOTOUTPUTFILE] [-o SVGOUTPUTFILE]
> >                               [-g IGNOREDIRECTORY] [-k SKIPPACKAGE]
> >                               [-s] [-u] [-l] [-f] [-b] [-v] [-q]
> >                               [--debug [0-9]]
> >
> > Recursively scan a directory for EDK II packages and generate GraphViz dot
> > input that is used to render a graph of package dependencies in SVG
> format.
> > Copyright (c) 2019, Intel Corporation. All rights reserved.
> >
> > optional arguments:
> >   -h, --help            show this help message and exit
> >   -w WORKSPACE, --workspace WORKSPACE
> >                         Directory to recursively scan for EDK II packages.
> >                         Default is current directory.
> >   -p PACKAGESPATH, --packages-path PACKAGESPATH
> >                         List of directories to recursively scan for EDK II
> >                         packages.
> >   -d DOTOUTPUTFILE, --dot-output DOTOUTPUTFILE
> >                         DOT output filename.
> >   -o OUTPUTFILE, --output OUTPUTFILE
> >                         SVG output filename.
> >   -g IGNOREDIRECTORY, --ignore-directory IGNOREDIRECTORY
> >                         Name of directory to ignore. Option can be repeated
> >                         to ignore multiple directories.
> 
> 1. Name of directory to ignore when scanning for EDKII Module INFs.
> 
> >   -k SKIPPACKAGE, --skip-package SKIPPACKAGE
> >                         Name of EDK II Package DEC file to skip. Option can
> >                         be repeated to skip multiple EDK II packages.
> 
> 2. Name of EDK II Package DEC file (including .DEC suffix) to skip. Option can
> Be repeated to skip multiple EDK II packages.
> 
> >   -s, --self-dependency
> >                         Include self links in dependency graph. Default is
> >                         disabled.
> >   -u, --unresolved      Include unresolved EDK II packages in dependency
> >                         graph. Default is disabled.
> >   -l, --label           Label links with the number of EDK II package
> >                         dependencies. Default is disabled.
> >   -f, --full-paths      Label package nodes with full path to EDK II
> >                         package.  Default is disabled.
> >   -b, --web-browser     Display SVG output file in default web browser.
> >                         Default is disabled.
> >   -v, --verbose         Increase output messages
> >   -q, --quiet           Reduce output messages
> >   --debug [0-9]         Set debug level
> >
> > Cc: Ray Ni <ray.ni@intel.com>
> > Cc: Bob Feng <bob.c.feng@intel.com>
> > Cc: Liming Gao <liming.gao@intel.com>
> > Cc: Sean Brogan <sean.brogan@microsoft.com>
> > Cc: Bret Barkelew <Bret.Barkelew@microsoft.com>
> > Signed-off-by: Michael D Kinney <michael.d.kinney@intel.com>
> > ---
> >  BaseTools/Scripts/PackageDependencyGraph.py | 296
> > ++++++++++++++++++++
> >  1 file changed, 296 insertions(+)
> >  create mode 100644 BaseTools/Scripts/PackageDependencyGraph.py
> >
> > diff --git a/BaseTools/Scripts/PackageDependencyGraph.py
> > b/BaseTools/Scripts/PackageDependencyGraph.py
> > new file mode 100644
> > index 0000000000..b3c8e41774
> > --- /dev/null
> > +++ b/BaseTools/Scripts/PackageDependencyGraph.py
> > @@ -0,0 +1,296 @@
> > +# @file
> > +# Recursively scan a directory for EDK II packages and generate GraphViz
> > dot
> > +# input that is used to render a graph of package dependencies in SVG
> > format.
> > +#
> > +# Copyright (c) 2019, Intel Corporation. All rights reserved.<BR>
> > +# SPDX-License-Identifier: BSD-2-Clause-Patent
> > +#
> > +##
> > +
> > +import os
> > +import sys
> > +import argparse
> > +import subprocess
> > +import webbrowser
> > +import networkx as nx
> > +from edk2toollib.uefi.edk2.parsers.inf_parser import InfParser
> > +
> > +#
> > +# Globals for help information
> > +#
> > +__prog__        = 'PackageDependencyGraph'
> > +__copyright__   = 'Copyright (c) 2019, Intel Corporation. All rights
> reserved.'
> > +__description__ = '''Recursively scan a directory for EDK II packages and
> > +generate GraphViz dot input that is used to render a graph of package
> > +dependencies in SVG format.'''
> > +
> > +if __name__ == '__main__':
> > +
> > +    #
> > +    # Create command line argument parser object
> > +    #
> > +    parser = argparse.ArgumentParser (prog = __prog__,
> > +                                      description = __description__ + '\n' + __copyright__,
> > +                                      conflict_handler = 'resolve')
> > +    parser.add_argument ("-w", "--workspace", dest = 'Workspace', default
> =
> > os.curdir,
> > +                         help = "Directory to recursively scan for EDK II packages.
> > Default is current directory.")
> > +    parser.add_argument ("-p", "--packages-path", dest = 'PackagesPath',
> > default = None,
> > +                         help = "List of directories to recursively scan for EDK II
> > packages.")
> > +    parser.add_argument ("-d", "--dot-output", dest = 'DotOutputFile',
> > +                         help = "DOT output filename.")
> > +    parser.add_argument ("-o", "--output", dest = 'SvgOutputFile',
> > +                         help = "SVG output filename.")
> > +    parser.add_argument ("-g", "--ignore-directory", dest =
> 'IgnoreDirectory',
> > action='append', default=[],
> > +                         help = "Name of directory to ignore.  Option can be repeated
> to
> > ignore multiple directories.")
> > +    parser.add_argument ("-k", "--skip-package", dest = 'SkipPackage',
> > action='append', default=[],
> > +                         help = "Name of EDK II Package DEC file to skip.  Option can
> be
> > repeated to skip multiple EDK II packages.")
> > +    parser.add_argument ("-s", "--self-dependency", dest =
> > 'SelfDependency', action = "store_true", default = False,
> > +                         help = "Include self links in dependency graph.  Default is
> > disabled.")
> > +    parser.add_argument ("-u", "--unresolved", dest = 'Unresolved', action
> =
> > "store_true", default=False,
> > +                         help = "Include unresolved EDK II packages in dependency
> > graph.  Default is disabled.")
> > +    parser.add_argument ("-l", "--label", dest = 'Label', action =
> "store_true",
> > default=False,
> > +                         help = "Label links with the number of EDK II package
> > dependencies.  Default is disabled.")
> > +    parser.add_argument ("-f", "--full-paths", dest = 'FullPaths', action =
> > "store_true", default=False,
> > +                         help = "Label package nodes with full path to EDK II package.
> > Default is disabled.")
> > +    parser.add_argument ("-b", "--web-browser", dest = 'WebBrowser',
> > action = "store_true", default=False,
> > +                         help = "Display SVG output file in default web browser.
> Default
> > is disabled.")
> > +    parser.add_argument ("-v", "--verbose", dest = 'Verbose', action =
> > "store_true",
> > +                         help = "Increase output messages")
> > +    parser.add_argument ("-q", "--quiet", dest = 'Quiet', action =
> > "store_true",
> > +                         help = "Reduce output messages")
> > +    parser.add_argument ("--debug", dest = 'Debug', type = int, metavar =
> > '[0-9]', choices = range (0, 10), default = 0,
> > +                         help = "Set debug level")
> > +
> > +    #
> > +    # Parse command line arguments
> > +    #
> > +    args = parser.parse_args ()
> > +
> > +    #
> > +    # Find all EDK II package DEC files
> > +    #
> > +    Components = {}
> > +    SearchPaths = [args.Workspace]
> > +    if args.PackagesPath:
> > +        SearchPaths += args.PackagesPath.split(os.pathsep)
> > +    SearchPaths = [os.path.realpath(x) for x in SearchPaths]
> > +    for SearchPath in SearchPaths:
> > +        for root, dirs, files in os.walk (SearchPath):
> > +            for name in files:
> > +                FilePath = os.path.join (root, name)
> > +                if
> > set(FilePath.split(os.sep)).intersection(set(args.IgnoreDirectory)) != set():
> > +                    if args.Verbose:
> > +                        print ('IGNORE:' + FilePath)
> > +                    continue
> > +                if os.path.splitext(FilePath)[1].lower() in ['.dec']:
> > +                    DecFile = os.path.realpath (FilePath)
> > +                    if os.path.split(DecFile)[1] in args.SkipPackage:
> > +                        if args.Verbose:
> > +                            print ('SKIP:' + DecFile)
> > +                        continue
> > +                    if DecFile not in Components:
> > +                        if args.Verbose:
> > +                            print ('PACKAGE:' + DecFile)
> > +                        Components[DecFile] = {}
> > +
> > +    #
> > +    # Find EDK II component INF files in each EDK II package
> > +    #
> > +    PackageLabels = {}
> > +    UnresolvedPackages = []
> > +    AmbiguousDependencies = []
> > +    for DecFile in Components:
> > +        DecPath = os.path.split (DecFile)[0]
> > +        if DecFile not in PackageLabels:
> > +            PackageLabels[DecFile] = (DecFile.replace(os.path.sep,'\\n'), 'white')
> > +            if not args.FullPaths:
> > +                PackagePath = os.path.relpath(DecFile, os.path.split(DecPath)[0])
> > +                PackageLabels[DecFile] = (PackagePath.replace(os.path.sep,'\\n'),
> > 'white')
> > +        for root, dirs, files in os.walk (DecPath):
> > +            for name in files:
> > +                FilePath = os.path.join(root, name)
> > +                if
> > set(FilePath.split(os.sep)).intersection(set(args.IgnoreDirectory)) != set():
> > +                    if args.Verbose:
> > +                        print ('IGNORE:' + FilePath)
> > +                    continue
> > +                if os.path.splitext(FilePath)[1].lower() in ['.inf']:
> > +                    InfFile = os.path.realpath (FilePath)
> > +                    Inf = InfParser ()
> > +                    Inf.ParseFile (InfFile)
> > +                    DependentPackages = []
> > +                    for Dependency in Inf.PackagesUsed:
> > +                        Dependency = os.path.normpath(Dependency)
> > +                        if os.path.split(Dependency)[1] in args.SkipPackage:
> > +                            if args.Verbose:
> > +                                print ('SKIP:' + Dependency)
> > +                            continue
> > +                        Found = False
> > +                        for SearchPath in SearchPaths:
> > +                            PackagePath = os.path.realpath(os.path.join(SearchPath,
> > Dependency))
> > +                            if os.path.exists(PackagePath):
> > +                                DependentPackages.append(PackagePath)
> > +                                if not args.FullPaths:
> > +                                    PackageLabels[PackagePath] =
> > (Dependency.replace(os.path.sep,'\\n'), 'white')
> > +                                Found = True
> > +                                break
> > +                        if not Found:
> > +                            Count = 0
> > +                            Match = ''
> > +                            for DecFile2 in Components:
> > +                                if DecFile2.endswith(Dependency):
> > +                                    if Count == 0:
> > +                                        Match = DecFile2
> > +                                    Count = Count + 1
> > +                            if Count > 1:
> > +                                AmbiguousDependencies.append (Dependency)
> > +                            if Count == 1:
> > +                                DependentPackages.append(Match)
> > +                                if not args.FullPaths:
> > +                                    PackageLabels[Match] =
> > (Dependency.replace(os.path.sep,'\\n'), 'white')
> > +                                Found = True
> > +                        if not Found and args.Unresolved:
> > +                            if args.Verbose:
> > +                                print ('WARNING: Dependent package not found ' +
> > Dependency)
> > +                            DependentPackages.append(Dependency)
> > +                            UnresolvedPackages.append(Dependency)
> > +                            PackageLabels[Dependency] =
> > (Dependency.replace(os.path.sep,'\\n'), 'white')
> > +                    Components[DecFile][InfFile] = DependentPackages
> > +    if AmbiguousDependencies:
> > +        for Dependency in set(AmbiguousDependencies):
> > +            print ('ERROR: MULTIPLE:   ' + Dependency)
> > +        print ('Use --packages-path to provide search priority.')
> > +        sys.exit(1)
> > +
> > +    #
> > +    # Generate GraphViz dot input file contents.
> > +    # Use networkx to detect circular dependencies.
> > +    #
> > +    MaxWeight = 0
> > +    MaxWeightDecFile = list(Components.keys())[0]
> > +    Graph = nx.DiGraph()
> > +    Edges = []
> > +    for DecFile in Components:
> > +        if args.Verbose:
> > +            print ('PACKAGE DEPENDENCIES:' + DecFile)
> > +        AllDependencies = []
> > +        Dependencies = set()
> > +        for InfFile in Components[DecFile]:
> > +            AllDependencies += Components[DecFile][InfFile]
> > +            Dependencies = Dependencies.union(Components[DecFile][InfFile])
> > +        if not args.SelfDependency:
> > +            Dependencies = Dependencies.difference(set([DecFile]))
> > +        for Dependency in Dependencies:
> > +            Weight = AllDependencies.count(Dependency)
> > +            if Weight > MaxWeight:
> > +                MaxWeight = Weight
> > +                MaxWeightDecFile = DecFile
> > +            if args.Verbose:
> > +                print ('  DEPENDENCY: Weight(' + str(Weight) + ') ' + Dependency)
> > +            Edges.append('  "{Package}" -> "{Dependency}" [label =
> > "{Weight}"];'.format(
> > +                Package    = DecFile,
> > +                Dependency = Dependency,
> > +                Weight     = str(Weight) if args.Label else ''
> > +                ))
> > +            Graph.add_edge(DecFile, Dependency)
> > +    Edges.sort()
> > +
> > +    #
> > +    # Set fill color to yellow if a packages is part of a circular dependency
> > +    #
> > +    for Node in set([x for y in list(nx.simple_cycles(Graph)) for x in y]):
> > +        PackageLabels[Node] = (PackageLabels[Node][0], 'yellow')
> > +        print ('ERROR: CIRCULAR:   ' + Node)
> > +
> > +    #
> > +    # Set fill color to pink if a package that is nested inside another package.
> > +    # Set fill color to orange if a package is nested inside another package
> and
> > +    # is part of a circular dependency.
> > +    #
> > +    for DecFile in Components:
> > +        DecPath = os.path.split (DecFile)[0]
> > +        for DecFile2 in Components:
> > +            if DecFile2 == DecFile:
> > +                continue
> > +            if os.path.commonpath ([DecPath, DecFile2]) != DecPath:
> > +                continue
> > +            if len(DecFile2) < len(DecPath):
> > +                DecFile2 = DecFile
> > +            print ('ERROR: NESTED:     ' + DecFile2)
> > +            if PackageLabels[DecFile2][1] == 'yellow':
> > +                PackageLabels[DecFile2] = (PackageLabels[DecFile2][0], 'orange')
> > +            else:
> > +                PackageLabels[DecFile2] = (PackageLabels[DecFile2][0], 'pink')
> > +
> > +    #
> > +    # Set fill color to red for nodes that are unresolved
> > +    #
> > +    for Node in set(UnresolvedPackages):
> > +        PackageLabels[Node] = (PackageLabels[Node][0], 'red')
> > +        print ('ERROR: UNRESOLVED: ' + Node)
> > +
> > +    #
> > +    # Add node statements to set node label and fill color
> > +    #
> > +    Nodes = []
> > +    for Package in PackageLabels:
> > +        Nodes.append('  "{Package}"
> [label="{Label}",fillcolor={Color}];'.format(
> > +            Package = Package,
> > +            Label = PackageLabels[Package][0],
> > +            Color = PackageLabels[Package][1]
> > +            ))
> > +    Nodes.sort()
> > +
> > +    #
> > +    # Generate dot file from Nodes and Edges and add a Legend at top of
> > graph
> > +    #
> > +    Dot = []
> > +    Dot.append('digraph {')
> > +    Dot.append('  rankdir=BT;')
> > +    Dot.append('  node [shape=Mrecord,style=filled];')
> > +    Dot.append('')
> > +    Dot = Dot + Nodes
> > +    Dot.append('')
> > +    Dot = Dot + Edges
> > +    Dot.append('')
> > +    Dot.append('  subgraph legend {')
> > +    Dot.append('    rank=sink;')
> > +    Dot.append('    Unresolved     [label="Unresolved Dependency",
> > fillcolor=red];')
> > +    Dot.append('    Circular       [label="Circular Dependency",
> > fillcolor=yellow];')
> > +    Dot.append('    Nested         [label="Nested Package",
> > fillcolor=pink];')
> > +    Dot.append('    NestedCircular [label="Nested Package with Circular
> > Dependency",fillcolor=orange];')
> > +    Dot.append('  }')
> > +    if MaxWeightDecFile:
> > +        Dot.append('  Unresolved->"' + MaxWeightDecFile + '" [style=invis];')
> > +    Dot.append('}')
> > +
> > +    if args.DotOutputFile:
> > +        #
> > +        # Write GraphViz dot file contents to DotOutputFile
> > +        #
> > +        with open(os.path.realpath(args.DotOutputFile), 'w') as File:
> > +            File.write('\n'.join(Dot))
> > +    if args.SvgOutputFile:
> > +        #
> > +        # Use GraphViz 'dot' command to generate SVG output file
> > +        #
> > +        args.SvgOutputFile = os.path.realpath(args.SvgOutputFile)
> > +        try:
> > +            Process = subprocess.Popen('dot -Tsvg',
> > +              stdin=subprocess.PIPE,
> > +              stdout=open(args.SvgOutputFile, 'w'),
> > +              stderr=subprocess.PIPE,
> > +              shell=True
> > +              )
> > +            Process.stdin.write ('\n'.join(Dot).encode())
> > +            Process.communicate()
> > +            if Process.returncode != 0:
> > +                print ("ERROR: Can not run GraphViz 'dot' command.  Check install
> > and path.")
> > +                sys.exit(Process.returncode)
> > +        except:
> > +            print ("ERROR: Can not run GraphViz 'dot' command.  Check install
> and
> > path.")
> > +            sys.exit(1)
> > +        #
> > +        # Display SVG file in default web browser
> > +        #
> > +        if args.WebBrowser:
> > +            webbrowser.open(args.SvgOutputFile)
> > --
> > 2.21.0.windows.1
> 
> 
> 

Re: [edk2-devel] [Patch 1/1] BaseTools/Scripts: Add package dependency graphing tool

[Steven Shi]

[-- Attachment #1: Type: text/plain, Size: 24778 bytes --]

Ray,

I think you are asking for a module-level dependency tool. 😊



Mike,

This tool is cool. Besides the package level, do you think we can go further to figure out a module level dependency tool? With module level dependency info, we can define more interesting and comprehensive Modularity Metrics to measure the edk2 architecture quality (Architectural Technical Debt).







Thanks



Steven Shi

Intel\SSG\SFE\FIE Firmware Infrastructure





> -----Original Message-----

> From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Ni, Ray

> Sent: Monday, December 16, 2019 4:41 PM

> To: devel@edk2.groups.io; Ni, Ray <ray.ni@intel.com>; Kinney, Michael D

> <michael.d.kinney@intel.com>

> Cc: Feng, Bob C <bob.c.feng@intel.com>; Gao, Liming

> <liming.gao@intel.com>; Sean Brogan <sean.brogan@microsoft.com>; Bret

> Barkelew <Bret.Barkelew@microsoft.com>

> Subject: Re: [edk2-devel] [Patch 1/1] BaseTools/Scripts: Add package

> dependency graphing tool

>

> Mike,

> This pkg dep tool can tell through weight when a module depends on a new

> pkg.

> But it cannot tell when a module depends on another

> PCD/Protocol/Guid/Ppi/Library which belongs to an already-depended pkg.

>

> Failure to tell such information may make a bad module (that violates the

> dependency rules) worse (wrongly depend on more interfaces).

>

> Do you think that the tool can be enhanced in future to detect the case?

>

> Thanks,

> Ray

>

> > -----Original Message-----

> > From: devel@edk2.groups.io<mailto:devel@edk2.groups.io> <devel@edk2.groups.io<mailto:devel@edk2.groups.io>> On Behalf Of Ni,

> Ray

> > Sent: Monday, December 16, 2019 1:57 PM

> > To: Kinney, Michael D <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>>;

> devel@edk2.groups.io<mailto:devel@edk2.groups.io>

> > Cc: Feng, Bob C <bob.c.feng@intel.com<mailto:bob.c.feng@intel.com>>; Gao, Liming

> > <liming.gao@intel.com<mailto:liming.gao@intel.com>>; Sean Brogan <sean.brogan@microsoft.com<mailto:sean.brogan@microsoft.com>>;

> Bret

> > Barkelew <Bret.Barkelew@microsoft.com<mailto:Bret.Barkelew@microsoft.com>>

> > Subject: Re: [edk2-devel] [Patch 1/1] BaseTools/Scripts: Add package

> > dependency graphing tool

> >

> > Mike,

> > 2 minor comments regarding the help string in below.

> >

> > > -----Original Message-----

> > > From: Kinney, Michael D <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>>

> > > Sent: Saturday, December 14, 2019 3:45 AM

> > > To: devel@edk2.groups.io<mailto:devel@edk2.groups.io>

> > > Cc: Ni, Ray <ray.ni@intel.com<mailto:ray.ni@intel.com>>; Feng, Bob C <bob.c.feng@intel.com<mailto:bob.c.feng@intel.com>>;

> Gao,

> > > Liming <liming.gao@intel.com<mailto:liming.gao@intel.com>>; Sean Brogan

> > <sean.brogan@microsoft.com<mailto:sean.brogan@microsoft.com>>;

> > > Bret Barkelew <Bret.Barkelew@microsoft.com<mailto:Bret.Barkelew@microsoft.com>>

> > > Subject: [Patch 1/1] BaseTools/Scripts: Add package dependency

> graphing

> > > tool

> > >

> > > https://bugzilla.tianocore.org/show_bug.cgi?id=2161

> > >

> > > Add python script that recursively scans a directory for EDK II

> > > packages and generates GraphViz dot input that is used to render

> > > a graph of package dependencies in SVG format.

> > >

> > > Detects following error/warning conditions:

> > > * Ambiguous dependencies (multiple matches)

> > > * Unresolved dependencies

> > > * Circular dependencies

> > > * Nested packages

> > >

> > > usage: PackageDependencyGraph [-h] [-w WORKSPACE] [-p

> > PACKAGESPATH]

> > >                               [-d DOTOUTPUTFILE] [-o SVGOUTPUTFILE]

> > >                               [-g IGNOREDIRECTORY] [-k SKIPPACKAGE]

> > >                               [-s] [-u] [-l] [-f] [-b] [-v] [-q]

> > >                               [--debug [0-9]]

> > >

> > > Recursively scan a directory for EDK II packages and generate GraphViz

> dot

> > > input that is used to render a graph of package dependencies in SVG

> > format.

> > > Copyright (c) 2019, Intel Corporation. All rights reserved.

> > >

> > > optional arguments:

> > >   -h, --help            show this help message and exit

> > >   -w WORKSPACE, --workspace WORKSPACE

> > >                         Directory to recursively scan for EDK II packages.

> > >                         Default is current directory.

> > >   -p PACKAGESPATH, --packages-path PACKAGESPATH

> > >                         List of directories to recursively scan for EDK II

> > >                         packages.

> > >   -d DOTOUTPUTFILE, --dot-output DOTOUTPUTFILE

> > >                         DOT output filename.

> > >   -o OUTPUTFILE, --output OUTPUTFILE

> > >                         SVG output filename.

> > >   -g IGNOREDIRECTORY, --ignore-directory IGNOREDIRECTORY

> > >                         Name of directory to ignore. Option can be repeated

> > >                         to ignore multiple directories.

> >

> > 1. Name of directory to ignore when scanning for EDKII Module INFs.

> >

> > >   -k SKIPPACKAGE, --skip-package SKIPPACKAGE

> > >                         Name of EDK II Package DEC file to skip. Option can

> > >                         be repeated to skip multiple EDK II packages.

> >

> > 2. Name of EDK II Package DEC file (including .DEC suffix) to skip. Option

> can

> > Be repeated to skip multiple EDK II packages.

> >

> > >   -s, --self-dependency

> > >                         Include self links in dependency graph. Default is

> > >                         disabled.

> > >   -u, --unresolved      Include unresolved EDK II packages in dependency

> > >                         graph. Default is disabled.

> > >   -l, --label           Label links with the number of EDK II package

> > >                         dependencies. Default is disabled.

> > >   -f, --full-paths      Label package nodes with full path to EDK II

> > >                         package.  Default is disabled.

> > >   -b, --web-browser     Display SVG output file in default web browser.

> > >                         Default is disabled.

> > >   -v, --verbose         Increase output messages

> > >   -q, --quiet           Reduce output messages

> > >   --debug [0-9]         Set debug level

> > >

> > > Cc: Ray Ni <ray.ni@intel.com<mailto:ray.ni@intel.com>>

> > > Cc: Bob Feng <bob.c.feng@intel.com<mailto:bob.c.feng@intel.com>>

> > > Cc: Liming Gao <liming.gao@intel.com<mailto:liming.gao@intel.com>>

> > > Cc: Sean Brogan <sean.brogan@microsoft.com<mailto:sean.brogan@microsoft.com>>

> > > Cc: Bret Barkelew <Bret.Barkelew@microsoft.com<mailto:Bret.Barkelew@microsoft.com>>

> > > Signed-off-by: Michael D Kinney <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>>

> > > ---

> > >  BaseTools/Scripts/PackageDependencyGraph.py | 296

> > > ++++++++++++++++++++

> > >  1 file changed, 296 insertions(+)

> > >  create mode 100644 BaseTools/Scripts/PackageDependencyGraph.py

> > >

> > > diff --git a/BaseTools/Scripts/PackageDependencyGraph.py

> > > b/BaseTools/Scripts/PackageDependencyGraph.py

> > > new file mode 100644

> > > index 0000000000..b3c8e41774

> > > --- /dev/null

> > > +++ b/BaseTools/Scripts/PackageDependencyGraph.py

> > > @@ -0,0 +1,296 @@

> > > +# @file

> > > +# Recursively scan a directory for EDK II packages and generate

> GraphViz

> > > dot

> > > +# input that is used to render a graph of package dependencies in SVG

> > > format.

> > > +#

> > > +# Copyright (c) 2019, Intel Corporation. All rights reserved.<BR>

> > > +# SPDX-License-Identifier: BSD-2-Clause-Patent

> > > +#

> > > +##

> > > +

> > > +import os

> > > +import sys

> > > +import argparse

> > > +import subprocess

> > > +import webbrowser

> > > +import networkx as nx

> > > +from edk2toollib.uefi.edk2.parsers.inf_parser import InfParser

> > > +

> > > +#

> > > +# Globals for help information

> > > +#

> > > +__prog__        = 'PackageDependencyGraph'

> > > +__copyright__   = 'Copyright (c) 2019, Intel Corporation. All rights

> > reserved.'

> > > +__description__ = '''Recursively scan a directory for EDK II packages and

> > > +generate GraphViz dot input that is used to render a graph of package

> > > +dependencies in SVG format.'''

> > > +

> > > +if __name__ == '__main__':

> > > +

> > > +    #

> > > +    # Create command line argument parser object

> > > +    #

> > > +    parser = argparse.ArgumentParser (prog = __prog__,

> > > +                                      description = __description__ + '\n' +

> __copyright__,

> > > +                                      conflict_handler = 'resolve')

> > > +    parser.add_argument ("-w", "--workspace", dest = 'Workspace',

> default

> > =

> > > os.curdir,

> > > +                         help = "Directory to recursively scan for EDK II packages.

> > > Default is current directory.")

> > > +    parser.add_argument ("-p", "--packages-path", dest = 'PackagesPath',

> > > default = None,

> > > +                         help = "List of directories to recursively scan for EDK II

> > > packages.")

> > > +    parser.add_argument ("-d", "--dot-output", dest = 'DotOutputFile',

> > > +                         help = "DOT output filename.")

> > > +    parser.add_argument ("-o", "--output", dest = 'SvgOutputFile',

> > > +                         help = "SVG output filename.")

> > > +    parser.add_argument ("-g", "--ignore-directory", dest =

> > 'IgnoreDirectory',

> > > action='append', default=[],

> > > +                         help = "Name of directory to ignore.  Option can be

> repeated

> > to

> > > ignore multiple directories.")

> > > +    parser.add_argument ("-k", "--skip-package", dest = 'SkipPackage',

> > > action='append', default=[],

> > > +                         help = "Name of EDK II Package DEC file to skip.  Option

> can

> > be

> > > repeated to skip multiple EDK II packages.")

> > > +    parser.add_argument ("-s", "--self-dependency", dest =

> > > 'SelfDependency', action = "store_true", default = False,

> > > +                         help = "Include self links in dependency graph.  Default is

> > > disabled.")

> > > +    parser.add_argument ("-u", "--unresolved", dest = 'Unresolved',

> action

> > =

> > > "store_true", default=False,

> > > +                         help = "Include unresolved EDK II packages in dependency

> > > graph.  Default is disabled.")

> > > +    parser.add_argument ("-l", "--label", dest = 'Label', action =

> > "store_true",

> > > default=False,

> > > +                         help = "Label links with the number of EDK II package

> > > dependencies.  Default is disabled.")

> > > +    parser.add_argument ("-f", "--full-paths", dest = 'FullPaths', action =

> > > "store_true", default=False,

> > > +                         help = "Label package nodes with full path to EDK II

> package.

> > > Default is disabled.")

> > > +    parser.add_argument ("-b", "--web-browser", dest = 'WebBrowser',

> > > action = "store_true", default=False,

> > > +                         help = "Display SVG output file in default web browser.

> > Default

> > > is disabled.")

> > > +    parser.add_argument ("-v", "--verbose", dest = 'Verbose', action =

> > > "store_true",

> > > +                         help = "Increase output messages")

> > > +    parser.add_argument ("-q", "--quiet", dest = 'Quiet', action =

> > > "store_true",

> > > +                         help = "Reduce output messages")

> > > +    parser.add_argument ("--debug", dest = 'Debug', type = int, metavar =

> > > '[0-9]', choices = range (0, 10), default = 0,

> > > +                         help = "Set debug level")

> > > +

> > > +    #

> > > +    # Parse command line arguments

> > > +    #

> > > +    args = parser.parse_args ()

> > > +

> > > +    #

> > > +    # Find all EDK II package DEC files

> > > +    #

> > > +    Components = {}

> > > +    SearchPaths = [args.Workspace]

> > > +    if args.PackagesPath:

> > > +        SearchPaths += args.PackagesPath.split(os.pathsep)

> > > +    SearchPaths = [os.path.realpath(x) for x in SearchPaths]

> > > +    for SearchPath in SearchPaths:

> > > +        for root, dirs, files in os.walk (SearchPath):

> > > +            for name in files:

> > > +                FilePath = os.path.join (root, name)

> > > +                if

> > > set(FilePath.split(os.sep)).intersection(set(args.IgnoreDirectory)) != set():

> > > +                    if args.Verbose:

> > > +                        print ('IGNORE:' + FilePath)

> > > +                    continue

> > > +                if os.path.splitext(FilePath)[1].lower() in ['.dec']:

> > > +                    DecFile = os.path.realpath (FilePath)

> > > +                    if os.path.split(DecFile)[1] in args.SkipPackage:

> > > +                        if args.Verbose:

> > > +                            print ('SKIP:' + DecFile)

> > > +                        continue

> > > +                    if DecFile not in Components:

> > > +                        if args.Verbose:

> > > +                            print ('PACKAGE:' + DecFile)

> > > +                        Components[DecFile] = {}

> > > +

> > > +    #

> > > +    # Find EDK II component INF files in each EDK II package

> > > +    #

> > > +    PackageLabels = {}

> > > +    UnresolvedPackages = []

> > > +    AmbiguousDependencies = []

> > > +    for DecFile in Components:

> > > +        DecPath = os.path.split (DecFile)[0]

> > > +        if DecFile not in PackageLabels:

> > > +            PackageLabels[DecFile] = (DecFile.replace(os.path.sep,'\\n'),

> 'white')

> > > +            if not args.FullPaths:

> > > +                PackagePath = os.path.relpath(DecFile,

> os.path.split(DecPath)[0])

> > > +                PackageLabels[DecFile] =

> (PackagePath.replace(os.path.sep,'\\n'),

> > > 'white')

> > > +        for root, dirs, files in os.walk (DecPath):

> > > +            for name in files:

> > > +                FilePath = os.path.join(root, name)

> > > +                if

> > > set(FilePath.split(os.sep)).intersection(set(args.IgnoreDirectory)) != set():

> > > +                    if args.Verbose:

> > > +                        print ('IGNORE:' + FilePath)

> > > +                    continue

> > > +                if os.path.splitext(FilePath)[1].lower() in ['.inf']:

> > > +                    InfFile = os.path.realpath (FilePath)

> > > +                    Inf = InfParser ()

> > > +                    Inf.ParseFile (InfFile)

> > > +                    DependentPackages = []

> > > +                    for Dependency in Inf.PackagesUsed:

> > > +                        Dependency = os.path.normpath(Dependency)

> > > +                        if os.path.split(Dependency)[1] in args.SkipPackage:

> > > +                            if args.Verbose:

> > > +                                print ('SKIP:' + Dependency)

> > > +                            continue

> > > +                        Found = False

> > > +                        for SearchPath in SearchPaths:

> > > +                            PackagePath = os.path.realpath(os.path.join(SearchPath,

> > > Dependency))

> > > +                            if os.path.exists(PackagePath):

> > > +                                DependentPackages.append(PackagePath)

> > > +                                if not args.FullPaths:

> > > +                                    PackageLabels[PackagePath] =

> > > (Dependency.replace(os.path.sep,'\\n'), 'white')

> > > +                                Found = True

> > > +                                break

> > > +                        if not Found:

> > > +                            Count = 0

> > > +                            Match = ''

> > > +                            for DecFile2 in Components:

> > > +                                if DecFile2.endswith(Dependency):

> > > +                                    if Count == 0:

> > > +                                        Match = DecFile2

> > > +                                    Count = Count + 1

> > > +                            if Count > 1:

> > > +                                AmbiguousDependencies.append (Dependency)

> > > +                            if Count == 1:

> > > +                                DependentPackages.append(Match)

> > > +                                if not args.FullPaths:

> > > +                                    PackageLabels[Match] =

> > > (Dependency.replace(os.path.sep,'\\n'), 'white')

> > > +                                Found = True

> > > +                        if not Found and args.Unresolved:

> > > +                            if args.Verbose:

> > > +                                print ('WARNING: Dependent package not found ' +

> > > Dependency)

> > > +                            DependentPackages.append(Dependency)

> > > +                            UnresolvedPackages.append(Dependency)

> > > +                            PackageLabels[Dependency] =

> > > (Dependency.replace(os.path.sep,'\\n'), 'white')

> > > +                    Components[DecFile][InfFile] = DependentPackages

> > > +    if AmbiguousDependencies:

> > > +        for Dependency in set(AmbiguousDependencies):

> > > +            print ('ERROR: MULTIPLE:   ' + Dependency)

> > > +        print ('Use --packages-path to provide search priority.')

> > > +        sys.exit(1)

> > > +

> > > +    #

> > > +    # Generate GraphViz dot input file contents.

> > > +    # Use networkx to detect circular dependencies.

> > > +    #

> > > +    MaxWeight = 0

> > > +    MaxWeightDecFile = list(Components.keys())[0]

> > > +    Graph = nx.DiGraph()

> > > +    Edges = []

> > > +    for DecFile in Components:

> > > +        if args.Verbose:

> > > +            print ('PACKAGE DEPENDENCIES:' + DecFile)

> > > +        AllDependencies = []

> > > +        Dependencies = set()

> > > +        for InfFile in Components[DecFile]:

> > > +            AllDependencies += Components[DecFile][InfFile]

> > > +            Dependencies =

> Dependencies.union(Components[DecFile][InfFile])

> > > +        if not args.SelfDependency:

> > > +            Dependencies = Dependencies.difference(set([DecFile]))

> > > +        for Dependency in Dependencies:

> > > +            Weight = AllDependencies.count(Dependency)

> > > +            if Weight > MaxWeight:

> > > +                MaxWeight = Weight

> > > +                MaxWeightDecFile = DecFile

> > > +            if args.Verbose:

> > > +                print ('  DEPENDENCY: Weight(' + str(Weight) + ') ' +

> Dependency)

> > > +            Edges.append('  "{Package}" -> "{Dependency}" [label =

> > > "{Weight}"];'.format(

> > > +                Package    = DecFile,

> > > +                Dependency = Dependency,

> > > +                Weight     = str(Weight) if args.Label else ''

> > > +                ))

> > > +            Graph.add_edge(DecFile, Dependency)

> > > +    Edges.sort()

> > > +

> > > +    #

> > > +    # Set fill color to yellow if a packages is part of a circular dependency

> > > +    #

> > > +    for Node in set([x for y in list(nx.simple_cycles(Graph)) for x in y]):

> > > +        PackageLabels[Node] = (PackageLabels[Node][0], 'yellow')

> > > +        print ('ERROR: CIRCULAR:   ' + Node)

> > > +

> > > +    #

> > > +    # Set fill color to pink if a package that is nested inside another

> package.

> > > +    # Set fill color to orange if a package is nested inside another package

> > and

> > > +    # is part of a circular dependency.

> > > +    #

> > > +    for DecFile in Components:

> > > +        DecPath = os.path.split (DecFile)[0]

> > > +        for DecFile2 in Components:

> > > +            if DecFile2 == DecFile:

> > > +                continue

> > > +            if os.path.commonpath ([DecPath, DecFile2]) != DecPath:

> > > +                continue

> > > +            if len(DecFile2) < len(DecPath):

> > > +                DecFile2 = DecFile

> > > +            print ('ERROR: NESTED:     ' + DecFile2)

> > > +            if PackageLabels[DecFile2][1] == 'yellow':

> > > +                PackageLabels[DecFile2] = (PackageLabels[DecFile2][0],

> 'orange')

> > > +            else:

> > > +                PackageLabels[DecFile2] = (PackageLabels[DecFile2][0], 'pink')

> > > +

> > > +    #

> > > +    # Set fill color to red for nodes that are unresolved

> > > +    #

> > > +    for Node in set(UnresolvedPackages):

> > > +        PackageLabels[Node] = (PackageLabels[Node][0], 'red')

> > > +        print ('ERROR: UNRESOLVED: ' + Node)

> > > +

> > > +    #

> > > +    # Add node statements to set node label and fill color

> > > +    #

> > > +    Nodes = []

> > > +    for Package in PackageLabels:

> > > +        Nodes.append('  "{Package}"

> > [label="{Label}",fillcolor={Color}];'.format(

> > > +            Package = Package,

> > > +            Label = PackageLabels[Package][0],

> > > +            Color = PackageLabels[Package][1]

> > > +            ))

> > > +    Nodes.sort()

> > > +

> > > +    #

> > > +    # Generate dot file from Nodes and Edges and add a Legend at top of

> > > graph

> > > +    #

> > > +    Dot = []

> > > +    Dot.append('digraph {')

> > > +    Dot.append('  rankdir=BT;')

> > > +    Dot.append('  node [shape=Mrecord,style=filled];')

> > > +    Dot.append('')

> > > +    Dot = Dot + Nodes

> > > +    Dot.append('')

> > > +    Dot = Dot + Edges

> > > +    Dot.append('')

> > > +    Dot.append('  subgraph legend {')

> > > +    Dot.append('    rank=sink;')

> > > +    Dot.append('    Unresolved     [label="Unresolved Dependency",

> > > fillcolor=red];')

> > > +    Dot.append('    Circular       [label="Circular Dependency",

> > > fillcolor=yellow];')

> > > +    Dot.append('    Nested         [label="Nested Package",

> > > fillcolor=pink];')

> > > +    Dot.append('    NestedCircular [label="Nested Package with Circular

> > > Dependency",fillcolor=orange];')

> > > +    Dot.append('  }')

> > > +    if MaxWeightDecFile:

> > > +        Dot.append('  Unresolved->"' + MaxWeightDecFile + '" [style=invis];')

> > > +    Dot.append('}')

> > > +

> > > +    if args.DotOutputFile:

> > > +        #

> > > +        # Write GraphViz dot file contents to DotOutputFile

> > > +        #

> > > +        with open(os.path.realpath(args.DotOutputFile), 'w') as File:

> > > +            File.write('\n'.join(Dot))

> > > +    if args.SvgOutputFile:

> > > +        #

> > > +        # Use GraphViz 'dot' command to generate SVG output file

> > > +        #

> > > +        args.SvgOutputFile = os.path.realpath(args.SvgOutputFile)

> > > +        try:

> > > +            Process = subprocess.Popen('dot -Tsvg',

> > > +              stdin=subprocess.PIPE,

> > > +              stdout=open(args.SvgOutputFile, 'w'),

> > > +              stderr=subprocess.PIPE,

> > > +              shell=True

> > > +              )

> > > +            Process.stdin.write ('\n'.join(Dot).encode())

> > > +            Process.communicate()

> > > +            if Process.returncode != 0:

> > > +                print ("ERROR: Can not run GraphViz 'dot' command.  Check

> install

> > > and path.")

> > > +                sys.exit(Process.returncode)

> > > +        except:

> > > +            print ("ERROR: Can not run GraphViz 'dot' command.  Check

> install

> > and

> > > path.")

> > > +            sys.exit(1)

> > > +        #

> > > +        # Display SVG file in default web browser

> > > +        #

> > > +        if args.WebBrowser:

> > > +            webbrowser.open(args.SvgOutputFile)

> > > --

> > > 2.21.0.windows.1

> >

> >

> >

>

>

> 



[-- Attachment #2: Type: text/html, Size: 67697 bytes --]

Re: [edk2-devel] [Patch 1/1] BaseTools/Scripts: Add package dependency graphing tool

[Michael D Kinney]

Ray,

Yes.  More features can be added.  Can you add to the BZ some
examples of the output you would like to see.

https://bugzilla.tianocore.org/show_bug.cgi?id=2161

Thanks,

Mike

> -----Original Message-----
> From: Ni, Ray <ray.ni@intel.com>
> Sent: Monday, December 16, 2019 12:41 AM
> To: devel@edk2.groups.io; Ni, Ray <ray.ni@intel.com>;
> Kinney, Michael D <michael.d.kinney@intel.com>
> Cc: Feng, Bob C <bob.c.feng@intel.com>; Gao, Liming
> <liming.gao@intel.com>; Sean Brogan
> <sean.brogan@microsoft.com>; Bret Barkelew
> <Bret.Barkelew@microsoft.com>
> Subject: RE: [edk2-devel] [Patch 1/1]
> BaseTools/Scripts: Add package dependency graphing tool
> 
> Mike,
> This pkg dep tool can tell through weight when a module
> depends on a new pkg.
> But it cannot tell when a module depends on another
> PCD/Protocol/Guid/Ppi/Library which belongs to an
> already-depended pkg.
> 
> Failure to tell such information may make a bad module
> (that violates the dependency rules) worse (wrongly
> depend on more interfaces).
> 
> Do you think that the tool can be enhanced in future to
> detect the case?
> 
> Thanks,
> Ray
> 
> > -----Original Message-----
> > From: devel@edk2.groups.io <devel@edk2.groups.io> On
> Behalf Of Ni, Ray
> > Sent: Monday, December 16, 2019 1:57 PM
> > To: Kinney, Michael D <michael.d.kinney@intel.com>;
> devel@edk2.groups.io
> > Cc: Feng, Bob C <bob.c.feng@intel.com>; Gao, Liming
> > <liming.gao@intel.com>; Sean Brogan
> <sean.brogan@microsoft.com>; Bret
> > Barkelew <Bret.Barkelew@microsoft.com>
> > Subject: Re: [edk2-devel] [Patch 1/1]
> BaseTools/Scripts: Add package
> > dependency graphing tool
> >
> > Mike,
> > 2 minor comments regarding the help string in below.
> >
> > > -----Original Message-----
> > > From: Kinney, Michael D
> <michael.d.kinney@intel.com>
> > > Sent: Saturday, December 14, 2019 3:45 AM
> > > To: devel@edk2.groups.io
> > > Cc: Ni, Ray <ray.ni@intel.com>; Feng, Bob C
> <bob.c.feng@intel.com>; Gao,
> > > Liming <liming.gao@intel.com>; Sean Brogan
> > <sean.brogan@microsoft.com>;
> > > Bret Barkelew <Bret.Barkelew@microsoft.com>
> > > Subject: [Patch 1/1] BaseTools/Scripts: Add package
> dependency graphing
> > > tool
> > >
> > > https://bugzilla.tianocore.org/show_bug.cgi?id=2161
> > >
> > > Add python script that recursively scans a
> directory for EDK II
> > > packages and generates GraphViz dot input that is
> used to render
> > > a graph of package dependencies in SVG format.
> > >
> > > Detects following error/warning conditions:
> > > * Ambiguous dependencies (multiple matches)
> > > * Unresolved dependencies
> > > * Circular dependencies
> > > * Nested packages
> > >
> > > usage: PackageDependencyGraph [-h] [-w WORKSPACE]
> [-p
> > PACKAGESPATH]
> > >                               [-d DOTOUTPUTFILE] [-
> o SVGOUTPUTFILE]
> > >                               [-g IGNOREDIRECTORY]
> [-k SKIPPACKAGE]
> > >                               [-s] [-u] [-l] [-f]
> [-b] [-v] [-q]
> > >                               [--debug [0-9]]
> > >
> > > Recursively scan a directory for EDK II packages
> and generate GraphViz dot
> > > input that is used to render a graph of package
> dependencies in SVG
> > format.
> > > Copyright (c) 2019, Intel Corporation. All rights
> reserved.
> > >
> > > optional arguments:
> > >   -h, --help            show this help message and
> exit
> > >   -w WORKSPACE, --workspace WORKSPACE
> > >                         Directory to recursively
> scan for EDK II packages.
> > >                         Default is current
> directory.
> > >   -p PACKAGESPATH, --packages-path PACKAGESPATH
> > >                         List of directories to
> recursively scan for EDK II
> > >                         packages.
> > >   -d DOTOUTPUTFILE, --dot-output DOTOUTPUTFILE
> > >                         DOT output filename.
> > >   -o OUTPUTFILE, --output OUTPUTFILE
> > >                         SVG output filename.
> > >   -g IGNOREDIRECTORY, --ignore-directory
> IGNOREDIRECTORY
> > >                         Name of directory to
> ignore. Option can be repeated
> > >                         to ignore multiple
> directories.
> >
> > 1. Name of directory to ignore when scanning for
> EDKII Module INFs.
> >
> > >   -k SKIPPACKAGE, --skip-package SKIPPACKAGE
> > >                         Name of EDK II Package DEC
> file to skip. Option can
> > >                         be repeated to skip
> multiple EDK II packages.
> >
> > 2. Name of EDK II Package DEC file (including .DEC
> suffix) to skip. Option can
> > Be repeated to skip multiple EDK II packages.
> >
> > >   -s, --self-dependency
> > >                         Include self links in
> dependency graph. Default is
> > >                         disabled.
> > >   -u, --unresolved      Include unresolved EDK II
> packages in dependency
> > >                         graph. Default is disabled.
> > >   -l, --label           Label links with the number
> of EDK II package
> > >                         dependencies. Default is
> disabled.
> > >   -f, --full-paths      Label package nodes with
> full path to EDK II
> > >                         package.  Default is
> disabled.
> > >   -b, --web-browser     Display SVG output file in
> default web browser.
> > >                         Default is disabled.
> > >   -v, --verbose         Increase output messages
> > >   -q, --quiet           Reduce output messages
> > >   --debug [0-9]         Set debug level
> > >
> > > Cc: Ray Ni <ray.ni@intel.com>
> > > Cc: Bob Feng <bob.c.feng@intel.com>
> > > Cc: Liming Gao <liming.gao@intel.com>
> > > Cc: Sean Brogan <sean.brogan@microsoft.com>
> > > Cc: Bret Barkelew <Bret.Barkelew@microsoft.com>
> > > Signed-off-by: Michael D Kinney
> <michael.d.kinney@intel.com>
> > > ---
> > >  BaseTools/Scripts/PackageDependencyGraph.py | 296
> > > ++++++++++++++++++++
> > >  1 file changed, 296 insertions(+)
> > >  create mode 100644
> BaseTools/Scripts/PackageDependencyGraph.py
> > >
> > > diff --git
> a/BaseTools/Scripts/PackageDependencyGraph.py
> > > b/BaseTools/Scripts/PackageDependencyGraph.py
> > > new file mode 100644
> > > index 0000000000..b3c8e41774
> > > --- /dev/null
> > > +++ b/BaseTools/Scripts/PackageDependencyGraph.py
> > > @@ -0,0 +1,296 @@
> > > +# @file
> > > +# Recursively scan a directory for EDK II packages
> and generate GraphViz
> > > dot
> > > +# input that is used to render a graph of package
> dependencies in SVG
> > > format.
> > > +#
> > > +# Copyright (c) 2019, Intel Corporation. All
> rights reserved.<BR>
> > > +# SPDX-License-Identifier: BSD-2-Clause-Patent
> > > +#
> > > +##
> > > +
> > > +import os
> > > +import sys
> > > +import argparse
> > > +import subprocess
> > > +import webbrowser
> > > +import networkx as nx
> > > +from edk2toollib.uefi.edk2.parsers.inf_parser
> import InfParser
> > > +
> > > +#
> > > +# Globals for help information
> > > +#
> > > +__prog__        = 'PackageDependencyGraph'
> > > +__copyright__   = 'Copyright (c) 2019, Intel
> Corporation. All rights
> > reserved.'
> > > +__description__ = '''Recursively scan a directory
> for EDK II packages and
> > > +generate GraphViz dot input that is used to render
> a graph of package
> > > +dependencies in SVG format.'''
> > > +
> > > +if __name__ == '__main__':
> > > +
> > > +    #
> > > +    # Create command line argument parser object
> > > +    #
> > > +    parser = argparse.ArgumentParser (prog =
> __prog__,
> > > +                                      description
> = __description__ + '\n' + __copyright__,
> > > +
> conflict_handler = 'resolve')
> > > +    parser.add_argument ("-w", "--workspace", dest
> = 'Workspace', default
> > =
> > > os.curdir,
> > > +                         help = "Directory to
> recursively scan for EDK II packages.
> > > Default is current directory.")
> > > +    parser.add_argument ("-p", "--packages-path",
> dest = 'PackagesPath',
> > > default = None,
> > > +                         help = "List of
> directories to recursively scan for EDK II
> > > packages.")
> > > +    parser.add_argument ("-d", "--dot-output",
> dest = 'DotOutputFile',
> > > +                         help = "DOT output
> filename.")
> > > +    parser.add_argument ("-o", "--output", dest =
> 'SvgOutputFile',
> > > +                         help = "SVG output
> filename.")
> > > +    parser.add_argument ("-g", "--ignore-
> directory", dest =
> > 'IgnoreDirectory',
> > > action='append', default=[],
> > > +                         help = "Name of directory
> to ignore.  Option can be repeated
> > to
> > > ignore multiple directories.")
> > > +    parser.add_argument ("-k", "--skip-package",
> dest = 'SkipPackage',
> > > action='append', default=[],
> > > +                         help = "Name of EDK II
> Package DEC file to skip.  Option can
> > be
> > > repeated to skip multiple EDK II packages.")
> > > +    parser.add_argument ("-s", "--self-
> dependency", dest =
> > > 'SelfDependency', action = "store_true", default =
> False,
> > > +                         help = "Include self
> links in dependency graph.  Default is
> > > disabled.")
> > > +    parser.add_argument ("-u", "--unresolved",
> dest = 'Unresolved', action
> > =
> > > "store_true", default=False,
> > > +                         help = "Include
> unresolved EDK II packages in dependency
> > > graph.  Default is disabled.")
> > > +    parser.add_argument ("-l", "--label", dest =
> 'Label', action =
> > "store_true",
> > > default=False,
> > > +                         help = "Label links with
> the number of EDK II package
> > > dependencies.  Default is disabled.")
> > > +    parser.add_argument ("-f", "--full-paths",
> dest = 'FullPaths', action =
> > > "store_true", default=False,
> > > +                         help = "Label package
> nodes with full path to EDK II package.
> > > Default is disabled.")
> > > +    parser.add_argument ("-b", "--web-browser",
> dest = 'WebBrowser',
> > > action = "store_true", default=False,
> > > +                         help = "Display SVG
> output file in default web browser.
> > Default
> > > is disabled.")
> > > +    parser.add_argument ("-v", "--verbose", dest =
> 'Verbose', action =
> > > "store_true",
> > > +                         help = "Increase output
> messages")
> > > +    parser.add_argument ("-q", "--quiet", dest =
> 'Quiet', action =
> > > "store_true",
> > > +                         help = "Reduce output
> messages")
> > > +    parser.add_argument ("--debug", dest =
> 'Debug', type = int, metavar =
> > > '[0-9]', choices = range (0, 10), default = 0,
> > > +                         help = "Set debug level")
> > > +
> > > +    #
> > > +    # Parse command line arguments
> > > +    #
> > > +    args = parser.parse_args ()
> > > +
> > > +    #
> > > +    # Find all EDK II package DEC files
> > > +    #
> > > +    Components = {}
> > > +    SearchPaths = [args.Workspace]
> > > +    if args.PackagesPath:
> > > +        SearchPaths +=
> args.PackagesPath.split(os.pathsep)
> > > +    SearchPaths = [os.path.realpath(x) for x in
> SearchPaths]
> > > +    for SearchPath in SearchPaths:
> > > +        for root, dirs, files in os.walk
> (SearchPath):
> > > +            for name in files:
> > > +                FilePath = os.path.join (root,
> name)
> > > +                if
> > >
> set(FilePath.split(os.sep)).intersection(set(args.Ignor
> eDirectory)) != set():
> > > +                    if args.Verbose:
> > > +                        print ('IGNORE:' +
> FilePath)
> > > +                    continue
> > > +                if
> os.path.splitext(FilePath)[1].lower() in ['.dec']:
> > > +                    DecFile = os.path.realpath
> (FilePath)
> > > +                    if os.path.split(DecFile)[1]
> in args.SkipPackage:
> > > +                        if args.Verbose:
> > > +                            print ('SKIP:' +
> DecFile)
> > > +                        continue
> > > +                    if DecFile not in Components:
> > > +                        if args.Verbose:
> > > +                            print ('PACKAGE:' +
> DecFile)
> > > +                        Components[DecFile] = {}
> > > +
> > > +    #
> > > +    # Find EDK II component INF files in each EDK
> II package
> > > +    #
> > > +    PackageLabels = {}
> > > +    UnresolvedPackages = []
> > > +    AmbiguousDependencies = []
> > > +    for DecFile in Components:
> > > +        DecPath = os.path.split (DecFile)[0]
> > > +        if DecFile not in PackageLabels:
> > > +            PackageLabels[DecFile] =
> (DecFile.replace(os.path.sep,'\\n'), 'white')
> > > +            if not args.FullPaths:
> > > +                PackagePath =
> os.path.relpath(DecFile, os.path.split(DecPath)[0])
> > > +                PackageLabels[DecFile] =
> (PackagePath.replace(os.path.sep,'\\n'),
> > > 'white')
> > > +        for root, dirs, files in os.walk
> (DecPath):
> > > +            for name in files:
> > > +                FilePath = os.path.join(root,
> name)
> > > +                if
> > >
> set(FilePath.split(os.sep)).intersection(set(args.Ignor
> eDirectory)) != set():
> > > +                    if args.Verbose:
> > > +                        print ('IGNORE:' +
> FilePath)
> > > +                    continue
> > > +                if
> os.path.splitext(FilePath)[1].lower() in ['.inf']:
> > > +                    InfFile = os.path.realpath
> (FilePath)
> > > +                    Inf = InfParser ()
> > > +                    Inf.ParseFile (InfFile)
> > > +                    DependentPackages = []
> > > +                    for Dependency in
> Inf.PackagesUsed:
> > > +                        Dependency =
> os.path.normpath(Dependency)
> > > +                        if
> os.path.split(Dependency)[1] in args.SkipPackage:
> > > +                            if args.Verbose:
> > > +                                print ('SKIP:' +
> Dependency)
> > > +                            continue
> > > +                        Found = False
> > > +                        for SearchPath in
> SearchPaths:
> > > +                            PackagePath =
> os.path.realpath(os.path.join(SearchPath,
> > > Dependency))
> > > +                            if
> os.path.exists(PackagePath):
> > > +
> DependentPackages.append(PackagePath)
> > > +                                if not
> args.FullPaths:
> > > +
> PackageLabels[PackagePath] =
> > > (Dependency.replace(os.path.sep,'\\n'), 'white')
> > > +                                Found = True
> > > +                                break
> > > +                        if not Found:
> > > +                            Count = 0
> > > +                            Match = ''
> > > +                            for DecFile2 in
> Components:
> > > +                                if
> DecFile2.endswith(Dependency):
> > > +                                    if Count == 0:
> > > +                                        Match =
> DecFile2
> > > +                                    Count = Count
> + 1
> > > +                            if Count > 1:
> > > +
> AmbiguousDependencies.append (Dependency)
> > > +                            if Count == 1:
> > > +
> DependentPackages.append(Match)
> > > +                                if not
> args.FullPaths:
> > > +
> PackageLabels[Match] =
> > > (Dependency.replace(os.path.sep,'\\n'), 'white')
> > > +                                Found = True
> > > +                        if not Found and
> args.Unresolved:
> > > +                            if args.Verbose:
> > > +                                print ('WARNING:
> Dependent package not found ' +
> > > Dependency)
> > > +
> DependentPackages.append(Dependency)
> > > +
> UnresolvedPackages.append(Dependency)
> > > +
> PackageLabels[Dependency] =
> > > (Dependency.replace(os.path.sep,'\\n'), 'white')
> > > +                    Components[DecFile][InfFile] =
> DependentPackages
> > > +    if AmbiguousDependencies:
> > > +        for Dependency in
> set(AmbiguousDependencies):
> > > +            print ('ERROR: MULTIPLE:   ' +
> Dependency)
> > > +        print ('Use --packages-path to provide
> search priority.')
> > > +        sys.exit(1)
> > > +
> > > +    #
> > > +    # Generate GraphViz dot input file contents.
> > > +    # Use networkx to detect circular
> dependencies.
> > > +    #
> > > +    MaxWeight = 0
> > > +    MaxWeightDecFile = list(Components.keys())[0]
> > > +    Graph = nx.DiGraph()
> > > +    Edges = []
> > > +    for DecFile in Components:
> > > +        if args.Verbose:
> > > +            print ('PACKAGE DEPENDENCIES:' +
> DecFile)
> > > +        AllDependencies = []
> > > +        Dependencies = set()
> > > +        for InfFile in Components[DecFile]:
> > > +            AllDependencies +=
> Components[DecFile][InfFile]
> > > +            Dependencies =
> Dependencies.union(Components[DecFile][InfFile])
> > > +        if not args.SelfDependency:
> > > +            Dependencies =
> Dependencies.difference(set([DecFile]))
> > > +        for Dependency in Dependencies:
> > > +            Weight =
> AllDependencies.count(Dependency)
> > > +            if Weight > MaxWeight:
> > > +                MaxWeight = Weight
> > > +                MaxWeightDecFile = DecFile
> > > +            if args.Verbose:
> > > +                print ('  DEPENDENCY: Weight(' +
> str(Weight) + ') ' + Dependency)
> > > +            Edges.append('  "{Package}" ->
> "{Dependency}" [label =
> > > "{Weight}"];'.format(
> > > +                Package    = DecFile,
> > > +                Dependency = Dependency,
> > > +                Weight     = str(Weight) if
> args.Label else ''
> > > +                ))
> > > +            Graph.add_edge(DecFile, Dependency)
> > > +    Edges.sort()
> > > +
> > > +    #
> > > +    # Set fill color to yellow if a packages is
> part of a circular dependency
> > > +    #
> > > +    for Node in set([x for y in
> list(nx.simple_cycles(Graph)) for x in y]):
> > > +        PackageLabels[Node] =
> (PackageLabels[Node][0], 'yellow')
> > > +        print ('ERROR: CIRCULAR:   ' + Node)
> > > +
> > > +    #
> > > +    # Set fill color to pink if a package that is
> nested inside another package.
> > > +    # Set fill color to orange if a package is
> nested inside another package
> > and
> > > +    # is part of a circular dependency.
> > > +    #
> > > +    for DecFile in Components:
> > > +        DecPath = os.path.split (DecFile)[0]
> > > +        for DecFile2 in Components:
> > > +            if DecFile2 == DecFile:
> > > +                continue
> > > +            if os.path.commonpath ([DecPath,
> DecFile2]) != DecPath:
> > > +                continue
> > > +            if len(DecFile2) < len(DecPath):
> > > +                DecFile2 = DecFile
> > > +            print ('ERROR: NESTED:     ' +
> DecFile2)
> > > +            if PackageLabels[DecFile2][1] ==
> 'yellow':
> > > +                PackageLabels[DecFile2] =
> (PackageLabels[DecFile2][0], 'orange')
> > > +            else:
> > > +                PackageLabels[DecFile2] =
> (PackageLabels[DecFile2][0], 'pink')
> > > +
> > > +    #
> > > +    # Set fill color to red for nodes that are
> unresolved
> > > +    #
> > > +    for Node in set(UnresolvedPackages):
> > > +        PackageLabels[Node] =
> (PackageLabels[Node][0], 'red')
> > > +        print ('ERROR: UNRESOLVED: ' + Node)
> > > +
> > > +    #
> > > +    # Add node statements to set node label and
> fill color
> > > +    #
> > > +    Nodes = []
> > > +    for Package in PackageLabels:
> > > +        Nodes.append('  "{Package}"
> > [label="{Label}",fillcolor={Color}];'.format(
> > > +            Package = Package,
> > > +            Label = PackageLabels[Package][0],
> > > +            Color = PackageLabels[Package][1]
> > > +            ))
> > > +    Nodes.sort()
> > > +
> > > +    #
> > > +    # Generate dot file from Nodes and Edges and
> add a Legend at top of
> > > graph
> > > +    #
> > > +    Dot = []
> > > +    Dot.append('digraph {')
> > > +    Dot.append('  rankdir=BT;')
> > > +    Dot.append('  node
> [shape=Mrecord,style=filled];')
> > > +    Dot.append('')
> > > +    Dot = Dot + Nodes
> > > +    Dot.append('')
> > > +    Dot = Dot + Edges
> > > +    Dot.append('')
> > > +    Dot.append('  subgraph legend {')
> > > +    Dot.append('    rank=sink;')
> > > +    Dot.append('    Unresolved
> [label="Unresolved Dependency",
> > > fillcolor=red];')
> > > +    Dot.append('    Circular
> [label="Circular Dependency",
> > > fillcolor=yellow];')
> > > +    Dot.append('    Nested         [label="Nested
> Package",
> > > fillcolor=pink];')
> > > +    Dot.append('    NestedCircular [label="Nested
> Package with Circular
> > > Dependency",fillcolor=orange];')
> > > +    Dot.append('  }')
> > > +    if MaxWeightDecFile:
> > > +        Dot.append('  Unresolved->"' +
> MaxWeightDecFile + '" [style=invis];')
> > > +    Dot.append('}')
> > > +
> > > +    if args.DotOutputFile:
> > > +        #
> > > +        # Write GraphViz dot file contents to
> DotOutputFile
> > > +        #
> > > +        with
> open(os.path.realpath(args.DotOutputFile), 'w') as
> File:
> > > +            File.write('\n'.join(Dot))
> > > +    if args.SvgOutputFile:
> > > +        #
> > > +        # Use GraphViz 'dot' command to generate
> SVG output file
> > > +        #
> > > +        args.SvgOutputFile =
> os.path.realpath(args.SvgOutputFile)
> > > +        try:
> > > +            Process = subprocess.Popen('dot -
> Tsvg',
> > > +              stdin=subprocess.PIPE,
> > > +              stdout=open(args.SvgOutputFile,
> 'w'),
> > > +              stderr=subprocess.PIPE,
> > > +              shell=True
> > > +              )
> > > +            Process.stdin.write
> ('\n'.join(Dot).encode())
> > > +            Process.communicate()
> > > +            if Process.returncode != 0:
> > > +                print ("ERROR: Can not run
> GraphViz 'dot' command.  Check install
> > > and path.")
> > > +                sys.exit(Process.returncode)
> > > +        except:
> > > +            print ("ERROR: Can not run GraphViz
> 'dot' command.  Check install
> > and
> > > path.")
> > > +            sys.exit(1)
> > > +        #
> > > +        # Display SVG file in default web browser
> > > +        #
> > > +        if args.WebBrowser:
> > > +            webbrowser.open(args.SvgOutputFile)
> > > --
> > > 2.21.0.windows.1
> >
> >
> > 

Re: [edk2-devel] [Patch 1/1] BaseTools/Scripts: Add package dependency graphing tool

[Michael D Kinney]

[-- Attachment #1: Type: text/plain, Size: 25742 bytes --]

Steve,

Yes.  I think a Module scoped view and a Platform scoped view would be good additions.

Can you please enter new BZ feature requests for these and add a summary of the features you would like to see in each?

Thanks,

Mike

From: Shi, Steven <steven.shi@intel.com>
Sent: Monday, December 16, 2019 12:48 AM
To: devel@edk2.groups.io; Ni, Ray <ray.ni@intel.com>; Kinney, Michael D <michael.d.kinney@intel.com>
Cc: Feng, Bob C <bob.c.feng@intel.com>; Gao, Liming <liming.gao@intel.com>; Sean Brogan <sean.brogan@microsoft.com>; Bret Barkelew <Bret.Barkelew@microsoft.com>
Subject: RE: [edk2-devel] [Patch 1/1] BaseTools/Scripts: Add package dependency graphing tool


Ray,

I think you are asking for a module-level dependency tool. 😊



Mike,

This tool is cool. Besides the package level, do you think we can go further to figure out a module level dependency tool? With module level dependency info, we can define more interesting and comprehensive Modularity Metrics to measure the edk2 architecture quality (Architectural Technical Debt).







Thanks



Steven Shi

Intel\SSG\SFE\FIE Firmware Infrastructure





> -----Original Message-----

> From: devel@edk2.groups.io<mailto:devel@edk2.groups.io> <devel@edk2.groups.io<mailto:devel@edk2.groups.io>> On Behalf Of Ni, Ray

> Sent: Monday, December 16, 2019 4:41 PM

> To: devel@edk2.groups.io<mailto:devel@edk2.groups.io>; Ni, Ray <ray.ni@intel.com<mailto:ray.ni@intel.com>>; Kinney, Michael D

> <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>>

> Cc: Feng, Bob C <bob.c.feng@intel.com<mailto:bob.c.feng@intel.com>>; Gao, Liming

> <liming.gao@intel.com<mailto:liming.gao@intel.com>>; Sean Brogan <sean.brogan@microsoft.com<mailto:sean.brogan@microsoft.com>>; Bret

> Barkelew <Bret.Barkelew@microsoft.com<mailto:Bret.Barkelew@microsoft.com>>

> Subject: Re: [edk2-devel] [Patch 1/1] BaseTools/Scripts: Add package

> dependency graphing tool

>

> Mike,

> This pkg dep tool can tell through weight when a module depends on a new

> pkg.

> But it cannot tell when a module depends on another

> PCD/Protocol/Guid/Ppi/Library which belongs to an already-depended pkg.

>

> Failure to tell such information may make a bad module (that violates the

> dependency rules) worse (wrongly depend on more interfaces).

>

> Do you think that the tool can be enhanced in future to detect the case?

>

> Thanks,

> Ray

>

> > -----Original Message-----

> > From: devel@edk2.groups.io<mailto:devel@edk2.groups.io> <devel@edk2.groups.io<mailto:devel@edk2.groups.io>> On Behalf Of Ni,

> Ray

> > Sent: Monday, December 16, 2019 1:57 PM

> > To: Kinney, Michael D <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>>;

> devel@edk2.groups.io<mailto:devel@edk2.groups.io>

> > Cc: Feng, Bob C <bob.c.feng@intel.com<mailto:bob.c.feng@intel.com>>; Gao, Liming

> > <liming.gao@intel.com<mailto:liming.gao@intel.com>>; Sean Brogan <sean.brogan@microsoft.com<mailto:sean.brogan@microsoft.com>>;

> Bret

> > Barkelew <Bret.Barkelew@microsoft.com<mailto:Bret.Barkelew@microsoft.com>>

> > Subject: Re: [edk2-devel] [Patch 1/1] BaseTools/Scripts: Add package

> > dependency graphing tool

> >

> > Mike,

> > 2 minor comments regarding the help string in below.

> >

> > > -----Original Message-----

> > > From: Kinney, Michael D <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>>

> > > Sent: Saturday, December 14, 2019 3:45 AM

> > > To: devel@edk2.groups.io<mailto:devel@edk2.groups.io>

> > > Cc: Ni, Ray <ray.ni@intel.com<mailto:ray.ni@intel.com>>; Feng, Bob C <bob.c.feng@intel.com<mailto:bob.c.feng@intel.com>>;

> Gao,

> > > Liming <liming.gao@intel.com<mailto:liming.gao@intel.com>>; Sean Brogan

> > <sean.brogan@microsoft.com<mailto:sean.brogan@microsoft.com>>;

> > > Bret Barkelew <Bret.Barkelew@microsoft.com<mailto:Bret.Barkelew@microsoft.com>>

> > > Subject: [Patch 1/1] BaseTools/Scripts: Add package dependency

> graphing

> > > tool

> > >

> > > https://bugzilla.tianocore.org/show_bug.cgi?id=2161

> > >

> > > Add python script that recursively scans a directory for EDK II

> > > packages and generates GraphViz dot input that is used to render

> > > a graph of package dependencies in SVG format.

> > >

> > > Detects following error/warning conditions:

> > > * Ambiguous dependencies (multiple matches)

> > > * Unresolved dependencies

> > > * Circular dependencies

> > > * Nested packages

> > >

> > > usage: PackageDependencyGraph [-h] [-w WORKSPACE] [-p

> > PACKAGESPATH]

> > >                               [-d DOTOUTPUTFILE] [-o SVGOUTPUTFILE]

> > >                               [-g IGNOREDIRECTORY] [-k SKIPPACKAGE]

> > >                               [-s] [-u] [-l] [-f] [-b] [-v] [-q]

> > >                               [--debug [0-9]]

> > >

> > > Recursively scan a directory for EDK II packages and generate GraphViz

> dot

> > > input that is used to render a graph of package dependencies in SVG

> > format.

> > > Copyright (c) 2019, Intel Corporation. All rights reserved.

> > >

> > > optional arguments:

> > >   -h, --help            show this help message and exit

> > >   -w WORKSPACE, --workspace WORKSPACE

> > >                         Directory to recursively scan for EDK II packages.

> > >                         Default is current directory.

> > >   -p PACKAGESPATH, --packages-path PACKAGESPATH

> > >                         List of directories to recursively scan for EDK II

> > >                         packages.

> > >   -d DOTOUTPUTFILE, --dot-output DOTOUTPUTFILE

> > >                         DOT output filename.

> > >   -o OUTPUTFILE, --output OUTPUTFILE

> > >                         SVG output filename.

> > >   -g IGNOREDIRECTORY, --ignore-directory IGNOREDIRECTORY

> > >                         Name of directory to ignore. Option can be repeated

> > >                         to ignore multiple directories.

> >

> > 1. Name of directory to ignore when scanning for EDKII Module INFs.

> >

> > >   -k SKIPPACKAGE, --skip-package SKIPPACKAGE

> > >                         Name of EDK II Package DEC file to skip. Option can

> > >                         be repeated to skip multiple EDK II packages.

> >

> > 2. Name of EDK II Package DEC file (including .DEC suffix) to skip. Option

> can

> > Be repeated to skip multiple EDK II packages.

> >

> > >   -s, --self-dependency

> > >                         Include self links in dependency graph. Default is

> > >                         disabled.

> > >   -u, --unresolved      Include unresolved EDK II packages in dependency

> > >                         graph. Default is disabled.

> > >   -l, --label           Label links with the number of EDK II package

> > >                         dependencies. Default is disabled.

> > >   -f, --full-paths      Label package nodes with full path to EDK II

> > >                         package.  Default is disabled.

> > >   -b, --web-browser     Display SVG output file in default web browser.

> > >                         Default is disabled.

> > >   -v, --verbose         Increase output messages

> > >   -q, --quiet           Reduce output messages

> > >   --debug [0-9]         Set debug level

> > >

> > > Cc: Ray Ni <ray.ni@intel.com<mailto:ray.ni@intel.com>>

> > > Cc: Bob Feng <bob.c.feng@intel.com<mailto:bob.c.feng@intel.com>>

> > > Cc: Liming Gao <liming.gao@intel.com<mailto:liming.gao@intel.com>>

> > > Cc: Sean Brogan <sean.brogan@microsoft.com<mailto:sean.brogan@microsoft.com>>

> > > Cc: Bret Barkelew <Bret.Barkelew@microsoft.com<mailto:Bret.Barkelew@microsoft.com>>

> > > Signed-off-by: Michael D Kinney <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>>

> > > ---

> > >  BaseTools/Scripts/PackageDependencyGraph.py | 296

> > > ++++++++++++++++++++

> > >  1 file changed, 296 insertions(+)

> > >  create mode 100644 BaseTools/Scripts/PackageDependencyGraph.py

> > >

> > > diff --git a/BaseTools/Scripts/PackageDependencyGraph.py

> > > b/BaseTools/Scripts/PackageDependencyGraph.py

> > > new file mode 100644

> > > index 0000000000..b3c8e41774

> > > --- /dev/null

> > > +++ b/BaseTools/Scripts/PackageDependencyGraph.py

> > > @@ -0,0 +1,296 @@

> > > +# @file

> > > +# Recursively scan a directory for EDK II packages and generate

> GraphViz

> > > dot

> > > +# input that is used to render a graph of package dependencies in SVG

> > > format.

> > > +#

> > > +# Copyright (c) 2019, Intel Corporation. All rights reserved.<BR>

> > > +# SPDX-License-Identifier: BSD-2-Clause-Patent

> > > +#

> > > +##

> > > +

> > > +import os

> > > +import sys

> > > +import argparse

> > > +import subprocess

> > > +import webbrowser

> > > +import networkx as nx

> > > +from edk2toollib.uefi.edk2.parsers.inf_parser import InfParser

> > > +

> > > +#

> > > +# Globals for help information

> > > +#

> > > +__prog__        = 'PackageDependencyGraph'

> > > +__copyright__   = 'Copyright (c) 2019, Intel Corporation. All rights

> > reserved.'

> > > +__description__ = '''Recursively scan a directory for EDK II packages and

> > > +generate GraphViz dot input that is used to render a graph of package

> > > +dependencies in SVG format.'''

> > > +

> > > +if __name__ == '__main__':

> > > +

> > > +    #

> > > +    # Create command line argument parser object

> > > +    #

> > > +    parser = argparse.ArgumentParser (prog = __prog__,

> > > +                                      description = __description__ + '\n' +

> __copyright__,

> > > +                                      conflict_handler = 'resolve')

> > > +    parser.add_argument ("-w", "--workspace", dest = 'Workspace',

> default

> > =

> > > os.curdir,

> > > +                         help = "Directory to recursively scan for EDK II packages.

> > > Default is current directory.")

> > > +    parser.add_argument ("-p", "--packages-path", dest = 'PackagesPath',

> > > default = None,

> > > +                         help = "List of directories to recursively scan for EDK II

> > > packages.")

> > > +    parser.add_argument ("-d", "--dot-output", dest = 'DotOutputFile',

> > > +                         help = "DOT output filename.")

> > > +    parser.add_argument ("-o", "--output", dest = 'SvgOutputFile',

> > > +                         help = "SVG output filename.")

> > > +    parser.add_argument ("-g", "--ignore-directory", dest =

> > 'IgnoreDirectory',

> > > action='append', default=[],

> > > +                         help = "Name of directory to ignore.  Option can be

> repeated

> > to

> > > ignore multiple directories.")

> > > +    parser.add_argument ("-k", "--skip-package", dest = 'SkipPackage',

> > > action='append', default=[],

> > > +                         help = "Name of EDK II Package DEC file to skip.  Option

> can

> > be

> > > repeated to skip multiple EDK II packages.")

> > > +    parser.add_argument ("-s", "--self-dependency", dest =

> > > 'SelfDependency', action = "store_true", default = False,

> > > +                         help = "Include self links in dependency graph.  Default is

> > > disabled.")

> > > +    parser.add_argument ("-u", "--unresolved", dest = 'Unresolved',

> action

> > =

> > > "store_true", default=False,

> > > +                         help = "Include unresolved EDK II packages in dependency

> > > graph.  Default is disabled.")

> > > +    parser.add_argument ("-l", "--label", dest = 'Label', action =

> > "store_true",

> > > default=False,

> > > +                         help = "Label links with the number of EDK II package

> > > dependencies.  Default is disabled.")

> > > +    parser.add_argument ("-f", "--full-paths", dest = 'FullPaths', action =

> > > "store_true", default=False,

> > > +                         help = "Label package nodes with full path to EDK II

> package.

> > > Default is disabled.")

> > > +    parser.add_argument ("-b", "--web-browser", dest = 'WebBrowser',

> > > action = "store_true", default=False,

> > > +                         help = "Display SVG output file in default web browser.

> > Default

> > > is disabled.")

> > > +    parser.add_argument ("-v", "--verbose", dest = 'Verbose', action =

> > > "store_true",

> > > +                         help = "Increase output messages")

> > > +    parser.add_argument ("-q", "--quiet", dest = 'Quiet', action =

> > > "store_true",

> > > +                         help = "Reduce output messages")

> > > +    parser.add_argument ("--debug", dest = 'Debug', type = int, metavar =

> > > '[0-9]', choices = range (0, 10), default = 0,

> > > +                         help = "Set debug level")

> > > +

> > > +    #

> > > +    # Parse command line arguments

> > > +    #

> > > +    args = parser.parse_args ()

> > > +

> > > +    #

> > > +    # Find all EDK II package DEC files

> > > +    #

> > > +    Components = {}

> > > +    SearchPaths = [args.Workspace]

> > > +    if args.PackagesPath:

> > > +        SearchPaths += args.PackagesPath.split(os.pathsep)

> > > +    SearchPaths = [os.path.realpath(x) for x in SearchPaths]

> > > +    for SearchPath in SearchPaths:

> > > +        for root, dirs, files in os.walk (SearchPath):

> > > +            for name in files:

> > > +                FilePath = os.path.join (root, name)

> > > +                if

> > > set(FilePath.split(os.sep)).intersection(set(args.IgnoreDirectory)) != set():

> > > +                    if args.Verbose:

> > > +                        print ('IGNORE:' + FilePath)

> > > +                    continue

> > > +                if os.path.splitext(FilePath)[1].lower() in ['.dec']:

> > > +                    DecFile = os.path.realpath (FilePath)

> > > +                    if os.path.split(DecFile)[1] in args.SkipPackage:

> > > +                        if args.Verbose:

> > > +                            print ('SKIP:' + DecFile)

> > > +                        continue

> > > +                    if DecFile not in Components:

> > > +                        if args.Verbose:

> > > +                            print ('PACKAGE:' + DecFile)

> > > +                        Components[DecFile] = {}

> > > +

> > > +    #

> > > +    # Find EDK II component INF files in each EDK II package

> > > +    #

> > > +    PackageLabels = {}

> > > +    UnresolvedPackages = []

> > > +    AmbiguousDependencies = []

> > > +    for DecFile in Components:

> > > +        DecPath = os.path.split (DecFile)[0]

> > > +        if DecFile not in PackageLabels:

> > > +            PackageLabels[DecFile] = (DecFile.replace(os.path.sep,'\\n'),

> 'white')

> > > +            if not args.FullPaths:

> > > +                PackagePath = os.path.relpath(DecFile,

> os.path.split(DecPath)[0])

> > > +                PackageLabels[DecFile] =

> (PackagePath.replace(os.path.sep,'\\n'),

> > > 'white')

> > > +        for root, dirs, files in os.walk (DecPath):

> > > +            for name in files:

> > > +                FilePath = os.path.join(root, name)

> > > +                if

> > > set(FilePath.split(os.sep)).intersection(set(args.IgnoreDirectory)) != set():

> > > +                    if args.Verbose:

> > > +                        print ('IGNORE:' + FilePath)

> > > +                    continue

> > > +                if os.path.splitext(FilePath)[1].lower() in ['.inf']:

> > > +                    InfFile = os.path.realpath (FilePath)

> > > +                    Inf = InfParser ()

> > > +                    Inf.ParseFile (InfFile)

> > > +                    DependentPackages = []

> > > +                    for Dependency in Inf.PackagesUsed:

> > > +                        Dependency = os.path.normpath(Dependency)

> > > +                        if os.path.split(Dependency)[1] in args.SkipPackage:

> > > +                            if args.Verbose:

> > > +                                print ('SKIP:' + Dependency)

> > > +                            continue

> > > +                        Found = False

> > > +                        for SearchPath in SearchPaths:

> > > +                            PackagePath = os.path.realpath(os.path.join(SearchPath,

> > > Dependency))

> > > +                            if os.path.exists(PackagePath):

> > > +                                DependentPackages.append(PackagePath)

> > > +                                if not args.FullPaths:

> > > +                                    PackageLabels[PackagePath] =

> > > (Dependency.replace(os.path.sep,'\\n'), 'white')

> > > +                                Found = True

> > > +                                break

> > > +                        if not Found:

> > > +                            Count = 0

> > > +                            Match = ''

> > > +                            for DecFile2 in Components:

> > > +                                if DecFile2.endswith(Dependency):

> > > +                                    if Count == 0:

> > > +                                        Match = DecFile2

> > > +                                    Count = Count + 1

> > > +                            if Count > 1:

> > > +                                AmbiguousDependencies.append (Dependency)

> > > +                            if Count == 1:

> > > +                                DependentPackages.append(Match)

> > > +                                if not args.FullPaths:

> > > +                                    PackageLabels[Match] =

> > > (Dependency.replace(os.path.sep,'\\n'), 'white')

> > > +                                Found = True

> > > +                        if not Found and args.Unresolved:

> > > +                            if args.Verbose:

> > > +                                print ('WARNING: Dependent package not found ' +

> > > Dependency)

> > > +                            DependentPackages.append(Dependency)

> > > +                            UnresolvedPackages.append(Dependency)

> > > +                            PackageLabels[Dependency] =

> > > (Dependency.replace(os.path.sep,'\\n'), 'white')

> > > +                    Components[DecFile][InfFile] = DependentPackages

> > > +    if AmbiguousDependencies:

> > > +        for Dependency in set(AmbiguousDependencies):

> > > +            print ('ERROR: MULTIPLE:   ' + Dependency)

> > > +        print ('Use --packages-path to provide search priority.')

> > > +        sys.exit(1)

> > > +

> > > +    #

> > > +    # Generate GraphViz dot input file contents.

> > > +    # Use networkx to detect circular dependencies.

> > > +    #

> > > +    MaxWeight = 0

> > > +    MaxWeightDecFile = list(Components.keys())[0]

> > > +    Graph = nx.DiGraph()

> > > +    Edges = []

> > > +    for DecFile in Components:

> > > +        if args.Verbose:

> > > +            print ('PACKAGE DEPENDENCIES:' + DecFile)

> > > +        AllDependencies = []

> > > +        Dependencies = set()

> > > +        for InfFile in Components[DecFile]:

> > > +            AllDependencies += Components[DecFile][InfFile]

> > > +            Dependencies =

> Dependencies.union(Components[DecFile][InfFile])

> > > +        if not args.SelfDependency:

> > > +            Dependencies = Dependencies.difference(set([DecFile]))

> > > +        for Dependency in Dependencies:

> > > +            Weight = AllDependencies.count(Dependency)

> > > +            if Weight > MaxWeight:

> > > +                MaxWeight = Weight

> > > +                MaxWeightDecFile = DecFile

> > > +            if args.Verbose:

> > > +                print ('  DEPENDENCY: Weight(' + str(Weight) + ') ' +

> Dependency)

> > > +            Edges.append('  "{Package}" -> "{Dependency}" [label =

> > > "{Weight}"];'.format(

> > > +                Package    = DecFile,

> > > +                Dependency = Dependency,

> > > +                Weight     = str(Weight) if args.Label else ''

> > > +                ))

> > > +            Graph.add_edge(DecFile, Dependency)

> > > +    Edges.sort()

> > > +

> > > +    #

> > > +    # Set fill color to yellow if a packages is part of a circular dependency

> > > +    #

> > > +    for Node in set([x for y in list(nx.simple_cycles(Graph)) for x in y]):

> > > +        PackageLabels[Node] = (PackageLabels[Node][0], 'yellow')

> > > +        print ('ERROR: CIRCULAR:   ' + Node)

> > > +

> > > +    #

> > > +    # Set fill color to pink if a package that is nested inside another

> package.

> > > +    # Set fill color to orange if a package is nested inside another package

> > and

> > > +    # is part of a circular dependency.

> > > +    #

> > > +    for DecFile in Components:

> > > +        DecPath = os.path.split (DecFile)[0]

> > > +        for DecFile2 in Components:

> > > +            if DecFile2 == DecFile:

> > > +                continue

> > > +            if os.path.commonpath ([DecPath, DecFile2]) != DecPath:

> > > +                continue

> > > +            if len(DecFile2) < len(DecPath):

> > > +                DecFile2 = DecFile

> > > +            print ('ERROR: NESTED:     ' + DecFile2)

> > > +            if PackageLabels[DecFile2][1] == 'yellow':

> > > +                PackageLabels[DecFile2] = (PackageLabels[DecFile2][0],

> 'orange')

> > > +            else:

> > > +                PackageLabels[DecFile2] = (PackageLabels[DecFile2][0], 'pink')

> > > +

> > > +    #

> > > +    # Set fill color to red for nodes that are unresolved

> > > +    #

> > > +    for Node in set(UnresolvedPackages):

> > > +        PackageLabels[Node] = (PackageLabels[Node][0], 'red')

> > > +        print ('ERROR: UNRESOLVED: ' + Node)

> > > +

> > > +    #

> > > +    # Add node statements to set node label and fill color

> > > +    #

> > > +    Nodes = []

> > > +    for Package in PackageLabels:

> > > +        Nodes.append('  "{Package}"

> > [label="{Label}",fillcolor={Color}];'.format(

> > > +            Package = Package,

> > > +            Label = PackageLabels[Package][0],

> > > +            Color = PackageLabels[Package][1]

> > > +            ))

> > > +    Nodes.sort()

> > > +

> > > +    #

> > > +    # Generate dot file from Nodes and Edges and add a Legend at top of

> > > graph

> > > +    #

> > > +    Dot = []

> > > +    Dot.append('digraph {')

> > > +    Dot.append('  rankdir=BT;')

> > > +    Dot.append('  node [shape=Mrecord,style=filled];')

> > > +    Dot.append('')

> > > +    Dot = Dot + Nodes

> > > +    Dot.append('')

> > > +    Dot = Dot + Edges

> > > +    Dot.append('')

> > > +    Dot.append('  subgraph legend {')

> > > +    Dot.append('    rank=sink;')

> > > +    Dot.append('    Unresolved     [label="Unresolved Dependency",

> > > fillcolor=red];')

> > > +    Dot.append('    Circular       [label="Circular Dependency",

> > > fillcolor=yellow];')

> > > +    Dot.append('    Nested         [label="Nested Package",

> > > fillcolor=pink];')

> > > +    Dot.append('    NestedCircular [label="Nested Package with Circular

> > > Dependency",fillcolor=orange];')

> > > +    Dot.append('  }')

> > > +    if MaxWeightDecFile:

> > > +        Dot.append('  Unresolved->"' + MaxWeightDecFile + '" [style=invis];')

> > > +    Dot.append('}')

> > > +

> > > +    if args.DotOutputFile:

> > > +        #

> > > +        # Write GraphViz dot file contents to DotOutputFile

> > > +        #

> > > +        with open(os.path.realpath(args.DotOutputFile), 'w') as File:

> > > +            File.write('\n'.join(Dot))

> > > +    if args.SvgOutputFile:

> > > +        #

> > > +        # Use GraphViz 'dot' command to generate SVG output file

> > > +        #

> > > +        args.SvgOutputFile = os.path.realpath(args.SvgOutputFile)

> > > +        try:

> > > +            Process = subprocess.Popen('dot -Tsvg',

> > > +              stdin=subprocess.PIPE,

> > > +              stdout=open(args.SvgOutputFile, 'w'),

> > > +              stderr=subprocess.PIPE,

> > > +              shell=True

> > > +              )

> > > +            Process.stdin.write ('\n'.join(Dot).encode())

> > > +            Process.communicate()

> > > +            if Process.returncode != 0:

> > > +                print ("ERROR: Can not run GraphViz 'dot' command.  Check

> install

> > > and path.")

> > > +                sys.exit(Process.returncode)

> > > +        except:

> > > +            print ("ERROR: Can not run GraphViz 'dot' command.  Check

> install

> > and

> > > path.")

> > > +            sys.exit(1)

> > > +        #

> > > +        # Display SVG file in default web browser

> > > +        #

> > > +        if args.WebBrowser:

> > > +            webbrowser.open(args.SvgOutputFile)

> > > --

> > > 2.21.0.windows.1

> >

> >

> >

>

>

> 



[-- Attachment #2: Type: text/html, Size: 113125 bytes --]

Re: [edk2-devel] [Patch 1/1] BaseTools/Scripts: Add package dependency graphing tool

[Sean]

[-- Attachment #1: Type: text/plain, Size: 27283 bytes --]

Mike/Steven/Ray,

We have had a tool for a couple of years with similar functionality.  It is a python tool you run on your code base and it creates a single (local) webpage that you can then use to build charts.

Please give it a try and see if you think it is useful.

Here is the tool.
https://github.com/spbrogan/edk2_dep

And here is a html report run on edk2
https://github.com/spbrogan/edk2_dep/blob/sample/edk2/Edk2_core.html

Thanks
Sean


From: Kinney, Michael D <michael.d.kinney@intel.com>
Sent: Monday, December 16, 2019 9:16 AM
To: Shi, Steven <steven.shi@intel.com>; devel@edk2.groups.io; Ni, Ray <ray.ni@intel.com>; Kinney, Michael D <michael.d.kinney@intel.com>
Cc: Feng, Bob C <bob.c.feng@intel.com>; Gao, Liming <liming.gao@intel.com>; Sean Brogan <sean.brogan@microsoft.com>; Bret Barkelew <Bret.Barkelew@microsoft.com>
Subject: [EXTERNAL] RE: [edk2-devel] [Patch 1/1] BaseTools/Scripts: Add package dependency graphing tool

Steve,

Yes.  I think a Module scoped view and a Platform scoped view would be good additions.

Can you please enter new BZ feature requests for these and add a summary of the features you would like to see in each?

Thanks,

Mike

From: Shi, Steven <steven.shi@intel.com<mailto:steven.shi@intel.com>>
Sent: Monday, December 16, 2019 12:48 AM
To: devel@edk2.groups.io<mailto:devel@edk2.groups.io>; Ni, Ray <ray.ni@intel.com<mailto:ray.ni@intel.com>>; Kinney, Michael D <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>>
Cc: Feng, Bob C <bob.c.feng@intel.com<mailto:bob.c.feng@intel.com>>; Gao, Liming <liming.gao@intel.com<mailto:liming.gao@intel.com>>; Sean Brogan <sean.brogan@microsoft.com<mailto:sean.brogan@microsoft.com>>; Bret Barkelew <Bret.Barkelew@microsoft.com<mailto:Bret.Barkelew@microsoft.com>>
Subject: RE: [edk2-devel] [Patch 1/1] BaseTools/Scripts: Add package dependency graphing tool


Ray,

I think you are asking for a module-level dependency tool. 😊



Mike,

This tool is cool. Besides the package level, do you think we can go further to figure out a module level dependency tool? With module level dependency info, we can define more interesting and comprehensive Modularity Metrics to measure the edk2 architecture quality (Architectural Technical Debt).







Thanks



Steven Shi

Intel\SSG\SFE\FIE Firmware Infrastructure





> -----Original Message-----

> From: devel@edk2.groups.io<mailto:devel@edk2.groups.io> <devel@edk2.groups.io<mailto:devel@edk2.groups.io>> On Behalf Of Ni, Ray

> Sent: Monday, December 16, 2019 4:41 PM

> To: devel@edk2.groups.io<mailto:devel@edk2.groups.io>; Ni, Ray <ray.ni@intel.com<mailto:ray.ni@intel.com>>; Kinney, Michael D

> <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>>

> Cc: Feng, Bob C <bob.c.feng@intel.com<mailto:bob.c.feng@intel.com>>; Gao, Liming

> <liming.gao@intel.com<mailto:liming.gao@intel.com>>; Sean Brogan <sean.brogan@microsoft.com<mailto:sean.brogan@microsoft.com>>; Bret

> Barkelew <Bret.Barkelew@microsoft.com<mailto:Bret.Barkelew@microsoft.com>>

> Subject: Re: [edk2-devel] [Patch 1/1] BaseTools/Scripts: Add package

> dependency graphing tool

>

> Mike,

> This pkg dep tool can tell through weight when a module depends on a new

> pkg.

> But it cannot tell when a module depends on another

> PCD/Protocol/Guid/Ppi/Library which belongs to an already-depended pkg.

>

> Failure to tell such information may make a bad module (that violates the

> dependency rules) worse (wrongly depend on more interfaces).

>

> Do you think that the tool can be enhanced in future to detect the case?

>

> Thanks,

> Ray

>

> > -----Original Message-----

> > From: devel@edk2.groups.io<mailto:devel@edk2.groups.io> <devel@edk2.groups.io<mailto:devel@edk2.groups.io>> On Behalf Of Ni,

> Ray

> > Sent: Monday, December 16, 2019 1:57 PM

> > To: Kinney, Michael D <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>>;

> devel@edk2.groups.io<mailto:devel@edk2.groups.io>

> > Cc: Feng, Bob C <bob.c.feng@intel.com<mailto:bob.c.feng@intel.com>>; Gao, Liming

> > <liming.gao@intel.com<mailto:liming.gao@intel.com>>; Sean Brogan <sean.brogan@microsoft.com<mailto:sean.brogan@microsoft.com>>;

> Bret

> > Barkelew <Bret.Barkelew@microsoft.com<mailto:Bret.Barkelew@microsoft.com>>

> > Subject: Re: [edk2-devel] [Patch 1/1] BaseTools/Scripts: Add package

> > dependency graphing tool

> >

> > Mike,

> > 2 minor comments regarding the help string in below.

> >

> > > -----Original Message-----

> > > From: Kinney, Michael D <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>>

> > > Sent: Saturday, December 14, 2019 3:45 AM

> > > To: devel@edk2.groups.io<mailto:devel@edk2.groups.io>

> > > Cc: Ni, Ray <ray.ni@intel.com<mailto:ray.ni@intel.com>>; Feng, Bob C <bob.c.feng@intel.com<mailto:bob.c.feng@intel.com>>;

> Gao,

> > > Liming <liming.gao@intel.com<mailto:liming.gao@intel.com>>; Sean Brogan

> > <sean.brogan@microsoft.com<mailto:sean.brogan@microsoft.com>>;

> > > Bret Barkelew <Bret.Barkelew@microsoft.com<mailto:Bret.Barkelew@microsoft.com>>

> > > Subject: [Patch 1/1] BaseTools/Scripts: Add package dependency

> graphing

> > > tool

> > >

> > > https://bugzilla.tianocore.org/show_bug.cgi?id=2161<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugzilla.tianocore.org%2Fshow_bug.cgi%3Fid%3D2161&data=02%7C01%7Csean.brogan%40microsoft.com%7Cf7fb1953185b4a17f27408d7824ba488%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637121133718397648&sdata=NE82lFZmNfIcNjUfo49D%2Bksn94atxzdx7F2j%2FWSME0E%3D&reserved=0>

> > >

> > > Add python script that recursively scans a directory for EDK II

> > > packages and generates GraphViz dot input that is used to render

> > > a graph of package dependencies in SVG format.

> > >

> > > Detects following error/warning conditions:

> > > * Ambiguous dependencies (multiple matches)

> > > * Unresolved dependencies

> > > * Circular dependencies

> > > * Nested packages

> > >

> > > usage: PackageDependencyGraph [-h] [-w WORKSPACE] [-p

> > PACKAGESPATH]

> > >                               [-d DOTOUTPUTFILE] [-o SVGOUTPUTFILE]

> > >                               [-g IGNOREDIRECTORY] [-k SKIPPACKAGE]

> > >                               [-s] [-u] [-l] [-f] [-b] [-v] [-q]

> > >                               [--debug [0-9]]

> > >

> > > Recursively scan a directory for EDK II packages and generate GraphViz

> dot

> > > input that is used to render a graph of package dependencies in SVG

> > format.

> > > Copyright (c) 2019, Intel Corporation. All rights reserved.

> > >

> > > optional arguments:

> > >   -h, --help            show this help message and exit

> > >   -w WORKSPACE, --workspace WORKSPACE

> > >                         Directory to recursively scan for EDK II packages.

> > >                         Default is current directory.

> > >   -p PACKAGESPATH, --packages-path PACKAGESPATH

> > >                         List of directories to recursively scan for EDK II

> > >                         packages.

> > >   -d DOTOUTPUTFILE, --dot-output DOTOUTPUTFILE

> > >                         DOT output filename.

> > >   -o OUTPUTFILE, --output OUTPUTFILE

> > >                         SVG output filename.

> > >   -g IGNOREDIRECTORY, --ignore-directory IGNOREDIRECTORY

> > >                         Name of directory to ignore. Option can be repeated

> > >                         to ignore multiple directories.

> >

> > 1. Name of directory to ignore when scanning for EDKII Module INFs.

> >

> > >   -k SKIPPACKAGE, --skip-package SKIPPACKAGE

> > >                         Name of EDK II Package DEC file to skip. Option can

> > >                         be repeated to skip multiple EDK II packages.

> >

> > 2. Name of EDK II Package DEC file (including .DEC suffix) to skip. Option

> can

> > Be repeated to skip multiple EDK II packages.

> >

> > >   -s, --self-dependency

> > >                         Include self links in dependency graph. Default is

> > >                         disabled.

> > >   -u, --unresolved      Include unresolved EDK II packages in dependency

> > >                         graph. Default is disabled.

> > >   -l, --label           Label links with the number of EDK II package

> > >                         dependencies. Default is disabled.

> > >   -f, --full-paths      Label package nodes with full path to EDK II

> > >                         package.  Default is disabled.

> > >   -b, --web-browser     Display SVG output file in default web browser.

> > >                         Default is disabled.

> > >   -v, --verbose         Increase output messages

> > >   -q, --quiet           Reduce output messages

> > >   --debug [0-9]         Set debug level

> > >

> > > Cc: Ray Ni <ray.ni@intel.com<mailto:ray.ni@intel.com>>

> > > Cc: Bob Feng <bob.c.feng@intel.com<mailto:bob.c.feng@intel.com>>

> > > Cc: Liming Gao <liming.gao@intel.com<mailto:liming.gao@intel.com>>

> > > Cc: Sean Brogan <sean.brogan@microsoft.com<mailto:sean.brogan@microsoft.com>>

> > > Cc: Bret Barkelew <Bret.Barkelew@microsoft.com<mailto:Bret.Barkelew@microsoft.com>>

> > > Signed-off-by: Michael D Kinney <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>>

> > > ---

> > >  BaseTools/Scripts/PackageDependencyGraph.py | 296

> > > ++++++++++++++++++++

> > >  1 file changed, 296 insertions(+)

> > >  create mode 100644 BaseTools/Scripts/PackageDependencyGraph.py

> > >

> > > diff --git a/BaseTools/Scripts/PackageDependencyGraph.py

> > > b/BaseTools/Scripts/PackageDependencyGraph.py

> > > new file mode 100644

> > > index 0000000000..b3c8e41774

> > > --- /dev/null

> > > +++ b/BaseTools/Scripts/PackageDependencyGraph.py

> > > @@ -0,0 +1,296 @@

> > > +# @file

> > > +# Recursively scan a directory for EDK II packages and generate

> GraphViz

> > > dot

> > > +# input that is used to render a graph of package dependencies in SVG

> > > format.

> > > +#

> > > +# Copyright (c) 2019, Intel Corporation. All rights reserved.<BR>

> > > +# SPDX-License-Identifier: BSD-2-Clause-Patent

> > > +#

> > > +##

> > > +

> > > +import os

> > > +import sys

> > > +import argparse

> > > +import subprocess

> > > +import webbrowser

> > > +import networkx as nx

> > > +from edk2toollib.uefi.edk2.parsers.inf_parser import InfParser

> > > +

> > > +#

> > > +# Globals for help information

> > > +#

> > > +__prog__        = 'PackageDependencyGraph'

> > > +__copyright__   = 'Copyright (c) 2019, Intel Corporation. All rights

> > reserved.'

> > > +__description__ = '''Recursively scan a directory for EDK II packages and

> > > +generate GraphViz dot input that is used to render a graph of package

> > > +dependencies in SVG format.'''

> > > +

> > > +if __name__ == '__main__':

> > > +

> > > +    #

> > > +    # Create command line argument parser object

> > > +    #

> > > +    parser = argparse.ArgumentParser (prog = __prog__,

> > > +                                      description = __description__ + '\n' +

> __copyright__,

> > > +                                      conflict_handler = 'resolve')

> > > +    parser.add_argument ("-w", "--workspace", dest = 'Workspace',

> default

> > =

> > > os.curdir,

> > > +                         help = "Directory to recursively scan for EDK II packages.

> > > Default is current directory.")

> > > +    parser.add_argument ("-p", "--packages-path", dest = 'PackagesPath',

> > > default = None,

> > > +                         help = "List of directories to recursively scan for EDK II

> > > packages.")

> > > +    parser.add_argument ("-d", "--dot-output", dest = 'DotOutputFile',

> > > +                         help = "DOT output filename.")

> > > +    parser.add_argument ("-o", "--output", dest = 'SvgOutputFile',

> > > +                         help = "SVG output filename.")

> > > +    parser.add_argument ("-g", "--ignore-directory", dest =

> > 'IgnoreDirectory',

> > > action='append', default=[],

> > > +                         help = "Name of directory to ignore.  Option can be

> repeated

> > to

> > > ignore multiple directories.")

> > > +    parser.add_argument ("-k", "--skip-package", dest = 'SkipPackage',

> > > action='append', default=[],

> > > +                         help = "Name of EDK II Package DEC file to skip.  Option

> can

> > be

> > > repeated to skip multiple EDK II packages.")

> > > +    parser.add_argument ("-s", "--self-dependency", dest =

> > > 'SelfDependency', action = "store_true", default = False,

> > > +                         help = "Include self links in dependency graph.  Default is

> > > disabled.")

> > > +    parser.add_argument ("-u", "--unresolved", dest = 'Unresolved',

> action

> > =

> > > "store_true", default=False,

> > > +                         help = "Include unresolved EDK II packages in dependency

> > > graph.  Default is disabled.")

> > > +    parser.add_argument ("-l", "--label", dest = 'Label', action =

> > "store_true",

> > > default=False,

> > > +                         help = "Label links with the number of EDK II package

> > > dependencies.  Default is disabled.")

> > > +    parser.add_argument ("-f", "--full-paths", dest = 'FullPaths', action =

> > > "store_true", default=False,

> > > +                         help = "Label package nodes with full path to EDK II

> package.

> > > Default is disabled.")

> > > +    parser.add_argument ("-b", "--web-browser", dest = 'WebBrowser',

> > > action = "store_true", default=False,

> > > +                         help = "Display SVG output file in default web browser.

> > Default

> > > is disabled.")

> > > +    parser.add_argument ("-v", "--verbose", dest = 'Verbose', action =

> > > "store_true",

> > > +                         help = "Increase output messages")

> > > +    parser.add_argument ("-q", "--quiet", dest = 'Quiet', action =

> > > "store_true",

> > > +                         help = "Reduce output messages")

> > > +    parser.add_argument ("--debug", dest = 'Debug', type = int, metavar =

> > > '[0-9]', choices = range (0, 10), default = 0,

> > > +                         help = "Set debug level")

> > > +

> > > +    #

> > > +    # Parse command line arguments

> > > +    #

> > > +    args = parser.parse_args ()

> > > +

> > > +    #

> > > +    # Find all EDK II package DEC files

> > > +    #

> > > +    Components = {}

> > > +    SearchPaths = [args.Workspace]

> > > +    if args.PackagesPath:

> > > +        SearchPaths += args.PackagesPath.split(os.pathsep)

> > > +    SearchPaths = [os.path.realpath(x) for x in SearchPaths]

> > > +    for SearchPath in SearchPaths:

> > > +        for root, dirs, files in os.walk (SearchPath):

> > > +            for name in files:

> > > +                FilePath = os.path.join (root, name)

> > > +                if

> > > set(FilePath.split(os.sep)).intersection(set(args.IgnoreDirectory)) != set():

> > > +                    if args.Verbose:

> > > +                        print ('IGNORE:' + FilePath)

> > > +                    continue

> > > +                if os.path.splitext(FilePath)[1].lower() in ['.dec']:

> > > +                    DecFile = os.path.realpath (FilePath)

> > > +                    if os.path.split(DecFile)[1] in args.SkipPackage:

> > > +                        if args.Verbose:

> > > +                            print ('SKIP:' + DecFile)

> > > +                        continue

> > > +                    if DecFile not in Components:

> > > +                        if args.Verbose:

> > > +                            print ('PACKAGE:' + DecFile)

> > > +                        Components[DecFile] = {}

> > > +

> > > +    #

> > > +    # Find EDK II component INF files in each EDK II package

> > > +    #

> > > +    PackageLabels = {}

> > > +    UnresolvedPackages = []

> > > +    AmbiguousDependencies = []

> > > +    for DecFile in Components:

> > > +        DecPath = os.path.split (DecFile)[0]

> > > +        if DecFile not in PackageLabels:

> > > +            PackageLabels[DecFile] = (DecFile.replace(os.path.sep,'\\n'),

> 'white')

> > > +            if not args.FullPaths:

> > > +                PackagePath = os.path.relpath(DecFile,

> os.path.split(DecPath)[0])

> > > +                PackageLabels[DecFile] =

> (PackagePath.replace(os.path.sep,'\\n'),

> > > 'white')

> > > +        for root, dirs, files in os.walk (DecPath):

> > > +            for name in files:

> > > +                FilePath = os.path.join(root, name)

> > > +                if

> > > set(FilePath.split(os.sep)).intersection(set(args.IgnoreDirectory)) != set():

> > > +                    if args.Verbose:

> > > +                        print ('IGNORE:' + FilePath)

> > > +                    continue

> > > +                if os.path.splitext(FilePath)[1].lower() in ['.inf']:

> > > +                    InfFile = os.path.realpath (FilePath)

> > > +                    Inf = InfParser ()

> > > +                    Inf.ParseFile (InfFile)

> > > +                    DependentPackages = []

> > > +                    for Dependency in Inf.PackagesUsed:

> > > +                        Dependency = os.path.normpath(Dependency)

> > > +                        if os.path.split(Dependency)[1] in args.SkipPackage:

> > > +                            if args.Verbose:

> > > +                                print ('SKIP:' + Dependency)

> > > +                            continue

> > > +                        Found = False

> > > +                        for SearchPath in SearchPaths:

> > > +                            PackagePath = os.path.realpath(os.path.join(SearchPath,

> > > Dependency))

> > > +                            if os.path.exists(PackagePath):

> > > +                                DependentPackages.append(PackagePath)

> > > +                                if not args.FullPaths:

> > > +                                    PackageLabels[PackagePath] =

> > > (Dependency.replace(os.path.sep,'\\n'), 'white')

> > > +                                Found = True

> > > +                                break

> > > +                        if not Found:

> > > +                            Count = 0

> > > +                            Match = ''

> > > +                            for DecFile2 in Components:

> > > +                                if DecFile2.endswith(Dependency):

> > > +                                    if Count == 0:

> > > +                                        Match = DecFile2

> > > +                                    Count = Count + 1

> > > +                            if Count > 1:

> > > +                                AmbiguousDependencies.append (Dependency)

> > > +                            if Count == 1:

> > > +                                DependentPackages.append(Match)

> > > +                                if not args.FullPaths:

> > > +                                    PackageLabels[Match] =

> > > (Dependency.replace(os.path.sep,'\\n'), 'white')

> > > +                                Found = True

> > > +                        if not Found and args.Unresolved:

> > > +                            if args.Verbose:

> > > +                                print ('WARNING: Dependent package not found ' +

> > > Dependency)

> > > +                            DependentPackages.append(Dependency)

> > > +                            UnresolvedPackages.append(Dependency)

> > > +                            PackageLabels[Dependency] =

> > > (Dependency.replace(os.path.sep,'\\n'), 'white')

> > > +                    Components[DecFile][InfFile] = DependentPackages

> > > +    if AmbiguousDependencies:

> > > +        for Dependency in set(AmbiguousDependencies):

> > > +            print ('ERROR: MULTIPLE:   ' + Dependency)

> > > +        print ('Use --packages-path to provide search priority.')

> > > +        sys.exit(1)

> > > +

> > > +    #

> > > +    # Generate GraphViz dot input file contents.

> > > +    # Use networkx to detect circular dependencies.

> > > +    #

> > > +    MaxWeight = 0

> > > +    MaxWeightDecFile = list(Components.keys())[0]

> > > +    Graph = nx.DiGraph()

> > > +    Edges = []

> > > +    for DecFile in Components:

> > > +        if args.Verbose:

> > > +            print ('PACKAGE DEPENDENCIES:' + DecFile)

> > > +        AllDependencies = []

> > > +        Dependencies = set()

> > > +        for InfFile in Components[DecFile]:

> > > +            AllDependencies += Components[DecFile][InfFile]

> > > +            Dependencies =

> Dependencies.union(Components[DecFile][InfFile])

> > > +        if not args.SelfDependency:

> > > +            Dependencies = Dependencies.difference(set([DecFile]))

> > > +        for Dependency in Dependencies:

> > > +            Weight = AllDependencies.count(Dependency)

> > > +            if Weight > MaxWeight:

> > > +                MaxWeight = Weight

> > > +                MaxWeightDecFile = DecFile

> > > +            if args.Verbose:

> > > +                print ('  DEPENDENCY: Weight(' + str(Weight) + ') ' +

> Dependency)

> > > +            Edges.append('  "{Package}" -> "{Dependency}" [label =

> > > "{Weight}"];'.format(

> > > +                Package    = DecFile,

> > > +                Dependency = Dependency,

> > > +                Weight     = str(Weight) if args.Label else ''

> > > +                ))

> > > +            Graph.add_edge(DecFile, Dependency)

> > > +    Edges.sort()

> > > +

> > > +    #

> > > +    # Set fill color to yellow if a packages is part of a circular dependency

> > > +    #

> > > +    for Node in set([x for y in list(nx.simple_cycles(Graph)) for x in y]):

> > > +        PackageLabels[Node] = (PackageLabels[Node][0], 'yellow')

> > > +        print ('ERROR: CIRCULAR:   ' + Node)

> > > +

> > > +    #

> > > +    # Set fill color to pink if a package that is nested inside another

> package.

> > > +    # Set fill color to orange if a package is nested inside another package

> > and

> > > +    # is part of a circular dependency.

> > > +    #

> > > +    for DecFile in Components:

> > > +        DecPath = os.path.split (DecFile)[0]

> > > +        for DecFile2 in Components:

> > > +            if DecFile2 == DecFile:

> > > +                continue

> > > +            if os.path.commonpath ([DecPath, DecFile2]) != DecPath:

> > > +                continue

> > > +            if len(DecFile2) < len(DecPath):

> > > +                DecFile2 = DecFile

> > > +            print ('ERROR: NESTED:     ' + DecFile2)

> > > +            if PackageLabels[DecFile2][1] == 'yellow':

> > > +                PackageLabels[DecFile2] = (PackageLabels[DecFile2][0],

> 'orange')

> > > +            else:

> > > +                PackageLabels[DecFile2] = (PackageLabels[DecFile2][0], 'pink')

> > > +

> > > +    #

> > > +    # Set fill color to red for nodes that are unresolved

> > > +    #

> > > +    for Node in set(UnresolvedPackages):

> > > +        PackageLabels[Node] = (PackageLabels[Node][0], 'red')

> > > +        print ('ERROR: UNRESOLVED: ' + Node)

> > > +

> > > +    #

> > > +    # Add node statements to set node label and fill color

> > > +    #

> > > +    Nodes = []

> > > +    for Package in PackageLabels:

> > > +        Nodes.append('  "{Package}"

> > [label="{Label}",fillcolor={Color}];'.format(

> > > +            Package = Package,

> > > +            Label = PackageLabels[Package][0],

> > > +            Color = PackageLabels[Package][1]

> > > +            ))

> > > +    Nodes.sort()

> > > +

> > > +    #

> > > +    # Generate dot file from Nodes and Edges and add a Legend at top of

> > > graph

> > > +    #

> > > +    Dot = []

> > > +    Dot.append('digraph {')

> > > +    Dot.append('  rankdir=BT;')

> > > +    Dot.append('  node [shape=Mrecord,style=filled];')

> > > +    Dot.append('')

> > > +    Dot = Dot + Nodes

> > > +    Dot.append('')

> > > +    Dot = Dot + Edges

> > > +    Dot.append('')

> > > +    Dot.append('  subgraph legend {')

> > > +    Dot.append('    rank=sink;')

> > > +    Dot.append('    Unresolved     [label="Unresolved Dependency",

> > > fillcolor=red];')

> > > +    Dot.append('    Circular       [label="Circular Dependency",

> > > fillcolor=yellow];')

> > > +    Dot.append('    Nested         [label="Nested Package",

> > > fillcolor=pink];')

> > > +    Dot.append('    NestedCircular [label="Nested Package with Circular

> > > Dependency",fillcolor=orange];')

> > > +    Dot.append('  }')

> > > +    if MaxWeightDecFile:

> > > +        Dot.append('  Unresolved->"' + MaxWeightDecFile + '" [style=invis];')

> > > +    Dot.append('}')

> > > +

> > > +    if args.DotOutputFile:

> > > +        #

> > > +        # Write GraphViz dot file contents to DotOutputFile

> > > +        #

> > > +        with open(os.path.realpath(args.DotOutputFile), 'w') as File:

> > > +            File.write('\n'.join(Dot))

> > > +    if args.SvgOutputFile:

> > > +        #

> > > +        # Use GraphViz 'dot' command to generate SVG output file

> > > +        #

> > > +        args.SvgOutputFile = os.path.realpath(args.SvgOutputFile)

> > > +        try:

> > > +            Process = subprocess.Popen('dot -Tsvg',

> > > +              stdin=subprocess.PIPE,

> > > +              stdout=open(args.SvgOutputFile, 'w'),

> > > +              stderr=subprocess.PIPE,

> > > +              shell=True

> > > +              )

> > > +            Process.stdin.write ('\n'.join(Dot).encode())

> > > +            Process.communicate()

> > > +            if Process.returncode != 0:

> > > +                print ("ERROR: Can not run GraphViz 'dot' command.  Check

> install

> > > and path.")

> > > +                sys.exit(Process.returncode)

> > > +        except:

> > > +            print ("ERROR: Can not run GraphViz 'dot' command.  Check

> install

> > and

> > > path.")

> > > +            sys.exit(1)

> > > +        #

> > > +        # Display SVG file in default web browser

> > > +        #

> > > +        if args.WebBrowser:

> > > +            webbrowser.open(args.SvgOutputFile)

> > > --

> > > 2.21.0.windows.1

> >

> >

> >

>

>

> 



[-- Attachment #2: Type: text/html, Size: 78170 bytes --]

Re: [edk2-devel] [Patch 1/1] BaseTools/Scripts: Add package dependency graphing tool

[Ni, Ray]

[-- Attachment #1: Type: text/plain, Size: 28348 bytes --]

Sean,
This is a very cool tool! We could learn from your tool source code to know how to use DecParser to get more detailed information, like which interfaces (PPI/Protocol/Guid/PCD/Lib) from which package a module is depending on.

Thanks,
Ray


From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Sean via Groups.Io
Sent: Tuesday, December 17, 2019 3:46 AM
To: Kinney, Michael D <michael.d.kinney@intel.com>; Shi, Steven <steven.shi@intel.com>; devel@edk2.groups.io; Ni, Ray <ray.ni@intel.com>; Kinney, Michael D <michael.d.kinney@intel.com>
Cc: Feng, Bob C <bob.c.feng@intel.com>; Gao, Liming <liming.gao@intel.com>; Bret Barkelew <Bret.Barkelew@microsoft.com>
Subject: Re: [edk2-devel] [Patch 1/1] BaseTools/Scripts: Add package dependency graphing tool

Mike/Steven/Ray,

We have had a tool for a couple of years with similar functionality.  It is a python tool you run on your code base and it creates a single (local) webpage that you can then use to build charts.

Please give it a try and see if you think it is useful.

Here is the tool.
https://github.com/spbrogan/edk2_dep

And here is a html report run on edk2
https://github.com/spbrogan/edk2_dep/blob/sample/edk2/Edk2_core.html

Thanks
Sean


From: Kinney, Michael D <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>>
Sent: Monday, December 16, 2019 9:16 AM
To: Shi, Steven <steven.shi@intel.com<mailto:steven.shi@intel.com>>; devel@edk2.groups.io<mailto:devel@edk2.groups.io>; Ni, Ray <ray.ni@intel.com<mailto:ray.ni@intel.com>>; Kinney, Michael D <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>>
Cc: Feng, Bob C <bob.c.feng@intel.com<mailto:bob.c.feng@intel.com>>; Gao, Liming <liming.gao@intel.com<mailto:liming.gao@intel.com>>; Sean Brogan <sean.brogan@microsoft.com<mailto:sean.brogan@microsoft.com>>; Bret Barkelew <Bret.Barkelew@microsoft.com<mailto:Bret.Barkelew@microsoft.com>>
Subject: [EXTERNAL] RE: [edk2-devel] [Patch 1/1] BaseTools/Scripts: Add package dependency graphing tool

Steve,

Yes.  I think a Module scoped view and a Platform scoped view would be good additions.

Can you please enter new BZ feature requests for these and add a summary of the features you would like to see in each?

Thanks,

Mike

From: Shi, Steven <steven.shi@intel.com<mailto:steven.shi@intel.com>>
Sent: Monday, December 16, 2019 12:48 AM
To: devel@edk2.groups.io<mailto:devel@edk2.groups.io>; Ni, Ray <ray.ni@intel.com<mailto:ray.ni@intel.com>>; Kinney, Michael D <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>>
Cc: Feng, Bob C <bob.c.feng@intel.com<mailto:bob.c.feng@intel.com>>; Gao, Liming <liming.gao@intel.com<mailto:liming.gao@intel.com>>; Sean Brogan <sean.brogan@microsoft.com<mailto:sean.brogan@microsoft.com>>; Bret Barkelew <Bret.Barkelew@microsoft.com<mailto:Bret.Barkelew@microsoft.com>>
Subject: RE: [edk2-devel] [Patch 1/1] BaseTools/Scripts: Add package dependency graphing tool


Ray,

I think you are asking for a module-level dependency tool. 😊



Mike,

This tool is cool. Besides the package level, do you think we can go further to figure out a module level dependency tool? With module level dependency info, we can define more interesting and comprehensive Modularity Metrics to measure the edk2 architecture quality (Architectural Technical Debt).







Thanks



Steven Shi

Intel\SSG\SFE\FIE Firmware Infrastructure





> -----Original Message-----

> From: devel@edk2.groups.io<mailto:devel@edk2.groups.io> <devel@edk2.groups.io<mailto:devel@edk2.groups.io>> On Behalf Of Ni, Ray

> Sent: Monday, December 16, 2019 4:41 PM

> To: devel@edk2.groups.io<mailto:devel@edk2.groups.io>; Ni, Ray <ray.ni@intel.com<mailto:ray.ni@intel.com>>; Kinney, Michael D

> <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>>

> Cc: Feng, Bob C <bob.c.feng@intel.com<mailto:bob.c.feng@intel.com>>; Gao, Liming

> <liming.gao@intel.com<mailto:liming.gao@intel.com>>; Sean Brogan <sean.brogan@microsoft.com<mailto:sean.brogan@microsoft.com>>; Bret

> Barkelew <Bret.Barkelew@microsoft.com<mailto:Bret.Barkelew@microsoft.com>>

> Subject: Re: [edk2-devel] [Patch 1/1] BaseTools/Scripts: Add package

> dependency graphing tool

>

> Mike,

> This pkg dep tool can tell through weight when a module depends on a new

> pkg.

> But it cannot tell when a module depends on another

> PCD/Protocol/Guid/Ppi/Library which belongs to an already-depended pkg.

>

> Failure to tell such information may make a bad module (that violates the

> dependency rules) worse (wrongly depend on more interfaces).

>

> Do you think that the tool can be enhanced in future to detect the case?

>

> Thanks,

> Ray

>

> > -----Original Message-----

> > From: devel@edk2.groups.io<mailto:devel@edk2.groups.io> <devel@edk2.groups.io<mailto:devel@edk2.groups.io>> On Behalf Of Ni,

> Ray

> > Sent: Monday, December 16, 2019 1:57 PM

> > To: Kinney, Michael D <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>>;

> devel@edk2.groups.io<mailto:devel@edk2.groups.io>

> > Cc: Feng, Bob C <bob.c.feng@intel.com<mailto:bob.c.feng@intel.com>>; Gao, Liming

> > <liming.gao@intel.com<mailto:liming.gao@intel.com>>; Sean Brogan <sean.brogan@microsoft.com<mailto:sean.brogan@microsoft.com>>;

> Bret

> > Barkelew <Bret.Barkelew@microsoft.com<mailto:Bret.Barkelew@microsoft.com>>

> > Subject: Re: [edk2-devel] [Patch 1/1] BaseTools/Scripts: Add package

> > dependency graphing tool

> >

> > Mike,

> > 2 minor comments regarding the help string in below.

> >

> > > -----Original Message-----

> > > From: Kinney, Michael D <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>>

> > > Sent: Saturday, December 14, 2019 3:45 AM

> > > To: devel@edk2.groups.io<mailto:devel@edk2.groups.io>

> > > Cc: Ni, Ray <ray.ni@intel.com<mailto:ray.ni@intel.com>>; Feng, Bob C <bob.c.feng@intel.com<mailto:bob.c.feng@intel.com>>;

> Gao,

> > > Liming <liming.gao@intel.com<mailto:liming.gao@intel.com>>; Sean Brogan

> > <sean.brogan@microsoft.com<mailto:sean.brogan@microsoft.com>>;

> > > Bret Barkelew <Bret.Barkelew@microsoft.com<mailto:Bret.Barkelew@microsoft.com>>

> > > Subject: [Patch 1/1] BaseTools/Scripts: Add package dependency

> graphing

> > > tool

> > >

> > > https://bugzilla.tianocore.org/show_bug.cgi?id=2161<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugzilla.tianocore.org%2Fshow_bug.cgi%3Fid%3D2161&data=02%7C01%7Csean.brogan%40microsoft.com%7Cf7fb1953185b4a17f27408d7824ba488%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637121133718397648&sdata=NE82lFZmNfIcNjUfo49D%2Bksn94atxzdx7F2j%2FWSME0E%3D&reserved=0>

> > >

> > > Add python script that recursively scans a directory for EDK II

> > > packages and generates GraphViz dot input that is used to render

> > > a graph of package dependencies in SVG format.

> > >

> > > Detects following error/warning conditions:

> > > * Ambiguous dependencies (multiple matches)

> > > * Unresolved dependencies

> > > * Circular dependencies

> > > * Nested packages

> > >

> > > usage: PackageDependencyGraph [-h] [-w WORKSPACE] [-p

> > PACKAGESPATH]

> > >                               [-d DOTOUTPUTFILE] [-o SVGOUTPUTFILE]

> > >                               [-g IGNOREDIRECTORY] [-k SKIPPACKAGE]

> > >                               [-s] [-u] [-l] [-f] [-b] [-v] [-q]

> > >                               [--debug [0-9]]

> > >

> > > Recursively scan a directory for EDK II packages and generate GraphViz

> dot

> > > input that is used to render a graph of package dependencies in SVG

> > format.

> > > Copyright (c) 2019, Intel Corporation. All rights reserved.

> > >

> > > optional arguments:

> > >   -h, --help            show this help message and exit

> > >   -w WORKSPACE, --workspace WORKSPACE

> > >                         Directory to recursively scan for EDK II packages.

> > >                         Default is current directory.

> > >   -p PACKAGESPATH, --packages-path PACKAGESPATH

> > >                         List of directories to recursively scan for EDK II

> > >                         packages.

> > >   -d DOTOUTPUTFILE, --dot-output DOTOUTPUTFILE

> > >                         DOT output filename.

> > >   -o OUTPUTFILE, --output OUTPUTFILE

> > >                         SVG output filename.

> > >   -g IGNOREDIRECTORY, --ignore-directory IGNOREDIRECTORY

> > >                         Name of directory to ignore. Option can be repeated

> > >                         to ignore multiple directories.

> >

> > 1. Name of directory to ignore when scanning for EDKII Module INFs.

> >

> > >   -k SKIPPACKAGE, --skip-package SKIPPACKAGE

> > >                         Name of EDK II Package DEC file to skip. Option can

> > >                         be repeated to skip multiple EDK II packages.

> >

> > 2. Name of EDK II Package DEC file (including .DEC suffix) to skip. Option

> can

> > Be repeated to skip multiple EDK II packages.

> >

> > >   -s, --self-dependency

> > >                         Include self links in dependency graph. Default is

> > >                         disabled.

> > >   -u, --unresolved      Include unresolved EDK II packages in dependency

> > >                         graph. Default is disabled.

> > >   -l, --label           Label links with the number of EDK II package

> > >                         dependencies. Default is disabled.

> > >   -f, --full-paths      Label package nodes with full path to EDK II

> > >                         package.  Default is disabled.

> > >   -b, --web-browser     Display SVG output file in default web browser.

> > >                         Default is disabled.

> > >   -v, --verbose         Increase output messages

> > >   -q, --quiet           Reduce output messages

> > >   --debug [0-9]         Set debug level

> > >

> > > Cc: Ray Ni <ray.ni@intel.com<mailto:ray.ni@intel.com>>

> > > Cc: Bob Feng <bob.c.feng@intel.com<mailto:bob.c.feng@intel.com>>

> > > Cc: Liming Gao <liming.gao@intel.com<mailto:liming.gao@intel.com>>

> > > Cc: Sean Brogan <sean.brogan@microsoft.com<mailto:sean.brogan@microsoft.com>>

> > > Cc: Bret Barkelew <Bret.Barkelew@microsoft.com<mailto:Bret.Barkelew@microsoft.com>>

> > > Signed-off-by: Michael D Kinney <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>>

> > > ---

> > >  BaseTools/Scripts/PackageDependencyGraph.py | 296

> > > ++++++++++++++++++++

> > >  1 file changed, 296 insertions(+)

> > >  create mode 100644 BaseTools/Scripts/PackageDependencyGraph.py

> > >

> > > diff --git a/BaseTools/Scripts/PackageDependencyGraph.py

> > > b/BaseTools/Scripts/PackageDependencyGraph.py

> > > new file mode 100644

> > > index 0000000000..b3c8e41774

> > > --- /dev/null

> > > +++ b/BaseTools/Scripts/PackageDependencyGraph.py

> > > @@ -0,0 +1,296 @@

> > > +# @file

> > > +# Recursively scan a directory for EDK II packages and generate

> GraphViz

> > > dot

> > > +# input that is used to render a graph of package dependencies in SVG

> > > format.

> > > +#

> > > +# Copyright (c) 2019, Intel Corporation. All rights reserved.<BR>

> > > +# SPDX-License-Identifier: BSD-2-Clause-Patent

> > > +#

> > > +##

> > > +

> > > +import os

> > > +import sys

> > > +import argparse

> > > +import subprocess

> > > +import webbrowser

> > > +import networkx as nx

> > > +from edk2toollib.uefi.edk2.parsers.inf_parser import InfParser

> > > +

> > > +#

> > > +# Globals for help information

> > > +#

> > > +__prog__        = 'PackageDependencyGraph'

> > > +__copyright__   = 'Copyright (c) 2019, Intel Corporation. All rights

> > reserved.'

> > > +__description__ = '''Recursively scan a directory for EDK II packages and

> > > +generate GraphViz dot input that is used to render a graph of package

> > > +dependencies in SVG format.'''

> > > +

> > > +if __name__ == '__main__':

> > > +

> > > +    #

> > > +    # Create command line argument parser object

> > > +    #

> > > +    parser = argparse.ArgumentParser (prog = __prog__,

> > > +                                      description = __description__ + '\n' +

> __copyright__,

> > > +                                      conflict_handler = 'resolve')

> > > +    parser.add_argument ("-w", "--workspace", dest = 'Workspace',

> default

> > =

> > > os.curdir,

> > > +                         help = "Directory to recursively scan for EDK II packages.

> > > Default is current directory.")

> > > +    parser.add_argument ("-p", "--packages-path", dest = 'PackagesPath',

> > > default = None,

> > > +                         help = "List of directories to recursively scan for EDK II

> > > packages.")

> > > +    parser.add_argument ("-d", "--dot-output", dest = 'DotOutputFile',

> > > +                         help = "DOT output filename.")

> > > +    parser.add_argument ("-o", "--output", dest = 'SvgOutputFile',

> > > +                         help = "SVG output filename.")

> > > +    parser.add_argument ("-g", "--ignore-directory", dest =

> > 'IgnoreDirectory',

> > > action='append', default=[],

> > > +                         help = "Name of directory to ignore.  Option can be

> repeated

> > to

> > > ignore multiple directories.")

> > > +    parser.add_argument ("-k", "--skip-package", dest = 'SkipPackage',

> > > action='append', default=[],

> > > +                         help = "Name of EDK II Package DEC file to skip.  Option

> can

> > be

> > > repeated to skip multiple EDK II packages.")

> > > +    parser.add_argument ("-s", "--self-dependency", dest =

> > > 'SelfDependency', action = "store_true", default = False,

> > > +                         help = "Include self links in dependency graph.  Default is

> > > disabled.")

> > > +    parser.add_argument ("-u", "--unresolved", dest = 'Unresolved',

> action

> > =

> > > "store_true", default=False,

> > > +                         help = "Include unresolved EDK II packages in dependency

> > > graph.  Default is disabled.")

> > > +    parser.add_argument ("-l", "--label", dest = 'Label', action =

> > "store_true",

> > > default=False,

> > > +                         help = "Label links with the number of EDK II package

> > > dependencies.  Default is disabled.")

> > > +    parser.add_argument ("-f", "--full-paths", dest = 'FullPaths', action =

> > > "store_true", default=False,

> > > +                         help = "Label package nodes with full path to EDK II

> package.

> > > Default is disabled.")

> > > +    parser.add_argument ("-b", "--web-browser", dest = 'WebBrowser',

> > > action = "store_true", default=False,

> > > +                         help = "Display SVG output file in default web browser.

> > Default

> > > is disabled.")

> > > +    parser.add_argument ("-v", "--verbose", dest = 'Verbose', action =

> > > "store_true",

> > > +                         help = "Increase output messages")

> > > +    parser.add_argument ("-q", "--quiet", dest = 'Quiet', action =

> > > "store_true",

> > > +                         help = "Reduce output messages")

> > > +    parser.add_argument ("--debug", dest = 'Debug', type = int, metavar =

> > > '[0-9]', choices = range (0, 10), default = 0,

> > > +                         help = "Set debug level")

> > > +

> > > +    #

> > > +    # Parse command line arguments

> > > +    #

> > > +    args = parser.parse_args ()

> > > +

> > > +    #

> > > +    # Find all EDK II package DEC files

> > > +    #

> > > +    Components = {}

> > > +    SearchPaths = [args.Workspace]

> > > +    if args.PackagesPath:

> > > +        SearchPaths += args.PackagesPath.split(os.pathsep)

> > > +    SearchPaths = [os.path.realpath(x) for x in SearchPaths]

> > > +    for SearchPath in SearchPaths:

> > > +        for root, dirs, files in os.walk (SearchPath):

> > > +            for name in files:

> > > +                FilePath = os.path.join (root, name)

> > > +                if

> > > set(FilePath.split(os.sep)).intersection(set(args.IgnoreDirectory)) != set():

> > > +                    if args.Verbose:

> > > +                        print ('IGNORE:' + FilePath)

> > > +                    continue

> > > +                if os.path.splitext(FilePath)[1].lower() in ['.dec']:

> > > +                    DecFile = os.path.realpath (FilePath)

> > > +                    if os.path.split(DecFile)[1] in args.SkipPackage:

> > > +                        if args.Verbose:

> > > +                            print ('SKIP:' + DecFile)

> > > +                        continue

> > > +                    if DecFile not in Components:

> > > +                        if args.Verbose:

> > > +                            print ('PACKAGE:' + DecFile)

> > > +                        Components[DecFile] = {}

> > > +

> > > +    #

> > > +    # Find EDK II component INF files in each EDK II package

> > > +    #

> > > +    PackageLabels = {}

> > > +    UnresolvedPackages = []

> > > +    AmbiguousDependencies = []

> > > +    for DecFile in Components:

> > > +        DecPath = os.path.split (DecFile)[0]

> > > +        if DecFile not in PackageLabels:

> > > +            PackageLabels[DecFile] = (DecFile.replace(os.path.sep,'\\n'),

> 'white')

> > > +            if not args.FullPaths:

> > > +                PackagePath = os.path.relpath(DecFile,

> os.path.split(DecPath)[0])

> > > +                PackageLabels[DecFile] =

> (PackagePath.replace(os.path.sep,'\\n'),

> > > 'white')

> > > +        for root, dirs, files in os.walk (DecPath):

> > > +            for name in files:

> > > +                FilePath = os.path.join(root, name)

> > > +                if

> > > set(FilePath.split(os.sep)).intersection(set(args.IgnoreDirectory)) != set():

> > > +                    if args.Verbose:

> > > +                        print ('IGNORE:' + FilePath)

> > > +                    continue

> > > +                if os.path.splitext(FilePath)[1].lower() in ['.inf']:

> > > +                    InfFile = os.path.realpath (FilePath)

> > > +                    Inf = InfParser ()

> > > +                    Inf.ParseFile (InfFile)

> > > +                    DependentPackages = []

> > > +                    for Dependency in Inf.PackagesUsed:

> > > +                        Dependency = os.path.normpath(Dependency)

> > > +                        if os.path.split(Dependency)[1] in args.SkipPackage:

> > > +                            if args.Verbose:

> > > +                                print ('SKIP:' + Dependency)

> > > +                            continue

> > > +                        Found = False

> > > +                        for SearchPath in SearchPaths:

> > > +                            PackagePath = os.path.realpath(os.path.join(SearchPath,

> > > Dependency))

> > > +                            if os.path.exists(PackagePath):

> > > +                                DependentPackages.append(PackagePath)

> > > +                                if not args.FullPaths:

> > > +                                    PackageLabels[PackagePath] =

> > > (Dependency.replace(os.path.sep,'\\n'), 'white')

> > > +                                Found = True

> > > +                                break

> > > +                        if not Found:

> > > +                            Count = 0

> > > +                            Match = ''

> > > +                            for DecFile2 in Components:

> > > +                                if DecFile2.endswith(Dependency):

> > > +                                    if Count == 0:

> > > +                                        Match = DecFile2

> > > +                                    Count = Count + 1

> > > +                            if Count > 1:

> > > +                                AmbiguousDependencies.append (Dependency)

> > > +                            if Count == 1:

> > > +                                DependentPackages.append(Match)

> > > +                                if not args.FullPaths:

> > > +                                    PackageLabels[Match] =

> > > (Dependency.replace(os.path.sep,'\\n'), 'white')

> > > +                                Found = True

> > > +                        if not Found and args.Unresolved:

> > > +                            if args.Verbose:

> > > +                                print ('WARNING: Dependent package not found ' +

> > > Dependency)

> > > +                            DependentPackages.append(Dependency)

> > > +                            UnresolvedPackages.append(Dependency)

> > > +                            PackageLabels[Dependency] =

> > > (Dependency.replace(os.path.sep,'\\n'), 'white')

> > > +                    Components[DecFile][InfFile] = DependentPackages

> > > +    if AmbiguousDependencies:

> > > +        for Dependency in set(AmbiguousDependencies):

> > > +            print ('ERROR: MULTIPLE:   ' + Dependency)

> > > +        print ('Use --packages-path to provide search priority.')

> > > +        sys.exit(1)

> > > +

> > > +    #

> > > +    # Generate GraphViz dot input file contents.

> > > +    # Use networkx to detect circular dependencies.

> > > +    #

> > > +    MaxWeight = 0

> > > +    MaxWeightDecFile = list(Components.keys())[0]

> > > +    Graph = nx.DiGraph()

> > > +    Edges = []

> > > +    for DecFile in Components:

> > > +        if args.Verbose:

> > > +            print ('PACKAGE DEPENDENCIES:' + DecFile)

> > > +        AllDependencies = []

> > > +        Dependencies = set()

> > > +        for InfFile in Components[DecFile]:

> > > +            AllDependencies += Components[DecFile][InfFile]

> > > +            Dependencies =

> Dependencies.union(Components[DecFile][InfFile])

> > > +        if not args.SelfDependency:

> > > +            Dependencies = Dependencies.difference(set([DecFile]))

> > > +        for Dependency in Dependencies:

> > > +            Weight = AllDependencies.count(Dependency)

> > > +            if Weight > MaxWeight:

> > > +                MaxWeight = Weight

> > > +                MaxWeightDecFile = DecFile

> > > +            if args.Verbose:

> > > +                print ('  DEPENDENCY: Weight(' + str(Weight) + ') ' +

> Dependency)

> > > +            Edges.append('  "{Package}" -> "{Dependency}" [label =

> > > "{Weight}"];'.format(

> > > +                Package    = DecFile,

> > > +                Dependency = Dependency,

> > > +                Weight     = str(Weight) if args.Label else ''

> > > +                ))

> > > +            Graph.add_edge(DecFile, Dependency)

> > > +    Edges.sort()

> > > +

> > > +    #

> > > +    # Set fill color to yellow if a packages is part of a circular dependency

> > > +    #

> > > +    for Node in set([x for y in list(nx.simple_cycles(Graph)) for x in y]):

> > > +        PackageLabels[Node] = (PackageLabels[Node][0], 'yellow')

> > > +        print ('ERROR: CIRCULAR:   ' + Node)

> > > +

> > > +    #

> > > +    # Set fill color to pink if a package that is nested inside another

> package.

> > > +    # Set fill color to orange if a package is nested inside another package

> > and

> > > +    # is part of a circular dependency.

> > > +    #

> > > +    for DecFile in Components:

> > > +        DecPath = os.path.split (DecFile)[0]

> > > +        for DecFile2 in Components:

> > > +            if DecFile2 == DecFile:

> > > +                continue

> > > +            if os.path.commonpath ([DecPath, DecFile2]) != DecPath:

> > > +                continue

> > > +            if len(DecFile2) < len(DecPath):

> > > +                DecFile2 = DecFile

> > > +            print ('ERROR: NESTED:     ' + DecFile2)

> > > +            if PackageLabels[DecFile2][1] == 'yellow':

> > > +                PackageLabels[DecFile2] = (PackageLabels[DecFile2][0],

> 'orange')

> > > +            else:

> > > +                PackageLabels[DecFile2] = (PackageLabels[DecFile2][0], 'pink')

> > > +

> > > +    #

> > > +    # Set fill color to red for nodes that are unresolved

> > > +    #

> > > +    for Node in set(UnresolvedPackages):

> > > +        PackageLabels[Node] = (PackageLabels[Node][0], 'red')

> > > +        print ('ERROR: UNRESOLVED: ' + Node)

> > > +

> > > +    #

> > > +    # Add node statements to set node label and fill color

> > > +    #

> > > +    Nodes = []

> > > +    for Package in PackageLabels:

> > > +        Nodes.append('  "{Package}"

> > [label="{Label}",fillcolor={Color}];'.format(

> > > +            Package = Package,

> > > +            Label = PackageLabels[Package][0],

> > > +            Color = PackageLabels[Package][1]

> > > +            ))

> > > +    Nodes.sort()

> > > +

> > > +    #

> > > +    # Generate dot file from Nodes and Edges and add a Legend at top of

> > > graph

> > > +    #

> > > +    Dot = []

> > > +    Dot.append('digraph {')

> > > +    Dot.append('  rankdir=BT;')

> > > +    Dot.append('  node [shape=Mrecord,style=filled];')

> > > +    Dot.append('')

> > > +    Dot = Dot + Nodes

> > > +    Dot.append('')

> > > +    Dot = Dot + Edges

> > > +    Dot.append('')

> > > +    Dot.append('  subgraph legend {')

> > > +    Dot.append('    rank=sink;')

> > > +    Dot.append('    Unresolved     [label="Unresolved Dependency",

> > > fillcolor=red];')

> > > +    Dot.append('    Circular       [label="Circular Dependency",

> > > fillcolor=yellow];')

> > > +    Dot.append('    Nested         [label="Nested Package",

> > > fillcolor=pink];')

> > > +    Dot.append('    NestedCircular [label="Nested Package with Circular

> > > Dependency",fillcolor=orange];')

> > > +    Dot.append('  }')

> > > +    if MaxWeightDecFile:

> > > +        Dot.append('  Unresolved->"' + MaxWeightDecFile + '" [style=invis];')

> > > +    Dot.append('}')

> > > +

> > > +    if args.DotOutputFile:

> > > +        #

> > > +        # Write GraphViz dot file contents to DotOutputFile

> > > +        #

> > > +        with open(os.path.realpath(args.DotOutputFile), 'w') as File:

> > > +            File.write('\n'.join(Dot))

> > > +    if args.SvgOutputFile:

> > > +        #

> > > +        # Use GraphViz 'dot' command to generate SVG output file

> > > +        #

> > > +        args.SvgOutputFile = os.path.realpath(args.SvgOutputFile)

> > > +        try:

> > > +            Process = subprocess.Popen('dot -Tsvg',

> > > +              stdin=subprocess.PIPE,

> > > +              stdout=open(args.SvgOutputFile, 'w'),

> > > +              stderr=subprocess.PIPE,

> > > +              shell=True

> > > +              )

> > > +            Process.stdin.write ('\n'.join(Dot).encode())

> > > +            Process.communicate()

> > > +            if Process.returncode != 0:

> > > +                print ("ERROR: Can not run GraphViz 'dot' command.  Check

> install

> > > and path.")

> > > +                sys.exit(Process.returncode)

> > > +        except:

> > > +            print ("ERROR: Can not run GraphViz 'dot' command.  Check

> install

> > and

> > > path.")

> > > +            sys.exit(1)

> > > +        #

> > > +        # Display SVG file in default web browser

> > > +        #

> > > +        if args.WebBrowser:

> > > +            webbrowser.open(args.SvgOutputFile)

> > > --

> > > 2.21.0.windows.1

> >

> >

> >

>

>

>




[-- Attachment #2: Type: text/html, Size: 82059 bytes --]

Re: [edk2-devel] [Patch 1/1] BaseTools/Scripts: Add package dependency graphing tool

[Ni, Ray]

Done.

> -----Original Message-----
> From: Kinney, Michael D <michael.d.kinney@intel.com>
> Sent: Tuesday, December 17, 2019 1:15 AM
> To: Ni, Ray <ray.ni@intel.com>; devel@edk2.groups.io; Kinney, Michael D
> <michael.d.kinney@intel.com>
> Cc: Feng, Bob C <bob.c.feng@intel.com>; Gao, Liming
> <liming.gao@intel.com>; Sean Brogan <sean.brogan@microsoft.com>; Bret
> Barkelew <Bret.Barkelew@microsoft.com>
> Subject: RE: [edk2-devel] [Patch 1/1] BaseTools/Scripts: Add package
> dependency graphing tool
> 
> Ray,
> 
> Yes.  More features can be added.  Can you add to the BZ some
> examples of the output you would like to see.
> 
> https://bugzilla.tianocore.org/show_bug.cgi?id=2161
> 
> Thanks,
> 
> Mike
> 
> > -----Original Message-----
> > From: Ni, Ray <ray.ni@intel.com>
> > Sent: Monday, December 16, 2019 12:41 AM
> > To: devel@edk2.groups.io; Ni, Ray <ray.ni@intel.com>;
> > Kinney, Michael D <michael.d.kinney@intel.com>
> > Cc: Feng, Bob C <bob.c.feng@intel.com>; Gao, Liming
> > <liming.gao@intel.com>; Sean Brogan
> > <sean.brogan@microsoft.com>; Bret Barkelew
> > <Bret.Barkelew@microsoft.com>
> > Subject: RE: [edk2-devel] [Patch 1/1]
> > BaseTools/Scripts: Add package dependency graphing tool
> >
> > Mike,
> > This pkg dep tool can tell through weight when a module
> > depends on a new pkg.
> > But it cannot tell when a module depends on another
> > PCD/Protocol/Guid/Ppi/Library which belongs to an
> > already-depended pkg.
> >
> > Failure to tell such information may make a bad module
> > (that violates the dependency rules) worse (wrongly
> > depend on more interfaces).
> >
> > Do you think that the tool can be enhanced in future to
> > detect the case?
> >
> > Thanks,
> > Ray
> >
> > > -----Original Message-----
> > > From: devel@edk2.groups.io <devel@edk2.groups.io> On
> > Behalf Of Ni, Ray
> > > Sent: Monday, December 16, 2019 1:57 PM
> > > To: Kinney, Michael D <michael.d.kinney@intel.com>;
> > devel@edk2.groups.io
> > > Cc: Feng, Bob C <bob.c.feng@intel.com>; Gao, Liming
> > > <liming.gao@intel.com>; Sean Brogan
> > <sean.brogan@microsoft.com>; Bret
> > > Barkelew <Bret.Barkelew@microsoft.com>
> > > Subject: Re: [edk2-devel] [Patch 1/1]
> > BaseTools/Scripts: Add package
> > > dependency graphing tool
> > >
> > > Mike,
> > > 2 minor comments regarding the help string in below.
> > >
> > > > -----Original Message-----
> > > > From: Kinney, Michael D
> > <michael.d.kinney@intel.com>
> > > > Sent: Saturday, December 14, 2019 3:45 AM
> > > > To: devel@edk2.groups.io
> > > > Cc: Ni, Ray <ray.ni@intel.com>; Feng, Bob C
> > <bob.c.feng@intel.com>; Gao,
> > > > Liming <liming.gao@intel.com>; Sean Brogan
> > > <sean.brogan@microsoft.com>;
> > > > Bret Barkelew <Bret.Barkelew@microsoft.com>
> > > > Subject: [Patch 1/1] BaseTools/Scripts: Add package
> > dependency graphing
> > > > tool
> > > >
> > > > https://bugzilla.tianocore.org/show_bug.cgi?id=2161
> > > >
> > > > Add python script that recursively scans a
> > directory for EDK II
> > > > packages and generates GraphViz dot input that is
> > used to render
> > > > a graph of package dependencies in SVG format.
> > > >
> > > > Detects following error/warning conditions:
> > > > * Ambiguous dependencies (multiple matches)
> > > > * Unresolved dependencies
> > > > * Circular dependencies
> > > > * Nested packages
> > > >
> > > > usage: PackageDependencyGraph [-h] [-w WORKSPACE]
> > [-p
> > > PACKAGESPATH]
> > > >                               [-d DOTOUTPUTFILE] [-
> > o SVGOUTPUTFILE]
> > > >                               [-g IGNOREDIRECTORY]
> > [-k SKIPPACKAGE]
> > > >                               [-s] [-u] [-l] [-f]
> > [-b] [-v] [-q]
> > > >                               [--debug [0-9]]
> > > >
> > > > Recursively scan a directory for EDK II packages
> > and generate GraphViz dot
> > > > input that is used to render a graph of package
> > dependencies in SVG
> > > format.
> > > > Copyright (c) 2019, Intel Corporation. All rights
> > reserved.
> > > >
> > > > optional arguments:
> > > >   -h, --help            show this help message and
> > exit
> > > >   -w WORKSPACE, --workspace WORKSPACE
> > > >                         Directory to recursively
> > scan for EDK II packages.
> > > >                         Default is current
> > directory.
> > > >   -p PACKAGESPATH, --packages-path PACKAGESPATH
> > > >                         List of directories to
> > recursively scan for EDK II
> > > >                         packages.
> > > >   -d DOTOUTPUTFILE, --dot-output DOTOUTPUTFILE
> > > >                         DOT output filename.
> > > >   -o OUTPUTFILE, --output OUTPUTFILE
> > > >                         SVG output filename.
> > > >   -g IGNOREDIRECTORY, --ignore-directory
> > IGNOREDIRECTORY
> > > >                         Name of directory to
> > ignore. Option can be repeated
> > > >                         to ignore multiple
> > directories.
> > >
> > > 1. Name of directory to ignore when scanning for
> > EDKII Module INFs.
> > >
> > > >   -k SKIPPACKAGE, --skip-package SKIPPACKAGE
> > > >                         Name of EDK II Package DEC
> > file to skip. Option can
> > > >                         be repeated to skip
> > multiple EDK II packages.
> > >
> > > 2. Name of EDK II Package DEC file (including .DEC
> > suffix) to skip. Option can
> > > Be repeated to skip multiple EDK II packages.
> > >
> > > >   -s, --self-dependency
> > > >                         Include self links in
> > dependency graph. Default is
> > > >                         disabled.
> > > >   -u, --unresolved      Include unresolved EDK II
> > packages in dependency
> > > >                         graph. Default is disabled.
> > > >   -l, --label           Label links with the number
> > of EDK II package
> > > >                         dependencies. Default is
> > disabled.
> > > >   -f, --full-paths      Label package nodes with
> > full path to EDK II
> > > >                         package.  Default is
> > disabled.
> > > >   -b, --web-browser     Display SVG output file in
> > default web browser.
> > > >                         Default is disabled.
> > > >   -v, --verbose         Increase output messages
> > > >   -q, --quiet           Reduce output messages
> > > >   --debug [0-9]         Set debug level
> > > >
> > > > Cc: Ray Ni <ray.ni@intel.com>
> > > > Cc: Bob Feng <bob.c.feng@intel.com>
> > > > Cc: Liming Gao <liming.gao@intel.com>
> > > > Cc: Sean Brogan <sean.brogan@microsoft.com>
> > > > Cc: Bret Barkelew <Bret.Barkelew@microsoft.com>
> > > > Signed-off-by: Michael D Kinney
> > <michael.d.kinney@intel.com>
> > > > ---
> > > >  BaseTools/Scripts/PackageDependencyGraph.py | 296
> > > > ++++++++++++++++++++
> > > >  1 file changed, 296 insertions(+)
> > > >  create mode 100644
> > BaseTools/Scripts/PackageDependencyGraph.py
> > > >
> > > > diff --git
> > a/BaseTools/Scripts/PackageDependencyGraph.py
> > > > b/BaseTools/Scripts/PackageDependencyGraph.py
> > > > new file mode 100644
> > > > index 0000000000..b3c8e41774
> > > > --- /dev/null
> > > > +++ b/BaseTools/Scripts/PackageDependencyGraph.py
> > > > @@ -0,0 +1,296 @@
> > > > +# @file
> > > > +# Recursively scan a directory for EDK II packages
> > and generate GraphViz
> > > > dot
> > > > +# input that is used to render a graph of package
> > dependencies in SVG
> > > > format.
> > > > +#
> > > > +# Copyright (c) 2019, Intel Corporation. All
> > rights reserved.<BR>
> > > > +# SPDX-License-Identifier: BSD-2-Clause-Patent
> > > > +#
> > > > +##
> > > > +
> > > > +import os
> > > > +import sys
> > > > +import argparse
> > > > +import subprocess
> > > > +import webbrowser
> > > > +import networkx as nx
> > > > +from edk2toollib.uefi.edk2.parsers.inf_parser
> > import InfParser
> > > > +
> > > > +#
> > > > +# Globals for help information
> > > > +#
> > > > +__prog__        = 'PackageDependencyGraph'
> > > > +__copyright__   = 'Copyright (c) 2019, Intel
> > Corporation. All rights
> > > reserved.'
> > > > +__description__ = '''Recursively scan a directory
> > for EDK II packages and
> > > > +generate GraphViz dot input that is used to render
> > a graph of package
> > > > +dependencies in SVG format.'''
> > > > +
> > > > +if __name__ == '__main__':
> > > > +
> > > > +    #
> > > > +    # Create command line argument parser object
> > > > +    #
> > > > +    parser = argparse.ArgumentParser (prog =
> > __prog__,
> > > > +                                      description
> > = __description__ + '\n' + __copyright__,
> > > > +
> > conflict_handler = 'resolve')
> > > > +    parser.add_argument ("-w", "--workspace", dest
> > = 'Workspace', default
> > > =
> > > > os.curdir,
> > > > +                         help = "Directory to
> > recursively scan for EDK II packages.
> > > > Default is current directory.")
> > > > +    parser.add_argument ("-p", "--packages-path",
> > dest = 'PackagesPath',
> > > > default = None,
> > > > +                         help = "List of
> > directories to recursively scan for EDK II
> > > > packages.")
> > > > +    parser.add_argument ("-d", "--dot-output",
> > dest = 'DotOutputFile',
> > > > +                         help = "DOT output
> > filename.")
> > > > +    parser.add_argument ("-o", "--output", dest =
> > 'SvgOutputFile',
> > > > +                         help = "SVG output
> > filename.")
> > > > +    parser.add_argument ("-g", "--ignore-
> > directory", dest =
> > > 'IgnoreDirectory',
> > > > action='append', default=[],
> > > > +                         help = "Name of directory
> > to ignore.  Option can be repeated
> > > to
> > > > ignore multiple directories.")
> > > > +    parser.add_argument ("-k", "--skip-package",
> > dest = 'SkipPackage',
> > > > action='append', default=[],
> > > > +                         help = "Name of EDK II
> > Package DEC file to skip.  Option can
> > > be
> > > > repeated to skip multiple EDK II packages.")
> > > > +    parser.add_argument ("-s", "--self-
> > dependency", dest =
> > > > 'SelfDependency', action = "store_true", default =
> > False,
> > > > +                         help = "Include self
> > links in dependency graph.  Default is
> > > > disabled.")
> > > > +    parser.add_argument ("-u", "--unresolved",
> > dest = 'Unresolved', action
> > > =
> > > > "store_true", default=False,
> > > > +                         help = "Include
> > unresolved EDK II packages in dependency
> > > > graph.  Default is disabled.")
> > > > +    parser.add_argument ("-l", "--label", dest =
> > 'Label', action =
> > > "store_true",
> > > > default=False,
> > > > +                         help = "Label links with
> > the number of EDK II package
> > > > dependencies.  Default is disabled.")
> > > > +    parser.add_argument ("-f", "--full-paths",
> > dest = 'FullPaths', action =
> > > > "store_true", default=False,
> > > > +                         help = "Label package
> > nodes with full path to EDK II package.
> > > > Default is disabled.")
> > > > +    parser.add_argument ("-b", "--web-browser",
> > dest = 'WebBrowser',
> > > > action = "store_true", default=False,
> > > > +                         help = "Display SVG
> > output file in default web browser.
> > > Default
> > > > is disabled.")
> > > > +    parser.add_argument ("-v", "--verbose", dest =
> > 'Verbose', action =
> > > > "store_true",
> > > > +                         help = "Increase output
> > messages")
> > > > +    parser.add_argument ("-q", "--quiet", dest =
> > 'Quiet', action =
> > > > "store_true",
> > > > +                         help = "Reduce output
> > messages")
> > > > +    parser.add_argument ("--debug", dest =
> > 'Debug', type = int, metavar =
> > > > '[0-9]', choices = range (0, 10), default = 0,
> > > > +                         help = "Set debug level")
> > > > +
> > > > +    #
> > > > +    # Parse command line arguments
> > > > +    #
> > > > +    args = parser.parse_args ()
> > > > +
> > > > +    #
> > > > +    # Find all EDK II package DEC files
> > > > +    #
> > > > +    Components = {}
> > > > +    SearchPaths = [args.Workspace]
> > > > +    if args.PackagesPath:
> > > > +        SearchPaths +=
> > args.PackagesPath.split(os.pathsep)
> > > > +    SearchPaths = [os.path.realpath(x) for x in
> > SearchPaths]
> > > > +    for SearchPath in SearchPaths:
> > > > +        for root, dirs, files in os.walk
> > (SearchPath):
> > > > +            for name in files:
> > > > +                FilePath = os.path.join (root,
> > name)
> > > > +                if
> > > >
> > set(FilePath.split(os.sep)).intersection(set(args.Ignor
> > eDirectory)) != set():
> > > > +                    if args.Verbose:
> > > > +                        print ('IGNORE:' +
> > FilePath)
> > > > +                    continue
> > > > +                if
> > os.path.splitext(FilePath)[1].lower() in ['.dec']:
> > > > +                    DecFile = os.path.realpath
> > (FilePath)
> > > > +                    if os.path.split(DecFile)[1]
> > in args.SkipPackage:
> > > > +                        if args.Verbose:
> > > > +                            print ('SKIP:' +
> > DecFile)
> > > > +                        continue
> > > > +                    if DecFile not in Components:
> > > > +                        if args.Verbose:
> > > > +                            print ('PACKAGE:' +
> > DecFile)
> > > > +                        Components[DecFile] = {}
> > > > +
> > > > +    #
> > > > +    # Find EDK II component INF files in each EDK
> > II package
> > > > +    #
> > > > +    PackageLabels = {}
> > > > +    UnresolvedPackages = []
> > > > +    AmbiguousDependencies = []
> > > > +    for DecFile in Components:
> > > > +        DecPath = os.path.split (DecFile)[0]
> > > > +        if DecFile not in PackageLabels:
> > > > +            PackageLabels[DecFile] =
> > (DecFile.replace(os.path.sep,'\\n'), 'white')
> > > > +            if not args.FullPaths:
> > > > +                PackagePath =
> > os.path.relpath(DecFile, os.path.split(DecPath)[0])
> > > > +                PackageLabels[DecFile] =
> > (PackagePath.replace(os.path.sep,'\\n'),
> > > > 'white')
> > > > +        for root, dirs, files in os.walk
> > (DecPath):
> > > > +            for name in files:
> > > > +                FilePath = os.path.join(root,
> > name)
> > > > +                if
> > > >
> > set(FilePath.split(os.sep)).intersection(set(args.Ignor
> > eDirectory)) != set():
> > > > +                    if args.Verbose:
> > > > +                        print ('IGNORE:' +
> > FilePath)
> > > > +                    continue
> > > > +                if
> > os.path.splitext(FilePath)[1].lower() in ['.inf']:
> > > > +                    InfFile = os.path.realpath
> > (FilePath)
> > > > +                    Inf = InfParser ()
> > > > +                    Inf.ParseFile (InfFile)
> > > > +                    DependentPackages = []
> > > > +                    for Dependency in
> > Inf.PackagesUsed:
> > > > +                        Dependency =
> > os.path.normpath(Dependency)
> > > > +                        if
> > os.path.split(Dependency)[1] in args.SkipPackage:
> > > > +                            if args.Verbose:
> > > > +                                print ('SKIP:' +
> > Dependency)
> > > > +                            continue
> > > > +                        Found = False
> > > > +                        for SearchPath in
> > SearchPaths:
> > > > +                            PackagePath =
> > os.path.realpath(os.path.join(SearchPath,
> > > > Dependency))
> > > > +                            if
> > os.path.exists(PackagePath):
> > > > +
> > DependentPackages.append(PackagePath)
> > > > +                                if not
> > args.FullPaths:
> > > > +
> > PackageLabels[PackagePath] =
> > > > (Dependency.replace(os.path.sep,'\\n'), 'white')
> > > > +                                Found = True
> > > > +                                break
> > > > +                        if not Found:
> > > > +                            Count = 0
> > > > +                            Match = ''
> > > > +                            for DecFile2 in
> > Components:
> > > > +                                if
> > DecFile2.endswith(Dependency):
> > > > +                                    if Count == 0:
> > > > +                                        Match =
> > DecFile2
> > > > +                                    Count = Count
> > + 1
> > > > +                            if Count > 1:
> > > > +
> > AmbiguousDependencies.append (Dependency)
> > > > +                            if Count == 1:
> > > > +
> > DependentPackages.append(Match)
> > > > +                                if not
> > args.FullPaths:
> > > > +
> > PackageLabels[Match] =
> > > > (Dependency.replace(os.path.sep,'\\n'), 'white')
> > > > +                                Found = True
> > > > +                        if not Found and
> > args.Unresolved:
> > > > +                            if args.Verbose:
> > > > +                                print ('WARNING:
> > Dependent package not found ' +
> > > > Dependency)
> > > > +
> > DependentPackages.append(Dependency)
> > > > +
> > UnresolvedPackages.append(Dependency)
> > > > +
> > PackageLabels[Dependency] =
> > > > (Dependency.replace(os.path.sep,'\\n'), 'white')
> > > > +                    Components[DecFile][InfFile] =
> > DependentPackages
> > > > +    if AmbiguousDependencies:
> > > > +        for Dependency in
> > set(AmbiguousDependencies):
> > > > +            print ('ERROR: MULTIPLE:   ' +
> > Dependency)
> > > > +        print ('Use --packages-path to provide
> > search priority.')
> > > > +        sys.exit(1)
> > > > +
> > > > +    #
> > > > +    # Generate GraphViz dot input file contents.
> > > > +    # Use networkx to detect circular
> > dependencies.
> > > > +    #
> > > > +    MaxWeight = 0
> > > > +    MaxWeightDecFile = list(Components.keys())[0]
> > > > +    Graph = nx.DiGraph()
> > > > +    Edges = []
> > > > +    for DecFile in Components:
> > > > +        if args.Verbose:
> > > > +            print ('PACKAGE DEPENDENCIES:' +
> > DecFile)
> > > > +        AllDependencies = []
> > > > +        Dependencies = set()
> > > > +        for InfFile in Components[DecFile]:
> > > > +            AllDependencies +=
> > Components[DecFile][InfFile]
> > > > +            Dependencies =
> > Dependencies.union(Components[DecFile][InfFile])
> > > > +        if not args.SelfDependency:
> > > > +            Dependencies =
> > Dependencies.difference(set([DecFile]))
> > > > +        for Dependency in Dependencies:
> > > > +            Weight =
> > AllDependencies.count(Dependency)
> > > > +            if Weight > MaxWeight:
> > > > +                MaxWeight = Weight
> > > > +                MaxWeightDecFile = DecFile
> > > > +            if args.Verbose:
> > > > +                print ('  DEPENDENCY: Weight(' +
> > str(Weight) + ') ' + Dependency)
> > > > +            Edges.append('  "{Package}" ->
> > "{Dependency}" [label =
> > > > "{Weight}"];'.format(
> > > > +                Package    = DecFile,
> > > > +                Dependency = Dependency,
> > > > +                Weight     = str(Weight) if
> > args.Label else ''
> > > > +                ))
> > > > +            Graph.add_edge(DecFile, Dependency)
> > > > +    Edges.sort()
> > > > +
> > > > +    #
> > > > +    # Set fill color to yellow if a packages is
> > part of a circular dependency
> > > > +    #
> > > > +    for Node in set([x for y in
> > list(nx.simple_cycles(Graph)) for x in y]):
> > > > +        PackageLabels[Node] =
> > (PackageLabels[Node][0], 'yellow')
> > > > +        print ('ERROR: CIRCULAR:   ' + Node)
> > > > +
> > > > +    #
> > > > +    # Set fill color to pink if a package that is
> > nested inside another package.
> > > > +    # Set fill color to orange if a package is
> > nested inside another package
> > > and
> > > > +    # is part of a circular dependency.
> > > > +    #
> > > > +    for DecFile in Components:
> > > > +        DecPath = os.path.split (DecFile)[0]
> > > > +        for DecFile2 in Components:
> > > > +            if DecFile2 == DecFile:
> > > > +                continue
> > > > +            if os.path.commonpath ([DecPath,
> > DecFile2]) != DecPath:
> > > > +                continue
> > > > +            if len(DecFile2) < len(DecPath):
> > > > +                DecFile2 = DecFile
> > > > +            print ('ERROR: NESTED:     ' +
> > DecFile2)
> > > > +            if PackageLabels[DecFile2][1] ==
> > 'yellow':
> > > > +                PackageLabels[DecFile2] =
> > (PackageLabels[DecFile2][0], 'orange')
> > > > +            else:
> > > > +                PackageLabels[DecFile2] =
> > (PackageLabels[DecFile2][0], 'pink')
> > > > +
> > > > +    #
> > > > +    # Set fill color to red for nodes that are
> > unresolved
> > > > +    #
> > > > +    for Node in set(UnresolvedPackages):
> > > > +        PackageLabels[Node] =
> > (PackageLabels[Node][0], 'red')
> > > > +        print ('ERROR: UNRESOLVED: ' + Node)
> > > > +
> > > > +    #
> > > > +    # Add node statements to set node label and
> > fill color
> > > > +    #
> > > > +    Nodes = []
> > > > +    for Package in PackageLabels:
> > > > +        Nodes.append('  "{Package}"
> > > [label="{Label}",fillcolor={Color}];'.format(
> > > > +            Package = Package,
> > > > +            Label = PackageLabels[Package][0],
> > > > +            Color = PackageLabels[Package][1]
> > > > +            ))
> > > > +    Nodes.sort()
> > > > +
> > > > +    #
> > > > +    # Generate dot file from Nodes and Edges and
> > add a Legend at top of
> > > > graph
> > > > +    #
> > > > +    Dot = []
> > > > +    Dot.append('digraph {')
> > > > +    Dot.append('  rankdir=BT;')
> > > > +    Dot.append('  node
> > [shape=Mrecord,style=filled];')
> > > > +    Dot.append('')
> > > > +    Dot = Dot + Nodes
> > > > +    Dot.append('')
> > > > +    Dot = Dot + Edges
> > > > +    Dot.append('')
> > > > +    Dot.append('  subgraph legend {')
> > > > +    Dot.append('    rank=sink;')
> > > > +    Dot.append('    Unresolved
> > [label="Unresolved Dependency",
> > > > fillcolor=red];')
> > > > +    Dot.append('    Circular
> > [label="Circular Dependency",
> > > > fillcolor=yellow];')
> > > > +    Dot.append('    Nested         [label="Nested
> > Package",
> > > > fillcolor=pink];')
> > > > +    Dot.append('    NestedCircular [label="Nested
> > Package with Circular
> > > > Dependency",fillcolor=orange];')
> > > > +    Dot.append('  }')
> > > > +    if MaxWeightDecFile:
> > > > +        Dot.append('  Unresolved->"' +
> > MaxWeightDecFile + '" [style=invis];')
> > > > +    Dot.append('}')
> > > > +
> > > > +    if args.DotOutputFile:
> > > > +        #
> > > > +        # Write GraphViz dot file contents to
> > DotOutputFile
> > > > +        #
> > > > +        with
> > open(os.path.realpath(args.DotOutputFile), 'w') as
> > File:
> > > > +            File.write('\n'.join(Dot))
> > > > +    if args.SvgOutputFile:
> > > > +        #
> > > > +        # Use GraphViz 'dot' command to generate
> > SVG output file
> > > > +        #
> > > > +        args.SvgOutputFile =
> > os.path.realpath(args.SvgOutputFile)
> > > > +        try:
> > > > +            Process = subprocess.Popen('dot -
> > Tsvg',
> > > > +              stdin=subprocess.PIPE,
> > > > +              stdout=open(args.SvgOutputFile,
> > 'w'),
> > > > +              stderr=subprocess.PIPE,
> > > > +              shell=True
> > > > +              )
> > > > +            Process.stdin.write
> > ('\n'.join(Dot).encode())
> > > > +            Process.communicate()
> > > > +            if Process.returncode != 0:
> > > > +                print ("ERROR: Can not run
> > GraphViz 'dot' command.  Check install
> > > > and path.")
> > > > +                sys.exit(Process.returncode)
> > > > +        except:
> > > > +            print ("ERROR: Can not run GraphViz
> > 'dot' command.  Check install
> > > and
> > > > path.")
> > > > +            sys.exit(1)
> > > > +        #
> > > > +        # Display SVG file in default web browser
> > > > +        #
> > > > +        if args.WebBrowser:
> > > > +            webbrowser.open(args.SvgOutputFile)
> > > > --
> > > > 2.21.0.windows.1
> > >
> > >
> > > 

Re: [edk2-devel] [Patch 1/1] BaseTools/Scripts: Add package dependency graphing tool

[Steven Shi]

[-- Attachment #1.1: Type: text/plain, Size: 30371 bytes --]

Hi Sean,
Your tool is cool. I wish we could merge you and Mike’s tool together.

I create a dependency html report with your tool in attachment. I have below feedback:

  1.  The filters response in Dependency Explorer tab are slow.
  2.  Only have fanout views, but no fanin views. I have a Stability Metric to measure the stability of a module, and I need the module fanin data. For a given module m, let fanin(m) be the set of modules that depend on m and let fanout(m) be the set of modules that m depends on. Following Martin [1][2], the Instability I(m) of m is defined as fanout(m)/( fanin(m) + fanout(m)). I’m OK that the fanin(m) data only make sense for a real platform build, and only need to generate it if input a platform build report.
  3.  If possible, please combine and output all the dependency data like below as a single Json file, which will be easy for other metrics tool to consume.

    ModuleDeps = []  # hold all deps

    ModuleList = []

    AsBuiltMappings = []

    LibraryClassInstanceDict = {}


[1] https://fi.ort.edu.uy/innovaportal/file/2032/1/design_principles.pdf, page 23, Stability Metrics
[2] https://ieeexplore.ieee.org/abstract/document/4027146, chapter 6.3,  Module Interaction Stability Index

Thanks
Steven



From: Ni, Ray <ray.ni@intel.com>
Sent: Tuesday, December 17, 2019 10:49 AM
To: devel@edk2.groups.io; sean.brogan@microsoft.com; Kinney, Michael D <michael.d.kinney@intel.com>; Shi, Steven <steven.shi@intel.com>
Cc: Feng, Bob C <bob.c.feng@intel.com>; Gao, Liming <liming.gao@intel.com>; Bret Barkelew <Bret.Barkelew@microsoft.com>
Subject: RE: [edk2-devel] [Patch 1/1] BaseTools/Scripts: Add package dependency graphing tool

Sean,
This is a very cool tool! We could learn from your tool source code to know how to use DecParser to get more detailed information, like which interfaces (PPI/Protocol/Guid/PCD/Lib) from which package a module is depending on.

Thanks,
Ray


From: devel@edk2.groups.io<mailto:devel@edk2.groups.io> <devel@edk2.groups.io<mailto:devel@edk2.groups.io>> On Behalf Of Sean via Groups.Io
Sent: Tuesday, December 17, 2019 3:46 AM
To: Kinney, Michael D <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>>; Shi, Steven <steven.shi@intel.com<mailto:steven.shi@intel.com>>; devel@edk2.groups.io<mailto:devel@edk2.groups.io>; Ni, Ray <ray.ni@intel.com<mailto:ray.ni@intel.com>>; Kinney, Michael D <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>>
Cc: Feng, Bob C <bob.c.feng@intel.com<mailto:bob.c.feng@intel.com>>; Gao, Liming <liming.gao@intel.com<mailto:liming.gao@intel.com>>; Bret Barkelew <Bret.Barkelew@microsoft.com<mailto:Bret.Barkelew@microsoft.com>>
Subject: Re: [edk2-devel] [Patch 1/1] BaseTools/Scripts: Add package dependency graphing tool

Mike/Steven/Ray,

We have had a tool for a couple of years with similar functionality.  It is a python tool you run on your code base and it creates a single (local) webpage that you can then use to build charts.

Please give it a try and see if you think it is useful.

Here is the tool.
https://github.com/spbrogan/edk2_dep

And here is a html report run on edk2
https://github.com/spbrogan/edk2_dep/blob/sample/edk2/Edk2_core.html

Thanks
Sean


From: Kinney, Michael D <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>>
Sent: Monday, December 16, 2019 9:16 AM
To: Shi, Steven <steven.shi@intel.com<mailto:steven.shi@intel.com>>; devel@edk2.groups.io<mailto:devel@edk2.groups.io>; Ni, Ray <ray.ni@intel.com<mailto:ray.ni@intel.com>>; Kinney, Michael D <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>>
Cc: Feng, Bob C <bob.c.feng@intel.com<mailto:bob.c.feng@intel.com>>; Gao, Liming <liming.gao@intel.com<mailto:liming.gao@intel.com>>; Sean Brogan <sean.brogan@microsoft.com<mailto:sean.brogan@microsoft.com>>; Bret Barkelew <Bret.Barkelew@microsoft.com<mailto:Bret.Barkelew@microsoft.com>>
Subject: [EXTERNAL] RE: [edk2-devel] [Patch 1/1] BaseTools/Scripts: Add package dependency graphing tool

Steve,

Yes.  I think a Module scoped view and a Platform scoped view would be good additions.

Can you please enter new BZ feature requests for these and add a summary of the features you would like to see in each?

Thanks,

Mike

From: Shi, Steven <steven.shi@intel.com<mailto:steven.shi@intel.com>>
Sent: Monday, December 16, 2019 12:48 AM
To: devel@edk2.groups.io<mailto:devel@edk2.groups.io>; Ni, Ray <ray.ni@intel.com<mailto:ray.ni@intel.com>>; Kinney, Michael D <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>>
Cc: Feng, Bob C <bob.c.feng@intel.com<mailto:bob.c.feng@intel.com>>; Gao, Liming <liming.gao@intel.com<mailto:liming.gao@intel.com>>; Sean Brogan <sean.brogan@microsoft.com<mailto:sean.brogan@microsoft.com>>; Bret Barkelew <Bret.Barkelew@microsoft.com<mailto:Bret.Barkelew@microsoft.com>>
Subject: RE: [edk2-devel] [Patch 1/1] BaseTools/Scripts: Add package dependency graphing tool


Ray,

I think you are asking for a module-level dependency tool. 😊



Mike,

This tool is cool. Besides the package level, do you think we can go further to figure out a module level dependency tool? With module level dependency info, we can define more interesting and comprehensive Modularity Metrics to measure the edk2 architecture quality (Architectural Technical Debt).







Thanks



Steven Shi

Intel\SSG\SFE\FIE Firmware Infrastructure





> -----Original Message-----

> From: devel@edk2.groups.io<mailto:devel@edk2.groups.io> <devel@edk2.groups.io<mailto:devel@edk2.groups.io>> On Behalf Of Ni, Ray

> Sent: Monday, December 16, 2019 4:41 PM

> To: devel@edk2.groups.io<mailto:devel@edk2.groups.io>; Ni, Ray <ray.ni@intel.com<mailto:ray.ni@intel.com>>; Kinney, Michael D

> <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>>

> Cc: Feng, Bob C <bob.c.feng@intel.com<mailto:bob.c.feng@intel.com>>; Gao, Liming

> <liming.gao@intel.com<mailto:liming.gao@intel.com>>; Sean Brogan <sean.brogan@microsoft.com<mailto:sean.brogan@microsoft.com>>; Bret

> Barkelew <Bret.Barkelew@microsoft.com<mailto:Bret.Barkelew@microsoft.com>>

> Subject: Re: [edk2-devel] [Patch 1/1] BaseTools/Scripts: Add package

> dependency graphing tool

>

> Mike,

> This pkg dep tool can tell through weight when a module depends on a new

> pkg.

> But it cannot tell when a module depends on another

> PCD/Protocol/Guid/Ppi/Library which belongs to an already-depended pkg.

>

> Failure to tell such information may make a bad module (that violates the

> dependency rules) worse (wrongly depend on more interfaces).

>

> Do you think that the tool can be enhanced in future to detect the case?

>

> Thanks,

> Ray

>

> > -----Original Message-----

> > From: devel@edk2.groups.io<mailto:devel@edk2.groups.io> <devel@edk2.groups.io<mailto:devel@edk2.groups.io>> On Behalf Of Ni,

> Ray

> > Sent: Monday, December 16, 2019 1:57 PM

> > To: Kinney, Michael D <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>>;

> devel@edk2.groups.io<mailto:devel@edk2.groups.io>

> > Cc: Feng, Bob C <bob.c.feng@intel.com<mailto:bob.c.feng@intel.com>>; Gao, Liming

> > <liming.gao@intel.com<mailto:liming.gao@intel.com>>; Sean Brogan <sean.brogan@microsoft.com<mailto:sean.brogan@microsoft.com>>;

> Bret

> > Barkelew <Bret.Barkelew@microsoft.com<mailto:Bret.Barkelew@microsoft.com>>

> > Subject: Re: [edk2-devel] [Patch 1/1] BaseTools/Scripts: Add package

> > dependency graphing tool

> >

> > Mike,

> > 2 minor comments regarding the help string in below.

> >

> > > -----Original Message-----

> > > From: Kinney, Michael D <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>>

> > > Sent: Saturday, December 14, 2019 3:45 AM

> > > To: devel@edk2.groups.io<mailto:devel@edk2.groups.io>

> > > Cc: Ni, Ray <ray.ni@intel.com<mailto:ray.ni@intel.com>>; Feng, Bob C <bob.c.feng@intel.com<mailto:bob.c.feng@intel.com>>;

> Gao,

> > > Liming <liming.gao@intel.com<mailto:liming.gao@intel.com>>; Sean Brogan

> > <sean.brogan@microsoft.com<mailto:sean.brogan@microsoft.com>>;

> > > Bret Barkelew <Bret.Barkelew@microsoft.com<mailto:Bret.Barkelew@microsoft.com>>

> > > Subject: [Patch 1/1] BaseTools/Scripts: Add package dependency

> graphing

> > > tool

> > >

> > > https://bugzilla.tianocore.org/show_bug.cgi?id=2161<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugzilla.tianocore.org%2Fshow_bug.cgi%3Fid%3D2161&data=02%7C01%7Csean.brogan%40microsoft.com%7Cf7fb1953185b4a17f27408d7824ba488%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637121133718397648&sdata=NE82lFZmNfIcNjUfo49D%2Bksn94atxzdx7F2j%2FWSME0E%3D&reserved=0>

> > >

> > > Add python script that recursively scans a directory for EDK II

> > > packages and generates GraphViz dot input that is used to render

> > > a graph of package dependencies in SVG format.

> > >

> > > Detects following error/warning conditions:

> > > * Ambiguous dependencies (multiple matches)

> > > * Unresolved dependencies

> > > * Circular dependencies

> > > * Nested packages

> > >

> > > usage: PackageDependencyGraph [-h] [-w WORKSPACE] [-p

> > PACKAGESPATH]

> > >                               [-d DOTOUTPUTFILE] [-o SVGOUTPUTFILE]

> > >                               [-g IGNOREDIRECTORY] [-k SKIPPACKAGE]

> > >                               [-s] [-u] [-l] [-f] [-b] [-v] [-q]

> > >                               [--debug [0-9]]

> > >

> > > Recursively scan a directory for EDK II packages and generate GraphViz

> dot

> > > input that is used to render a graph of package dependencies in SVG

> > format.

> > > Copyright (c) 2019, Intel Corporation. All rights reserved.

> > >

> > > optional arguments:

> > >   -h, --help            show this help message and exit

> > >   -w WORKSPACE, --workspace WORKSPACE

> > >                         Directory to recursively scan for EDK II packages.

> > >                         Default is current directory.

> > >   -p PACKAGESPATH, --packages-path PACKAGESPATH

> > >                         List of directories to recursively scan for EDK II

> > >                         packages.

> > >   -d DOTOUTPUTFILE, --dot-output DOTOUTPUTFILE

> > >                         DOT output filename.

> > >   -o OUTPUTFILE, --output OUTPUTFILE

> > >                         SVG output filename.

> > >   -g IGNOREDIRECTORY, --ignore-directory IGNOREDIRECTORY

> > >                         Name of directory to ignore. Option can be repeated

> > >                         to ignore multiple directories.

> >

> > 1. Name of directory to ignore when scanning for EDKII Module INFs.

> >

> > >   -k SKIPPACKAGE, --skip-package SKIPPACKAGE

> > >                         Name of EDK II Package DEC file to skip. Option can

> > >                         be repeated to skip multiple EDK II packages.

> >

> > 2. Name of EDK II Package DEC file (including .DEC suffix) to skip. Option

> can

> > Be repeated to skip multiple EDK II packages.

> >

> > >   -s, --self-dependency

> > >                         Include self links in dependency graph. Default is

> > >                         disabled.

> > >   -u, --unresolved      Include unresolved EDK II packages in dependency

> > >                         graph. Default is disabled.

> > >   -l, --label           Label links with the number of EDK II package

> > >                         dependencies. Default is disabled.

> > >   -f, --full-paths      Label package nodes with full path to EDK II

> > >                         package.  Default is disabled.

> > >   -b, --web-browser     Display SVG output file in default web browser.

> > >                         Default is disabled.

> > >   -v, --verbose         Increase output messages

> > >   -q, --quiet           Reduce output messages

> > >   --debug [0-9]         Set debug level

> > >

> > > Cc: Ray Ni <ray.ni@intel.com<mailto:ray.ni@intel.com>>

> > > Cc: Bob Feng <bob.c.feng@intel.com<mailto:bob.c.feng@intel.com>>

> > > Cc: Liming Gao <liming.gao@intel.com<mailto:liming.gao@intel.com>>

> > > Cc: Sean Brogan <sean.brogan@microsoft.com<mailto:sean.brogan@microsoft.com>>

> > > Cc: Bret Barkelew <Bret.Barkelew@microsoft.com<mailto:Bret.Barkelew@microsoft.com>>

> > > Signed-off-by: Michael D Kinney <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>>

> > > ---

> > >  BaseTools/Scripts/PackageDependencyGraph.py | 296

> > > ++++++++++++++++++++

> > >  1 file changed, 296 insertions(+)

> > >  create mode 100644 BaseTools/Scripts/PackageDependencyGraph.py

> > >

> > > diff --git a/BaseTools/Scripts/PackageDependencyGraph.py

> > > b/BaseTools/Scripts/PackageDependencyGraph.py

> > > new file mode 100644

> > > index 0000000000..b3c8e41774

> > > --- /dev/null

> > > +++ b/BaseTools/Scripts/PackageDependencyGraph.py

> > > @@ -0,0 +1,296 @@

> > > +# @file

> > > +# Recursively scan a directory for EDK II packages and generate

> GraphViz

> > > dot

> > > +# input that is used to render a graph of package dependencies in SVG

> > > format.

> > > +#

> > > +# Copyright (c) 2019, Intel Corporation. All rights reserved.<BR>

> > > +# SPDX-License-Identifier: BSD-2-Clause-Patent

> > > +#

> > > +##

> > > +

> > > +import os

> > > +import sys

> > > +import argparse

> > > +import subprocess

> > > +import webbrowser

> > > +import networkx as nx

> > > +from edk2toollib.uefi.edk2.parsers.inf_parser import InfParser

> > > +

> > > +#

> > > +# Globals for help information

> > > +#

> > > +__prog__        = 'PackageDependencyGraph'

> > > +__copyright__   = 'Copyright (c) 2019, Intel Corporation. All rights

> > reserved.'

> > > +__description__ = '''Recursively scan a directory for EDK II packages and

> > > +generate GraphViz dot input that is used to render a graph of package

> > > +dependencies in SVG format.'''

> > > +

> > > +if __name__ == '__main__':

> > > +

> > > +    #

> > > +    # Create command line argument parser object

> > > +    #

> > > +    parser = argparse.ArgumentParser (prog = __prog__,

> > > +                                      description = __description__ + '\n' +

> __copyright__,

> > > +                                      conflict_handler = 'resolve')

> > > +    parser.add_argument ("-w", "--workspace", dest = 'Workspace',

> default

> > =

> > > os.curdir,

> > > +                         help = "Directory to recursively scan for EDK II packages.

> > > Default is current directory.")

> > > +    parser.add_argument ("-p", "--packages-path", dest = 'PackagesPath',

> > > default = None,

> > > +                         help = "List of directories to recursively scan for EDK II

> > > packages.")

> > > +    parser.add_argument ("-d", "--dot-output", dest = 'DotOutputFile',

> > > +                         help = "DOT output filename.")

> > > +    parser.add_argument ("-o", "--output", dest = 'SvgOutputFile',

> > > +                         help = "SVG output filename.")

> > > +    parser.add_argument ("-g", "--ignore-directory", dest =

> > 'IgnoreDirectory',

> > > action='append', default=[],

> > > +                         help = "Name of directory to ignore.  Option can be

> repeated

> > to

> > > ignore multiple directories.")

> > > +    parser.add_argument ("-k", "--skip-package", dest = 'SkipPackage',

> > > action='append', default=[],

> > > +                         help = "Name of EDK II Package DEC file to skip.  Option

> can

> > be

> > > repeated to skip multiple EDK II packages.")

> > > +    parser.add_argument ("-s", "--self-dependency", dest =

> > > 'SelfDependency', action = "store_true", default = False,

> > > +                         help = "Include self links in dependency graph.  Default is

> > > disabled.")

> > > +    parser.add_argument ("-u", "--unresolved", dest = 'Unresolved',

> action

> > =

> > > "store_true", default=False,

> > > +                         help = "Include unresolved EDK II packages in dependency

> > > graph.  Default is disabled.")

> > > +    parser.add_argument ("-l", "--label", dest = 'Label', action =

> > "store_true",

> > > default=False,

> > > +                         help = "Label links with the number of EDK II package

> > > dependencies.  Default is disabled.")

> > > +    parser.add_argument ("-f", "--full-paths", dest = 'FullPaths', action =

> > > "store_true", default=False,

> > > +                         help = "Label package nodes with full path to EDK II

> package.

> > > Default is disabled.")

> > > +    parser.add_argument ("-b", "--web-browser", dest = 'WebBrowser',

> > > action = "store_true", default=False,

> > > +                         help = "Display SVG output file in default web browser.

> > Default

> > > is disabled.")

> > > +    parser.add_argument ("-v", "--verbose", dest = 'Verbose', action =

> > > "store_true",

> > > +                         help = "Increase output messages")

> > > +    parser.add_argument ("-q", "--quiet", dest = 'Quiet', action =

> > > "store_true",

> > > +                         help = "Reduce output messages")

> > > +    parser.add_argument ("--debug", dest = 'Debug', type = int, metavar =

> > > '[0-9]', choices = range (0, 10), default = 0,

> > > +                         help = "Set debug level")

> > > +

> > > +    #

> > > +    # Parse command line arguments

> > > +    #

> > > +    args = parser.parse_args ()

> > > +

> > > +    #

> > > +    # Find all EDK II package DEC files

> > > +    #

> > > +    Components = {}

> > > +    SearchPaths = [args.Workspace]

> > > +    if args.PackagesPath:

> > > +        SearchPaths += args.PackagesPath.split(os.pathsep)

> > > +    SearchPaths = [os.path.realpath(x) for x in SearchPaths]

> > > +    for SearchPath in SearchPaths:

> > > +        for root, dirs, files in os.walk (SearchPath):

> > > +            for name in files:

> > > +                FilePath = os.path.join (root, name)

> > > +                if

> > > set(FilePath.split(os.sep)).intersection(set(args.IgnoreDirectory)) != set():

> > > +                    if args.Verbose:

> > > +                        print ('IGNORE:' + FilePath)

> > > +                    continue

> > > +                if os.path.splitext(FilePath)[1].lower() in ['.dec']:

> > > +                    DecFile = os.path.realpath (FilePath)

> > > +                    if os.path.split(DecFile)[1] in args.SkipPackage:

> > > +                        if args.Verbose:

> > > +                            print ('SKIP:' + DecFile)

> > > +                        continue

> > > +                    if DecFile not in Components:

> > > +                        if args.Verbose:

> > > +                            print ('PACKAGE:' + DecFile)

> > > +                        Components[DecFile] = {}

> > > +

> > > +    #

> > > +    # Find EDK II component INF files in each EDK II package

> > > +    #

> > > +    PackageLabels = {}

> > > +    UnresolvedPackages = []

> > > +    AmbiguousDependencies = []

> > > +    for DecFile in Components:

> > > +        DecPath = os.path.split (DecFile)[0]

> > > +        if DecFile not in PackageLabels:

> > > +            PackageLabels[DecFile] = (DecFile.replace(os.path.sep,'\\n'),

> 'white')

> > > +            if not args.FullPaths:

> > > +                PackagePath = os.path.relpath(DecFile,

> os.path.split(DecPath)[0])

> > > +                PackageLabels[DecFile] =

> (PackagePath.replace(os.path.sep,'\\n'),

> > > 'white')

> > > +        for root, dirs, files in os.walk (DecPath):

> > > +            for name in files:

> > > +                FilePath = os.path.join(root, name)

> > > +                if

> > > set(FilePath.split(os.sep)).intersection(set(args.IgnoreDirectory)) != set():

> > > +                    if args.Verbose:

> > > +                        print ('IGNORE:' + FilePath)

> > > +                    continue

> > > +                if os.path.splitext(FilePath)[1].lower() in ['.inf']:

> > > +                    InfFile = os.path.realpath (FilePath)

> > > +                    Inf = InfParser ()

> > > +                    Inf.ParseFile (InfFile)

> > > +                    DependentPackages = []

> > > +                    for Dependency in Inf.PackagesUsed:

> > > +                        Dependency = os.path.normpath(Dependency)

> > > +                        if os.path.split(Dependency)[1] in args.SkipPackage:

> > > +                            if args.Verbose:

> > > +                                print ('SKIP:' + Dependency)

> > > +                            continue

> > > +                        Found = False

> > > +                        for SearchPath in SearchPaths:

> > > +                            PackagePath = os.path.realpath(os.path.join(SearchPath,

> > > Dependency))

> > > +                            if os.path.exists(PackagePath):

> > > +                                DependentPackages.append(PackagePath)

> > > +                                if not args.FullPaths:

> > > +                                    PackageLabels[PackagePath] =

> > > (Dependency.replace(os.path.sep,'\\n'), 'white')

> > > +                                Found = True

> > > +                                break

> > > +                        if not Found:

> > > +                            Count = 0

> > > +                            Match = ''

> > > +                            for DecFile2 in Components:

> > > +                                if DecFile2.endswith(Dependency):

> > > +                                    if Count == 0:

> > > +                                        Match = DecFile2

> > > +                                    Count = Count + 1

> > > +                            if Count > 1:

> > > +                                AmbiguousDependencies.append (Dependency)

> > > +                            if Count == 1:

> > > +                                DependentPackages.append(Match)

> > > +                                if not args.FullPaths:

> > > +                                    PackageLabels[Match] =

> > > (Dependency.replace(os.path.sep,'\\n'), 'white')

> > > +                                Found = True

> > > +                        if not Found and args.Unresolved:

> > > +                            if args.Verbose:

> > > +                                print ('WARNING: Dependent package not found ' +

> > > Dependency)

> > > +                            DependentPackages.append(Dependency)

> > > +                            UnresolvedPackages.append(Dependency)

> > > +                            PackageLabels[Dependency] =

> > > (Dependency.replace(os.path.sep,'\\n'), 'white')

> > > +                    Components[DecFile][InfFile] = DependentPackages

> > > +    if AmbiguousDependencies:

> > > +        for Dependency in set(AmbiguousDependencies):

> > > +            print ('ERROR: MULTIPLE:   ' + Dependency)

> > > +        print ('Use --packages-path to provide search priority.')

> > > +        sys.exit(1)

> > > +

> > > +    #

> > > +    # Generate GraphViz dot input file contents.

> > > +    # Use networkx to detect circular dependencies.

> > > +    #

> > > +    MaxWeight = 0

> > > +    MaxWeightDecFile = list(Components.keys())[0]

> > > +    Graph = nx.DiGraph()

> > > +    Edges = []

> > > +    for DecFile in Components:

> > > +        if args.Verbose:

> > > +            print ('PACKAGE DEPENDENCIES:' + DecFile)

> > > +        AllDependencies = []

> > > +        Dependencies = set()

> > > +        for InfFile in Components[DecFile]:

> > > +            AllDependencies += Components[DecFile][InfFile]

> > > +            Dependencies =

> Dependencies.union(Components[DecFile][InfFile])

> > > +        if not args.SelfDependency:

> > > +            Dependencies = Dependencies.difference(set([DecFile]))

> > > +        for Dependency in Dependencies:

> > > +            Weight = AllDependencies.count(Dependency)

> > > +            if Weight > MaxWeight:

> > > +                MaxWeight = Weight

> > > +                MaxWeightDecFile = DecFile

> > > +            if args.Verbose:

> > > +                print ('  DEPENDENCY: Weight(' + str(Weight) + ') ' +

> Dependency)

> > > +            Edges.append('  "{Package}" -> "{Dependency}" [label =

> > > "{Weight}"];'.format(

> > > +                Package    = DecFile,

> > > +                Dependency = Dependency,

> > > +                Weight     = str(Weight) if args.Label else ''

> > > +                ))

> > > +            Graph.add_edge(DecFile, Dependency)

> > > +    Edges.sort()

> > > +

> > > +    #

> > > +    # Set fill color to yellow if a packages is part of a circular dependency

> > > +    #

> > > +    for Node in set([x for y in list(nx.simple_cycles(Graph)) for x in y]):

> > > +        PackageLabels[Node] = (PackageLabels[Node][0], 'yellow')

> > > +        print ('ERROR: CIRCULAR:   ' + Node)

> > > +

> > > +    #

> > > +    # Set fill color to pink if a package that is nested inside another

> package.

> > > +    # Set fill color to orange if a package is nested inside another package

> > and

> > > +    # is part of a circular dependency.

> > > +    #

> > > +    for DecFile in Components:

> > > +        DecPath = os.path.split (DecFile)[0]

> > > +        for DecFile2 in Components:

> > > +            if DecFile2 == DecFile:

> > > +                continue

> > > +            if os.path.commonpath ([DecPath, DecFile2]) != DecPath:

> > > +                continue

> > > +            if len(DecFile2) < len(DecPath):

> > > +                DecFile2 = DecFile

> > > +            print ('ERROR: NESTED:     ' + DecFile2)

> > > +            if PackageLabels[DecFile2][1] == 'yellow':

> > > +                PackageLabels[DecFile2] = (PackageLabels[DecFile2][0],

> 'orange')

> > > +            else:

> > > +                PackageLabels[DecFile2] = (PackageLabels[DecFile2][0], 'pink')

> > > +

> > > +    #

> > > +    # Set fill color to red for nodes that are unresolved

> > > +    #

> > > +    for Node in set(UnresolvedPackages):

> > > +        PackageLabels[Node] = (PackageLabels[Node][0], 'red')

> > > +        print ('ERROR: UNRESOLVED: ' + Node)

> > > +

> > > +    #

> > > +    # Add node statements to set node label and fill color

> > > +    #

> > > +    Nodes = []

> > > +    for Package in PackageLabels:

> > > +        Nodes.append('  "{Package}"

> > [label="{Label}",fillcolor={Color}];'.format(

> > > +            Package = Package,

> > > +            Label = PackageLabels[Package][0],

> > > +            Color = PackageLabels[Package][1]

> > > +            ))

> > > +    Nodes.sort()

> > > +

> > > +    #

> > > +    # Generate dot file from Nodes and Edges and add a Legend at top of

> > > graph

> > > +    #

> > > +    Dot = []

> > > +    Dot.append('digraph {')

> > > +    Dot.append('  rankdir=BT;')

> > > +    Dot.append('  node [shape=Mrecord,style=filled];')

> > > +    Dot.append('')

> > > +    Dot = Dot + Nodes

> > > +    Dot.append('')

> > > +    Dot = Dot + Edges

> > > +    Dot.append('')

> > > +    Dot.append('  subgraph legend {')

> > > +    Dot.append('    rank=sink;')

> > > +    Dot.append('    Unresolved     [label="Unresolved Dependency",

> > > fillcolor=red];')

> > > +    Dot.append('    Circular       [label="Circular Dependency",

> > > fillcolor=yellow];')

> > > +    Dot.append('    Nested         [label="Nested Package",

> > > fillcolor=pink];')

> > > +    Dot.append('    NestedCircular [label="Nested Package with Circular

> > > Dependency",fillcolor=orange];')

> > > +    Dot.append('  }')

> > > +    if MaxWeightDecFile:

> > > +        Dot.append('  Unresolved->"' + MaxWeightDecFile + '" [style=invis];')

> > > +    Dot.append('}')

> > > +

> > > +    if args.DotOutputFile:

> > > +        #

> > > +        # Write GraphViz dot file contents to DotOutputFile

> > > +        #

> > > +        with open(os.path.realpath(args.DotOutputFile), 'w') as File:

> > > +            File.write('\n'.join(Dot))

> > > +    if args.SvgOutputFile:

> > > +        #

> > > +        # Use GraphViz 'dot' command to generate SVG output file

> > > +        #

> > > +        args.SvgOutputFile = os.path.realpath(args.SvgOutputFile)

> > > +        try:

> > > +            Process = subprocess.Popen('dot -Tsvg',

> > > +              stdin=subprocess.PIPE,

> > > +              stdout=open(args.SvgOutputFile, 'w'),

> > > +              stderr=subprocess.PIPE,

> > > +              shell=True

> > > +              )

> > > +            Process.stdin.write ('\n'.join(Dot).encode())

> > > +            Process.communicate()

> > > +            if Process.returncode != 0:

> > > +                print ("ERROR: Can not run GraphViz 'dot' command.  Check

> install

> > > and path.")

> > > +                sys.exit(Process.returncode)

> > > +        except:

> > > +            print ("ERROR: Can not run GraphViz 'dot' command.  Check

> install

> > and

> > > path.")

> > > +            sys.exit(1)

> > > +        #

> > > +        # Display SVG file in default web browser

> > > +        #

> > > +        if args.WebBrowser:

> > > +            webbrowser.open(args.SvgOutputFile)

> > > --

> > > 2.21.0.windows.1

> >

> >

> >

>

>

>




[-- Attachment #1.2: Type: text/html, Size: 86215 bytes --]

[-- Attachment #2: dep_tool.zip --]
[-- Type: application/x-zip-compressed, Size: 301758 bytes --]

Re: [edk2-devel] [Patch 1/1] BaseTools/Scripts: Add package dependency graphing tool

[Steven Shi]

[-- Attachment #1: Type: text/plain, Size: 28031 bytes --]

I reuse the BZ 2161 and add my requirement there as below.
https://bugzilla.tianocore.org/show_bug.cgi?id=2161

1. If without platform build log or DSC info, the tool only need to output module fanout build dependency info which includes what APIs (PPI/Protocol/Guid/PCD/Lib) a module depends on. No need to tell what these API's implementation instance are  (e.g. you don't know which library instance implement the library class/API without platform build info).
2. If with platform build log or DSC info, the tool need to output both module fanin and fanout info. And for the fanin and fanout info, the tool need output both APIs and their implementation instances paths (only need library instance module path and Static PCDs which impact a module build).
3. Please be aware the above dependency is only for the build dependency, not for the functionality dependency. Because the Uefi PPI/Protocol/Guid API's instances are always dynamic resolved/binding, and are not really needed in other module's static build, so only the PPI/Protocol/Guid API's definitions, not their instances, are the dependency of other module. But the library and static PCD are different, and their instances are necessary for other depending module pass build, so the library and static PCD instances are the dependency of other module.
4. Besides graph views, please combine and output all the dependency data as a single Json file, which will be easy for other metrics tool to consume.


Thanks

Steven Shi
Intel\SSG\SFE\FIE Firmware Infrastructure


From: Kinney, Michael D <michael.d.kinney@intel.com>
Sent: Tuesday, December 17, 2019 1:16 AM
To: Shi, Steven <steven.shi@intel.com>; devel@edk2.groups.io; Ni, Ray <ray.ni@intel.com>; Kinney, Michael D <michael.d.kinney@intel.com>
Cc: Feng, Bob C <bob.c.feng@intel.com>; Gao, Liming <liming.gao@intel.com>; Sean Brogan <sean.brogan@microsoft.com>; Bret Barkelew <Bret.Barkelew@microsoft.com>
Subject: RE: [edk2-devel] [Patch 1/1] BaseTools/Scripts: Add package dependency graphing tool

Steve,

Yes.  I think a Module scoped view and a Platform scoped view would be good additions.

Can you please enter new BZ feature requests for these and add a summary of the features you would like to see in each?

Thanks,

Mike

From: Shi, Steven <steven.shi@intel.com<mailto:steven.shi@intel.com>>
Sent: Monday, December 16, 2019 12:48 AM
To: devel@edk2.groups.io<mailto:devel@edk2.groups.io>; Ni, Ray <ray.ni@intel.com<mailto:ray.ni@intel.com>>; Kinney, Michael D <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>>
Cc: Feng, Bob C <bob.c.feng@intel.com<mailto:bob.c.feng@intel.com>>; Gao, Liming <liming.gao@intel.com<mailto:liming.gao@intel.com>>; Sean Brogan <sean.brogan@microsoft.com<mailto:sean.brogan@microsoft.com>>; Bret Barkelew <Bret.Barkelew@microsoft.com<mailto:Bret.Barkelew@microsoft.com>>
Subject: RE: [edk2-devel] [Patch 1/1] BaseTools/Scripts: Add package dependency graphing tool


Ray,

I think you are asking for a module-level dependency tool. 😊



Mike,

This tool is cool. Besides the package level, do you think we can go further to figure out a module level dependency tool? With module level dependency info, we can define more interesting and comprehensive Modularity Metrics to measure the edk2 architecture quality (Architectural Technical Debt).







Thanks



Steven Shi

Intel\SSG\SFE\FIE Firmware Infrastructure





> -----Original Message-----

> From: devel@edk2.groups.io<mailto:devel@edk2.groups.io> <devel@edk2.groups.io<mailto:devel@edk2.groups.io>> On Behalf Of Ni, Ray

> Sent: Monday, December 16, 2019 4:41 PM

> To: devel@edk2.groups.io<mailto:devel@edk2.groups.io>; Ni, Ray <ray.ni@intel.com<mailto:ray.ni@intel.com>>; Kinney, Michael D

> <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>>

> Cc: Feng, Bob C <bob.c.feng@intel.com<mailto:bob.c.feng@intel.com>>; Gao, Liming

> <liming.gao@intel.com<mailto:liming.gao@intel.com>>; Sean Brogan <sean.brogan@microsoft.com<mailto:sean.brogan@microsoft.com>>; Bret

> Barkelew <Bret.Barkelew@microsoft.com<mailto:Bret.Barkelew@microsoft.com>>

> Subject: Re: [edk2-devel] [Patch 1/1] BaseTools/Scripts: Add package

> dependency graphing tool

>

> Mike,

> This pkg dep tool can tell through weight when a module depends on a new

> pkg.

> But it cannot tell when a module depends on another

> PCD/Protocol/Guid/Ppi/Library which belongs to an already-depended pkg.

>

> Failure to tell such information may make a bad module (that violates the

> dependency rules) worse (wrongly depend on more interfaces).

>

> Do you think that the tool can be enhanced in future to detect the case?

>

> Thanks,

> Ray

>

> > -----Original Message-----

> > From: devel@edk2.groups.io<mailto:devel@edk2.groups.io> <devel@edk2.groups.io<mailto:devel@edk2.groups.io>> On Behalf Of Ni,

> Ray

> > Sent: Monday, December 16, 2019 1:57 PM

> > To: Kinney, Michael D <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>>;

> devel@edk2.groups.io<mailto:devel@edk2.groups.io>

> > Cc: Feng, Bob C <bob.c.feng@intel.com<mailto:bob.c.feng@intel.com>>; Gao, Liming

> > <liming.gao@intel.com<mailto:liming.gao@intel.com>>; Sean Brogan <sean.brogan@microsoft.com<mailto:sean.brogan@microsoft.com>>;

> Bret

> > Barkelew <Bret.Barkelew@microsoft.com<mailto:Bret.Barkelew@microsoft.com>>

> > Subject: Re: [edk2-devel] [Patch 1/1] BaseTools/Scripts: Add package

> > dependency graphing tool

> >

> > Mike,

> > 2 minor comments regarding the help string in below.

> >

> > > -----Original Message-----

> > > From: Kinney, Michael D <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>>

> > > Sent: Saturday, December 14, 2019 3:45 AM

> > > To: devel@edk2.groups.io<mailto:devel@edk2.groups.io>

> > > Cc: Ni, Ray <ray.ni@intel.com<mailto:ray.ni@intel.com>>; Feng, Bob C <bob.c.feng@intel.com<mailto:bob.c.feng@intel.com>>;

> Gao,

> > > Liming <liming.gao@intel.com<mailto:liming.gao@intel.com>>; Sean Brogan

> > <sean.brogan@microsoft.com<mailto:sean.brogan@microsoft.com>>;

> > > Bret Barkelew <Bret.Barkelew@microsoft.com<mailto:Bret.Barkelew@microsoft.com>>

> > > Subject: [Patch 1/1] BaseTools/Scripts: Add package dependency

> graphing

> > > tool

> > >

> > > https://bugzilla.tianocore.org/show_bug.cgi?id=2161

> > >

> > > Add python script that recursively scans a directory for EDK II

> > > packages and generates GraphViz dot input that is used to render

> > > a graph of package dependencies in SVG format.

> > >

> > > Detects following error/warning conditions:

> > > * Ambiguous dependencies (multiple matches)

> > > * Unresolved dependencies

> > > * Circular dependencies

> > > * Nested packages

> > >

> > > usage: PackageDependencyGraph [-h] [-w WORKSPACE] [-p

> > PACKAGESPATH]

> > >                               [-d DOTOUTPUTFILE] [-o SVGOUTPUTFILE]

> > >                               [-g IGNOREDIRECTORY] [-k SKIPPACKAGE]

> > >                               [-s] [-u] [-l] [-f] [-b] [-v] [-q]

> > >                               [--debug [0-9]]

> > >

> > > Recursively scan a directory for EDK II packages and generate GraphViz

> dot

> > > input that is used to render a graph of package dependencies in SVG

> > format.

> > > Copyright (c) 2019, Intel Corporation. All rights reserved.

> > >

> > > optional arguments:

> > >   -h, --help            show this help message and exit

> > >   -w WORKSPACE, --workspace WORKSPACE

> > >                         Directory to recursively scan for EDK II packages.

> > >                         Default is current directory.

> > >   -p PACKAGESPATH, --packages-path PACKAGESPATH

> > >                         List of directories to recursively scan for EDK II

> > >                         packages.

> > >   -d DOTOUTPUTFILE, --dot-output DOTOUTPUTFILE

> > >                         DOT output filename.

> > >   -o OUTPUTFILE, --output OUTPUTFILE

> > >                         SVG output filename.

> > >   -g IGNOREDIRECTORY, --ignore-directory IGNOREDIRECTORY

> > >                         Name of directory to ignore. Option can be repeated

> > >                         to ignore multiple directories.

> >

> > 1. Name of directory to ignore when scanning for EDKII Module INFs.

> >

> > >   -k SKIPPACKAGE, --skip-package SKIPPACKAGE

> > >                         Name of EDK II Package DEC file to skip. Option can

> > >                         be repeated to skip multiple EDK II packages.

> >

> > 2. Name of EDK II Package DEC file (including .DEC suffix) to skip. Option

> can

> > Be repeated to skip multiple EDK II packages.

> >

> > >   -s, --self-dependency

> > >                         Include self links in dependency graph. Default is

> > >                         disabled.

> > >   -u, --unresolved      Include unresolved EDK II packages in dependency

> > >                         graph. Default is disabled.

> > >   -l, --label           Label links with the number of EDK II package

> > >                         dependencies. Default is disabled.

> > >   -f, --full-paths      Label package nodes with full path to EDK II

> > >                         package.  Default is disabled.

> > >   -b, --web-browser     Display SVG output file in default web browser.

> > >                         Default is disabled.

> > >   -v, --verbose         Increase output messages

> > >   -q, --quiet           Reduce output messages

> > >   --debug [0-9]         Set debug level

> > >

> > > Cc: Ray Ni <ray.ni@intel.com<mailto:ray.ni@intel.com>>

> > > Cc: Bob Feng <bob.c.feng@intel.com<mailto:bob.c.feng@intel.com>>

> > > Cc: Liming Gao <liming.gao@intel.com<mailto:liming.gao@intel.com>>

> > > Cc: Sean Brogan <sean.brogan@microsoft.com<mailto:sean.brogan@microsoft.com>>

> > > Cc: Bret Barkelew <Bret.Barkelew@microsoft.com<mailto:Bret.Barkelew@microsoft.com>>

> > > Signed-off-by: Michael D Kinney <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>>

> > > ---

> > >  BaseTools/Scripts/PackageDependencyGraph.py | 296

> > > ++++++++++++++++++++

> > >  1 file changed, 296 insertions(+)

> > >  create mode 100644 BaseTools/Scripts/PackageDependencyGraph.py

> > >

> > > diff --git a/BaseTools/Scripts/PackageDependencyGraph.py

> > > b/BaseTools/Scripts/PackageDependencyGraph.py

> > > new file mode 100644

> > > index 0000000000..b3c8e41774

> > > --- /dev/null

> > > +++ b/BaseTools/Scripts/PackageDependencyGraph.py

> > > @@ -0,0 +1,296 @@

> > > +# @file

> > > +# Recursively scan a directory for EDK II packages and generate

> GraphViz

> > > dot

> > > +# input that is used to render a graph of package dependencies in SVG

> > > format.

> > > +#

> > > +# Copyright (c) 2019, Intel Corporation. All rights reserved.<BR>

> > > +# SPDX-License-Identifier: BSD-2-Clause-Patent

> > > +#

> > > +##

> > > +

> > > +import os

> > > +import sys

> > > +import argparse

> > > +import subprocess

> > > +import webbrowser

> > > +import networkx as nx

> > > +from edk2toollib.uefi.edk2.parsers.inf_parser import InfParser

> > > +

> > > +#

> > > +# Globals for help information

> > > +#

> > > +__prog__        = 'PackageDependencyGraph'

> > > +__copyright__   = 'Copyright (c) 2019, Intel Corporation. All rights

> > reserved.'

> > > +__description__ = '''Recursively scan a directory for EDK II packages and

> > > +generate GraphViz dot input that is used to render a graph of package

> > > +dependencies in SVG format.'''

> > > +

> > > +if __name__ == '__main__':

> > > +

> > > +    #

> > > +    # Create command line argument parser object

> > > +    #

> > > +    parser = argparse.ArgumentParser (prog = __prog__,

> > > +                                      description = __description__ + '\n' +

> __copyright__,

> > > +                                      conflict_handler = 'resolve')

> > > +    parser.add_argument ("-w", "--workspace", dest = 'Workspace',

> default

> > =

> > > os.curdir,

> > > +                         help = "Directory to recursively scan for EDK II packages.

> > > Default is current directory.")

> > > +    parser.add_argument ("-p", "--packages-path", dest = 'PackagesPath',

> > > default = None,

> > > +                         help = "List of directories to recursively scan for EDK II

> > > packages.")

> > > +    parser.add_argument ("-d", "--dot-output", dest = 'DotOutputFile',

> > > +                         help = "DOT output filename.")

> > > +    parser.add_argument ("-o", "--output", dest = 'SvgOutputFile',

> > > +                         help = "SVG output filename.")

> > > +    parser.add_argument ("-g", "--ignore-directory", dest =

> > 'IgnoreDirectory',

> > > action='append', default=[],

> > > +                         help = "Name of directory to ignore.  Option can be

> repeated

> > to

> > > ignore multiple directories.")

> > > +    parser.add_argument ("-k", "--skip-package", dest = 'SkipPackage',

> > > action='append', default=[],

> > > +                         help = "Name of EDK II Package DEC file to skip.  Option

> can

> > be

> > > repeated to skip multiple EDK II packages.")

> > > +    parser.add_argument ("-s", "--self-dependency", dest =

> > > 'SelfDependency', action = "store_true", default = False,

> > > +                         help = "Include self links in dependency graph.  Default is

> > > disabled.")

> > > +    parser.add_argument ("-u", "--unresolved", dest = 'Unresolved',

> action

> > =

> > > "store_true", default=False,

> > > +                         help = "Include unresolved EDK II packages in dependency

> > > graph.  Default is disabled.")

> > > +    parser.add_argument ("-l", "--label", dest = 'Label', action =

> > "store_true",

> > > default=False,

> > > +                         help = "Label links with the number of EDK II package

> > > dependencies.  Default is disabled.")

> > > +    parser.add_argument ("-f", "--full-paths", dest = 'FullPaths', action =

> > > "store_true", default=False,

> > > +                         help = "Label package nodes with full path to EDK II

> package.

> > > Default is disabled.")

> > > +    parser.add_argument ("-b", "--web-browser", dest = 'WebBrowser',

> > > action = "store_true", default=False,

> > > +                         help = "Display SVG output file in default web browser.

> > Default

> > > is disabled.")

> > > +    parser.add_argument ("-v", "--verbose", dest = 'Verbose', action =

> > > "store_true",

> > > +                         help = "Increase output messages")

> > > +    parser.add_argument ("-q", "--quiet", dest = 'Quiet', action =

> > > "store_true",

> > > +                         help = "Reduce output messages")

> > > +    parser.add_argument ("--debug", dest = 'Debug', type = int, metavar =

> > > '[0-9]', choices = range (0, 10), default = 0,

> > > +                         help = "Set debug level")

> > > +

> > > +    #

> > > +    # Parse command line arguments

> > > +    #

> > > +    args = parser.parse_args ()

> > > +

> > > +    #

> > > +    # Find all EDK II package DEC files

> > > +    #

> > > +    Components = {}

> > > +    SearchPaths = [args.Workspace]

> > > +    if args.PackagesPath:

> > > +        SearchPaths += args.PackagesPath.split(os.pathsep)

> > > +    SearchPaths = [os.path.realpath(x) for x in SearchPaths]

> > > +    for SearchPath in SearchPaths:

> > > +        for root, dirs, files in os.walk (SearchPath):

> > > +            for name in files:

> > > +                FilePath = os.path.join (root, name)

> > > +                if

> > > set(FilePath.split(os.sep)).intersection(set(args.IgnoreDirectory)) != set():

> > > +                    if args.Verbose:

> > > +                        print ('IGNORE:' + FilePath)

> > > +                    continue

> > > +                if os.path.splitext(FilePath)[1].lower() in ['.dec']:

> > > +                    DecFile = os.path.realpath (FilePath)

> > > +                    if os.path.split(DecFile)[1] in args.SkipPackage:

> > > +                        if args.Verbose:

> > > +                            print ('SKIP:' + DecFile)

> > > +                        continue

> > > +                    if DecFile not in Components:

> > > +                        if args.Verbose:

> > > +                            print ('PACKAGE:' + DecFile)

> > > +                        Components[DecFile] = {}

> > > +

> > > +    #

> > > +    # Find EDK II component INF files in each EDK II package

> > > +    #

> > > +    PackageLabels = {}

> > > +    UnresolvedPackages = []

> > > +    AmbiguousDependencies = []

> > > +    for DecFile in Components:

> > > +        DecPath = os.path.split (DecFile)[0]

> > > +        if DecFile not in PackageLabels:

> > > +            PackageLabels[DecFile] = (DecFile.replace(os.path.sep,'\\n'),

> 'white')

> > > +            if not args.FullPaths:

> > > +                PackagePath = os.path.relpath(DecFile,

> os.path.split(DecPath)[0])

> > > +                PackageLabels[DecFile] =

> (PackagePath.replace(os.path.sep,'\\n'),

> > > 'white')

> > > +        for root, dirs, files in os.walk (DecPath):

> > > +            for name in files:

> > > +                FilePath = os.path.join(root, name)

> > > +                if

> > > set(FilePath.split(os.sep)).intersection(set(args.IgnoreDirectory)) != set():

> > > +                    if args.Verbose:

> > > +                        print ('IGNORE:' + FilePath)

> > > +                    continue

> > > +                if os.path.splitext(FilePath)[1].lower() in ['.inf']:

> > > +                    InfFile = os.path.realpath (FilePath)

> > > +                    Inf = InfParser ()

> > > +                    Inf.ParseFile (InfFile)

> > > +                    DependentPackages = []

> > > +                    for Dependency in Inf.PackagesUsed:

> > > +                        Dependency = os.path.normpath(Dependency)

> > > +                        if os.path.split(Dependency)[1] in args.SkipPackage:

> > > +                            if args.Verbose:

> > > +                                print ('SKIP:' + Dependency)

> > > +                            continue

> > > +                        Found = False

> > > +                        for SearchPath in SearchPaths:

> > > +                            PackagePath = os.path.realpath(os.path.join(SearchPath,

> > > Dependency))

> > > +                            if os.path.exists(PackagePath):

> > > +                                DependentPackages.append(PackagePath)

> > > +                                if not args.FullPaths:

> > > +                                    PackageLabels[PackagePath] =

> > > (Dependency.replace(os.path.sep,'\\n'), 'white')

> > > +                                Found = True

> > > +                                break

> > > +                        if not Found:

> > > +                            Count = 0

> > > +                            Match = ''

> > > +                            for DecFile2 in Components:

> > > +                                if DecFile2.endswith(Dependency):

> > > +                                    if Count == 0:

> > > +                                        Match = DecFile2

> > > +                                    Count = Count + 1

> > > +                            if Count > 1:

> > > +                                AmbiguousDependencies.append (Dependency)

> > > +                            if Count == 1:

> > > +                                DependentPackages.append(Match)

> > > +                                if not args.FullPaths:

> > > +                                    PackageLabels[Match] =

> > > (Dependency.replace(os.path.sep,'\\n'), 'white')

> > > +                                Found = True

> > > +                        if not Found and args.Unresolved:

> > > +                            if args.Verbose:

> > > +                                print ('WARNING: Dependent package not found ' +

> > > Dependency)

> > > +                            DependentPackages.append(Dependency)

> > > +                            UnresolvedPackages.append(Dependency)

> > > +                            PackageLabels[Dependency] =

> > > (Dependency.replace(os.path.sep,'\\n'), 'white')

> > > +                    Components[DecFile][InfFile] = DependentPackages

> > > +    if AmbiguousDependencies:

> > > +        for Dependency in set(AmbiguousDependencies):

> > > +            print ('ERROR: MULTIPLE:   ' + Dependency)

> > > +        print ('Use --packages-path to provide search priority.')

> > > +        sys.exit(1)

> > > +

> > > +    #

> > > +    # Generate GraphViz dot input file contents.

> > > +    # Use networkx to detect circular dependencies.

> > > +    #

> > > +    MaxWeight = 0

> > > +    MaxWeightDecFile = list(Components.keys())[0]

> > > +    Graph = nx.DiGraph()

> > > +    Edges = []

> > > +    for DecFile in Components:

> > > +        if args.Verbose:

> > > +            print ('PACKAGE DEPENDENCIES:' + DecFile)

> > > +        AllDependencies = []

> > > +        Dependencies = set()

> > > +        for InfFile in Components[DecFile]:

> > > +            AllDependencies += Components[DecFile][InfFile]

> > > +            Dependencies =

> Dependencies.union(Components[DecFile][InfFile])

> > > +        if not args.SelfDependency:

> > > +            Dependencies = Dependencies.difference(set([DecFile]))

> > > +        for Dependency in Dependencies:

> > > +            Weight = AllDependencies.count(Dependency)

> > > +            if Weight > MaxWeight:

> > > +                MaxWeight = Weight

> > > +                MaxWeightDecFile = DecFile

> > > +            if args.Verbose:

> > > +                print ('  DEPENDENCY: Weight(' + str(Weight) + ') ' +

> Dependency)

> > > +            Edges.append('  "{Package}" -> "{Dependency}" [label =

> > > "{Weight}"];'.format(

> > > +                Package    = DecFile,

> > > +                Dependency = Dependency,

> > > +                Weight     = str(Weight) if args.Label else ''

> > > +                ))

> > > +            Graph.add_edge(DecFile, Dependency)

> > > +    Edges.sort()

> > > +

> > > +    #

> > > +    # Set fill color to yellow if a packages is part of a circular dependency

> > > +    #

> > > +    for Node in set([x for y in list(nx.simple_cycles(Graph)) for x in y]):

> > > +        PackageLabels[Node] = (PackageLabels[Node][0], 'yellow')

> > > +        print ('ERROR: CIRCULAR:   ' + Node)

> > > +

> > > +    #

> > > +    # Set fill color to pink if a package that is nested inside another

> package.

> > > +    # Set fill color to orange if a package is nested inside another package

> > and

> > > +    # is part of a circular dependency.

> > > +    #

> > > +    for DecFile in Components:

> > > +        DecPath = os.path.split (DecFile)[0]

> > > +        for DecFile2 in Components:

> > > +            if DecFile2 == DecFile:

> > > +                continue

> > > +            if os.path.commonpath ([DecPath, DecFile2]) != DecPath:

> > > +                continue

> > > +            if len(DecFile2) < len(DecPath):

> > > +                DecFile2 = DecFile

> > > +            print ('ERROR: NESTED:     ' + DecFile2)

> > > +            if PackageLabels[DecFile2][1] == 'yellow':

> > > +                PackageLabels[DecFile2] = (PackageLabels[DecFile2][0],

> 'orange')

> > > +            else:

> > > +                PackageLabels[DecFile2] = (PackageLabels[DecFile2][0], 'pink')

> > > +

> > > +    #

> > > +    # Set fill color to red for nodes that are unresolved

> > > +    #

> > > +    for Node in set(UnresolvedPackages):

> > > +        PackageLabels[Node] = (PackageLabels[Node][0], 'red')

> > > +        print ('ERROR: UNRESOLVED: ' + Node)

> > > +

> > > +    #

> > > +    # Add node statements to set node label and fill color

> > > +    #

> > > +    Nodes = []

> > > +    for Package in PackageLabels:

> > > +        Nodes.append('  "{Package}"

> > [label="{Label}",fillcolor={Color}];'.format(

> > > +            Package = Package,

> > > +            Label = PackageLabels[Package][0],

> > > +            Color = PackageLabels[Package][1]

> > > +            ))

> > > +    Nodes.sort()

> > > +

> > > +    #

> > > +    # Generate dot file from Nodes and Edges and add a Legend at top of

> > > graph

> > > +    #

> > > +    Dot = []

> > > +    Dot.append('digraph {')

> > > +    Dot.append('  rankdir=BT;')

> > > +    Dot.append('  node [shape=Mrecord,style=filled];')

> > > +    Dot.append('')

> > > +    Dot = Dot + Nodes

> > > +    Dot.append('')

> > > +    Dot = Dot + Edges

> > > +    Dot.append('')

> > > +    Dot.append('  subgraph legend {')

> > > +    Dot.append('    rank=sink;')

> > > +    Dot.append('    Unresolved     [label="Unresolved Dependency",

> > > fillcolor=red];')

> > > +    Dot.append('    Circular       [label="Circular Dependency",

> > > fillcolor=yellow];')

> > > +    Dot.append('    Nested         [label="Nested Package",

> > > fillcolor=pink];')

> > > +    Dot.append('    NestedCircular [label="Nested Package with Circular

> > > Dependency",fillcolor=orange];')

> > > +    Dot.append('  }')

> > > +    if MaxWeightDecFile:

> > > +        Dot.append('  Unresolved->"' + MaxWeightDecFile + '" [style=invis];')

> > > +    Dot.append('}')

> > > +

> > > +    if args.DotOutputFile:

> > > +        #

> > > +        # Write GraphViz dot file contents to DotOutputFile

> > > +        #

> > > +        with open(os.path.realpath(args.DotOutputFile), 'w') as File:

> > > +            File.write('\n'.join(Dot))

> > > +    if args.SvgOutputFile:

> > > +        #

> > > +        # Use GraphViz 'dot' command to generate SVG output file

> > > +        #

> > > +        args.SvgOutputFile = os.path.realpath(args.SvgOutputFile)

> > > +        try:

> > > +            Process = subprocess.Popen('dot -Tsvg',

> > > +              stdin=subprocess.PIPE,

> > > +              stdout=open(args.SvgOutputFile, 'w'),

> > > +              stderr=subprocess.PIPE,

> > > +              shell=True

> > > +              )

> > > +            Process.stdin.write ('\n'.join(Dot).encode())

> > > +            Process.communicate()

> > > +            if Process.returncode != 0:

> > > +                print ("ERROR: Can not run GraphViz 'dot' command.  Check

> install

> > > and path.")

> > > +                sys.exit(Process.returncode)

> > > +        except:

> > > +            print ("ERROR: Can not run GraphViz 'dot' command.  Check

> install

> > and

> > > path.")

> > > +            sys.exit(1)

> > > +        #

> > > +        # Display SVG file in default web browser

> > > +        #

> > > +        if args.WebBrowser:

> > > +            webbrowser.open(args.SvgOutputFile)

> > > --

> > > 2.21.0.windows.1

> >

> >

> >

>

>

> 



[-- Attachment #2: Type: text/html, Size: 80142 bytes --]