From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by mx.groups.io with SMTP id smtpd.web09.3238.1576475795353933190 for ; Sun, 15 Dec 2019 21:56:35 -0800 Authentication-Results: mx.groups.io; dkim=missing; spf=pass (domain: intel.com, ip: 134.134.136.24, mailfrom: ray.ni@intel.com) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by orsmga102.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 15 Dec 2019 21:56:34 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.69,320,1571727600"; d="scan'208";a="246906782" Received: from fmsmsx103.amr.corp.intel.com ([10.18.124.201]) by fmsmga002.fm.intel.com with ESMTP; 15 Dec 2019 21:56:34 -0800 Received: from shsmsx151.ccr.corp.intel.com (10.239.6.50) by FMSMSX103.amr.corp.intel.com (10.18.124.201) with Microsoft SMTP Server (TLS) id 14.3.439.0; Sun, 15 Dec 2019 21:56:34 -0800 Received: from shsmsx104.ccr.corp.intel.com ([169.254.5.90]) by SHSMSX151.ccr.corp.intel.com ([169.254.3.214]) with mapi id 14.03.0439.000; Mon, 16 Dec 2019 13:56:33 +0800 From: "Ni, Ray" To: "Kinney, Michael D" , "devel@edk2.groups.io" CC: "Feng, Bob C" , "Gao, Liming" , Sean Brogan , "Bret Barkelew" Subject: Re: [Patch 1/1] BaseTools/Scripts: Add package dependency graphing tool Thread-Topic: [Patch 1/1] BaseTools/Scripts: Add package dependency graphing tool Thread-Index: AQHVse3J5mlN2J1SSUCegIkulR/HMae8Q7kg Date: Mon, 16 Dec 2019 05:56:32 +0000 Message-ID: <734D49CCEBEEF84792F5B80ED585239D5C39EAFF@SHSMSX104.ccr.corp.intel.com> References: <20191213194449.7036-1-michael.d.kinney@intel.com> <20191213194449.7036-2-michael.d.kinney@intel.com> In-Reply-To: <20191213194449.7036-2-michael.d.kinney@intel.com> Accept-Language: en-US, zh-CN X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.239.127.40] MIME-Version: 1.0 Return-Path: ray.ni@intel.com Content-Language: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Mike, 2 minor comments regarding the help string in below. > -----Original Message----- > From: Kinney, Michael D > Sent: Saturday, December 14, 2019 3:45 AM > To: devel@edk2.groups.io > Cc: Ni, Ray ; Feng, Bob C ; Gao, > Liming ; Sean Brogan ; > Bret Barkelew > Subject: [Patch 1/1] BaseTools/Scripts: Add package dependency graphing > tool >=20 > https://bugzilla.tianocore.org/show_bug.cgi?id=3D2161 >=20 > Add python script that recursively scans a directory for EDK II > packages and generates GraphViz dot input that is used to render > a graph of package dependencies in SVG format. >=20 > Detects following error/warning conditions: > * Ambiguous dependencies (multiple matches) > * Unresolved dependencies > * Circular dependencies > * Nested packages >=20 > usage: PackageDependencyGraph [-h] [-w WORKSPACE] [-p PACKAGESPATH] > [-d DOTOUTPUTFILE] [-o SVGOUTPUTFILE] > [-g IGNOREDIRECTORY] [-k SKIPPACKAGE] > [-s] [-u] [-l] [-f] [-b] [-v] [-q] > [--debug [0-9]] >=20 > Recursively scan a directory for EDK II packages and generate GraphViz do= t > input that is used to render a graph of package dependencies in SVG forma= t. > Copyright (c) 2019, Intel Corporation. All rights reserved. >=20 > optional arguments: > -h, --help show this help message and exit > -w WORKSPACE, --workspace WORKSPACE > Directory to recursively scan for EDK II packages= . > Default is current directory. > -p PACKAGESPATH, --packages-path PACKAGESPATH > List of directories to recursively scan for EDK I= I > packages. > -d DOTOUTPUTFILE, --dot-output DOTOUTPUTFILE > DOT output filename. > -o OUTPUTFILE, --output OUTPUTFILE > SVG output filename. > -g IGNOREDIRECTORY, --ignore-directory IGNOREDIRECTORY > Name of directory to ignore. Option can be repeat= ed > to ignore multiple directories. 1. Name of directory to ignore when scanning for EDKII Module INFs. > -k SKIPPACKAGE, --skip-package SKIPPACKAGE > Name of EDK II Package DEC file to skip. Option c= an > be repeated to skip multiple EDK II packages. 2. Name of EDK II Package DEC file (including .DEC suffix) to skip. Option = can Be repeated to skip multiple EDK II packages. > -s, --self-dependency > Include self links in dependency graph. Default i= s > disabled. > -u, --unresolved Include unresolved EDK II packages in dependency > graph. Default is disabled. > -l, --label Label links with the number of EDK II package > dependencies. Default is disabled. > -f, --full-paths Label package nodes with full path to EDK II > package. Default is disabled. > -b, --web-browser Display SVG output file in default web browser. > Default is disabled. > -v, --verbose Increase output messages > -q, --quiet Reduce output messages > --debug [0-9] Set debug level >=20 > Cc: Ray Ni > Cc: Bob Feng > Cc: Liming Gao > Cc: Sean Brogan > Cc: Bret Barkelew > Signed-off-by: Michael D Kinney > --- > BaseTools/Scripts/PackageDependencyGraph.py | 296 > ++++++++++++++++++++ > 1 file changed, 296 insertions(+) > create mode 100644 BaseTools/Scripts/PackageDependencyGraph.py >=20 > diff --git a/BaseTools/Scripts/PackageDependencyGraph.py > b/BaseTools/Scripts/PackageDependencyGraph.py > new file mode 100644 > index 0000000000..b3c8e41774 > --- /dev/null > +++ b/BaseTools/Scripts/PackageDependencyGraph.py > @@ -0,0 +1,296 @@ > +# @file > +# Recursively scan a directory for EDK II packages and generate GraphViz > dot > +# input that is used to render a graph of package dependencies in SVG > format. > +# > +# Copyright (c) 2019, Intel Corporation. All rights reserved.
> +# SPDX-License-Identifier: BSD-2-Clause-Patent > +# > +## > + > +import os > +import sys > +import argparse > +import subprocess > +import webbrowser > +import networkx as nx > +from edk2toollib.uefi.edk2.parsers.inf_parser import InfParser > + > +# > +# Globals for help information > +# > +__prog__ =3D 'PackageDependencyGraph' > +__copyright__ =3D 'Copyright (c) 2019, Intel Corporation. All rights r= eserved.' > +__description__ =3D '''Recursively scan a directory for EDK II packages = and > +generate GraphViz dot input that is used to render a graph of package > +dependencies in SVG format.''' > + > +if __name__ =3D=3D '__main__': > + > + # > + # Create command line argument parser object > + # > + parser =3D argparse.ArgumentParser (prog =3D __prog__, > + description =3D __description__ + = '\n' + __copyright__, > + conflict_handler =3D 'resolve') > + parser.add_argument ("-w", "--workspace", dest =3D 'Workspace', defa= ult =3D > os.curdir, > + help =3D "Directory to recursively scan for EDK= II packages. > Default is current directory.") > + parser.add_argument ("-p", "--packages-path", dest =3D 'PackagesPath= ', > default =3D None, > + help =3D "List of directories to recursively sc= an for EDK II > packages.") > + parser.add_argument ("-d", "--dot-output", dest =3D 'DotOutputFile', > + help =3D "DOT output filename.") > + parser.add_argument ("-o", "--output", dest =3D 'SvgOutputFile', > + help =3D "SVG output filename.") > + parser.add_argument ("-g", "--ignore-directory", dest =3D 'IgnoreDir= ectory', > action=3D'append', default=3D[], > + help =3D "Name of directory to ignore. Option = can be repeated to > ignore multiple directories.") > + parser.add_argument ("-k", "--skip-package", dest =3D 'SkipPackage', > action=3D'append', default=3D[], > + help =3D "Name of EDK II Package DEC file to sk= ip. Option can be > repeated to skip multiple EDK II packages.") > + parser.add_argument ("-s", "--self-dependency", dest =3D > 'SelfDependency', action =3D "store_true", default =3D False, > + help =3D "Include self links in dependency grap= h. Default is > disabled.") > + parser.add_argument ("-u", "--unresolved", dest =3D 'Unresolved', ac= tion =3D > "store_true", default=3DFalse, > + help =3D "Include unresolved EDK II packages in= dependency > graph. Default is disabled.") > + parser.add_argument ("-l", "--label", dest =3D 'Label', action =3D "= store_true", > default=3DFalse, > + help =3D "Label links with the number of EDK II= package > dependencies. Default is disabled.") > + parser.add_argument ("-f", "--full-paths", dest =3D 'FullPaths', act= ion =3D > "store_true", default=3DFalse, > + help =3D "Label package nodes with full path to= EDK II package. > Default is disabled.") > + parser.add_argument ("-b", "--web-browser", dest =3D 'WebBrowser', > action =3D "store_true", default=3DFalse, > + help =3D "Display SVG output file in default we= b browser. Default > is disabled.") > + parser.add_argument ("-v", "--verbose", dest =3D 'Verbose', action = =3D > "store_true", > + help =3D "Increase output messages") > + parser.add_argument ("-q", "--quiet", dest =3D 'Quiet', action =3D > "store_true", > + help =3D "Reduce output messages") > + parser.add_argument ("--debug", dest =3D 'Debug', type =3D int, meta= var =3D > '[0-9]', choices =3D range (0, 10), default =3D 0, > + help =3D "Set debug level") > + > + # > + # Parse command line arguments > + # > + args =3D parser.parse_args () > + > + # > + # Find all EDK II package DEC files > + # > + Components =3D {} > + SearchPaths =3D [args.Workspace] > + if args.PackagesPath: > + SearchPaths +=3D args.PackagesPath.split(os.pathsep) > + SearchPaths =3D [os.path.realpath(x) for x in SearchPaths] > + for SearchPath in SearchPaths: > + for root, dirs, files in os.walk (SearchPath): > + for name in files: > + FilePath =3D os.path.join (root, name) > + if > set(FilePath.split(os.sep)).intersection(set(args.IgnoreDirectory)) !=3D = set(): > + if args.Verbose: > + print ('IGNORE:' + FilePath) > + continue > + if os.path.splitext(FilePath)[1].lower() in ['.dec']: > + DecFile =3D os.path.realpath (FilePath) > + if os.path.split(DecFile)[1] in args.SkipPackage: > + if args.Verbose: > + print ('SKIP:' + DecFile) > + continue > + if DecFile not in Components: > + if args.Verbose: > + print ('PACKAGE:' + DecFile) > + Components[DecFile] =3D {} > + > + # > + # Find EDK II component INF files in each EDK II package > + # > + PackageLabels =3D {} > + UnresolvedPackages =3D [] > + AmbiguousDependencies =3D [] > + for DecFile in Components: > + DecPath =3D os.path.split (DecFile)[0] > + if DecFile not in PackageLabels: > + PackageLabels[DecFile] =3D (DecFile.replace(os.path.sep,'\\n= '), 'white') > + if not args.FullPaths: > + PackagePath =3D os.path.relpath(DecFile, os.path.split(D= ecPath)[0]) > + PackageLabels[DecFile] =3D (PackagePath.replace(os.path.= sep,'\\n'), > 'white') > + for root, dirs, files in os.walk (DecPath): > + for name in files: > + FilePath =3D os.path.join(root, name) > + if > set(FilePath.split(os.sep)).intersection(set(args.IgnoreDirectory)) !=3D = set(): > + if args.Verbose: > + print ('IGNORE:' + FilePath) > + continue > + if os.path.splitext(FilePath)[1].lower() in ['.inf']: > + InfFile =3D os.path.realpath (FilePath) > + Inf =3D InfParser () > + Inf.ParseFile (InfFile) > + DependentPackages =3D [] > + for Dependency in Inf.PackagesUsed: > + Dependency =3D os.path.normpath(Dependency) > + if os.path.split(Dependency)[1] in args.SkipPack= age: > + if args.Verbose: > + print ('SKIP:' + Dependency) > + continue > + Found =3D False > + for SearchPath in SearchPaths: > + PackagePath =3D os.path.realpath(os.path.joi= n(SearchPath, > Dependency)) > + if os.path.exists(PackagePath): > + DependentPackages.append(PackagePath) > + if not args.FullPaths: > + PackageLabels[PackagePath] =3D > (Dependency.replace(os.path.sep,'\\n'), 'white') > + Found =3D True > + break > + if not Found: > + Count =3D 0 > + Match =3D '' > + for DecFile2 in Components: > + if DecFile2.endswith(Dependency): > + if Count =3D=3D 0: > + Match =3D DecFile2 > + Count =3D Count + 1 > + if Count > 1: > + AmbiguousDependencies.append (Dependency= ) > + if Count =3D=3D 1: > + DependentPackages.append(Match) > + if not args.FullPaths: > + PackageLabels[Match] =3D > (Dependency.replace(os.path.sep,'\\n'), 'white') > + Found =3D True > + if not Found and args.Unresolved: > + if args.Verbose: > + print ('WARNING: Dependent package not f= ound ' + > Dependency) > + DependentPackages.append(Dependency) > + UnresolvedPackages.append(Dependency) > + PackageLabels[Dependency] =3D > (Dependency.replace(os.path.sep,'\\n'), 'white') > + Components[DecFile][InfFile] =3D DependentPackages > + if AmbiguousDependencies: > + for Dependency in set(AmbiguousDependencies): > + print ('ERROR: MULTIPLE: ' + Dependency) > + print ('Use --packages-path to provide search priority.') > + sys.exit(1) > + > + # > + # Generate GraphViz dot input file contents. > + # Use networkx to detect circular dependencies. > + # > + MaxWeight =3D 0 > + MaxWeightDecFile =3D list(Components.keys())[0] > + Graph =3D nx.DiGraph() > + Edges =3D [] > + for DecFile in Components: > + if args.Verbose: > + print ('PACKAGE DEPENDENCIES:' + DecFile) > + AllDependencies =3D [] > + Dependencies =3D set() > + for InfFile in Components[DecFile]: > + AllDependencies +=3D Components[DecFile][InfFile] > + Dependencies =3D Dependencies.union(Components[DecFile][InfF= ile]) > + if not args.SelfDependency: > + Dependencies =3D Dependencies.difference(set([DecFile])) > + for Dependency in Dependencies: > + Weight =3D AllDependencies.count(Dependency) > + if Weight > MaxWeight: > + MaxWeight =3D Weight > + MaxWeightDecFile =3D DecFile > + if args.Verbose: > + print (' DEPENDENCY: Weight(' + str(Weight) + ') ' + De= pendency) > + Edges.append(' "{Package}" -> "{Dependency}" [label =3D > "{Weight}"];'.format( > + Package =3D DecFile, > + Dependency =3D Dependency, > + Weight =3D str(Weight) if args.Label else '' > + )) > + Graph.add_edge(DecFile, Dependency) > + Edges.sort() > + > + # > + # Set fill color to yellow if a packages is part of a circular depen= dency > + # > + for Node in set([x for y in list(nx.simple_cycles(Graph)) for x in y= ]): > + PackageLabels[Node] =3D (PackageLabels[Node][0], 'yellow') > + print ('ERROR: CIRCULAR: ' + Node) > + > + # > + # Set fill color to pink if a package that is nested inside another = package. > + # Set fill color to orange if a package is nested inside another pac= kage and > + # is part of a circular dependency. > + # > + for DecFile in Components: > + DecPath =3D os.path.split (DecFile)[0] > + for DecFile2 in Components: > + if DecFile2 =3D=3D DecFile: > + continue > + if os.path.commonpath ([DecPath, DecFile2]) !=3D DecPath: > + continue > + if len(DecFile2) < len(DecPath): > + DecFile2 =3D DecFile > + print ('ERROR: NESTED: ' + DecFile2) > + if PackageLabels[DecFile2][1] =3D=3D 'yellow': > + PackageLabels[DecFile2] =3D (PackageLabels[DecFile2][0],= 'orange') > + else: > + PackageLabels[DecFile2] =3D (PackageLabels[DecFile2][0],= 'pink') > + > + # > + # Set fill color to red for nodes that are unresolved > + # > + for Node in set(UnresolvedPackages): > + PackageLabels[Node] =3D (PackageLabels[Node][0], 'red') > + print ('ERROR: UNRESOLVED: ' + Node) > + > + # > + # Add node statements to set node label and fill color > + # > + Nodes =3D [] > + for Package in PackageLabels: > + Nodes.append(' "{Package}" [label=3D"{Label}",fillcolor=3D{Colo= r}];'.format( > + Package =3D Package, > + Label =3D PackageLabels[Package][0], > + Color =3D PackageLabels[Package][1] > + )) > + Nodes.sort() > + > + # > + # Generate dot file from Nodes and Edges and add a Legend at top of > graph > + # > + Dot =3D [] > + Dot.append('digraph {') > + Dot.append(' rankdir=3DBT;') > + Dot.append(' node [shape=3DMrecord,style=3Dfilled];') > + Dot.append('') > + Dot =3D Dot + Nodes > + Dot.append('') > + Dot =3D Dot + Edges > + Dot.append('') > + Dot.append(' subgraph legend {') > + Dot.append(' rank=3Dsink;') > + Dot.append(' Unresolved [label=3D"Unresolved Dependency", > fillcolor=3Dred];') > + Dot.append(' Circular [label=3D"Circular Dependency", > fillcolor=3Dyellow];') > + Dot.append(' Nested [label=3D"Nested Package", > fillcolor=3Dpink];') > + Dot.append(' NestedCircular [label=3D"Nested Package with Circula= r > Dependency",fillcolor=3Dorange];') > + Dot.append(' }') > + if MaxWeightDecFile: > + Dot.append(' Unresolved->"' + MaxWeightDecFile + '" [style=3Din= vis];') > + Dot.append('}') > + > + if args.DotOutputFile: > + # > + # Write GraphViz dot file contents to DotOutputFile > + # > + with open(os.path.realpath(args.DotOutputFile), 'w') as File: > + File.write('\n'.join(Dot)) > + if args.SvgOutputFile: > + # > + # Use GraphViz 'dot' command to generate SVG output file > + # > + args.SvgOutputFile =3D os.path.realpath(args.SvgOutputFile) > + try: > + Process =3D subprocess.Popen('dot -Tsvg', > + stdin=3Dsubprocess.PIPE, > + stdout=3Dopen(args.SvgOutputFile, 'w'), > + stderr=3Dsubprocess.PIPE, > + shell=3DTrue > + ) > + Process.stdin.write ('\n'.join(Dot).encode()) > + Process.communicate() > + if Process.returncode !=3D 0: > + print ("ERROR: Can not run GraphViz 'dot' command. Chec= k install > and path.") > + sys.exit(Process.returncode) > + except: > + print ("ERROR: Can not run GraphViz 'dot' command. Check in= stall and > path.") > + sys.exit(1) > + # > + # Display SVG file in default web browser > + # > + if args.WebBrowser: > + webbrowser.open(args.SvgOutputFile) > -- > 2.21.0.windows.1