From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by mx.groups.io with SMTP id smtpd.web10.4187.1576485636503146051 for ; Mon, 16 Dec 2019 00:40:36 -0800 Authentication-Results: mx.groups.io; dkim=missing; spf=pass (domain: intel.com, ip: 192.55.52.93, mailfrom: ray.ni@intel.com) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga007.fm.intel.com ([10.253.24.52]) by fmsmga102.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 16 Dec 2019 00:40:36 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.69,321,1571727600"; d="scan'208";a="211987353" Received: from fmsmsx103.amr.corp.intel.com ([10.18.124.201]) by fmsmga007.fm.intel.com with ESMTP; 16 Dec 2019 00:40:36 -0800 Received: from fmsmsx123.amr.corp.intel.com (10.18.125.38) by FMSMSX103.amr.corp.intel.com (10.18.124.201) with Microsoft SMTP Server (TLS) id 14.3.439.0; Mon, 16 Dec 2019 00:40:35 -0800 Received: from shsmsx107.ccr.corp.intel.com (10.239.4.96) by fmsmsx123.amr.corp.intel.com (10.18.125.38) with Microsoft SMTP Server (TLS) id 14.3.439.0; Mon, 16 Dec 2019 00:40:35 -0800 Received: from shsmsx104.ccr.corp.intel.com ([169.254.5.90]) by SHSMSX107.ccr.corp.intel.com ([169.254.9.164]) with mapi id 14.03.0439.000; Mon, 16 Dec 2019 16:40:33 +0800 From: "Ni, Ray" To: "devel@edk2.groups.io" , "Ni, Ray" , "Kinney, Michael D" CC: "Feng, Bob C" , "Gao, Liming" , Sean Brogan , "Bret Barkelew" Subject: Re: [edk2-devel] [Patch 1/1] BaseTools/Scripts: Add package dependency graphing tool Thread-Topic: [edk2-devel] [Patch 1/1] BaseTools/Scripts: Add package dependency graphing tool Thread-Index: AQHVse3J5mlN2J1SSUCegIkulR/HMae8Q7kggAAVGZA= Date: Mon, 16 Dec 2019 08:40:32 +0000 Message-ID: <734D49CCEBEEF84792F5B80ED585239D5C39EC8B@SHSMSX104.ccr.corp.intel.com> References: <20191213194449.7036-1-michael.d.kinney@intel.com> <20191213194449.7036-2-michael.d.kinney@intel.com> <15E0C4623803B81C.31060@groups.io> In-Reply-To: <15E0C4623803B81C.31060@groups.io> Accept-Language: en-US, zh-CN X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.239.127.40] MIME-Version: 1.0 Return-Path: ray.ni@intel.com Content-Language: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Mike, This pkg dep tool can tell through weight when a module depends on a new p= kg. But it cannot tell when a module depends on another PCD/Protocol/Guid/Ppi/= Library which belongs to an already-depended pkg. Failure to tell such information may make a bad module (that violates the = dependency rules) worse (wrongly depend on more interfaces). Do you think that the tool can be enhanced in future to detect the case? Thanks, Ray > -----Original Message----- > From: devel@edk2.groups.io On Behalf Of Ni, Ray > Sent: Monday, December 16, 2019 1:57 PM > To: Kinney, Michael D ; devel@edk2.groups.io > Cc: Feng, Bob C ; Gao, Liming > ; Sean Brogan ; Bret > Barkelew > Subject: Re: [edk2-devel] [Patch 1/1] BaseTools/Scripts: Add package > dependency graphing tool >=20 > Mike, > 2 minor comments regarding the help string in below. >=20 > > -----Original Message----- > > From: Kinney, Michael D > > Sent: Saturday, December 14, 2019 3:45 AM > > To: devel@edk2.groups.io > > Cc: Ni, Ray ; Feng, Bob C ; Ga= o, > > Liming ; Sean Brogan > ; > > Bret Barkelew > > Subject: [Patch 1/1] BaseTools/Scripts: Add package dependency graphin= g > > tool > > > > https://bugzilla.tianocore.org/show_bug.cgi?id=3D2161 > > > > Add python script that recursively scans a directory for EDK II > > packages and generates GraphViz dot input that is used to render > > a graph of package dependencies in SVG format. > > > > Detects following error/warning conditions: > > * Ambiguous dependencies (multiple matches) > > * Unresolved dependencies > > * Circular dependencies > > * Nested packages > > > > usage: PackageDependencyGraph [-h] [-w WORKSPACE] [-p > PACKAGESPATH] > > [-d DOTOUTPUTFILE] [-o SVGOUTPUTFILE] > > [-g IGNOREDIRECTORY] [-k SKIPPACKAGE] > > [-s] [-u] [-l] [-f] [-b] [-v] [-q] > > [--debug [0-9]] > > > > Recursively scan a directory for EDK II packages and generate GraphViz= dot > > input that is used to render a graph of package dependencies in SVG > format. > > Copyright (c) 2019, Intel Corporation. All rights reserved. > > > > optional arguments: > > -h, --help show this help message and exit > > -w WORKSPACE, --workspace WORKSPACE > > Directory to recursively scan for EDK II packa= ges. > > Default is current directory. > > -p PACKAGESPATH, --packages-path PACKAGESPATH > > List of directories to recursively scan for ED= K II > > packages. > > -d DOTOUTPUTFILE, --dot-output DOTOUTPUTFILE > > DOT output filename. > > -o OUTPUTFILE, --output OUTPUTFILE > > SVG output filename. > > -g IGNOREDIRECTORY, --ignore-directory IGNOREDIRECTORY > > Name of directory to ignore. Option can be rep= eated > > to ignore multiple directories. >=20 > 1. Name of directory to ignore when scanning for EDKII Module INFs. >=20 > > -k SKIPPACKAGE, --skip-package SKIPPACKAGE > > Name of EDK II Package DEC file to skip. Optio= n can > > be repeated to skip multiple EDK II packages. >=20 > 2. Name of EDK II Package DEC file (including .DEC suffix) to skip. Opti= on can > Be repeated to skip multiple EDK II packages. >=20 > > -s, --self-dependency > > Include self links in dependency graph. Defaul= t is > > disabled. > > -u, --unresolved Include unresolved EDK II packages in dependen= cy > > graph. Default is disabled. > > -l, --label Label links with the number of EDK II package > > dependencies. Default is disabled. > > -f, --full-paths Label package nodes with full path to EDK II > > package. Default is disabled. > > -b, --web-browser Display SVG output file in default web browser= . > > Default is disabled. > > -v, --verbose Increase output messages > > -q, --quiet Reduce output messages > > --debug [0-9] Set debug level > > > > Cc: Ray Ni > > Cc: Bob Feng > > Cc: Liming Gao > > Cc: Sean Brogan > > Cc: Bret Barkelew > > Signed-off-by: Michael D Kinney > > --- > > BaseTools/Scripts/PackageDependencyGraph.py | 296 > > ++++++++++++++++++++ > > 1 file changed, 296 insertions(+) > > create mode 100644 BaseTools/Scripts/PackageDependencyGraph.py > > > > diff --git a/BaseTools/Scripts/PackageDependencyGraph.py > > b/BaseTools/Scripts/PackageDependencyGraph.py > > new file mode 100644 > > index 0000000000..b3c8e41774 > > --- /dev/null > > +++ b/BaseTools/Scripts/PackageDependencyGraph.py > > @@ -0,0 +1,296 @@ > > +# @file > > +# Recursively scan a directory for EDK II packages and generate Graph= Viz > > dot > > +# input that is used to render a graph of package dependencies in SVG > > format. > > +# > > +# Copyright (c) 2019, Intel Corporation. All rights reserved.
> > +# SPDX-License-Identifier: BSD-2-Clause-Patent > > +# > > +## > > + > > +import os > > +import sys > > +import argparse > > +import subprocess > > +import webbrowser > > +import networkx as nx > > +from edk2toollib.uefi.edk2.parsers.inf_parser import InfParser > > + > > +# > > +# Globals for help information > > +# > > +__prog__ =3D 'PackageDependencyGraph' > > +__copyright__ =3D 'Copyright (c) 2019, Intel Corporation. All right= s > reserved.' > > +__description__ =3D '''Recursively scan a directory for EDK II packag= es and > > +generate GraphViz dot input that is used to render a graph of package > > +dependencies in SVG format.''' > > + > > +if __name__ =3D=3D '__main__': > > + > > + # > > + # Create command line argument parser object > > + # > > + parser =3D argparse.ArgumentParser (prog =3D __prog__, > > + description =3D __description__= + '\n' + __copyright__, > > + conflict_handler =3D 'resolve') > > + parser.add_argument ("-w", "--workspace", dest =3D 'Workspace', d= efault > =3D > > os.curdir, > > + help =3D "Directory to recursively scan for = EDK II packages. > > Default is current directory.") > > + parser.add_argument ("-p", "--packages-path", dest =3D 'PackagesP= ath', > > default =3D None, > > + help =3D "List of directories to recursively= scan for EDK II > > packages.") > > + parser.add_argument ("-d", "--dot-output", dest =3D 'DotOutputFil= e', > > + help =3D "DOT output filename.") > > + parser.add_argument ("-o", "--output", dest =3D 'SvgOutputFile', > > + help =3D "SVG output filename.") > > + parser.add_argument ("-g", "--ignore-directory", dest =3D > 'IgnoreDirectory', > > action=3D'append', default=3D[], > > + help =3D "Name of directory to ignore. Opti= on can be repeated > to > > ignore multiple directories.") > > + parser.add_argument ("-k", "--skip-package", dest =3D 'SkipPackag= e', > > action=3D'append', default=3D[], > > + help =3D "Name of EDK II Package DEC file to= skip. Option can > be > > repeated to skip multiple EDK II packages.") > > + parser.add_argument ("-s", "--self-dependency", dest =3D > > 'SelfDependency', action =3D "store_true", default =3D False, > > + help =3D "Include self links in dependency g= raph. Default is > > disabled.") > > + parser.add_argument ("-u", "--unresolved", dest =3D 'Unresolved',= action > =3D > > "store_true", default=3DFalse, > > + help =3D "Include unresolved EDK II packages= in dependency > > graph. Default is disabled.") > > + parser.add_argument ("-l", "--label", dest =3D 'Label', action = =3D > "store_true", > > default=3DFalse, > > + help =3D "Label links with the number of EDK= II package > > dependencies. Default is disabled.") > > + parser.add_argument ("-f", "--full-paths", dest =3D 'FullPaths', = action =3D > > "store_true", default=3DFalse, > > + help =3D "Label package nodes with full path= to EDK II package. > > Default is disabled.") > > + parser.add_argument ("-b", "--web-browser", dest =3D 'WebBrowser'= , > > action =3D "store_true", default=3DFalse, > > + help =3D "Display SVG output file in default= web browser. > Default > > is disabled.") > > + parser.add_argument ("-v", "--verbose", dest =3D 'Verbose', actio= n =3D > > "store_true", > > + help =3D "Increase output messages") > > + parser.add_argument ("-q", "--quiet", dest =3D 'Quiet', action = =3D > > "store_true", > > + help =3D "Reduce output messages") > > + parser.add_argument ("--debug", dest =3D 'Debug', type =3D int, m= etavar =3D > > '[0-9]', choices =3D range (0, 10), default =3D 0, > > + help =3D "Set debug level") > > + > > + # > > + # Parse command line arguments > > + # > > + args =3D parser.parse_args () > > + > > + # > > + # Find all EDK II package DEC files > > + # > > + Components =3D {} > > + SearchPaths =3D [args.Workspace] > > + if args.PackagesPath: > > + SearchPaths +=3D args.PackagesPath.split(os.pathsep) > > + SearchPaths =3D [os.path.realpath(x) for x in SearchPaths] > > + for SearchPath in SearchPaths: > > + for root, dirs, files in os.walk (SearchPath): > > + for name in files: > > + FilePath =3D os.path.join (root, name) > > + if > > set(FilePath.split(os.sep)).intersection(set(args.IgnoreDirectory)) != =3D set(): > > + if args.Verbose: > > + print ('IGNORE:' + FilePath) > > + continue > > + if os.path.splitext(FilePath)[1].lower() in ['.dec']: > > + DecFile =3D os.path.realpath (FilePath) > > + if os.path.split(DecFile)[1] in args.SkipPackage: > > + if args.Verbose: > > + print ('SKIP:' + DecFile) > > + continue > > + if DecFile not in Components: > > + if args.Verbose: > > + print ('PACKAGE:' + DecFile) > > + Components[DecFile] =3D {} > > + > > + # > > + # Find EDK II component INF files in each EDK II package > > + # > > + PackageLabels =3D {} > > + UnresolvedPackages =3D [] > > + AmbiguousDependencies =3D [] > > + for DecFile in Components: > > + DecPath =3D os.path.split (DecFile)[0] > > + if DecFile not in PackageLabels: > > + PackageLabels[DecFile] =3D (DecFile.replace(os.path.sep,'= \\n'), 'white') > > + if not args.FullPaths: > > + PackagePath =3D os.path.relpath(DecFile, os.path.spli= t(DecPath)[0]) > > + PackageLabels[DecFile] =3D (PackagePath.replace(os.pa= th.sep,'\\n'), > > 'white') > > + for root, dirs, files in os.walk (DecPath): > > + for name in files: > > + FilePath =3D os.path.join(root, name) > > + if > > set(FilePath.split(os.sep)).intersection(set(args.IgnoreDirectory)) != =3D set(): > > + if args.Verbose: > > + print ('IGNORE:' + FilePath) > > + continue > > + if os.path.splitext(FilePath)[1].lower() in ['.inf']: > > + InfFile =3D os.path.realpath (FilePath) > > + Inf =3D InfParser () > > + Inf.ParseFile (InfFile) > > + DependentPackages =3D [] > > + for Dependency in Inf.PackagesUsed: > > + Dependency =3D os.path.normpath(Dependency) > > + if os.path.split(Dependency)[1] in args.SkipP= ackage: > > + if args.Verbose: > > + print ('SKIP:' + Dependency) > > + continue > > + Found =3D False > > + for SearchPath in SearchPaths: > > + PackagePath =3D os.path.realpath(os.path.= join(SearchPath, > > Dependency)) > > + if os.path.exists(PackagePath): > > + DependentPackages.append(PackagePath) > > + if not args.FullPaths: > > + PackageLabels[PackagePath] =3D > > (Dependency.replace(os.path.sep,'\\n'), 'white') > > + Found =3D True > > + break > > + if not Found: > > + Count =3D 0 > > + Match =3D '' > > + for DecFile2 in Components: > > + if DecFile2.endswith(Dependency): > > + if Count =3D=3D 0: > > + Match =3D DecFile2 > > + Count =3D Count + 1 > > + if Count > 1: > > + AmbiguousDependencies.append (Depende= ncy) > > + if Count =3D=3D 1: > > + DependentPackages.append(Match) > > + if not args.FullPaths: > > + PackageLabels[Match] =3D > > (Dependency.replace(os.path.sep,'\\n'), 'white') > > + Found =3D True > > + if not Found and args.Unresolved: > > + if args.Verbose: > > + print ('WARNING: Dependent package no= t found ' + > > Dependency) > > + DependentPackages.append(Dependency) > > + UnresolvedPackages.append(Dependency) > > + PackageLabels[Dependency] =3D > > (Dependency.replace(os.path.sep,'\\n'), 'white') > > + Components[DecFile][InfFile] =3D DependentPackage= s > > + if AmbiguousDependencies: > > + for Dependency in set(AmbiguousDependencies): > > + print ('ERROR: MULTIPLE: ' + Dependency) > > + print ('Use --packages-path to provide search priority.') > > + sys.exit(1) > > + > > + # > > + # Generate GraphViz dot input file contents. > > + # Use networkx to detect circular dependencies. > > + # > > + MaxWeight =3D 0 > > + MaxWeightDecFile =3D list(Components.keys())[0] > > + Graph =3D nx.DiGraph() > > + Edges =3D [] > > + for DecFile in Components: > > + if args.Verbose: > > + print ('PACKAGE DEPENDENCIES:' + DecFile) > > + AllDependencies =3D [] > > + Dependencies =3D set() > > + for InfFile in Components[DecFile]: > > + AllDependencies +=3D Components[DecFile][InfFile] > > + Dependencies =3D Dependencies.union(Components[DecFile][I= nfFile]) > > + if not args.SelfDependency: > > + Dependencies =3D Dependencies.difference(set([DecFile])) > > + for Dependency in Dependencies: > > + Weight =3D AllDependencies.count(Dependency) > > + if Weight > MaxWeight: > > + MaxWeight =3D Weight > > + MaxWeightDecFile =3D DecFile > > + if args.Verbose: > > + print (' DEPENDENCY: Weight(' + str(Weight) + ') ' += Dependency) > > + Edges.append(' "{Package}" -> "{Dependency}" [label =3D > > "{Weight}"];'.format( > > + Package =3D DecFile, > > + Dependency =3D Dependency, > > + Weight =3D str(Weight) if args.Label else '' > > + )) > > + Graph.add_edge(DecFile, Dependency) > > + Edges.sort() > > + > > + # > > + # Set fill color to yellow if a packages is part of a circular de= pendency > > + # > > + for Node in set([x for y in list(nx.simple_cycles(Graph)) for x i= n y]): > > + PackageLabels[Node] =3D (PackageLabels[Node][0], 'yellow') > > + print ('ERROR: CIRCULAR: ' + Node) > > + > > + # > > + # Set fill color to pink if a package that is nested inside anoth= er package. > > + # Set fill color to orange if a package is nested inside another = package > and > > + # is part of a circular dependency. > > + # > > + for DecFile in Components: > > + DecPath =3D os.path.split (DecFile)[0] > > + for DecFile2 in Components: > > + if DecFile2 =3D=3D DecFile: > > + continue > > + if os.path.commonpath ([DecPath, DecFile2]) !=3D DecPath: > > + continue > > + if len(DecFile2) < len(DecPath): > > + DecFile2 =3D DecFile > > + print ('ERROR: NESTED: ' + DecFile2) > > + if PackageLabels[DecFile2][1] =3D=3D 'yellow': > > + PackageLabels[DecFile2] =3D (PackageLabels[DecFile2][= 0], 'orange') > > + else: > > + PackageLabels[DecFile2] =3D (PackageLabels[DecFile2][= 0], 'pink') > > + > > + # > > + # Set fill color to red for nodes that are unresolved > > + # > > + for Node in set(UnresolvedPackages): > > + PackageLabels[Node] =3D (PackageLabels[Node][0], 'red') > > + print ('ERROR: UNRESOLVED: ' + Node) > > + > > + # > > + # Add node statements to set node label and fill color > > + # > > + Nodes =3D [] > > + for Package in PackageLabels: > > + Nodes.append(' "{Package}" > [label=3D"{Label}",fillcolor=3D{Color}];'.format( > > + Package =3D Package, > > + Label =3D PackageLabels[Package][0], > > + Color =3D PackageLabels[Package][1] > > + )) > > + Nodes.sort() > > + > > + # > > + # Generate dot file from Nodes and Edges and add a Legend at top = of > > graph > > + # > > + Dot =3D [] > > + Dot.append('digraph {') > > + Dot.append(' rankdir=3DBT;') > > + Dot.append(' node [shape=3DMrecord,style=3Dfilled];') > > + Dot.append('') > > + Dot =3D Dot + Nodes > > + Dot.append('') > > + Dot =3D Dot + Edges > > + Dot.append('') > > + Dot.append(' subgraph legend {') > > + Dot.append(' rank=3Dsink;') > > + Dot.append(' Unresolved [label=3D"Unresolved Dependency", > > fillcolor=3Dred];') > > + Dot.append(' Circular [label=3D"Circular Dependency", > > fillcolor=3Dyellow];') > > + Dot.append(' Nested [label=3D"Nested Package", > > fillcolor=3Dpink];') > > + Dot.append(' NestedCircular [label=3D"Nested Package with Circ= ular > > Dependency",fillcolor=3Dorange];') > > + Dot.append(' }') > > + if MaxWeightDecFile: > > + Dot.append(' Unresolved->"' + MaxWeightDecFile + '" [style= =3Dinvis];') > > + Dot.append('}') > > + > > + if args.DotOutputFile: > > + # > > + # Write GraphViz dot file contents to DotOutputFile > > + # > > + with open(os.path.realpath(args.DotOutputFile), 'w') as File: > > + File.write('\n'.join(Dot)) > > + if args.SvgOutputFile: > > + # > > + # Use GraphViz 'dot' command to generate SVG output file > > + # > > + args.SvgOutputFile =3D os.path.realpath(args.SvgOutputFile) > > + try: > > + Process =3D subprocess.Popen('dot -Tsvg', > > + stdin=3Dsubprocess.PIPE, > > + stdout=3Dopen(args.SvgOutputFile, 'w'), > > + stderr=3Dsubprocess.PIPE, > > + shell=3DTrue > > + ) > > + Process.stdin.write ('\n'.join(Dot).encode()) > > + Process.communicate() > > + if Process.returncode !=3D 0: > > + print ("ERROR: Can not run GraphViz 'dot' command. C= heck install > > and path.") > > + sys.exit(Process.returncode) > > + except: > > + print ("ERROR: Can not run GraphViz 'dot' command. Check= install > and > > path.") > > + sys.exit(1) > > + # > > + # Display SVG file in default web browser > > + # > > + if args.WebBrowser: > > + webbrowser.open(args.SvgOutputFile) > > -- > > 2.21.0.windows.1 >=20 >=20 >=20