From mboxrd@z Thu Jan  1 00:00:00 1970
Authentication-Results: mx.groups.io;
 dkim=missing; spf=pass (domain: intel.com, ip: 134.134.136.65, mailfrom: bob.c.feng@intel.com)
Received: from mga03.intel.com (mga03.intel.com [134.134.136.65])
 by groups.io with SMTP; Mon, 24 Jun 2019 06:25:49 -0700
X-Amp-Result: SKIPPED(no attachment in message)
X-Amp-File-Uploaded: False
Received: from fmsmga001.fm.intel.com ([10.253.24.23])
  by orsmga103.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 24 Jun 2019 06:25:48 -0700
X-ExtLoop1: 1
X-IronPort-AV: E=Sophos;i="5.63,412,1557212400"; 
   d="scan'208";a="182634946"
Received: from fmsmsx105.amr.corp.intel.com ([10.18.124.203])
  by fmsmga001.fm.intel.com with ESMTP; 24 Jun 2019 06:25:48 -0700
Received: from fmsmsx117.amr.corp.intel.com (10.18.116.17) by
 FMSMSX105.amr.corp.intel.com (10.18.124.203) with Microsoft SMTP Server (TLS)
 id 14.3.439.0; Mon, 24 Jun 2019 06:25:48 -0700
Received: from shsmsx106.ccr.corp.intel.com (10.239.4.159) by
 fmsmsx117.amr.corp.intel.com (10.18.116.17) with Microsoft SMTP Server (TLS)
 id 14.3.439.0; Mon, 24 Jun 2019 06:25:48 -0700
Received: from shsmsx105.ccr.corp.intel.com ([169.254.11.72]) by
 SHSMSX106.ccr.corp.intel.com ([169.254.10.89]) with mapi id 14.03.0439.000;
 Mon, 24 Jun 2019 21:25:46 +0800
From: "Bob Feng" <bob.c.feng@intel.com>
To: "Fan, ZhijuX" <zhijux.fan@intel.com>, "devel@edk2.groups.io"
	<devel@edk2.groups.io>
CC: "Gao, Liming" <liming.gao@intel.com>
Subject: Re: [PATCH V4] BaseTools:Add DetectNotUsedItem.py to Edk2\BaseTools\Scripts
Thread-Topic: [PATCH V4] BaseTools:Add DetectNotUsedItem.py to
 Edk2\BaseTools\Scripts
Thread-Index: AdUqdz3HJY+EOM9WQs6+ALaQrF7ffgAGRB+Q
Date: Mon, 24 Jun 2019 13:25:45 +0000
Message-ID: <08650203BA1BD64D8AD9B6D5D74A85D160B3208C@SHSMSX105.ccr.corp.intel.com>
References: <FAD0D7E0AE0FA54D987F6E72435CAFD50AF845B6@SHSMSX101.ccr.corp.intel.com>
In-Reply-To: <FAD0D7E0AE0FA54D987F6E72435CAFD50AF845B6@SHSMSX101.ccr.corp.intel.com>
Accept-Language: zh-CN, en-US
X-MS-Has-Attach: 
X-MS-TNEF-Correlator: 
x-titus-metadata-40: eyJDYXRlZ29yeUxhYmVscyI6IiIsIk1ldGFkYXRhIjp7Im5zIjoiaHR0cDpcL1wvd3d3LnRpdHVzLmNvbVwvbnNcL0ludGVsMyIsImlkIjoiOWZlNTEwMTItMTIwMC00NTM0LTgwMGEtMmFhNzIxOGI0ZWMzIiwicHJvcHMiOlt7Im4iOiJDVFBDbGFzc2lmaWNhdGlvbiIsInZhbHMiOlt7InZhbHVlIjoiQ1RQX05UIn1dfV19LCJTdWJqZWN0TGFiZWxzIjpbXSwiVE1DVmVyc2lvbiI6IjE3LjEwLjE4MDQuNDkiLCJUcnVzdGVkTGFiZWxIYXNoIjoiVCt5b1JSSGtFSDBCaG9mVklINFYxaGkwUmJVcWhiaDBERWhSc3pGckRTeWtxdmN5OE1PZlBJNUtTV2hsVktuSSJ9
x-ctpclassification: CTP_NT
dlp-product: dlpe-windows
dlp-version: 11.0.600.7
dlp-reaction: no-action
x-originating-ip: [10.239.127.40]
MIME-Version: 1.0
Return-Path: bob.c.feng@intel.com
Content-Language: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

Reviewed-by: Bob Feng <bob.c.feng@intel.com>

-----Original Message-----
From: Fan, ZhijuX=20
Sent: Monday, June 24, 2019 6:26 PM
To: devel@edk2.groups.io
Cc: Gao, Liming <liming.gao@intel.com>; Feng, Bob C <bob.c.feng@intel.com>
Subject: [PATCH V4] BaseTools:Add DetectNotUsedItem.py to Edk2\BaseTools\Sc=
ripts

BZ:https://bugzilla.tianocore.org/show_bug.cgi?id=3D1850

This script is used to Detect unreferenced PCD and GUID/Protocols/PPIs.
The input parameters are Dec file and package directory.

This script can be run in both Py2 and Py3.

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Zhiju.Fan <zhijux.fan@intel.com>
---
 BaseTools/Scripts/DetectNotUsedItem.py | 198 +++++++++++++++++++++++++++++=
++++
 1 file changed, 198 insertions(+)
 create mode 100644 BaseTools/Scripts/DetectNotUsedItem.py

diff --git a/BaseTools/Scripts/DetectNotUsedItem.py b/BaseTools/Scripts/Det=
ectNotUsedItem.py
new file mode 100644
index 0000000000..7e3568fcf9
--- /dev/null
+++ b/BaseTools/Scripts/DetectNotUsedItem.py
@@ -0,0 +1,198 @@
+## @file
+# Detect unreferenced PCD and GUID/Protocols/PPIs.
+#
+# Copyright (c) 2019, Intel Corporation. All rights reserved.
+#
+# SPDX-License-Identifier: BSD-2-Clause-Patent #
+
+'''
+DetectNotUsedItem
+'''
+import re
+import os
+import sys
+import argparse
+
+#
+# Globals for help information
+#
+__prog__ =3D 'DetectNotUsedItem'
+__version__ =3D '%s Version %s' % (__prog__, '0.1') __copyright__ =3D=20
+'Copyright (c) 2019, Intel Corporation. All rights reserved.'
+__description__ =3D "Detect unreferenced PCD and GUID/Protocols/PPIs.\n"
+
+SectionList =3D ["LibraryClasses", "Guids", "Ppis", "Protocols", "Pcd"]
+
+
+class PROCESS(object):
+
+    def __init__(self, DecPath, InfDirs):
+        self.Dec =3D DecPath
+        self.InfPath =3D InfDirs
+        self.Log =3D []
+
+    def ParserDscFdfInfFile(self):
+        AllContentList =3D []
+        for File in self.SearchbyExt([".dsc", ".fdf", ".inf"]):
+            AllContentList +=3D self.ParseDscFdfInfContent(File)
+        return AllContentList
+
+    # Search File by extension name
+    def SearchbyExt(self, ExtList):
+        FileList =3D []
+        for path in self.InfPath:
+            if type(ExtList) =3D=3D type(''):
+                for root, _, files in os.walk(path, topdown=3DTrue, follow=
links=3DFalse):
+                    for filename in files:
+                        if filename.endswith(ExtList):
+                            FileList.append(os.path.join(root, filename))
+            elif type(ExtList) =3D=3D type([]):
+                for root, _, files in os.walk(path, topdown=3DTrue, follow=
links=3DFalse):
+                    for filename in files:
+                        for Ext in ExtList:
+                            if filename.endswith(Ext):
+                                FileList.append(os.path.join(root, filenam=
e))
+        return FileList
+
+    # Parse DEC file to get Line number and Name
+    # return section name, the Item Name and comments line number
+    def ParseDecContent(self):
+        SectionRE =3D re.compile(r'\[(.*)\]')
+        Flag =3D False
+        Comments =3D {}
+        Comment_Line =3D []
+        ItemName =3D {}
+        with open(self.Dec, 'r') as F:
+            for Index, content in enumerate(F):
+                NotComment =3D not content.strip().startswith("#")
+                Section =3D SectionRE.findall(content)
+                if Section and NotComment:
+                    Flag =3D self.IsNeedParseSection(Section[0])
+                if Flag:
+                    Comment_Line.append(Index)
+                    if NotComment:
+                        if content !=3D "\n" and content !=3D "\r\n":
+                            ItemName[Index] =3D content.split('=3D')[0].sp=
lit('|')[0].split('#')[0].strip()
+                            Comments[Index] =3D Comment_Line
+                            Comment_Line =3D []
+        return ItemName, Comments
+
+    def IsNeedParseSection(self, SectionName):
+        for item in SectionList:
+            if item in SectionName:
+                return True
+        return False
+
+    # Parse DSC, FDF, INF File, remove comments, return Lines list
+    def ParseDscFdfInfContent(self, File):
+        with open(File, 'r') as F:
+            lines =3D F.readlines()
+        for Index in range(len(lines) - 1, -1, -1):
+            if lines[Index].strip().startswith("#") or lines[Index] =3D=3D=
 "\n" or lines[Index] =3D=3D "\r\n":
+                lines.remove(lines[Index])
+            elif "#" in lines[Index]:
+                lines[Index] =3D lines[Index].split("#")[0].strip()
+            else:
+                lines[Index] =3D lines[Index].strip()
+        return lines
+
+    def DetectNotUsedItem(self):
+        NotUsedItem =3D {}
+        DecItem, DecComments =3D self.ParseDecContent()
+        InfDscFdfContent =3D self.ParserDscFdfInfFile()
+        for LineNum in list(DecItem.keys()):
+            DecItemName =3D DecItem[LineNum]
+            Match_reg =3D re.compile("(?<![a-zA-Z0-9_-])%s(?![a-zA-Z0-9_-]=
)" % DecItemName)
+            MatchFlag =3D False
+            for Line in InfDscFdfContent:
+                if Match_reg.search(Line):
+                    MatchFlag =3D True
+                    break
+            if not MatchFlag:
+                NotUsedItem[LineNum] =3D DecItemName
+        self.Display(NotUsedItem)
+        return NotUsedItem, DecComments
+
+    def Display(self, UnuseDict):
+        print("DEC File:\n%s\n%s%s" % (self.Dec, "{:<15}".format("Line Num=
ber"), "{:<0}".format("Unused Item")))
+        self.Log.append(
+            "DEC File:\n%s\n%s%s\n" % (self.Dec, "{:<15}".format("Line Num=
ber"), "{:<0}".format("Unused Item")))
+        for num in list(sorted(UnuseDict.keys())):
+            ItemName =3D UnuseDict[num]
+            print("%s%s%s" % (" " * 3, "{:<12}".format(num + 1), "{:<1}".f=
ormat(ItemName)))
+            self.Log.append(("%s%s%s\n" % (" " * 3, "{:<12}".format(num=20
+ + 1), "{:<1}".format(ItemName))))
+
+    def Clean(self, UnUseDict, Comments):
+        removednum =3D []
+        for num in list(UnUseDict.keys()):
+            if num in list(Comments.keys()):
+                removednum +=3D Comments[num]
+        with open(self.Dec, 'r') as Dec:
+            lines =3D Dec.readlines()
+        try:
+            with open(self.Dec, 'w+') as T:
+                for linenum in range(len(lines)):
+                    if linenum in removednum:
+                        continue
+                    else:
+                        T.write(lines[linenum])
+            print("DEC File has been clean: %s" % (self.Dec))
+        except Exception as err:
+            print(err)
+
+
+class Main(object):
+
+    def mainprocess(self, Dec, Dirs, Isclean, LogPath):
+        for dir in Dirs:
+            if not os.path.exists(dir):
+                print("Error: Invalid path for '--dirs': %s" % dir)
+                sys.exit(1)
+        Pa =3D PROCESS(Dec, Dirs)
+        unuse, comment =3D Pa.DetectNotUsedItem()
+        if Isclean:
+            Pa.Clean(unuse, comment)
+        self.Logging(Pa.Log, LogPath)
+
+    def Logging(self, content, LogPath):
+        if LogPath:
+            try:
+                if os.path.isdir(LogPath):
+                    FilePath =3D os.path.dirname(LogPath)
+                    if not os.path.exists(FilePath):
+                        os.makedirs(FilePath)
+                with open(LogPath, 'w+') as log:
+                    for line in content:
+                        log.write(line)
+                print("Log save to file: %s" % LogPath)
+            except Exception as e:
+                print("Save log Error: %s" % e)
+
+
+def main():
+    parser =3D argparse.ArgumentParser(prog=3D__prog__,
+                                     description=3D__description__ + __cop=
yright__,
+                                     conflict_handler=3D'resolve')
+    parser.add_argument('-i', '--input', metavar=3D"", dest=3D'InputDec', =
help=3D"Input DEC file name.")
+    parser.add_argument('--dirs', metavar=3D"", action=3D'append', dest=3D=
'Dirs',
+                        help=3D"The package directory. To specify more dir=
ectories, please repeat this option.")
+    parser.add_argument('--clean', action=3D'store_true', default=3DFalse,=
 dest=3D'Clean',
+                        help=3D"Clean the unreferenced items from DEC file=
.")
+    parser.add_argument('--log', metavar=3D"", dest=3D"Logfile", default=
=3DFalse,
+                        help=3D"Put log in specified file as well as on co=
nsole.")
+    options =3D parser.parse_args()
+    if options.InputDec:
+        if not (os.path.exists(options.InputDec) and options.InputDec.ends=
with(".dec")):
+            print("Error: Invalid DEC file input: %s" % options.InputDec)
+        if options.Dirs:
+            M =3D Main()
+            M.mainprocess(options.InputDec, options.Dirs, options.Clean, o=
ptions.Logfile)
+        else:
+            print("Error: the following argument is required:'--dirs'.")
+    else:
+        print("Error: the following argument is=20
+required:'-i/--input'.")
+
+
+if __name__ =3D=3D '__main__':
+    main()
--
2.14.1.windows.1