[PATCH v3 00/17] Add a pipline to check Ecc issues for edk2 on open ci

[PATCH v3 00/17] Add a pipline to check Ecc issues for edk2 on open ci

[Zhang, Shenglei]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=y, Size: 4027 bytes --]

REF: https://bugzilla.tianocore.org/show_bug.cgi?id=2606
As planed we will enable Ecc check for edk2 on open ci. And they are
ready now, but these are V3 series. So I expect that contributors in
edk2 community can try using this script when reviewing. And I appreciate
receiving feedback and comments if someone find errors or false positive
issues.

I created a pipline of EccCheck for my forked edk2.
https://dev.azure.com/shengleizhang/shengleizhang/_build?definitionId=10&_a=summary

The patch series are big, so the commits are also pushed into my forked tree.
https://github.com/shenglei10/edk2/commits/ECC

Patches
1/17: EccCheck.py is a tool to report Ecc issues for commits. It can be run
     on azure servers for open ci, or locally. Its usage is like
     PatchCheck.py.
2/17: It's a lib necessary for py3 to run Ecc on azure servers. For local
     use, we need to type command
     "py -3 -m pip install antlr4-python3-runtime==4.7.1" first.
3/17: Windows-EccCheck.yml is a yaml file to configure the newly added
     pipline. Azure uses this to create a pipline.
4/17~17/17: We consider some cases that will report out Ecc issues but they won't
     be fixed, like submodule and industry standard related things. So we
     add two configuration fields "Exception" and "IgnoreFiles" for people
     to use. These patches add configuration in yaml files for Ecc check.

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Bret Barkelew <Bret.Barkelew@microsoft.com>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Sean Brogan <sean.brogan@microsoft.com>

v2: Update 1/17, fix the bug that the script can't hanlde multiple commits.

v3: Update 1/17, set the only workalbe workspace is edk2 root directory.
    Update 2/17, designate the version of antlr4 is 4.7.1.
    Add 4/17~17/17.

Shenglei Zhang (17):
  BaseTools/Scripts: Add EccCheck.py
  pip-requirements.txt: Add Ecc required lib
  .azurepiplines: Add a pipline to check ECC issues for commits
  MdeModulePkg/MdeModulePkg.ci.yaml: Add configuration for Ecc check
  ArmVirtPkg/ArmVirtPkg.ci.yaml: Add configuration for Ecc check
  CryptoPkg/CryptoPkg.ci.yaml： Add configuration for Ecc check
  EmulatorPkg/EmulatorPkg.ci.yaml: Add configuration for Ecc check
  FatPkg/FatPkg.ci.yaml: Add configuration for Ecc check
  FmpDevicePkg/FmpDevicePkg.ci.yaml: Add configuration for Ecc check
  MdePkg/MdePkg.ci.yaml: Add configuration for Ecc check
  NetworkPkg/NetworkPkg.ci.yaml: Add configuration for Ecc check
  OvmfPkg/OvmfPkg.ci.yaml: Add configuration for Ecc check
  PcAtChipsetPkg/PcAtChipsetPkg.ci.yaml: Add configuration for Ecc check
  SecurityPkg/SecurityPkg.ci.yaml: Add configuration for Ecc check
  ShellPkg/ShellPkg.ci.yaml: Add configuration for Ecc check
  UefiCpuPkg/UefiCpuPkg.ci.yaml: Add configuration for Ecc check
  UnitTestFrameworkPkg: Add configuration for Ecc check in yaml file

 .azurepipelines/Windows-EccCheck.yml          |  38 ++
 ArmVirtPkg/ArmVirtPkg.ci.yaml                 |  10 +
 BaseTools/Scripts/EccCheck.py                 | 437 ++++++++++++++++++
 CryptoPkg/CryptoPkg.ci.yaml                   |  10 +
 EmulatorPkg/EmulatorPkg.ci.yaml               |  10 +
 FatPkg/FatPkg.ci.yaml                         |  10 +
 FmpDevicePkg/FmpDevicePkg.ci.yaml             |  10 +
 MdeModulePkg/MdeModulePkg.ci.yaml             |  10 +
 MdePkg/MdePkg.ci.yaml                         |  10 +
 NetworkPkg/NetworkPkg.ci.yaml                 |  10 +
 OvmfPkg/OvmfPkg.ci.yaml                       |  10 +
 PcAtChipsetPkg/PcAtChipsetPkg.ci.yaml         |  10 +
 SecurityPkg/SecurityPkg.ci.yaml               |  10 +
 ShellPkg/ShellPkg.ci.yaml                     |  10 +
 UefiCpuPkg/UefiCpuPkg.ci.yaml                 |  10 +
 .../UnitTestFrameworkPkg.ci.yaml              |  10 +
 pip-requirements.txt                          |   1 +
 17 files changed, 616 insertions(+)
 create mode 100644 .azurepipelines/Windows-EccCheck.yml
 create mode 100644 BaseTools/Scripts/EccCheck.py

-- 
2.18.0.windows.1

[PATCH v3 01/17] BaseTools/Scripts: Add EccCheck.py

[Zhang, Shenglei]

REF: https://bugzilla.tianocore.org/show_bug.cgi?id=2606
EccCheck.py is a tool to report Ecc issues for commits, which will
be run on open ci.
But note not each kind of issue could be reported out.
It can only handle the issues, whose line number in CSV report
accurately map with their code in source code files. And comment
issues can also be handled.

Its usage is similar to PatchCheck.py. Type EccCheck.py -h and then
learn how to use it.

If a patch passes EccCheck, "Ecc Pass" will show up.
Otherwise, "Ecc error detected" alerts the users and the details
are also presented.

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Shenglei Zhang <shenglei.zhang@intel.com>
---
 BaseTools/Scripts/EccCheck.py | 437 ++++++++++++++++++++++++++++++++++
 1 file changed, 437 insertions(+)
 create mode 100644 BaseTools/Scripts/EccCheck.py

diff --git a/BaseTools/Scripts/EccCheck.py b/BaseTools/Scripts/EccCheck.py
new file mode 100644
index 000000000000..2848a1e43864
--- /dev/null
+++ b/BaseTools/Scripts/EccCheck.py
@@ -0,0 +1,437 @@
+## @file
+#  Check a patch for various format issues
+#
+#  Copyright (c) 2020, Intel Corporation. All rights reserved.<BR>
+#
+#  SPDX-License-Identifier: BSD-2-Clause-Patent
+#
+
+import os
+import re
+import csv
+import subprocess
+import argparse
+import sys
+import yaml
+import xml.dom.minidom
+
+__copyright__ = "Copyright (c) 2020, Intel Corporation  All rights reserved."
+ReModifyFile = re.compile(r'[B-Q,S-Z]+[\d]*\t(.*?)\n')
+FindModifyFile = re.compile(r'\+\+\+ b\/(.*)')
+LineScopePattern = (r'@@ -\d*\,*\d* \+\d*\,*\d* @@.*')
+LineNumRange = re.compile(r'@@ -\d*\,*\d* \+(\d*)\,*(\d*) @@.*')
+
+EnvList = os.environ
+GlobalSymbol = {}
+
+def AppendException(ExceptionList, ExceptionFile):
+    error_code_list = ExceptionList[::2]
+    keyword_list = ExceptionList[1::2]
+    domTree = xml.dom.minidom.parse(ExceptionFile)
+    rootNode = domTree.documentElement
+    for error_code, keyword in zip(error_code_list, keyword_list):
+        customer_node = domTree.createElement("Exception")
+        keyword_node = domTree.createElement("KeyWord")
+        keyword_node_text_value = domTree.createTextNode(keyword)
+        keyword_node.appendChild(keyword_node_text_value)
+        customer_node.appendChild(keyword_node)
+        error_code_node = domTree.createElement("ErrorID")
+        error_code_text_value = domTree.createTextNode(error_code)
+        error_code_node.appendChild(error_code_text_value)
+        customer_node.appendChild(error_code_node)
+        rootNode.appendChild(customer_node)
+
+    with open(ExceptionFile, 'w') as f:
+        domTree.writexml(f,indent='',addindent='',newl='\n',encoding='UTF-8')
+
+def GetPkgList(EnvList):
+    WORKDIR = EnvList['WORKDIR']
+    dirs = os.listdir(WORKDIR)
+    PkgList = []
+    for directory in dirs:
+        if directory.endswith('Pkg'):
+            PkgList.append(directory)
+    return PkgList
+
+def GenerateEccReport(EnvList, ModifyDirList, EccDiffRange):
+    IsECCNeed = False
+    IsTrue = True
+    PkgList = GetPkgList(EnvList)
+
+    for Line in ModifyDirList:
+        print('echo Run ECC tool for the commit in %s' % Line)
+        GlobalSymbol['GenerateEccReport'] = True
+        for Pkg in PkgList:
+            if Pkg in Line:
+                IsECCNeed = True
+                Ecc_cmd = ["ecc", "-c", "%WORKDIR%\BaseTools\Source\Python\Ecc\config.ini",
+                                  "-e", "%WORKDIR%\BaseTools\Source\Python\Ecc\exception.xml",
+                                  "-t", "%WORKDIR%\{}".format(Line),
+                                  "-r", "%WORKDIR%\Ecc.csv"]
+                _, _, result, return_code = ExecuteScript(Ecc_cmd, EnvList, shell=True)
+                if return_code != 0:
+                    IsTrue = False
+                    break
+
+        if not IsTrue:
+            print('Fail to run ECC tool')
+            GlobalSymbol['SCRIPT_ERROR'] = True
+            EndDelFile(EnvList)
+
+        if GlobalSymbol.get('GenerateEccReport'):
+            ParseEccReport(EnvList, EccDiffRange)
+        else:
+            print("Patch check tool or ECC tool don't detect error")
+
+    if IsECCNeed:
+        revert_cmd = ["git", "checkout", "--", "%WORKDIR%\BaseTools\Source\Python\Ecc\exception.xml"]
+        _, _, result, return_code = ExecuteScript(revert_cmd, EnvList, shell=True)
+    else:
+        print("Doesn't need run ECC check")
+        return
+
+def ParseEccReport(EnvList, EccDiffRange):
+    WorkDir = EnvList['WORKDIR']
+    EccLog = os.path.join(WorkDir, "Ecc.log")
+    EccsCsv = "Ecc.csv"
+    FileList = os.listdir(WorkDir)
+    rowLines = []
+    if EccsCsv in FileList:
+        with open(EccsCsv) as Csv:
+            reader = csv.reader(Csv)
+            for row in reader:
+                for ModifyFile in EccDiffRange:
+                    if ModifyFile in row[3]:
+                        for i in EccDiffRange[ModifyFile]:
+                            LineNo = int(row[4])
+                            if i[0] <= LineNo <= i[1]:
+                                row[0] = '\nEFI coding style error'
+                                row[1] = 'Error code: ' + row[1]
+                                row[3] = 'File: ' + row[3]
+                                row[4] = 'Line number: ' + row[4]
+                                rowLine = '\n  *'.join(row)
+                                rowLines.append(rowLine)
+                                break
+                        break
+    if rowLines:
+        GlobalSymbol['ECC_PASS'] = False
+
+    with open(EccLog, 'a') as EccFile:
+            AllLine = '\n'.join(rowLines)
+            AllLine = AllLine + '\n'
+            EccFile.writelines(AllLine)
+
+def RemoveFile(File):
+    if os.path.exists(File):
+        os.remove(File)
+
+def ExecuteScript(command, env_variables, collect_env=False,
+                  enable_std_pipe=False, shell=True):
+    """launches a process that executes a script/shell command passed to it
+
+        :param command: The command/script with its commandline
+            arguments to be executed
+        :type command:  List:String
+        :param env_variables: Environment variables passed to the process
+        :type env_variables: String
+        :param collect_env: Enables the collection of evironment variables
+            when process execution is done
+        :type collect_env: Boolean
+        :param enable_std_pipe: Enables process out to be piped to
+        :type enable_std_pipe: String
+        :returns: a tuple of std_out, stderr , environment variables,
+            return code
+        :rtype: Tuple: (std_out, stderr , enVar, return_code)
+    """
+
+    env_marker = '-----env-----'
+    env = {}
+    kwarg = {"env": env_variables,
+             "universal_newlines": True,
+             "shell": shell,
+             "cwd": env_variables["WORKSPACE"]}
+
+    if enable_std_pipe or collect_env:
+        kwarg["stdout"] = subprocess.PIPE
+        kwarg["stderr"] = subprocess.PIPE
+
+    if collect_env:
+        # get the binary that prints environment variables based on os
+        if os.name == 'nt':
+            get_var_command = "set"
+        else:
+            get_var_command = "env"
+        # modify the command to print the environment variables
+        if isinstance(command, list):
+            command += ["&&", "echo", env_marker, "&&",
+                        get_var_command, "&&", "echo", env_marker]
+        else:
+            command += " " + " ".join(["&&", "echo", env_marker,
+                                       "&&", get_var_command,
+                                       "&&", "echo", env_marker])
+
+    # execute the command
+    execute = subprocess.Popen(command, **kwarg)
+    std_out, stderr = execute.communicate()
+    code = execute.returncode
+
+    # wait for process to be done
+    execute.wait()
+
+    # if collect enviroment variables
+    if collect_env:
+        # get the new environment variables
+        std_out, env = GetEnvironmentVariables(std_out, env_marker)
+    return (std_out, stderr, env, code)
+
+def GetEnvironmentVariables(std_out_str, marker):
+    """Gets the environment variables from a process
+
+        :param std_out_str: The std_out pipe
+        :type std_out_str: String
+        :param marker: A begining and end mark of environment
+            variables printed to std_out
+        :type marker: String
+        :returns: The environment variables read from the process' std_out pipe
+        :rtype: Tuple
+    """
+    start_env_update = False
+    environment_vars = {}
+    out_put = ""
+    for line in std_out_str.split("\n"):
+        if start_env_update and len(line.split("=")) == 2:
+            key, value = line.split("=")
+            environment_vars[key] = value
+        else:
+            out_put += "\n" + line.replace(marker, "")
+
+        if marker in line:
+            if start_env_update:
+                start_env_update = False
+            else:
+                start_env_update = True
+    return (out_put, environment_vars)
+
+def EndDelFile(EnvList):
+    WORKDIR = EnvList['WORKDIR']
+    ModifyFileListLog = os.path.join(WORKDIR, 'PatchModifyFiles.log')
+    RemoveFile(ModifyFileListLog)
+    PatchLog = os.path.join(WORKDIR, 'PatchFile.log')
+    RemoveFile(PatchLog)
+    FileLog = os.path.join(WORKDIR, "File.log")
+    RemoveFile(FileLog)
+    if GlobalSymbol.get('GenerateEccReport'):
+        FileList = os.listdir(WORKDIR)
+        CsvList = [os.path.join(WORKDIR, File) for File in FileList if File.endswith('.csv')]
+        for Csv in CsvList:
+            RemoveFile(Csv)
+    if GlobalSymbol.get('SCRIPT_ERROR'):
+        print('ECC tool detect error')
+        exit(1)
+
+def EdksetupRebuild(EnvList, WORKDIR):
+    EnvList['WORKDIR'] = WORKDIR
+    EnvList['WORKSPACE'] = WORKDIR
+    edk2_setup_cmd = ["edksetup", "Rebuild"]
+    _, _, result, return_code = ExecuteScript(edk2_setup_cmd, EnvList, collect_env=True, shell=True)
+    if return_code == 0:
+        for Key in result:
+            EnvList[Key] = result[Key]
+        return True
+    return False
+
+def GetDiffrange(EnvList, commit):
+    WORKDIR = EnvList['WORKDIR']
+    ModifyFileListLog = EnvList['ModifyFileListLog']
+    RangeDirectory = {}
+    PatchLog = os.path.join(WORKDIR, 'PatchFile.log')
+    Format_Patch_cmd = ["git", "show", str(commit), "--unified=0", ">", PatchLog]
+    _, _, result, return_code = ExecuteScript(Format_Patch_cmd, EnvList, shell=True)
+    if return_code != 0:
+        print('Fail to run GIT')
+        GlobalSymbol['SCRIPT_ERROR'] = True
+        EndDelFile(EnvList)
+    with open(PatchLog, encoding='utf8') as PatchLogFile:
+        Filelines = PatchLogFile.readlines()
+        IsDelete = True
+        StartCheck = False
+        for line in Filelines:
+            ModifyFile = FindModifyFile.findall(line)
+            if ModifyFile and not StartCheck and os.path.isfile(ModifyFile[0]):
+                ModifyFileCommentDic = GetCommentRange(EnvList, ModifyFile[0])
+                IsDelete = False
+                StartCheck = True
+                ModifyFilieDic = ModifyFile[0]
+                ModifyFilieDic = ModifyFilieDic.replace("/","\\")
+                RangeDirectory[ModifyFilieDic] = []
+            elif line.startswith('--- '):
+                StartCheck = False
+            elif re.match(LineScopePattern, line, re.I) and not IsDelete and StartCheck:
+                startline = LineNumRange.search(line).group(1)
+                linerange = LineNumRange.search(line).group(2)
+                if not linerange:
+                    linerange = '1'
+                RangeDirectory[ModifyFilieDic].append((int(startline), int(startline) + int(linerange) - 1))
+                for i in ModifyFileCommentDic:
+                    if i[0] <= int(startline) <= i[1]:
+                        RangeDirectory[ModifyFilieDic].append(i)
+    return RangeDirectory
+
+def GetCommentRange(EnvList, ModifyFile):
+    WORKDIR = EnvList['WORKDIR']
+    ModifyFilePath = os.path.join(WORKDIR, ModifyFile)
+    with open(ModifyFilePath) as f:
+        LineNo = 1
+        CommentRange = []
+        Start = False
+        for line in f:
+            if line.startswith('/**'):
+                startno = LineNo
+                Start = True
+            if line.startswith('**/') and Start:
+                endno = LineNo
+                Start = False
+                CommentRange.append((int(startno), int(endno)))
+            LineNo += 1
+
+    if CommentRange and CommentRange[0][0] == 1:
+        del CommentRange[0]
+    return CommentRange
+
+def GetModifyDir(EnvList, commit):
+    WORKDIR = EnvList['WORKDIR']
+    ModifyDirList = []
+    ModifyFileListLog = os.path.join(WORKDIR, 'PatchModifyFiles.log')
+    EnvList['ModifyFileListLog'] = ModifyFileListLog
+    Patch_Modify_cmd = ["git", "diff", "--name-status", str(commit), str(commit)+"~1", ">", ModifyFileListLog]
+    _, _, result, return_code = ExecuteScript(Patch_Modify_cmd, EnvList, shell=True)
+    if return_code != 0:
+        print('Fail to run GIT')
+        GlobalSymbol['SCRIPT_ERROR'] = True
+        EndDelFile(EnvList)
+    with open(ModifyFileListLog) as ModifyFile:
+        Filelines = ModifyFile.readlines()
+    for Line in Filelines:
+        FilePath = ReModifyFile.findall(Line)
+        if FilePath:
+            FileDir = os.path.dirname(FilePath[0])
+        else:
+            continue
+        PkgList = GetPkgList(EnvList)
+        if FileDir in PkgList or not FileDir:
+            continue
+        else:
+            ModifyDirList.append('%s'%FileDir)
+
+    ModifyDirList = list(set(ModifyDirList))
+    return ModifyDirList
+
+def ApplyConfig(EnvList, ModifyDirList, EccDiffRange):
+    WORKDIR = EnvList['WORKDIR']
+    ModifyPkgList = []
+    for ModifyDir in ModifyDirList:
+        ModifyPkg = ModifyDir.split("/")[0]
+        ModifyPkgList.append(ModifyPkg)
+    ModifyPkgList = list(set(ModifyPkgList))
+    for ModifyPkg in ModifyPkgList:
+        pkg_config_file = os.path.join(WORKDIR, ModifyPkg, ModifyPkg + ".ci.yaml")
+        if os.path.exists(pkg_config_file):
+            with open(pkg_config_file, 'r') as f:
+                pkg_config = yaml.safe_load(f)
+            if "EccCheck" in pkg_config:
+                EccConfig = pkg_config["EccCheck"]
+                #
+                # Add exceptions
+                #
+                ExceptionList = EccConfig["ExceptionList"]
+                ExceptionFile = os.path.join(WORKDIR, "BaseTools", "Source", "Python", "Ecc", "exception.xml")
+                if os.path.exists(ExceptionFile):
+                    AppendException(ExceptionList, ExceptionFile)
+                #
+                # Exclude ignored files
+                #
+                IgnoreFilesList = EccConfig['IgnoreFiles']
+                for ignore_file in IgnoreFilesList:
+                    ignore_file = ignore_file.replace("/", "\\")
+                    ignore_file = os.path.join(ModifyPkg, ignore_file)
+                    if ignore_file in EccDiffRange:
+                        del EccDiffRange[ignore_file]
+    return
+
+def CheckOneCommit(commit):
+    WORKDIR = os.getcwd()
+    if not WORKDIR.endswith('edk2') and not WORKDIR.endswith('Edk2') and not WORKDIR.endswith('\s'):
+        print(WORKDIR)
+        print("Error: invalid workspace.\nBefore using EccCheck.py, please change workspace to edk2 root directory!")
+        exit(1)
+    EdksetupRebuild(EnvList, WORKDIR)
+    ModifyDirList = GetModifyDir(EnvList, commit)
+    EccDiffRange = GetDiffrange(EnvList, commit)
+    ApplyConfig(EnvList, ModifyDirList, EccDiffRange)
+    GenerateEccReport(EnvList, ModifyDirList, EccDiffRange)
+    EndDelFile(EnvList)
+
+def parse_options():
+    parser = argparse.ArgumentParser(description=__copyright__)
+    parser.add_argument('commits', nargs='*',
+                        help='[commit(s) ID | number of commits, like "-3" means check first 3 commits]')
+    args = parser.parse_args()
+    return args
+
+def process_one_arg(self, arg):
+        if len(arg) >= 2 and arg[0] == '-':
+            count = int(arg[1:])
+        CheckOneArg(arg, self.count).ok
+
+
+def read_commit_list_from_git(start_commit, count):
+    cmd = ['git', 'rev-list', '--abbrev-commit', '--no-walk' ]
+    if count is not None:
+            cmd.append('--max-count=' + str(count))
+    cmd.append(start_commit)
+    p = subprocess.Popen(cmd,
+                stdout=subprocess.PIPE,
+                stderr=subprocess.STDOUT)
+    result = p.communicate()
+    out = result[0].decode('utf-8', 'ignore') if result[0] and result[0].find(b"fatal")!=0 else None
+    return out.split() if out else []
+
+def ReleaseReport(EnvList):
+        WORKDIR = EnvList['WORKDIR']
+        EccLog = os.path.join(WORKDIR, "ECC.log")
+        if GlobalSymbol['ECC_PASS']:
+            print('\n===================Ecc pass===================')
+            RemoveFile(EccLog)
+            return 0
+        else:
+            print('\n===================Ecc error detected===================')
+            with open(EccLog) as output:
+                print(output.read())
+            RemoveFile(EccLog)
+            return -1
+
+def main():
+    commits = parse_options().commits
+
+    if len(commits) == 0:
+        commits = [ 'HEAD' ]
+
+    if len(commits[0]) >= 2 and commits[0][0] == '-':
+        count = int(commits[0][1:])
+        commits = read_commit_list_from_git('HEAD', count)
+
+    """
+        This 'if' block is only used for creating pull request.
+    """
+    if ".." in commits[0]:
+        commits = read_commit_list_from_git(commits[0], None)
+
+    GlobalSymbol['ECC_PASS'] = True
+    for commit in commits:
+        CheckOneCommit(commit)
+
+    retval = ReleaseReport(EnvList)
+    return retval
+
+if __name__ == "__main__":
+    sys.exit(main())
-- 
2.18.0.windows.1

[PATCH v3 02/17] pip-requirements.txt: Add Ecc required lib

[Zhang, Shenglei]

antlr4-python3-runtime is a lib to support Ecc run with Py3.x.

Cc: Sean Brogan <sean.brogan@microsoft.com>
Cc: Bret Barkelew <Bret.Barkelew@microsoft.com>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Shenglei Zhang <shenglei.zhang@intel.com>
---
 pip-requirements.txt | 1 +
 1 file changed, 1 insertion(+)

diff --git a/pip-requirements.txt b/pip-requirements.txt
index 574dac43b1a6..0fecd37f2a83 100644
--- a/pip-requirements.txt
+++ b/pip-requirements.txt
@@ -14,3 +14,4 @@
 
 edk2-pytool-library==0.10.*
 edk2-pytool-extensions~=0.13.3
+antlr4-python3-runtime==4.7.1
-- 
2.18.0.windows.1

[PATCH v3 03/17] .azurepiplines: Add a pipline to check ECC issues for commits

[Zhang, Shenglei]

REF: https://bugzilla.tianocore.org/show_bug.cgi?id=2606
Add a pipeline to run the ECC checks on Azure Pipelines agents
for edk2 open ci.

Cc: Sean Brogan <sean.brogan@microsoft.com>
Cc: Bret Barkelew <Bret.Barkelew@microsoft.com>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Shenglei Zhang <shenglei.zhang@intel.com>
---
 .azurepipelines/Windows-EccCheck.yml | 38 ++++++++++++++++++++++++++++
 1 file changed, 38 insertions(+)
 create mode 100644 .azurepipelines/Windows-EccCheck.yml

diff --git a/.azurepipelines/Windows-EccCheck.yml b/.azurepipelines/Windows-EccCheck.yml
new file mode 100644
index 000000000000..0d20551efb5f
--- /dev/null
+++ b/.azurepipelines/Windows-EccCheck.yml
@@ -0,0 +1,38 @@
+## @file
+# Azure Pipielines YML file that evalues the patch series in a PR using the
+# python script BaseTools/Scripts/PatchCheck.py.
+#
+# NOTE: This example monitors pull requests against the edk2-ci branch.  Most
+# environments would replace 'edk2-ci' with 'master'.
+#
+# Copyright (c) 2019, Intel Corporation. All rights reserved.<BR>
+# SPDX-License-Identifier: BSD-2-Clause-Patent
+#
+# https://github.com/tianocore
+#
+##
+
+trigger: none
+
+pr:
+- master
+
+pool:
+  vmImage: 'windows-latest'
+
+steps:
+- checkout: self
+  clean: true
+
+- task: UsePythonVersion@0
+  inputs:
+    versionSpec: '3.8.x'
+    architecture: 'x64'
+
+- script: pip install -r pip-requirements.txt
+  displayName: 'Install/Upgrade pip modules'
+
+- script: |
+    git fetch origin $(System.PullRequest.TargetBranch):$(System.PullRequest.TargetBranch)
+    py -3 BaseTools/Scripts/EccCheck.py $(System.PullRequest.TargetBranch)..$(System.PullRequest.SourceCommitId)
+  displayName: 'Use EccCheck.py to verify patch series in pull request'
-- 
2.18.0.windows.1

[PATCH v3 04/17] MdeModulePkg/MdeModulePkg.ci.yaml: Add configuration for Ecc check

[Zhang, Shenglei]

Add configuration ExceptionList and IgnoreFiles for package config
files. So users can rely on this to ignore some Ecc issues.

Cc: Jian J Wang <jian.j.wang@intel.com>
Cc: Hao A Wu <hao.a.wu@intel.com>
Signed-off-by: Shenglei Zhang <shenglei.zhang@intel.com>
---
 MdeModulePkg/MdeModulePkg.ci.yaml | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/MdeModulePkg/MdeModulePkg.ci.yaml b/MdeModulePkg/MdeModulePkg.ci.yaml
index 1cfc1328390e..cfdac0a73f0c 100644
--- a/MdeModulePkg/MdeModulePkg.ci.yaml
+++ b/MdeModulePkg/MdeModulePkg.ci.yaml
@@ -5,6 +5,16 @@
 # SPDX-License-Identifier: BSD-2-Clause-Patent
 ##
 {
+    "EccCheck": {
+        ## Exception sample looks like below:
+        ## "ExceptionList": [
+        ##     "<ErrorID>", "<KeyWord>"
+        ## ]
+        "ExceptionList": [
+        ],
+        "IgnoreFiles": [
+        ]
+    },
     ## options defined ci/Plugin/CompilerPlugin
     "CompilerPlugin": {
         "DscPath": "MdeModulePkg.dsc"
-- 
2.18.0.windows.1

[PATCH v3 05/17] ArmVirtPkg/ArmVirtPkg.ci.yaml: Add configuration for Ecc check

[Zhang, Shenglei]

Add configuration ExceptionList and IgnoreFiles for package config
files. So users can rely on this to ignore some Ecc issues.

Cc: Laszlo Ersek <lersek@redhat.com>
Cc: Ard Biesheuvel <ard.biesheuvel@arm.com>
Cc: Leif Lindholm <leif@nuviainc.com>
Signed-off-by: Shenglei Zhang <shenglei.zhang@intel.com>
---
 ArmVirtPkg/ArmVirtPkg.ci.yaml | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/ArmVirtPkg/ArmVirtPkg.ci.yaml b/ArmVirtPkg/ArmVirtPkg.ci.yaml
index 4553725ee528..b52e621b3c5c 100644
--- a/ArmVirtPkg/ArmVirtPkg.ci.yaml
+++ b/ArmVirtPkg/ArmVirtPkg.ci.yaml
@@ -8,6 +8,16 @@
 # SPDX-License-Identifier: BSD-2-Clause-Patent
 ##
 {
+    "EccCheck": {
+        ## Exception sample looks like below:
+        ## "ExceptionList": [
+        ##     "<ErrorID>", "<KeyWord>"
+        ## ]
+        "ExceptionList": [
+        ],
+        "IgnoreFiles": [
+        ]
+    },
     ## options defined .pytool/Plugin/CompilerPlugin
     "CompilerPlugin": {
         "DscPath": "" # Don't support this test
-- 
2.18.0.windows.1

[PATCH v3 06/17] CryptoPkg/CryptoPkg.ci.yaml： Add configuration for Ecc check

[Zhang, Shenglei]

Add configuration ExceptionList and IgnoreFiles for package config
files. So users can rely on this to ignore some Ecc issues.

Cc: Jian J Wang <jian.j.wang@intel.com>
Cc: Xiaoyu Lu <xiaoyux.lu@intel.com>
Signed-off-by: Shenglei Zhang <shenglei.zhang@intel.com>
---
 CryptoPkg/CryptoPkg.ci.yaml | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/CryptoPkg/CryptoPkg.ci.yaml b/CryptoPkg/CryptoPkg.ci.yaml
index f54ebfb22e70..2c1a8a7f5072 100644
--- a/CryptoPkg/CryptoPkg.ci.yaml
+++ b/CryptoPkg/CryptoPkg.ci.yaml
@@ -5,6 +5,16 @@
 # SPDX-License-Identifier: BSD-2-Clause-Patent
 ##
 {
+    "EccCheck": {
+        ## Exception sample looks like below:
+        ## "ExceptionList": [
+        ##     "<ErrorID>", "<KeyWord>"
+        ## ]
+        "ExceptionList": [
+        ],
+        "IgnoreFiles": [
+        ]
+    },
     "CompilerPlugin": {
         "DscPath": "CryptoPkg.dsc"
     },
-- 
2.18.0.windows.1

[PATCH v3 07/17] EmulatorPkg/EmulatorPkg.ci.yaml: Add configuration for Ecc check

[Zhang, Shenglei]

Add configuration ExceptionList and IgnoreFiles for package config
files. So users can rely on this to ignore some Ecc issues.

Cc: Jordan Justen <jordan.l.justen@intel.com>
Cc: Andrew Fish <afish@apple.com>
Cc: Ray Ni <ray.ni@intel.com>
Signed-off-by: Shenglei Zhang <shenglei.zhang@intel.com>
---
 EmulatorPkg/EmulatorPkg.ci.yaml | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/EmulatorPkg/EmulatorPkg.ci.yaml b/EmulatorPkg/EmulatorPkg.ci.yaml
index 81f81780ec76..a5c932366c8a 100644
--- a/EmulatorPkg/EmulatorPkg.ci.yaml
+++ b/EmulatorPkg/EmulatorPkg.ci.yaml
@@ -8,6 +8,16 @@
 # SPDX-License-Identifier: BSD-2-Clause-Patent
 ##
 {
+    "EccCheck": {
+        ## Exception sample looks like below:
+        ## "ExceptionList": [
+        ##     "<ErrorID>", "<KeyWord>"
+        ## ]
+        "ExceptionList": [
+        ],
+        "IgnoreFiles": [
+        ]
+    },
     ## options defined .pytool/Plugin/CompilerPlugin
     "CompilerPlugin": {
         "DscPath": "" # Don't support this test
-- 
2.18.0.windows.1

[PATCH v3 08/17] FatPkg/FatPkg.ci.yaml: Add configuration for Ecc check

[Zhang, Shenglei]

Add configuration ExceptionList and IgnoreFiles for package config
files. So users can rely on this to ignore some Ecc issues.

Cc: Ray Ni <ray.ni@intel.com>
Signed-off-by: Shenglei Zhang <shenglei.zhang@intel.com>
---
 FatPkg/FatPkg.ci.yaml | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/FatPkg/FatPkg.ci.yaml b/FatPkg/FatPkg.ci.yaml
index 8b0fb1d4fcd5..82069d3a7534 100644
--- a/FatPkg/FatPkg.ci.yaml
+++ b/FatPkg/FatPkg.ci.yaml
@@ -5,6 +5,16 @@
 # SPDX-License-Identifier: BSD-2-Clause-Patent
 ##
 {
+    "EccCheck": {
+        ## Exception sample looks like below:
+        ## "ExceptionList": [
+        ##     "<ErrorID>", "<KeyWord>"
+        ## ]
+        "ExceptionList": [
+        ],
+        "IgnoreFiles": [
+        ]
+    },
     "CompilerPlugin": {
         "DscPath": "FatPkg.dsc"
     },
-- 
2.18.0.windows.1

[PATCH v3 09/17] FmpDevicePkg/FmpDevicePkg.ci.yaml: Add configuration for Ecc check

[Zhang, Shenglei]

Add configuration ExceptionList and IgnoreFiles for package config
files. So users can rely on this to ignore some Ecc issues.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Signed-off-by: Shenglei Zhang <shenglei.zhang@intel.com>
---
 FmpDevicePkg/FmpDevicePkg.ci.yaml | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/FmpDevicePkg/FmpDevicePkg.ci.yaml b/FmpDevicePkg/FmpDevicePkg.ci.yaml
index 74a0aefe8e49..99ffa4aae8f2 100644
--- a/FmpDevicePkg/FmpDevicePkg.ci.yaml
+++ b/FmpDevicePkg/FmpDevicePkg.ci.yaml
@@ -5,6 +5,16 @@
 # SPDX-License-Identifier: BSD-2-Clause-Patent
 ##
 {
+    "EccCheck": {
+        ## Exception sample looks like below:
+        ## "ExceptionList": [
+        ##     "<ErrorID>", "<KeyWord>"
+        ## ]
+        "ExceptionList": [
+        ],
+        "IgnoreFiles": [
+        ]
+    },
     "CompilerPlugin": {
         "DscPath": "FmpDevicePkg.dsc"
     },
-- 
2.18.0.windows.1

[PATCH v3 10/17] MdePkg/MdePkg.ci.yaml: Add configuration for Ecc check

[Zhang, Shenglei]

Add configuration ExceptionList and IgnoreFiles for package config
files. So users can rely on this to ignore some Ecc issues.

Cc: Michael D Kinney <michael.d.kinney@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Shenglei Zhang <shenglei.zhang@intel.com>
---
 MdePkg/MdePkg.ci.yaml | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/MdePkg/MdePkg.ci.yaml b/MdePkg/MdePkg.ci.yaml
index 3268f1535499..e779dac912b4 100644
--- a/MdePkg/MdePkg.ci.yaml
+++ b/MdePkg/MdePkg.ci.yaml
@@ -5,6 +5,16 @@
 # SPDX-License-Identifier: BSD-2-Clause-Patent
 ##
 {
+    "EccCheck": {
+        ## Exception sample looks like below:
+        ## "ExceptionList": [
+        ##     "<ErrorID>", "<KeyWord>"
+        ## ]
+        "ExceptionList": [
+        ],
+        "IgnoreFiles": [
+        ]
+    },
     ## options defined ci/Plugin/CompilerPlugin
     "CompilerPlugin": {
         "DscPath": "MdePkg.dsc"
-- 
2.18.0.windows.1

[PATCH v3 11/17] NetworkPkg/NetworkPkg.ci.yaml: Add configuration for Ecc check

[Zhang, Shenglei]

Add configuration ExceptionList and IgnoreFiles for package config
files. So users can rely on this to ignore some Ecc issues.

Cc: Maciej Rabeda <maciej.rabeda@linux.intel.com>
Cc: Jiaxin Wu <jiaxin.wu@intel.com>
Cc: Siyuan Fu <siyuan.fu@intel.com>
Signed-off-by: Shenglei Zhang <shenglei.zhang@intel.com>
---
 NetworkPkg/NetworkPkg.ci.yaml | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/NetworkPkg/NetworkPkg.ci.yaml b/NetworkPkg/NetworkPkg.ci.yaml
index 70f2e1014748..7cd61912c71e 100644
--- a/NetworkPkg/NetworkPkg.ci.yaml
+++ b/NetworkPkg/NetworkPkg.ci.yaml
@@ -5,6 +5,16 @@
 # SPDX-License-Identifier: BSD-2-Clause-Patent
 ##
 {
+    "EccCheck": {
+        ## Exception sample looks like below:
+        ## "ExceptionList": [
+        ##     "<ErrorID>", "<KeyWord>"
+        ## ]
+        "ExceptionList": [
+        ],
+        "IgnoreFiles": [
+        ]
+    },
     "CompilerPlugin": {
         "DscPath": "NetworkPkg.dsc"
     },
-- 
2.18.0.windows.1

[PATCH v3 12/17] OvmfPkg/OvmfPkg.ci.yaml: Add configuration for Ecc check

[Zhang, Shenglei]

Add configuration ExceptionList and IgnoreFiles for package config
files. So users can rely on this to ignore some Ecc issues.

Cc: Jordan Justen <jordan.l.justen@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Cc: Ard Biesheuvel <ard.biesheuvel@arm.com>
Signed-off-by: Shenglei Zhang <shenglei.zhang@intel.com>
---
 OvmfPkg/OvmfPkg.ci.yaml | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/OvmfPkg/OvmfPkg.ci.yaml b/OvmfPkg/OvmfPkg.ci.yaml
index 98992f0429ff..67561c0bace0 100644
--- a/OvmfPkg/OvmfPkg.ci.yaml
+++ b/OvmfPkg/OvmfPkg.ci.yaml
@@ -8,6 +8,16 @@
 # SPDX-License-Identifier: BSD-2-Clause-Patent
 ##
 {
+    "EccCheck": {
+        ## Exception sample looks like below:
+        ## "ExceptionList": [
+        ##     "<ErrorID>", "<KeyWord>"
+        ## ]
+        "ExceptionList": [
+        ],
+        "IgnoreFiles": [
+        ]
+    },
     ## options defined .pytool/Plugin/CompilerPlugin
     "CompilerPlugin": {
         "DscPath": "" # Don't support this test
-- 
2.18.0.windows.1

[PATCH v3 13/17] PcAtChipsetPkg/PcAtChipsetPkg.ci.yaml: Add configuration for Ecc check

[Zhang, Shenglei]

Add configuration ExceptionList and IgnoreFiles for package config
files. So users can rely on this to ignore some Ecc issues.

Cc: Ray Ni <ray.ni@intel.com>
Signed-off-by: Shenglei Zhang <shenglei.zhang@intel.com>
---
 PcAtChipsetPkg/PcAtChipsetPkg.ci.yaml | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/PcAtChipsetPkg/PcAtChipsetPkg.ci.yaml b/PcAtChipsetPkg/PcAtChipsetPkg.ci.yaml
index be470807bd9e..5809ff8e3b1c 100644
--- a/PcAtChipsetPkg/PcAtChipsetPkg.ci.yaml
+++ b/PcAtChipsetPkg/PcAtChipsetPkg.ci.yaml
@@ -5,6 +5,16 @@
 # SPDX-License-Identifier: BSD-2-Clause-Patent
 ##
 {
+    "EccCheck": {
+        ## Exception sample looks like below:
+        ## "ExceptionList": [
+        ##     "<ErrorID>", "<KeyWord>"
+        ## ]
+        "ExceptionList": [
+        ],
+        "IgnoreFiles": [
+        ]
+    },
     "CompilerPlugin": {
         "DscPath": "PcAtChipsetPkg.dsc"
     },
-- 
2.18.0.windows.1

[PATCH v3 14/17] SecurityPkg/SecurityPkg.ci.yaml: Add configuration for Ecc check

[Zhang, Shenglei]

Add configuration ExceptionList and IgnoreFiles for package config
files. So users can rely on this to ignore some Ecc issues.

Cc: Jiewen Yao <jiewen.yao@intel.com>
Cc: Jian J Wang <jian.j.wang@intel.com>
Cc: Chao Zhang <chao.b.zhang@intel.com>
Signed-off-by: Shenglei Zhang <shenglei.zhang@intel.com>
---
 SecurityPkg/SecurityPkg.ci.yaml | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/SecurityPkg/SecurityPkg.ci.yaml b/SecurityPkg/SecurityPkg.ci.yaml
index 953219053318..dc70ce642171 100644
--- a/SecurityPkg/SecurityPkg.ci.yaml
+++ b/SecurityPkg/SecurityPkg.ci.yaml
@@ -5,6 +5,16 @@
 # SPDX-License-Identifier: BSD-2-Clause-Patent
 ##
 {
+    "EccCheck": {
+        ## Exception sample looks like below:
+        ## "ExceptionList": [
+        ##     "<ErrorID>", "<KeyWord>"
+        ## ]
+        "ExceptionList": [
+        ],
+        "IgnoreFiles": [
+        ]
+    },
     "CompilerPlugin": {
         "DscPath": "SecurityPkg.dsc"
     },
-- 
2.18.0.windows.1

[PATCH v3 15/17] ShellPkg/ShellPkg.ci.yaml: Add configuration for Ecc check

[Zhang, Shenglei]

Add configuration ExceptionList and IgnoreFiles for package config
files. So users can rely on this to ignore some Ecc issues.

Cc: Ray Ni <ray.ni@intel.com>
Cc: Zhichao Gao <zhichao.gao@intel.com>
Signed-off-by: Shenglei Zhang <shenglei.zhang@intel.com>
---
 ShellPkg/ShellPkg.ci.yaml | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/ShellPkg/ShellPkg.ci.yaml b/ShellPkg/ShellPkg.ci.yaml
index 67de34a2556e..528bd2cc41cc 100644
--- a/ShellPkg/ShellPkg.ci.yaml
+++ b/ShellPkg/ShellPkg.ci.yaml
@@ -5,6 +5,16 @@
 # SPDX-License-Identifier: BSD-2-Clause-Patent
 ##
 {
+    "EccCheck": {
+        ## Exception sample looks like below:
+        ## "ExceptionList": [
+        ##     "<ErrorID>", "<KeyWord>"
+        ## ]
+        "ExceptionList": [
+        ],
+        "IgnoreFiles": [
+        ]
+    },
     "CompilerPlugin": {
         "DscPath": "ShellPkg.dsc"
     },
-- 
2.18.0.windows.1

[PATCH v3 16/17] UefiCpuPkg/UefiCpuPkg.ci.yaml: Add configuration for Ecc check

[Zhang, Shenglei]

Add configuration ExceptionList and IgnoreFiles for package config
files. So users can rely on this to ignore some Ecc issues.

Cc: Eric Dong <eric.dong@intel.com>
Cc: Ray Ni <ray.ni@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Signed-off-by: Shenglei Zhang <shenglei.zhang@intel.com>
---
 UefiCpuPkg/UefiCpuPkg.ci.yaml | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/UefiCpuPkg/UefiCpuPkg.ci.yaml b/UefiCpuPkg/UefiCpuPkg.ci.yaml
index 99e460a8b090..dc3ff362f7c2 100644
--- a/UefiCpuPkg/UefiCpuPkg.ci.yaml
+++ b/UefiCpuPkg/UefiCpuPkg.ci.yaml
@@ -5,6 +5,16 @@
 # SPDX-License-Identifier: BSD-2-Clause-Patent
 ##
 {
+    "EccCheck": {
+        ## Exception sample looks like below:
+        ## "ExceptionList": [
+        ##     "<ErrorID>", "<KeyWord>"
+        ## ]
+        "ExceptionList": [
+        ],
+        "IgnoreFiles": [
+        ]
+    },
     "CompilerPlugin": {
         "DscPath": "UefiCpuPkg.dsc"
     },
-- 
2.18.0.windows.1

[PATCH v3 17/17] UnitTestFrameworkPkg: Add configuration for Ecc check in yaml file

[Zhang, Shenglei]

Add configuration ExceptionList and IgnoreFiles for package config
files. So users can rely on this to ignore some Ecc issues.

Cc: Michael D Kinney <michael.d.kinney@intel.com>
Cc: Sean Brogan <sean.brogan@microsoft.com>
Cc: Bret Barkelew <Bret.Barkelew@microsoft.com>
Signed-off-by: Shenglei Zhang <shenglei.zhang@intel.com>
---
 UnitTestFrameworkPkg/UnitTestFrameworkPkg.ci.yaml | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/UnitTestFrameworkPkg/UnitTestFrameworkPkg.ci.yaml b/UnitTestFrameworkPkg/UnitTestFrameworkPkg.ci.yaml
index 51e172537f8a..7e9fc5d005fb 100644
--- a/UnitTestFrameworkPkg/UnitTestFrameworkPkg.ci.yaml
+++ b/UnitTestFrameworkPkg/UnitTestFrameworkPkg.ci.yaml
@@ -5,6 +5,16 @@
 # SPDX-License-Identifier: BSD-2-Clause-Patent
 ##
 {
+    "EccCheck": {
+        ## Exception sample looks like below:
+        ## "ExceptionList": [
+        ##     "<ErrorID>", "<KeyWord>"
+        ## ]
+        "ExceptionList": [
+        ],
+        "IgnoreFiles": [
+        ]
+    },
     ## options defined .pytool/Plugin/CompilerPlugin
     "CompilerPlugin": {
         "DscPath": "UnitTestFrameworkPkg.dsc"
-- 
2.18.0.windows.1

[PATCH v3 00/17] Add a pipline to check Ecc issues for edk2 on open ci

[Zhang, Shenglei]

REF: https://bugzilla.tianocore.org/show_bug.cgi?id=2606
As planed we will enable Ecc check for edk2 on open ci. And they are
ready now, but these are V3 series. So I expect that contributors in
edk2 community can try using this script when reviewing. And I appreciate
receiving feedback and comments if someone find errors or false positive
issues.

I created a pipline of EccCheck for my forked edk2.
https://dev.azure.com/shengleizhang/shengleizhang/_build?definitionId=10&_a=summary

The patch series are big, so the commits are also pushed into my forked tree.
https://github.com/shenglei10/edk2/commits/ECC

Patches
1/17: EccCheck.py is a tool to report Ecc issues for commits. It can be run
     on azure servers for open ci, or locally. Its usage is like
     PatchCheck.py.
2/17: It's a lib necessary for py3 to run Ecc on azure servers. For local
     use, we need to type command
     "py -3 -m pip install antlr4-python3-runtime==4.7.1" first.
3/17: Windows-EccCheck.yml is a yaml file to configure the newly added
     pipline. Azure uses this to create a pipline.
4/17~17/17: We consider some cases that will report out Ecc issues but they won't
     be fixed, like submodule and industry standard related things. So we
     add two configuration fields "Exception" and "IgnoreFiles" for people
     to use. These patches add configuration in yaml files for Ecc check.

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Bret Barkelew <Bret.Barkelew@microsoft.com>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Sean Brogan <sean.brogan@microsoft.com>

v2: Update 1/17, fix the bug that the script can't hanlde multiple commits.

v3: Update 1/17, set the only workalbe workspace is edk2 root directory.
    Update 2/17, designate the version of antlr4 is 4.7.1.
    Add 4/17~17/17.

Shenglei Zhang (17):
  BaseTools/Scripts: Add EccCheck.py
  pip-requirements.txt: Add Ecc required lib
  .azurepiplines: Add a pipline to check ECC issues for commits
  MdeModulePkg/MdeModulePkg.ci.yaml: Add configuration for Ecc check
  ArmVirtPkg/ArmVirtPkg.ci.yaml: Add configuration for Ecc check
  CryptoPkg/CryptoPkg.ci.yaml: Add configuration for Ecc check
  EmulatorPkg/EmulatorPkg.ci.yaml: Add configuration for Ecc check
  FatPkg/FatPkg.ci.yaml: Add configuration for Ecc check
  FmpDevicePkg/FmpDevicePkg.ci.yaml: Add configuration for Ecc check
  MdePkg/MdePkg.ci.yaml: Add configuration for Ecc check
  NetworkPkg/NetworkPkg.ci.yaml: Add configuration for Ecc check
  OvmfPkg/OvmfPkg.ci.yaml: Add configuration for Ecc check
  PcAtChipsetPkg/PcAtChipsetPkg.ci.yaml: Add configuration for Ecc check
  SecurityPkg/SecurityPkg.ci.yaml: Add configuration for Ecc check
  ShellPkg/ShellPkg.ci.yaml: Add configuration for Ecc check
  UefiCpuPkg/UefiCpuPkg.ci.yaml: Add configuration for Ecc check
  UnitTestFrameworkPkg: Add configuration for Ecc check in yaml file

 .azurepipelines/Windows-EccCheck.yml          |  38 ++
 ArmVirtPkg/ArmVirtPkg.ci.yaml                 |  10 +
 BaseTools/Scripts/EccCheck.py                 | 437 ++++++++++++++++++
 CryptoPkg/CryptoPkg.ci.yaml                   |  10 +
 EmulatorPkg/EmulatorPkg.ci.yaml               |  10 +
 FatPkg/FatPkg.ci.yaml                         |  10 +
 FmpDevicePkg/FmpDevicePkg.ci.yaml             |  10 +
 MdeModulePkg/MdeModulePkg.ci.yaml             |  10 +
 MdePkg/MdePkg.ci.yaml                         |  10 +
 NetworkPkg/NetworkPkg.ci.yaml                 |  10 +
 OvmfPkg/OvmfPkg.ci.yaml                       |  10 +
 PcAtChipsetPkg/PcAtChipsetPkg.ci.yaml         |  10 +
 SecurityPkg/SecurityPkg.ci.yaml               |  10 +
 ShellPkg/ShellPkg.ci.yaml                     |  10 +
 UefiCpuPkg/UefiCpuPkg.ci.yaml                 |  10 +
 .../UnitTestFrameworkPkg.ci.yaml              |  10 +
 pip-requirements.txt                          |   1 +
 17 files changed, 616 insertions(+)
 create mode 100644 .azurepipelines/Windows-EccCheck.yml
 create mode 100644 BaseTools/Scripts/EccCheck.py

-- 
2.18.0.windows.1

Re: [PATCH v3 11/17] NetworkPkg/NetworkPkg.ci.yaml: Add configuration for Ecc check

[Maciej Rabeda]

Reviewed-by: Maciej Rabeda <maciej.rabeda@linux.intel.com>

On 10-Jun-20 10:14, Shenglei Zhang wrote:
> Add configuration ExceptionList and IgnoreFiles for package config
> files. So users can rely on this to ignore some Ecc issues.
>
> Cc: Maciej Rabeda <maciej.rabeda@linux.intel.com>
> Cc: Jiaxin Wu <jiaxin.wu@intel.com>
> Cc: Siyuan Fu <siyuan.fu@intel.com>
> Signed-off-by: Shenglei Zhang <shenglei.zhang@intel.com>
> ---
>   NetworkPkg/NetworkPkg.ci.yaml | 10 ++++++++++
>   1 file changed, 10 insertions(+)
>
> diff --git a/NetworkPkg/NetworkPkg.ci.yaml b/NetworkPkg/NetworkPkg.ci.yaml
> index 70f2e1014748..7cd61912c71e 100644
> --- a/NetworkPkg/NetworkPkg.ci.yaml
> +++ b/NetworkPkg/NetworkPkg.ci.yaml
> @@ -5,6 +5,16 @@
>   # SPDX-License-Identifier: BSD-2-Clause-Patent
>   ##
>   {
> +    "EccCheck": {
> +        ## Exception sample looks like below:
> +        ## "ExceptionList": [
> +        ##     "<ErrorID>", "<KeyWord>"
> +        ## ]
> +        "ExceptionList": [
> +        ],
> +        "IgnoreFiles": [
> +        ]
> +    },
>       "CompilerPlugin": {
>           "DscPath": "NetworkPkg.dsc"
>       },

Re: [PATCH v3 05/17] ArmVirtPkg/ArmVirtPkg.ci.yaml: Add configuration for Ecc check

[Laszlo Ersek]

On 06/10/20 10:14, Shenglei Zhang wrote:
> Add configuration ExceptionList and IgnoreFiles for package config
> files. So users can rely on this to ignore some Ecc issues.
> 
> Cc: Laszlo Ersek <lersek@redhat.com>
> Cc: Ard Biesheuvel <ard.biesheuvel@arm.com>
> Cc: Leif Lindholm <leif@nuviainc.com>
> Signed-off-by: Shenglei Zhang <shenglei.zhang@intel.com>
> ---
>  ArmVirtPkg/ArmVirtPkg.ci.yaml | 10 ++++++++++
>  1 file changed, 10 insertions(+)
> 
> diff --git a/ArmVirtPkg/ArmVirtPkg.ci.yaml b/ArmVirtPkg/ArmVirtPkg.ci.yaml
> index 4553725ee528..b52e621b3c5c 100644
> --- a/ArmVirtPkg/ArmVirtPkg.ci.yaml
> +++ b/ArmVirtPkg/ArmVirtPkg.ci.yaml
> @@ -8,6 +8,16 @@
>  # SPDX-License-Identifier: BSD-2-Clause-Patent
>  ##
>  {
> +    "EccCheck": {
> +        ## Exception sample looks like below:
> +        ## "ExceptionList": [
> +        ##     "<ErrorID>", "<KeyWord>"
> +        ## ]
> +        "ExceptionList": [
> +        ],
> +        "IgnoreFiles": [
> +        ]
> +    },
>      ## options defined .pytool/Plugin/CompilerPlugin
>      "CompilerPlugin": {
>          "DscPath": "" # Don't support this test
> 

Acked-by: Laszlo Ersek <lersek@redhat.com>

Re: [PATCH v3 12/17] OvmfPkg/OvmfPkg.ci.yaml: Add configuration for Ecc check

[Laszlo Ersek]

On 06/10/20 10:14, Shenglei Zhang wrote:
> Add configuration ExceptionList and IgnoreFiles for package config
> files. So users can rely on this to ignore some Ecc issues.
> 
> Cc: Jordan Justen <jordan.l.justen@intel.com>
> Cc: Laszlo Ersek <lersek@redhat.com>
> Cc: Ard Biesheuvel <ard.biesheuvel@arm.com>
> Signed-off-by: Shenglei Zhang <shenglei.zhang@intel.com>
> ---
>  OvmfPkg/OvmfPkg.ci.yaml | 10 ++++++++++
>  1 file changed, 10 insertions(+)
> 
> diff --git a/OvmfPkg/OvmfPkg.ci.yaml b/OvmfPkg/OvmfPkg.ci.yaml
> index 98992f0429ff..67561c0bace0 100644
> --- a/OvmfPkg/OvmfPkg.ci.yaml
> +++ b/OvmfPkg/OvmfPkg.ci.yaml
> @@ -8,6 +8,16 @@
>  # SPDX-License-Identifier: BSD-2-Clause-Patent
>  ##
>  {
> +    "EccCheck": {
> +        ## Exception sample looks like below:
> +        ## "ExceptionList": [
> +        ##     "<ErrorID>", "<KeyWord>"
> +        ## ]
> +        "ExceptionList": [
> +        ],
> +        "IgnoreFiles": [
> +        ]
> +    },
>      ## options defined .pytool/Plugin/CompilerPlugin
>      "CompilerPlugin": {
>          "DscPath": "" # Don't support this test
> 

Acked-by: Laszlo Ersek <lersek@redhat.com>

Re: [PATCH v3 16/17] UefiCpuPkg/UefiCpuPkg.ci.yaml: Add configuration for Ecc check

[Laszlo Ersek]

On 06/10/20 10:14, Shenglei Zhang wrote:
> Add configuration ExceptionList and IgnoreFiles for package config
> files. So users can rely on this to ignore some Ecc issues.
> 
> Cc: Eric Dong <eric.dong@intel.com>
> Cc: Ray Ni <ray.ni@intel.com>
> Cc: Laszlo Ersek <lersek@redhat.com>
> Signed-off-by: Shenglei Zhang <shenglei.zhang@intel.com>
> ---
>  UefiCpuPkg/UefiCpuPkg.ci.yaml | 10 ++++++++++
>  1 file changed, 10 insertions(+)
> 
> diff --git a/UefiCpuPkg/UefiCpuPkg.ci.yaml b/UefiCpuPkg/UefiCpuPkg.ci.yaml
> index 99e460a8b090..dc3ff362f7c2 100644
> --- a/UefiCpuPkg/UefiCpuPkg.ci.yaml
> +++ b/UefiCpuPkg/UefiCpuPkg.ci.yaml
> @@ -5,6 +5,16 @@
>  # SPDX-License-Identifier: BSD-2-Clause-Patent
>  ##
>  {
> +    "EccCheck": {
> +        ## Exception sample looks like below:
> +        ## "ExceptionList": [
> +        ##     "<ErrorID>", "<KeyWord>"
> +        ## ]
> +        "ExceptionList": [
> +        ],
> +        "IgnoreFiles": [
> +        ]
> +    },
>      "CompilerPlugin": {
>          "DscPath": "UefiCpuPkg.dsc"
>      },
> 

Acked-by: Laszlo Ersek <lersek@redhat.com>