[PATCH v7 00/16] Add a plugin to check Ecc issues for edk2 on open ci

[PATCH v7 00/16] Add a plugin to check Ecc issues for edk2 on open ci

[Zhang, Shenglei]

REF: https://bugzilla.tianocore.org/show_bug.cgi?id=2606
As planed we will enable Ecc check for edk2 on open ci. And they are
ready now. I appreciate receiving feedback and comments if someone
find errors or false positive issues.

I created a pipline of EccCheck for my forked edk2. Welcome everyone to
create pull request to test the quality of this plugin.
My forked tree: https://github.com/shenglei10/edk2

And I also created some test cases for ECC plugin. Below are test cases.
https://github.com/shenglei10/edk2/tree/ECC
Results can be view in below azure server.
https://dev.azure.com/shengleizhang/shengleizhang/_build?definitionId=12&_a=summary

Patches
1/16: It's a lib necessary for py3 to run Ecc on azure servers.

2/16: EccCheck.py is a plugin to report Ecc issues for commits. It can be run
     on azure servers for open ci, or a local virtual environment.

3/16~16/16: We consider some cases that will report out Ecc issues but they won't
     be fixed, like submodule and industry standard related things. So we
     add two configuration fields "Exception" and "IgnoreFiles" for people
     to use. These patches add configuration in yaml files for Ecc check.

Cc: Bob Feng <bob.c.feng@intel.com>
Cc: Bret Barkelew <Bret.Barkelew@microsoft.com>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Sean Brogan <sean.brogan@microsoft.com>

v2: Update 1/17, fix the bug that the script can't hanlde multiple commits.

v3: Update 1/17, set the only workalbe workspace is edk2 root directory.
    Update 2/17, designate the version of antlr4 is 4.7.1.
    Add 4/17~17/17.

v4. Update 1/17, remove the function EdksetupRebuild(), instead add
    function SetupEnvironment(). Update variables' format and type hints
    to pass flake8 and mypy.

v5. Conver the former method to plugin solution, to align with
    other check points on open ci.

v6. The 1/16 patch is missed in v5 series. Now add it in v6.

v7. Fix a bug that Ecc plugin can not be run correctly under Linux OS.

Shenglei Zhang (16):
  pip-requirements.txt: Add Ecc required lib
  .pytool/Plugin: Add a plugin EccCheck
  MdeModulePkg/MdeModulePkg.ci.yaml: Add configuration for Ecc check
  ArmVirtPkg/ArmVirtPkg.ci.yaml: Add configuration for Ecc check
  CryptoPkg/CryptoPkg.ci.yaml: Add configuration for Ecc check
  EmulatorPkg/EmulatorPkg.ci.yaml: Add configuration for Ecc check
  FatPkg/FatPkg.ci.yaml: Add configuration for Ecc check
  FmpDevicePkg/FmpDevicePkg.ci.yaml: Add configuration for Ecc check
  MdePkg/MdePkg.ci.yaml: Add configuration for Ecc check
  NetworkPkg/NetworkPkg.ci.yaml: Add configuration for Ecc check
  OvmfPkg/OvmfPkg.ci.yaml: Add configuration for Ecc check
  PcAtChipsetPkg/PcAtChipsetPkg.ci.yaml: Add configuration for Ecc check
  SecurityPkg/SecurityPkg.ci.yaml: Add configuration for Ecc check
  ShellPkg/ShellPkg.ci.yaml: Add configuration for Ecc check
  UefiCpuPkg/UefiCpuPkg.ci.yaml: Add configuration for Ecc check
  UnitTestFrameworkPkg: Add configuration for Ecc check in yaml file

 .pytool/Plugin/EccCheck/EccCheck.py           | 267 ++++++++++++++++++
 .pytool/Plugin/EccCheck/EccCheck_plug_in.yaml |  11 +
 .pytool/Plugin/EccCheck/Readme.md             |  15 +
 ArmVirtPkg/ArmVirtPkg.ci.yaml                 |  11 +
 CryptoPkg/CryptoPkg.ci.yaml                   |  11 +
 EmulatorPkg/EmulatorPkg.ci.yaml               |  11 +
 FatPkg/FatPkg.ci.yaml                         |  11 +
 FmpDevicePkg/FmpDevicePkg.ci.yaml             |  11 +
 MdeModulePkg/MdeModulePkg.ci.yaml             |  11 +
 MdePkg/MdePkg.ci.yaml                         |  11 +
 NetworkPkg/NetworkPkg.ci.yaml                 |  11 +
 OvmfPkg/OvmfPkg.ci.yaml                       |  11 +
 PcAtChipsetPkg/PcAtChipsetPkg.ci.yaml         |  11 +
 SecurityPkg/SecurityPkg.ci.yaml               |  11 +
 ShellPkg/ShellPkg.ci.yaml                     |  11 +
 UefiCpuPkg/UefiCpuPkg.ci.yaml                 |  11 +
 .../UnitTestFrameworkPkg.ci.yaml              |  10 +
 pip-requirements.txt                          |   1 +
 18 files changed, 447 insertions(+)
 create mode 100644 .pytool/Plugin/EccCheck/EccCheck.py
 create mode 100644 .pytool/Plugin/EccCheck/EccCheck_plug_in.yaml
 create mode 100644 .pytool/Plugin/EccCheck/Readme.md

-- 
2.18.0.windows.1

[PATCH v7 01/16] pip-requirements.txt: Add Ecc required lib

[Zhang, Shenglei]

antlr4-python3-runtime is a lib to support Ecc run with Py3.x.

Cc: Sean Brogan <sean.brogan@microsoft.com>
Cc: Bret Barkelew <Bret.Barkelew@microsoft.com>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Shenglei Zhang <shenglei.zhang@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
---
 pip-requirements.txt | 1 +
 1 file changed, 1 insertion(+)

diff --git a/pip-requirements.txt b/pip-requirements.txt
index 574dac43b1a6..0fecd37f2a83 100644
--- a/pip-requirements.txt
+++ b/pip-requirements.txt
@@ -14,3 +14,4 @@
 
 edk2-pytool-library==0.10.*
 edk2-pytool-extensions~=0.13.3
+antlr4-python3-runtime==4.7.1
-- 
2.18.0.windows.1

[PATCH v7 02/16] .pytool/Plugin: Add a plugin EccCheck

[Zhang, Shenglei]

REF: https://bugzilla.tianocore.org/show_bug.cgi?id=2606
EccCheck is a plugin to report Ecc issues for code in pull request
, which will be run on open ci.
But note not each kind of issue could be reported out.
It can only handle the issues, whose line number in CSV report
accurately map with their code in source code files. And Ecc issues
about comments can also be handled.

Cc: Sean Brogan <sean.brogan@microsoft.com>
Cc: Bret Barkelew <Bret.Barkelew@microsoft.com>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Shenglei Zhang <shenglei.zhang@intel.com>
---
 .pytool/Plugin/EccCheck/EccCheck.py           | 267 ++++++++++++++++++
 .pytool/Plugin/EccCheck/EccCheck_plug_in.yaml |  11 +
 .pytool/Plugin/EccCheck/Readme.md             |  15 +
 3 files changed, 293 insertions(+)
 create mode 100644 .pytool/Plugin/EccCheck/EccCheck.py
 create mode 100644 .pytool/Plugin/EccCheck/EccCheck_plug_in.yaml
 create mode 100644 .pytool/Plugin/EccCheck/Readme.md

diff --git a/.pytool/Plugin/EccCheck/EccCheck.py b/.pytool/Plugin/EccCheck/EccCheck.py
new file mode 100644
index 000000000000..3857f472eda2
--- /dev/null
+++ b/.pytool/Plugin/EccCheck/EccCheck.py
@@ -0,0 +1,267 @@
+# @file EccCheck.py
+#
+# Copyright (c) 2020, Intel Corporation. All rights reserved.<BR>
+# SPDX-License-Identifier: BSD-2-Clause-Patent
+##
+
+import os
+import re
+import csv
+import xml.dom.minidom
+from typing import List, Dict, Tuple
+import logging
+from io import StringIO
+from edk2toolext.environment import shell_environment
+from edk2toolext.environment.plugintypes.ci_build_plugin import ICiBuildPlugin
+from edk2toolext.environment.var_dict import VarDict
+from edk2toollib.utility_functions import RunCmd
+
+
+class EccCheck(ICiBuildPlugin):
+    """
+    A CiBuildPlugin that finds the Ecc issues of newly added code in pull request.
+
+    Configuration options:
+    "EccCheck": {
+        "ExceptionList": [],
+        "IgnoreFiles": []
+    },
+    """
+
+    ReModifyFile = re.compile(r'[B-Q,S-Z]+[\d]*\t(.*)')
+    FindModifyFile = re.compile(r'\+\+\+ b\/(.*)')
+    LineScopePattern = (r'@@ -\d*\,*\d* \+\d*\,*\d* @@.*')
+    LineNumRange = re.compile(r'@@ -\d*\,*\d* \+(\d*)\,*(\d*) @@.*')
+
+    def GetTestName(self, packagename: str, environment: VarDict) -> tuple:
+        """ Provide the testcase name and classname for use in reporting
+            testclassname: a descriptive string for the testcase can include whitespace
+            classname: should be patterned <packagename>.<plugin>.<optionally any unique condition>
+
+            Args:
+              packagename: string containing name of package to build
+              environment: The VarDict for the test to run in
+            Returns:
+                a tuple containing the testcase name and the classname
+                (testcasename, classname)
+        """
+        return ("Check for efi coding style for " + packagename, packagename + ".EccCheck")
+
+    ##
+    # External function of plugin.  This function is used to perform the task of the ci_build_plugin Plugin
+    #
+    #   - package is the edk2 path to package.  This means workspace/packagepath relative.
+    #   - edk2path object configured with workspace and packages path
+    #   - PkgConfig Object (dict) for the pkg
+    #   - EnvConfig Object
+    #   - Plugin Manager Instance
+    #   - Plugin Helper Obj Instance
+    #   - Junit Logger
+    #   - output_stream the StringIO output stream from this plugin via logging
+    def RunBuildPlugin(self, packagename, Edk2pathObj, pkgconfig, environment, PLM, PLMHelper, tc, output_stream=None):
+        edk2_path = Edk2pathObj.WorkspacePath
+        python_path = os.path.join(edk2_path, "BaseTools", "Source", "Python")
+        env = shell_environment.GetEnvironment()
+        env.set_shell_var('PYTHONPATH', python_path)
+        env.set_shell_var('WORKSPACE', edk2_path)
+        self.ECC_PASS = True
+        self.ApplyConfig(pkgconfig, edk2_path, packagename)
+        modify_dir_list = self.GetModifyDir(packagename)
+        patch = self.GetDiff(packagename)
+        ecc_diff_range = self.GetDiffRange(patch, packagename, edk2_path)
+        self.GenerateEccReport(modify_dir_list, ecc_diff_range, edk2_path)
+        ecc_log = os.path.join(edk2_path, "Ecc.log")
+        if self.ECC_PASS:
+            tc.SetSuccess()
+            self.RemoveFile(ecc_log)
+            return 0
+        else:
+            with open(ecc_log, encoding='utf8') as output:
+                ecc_output = output.readlines()
+                for line in ecc_output:
+                    logging.error(line)
+            self.RemoveFile(ecc_log)
+            tc.SetFailed("EccCheck failed for {0}".format(packagename), "Ecc detected issues")
+            return 1
+
+    def GetDiff(self, pkg: str) -> List[str]:
+        return_buffer = StringIO()
+        params = "diff --unified=0 origin/master HEAD"
+        RunCmd("git", params, outstream=return_buffer)
+        p = return_buffer.getvalue().strip()
+        patch = p.split("\n")
+        return_buffer.close()
+
+        return patch
+
+    def RemoveFile(self, file: str) -> None:
+        if os.path.exists(file):
+            os.remove(file)
+        return
+
+    def GetModifyDir(self, pkg: str) -> List[str]:
+        return_buffer = StringIO()
+        params = "diff --name-status" + ' HEAD' + ' origin/master'
+        RunCmd("git", params, outstream=return_buffer)
+        p1 = return_buffer.getvalue().strip()
+        dir_list = p1.split("\n")
+        return_buffer.close()
+        modify_dir_list = []
+        for modify_dir in dir_list:
+            file_path = self.ReModifyFile.findall(modify_dir)
+            if file_path:
+                file_dir = os.path.dirname(file_path[0])
+            else:
+                continue
+            if pkg in file_dir and file_dir != pkg:
+                modify_dir_list.append('%s' % file_dir)
+            else:
+                continue
+
+        modify_dir_list = list(set(modify_dir_list))
+        return modify_dir_list
+
+    def GetDiffRange(self, patch_diff: List[str], pkg: str, workingdir: str) -> Dict[str, List[Tuple[int, int]]]:
+        IsDelete = True
+        StartCheck = False
+        range_directory: Dict[str, List[Tuple[int, int]]] = {}
+        for line in patch_diff:
+            modify_file = self.FindModifyFile.findall(line)
+            if modify_file and pkg in modify_file[0] and not StartCheck and os.path.isfile(modify_file[0]):
+                modify_file_comment_dic = self.GetCommentRange(modify_file[0], workingdir)
+                IsDelete = False
+                StartCheck = True
+                modify_file_dic = modify_file[0]
+                modify_file_dic = modify_file_dic.replace("/", os.sep)
+                range_directory[modify_file_dic] = []
+            elif line.startswith('--- '):
+                StartCheck = False
+            elif re.match(self.LineScopePattern, line, re.I) and not IsDelete and StartCheck:
+                start_line = self.LineNumRange.search(line).group(1)
+                line_range = self.LineNumRange.search(line).group(2)
+                if not line_range:
+                    line_range = '1'
+                range_directory[modify_file_dic].append((int(start_line), int(start_line) + int(line_range) - 1))
+                for i in modify_file_comment_dic:
+                    if int(i[0]) <= int(start_line) <= int(i[1]):
+                        range_directory[modify_file_dic].append(i)
+        return range_directory
+
+    def GetCommentRange(self, modify_file: str, workingdir: str) -> List[Tuple[int, int]]:
+        modify_file_path = os.path.join(workingdir, modify_file)
+        with open(modify_file_path) as f:
+            line_no = 1
+            comment_range: List[Tuple[int, int]] = []
+            Start = False
+            for line in f:
+                if line.startswith('/**'):
+                    start_no = line_no
+                    Start = True
+                if line.startswith('**/') and Start:
+                    end_no = line_no
+                    Start = False
+                    comment_range.append((int(start_no), int(end_no)))
+                line_no += 1
+
+        if comment_range and comment_range[0][0] == 1:
+            del comment_range[0]
+        return comment_range
+
+    def GenerateEccReport(self, modify_dir_list: List[str], ecc_diff_range: Dict[str, List[Tuple[int, int]]],
+                          edk2_path: str) -> None:
+        ecc_need = False
+        ecc_run = True
+        config = os.path.join(edk2_path, "BaseTools", "Source", "Python", "Ecc", "config.ini")
+        exception = os.path.join(edk2_path, "BaseTools", "Source", "Python", "Ecc", "exception.xml")
+        report = os.path.join(edk2_path, "Ecc.csv")
+        for modify_dir in modify_dir_list:
+            target = os.path.join(edk2_path, modify_dir)
+            logging.info('Run ECC tool for the commit in %s' % modify_dir)
+            ecc_need = True
+            ecc_params = "-c {0} -e {1} -t {2} -r {3}".format(config, exception, target, report)
+            return_code = RunCmd("Ecc", ecc_params, workingdir=edk2_path)
+            if return_code != 0:
+                ecc_run = False
+                break
+            if not ecc_run:
+                logging.error('Fail to run ECC tool')
+            self.ParseEccReport(ecc_diff_range, edk2_path)
+
+        if not ecc_need:
+            logging.info("Doesn't need run ECC check")
+
+        revert_params = "checkout -- {}".format(exception)
+        RunCmd("git", revert_params)
+        return
+
+    def ParseEccReport(self, ecc_diff_range: Dict[str, List[Tuple[int, int]]], edk2_path: str) -> None:
+        ecc_log = os.path.join(edk2_path, "Ecc.log")
+        ecc_csv = "Ecc.csv"
+        file = os.listdir(edk2_path)
+        row_lines = []
+        if ecc_csv in file:
+            with open(ecc_csv) as csv_file:
+                reader = csv.reader(csv_file)
+                for row in reader:
+                    for modify_file in ecc_diff_range:
+                        if modify_file in row[3]:
+                            for i in ecc_diff_range[modify_file]:
+                                line_no = int(row[4])
+                                if i[0] <= line_no <= i[1]:
+                                    row[0] = '\nEFI coding style error'
+                                    row[1] = 'Error code: ' + row[1]
+                                    row[3] = 'file: ' + row[3]
+                                    row[4] = 'Line number: ' + row[4]
+                                    row_line = '\n  *'.join(row)
+                                    row_lines.append(row_line)
+                                    break
+                            break
+        if row_lines:
+            self.ECC_PASS = False
+
+        with open(ecc_log, 'a') as log:
+            all_line = '\n'.join(row_lines)
+            all_line = all_line + '\n'
+            log.writelines(all_line)
+        return
+
+    def ApplyConfig(self, pkgconfig: Dict[str, List[str]], edk2_path: str, pkg: str) -> None:
+        if "IgnoreFiles" in pkgconfig:
+            for a in pkgconfig["IgnoreFiles"]:
+                a = os.path.join(edk2_path, pkg, a)
+                a = a.replace(os.sep, "/")
+                try:
+                    logging.info("Ignoring File {0}".format(a))
+                    self.RemoveFile(a)
+                except Exception:
+                    logging.info("EccCheck.IgnoreInf -> {0} not found in filesystem.  Invalid ignore file".format(a))
+        if "ExceptionList" in pkgconfig:
+            exception_list = pkgconfig["ExceptionList"]
+            exception_xml = os.path.join(edk2_path, "BaseTools", "Source", "Python", "Ecc", "exception.xml")
+            try:
+                logging.info("Appending exceptions")
+                self.AppendException(exception_list, exception_xml)
+            except Exception as e:
+                logging.error("Fail to apply exceptions")
+                raise e
+        return
+
+    def AppendException(self, exception_list: List[str], exception_xml: str) -> None:
+        error_code_list = exception_list[::2]
+        keyword_list = exception_list[1::2]
+        dom_tree = xml.dom.minidom.parse(exception_xml)
+        root_node = dom_tree.documentElement
+        for error_code, keyword in zip(error_code_list, keyword_list):
+            customer_node = dom_tree.createElement("Exception")
+            keyword_node = dom_tree.createElement("KeyWord")
+            keyword_node_text_value = dom_tree.createTextNode(keyword)
+            keyword_node.appendChild(keyword_node_text_value)
+            customer_node.appendChild(keyword_node)
+            error_code_node = dom_tree.createElement("ErrorID")
+            error_code_text_value = dom_tree.createTextNode(error_code)
+            error_code_node.appendChild(error_code_text_value)
+            customer_node.appendChild(error_code_node)
+            root_node.appendChild(customer_node)
+        with open(exception_xml, 'w') as f:
+            dom_tree.writexml(f, indent='', addindent='', newl='\n', encoding='UTF-8')
+        return
diff --git a/.pytool/Plugin/EccCheck/EccCheck_plug_in.yaml b/.pytool/Plugin/EccCheck/EccCheck_plug_in.yaml
new file mode 100644
index 000000000000..0d121ecb1861
--- /dev/null
+++ b/.pytool/Plugin/EccCheck/EccCheck_plug_in.yaml
@@ -0,0 +1,11 @@
+## @file
+# CiBuildPlugin used to check Ecc issues
+#
+# Copyright (c) 2020, Intel Corporation. All rights reserved.<BR>
+# SPDX-License-Identifier: BSD-2-Clause-Patent
+##
+{
+  "scope": "cibuild",
+  "name": "EccCheck Test",
+  "module": "EccCheck"
+}
diff --git a/.pytool/Plugin/EccCheck/Readme.md b/.pytool/Plugin/EccCheck/Readme.md
new file mode 100644
index 000000000000..85c439a5edf1
--- /dev/null
+++ b/.pytool/Plugin/EccCheck/Readme.md
@@ -0,0 +1,15 @@
+# EFI Coding style Check Plugin
+
+This CiBuildPlugin finds the Ecc issues of newly added code in pull request.
+
+## Configuration
+
+The plugin can be configured to ignore certain files and issues.
+
+"EccCheck": {
+        "ExceptionList": [],
+        "IgnoreFiles": []
+    },
+    """
+
+OPTIONAL List of file to ignore.
-- 
2.18.0.windows.1

[PATCH v7 03/16] MdeModulePkg/MdeModulePkg.ci.yaml: Add configuration for Ecc check

[Zhang, Shenglei]

Add configuration ExceptionList and IgnoreFiles for package config
files. So users can rely on this to ignore some Ecc issues.

Cc: Jian J Wang <jian.j.wang@intel.com>
Cc: Hao A Wu <hao.a.wu@intel.com>
Signed-off-by: Shenglei Zhang <shenglei.zhang@intel.com>
---
 MdeModulePkg/MdeModulePkg.ci.yaml | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/MdeModulePkg/MdeModulePkg.ci.yaml b/MdeModulePkg/MdeModulePkg.ci.yaml
index 1cfc1328390e..0ed929855417 100644
--- a/MdeModulePkg/MdeModulePkg.ci.yaml
+++ b/MdeModulePkg/MdeModulePkg.ci.yaml
@@ -2,9 +2,20 @@
 # CI configuration for MdeModulePkg
 #
 # Copyright (c) Microsoft Corporation
+# Copyright (c) 2020, Intel Corporation. All rights reserved.<BR>
 # SPDX-License-Identifier: BSD-2-Clause-Patent
 ##
 {
+    "EccCheck": {
+        ## Exception sample looks like below:
+        ## "ExceptionList": [
+        ##     "<ErrorID>", "<KeyWord>"
+        ## ]
+        "ExceptionList": [
+        ],
+        "IgnoreFiles": [
+        ]
+    },
     ## options defined ci/Plugin/CompilerPlugin
     "CompilerPlugin": {
         "DscPath": "MdeModulePkg.dsc"
-- 
2.18.0.windows.1

[PATCH v7 04/16] ArmVirtPkg/ArmVirtPkg.ci.yaml: Add configuration for Ecc check

[Zhang, Shenglei]

Add configuration ExceptionList and IgnoreFiles for package config
files. So users can rely on this to ignore some Ecc issues.

Cc: Laszlo Ersek <lersek@redhat.com>
Cc: Ard Biesheuvel <ard.biesheuvel@arm.com>
Cc: Leif Lindholm <leif@nuviainc.com>
Signed-off-by: Shenglei Zhang <shenglei.zhang@intel.com>
Acked-by: Laszlo Ersek <lersek@redhat.com>
---
 ArmVirtPkg/ArmVirtPkg.ci.yaml | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/ArmVirtPkg/ArmVirtPkg.ci.yaml b/ArmVirtPkg/ArmVirtPkg.ci.yaml
index 4553725ee528..3b321c806a62 100644
--- a/ArmVirtPkg/ArmVirtPkg.ci.yaml
+++ b/ArmVirtPkg/ArmVirtPkg.ci.yaml
@@ -5,9 +5,20 @@
 # used for code analysis.
 #
 # Copyright (c) Microsoft Corporation
+# Copyright (c) 2020, Intel Corporation. All rights reserved.<BR>
 # SPDX-License-Identifier: BSD-2-Clause-Patent
 ##
 {
+    "EccCheck": {
+        ## Exception sample looks like below:
+        ## "ExceptionList": [
+        ##     "<ErrorID>", "<KeyWord>"
+        ## ]
+        "ExceptionList": [
+        ],
+        "IgnoreFiles": [
+        ]
+    },
     ## options defined .pytool/Plugin/CompilerPlugin
     "CompilerPlugin": {
         "DscPath": "" # Don't support this test
-- 
2.18.0.windows.1

[PATCH v7 05/16] CryptoPkg/CryptoPkg.ci.yaml: Add configuration for Ecc check

[Zhang, Shenglei]

Add configuration ExceptionList and IgnoreFiles for package config
files. So users can rely on this to ignore some Ecc issues.

Cc: Jian J Wang <jian.j.wang@intel.com>
Cc: Xiaoyu Lu <xiaoyux.lu@intel.com>
Signed-off-by: Shenglei Zhang <shenglei.zhang@intel.com>
Reviewed-by: Guomin Jiang <guomin.jiang@intel.com>
Reviewed-by: Jian J Wang <jian.j.wang@intel.com>
---
 CryptoPkg/CryptoPkg.ci.yaml | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/CryptoPkg/CryptoPkg.ci.yaml b/CryptoPkg/CryptoPkg.ci.yaml
index f54ebfb22e70..c60d8bac6dee 100644
--- a/CryptoPkg/CryptoPkg.ci.yaml
+++ b/CryptoPkg/CryptoPkg.ci.yaml
@@ -2,9 +2,20 @@
 # CI configuration for CryptoPkg
 #
 # Copyright (c) Microsoft Corporation
+# Copyright (c) 2020, Intel Corporation. All rights reserved.<BR>
 # SPDX-License-Identifier: BSD-2-Clause-Patent
 ##
 {
+    "EccCheck": {
+        ## Exception sample looks like below:
+        ## "ExceptionList": [
+        ##     "<ErrorID>", "<KeyWord>"
+        ## ]
+        "ExceptionList": [
+        ],
+        "IgnoreFiles": [
+        ]
+    },
     "CompilerPlugin": {
         "DscPath": "CryptoPkg.dsc"
     },
-- 
2.18.0.windows.1

[PATCH v7 06/16] EmulatorPkg/EmulatorPkg.ci.yaml: Add configuration for Ecc check

[Zhang, Shenglei]

Add configuration ExceptionList and IgnoreFiles for package config
files. So users can rely on this to ignore some Ecc issues.

Cc: Jordan Justen <jordan.l.justen@intel.com>
Cc: Andrew Fish <afish@apple.com>
Cc: Ray Ni <ray.ni@intel.com>
Signed-off-by: Shenglei Zhang <shenglei.zhang@intel.com>
Acked-by: Ray Ni <ray.ni@intel.com>
---
 EmulatorPkg/EmulatorPkg.ci.yaml | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/EmulatorPkg/EmulatorPkg.ci.yaml b/EmulatorPkg/EmulatorPkg.ci.yaml
index 81f81780ec76..a664f4462e5d 100644
--- a/EmulatorPkg/EmulatorPkg.ci.yaml
+++ b/EmulatorPkg/EmulatorPkg.ci.yaml
@@ -5,9 +5,20 @@
 # used for code analysis.
 #
 # Copyright (c) Microsoft Corporation
+# Copyright (c) 2020, Intel Corporation. All rights reserved.<BR>
 # SPDX-License-Identifier: BSD-2-Clause-Patent
 ##
 {
+    "EccCheck": {
+        ## Exception sample looks like below:
+        ## "ExceptionList": [
+        ##     "<ErrorID>", "<KeyWord>"
+        ## ]
+        "ExceptionList": [
+        ],
+        "IgnoreFiles": [
+        ]
+    },
     ## options defined .pytool/Plugin/CompilerPlugin
     "CompilerPlugin": {
         "DscPath": "" # Don't support this test
-- 
2.18.0.windows.1

[PATCH v7 07/16] FatPkg/FatPkg.ci.yaml: Add configuration for Ecc check

[Zhang, Shenglei]

Add configuration ExceptionList and IgnoreFiles for package config
files. So users can rely on this to ignore some Ecc issues.

Cc: Ray Ni <ray.ni@intel.com>
Signed-off-by: Shenglei Zhang <shenglei.zhang@intel.com>
Reviewed-by: Guomin Jiang <guomin.jiang@intel.com>
---
 FatPkg/FatPkg.ci.yaml | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/FatPkg/FatPkg.ci.yaml b/FatPkg/FatPkg.ci.yaml
index 8b0fb1d4fcd5..e6a4c810976f 100644
--- a/FatPkg/FatPkg.ci.yaml
+++ b/FatPkg/FatPkg.ci.yaml
@@ -2,9 +2,20 @@
 # CI configuration for FatPkg
 #
 # Copyright (c) Microsoft Corporation
+# Copyright (c) 2020, Intel Corporation. All rights reserved.<BR>
 # SPDX-License-Identifier: BSD-2-Clause-Patent
 ##
 {
+    "EccCheck": {
+        ## Exception sample looks like below:
+        ## "ExceptionList": [
+        ##     "<ErrorID>", "<KeyWord>"
+        ## ]
+        "ExceptionList": [
+        ],
+        "IgnoreFiles": [
+        ]
+    },
     "CompilerPlugin": {
         "DscPath": "FatPkg.dsc"
     },
-- 
2.18.0.windows.1

[PATCH v7 08/16] FmpDevicePkg/FmpDevicePkg.ci.yaml: Add configuration for Ecc check

[Zhang, Shenglei]

Add configuration ExceptionList and IgnoreFiles for package config
files. So users can rely on this to ignore some Ecc issues.

Cc: Liming Gao <liming.gao@intel.com>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Signed-off-by: Shenglei Zhang <shenglei.zhang@intel.com>
Reviewed-by: Guomin Jiang <guomin.jiang@intel.com>
---
 FmpDevicePkg/FmpDevicePkg.ci.yaml | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/FmpDevicePkg/FmpDevicePkg.ci.yaml b/FmpDevicePkg/FmpDevicePkg.ci.yaml
index 74a0aefe8e49..b84400f06ffd 100644
--- a/FmpDevicePkg/FmpDevicePkg.ci.yaml
+++ b/FmpDevicePkg/FmpDevicePkg.ci.yaml
@@ -2,9 +2,20 @@
 # CI configuration for FmpDevicePkg
 #
 # Copyright (c) Microsoft Corporation
+# Copyright (c) 2020, Intel Corporation. All rights reserved.<BR>
 # SPDX-License-Identifier: BSD-2-Clause-Patent
 ##
 {
+    "EccCheck": {
+        ## Exception sample looks like below:
+        ## "ExceptionList": [
+        ##     "<ErrorID>", "<KeyWord>"
+        ## ]
+        "ExceptionList": [
+        ],
+        "IgnoreFiles": [
+        ]
+    },
     "CompilerPlugin": {
         "DscPath": "FmpDevicePkg.dsc"
     },
-- 
2.18.0.windows.1

[PATCH v7 09/16] MdePkg/MdePkg.ci.yaml: Add configuration for Ecc check

[Zhang, Shenglei]

Add configuration ExceptionList and IgnoreFiles for package config
files. So users can rely on this to ignore some Ecc issues.

Cc: Michael D Kinney <michael.d.kinney@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Signed-off-by: Shenglei Zhang <shenglei.zhang@intel.com>
Reviewed-by: Liming Gao <liming.gao@intel.com>
---
 MdePkg/MdePkg.ci.yaml | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/MdePkg/MdePkg.ci.yaml b/MdePkg/MdePkg.ci.yaml
index 3268f1535499..b6d7c57de83c 100644
--- a/MdePkg/MdePkg.ci.yaml
+++ b/MdePkg/MdePkg.ci.yaml
@@ -2,9 +2,20 @@
 # CI configuration for MdePkg
 #
 # Copyright (c) Microsoft Corporation
+# Copyright (c) 2020, Intel Corporation. All rights reserved.<BR>
 # SPDX-License-Identifier: BSD-2-Clause-Patent
 ##
 {
+    "EccCheck": {
+        ## Exception sample looks like below:
+        ## "ExceptionList": [
+        ##     "<ErrorID>", "<KeyWord>"
+        ## ]
+        "ExceptionList": [
+        ],
+        "IgnoreFiles": [
+        ]
+    },
     ## options defined ci/Plugin/CompilerPlugin
     "CompilerPlugin": {
         "DscPath": "MdePkg.dsc"
-- 
2.18.0.windows.1

[PATCH v7 10/16] NetworkPkg/NetworkPkg.ci.yaml: Add configuration for Ecc check

[Zhang, Shenglei]

Add configuration ExceptionList and IgnoreFiles for package config
files. So users can rely on this to ignore some Ecc issues.

Cc: Maciej Rabeda <maciej.rabeda@linux.intel.com>
Cc: Jiaxin Wu <jiaxin.wu@intel.com>
Cc: Siyuan Fu <siyuan.fu@intel.com>
Signed-off-by: Shenglei Zhang <shenglei.zhang@intel.com>
Reviewed-by: Maciej Rabeda <maciej.rabeda@linux.intel.com>
---
 NetworkPkg/NetworkPkg.ci.yaml | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/NetworkPkg/NetworkPkg.ci.yaml b/NetworkPkg/NetworkPkg.ci.yaml
index 70f2e1014748..d45faf0ac967 100644
--- a/NetworkPkg/NetworkPkg.ci.yaml
+++ b/NetworkPkg/NetworkPkg.ci.yaml
@@ -2,9 +2,20 @@
 # CI configuration for NetworkPkg
 #
 # Copyright (c) Microsoft Corporation
+# Copyright (c) 2020, Intel Corporation. All rights reserved.<BR>
 # SPDX-License-Identifier: BSD-2-Clause-Patent
 ##
 {
+    "EccCheck": {
+        ## Exception sample looks like below:
+        ## "ExceptionList": [
+        ##     "<ErrorID>", "<KeyWord>"
+        ## ]
+        "ExceptionList": [
+        ],
+        "IgnoreFiles": [
+        ]
+    },
     "CompilerPlugin": {
         "DscPath": "NetworkPkg.dsc"
     },
-- 
2.18.0.windows.1

[PATCH v7 11/16] OvmfPkg/OvmfPkg.ci.yaml: Add configuration for Ecc check

[Zhang, Shenglei]

Add configuration ExceptionList and IgnoreFiles for package config
files. So users can rely on this to ignore some Ecc issues.

Cc: Jordan Justen <jordan.l.justen@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Cc: Ard Biesheuvel <ard.biesheuvel@arm.com>
Signed-off-by: Shenglei Zhang <shenglei.zhang@intel.com>
Acked-by: Laszlo Ersek <lersek@redhat.com>
---
 OvmfPkg/OvmfPkg.ci.yaml | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/OvmfPkg/OvmfPkg.ci.yaml b/OvmfPkg/OvmfPkg.ci.yaml
index 98992f0429ff..e151cebf7020 100644
--- a/OvmfPkg/OvmfPkg.ci.yaml
+++ b/OvmfPkg/OvmfPkg.ci.yaml
@@ -5,9 +5,20 @@
 # used for code analysis.
 #
 # Copyright (c) Microsoft Corporation
+# Copyright (c) 2020, Intel Corporation. All rights reserved.<BR>
 # SPDX-License-Identifier: BSD-2-Clause-Patent
 ##
 {
+    "EccCheck": {
+        ## Exception sample looks like below:
+        ## "ExceptionList": [
+        ##     "<ErrorID>", "<KeyWord>"
+        ## ]
+        "ExceptionList": [
+        ],
+        "IgnoreFiles": [
+        ]
+    },
     ## options defined .pytool/Plugin/CompilerPlugin
     "CompilerPlugin": {
         "DscPath": "" # Don't support this test
-- 
2.18.0.windows.1

[PATCH v7 12/16] PcAtChipsetPkg/PcAtChipsetPkg.ci.yaml: Add configuration for Ecc check

[Zhang, Shenglei]

Add configuration ExceptionList and IgnoreFiles for package config
files. So users can rely on this to ignore some Ecc issues.

Cc: Ray Ni <ray.ni@intel.com>
Signed-off-by: Shenglei Zhang <shenglei.zhang@intel.com>
Acked-by: Ray Ni <ray.ni@intel.com>
---
 PcAtChipsetPkg/PcAtChipsetPkg.ci.yaml | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/PcAtChipsetPkg/PcAtChipsetPkg.ci.yaml b/PcAtChipsetPkg/PcAtChipsetPkg.ci.yaml
index be470807bd9e..c59287064d65 100644
--- a/PcAtChipsetPkg/PcAtChipsetPkg.ci.yaml
+++ b/PcAtChipsetPkg/PcAtChipsetPkg.ci.yaml
@@ -2,9 +2,20 @@
 # CI configuration for PcAtChipsetPkg
 #
 # Copyright (c) Microsoft Corporation
+# Copyright (c) 2020, Intel Corporation. All rights reserved.<BR>
 # SPDX-License-Identifier: BSD-2-Clause-Patent
 ##
 {
+    "EccCheck": {
+        ## Exception sample looks like below:
+        ## "ExceptionList": [
+        ##     "<ErrorID>", "<KeyWord>"
+        ## ]
+        "ExceptionList": [
+        ],
+        "IgnoreFiles": [
+        ]
+    },
     "CompilerPlugin": {
         "DscPath": "PcAtChipsetPkg.dsc"
     },
-- 
2.18.0.windows.1

[PATCH v7 13/16] SecurityPkg/SecurityPkg.ci.yaml: Add configuration for Ecc check

[Zhang, Shenglei]

Add configuration ExceptionList and IgnoreFiles for package config
files. So users can rely on this to ignore some Ecc issues.

Cc: Jiewen Yao <jiewen.yao@intel.com>
Cc: Jian J Wang <jian.j.wang@intel.com>
Cc: Chao Zhang <chao.b.zhang@intel.com>
Signed-off-by: Shenglei Zhang <shenglei.zhang@intel.com>
Reviewed-by: Jian J Wang <jian.j.wang@intel.com>
---
 SecurityPkg/SecurityPkg.ci.yaml | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/SecurityPkg/SecurityPkg.ci.yaml b/SecurityPkg/SecurityPkg.ci.yaml
index 953219053318..a0e143140875 100644
--- a/SecurityPkg/SecurityPkg.ci.yaml
+++ b/SecurityPkg/SecurityPkg.ci.yaml
@@ -2,9 +2,20 @@
 # CI configuration for SecurityPkg
 #
 # Copyright (c) Microsoft Corporation
+# Copyright (c) 2020, Intel Corporation. All rights reserved.<BR>
 # SPDX-License-Identifier: BSD-2-Clause-Patent
 ##
 {
+    "EccCheck": {
+        ## Exception sample looks like below:
+        ## "ExceptionList": [
+        ##     "<ErrorID>", "<KeyWord>"
+        ## ]
+        "ExceptionList": [
+        ],
+        "IgnoreFiles": [
+        ]
+    },
     "CompilerPlugin": {
         "DscPath": "SecurityPkg.dsc"
     },
-- 
2.18.0.windows.1

[PATCH v7 14/16] ShellPkg/ShellPkg.ci.yaml: Add configuration for Ecc check

[Zhang, Shenglei]

Add configuration ExceptionList and IgnoreFiles for package config
files. So users can rely on this to ignore some Ecc issues.

Cc: Ray Ni <ray.ni@intel.com>
Cc: Zhichao Gao <zhichao.gao@intel.com>
Signed-off-by: Shenglei Zhang <shenglei.zhang@intel.com>
Acked-by: Ray Ni <ray.ni@intel.com>
---
 ShellPkg/ShellPkg.ci.yaml | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/ShellPkg/ShellPkg.ci.yaml b/ShellPkg/ShellPkg.ci.yaml
index 67de34a2556e..eedc173738dc 100644
--- a/ShellPkg/ShellPkg.ci.yaml
+++ b/ShellPkg/ShellPkg.ci.yaml
@@ -2,9 +2,20 @@
 # CI configuration for ShellPkg
 #
 # Copyright (c) Microsoft Corporation
+# Copyright (c) 2020, Intel Corporation. All rights reserved.<BR>
 # SPDX-License-Identifier: BSD-2-Clause-Patent
 ##
 {
+    "EccCheck": {
+        ## Exception sample looks like below:
+        ## "ExceptionList": [
+        ##     "<ErrorID>", "<KeyWord>"
+        ## ]
+        "ExceptionList": [
+        ],
+        "IgnoreFiles": [
+        ]
+    },
     "CompilerPlugin": {
         "DscPath": "ShellPkg.dsc"
     },
-- 
2.18.0.windows.1

[PATCH v7 15/16] UefiCpuPkg/UefiCpuPkg.ci.yaml: Add configuration for Ecc check

[Zhang, Shenglei]

Add configuration ExceptionList and IgnoreFiles for package config
files. So users can rely on this to ignore some Ecc issues.

Cc: Eric Dong <eric.dong@intel.com>
Cc: Ray Ni <ray.ni@intel.com>
Cc: Laszlo Ersek <lersek@redhat.com>
Signed-off-by: Shenglei Zhang <shenglei.zhang@intel.com>
Acked-by: Ray Ni <ray.ni@intel.com>
Reviewed-by: Eric Dong <eric.dong@intel.com>
Acked-by: Laszlo Ersek <lersek@redhat.com>
---
 UefiCpuPkg/UefiCpuPkg.ci.yaml | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/UefiCpuPkg/UefiCpuPkg.ci.yaml b/UefiCpuPkg/UefiCpuPkg.ci.yaml
index 99e460a8b090..0e216344cd53 100644
--- a/UefiCpuPkg/UefiCpuPkg.ci.yaml
+++ b/UefiCpuPkg/UefiCpuPkg.ci.yaml
@@ -2,9 +2,20 @@
 # CI configuration for UefiCpuPkg
 #
 # Copyright (c) Microsoft Corporation
+# Copyright (c) 2020, Intel Corporation. All rights reserved.<BR>
 # SPDX-License-Identifier: BSD-2-Clause-Patent
 ##
 {
+    "EccCheck": {
+        ## Exception sample looks like below:
+        ## "ExceptionList": [
+        ##     "<ErrorID>", "<KeyWord>"
+        ## ]
+        "ExceptionList": [
+        ],
+        "IgnoreFiles": [
+        ]
+    },
     "CompilerPlugin": {
         "DscPath": "UefiCpuPkg.dsc"
     },
-- 
2.18.0.windows.1

[PATCH v7 16/16] UnitTestFrameworkPkg: Add configuration for Ecc check in yaml file

[Zhang, Shenglei]

Add configuration ExceptionList and IgnoreFiles for package config
files. So users can rely on this to ignore some Ecc issues.

Cc: Michael D Kinney <michael.d.kinney@intel.com>
Cc: Sean Brogan <sean.brogan@microsoft.com>
Cc: Bret Barkelew <Bret.Barkelew@microsoft.com>
Signed-off-by: Shenglei Zhang <shenglei.zhang@intel.com>
Acked-by: Ray Ni <ray.ni@intel.com>
---
 UnitTestFrameworkPkg/UnitTestFrameworkPkg.ci.yaml | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/UnitTestFrameworkPkg/UnitTestFrameworkPkg.ci.yaml b/UnitTestFrameworkPkg/UnitTestFrameworkPkg.ci.yaml
index 51e172537f8a..7e9fc5d005fb 100644
--- a/UnitTestFrameworkPkg/UnitTestFrameworkPkg.ci.yaml
+++ b/UnitTestFrameworkPkg/UnitTestFrameworkPkg.ci.yaml
@@ -5,6 +5,16 @@
 # SPDX-License-Identifier: BSD-2-Clause-Patent
 ##
 {
+    "EccCheck": {
+        ## Exception sample looks like below:
+        ## "ExceptionList": [
+        ##     "<ErrorID>", "<KeyWord>"
+        ## ]
+        "ExceptionList": [
+        ],
+        "IgnoreFiles": [
+        ]
+    },
     ## options defined .pytool/Plugin/CompilerPlugin
     "CompilerPlugin": {
         "DscPath": "UnitTestFrameworkPkg.dsc"
-- 
2.18.0.windows.1

Re: [edk2-devel] [PATCH v7 00/16] Add a plugin to check Ecc issues for edk2 on open ci

[Leif Lindholm]

On Mon, Jul 06, 2020 at 16:48:30 +0800, Zhang, Shenglei wrote:
> REF: https://bugzilla.tianocore.org/show_bug.cgi?id=2606
> As planed we will enable Ecc check for edk2 on open ci. And they are
> ready now. I appreciate receiving feedback and comments if someone
> find errors or false positive issues.
> 
> I created a pipline of EccCheck for my forked edk2. Welcome everyone to
> create pull request to test the quality of this plugin.
> My forked tree: https://github.com/shenglei10/edk2
> 
> And I also created some test cases for ECC plugin. Below are test cases.
> https://github.com/shenglei10/edk2/tree/ECC
> Results can be view in below azure server.
> https://dev.azure.com/shengleizhang/shengleizhang/_build?definitionId=12&_a=summary
> 
> Patches
> 1/16: It's a lib necessary for py3 to run Ecc on azure servers.
> 
> 2/16: EccCheck.py is a plugin to report Ecc issues for commits. It can be run
>      on azure servers for open ci, or a local virtual environment.
> 
> 3/16~16/16: We consider some cases that will report out Ecc issues but they won't
>      be fixed, like submodule and industry standard related things. So we
>      add two configuration fields "Exception" and "IgnoreFiles" for people
>      to use. These patches add configuration in yaml files for Ecc check.
> 
> Cc: Bob Feng <bob.c.feng@intel.com>
> Cc: Bret Barkelew <Bret.Barkelew@microsoft.com>
> Cc: Michael D Kinney <michael.d.kinney@intel.com>
> Cc: Liming Gao <liming.gao@intel.com>
> Cc: Sean Brogan <sean.brogan@microsoft.com>
> 
> v2: Update 1/17, fix the bug that the script can't hanlde multiple commits.
> 
> v3: Update 1/17, set the only workalbe workspace is edk2 root directory.
>     Update 2/17, designate the version of antlr4 is 4.7.1.
>     Add 4/17~17/17.
> 
> v4. Update 1/17, remove the function EdksetupRebuild(), instead add
>     function SetupEnvironment(). Update variables' format and type hints
>     to pass flake8 and mypy.
> 
> v5. Conver the former method to plugin solution, to align with
>     other check points on open ci.
> 
> v6. The 1/16 patch is missed in v5 series. Now add it in v6.
> 
> v7. Fix a bug that Ecc plugin can not be run correctly under Linux OS.

What is the bug? Where is it fixed?

/
    Leif

> Shenglei Zhang (16):
>   pip-requirements.txt: Add Ecc required lib
>   .pytool/Plugin: Add a plugin EccCheck
>   MdeModulePkg/MdeModulePkg.ci.yaml: Add configuration for Ecc check
>   ArmVirtPkg/ArmVirtPkg.ci.yaml: Add configuration for Ecc check
>   CryptoPkg/CryptoPkg.ci.yaml: Add configuration for Ecc check
>   EmulatorPkg/EmulatorPkg.ci.yaml: Add configuration for Ecc check
>   FatPkg/FatPkg.ci.yaml: Add configuration for Ecc check
>   FmpDevicePkg/FmpDevicePkg.ci.yaml: Add configuration for Ecc check
>   MdePkg/MdePkg.ci.yaml: Add configuration for Ecc check
>   NetworkPkg/NetworkPkg.ci.yaml: Add configuration for Ecc check
>   OvmfPkg/OvmfPkg.ci.yaml: Add configuration for Ecc check
>   PcAtChipsetPkg/PcAtChipsetPkg.ci.yaml: Add configuration for Ecc check
>   SecurityPkg/SecurityPkg.ci.yaml: Add configuration for Ecc check
>   ShellPkg/ShellPkg.ci.yaml: Add configuration for Ecc check
>   UefiCpuPkg/UefiCpuPkg.ci.yaml: Add configuration for Ecc check
>   UnitTestFrameworkPkg: Add configuration for Ecc check in yaml file
> 
>  .pytool/Plugin/EccCheck/EccCheck.py           | 267 ++++++++++++++++++
>  .pytool/Plugin/EccCheck/EccCheck_plug_in.yaml |  11 +
>  .pytool/Plugin/EccCheck/Readme.md             |  15 +
>  ArmVirtPkg/ArmVirtPkg.ci.yaml                 |  11 +
>  CryptoPkg/CryptoPkg.ci.yaml                   |  11 +
>  EmulatorPkg/EmulatorPkg.ci.yaml               |  11 +
>  FatPkg/FatPkg.ci.yaml                         |  11 +
>  FmpDevicePkg/FmpDevicePkg.ci.yaml             |  11 +
>  MdeModulePkg/MdeModulePkg.ci.yaml             |  11 +
>  MdePkg/MdePkg.ci.yaml                         |  11 +
>  NetworkPkg/NetworkPkg.ci.yaml                 |  11 +
>  OvmfPkg/OvmfPkg.ci.yaml                       |  11 +
>  PcAtChipsetPkg/PcAtChipsetPkg.ci.yaml         |  11 +
>  SecurityPkg/SecurityPkg.ci.yaml               |  11 +
>  ShellPkg/ShellPkg.ci.yaml                     |  11 +
>  UefiCpuPkg/UefiCpuPkg.ci.yaml                 |  11 +
>  .../UnitTestFrameworkPkg.ci.yaml              |  10 +
>  pip-requirements.txt                          |   1 +
>  18 files changed, 447 insertions(+)
>  create mode 100644 .pytool/Plugin/EccCheck/EccCheck.py
>  create mode 100644 .pytool/Plugin/EccCheck/EccCheck_plug_in.yaml
>  create mode 100644 .pytool/Plugin/EccCheck/Readme.md
> 
> -- 
> 2.18.0.windows.1
> 
> 
> 
> 

Re: [edk2-devel] [PATCH v7 00/16] Add a plugin to check Ecc issues for edk2 on open ci

[Zhang, Shenglei]

Comments below.

> -----Original Message-----
> From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Leif
> Lindholm
> Sent: Monday, July 6, 2020 4:55 PM
> To: devel@edk2.groups.io; Zhang, Shenglei <shenglei.zhang@intel.com>
> Cc: Feng, Bob C <bob.c.feng@intel.com>; Bret Barkelew
> <Bret.Barkelew@microsoft.com>; Kinney, Michael D
> <michael.d.kinney@intel.com>; Gao, Liming <liming.gao@intel.com>; Sean
> Brogan <sean.brogan@microsoft.com>
> Subject: Re: [edk2-devel] [PATCH v7 00/16] Add a plugin to check Ecc issues
> for edk2 on open ci
> 
> On Mon, Jul 06, 2020 at 16:48:30 +0800, Zhang, Shenglei wrote:
> > REF: https://bugzilla.tianocore.org/show_bug.cgi?id=2606
> > As planed we will enable Ecc check for edk2 on open ci. And they are
> > ready now. I appreciate receiving feedback and comments if someone
> > find errors or false positive issues.
> >
> > I created a pipline of EccCheck for my forked edk2. Welcome everyone to
> > create pull request to test the quality of this plugin.
> > My forked tree: https://github.com/shenglei10/edk2
> >
> > And I also created some test cases for ECC plugin. Below are test cases.
> > https://github.com/shenglei10/edk2/tree/ECC
> > Results can be view in below azure server.
> >
> https://dev.azure.com/shengleizhang/shengleizhang/_build?definitionId=12
> &_a=summary
> >
> > Patches
> > 1/16: It's a lib necessary for py3 to run Ecc on azure servers.
> >
> > 2/16: EccCheck.py is a plugin to report Ecc issues for commits. It can be run
> >      on azure servers for open ci, or a local virtual environment.
> >
> > 3/16~16/16: We consider some cases that will report out Ecc issues but they
> won't
> >      be fixed, like submodule and industry standard related things. So we
> >      add two configuration fields "Exception" and "IgnoreFiles" for people
> >      to use. These patches add configuration in yaml files for Ecc check.
> >
> > Cc: Bob Feng <bob.c.feng@intel.com>
> > Cc: Bret Barkelew <Bret.Barkelew@microsoft.com>
> > Cc: Michael D Kinney <michael.d.kinney@intel.com>
> > Cc: Liming Gao <liming.gao@intel.com>
> > Cc: Sean Brogan <sean.brogan@microsoft.com>
> >
> > v2: Update 1/17, fix the bug that the script can't hanlde multiple commits.
> >
> > v3: Update 1/17, set the only workalbe workspace is edk2 root directory.
> >     Update 2/17, designate the version of antlr4 is 4.7.1.
> >     Add 4/17~17/17.
> >
> > v4. Update 1/17, remove the function EdksetupRebuild(), instead add
> >     function SetupEnvironment(). Update variables' format and type hints
> >     to pass flake8 and mypy.
> >
> > v5. Conver the former method to plugin solution, to align with
> >     other check points on open ci.
> >
> > v6. The 1/16 patch is missed in v5 series. Now add it in v6.
> >
> > v7. Fix a bug that Ecc plugin can not be run correctly under Linux OS.
> 
> What is the bug? Where is it fixed?

V6 plugin pipeline link:
https://dev.azure.com/shengleizhang/shengleizhang/_build/results?buildId=282&view=logs&j=9b8d87a5-bd93-5358-8ffa-588e312c4376&t=282a77bb-da29-5801-6299-2bbe15d5aabb
v7 plugin pipeline link:
https://dev.azure.com/shengleizhang/shengleizhang/_build/results?buildId=316&view=logs&j=5e3952f2-9c8a-5394-82c1-a803bcab3ca3&t=7ea9e85e-5c3d-5fcf-865e-8449f16cc745

For the same pull request, v6 plugin can't report out the Ecc issues under Linux OS.

Thanks,
Shenglei

> 
> /
>     Leif
> 
> > Shenglei Zhang (16):
> >   pip-requirements.txt: Add Ecc required lib
> >   .pytool/Plugin: Add a plugin EccCheck
> >   MdeModulePkg/MdeModulePkg.ci.yaml: Add configuration for Ecc check
> >   ArmVirtPkg/ArmVirtPkg.ci.yaml: Add configuration for Ecc check
> >   CryptoPkg/CryptoPkg.ci.yaml: Add configuration for Ecc check
> >   EmulatorPkg/EmulatorPkg.ci.yaml: Add configuration for Ecc check
> >   FatPkg/FatPkg.ci.yaml: Add configuration for Ecc check
> >   FmpDevicePkg/FmpDevicePkg.ci.yaml: Add configuration for Ecc check
> >   MdePkg/MdePkg.ci.yaml: Add configuration for Ecc check
> >   NetworkPkg/NetworkPkg.ci.yaml: Add configuration for Ecc check
> >   OvmfPkg/OvmfPkg.ci.yaml: Add configuration for Ecc check
> >   PcAtChipsetPkg/PcAtChipsetPkg.ci.yaml: Add configuration for Ecc check
> >   SecurityPkg/SecurityPkg.ci.yaml: Add configuration for Ecc check
> >   ShellPkg/ShellPkg.ci.yaml: Add configuration for Ecc check
> >   UefiCpuPkg/UefiCpuPkg.ci.yaml: Add configuration for Ecc check
> >   UnitTestFrameworkPkg: Add configuration for Ecc check in yaml file
> >
> >  .pytool/Plugin/EccCheck/EccCheck.py           | 267 ++++++++++++++++++
> >  .pytool/Plugin/EccCheck/EccCheck_plug_in.yaml |  11 +
> >  .pytool/Plugin/EccCheck/Readme.md             |  15 +
> >  ArmVirtPkg/ArmVirtPkg.ci.yaml                 |  11 +
> >  CryptoPkg/CryptoPkg.ci.yaml                   |  11 +
> >  EmulatorPkg/EmulatorPkg.ci.yaml               |  11 +
> >  FatPkg/FatPkg.ci.yaml                         |  11 +
> >  FmpDevicePkg/FmpDevicePkg.ci.yaml             |  11 +
> >  MdeModulePkg/MdeModulePkg.ci.yaml             |  11 +
> >  MdePkg/MdePkg.ci.yaml                         |  11 +
> >  NetworkPkg/NetworkPkg.ci.yaml                 |  11 +
> >  OvmfPkg/OvmfPkg.ci.yaml                       |  11 +
> >  PcAtChipsetPkg/PcAtChipsetPkg.ci.yaml         |  11 +
> >  SecurityPkg/SecurityPkg.ci.yaml               |  11 +
> >  ShellPkg/ShellPkg.ci.yaml                     |  11 +
> >  UefiCpuPkg/UefiCpuPkg.ci.yaml                 |  11 +
> >  .../UnitTestFrameworkPkg.ci.yaml              |  10 +
> >  pip-requirements.txt                          |   1 +
> >  18 files changed, 447 insertions(+)
> >  create mode 100644 .pytool/Plugin/EccCheck/EccCheck.py
> >  create mode 100644 .pytool/Plugin/EccCheck/EccCheck_plug_in.yaml
> >  create mode 100644 .pytool/Plugin/EccCheck/Readme.md
> >
> > --
> > 2.18.0.windows.1
> >
> >
> >
> >
> 
> 

Re: [edk2-devel] [PATCH v7 00/16] Add a plugin to check Ecc issues for edk2 on open ci

[Leif Lindholm]

On Tue, Jul 07, 2020 at 02:24:14 +0000, Zhang, Shenglei wrote:
> Comments below.
> 
> > -----Original Message-----
> > From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Leif
> > Lindholm
> > Sent: Monday, July 6, 2020 4:55 PM
> > To: devel@edk2.groups.io; Zhang, Shenglei <shenglei.zhang@intel.com>
> > Cc: Feng, Bob C <bob.c.feng@intel.com>; Bret Barkelew
> > <Bret.Barkelew@microsoft.com>; Kinney, Michael D
> > <michael.d.kinney@intel.com>; Gao, Liming <liming.gao@intel.com>; Sean
> > Brogan <sean.brogan@microsoft.com>
> > Subject: Re: [edk2-devel] [PATCH v7 00/16] Add a plugin to check Ecc issues
> > for edk2 on open ci
> > 
> > On Mon, Jul 06, 2020 at 16:48:30 +0800, Zhang, Shenglei wrote:
> > > REF: https://bugzilla.tianocore.org/show_bug.cgi?id=2606
> > > As planed we will enable Ecc check for edk2 on open ci. And they are
> > > ready now. I appreciate receiving feedback and comments if someone
> > > find errors or false positive issues.
> > >
> > > I created a pipline of EccCheck for my forked edk2. Welcome everyone to
> > > create pull request to test the quality of this plugin.
> > > My forked tree: https://github.com/shenglei10/edk2
> > >
> > > And I also created some test cases for ECC plugin. Below are test cases.
> > > https://github.com/shenglei10/edk2/tree/ECC
> > > Results can be view in below azure server.
> > >
> > https://dev.azure.com/shengleizhang/shengleizhang/_build?definitionId=12
> > &_a=summary
> > >
> > > Patches
> > > 1/16: It's a lib necessary for py3 to run Ecc on azure servers.
> > >
> > > 2/16: EccCheck.py is a plugin to report Ecc issues for commits. It can be run
> > >      on azure servers for open ci, or a local virtual environment.
> > >
> > > 3/16~16/16: We consider some cases that will report out Ecc issues but they
> > won't
> > >      be fixed, like submodule and industry standard related things. So we
> > >      add two configuration fields "Exception" and "IgnoreFiles" for people
> > >      to use. These patches add configuration in yaml files for Ecc check.
> > >
> > > Cc: Bob Feng <bob.c.feng@intel.com>
> > > Cc: Bret Barkelew <Bret.Barkelew@microsoft.com>
> > > Cc: Michael D Kinney <michael.d.kinney@intel.com>
> > > Cc: Liming Gao <liming.gao@intel.com>
> > > Cc: Sean Brogan <sean.brogan@microsoft.com>
> > >
> > > v2: Update 1/17, fix the bug that the script can't hanlde multiple commits.
> > >
> > > v3: Update 1/17, set the only workalbe workspace is edk2 root directory.
> > >     Update 2/17, designate the version of antlr4 is 4.7.1.
> > >     Add 4/17~17/17.
> > >
> > > v4. Update 1/17, remove the function EdksetupRebuild(), instead add
> > >     function SetupEnvironment(). Update variables' format and type hints
> > >     to pass flake8 and mypy.
> > >
> > > v5. Conver the former method to plugin solution, to align with
> > >     other check points on open ci.
> > >
> > > v6. The 1/16 patch is missed in v5 series. Now add it in v6.
> > >
> > > v7. Fix a bug that Ecc plugin can not be run correctly under Linux OS.
> > 
> > What is the bug? Where is it fixed?
> 
> V6 plugin pipeline link:
> https://dev.azure.com/shengleizhang/shengleizhang/_build/results?buildId=282&view=logs&j=9b8d87a5-bd93-5358-8ffa-588e312c4376&t=282a77bb-da29-5801-6299-2bbe15d5aabb
> v7 plugin pipeline link:
> https://dev.azure.com/shengleizhang/shengleizhang/_build/results?buildId=316&view=logs&j=5e3952f2-9c8a-5394-82c1-a803bcab3ca3&t=7ea9e85e-5c3d-5fcf-865e-8449f16cc745
> 
> For the same pull request, v6 plugin can't report out the Ecc issues under Linux OS.

Right. So *what* was the bug and *where* (which patch or patches) was
it fixed? This is information that should (in an abbreviated form) be
provided in the cover letter.

"Fix a bug" is data, not information. The same is true for "can not be
run correctly". And neither tells me which patch I should look at.
If this has been covered in a specific discussion thread, a link to
the thread *and* mentioning which patch(es) it affects is sufficient.

Regards,

Leif

> 
> Thanks,
> Shenglei
> 
> > 
> > /
> >     Leif
> > 
> > > Shenglei Zhang (16):
> > >   pip-requirements.txt: Add Ecc required lib
> > >   .pytool/Plugin: Add a plugin EccCheck
> > >   MdeModulePkg/MdeModulePkg.ci.yaml: Add configuration for Ecc check
> > >   ArmVirtPkg/ArmVirtPkg.ci.yaml: Add configuration for Ecc check
> > >   CryptoPkg/CryptoPkg.ci.yaml: Add configuration for Ecc check
> > >   EmulatorPkg/EmulatorPkg.ci.yaml: Add configuration for Ecc check
> > >   FatPkg/FatPkg.ci.yaml: Add configuration for Ecc check
> > >   FmpDevicePkg/FmpDevicePkg.ci.yaml: Add configuration for Ecc check
> > >   MdePkg/MdePkg.ci.yaml: Add configuration for Ecc check
> > >   NetworkPkg/NetworkPkg.ci.yaml: Add configuration for Ecc check
> > >   OvmfPkg/OvmfPkg.ci.yaml: Add configuration for Ecc check
> > >   PcAtChipsetPkg/PcAtChipsetPkg.ci.yaml: Add configuration for Ecc check
> > >   SecurityPkg/SecurityPkg.ci.yaml: Add configuration for Ecc check
> > >   ShellPkg/ShellPkg.ci.yaml: Add configuration for Ecc check
> > >   UefiCpuPkg/UefiCpuPkg.ci.yaml: Add configuration for Ecc check
> > >   UnitTestFrameworkPkg: Add configuration for Ecc check in yaml file
> > >
> > >  .pytool/Plugin/EccCheck/EccCheck.py           | 267 ++++++++++++++++++
> > >  .pytool/Plugin/EccCheck/EccCheck_plug_in.yaml |  11 +
> > >  .pytool/Plugin/EccCheck/Readme.md             |  15 +
> > >  ArmVirtPkg/ArmVirtPkg.ci.yaml                 |  11 +
> > >  CryptoPkg/CryptoPkg.ci.yaml                   |  11 +
> > >  EmulatorPkg/EmulatorPkg.ci.yaml               |  11 +
> > >  FatPkg/FatPkg.ci.yaml                         |  11 +
> > >  FmpDevicePkg/FmpDevicePkg.ci.yaml             |  11 +
> > >  MdeModulePkg/MdeModulePkg.ci.yaml             |  11 +
> > >  MdePkg/MdePkg.ci.yaml                         |  11 +
> > >  NetworkPkg/NetworkPkg.ci.yaml                 |  11 +
> > >  OvmfPkg/OvmfPkg.ci.yaml                       |  11 +
> > >  PcAtChipsetPkg/PcAtChipsetPkg.ci.yaml         |  11 +
> > >  SecurityPkg/SecurityPkg.ci.yaml               |  11 +
> > >  ShellPkg/ShellPkg.ci.yaml                     |  11 +
> > >  UefiCpuPkg/UefiCpuPkg.ci.yaml                 |  11 +
> > >  .../UnitTestFrameworkPkg.ci.yaml              |  10 +
> > >  pip-requirements.txt                          |   1 +
> > >  18 files changed, 447 insertions(+)
> > >  create mode 100644 .pytool/Plugin/EccCheck/EccCheck.py
> > >  create mode 100644 .pytool/Plugin/EccCheck/EccCheck_plug_in.yaml
> > >  create mode 100644 .pytool/Plugin/EccCheck/Readme.md
> > >
> > > --
> > > 2.18.0.windows.1
> > >
> > >
> > >
> > >
> > 
> > 
> 

Re: [edk2-devel] [PATCH v7 00/16] Add a plugin to check Ecc issues for edk2 on open ci

[Zhang, Shenglei]

Comments below.

> -----Original Message-----
> From: Leif Lindholm <leif@nuviainc.com>
> Sent: Tuesday, July 7, 2020 7:57 PM
> To: Zhang, Shenglei <shenglei.zhang@intel.com>
> Cc: devel@edk2.groups.io; Feng, Bob C <bob.c.feng@intel.com>; Bret
> Barkelew <Bret.Barkelew@microsoft.com>; Kinney, Michael D
> <michael.d.kinney@intel.com>; Gao, Liming <liming.gao@intel.com>; Sean
> Brogan <sean.brogan@microsoft.com>
> Subject: Re: [edk2-devel] [PATCH v7 00/16] Add a plugin to check Ecc issues
> for edk2 on open ci
> 
> On Tue, Jul 07, 2020 at 02:24:14 +0000, Zhang, Shenglei wrote:
> > Comments below.
> >
> > > -----Original Message-----
> > > From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Leif
> > > Lindholm
> > > Sent: Monday, July 6, 2020 4:55 PM
> > > To: devel@edk2.groups.io; Zhang, Shenglei <shenglei.zhang@intel.com>
> > > Cc: Feng, Bob C <bob.c.feng@intel.com>; Bret Barkelew
> > > <Bret.Barkelew@microsoft.com>; Kinney, Michael D
> > > <michael.d.kinney@intel.com>; Gao, Liming <liming.gao@intel.com>;
> Sean
> > > Brogan <sean.brogan@microsoft.com>
> > > Subject: Re: [edk2-devel] [PATCH v7 00/16] Add a plugin to check Ecc
> issues
> > > for edk2 on open ci
> > >
> > > On Mon, Jul 06, 2020 at 16:48:30 +0800, Zhang, Shenglei wrote:
> > > > REF: https://bugzilla.tianocore.org/show_bug.cgi?id=2606
> > > > As planed we will enable Ecc check for edk2 on open ci. And they are
> > > > ready now. I appreciate receiving feedback and comments if someone
> > > > find errors or false positive issues.
> > > >
> > > > I created a pipline of EccCheck for my forked edk2. Welcome everyone
> to
> > > > create pull request to test the quality of this plugin.
> > > > My forked tree: https://github.com/shenglei10/edk2
> > > >
> > > > And I also created some test cases for ECC plugin. Below are test cases.
> > > > https://github.com/shenglei10/edk2/tree/ECC
> > > > Results can be view in below azure server.
> > > >
> > >
> https://dev.azure.com/shengleizhang/shengleizhang/_build?definitionId=12
> > > &_a=summary
> > > >
> > > > Patches
> > > > 1/16: It's a lib necessary for py3 to run Ecc on azure servers.
> > > >
> > > > 2/16: EccCheck.py is a plugin to report Ecc issues for commits. It can be
> run
> > > >      on azure servers for open ci, or a local virtual environment.
> > > >
> > > > 3/16~16/16: We consider some cases that will report out Ecc issues but
> they
> > > won't
> > > >      be fixed, like submodule and industry standard related things. So we
> > > >      add two configuration fields "Exception" and "IgnoreFiles" for people
> > > >      to use. These patches add configuration in yaml files for Ecc check.
> > > >
> > > > Cc: Bob Feng <bob.c.feng@intel.com>
> > > > Cc: Bret Barkelew <Bret.Barkelew@microsoft.com>
> > > > Cc: Michael D Kinney <michael.d.kinney@intel.com>
> > > > Cc: Liming Gao <liming.gao@intel.com>
> > > > Cc: Sean Brogan <sean.brogan@microsoft.com>
> > > >
> > > > v2: Update 1/17, fix the bug that the script can't hanlde multiple
> commits.
> > > >
> > > > v3: Update 1/17, set the only workalbe workspace is edk2 root directory.
> > > >     Update 2/17, designate the version of antlr4 is 4.7.1.
> > > >     Add 4/17~17/17.
> > > >
> > > > v4. Update 1/17, remove the function EdksetupRebuild(), instead add
> > > >     function SetupEnvironment(). Update variables' format and type
> hints
> > > >     to pass flake8 and mypy.
> > > >
> > > > v5. Conver the former method to plugin solution, to align with
> > > >     other check points on open ci.
> > > >
> > > > v6. The 1/16 patch is missed in v5 series. Now add it in v6.
> > > >
> > > > v7. Fix a bug that Ecc plugin can not be run correctly under Linux OS.
> > >
> > > What is the bug? Where is it fixed?
> >
> > V6 plugin pipeline link:
> >
> https://dev.azure.com/shengleizhang/shengleizhang/_build/results?buildId
> =282&view=logs&j=9b8d87a5-bd93-5358-8ffa-588e312c4376&t=282a77bb-
> da29-5801-6299-2bbe15d5aabb
> > v7 plugin pipeline link:
> >
> https://dev.azure.com/shengleizhang/shengleizhang/_build/results?buildId
> =316&view=logs&j=5e3952f2-9c8a-5394-82c1-a803bcab3ca3&t=7ea9e85e-
> 5c3d-5fcf-865e-8449f16cc745
> >
> > For the same pull request, v6 plugin can't report out the Ecc issues under
> Linux OS.
> 
> Right. So *what* was the bug and *where* (which patch or patches) was
> it fixed? This is information that should (in an abbreviated form) be
> provided in the cover letter.
> 
> "Fix a bug" is data, not information. The same is true for "can not be
> run correctly". And neither tells me which patch I should look at.
> If this has been covered in a specific discussion thread, a link to
> the thread *and* mentioning which patch(es) it affects is sufficient.
> 

OK. Let me re-answer your question.
The bug is that for the same pull request, v6 plugin can't report out the Ecc issues
under Linux OS. It's not difficult to find v6 plugin failed to report out Ecc issues and
v7 plugin managed to do that.
V6 plugin pipeline link:
https://dev.azure.com/shengleizhang/shengleizhang/_build/results?buildId=282&view=logs&j=9b8d87a5-bd93-5358-8ffa-588e312c4376&t=282a77bb-da29-5801-6299-2bbe15d5aabb
v7 plugin pipeline link:
https://dev.azure.com/shengleizhang/shengleizhang/_build/results?buildId=316&view=logs&j=5e3952f2-9c8a-5394-82c1-a803bcab3ca3&t=7ea9e85e-5c3d-5fcf-865e-8449f16cc745

And the update of v7 patch series is 2/16.
It fixed some case sensitive issues and separating symbol problems, like updating 'ecc.log' to 'Ecc.log' and updating '\' to'/'.
Above solution fixed the plugin failed to report out Ecc issues under Linux os.

Thanks,
Shenglei

> Regards,
> 
> Leif
> 
> >
> > Thanks,
> > Shenglei
> >
> > >
> > > /
> > >     Leif
> > >
> > > > Shenglei Zhang (16):
> > > >   pip-requirements.txt: Add Ecc required lib
> > > >   .pytool/Plugin: Add a plugin EccCheck
> > > >   MdeModulePkg/MdeModulePkg.ci.yaml: Add configuration for Ecc
> check
> > > >   ArmVirtPkg/ArmVirtPkg.ci.yaml: Add configuration for Ecc check
> > > >   CryptoPkg/CryptoPkg.ci.yaml: Add configuration for Ecc check
> > > >   EmulatorPkg/EmulatorPkg.ci.yaml: Add configuration for Ecc check
> > > >   FatPkg/FatPkg.ci.yaml: Add configuration for Ecc check
> > > >   FmpDevicePkg/FmpDevicePkg.ci.yaml: Add configuration for Ecc check
> > > >   MdePkg/MdePkg.ci.yaml: Add configuration for Ecc check
> > > >   NetworkPkg/NetworkPkg.ci.yaml: Add configuration for Ecc check
> > > >   OvmfPkg/OvmfPkg.ci.yaml: Add configuration for Ecc check
> > > >   PcAtChipsetPkg/PcAtChipsetPkg.ci.yaml: Add configuration for Ecc
> check
> > > >   SecurityPkg/SecurityPkg.ci.yaml: Add configuration for Ecc check
> > > >   ShellPkg/ShellPkg.ci.yaml: Add configuration for Ecc check
> > > >   UefiCpuPkg/UefiCpuPkg.ci.yaml: Add configuration for Ecc check
> > > >   UnitTestFrameworkPkg: Add configuration for Ecc check in yaml file
> > > >
> > > >  .pytool/Plugin/EccCheck/EccCheck.py           | 267
> ++++++++++++++++++
> > > >  .pytool/Plugin/EccCheck/EccCheck_plug_in.yaml |  11 +
> > > >  .pytool/Plugin/EccCheck/Readme.md             |  15 +
> > > >  ArmVirtPkg/ArmVirtPkg.ci.yaml                 |  11 +
> > > >  CryptoPkg/CryptoPkg.ci.yaml                   |  11 +
> > > >  EmulatorPkg/EmulatorPkg.ci.yaml               |  11 +
> > > >  FatPkg/FatPkg.ci.yaml                         |  11 +
> > > >  FmpDevicePkg/FmpDevicePkg.ci.yaml             |  11 +
> > > >  MdeModulePkg/MdeModulePkg.ci.yaml             |  11 +
> > > >  MdePkg/MdePkg.ci.yaml                         |  11 +
> > > >  NetworkPkg/NetworkPkg.ci.yaml                 |  11 +
> > > >  OvmfPkg/OvmfPkg.ci.yaml                       |  11 +
> > > >  PcAtChipsetPkg/PcAtChipsetPkg.ci.yaml         |  11 +
> > > >  SecurityPkg/SecurityPkg.ci.yaml               |  11 +
> > > >  ShellPkg/ShellPkg.ci.yaml                     |  11 +
> > > >  UefiCpuPkg/UefiCpuPkg.ci.yaml                 |  11 +
> > > >  .../UnitTestFrameworkPkg.ci.yaml              |  10 +
> > > >  pip-requirements.txt                          |   1 +
> > > >  18 files changed, 447 insertions(+)
> > > >  create mode 100644 .pytool/Plugin/EccCheck/EccCheck.py
> > > >  create mode 100644 .pytool/Plugin/EccCheck/EccCheck_plug_in.yaml
> > > >  create mode 100644 .pytool/Plugin/EccCheck/Readme.md
> > > >
> > > > --
> > > > 2.18.0.windows.1
> > > >
> > > >
> > > >
> > > >
> > >
> > > 
> >

Re: [edk2-devel] [PATCH v7 00/16] Add a plugin to check Ecc issues for edk2 on open ci

[Liming Gao]

Shenglei:
  I have no other comment. This patch set is good to me. Reviewed-by: Liming Gao <liming.gao@intel.com>

  If no other comment, I suggest to merge this patch set before end of this week. 

Thanks
Liming
-----Original Message-----
From: Zhang, Shenglei <shenglei.zhang@intel.com> 
Sent: 2020年7月8日 9:57
To: Leif Lindholm <leif@nuviainc.com>
Cc: devel@edk2.groups.io; Feng, Bob C <bob.c.feng@intel.com>; Bret Barkelew <Bret.Barkelew@microsoft.com>; Kinney, Michael D <michael.d.kinney@intel.com>; Gao, Liming <liming.gao@intel.com>; Sean Brogan <sean.brogan@microsoft.com>
Subject: RE: [edk2-devel] [PATCH v7 00/16] Add a plugin to check Ecc issues for edk2 on open ci

Comments below.

> -----Original Message-----
> From: Leif Lindholm <leif@nuviainc.com>
> Sent: Tuesday, July 7, 2020 7:57 PM
> To: Zhang, Shenglei <shenglei.zhang@intel.com>
> Cc: devel@edk2.groups.io; Feng, Bob C <bob.c.feng@intel.com>; Bret 
> Barkelew <Bret.Barkelew@microsoft.com>; Kinney, Michael D 
> <michael.d.kinney@intel.com>; Gao, Liming <liming.gao@intel.com>; Sean 
> Brogan <sean.brogan@microsoft.com>
> Subject: Re: [edk2-devel] [PATCH v7 00/16] Add a plugin to check Ecc 
> issues for edk2 on open ci
> 
> On Tue, Jul 07, 2020 at 02:24:14 +0000, Zhang, Shenglei wrote:
> > Comments below.
> >
> > > -----Original Message-----
> > > From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of 
> > > Leif Lindholm
> > > Sent: Monday, July 6, 2020 4:55 PM
> > > To: devel@edk2.groups.io; Zhang, Shenglei 
> > > <shenglei.zhang@intel.com>
> > > Cc: Feng, Bob C <bob.c.feng@intel.com>; Bret Barkelew 
> > > <Bret.Barkelew@microsoft.com>; Kinney, Michael D 
> > > <michael.d.kinney@intel.com>; Gao, Liming <liming.gao@intel.com>;
> Sean
> > > Brogan <sean.brogan@microsoft.com>
> > > Subject: Re: [edk2-devel] [PATCH v7 00/16] Add a plugin to check 
> > > Ecc
> issues
> > > for edk2 on open ci
> > >
> > > On Mon, Jul 06, 2020 at 16:48:30 +0800, Zhang, Shenglei wrote:
> > > > REF: https://bugzilla.tianocore.org/show_bug.cgi?id=2606
> > > > As planed we will enable Ecc check for edk2 on open ci. And they 
> > > > are ready now. I appreciate receiving feedback and comments if 
> > > > someone find errors or false positive issues.
> > > >
> > > > I created a pipline of EccCheck for my forked edk2. Welcome 
> > > > everyone
> to
> > > > create pull request to test the quality of this plugin.
> > > > My forked tree: https://github.com/shenglei10/edk2
> > > >
> > > > And I also created some test cases for ECC plugin. Below are test cases.
> > > > https://github.com/shenglei10/edk2/tree/ECC
> > > > Results can be view in below azure server.
> > > >
> > >
> https://dev.azure.com/shengleizhang/shengleizhang/_build?definitionId=
> 12
> > > &_a=summary
> > > >
> > > > Patches
> > > > 1/16: It's a lib necessary for py3 to run Ecc on azure servers.
> > > >
> > > > 2/16: EccCheck.py is a plugin to report Ecc issues for commits. 
> > > > It can be
> run
> > > >      on azure servers for open ci, or a local virtual environment.
> > > >
> > > > 3/16~16/16: We consider some cases that will report out Ecc 
> > > > issues but
> they
> > > won't
> > > >      be fixed, like submodule and industry standard related things. So we
> > > >      add two configuration fields "Exception" and "IgnoreFiles" for people
> > > >      to use. These patches add configuration in yaml files for Ecc check.
> > > >
> > > > Cc: Bob Feng <bob.c.feng@intel.com>
> > > > Cc: Bret Barkelew <Bret.Barkelew@microsoft.com>
> > > > Cc: Michael D Kinney <michael.d.kinney@intel.com>
> > > > Cc: Liming Gao <liming.gao@intel.com>
> > > > Cc: Sean Brogan <sean.brogan@microsoft.com>
> > > >
> > > > v2: Update 1/17, fix the bug that the script can't hanlde 
> > > > multiple
> commits.
> > > >
> > > > v3: Update 1/17, set the only workalbe workspace is edk2 root directory.
> > > >     Update 2/17, designate the version of antlr4 is 4.7.1.
> > > >     Add 4/17~17/17.
> > > >
> > > > v4. Update 1/17, remove the function EdksetupRebuild(), instead add
> > > >     function SetupEnvironment(). Update variables' format and 
> > > > type
> hints
> > > >     to pass flake8 and mypy.
> > > >
> > > > v5. Conver the former method to plugin solution, to align with
> > > >     other check points on open ci.
> > > >
> > > > v6. The 1/16 patch is missed in v5 series. Now add it in v6.
> > > >
> > > > v7. Fix a bug that Ecc plugin can not be run correctly under Linux OS.
> > >
> > > What is the bug? Where is it fixed?
> >
> > V6 plugin pipeline link:
> >
> https://dev.azure.com/shengleizhang/shengleizhang/_build/results?build
> Id
> =282&view=logs&j=9b8d87a5-bd93-5358-8ffa-588e312c4376&t=282a77bb-
> da29-5801-6299-2bbe15d5aabb
> > v7 plugin pipeline link:
> >
> https://dev.azure.com/shengleizhang/shengleizhang/_build/results?build
> Id
> =316&view=logs&j=5e3952f2-9c8a-5394-82c1-a803bcab3ca3&t=7ea9e85e-
> 5c3d-5fcf-865e-8449f16cc745
> >
> > For the same pull request, v6 plugin can't report out the Ecc issues 
> > under
> Linux OS.
> 
> Right. So *what* was the bug and *where* (which patch or patches) was 
> it fixed? This is information that should (in an abbreviated form) be 
> provided in the cover letter.
> 
> "Fix a bug" is data, not information. The same is true for "can not be 
> run correctly". And neither tells me which patch I should look at.
> If this has been covered in a specific discussion thread, a link to 
> the thread *and* mentioning which patch(es) it affects is sufficient.
> 

OK. Let me re-answer your question.
The bug is that for the same pull request, v6 plugin can't report out the Ecc issues under Linux OS. It's not difficult to find v6 plugin failed to report out Ecc issues and
v7 plugin managed to do that.
V6 plugin pipeline link:
https://dev.azure.com/shengleizhang/shengleizhang/_build/results?buildId=282&view=logs&j=9b8d87a5-bd93-5358-8ffa-588e312c4376&t=282a77bb-da29-5801-6299-2bbe15d5aabb
v7 plugin pipeline link:
https://dev.azure.com/shengleizhang/shengleizhang/_build/results?buildId=316&view=logs&j=5e3952f2-9c8a-5394-82c1-a803bcab3ca3&t=7ea9e85e-5c3d-5fcf-865e-8449f16cc745

And the update of v7 patch series is 2/16.
It fixed some case sensitive issues and separating symbol problems, like updating 'ecc.log' to 'Ecc.log' and updating '\' to'/'.
Above solution fixed the plugin failed to report out Ecc issues under Linux os.

Thanks,
Shenglei

> Regards,
> 
> Leif
> 
> >
> > Thanks,
> > Shenglei
> >
> > >
> > > /
> > >     Leif
> > >
> > > > Shenglei Zhang (16):
> > > >   pip-requirements.txt: Add Ecc required lib
> > > >   .pytool/Plugin: Add a plugin EccCheck
> > > >   MdeModulePkg/MdeModulePkg.ci.yaml: Add configuration for Ecc
> check
> > > >   ArmVirtPkg/ArmVirtPkg.ci.yaml: Add configuration for Ecc check
> > > >   CryptoPkg/CryptoPkg.ci.yaml: Add configuration for Ecc check
> > > >   EmulatorPkg/EmulatorPkg.ci.yaml: Add configuration for Ecc check
> > > >   FatPkg/FatPkg.ci.yaml: Add configuration for Ecc check
> > > >   FmpDevicePkg/FmpDevicePkg.ci.yaml: Add configuration for Ecc check
> > > >   MdePkg/MdePkg.ci.yaml: Add configuration for Ecc check
> > > >   NetworkPkg/NetworkPkg.ci.yaml: Add configuration for Ecc check
> > > >   OvmfPkg/OvmfPkg.ci.yaml: Add configuration for Ecc check
> > > >   PcAtChipsetPkg/PcAtChipsetPkg.ci.yaml: Add configuration for 
> > > > Ecc
> check
> > > >   SecurityPkg/SecurityPkg.ci.yaml: Add configuration for Ecc check
> > > >   ShellPkg/ShellPkg.ci.yaml: Add configuration for Ecc check
> > > >   UefiCpuPkg/UefiCpuPkg.ci.yaml: Add configuration for Ecc check
> > > >   UnitTestFrameworkPkg: Add configuration for Ecc check in yaml 
> > > > file
> > > >
> > > >  .pytool/Plugin/EccCheck/EccCheck.py           | 267
> ++++++++++++++++++
> > > >  .pytool/Plugin/EccCheck/EccCheck_plug_in.yaml |  11 +
> > > >  .pytool/Plugin/EccCheck/Readme.md             |  15 +
> > > >  ArmVirtPkg/ArmVirtPkg.ci.yaml                 |  11 +
> > > >  CryptoPkg/CryptoPkg.ci.yaml                   |  11 +
> > > >  EmulatorPkg/EmulatorPkg.ci.yaml               |  11 +
> > > >  FatPkg/FatPkg.ci.yaml                         |  11 +
> > > >  FmpDevicePkg/FmpDevicePkg.ci.yaml             |  11 +
> > > >  MdeModulePkg/MdeModulePkg.ci.yaml             |  11 +
> > > >  MdePkg/MdePkg.ci.yaml                         |  11 +
> > > >  NetworkPkg/NetworkPkg.ci.yaml                 |  11 +
> > > >  OvmfPkg/OvmfPkg.ci.yaml                       |  11 +
> > > >  PcAtChipsetPkg/PcAtChipsetPkg.ci.yaml         |  11 +
> > > >  SecurityPkg/SecurityPkg.ci.yaml               |  11 +
> > > >  ShellPkg/ShellPkg.ci.yaml                     |  11 +
> > > >  UefiCpuPkg/UefiCpuPkg.ci.yaml                 |  11 +
> > > >  .../UnitTestFrameworkPkg.ci.yaml              |  10 +
> > > >  pip-requirements.txt                          |   1 +
> > > >  18 files changed, 447 insertions(+)  create mode 100644 
> > > > .pytool/Plugin/EccCheck/EccCheck.py
> > > >  create mode 100644 
> > > > .pytool/Plugin/EccCheck/EccCheck_plug_in.yaml
> > > >  create mode 100644 .pytool/Plugin/EccCheck/Readme.md
> > > >
> > > > --
> > > > 2.18.0.windows.1
> > > >
> > > >
> > > >
> > > >
> > >
> > > 
> >

Re: [edk2-devel] [PATCH v7 03/16] MdeModulePkg/MdeModulePkg.ci.yaml: Add configuration for Ecc check

[Wu, Hao A]

> -----Original Message-----
> From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Zhang,
> Shenglei
> Sent: Monday, July 6, 2020 4:49 PM
> To: devel@edk2.groups.io
> Cc: Wang, Jian J <jian.j.wang@intel.com>; Wu, Hao A <hao.a.wu@intel.com>
> Subject: [edk2-devel] [PATCH v7 03/16]
> MdeModulePkg/MdeModulePkg.ci.yaml: Add configuration for Ecc check
> 
> Add configuration ExceptionList and IgnoreFiles for package config files. So users
> can rely on this to ignore some Ecc issues.
> 
> Cc: Jian J Wang <jian.j.wang@intel.com>
> Cc: Hao A Wu <hao.a.wu@intel.com>
> Signed-off-by: Shenglei Zhang <shenglei.zhang@intel.com>
> ---
>  MdeModulePkg/MdeModulePkg.ci.yaml | 11 +++++++++++
>  1 file changed, 11 insertions(+)
> 
> diff --git a/MdeModulePkg/MdeModulePkg.ci.yaml
> b/MdeModulePkg/MdeModulePkg.ci.yaml
> index 1cfc1328390e..0ed929855417 100644
> --- a/MdeModulePkg/MdeModulePkg.ci.yaml
> +++ b/MdeModulePkg/MdeModulePkg.ci.yaml
> @@ -2,9 +2,20 @@
>  # CI configuration for MdeModulePkg
>  #
>  # Copyright (c) Microsoft Corporation
> +# Copyright (c) 2020, Intel Corporation. All rights reserved.<BR>
>  # SPDX-License-Identifier: BSD-2-Clause-Patent  ##  {
> +    "EccCheck": {
> +        ## Exception sample looks like below:
> +        ## "ExceptionList": [
> +        ##     "<ErrorID>", "<KeyWord>"
> +        ## ]
> +        "ExceptionList": [
> +        ],
> +        "IgnoreFiles": [
> +        ]
> +    },


Acked-by: Hao A Wu <hao.a.wu@intel.com>

Best Regards,
Hao Wu


>      ## options defined ci/Plugin/CompilerPlugin
>      "CompilerPlugin": {
>          "DscPath": "MdeModulePkg.dsc"
> --
> 2.18.0.windows.1
> 
> 
>