From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail02.groups.io (mail02.groups.io [66.175.222.108]) by spool.mail.gandi.net (Postfix) with ESMTPS id D0342AC0144 for ; Wed, 18 Oct 2023 01:05:24 +0000 (UTC) DKIM-Signature: a=rsa-sha256; bh=ghFUMniSFKseGc5hyeSHI5FUv1q0PE+PGJ8OFIo6I9o=; c=relaxed/simple; d=groups.io; h=DKIM-Filter:From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References:MIME-Version:Precedence:List-Subscribe:List-Help:Sender:List-Id:Mailing-List:Delivered-To:Reply-To:List-Unsubscribe-Post:List-Unsubscribe:Content-Transfer-Encoding; s=20140610; t=1697591123; v=1; b=epw3mMtqbC1cvhzTyKN8HtOyv1lrOGndnyV8SmXiH5s8GN45dNgOHPXrzge5b/eV2aMBBijw eaQMKiVs4trsDNovry9sVIil20VddoYp+T3QaFLAehCmrRNK2Mxonv/OrGzlMK+D91/2Ryz9Tp5 U6/Eb7gXf/unLmk5sgQIBTOE= X-Received: by 127.0.0.2 with SMTP id JI5bYY7687511x5e9pbhLcEi; Tue, 17 Oct 2023 18:05:23 -0700 X-Received: from linux.microsoft.com (linux.microsoft.com [13.77.154.182]) by mx.groups.io with SMTP id smtpd.web10.272336.1697591123012868554 for ; Tue, 17 Oct 2023 18:05:23 -0700 X-Received: from localhost.localdomain (unknown [47.201.241.95]) by linux.microsoft.com (Postfix) with ESMTPSA id 3B01E20B74C2; Tue, 17 Oct 2023 18:05:22 -0700 (PDT) DKIM-Filter: OpenDKIM Filter v2.11.0 linux.microsoft.com 3B01E20B74C2 From: "Michael Kubacki" To: devel@edk2.groups.io Cc: Sean Brogan , Michael D Kinney Subject: [edk2-devel] [PATCH v3 5/7] .github/workflows/codeql.yml: Add CodeQL workflow Date: Tue, 17 Oct 2023 21:04:43 -0400 Message-ID: <20231018010445.528-6-mikuback@linux.microsoft.com> In-Reply-To: <20231018010445.528-1-mikuback@linux.microsoft.com> References: <20231018010445.528-1-mikuback@linux.microsoft.com> MIME-Version: 1.0 Precedence: Bulk List-Subscribe: List-Help: Sender: devel@edk2.groups.io List-Id: Mailing-List: list devel@edk2.groups.io; contact devel+owner@edk2.groups.io Reply-To: devel@edk2.groups.io,mikuback@linux.microsoft.com List-Unsubscribe-Post: List-Unsubscribe=One-Click List-Unsubscribe: X-Gm-Message-State: oxD73Xc5KSgQsC6kW6vlzSZLx7686176AA= Content-Transfer-Encoding: quoted-printable X-GND-Status: LEGIT Authentication-Results: spool.mail.gandi.net; dkim=pass header.d=groups.io header.s=20140610 header.b=epw3mMtq; dmarc=fail reason="SPF not aligned (relaxed), DKIM not aligned (relaxed)" header.from=linux.microsoft.com (policy=none); spf=pass (spool.mail.gandi.net: domain of bounce@groups.io designates 66.175.222.108 as permitted sender) smtp.mailfrom=bounce@groups.io From: Michael Kubacki Adds a workflow to run CodeQL against all packages built in .pytool/CISettings.py. The following is done: 1. Determine which packages to build against. Those that support are managed by .pytool/CISettings.py will be selected. For each package: 2. Determine how to interact with the package. Such as whether `stuart_ci_setup` or `stuart_setup` should be used. 3. Perform supported Stuart steps for setup and update. 4. Discover the CodeQL plugin directory in the repo. 5. Attempt to load the CodeQL CLI specific to the host OS from a GitHub cache. 6. Perform the build. 7. Clean up some files after build to improve robustness. 8. Upload the CodeQL results (generated SARIF file) to GitHub Code Scanning. The results will be associated with the trigger of the workflow. After each step that can upload logs such as the setup, update, and build steps the logs are uploaded as an artifact to the workflow run. This allows easy debugging in case there's an error in the step. The SARIF file is also uploaded to the workflow run so it can be downloaded and analyzed. Cc: Sean Brogan Cc: Michael D Kinney Signed-off-by: Michael Kubacki --- .github/workflows/codeql.yml | 338 ++++++++++++++++++++ 1 file changed, 338 insertions(+) diff --git a/.github/workflows/codeql.yml b/.github/workflows/codeql.yml new file mode 100644 index 000000000000..72ece9dcb446 --- /dev/null +++ b/.github/workflows/codeql.yml @@ -0,0 +1,338 @@ +# This workflow runs CodeQL against the repository. +# +# Results are uploaded to GitHub Code Scanning. +# +# Due to a known issue with the CodeQL extractor when building the edk2 +# codebase on Linux systems, only Windows agents are used for build with +# the VS toolchain. +# +# Copyright (c) Microsoft Corporation. +# SPDX-License-Identifier: BSD-2-Clause-Patent + +name: "CodeQL" + +on: + push: + branches: + - master + pull_request: + branches: + - master + paths-ignore: + - '!**.c' + - '!**.h' + +jobs: + analyze: + name: Analyze + runs-on: windows-2019 + permissions: + actions: read + contents: read + security-events: write + + strategy: + fail-fast: false + matrix: + include: + - Package: "ArmPkg" + ArchList: "IA32,X64" + - Package: "CryptoPkg" + ArchList: "IA32" + - Package: "CryptoPkg" + ArchList: "X64" + - Package: "DynamicTablesPkg" + ArchList: "IA32,X64" + - Package: "FatPkg" + ArchList: "IA32,X64" + - Package: "FmpDevicePkg" + ArchList: "IA32,X64" + - Package: "IntelFsp2Pkg" + ArchList: "IA32,X64" + - Package: "IntelFsp2WrapperPkg" + ArchList: "IA32,X64" + - Package: "MdeModulePkg" + ArchList: "IA32" + - Package: "MdeModulePkg" + ArchList: "X64" + - Package: "MdePkg" + ArchList: "IA32,X64" + - Package: "PcAtChipsetPkg" + ArchList: "IA32,X64" + - Package: "PrmPkg" + ArchList: "IA32,X64" + - Package: "SecurityPkg" + ArchList: "IA32,X64" + - Package: "ShellPkg" + ArchList: "IA32,X64" + - Package: "SourceLevelDebugPkg" + ArchList: "IA32,X64" + - Package: "StandaloneMmPkg" + ArchList: "IA32,X64" + - Package: "UefiCpuPkg" + ArchList: "IA32,X64" + - Package: "UnitTestFrameworkPkg" + ArchList: "IA32,X64" + + steps: + - name: Checkout repository + uses: actions/checkout@v4 + + - name: Install Python + uses: actions/setup-python@v4 + with: + python-version: '3.11' + cache: 'pip' + cache-dependency-path: 'pip-requirements.txt' + + - name: Use Git Long Paths on Windows + if: runner.os =3D=3D 'Windows' + shell: pwsh + run: | + git config --system core.longpaths true + + - name: Install/Upgrade pip Modules + run: pip install -r pip-requirements.txt --upgrade requests + + - name: Determine CI Settings File Supported Operations + id: get_ci_file_operations + shell: python + run: | + import importlib + import os + import sys + from pathlib import Path + from edk2toolext.invocables.edk2_ci_setup import CiSetupSettings= Manager + from edk2toolext.invocables.edk2_setup import SetupSettingsManag= er + + # Find the repo CI Settings file + ci_settings_file =3D list(Path(os.environ['GITHUB_WORKSPACE']).r= glob('.pytool/CISettings.py')) + + # Note: At this point, submodules have not been pulled, only one= CI Settings file should exist + if len(ci_settings_file) !=3D 1 or not ci_settings_file[0].is_fi= le(): + print("::error title=3DWorkspace Error!::Failed to find CI S= ettings file!") + sys.exit(1) + + ci_settings_file =3D ci_settings_file[0] + + # Try Finding the Settings class in the file + module_name =3D 'ci_settings' + + spec =3D importlib.util.spec_from_file_location(module_name, ci_= settings_file) + module =3D importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + + try: + settings =3D getattr(module, 'Settings') + except AttributeError: + print("::error title=3DWorkspace Error!::Failed to find Sett= ings class in CI Settings file!") + sys.exit(1) + + # Determine Which Operations Are Supported by the Settings Class + ci_setup_supported =3D issubclass(settings, CiSetupSettingsManag= er) + setup_supported =3D issubclass(settings, SetupSettingsManager) + + with open(os.environ['GITHUB_OUTPUT'], 'a') as fh: + print(f'ci_setup_supported=3D{str(ci_setup_supported).lower(= )}', file=3Dfh) + print(f'setup_supported=3D{str(setup_supported).lower()}', f= ile=3Dfh) + + - name: Setup + if: steps.get_ci_file_operations.outputs.setup_supported =3D=3D 't= rue' + run: stuart_setup -c .pytool/CISettings.py -t DEBUG -a ${{ matrix.= ArchList }} TOOL_CHAIN_TAG=3DVS2019 + + - name: Upload Setup Log As An Artifact + uses: actions/upload-artifact@v3 + if: (success() || failure()) && steps.get_ci_file_operations.outpu= ts.setup_supported =3D=3D 'true' + with: + name: ${{ matrix.Package }}-Logs + path: | + **/SETUPLOG.txt + retention-days: 7 + if-no-files-found: ignore + + - name: CI Setup + if: steps.get_ci_file_operations.outputs.ci_setup_supported =3D=3D= 'true' + run: stuart_ci_setup -c .pytool/CISettings.py -t DEBUG -a ${{ matr= ix.ArchList }} TOOL_CHAIN_TAG=3DVS2019 + + - name: Upload CI Setup Log As An Artifact + uses: actions/upload-artifact@v3 + if: (success() || failure()) && steps.get_ci_file_operations.outpu= ts.ci_setup_supported =3D=3D 'true' + with: + name: ${{ matrix.Package }}-Logs + path: | + **/CISETUP.txt + retention-days: 7 + if-no-files-found: ignore + + - name: Update + run: stuart_update -c .pytool/CISettings.py -t DEBUG -a ${{ matrix= .ArchList }} TOOL_CHAIN_TAG=3DVS2019 + + - name: Upload Update Log As An Artifact + uses: actions/upload-artifact@v3 + if: success() || failure() + with: + name: ${{ matrix.Package }}-Logs + path: | + **/UPDATE_LOG.txt + retention-days: 7 + if-no-files-found: ignore + + - name: Build Tools From Source + run: python BaseTools/Edk2ToolsBuild.py -t VS2019 + + - name: Find CodeQL Plugin Directory + id: find_dir + shell: python + run: | + import os + import sys + from pathlib import Path + + # Find the plugin directory that contains the CodeQL plugin + plugin_dir =3D list(Path(os.environ['GITHUB_WORKSPACE']).rglob('= BaseTools/Plugin/CodeQL')) + + # This should only be found once + if len(plugin_dir) =3D=3D 1: + plugin_dir =3D str(plugin_dir[0]) + + with open(os.environ['GITHUB_OUTPUT'], 'a') as fh: + print(f'codeql_plugin_dir=3D{plugin_dir}', file=3Dfh) + else: + print("::error title=3DWorkspace Error!::Failed to find Code= QL plugin directory!") + sys.exit(1) + + - name: Get CodeQL CLI Cache Data + id: cache_key_gen + env: + CODEQL_PLUGIN_DIR: ${{ steps.find_dir.outputs.codeql_plugin_dir = }} + shell: python + run: | + import os + import yaml + + codeql_cli_ext_dep_name =3D 'codeqlcli_windows_ext_dep' + codeql_plugin_file =3D os.path.join(os.environ['CODEQL_PLUGIN_DI= R'], codeql_cli_ext_dep_name + '.yaml') + + with open (codeql_plugin_file) as pf: + codeql_cli_ext_dep =3D yaml.safe_load(pf) + + cache_key_name =3D codeql_cli_ext_dep['name'] + cache_key_version =3D codeql_cli_ext_dep['version'] + cache_key =3D f'{cache_key_name}-{cache_key_version}' + + codeql_plugin_cli_ext_dep_dir =3D os.path.join(os.environ['C= ODEQL_PLUGIN_DIR'], codeql_cli_ext_dep['name'].strip() + '_extdep') + + with open(os.environ['GITHUB_OUTPUT'], 'a') as fh: + print(f'codeql_cli_cache_key=3D{cache_key}', file=3Dfh) + print(f'codeql_cli_ext_dep_dir=3D{codeql_plugin_cli_ext_= dep_dir}', file=3Dfh) + + - name: Attempt to Load CodeQL CLI From Cache + id: codeqlcli_cache + uses: actions/cache@v3 + with: + path: ${{ steps.cache_key_gen.outputs.codeql_cli_ext_dep_dir }} + key: ${{ steps.cache_key_gen.outputs.codeql_cli_cache_key }} + + - name: Download CodeQL CLI + if: steps.codeqlcli_cache.outputs.cache-hit !=3D 'true' + run: stuart_update -c .pytool/CISettings.py -t DEBUG -a ${{ matrix= .ArchList }} TOOL_CHAIN_TAG=3DVS2019 --codeql + + - name: Remove CI Plugins Irrelevant to CodeQL + shell: python + env: + CODEQL_PLUGIN_DIR: ${{ steps.find_dir.outputs.codeql_plugin_dir = }} + run: | + import os + import shutil + from pathlib import Path + + # Only these two plugins are needed for CodeQL + plugins_to_keep =3D ['CompilerPlugin'] + + plugin_dir =3D Path('.pytool/Plugin').absolute() + if plugin_dir.is_dir(): + for dir in plugin_dir.iterdir(): + if str(dir.stem) not in plugins_to_keep: + shutil.rmtree(str(dir.absolute()), ignore_errors=3DT= rue) + + - name: CI Build + env: + STUART_CODEQL_PATH: ${{ steps.cache_key_gen.outputs.codeql_cli_e= xt_dep_dir }} + run: stuart_ci_build -c .pytool/CISettings.py -t DEBUG -p ${{ matr= ix.Package }} -a ${{ matrix.ArchList }} TOOL_CHAIN_TAG=3DVS2019 --codeql + + - name: Build Cleanup + id: build_cleanup + shell: python + run: | + import os + import shutil + from pathlib import Path + + dirs_to_delete =3D ['ia32', 'x64', 'arm', 'aarch64'] + + def delete_dirs(path: Path): + if path.exists() and path.is_dir(): + if path.name.lower() in dirs_to_delete: + print(f'Removed {str(path)}') + shutil.rmtree(path) + return + + for child_dir in path.iterdir(): + delete_dirs(child_dir) + + build_path =3D Path(os.environ['GITHUB_WORKSPACE'], 'Build') + delete_dirs(build_path) + + - name: Upload Build Logs As An Artifact + uses: actions/upload-artifact@v3 + if: success() || failure() + with: + name: ${{ matrix.Package }}-Logs + path: | + **/BUILD_REPORT.TXT + **/OVERRIDELOG.TXT + **/BUILDLOG_*.md + **/BUILDLOG_*.txt + **/CI_*.md + **/CI_*.txt + retention-days: 7 + if-no-files-found: ignore + + - name: Prepare Env Data for CodeQL Upload + id: env_data + env: + PACKAGE_NAME: ${{ matrix.Package }} + shell: python + run: | + import os + + package =3D os.environ['PACKAGE_NAME'].strip().lower() + directory_name =3D 'codeql-analysis-' + package + '-debug' + file_name =3D 'codeql-db-' + package + '-debug-0.sarif' + sarif_path =3D os.path.join('Build', directory_name, file_name) + + with open(os.environ['GITHUB_OUTPUT'], 'a') as fh: + if os.path.isfile(sarif_path): + print(f'upload_sarif_file=3Dtrue', file=3Dfh) + print(f'sarif_file_path=3D{sarif_path}', file=3Dfh) + else: + print(f'upload_sarif_file=3Dfalse', file=3Dfh) + + - name: Upload CodeQL Results (SARIF) As An Artifact + uses: actions/upload-artifact@v3 + if: steps.env_data.outputs.upload_sarif_file =3D=3D 'true' + with: + name: ${{ matrix.Package }}-CodeQL-SARIF + path: ${{ steps.env_data.outputs.sarif_file_path }} + retention-days: 14 + if-no-files-found: warn + + - name: Upload CodeQL Results (SARIF) To GitHub Code Scanning + uses: github/codeql-action/upload-sarif@v2 + if: steps.env_data.outputs.upload_sarif_file =3D=3D 'true' + with: + # Path to SARIF file relative to the root of the repository. + sarif_file: ${{ steps.env_data.outputs.sarif_file_path }} + # Optional category for the results. Used to differentiate multi= ple results for one commit. + # Each package is a separate category. + category: ${{ matrix.Package }} --=20 2.42.0.windows.2 -=-=-=-=-=-=-=-=-=-=-=- Groups.io Links: You receive all messages sent to this group. View/Reply Online (#109699): https://edk2.groups.io/g/devel/message/109699 Mute This Topic: https://groups.io/mt/102031061/7686176 Group Owner: devel+owner@edk2.groups.io Unsubscribe: https://edk2.groups.io/g/devel/unsub [rebecca@openfw.io] -=-=-=-=-=-=-=-=-=-=-=-