[edk2-wiki][PATCH v2 1/1] Add Uncrustify formatting overview

["Michael Kubacki" <michael.kubacki@outlook.com>]

From: Michael Kubacki <michael.kubacki@microsoft.com>

This wiki page describes how Uncrustify is used in EDK II and how
to install and use Uncrustify during the EDK II code development
process.

Cc: Andrew Fish <afish@apple.com>
Cc: Leif Lindholm <leif@nuviainc.com>
Cc: Michael D Kinney <michael.d.kinney@intel.com>
Cc: Liming Gao <gaoliming@byosoft.com.cn>
Signed-off-by: Michael Kubacki <michael.kubacki@microsoft.com>
---

Notes:
    v2 changes:
    
    - Add UncrustifyCheck CI plugin readme and config file links
    - Update manual usage instructions to include Linux instructions
    - Clean up manual usage instructions
    - Minor formatting changes

 EDK-II-Code-Formatting.md | 247 ++++++++++++++++++++
 1 file changed, 247 insertions(+)

diff --git a/EDK-II-Code-Formatting.md b/EDK-II-Code-Formatting.md
new file mode 100644
index 000000000000..e671663e5dcd
--- /dev/null
+++ b/EDK-II-Code-Formatting.md
@@ -0,0 +1,247 @@
+# EDK II Code Formatting
+
+To better achieve the goals of the [EDK II C Coding Standards Specification](https://edk2-docs.gitbook.io/edk-ii-c-coding-standards-specification/),
+EDK II code formatting is automated using a source code beautifier called Uncrustify. Uncrustify is compatible with
+C/C++ in addition to other languages. In EDK II, it is used to format C language source code.
+
+## Uncrustify in the edk2 Repository
+
+The Uncrustify collateral in the edk2 repository contains all of the resources needed to get the Uncrustify application
+and run it with the same settings as other developers.
+
+> Uncrustify is automatically run against code submitted in edk2 as a continuous integration (CI) plugin called
+"UncrustifyCheck". The plugin is available in the following directory:
+[.pytool/Plugin/UncrustifyCheck](https://github.com/tianocore/edk2/tree/master/.pytool/Plugin/UncrustifyCheck)
+
+The UncrustifyCheck plugin in edk2 contains the following files used to check code for compliance to the coding
+standard:
+
+* **default_file_header.txt** - A text file containing a template that is placed at the top of files missing a file
+  header.
+* **default_function_header.txt** - A text file containing a template that is placed above functions that are missing
+  a function header.
+* **Readme.md** - A file that contains details about how the plugin works and how to use it.
+* **uncrustify_ext_dep.yml** - An "external dependency" file that is used to get the current version of the Uncrustify
+  application used by the plugin. This file contains the NuGet feed URL and the version currently used.
+* **uncrustify_plug_in.yaml** - A file that contains information to describe the plugin to the build environment.
+* **uncrustify.cfg** - A file used by the Uncrustify application to control how it formats code. If you want to tweak
+  particular formatting details, this is the place to start.
+* **UncrustifyCheck.py** - The actual Python file that is the CI plugin. Like all CI plugins, this plugin can be run
+  in local CI and server CI.
+
+## EDK II Uncrustify Fork
+
+Due to nuances in the way EDK II formats code, some changes were made to the upstream Uncrustify application. These
+were changes that could not be made purely through the Uncrustify configuration file. For more details about the fork,
+please visit that project overview in the link below.
+
+* Uncrustify upstream repository: https://github.com/uncrustify/uncrustify
+* Uncrustify EDK II fork repository (in Project Mu): https://dev.azure.com/projectmu/Uncrustify
+
+## Developer Workflow
+
+Developers must install Uncrustify and run the application against their code before sending patch review emails or
+submitting pull requests. Pull requests run against EDK II CI which includes the UncrustifyCheck CI plugin.
+
+Fortunately, Uncrustify can be installed quickly, you can format your code quickly locally, and you can verify the
+code against the UncrustifyCheck CI plugin before sending it to others.
+
+The recommended flow is:
+
+1. Clone the edk2 source code repository
+2. Use the `stuart*` commands to pull the Uncrustify application into the edk2 workspace
+3. Set up the ability to run Uncrustify locally (for example, using the Visual Studio Code Uncrustify plugin)
+4. Make and test code changes
+5. Run EDK II CI locally to verify UncrustifyCheck passes
+6. Send the code patch to the EDK II mailing list
+
+## Installing Uncrustify
+
+Uncrustify is a portable executable that is built in the [EDK II Uncrustify fork repository](https://dev.azure.com/projectmu/Uncrustify)
+and ultimately published into a NuGet feed in that fork project.
+
+### Recommended Installation: In edk2 repository
+
+It is strongly recommended to follow this flow. It sets up the workspace to work with local CI and automatically gets
+the current supported version of the application.
+
+The Uncrustify tool is installed automatically when the Pytools environment is used and the `stuart*` commands are run
+to complete environment setup. Review the edk2 [.pytool/Readme.md](https://github.com/tianocore/edk2/tree/master/.pytool#running-ci-locally)
+file for details on `stuart` and this overall flow.
+
+After running the `stuart_update` command, the Uncrustify application content should be brought down into
+`.pytool\Plugin\UncrustifyCheck\mu-uncrustify-release_extdep` in your edk2 workspace. The contents of this directory
+now represent the contents of the NuGet package and it should contain a Linux and Windows executable of the
+application.
+
+### Manual Installation: Download from the fork project
+
+The release pipeline in the EDK II Uncrustify fork project contains the build information for each release. Each build
+in this pipeline represents a release. By going to a specific build, the details mapping the build to source code
+(such as the branch and commit) are present.
+
+The build content is published as a NuGet package to a NuGet feed. This is the same feed, the recommended installation
+instructions automatically pull from. The NuGet feed is available in the ["Artifacts"](https://dev.azure.com/projectmu/Uncrustify/_packaging?_a=feed&feed=mu_uncrustify)
+section of the fork project. If you hover/click on a specific package entry (e.g. "mu-uncrustify-release"), a set of
+three ellipsis will appear. Click the ellipsis and a context menu will appear. The NuGet package can be downloaded
+by clicking "Download \<x.y.z\>".
+
+Once downloaded, the `.nupkg` file can be treated as a zip file. If the file is opened as a zip file, the executable
+can be found in the `mu-uncrustify-<debug/release>` directory.
+
+## How to Run Uncrustify
+
+Once Uncrustify is installed, you can run the application in a number of ways. In all cases, you should be using the
+Uncrustify application built from the [Uncrustify EDK II fork](https://dev.azure.com/projectmu/Uncrustify) and the
+[Uncrustify configuration file](https://github.com/tianocore/edk2/blob/master/.pytool/Plugin/UncrustifyCheck/uncrustify.cfg)
+currently checked into edk2.
+
+### Recommended Usage: Visual Studio (VS) Code Plugin
+
+The Visual Studio Code plugin provides a way to seamlessly run Uncrustify against code at anytime in the editor and
+the configuration details are set once in the editor configuration file.
+
+1. Install the Uncrustify VS Code extension:
+   > Name: Uncrustify
+     Id: zachflower.uncrustify
+     Description: Code format using uncrustify
+     Publisher: Zachary Flower
+     VS Marketplace Link: <https://marketplace.visualstudio.com/items?itemName=zachflower.uncrustify>
+
+2. Configure the Uncrustify plugin for your local setup by adding the following to your VS Code settings.json file:
+   (Windows example)
+
+   ```json
+   "uncrustify.configPath.windows": "path_to_your_config_file",
+   "uncrustify.executablePath.windows": "path_to_your_uncrustify_executable"
+   ```
+
+   > Windows Example:
+   >
+   >```json
+   >"uncrustify.configPath.windows": "D:/src/edk2/.pytool/Plugin/UncrustifyCheck/uncrustify.cfg",
+   >"uncrustify.executablePath.windows": "D:/src/edk2/.pytool/Plugin/UncrustifyCheck/mu-uncrustify-release_extdep/Windows-x86/uncrustify.exe"
+   >```
+
+3. Open a C source code file, _`Ctrl+Shift+P` -> Format Document With... -> Configure Default Formatter -> Uncrustify_
+
+4. Then, _`Ctrl+Shift+P` -> Format Document_ any time you would like to format your source code file with Uncrustify
+
+### Manual Usage: Run in a Terminal
+
+These instructions are written for Windows 10. These activities could be further automated into a high-level script
+but that has not been done yet.
+
+1. Generate a list of the files to run against. This example generates a recursive list of all .c and .h files.
+
+   * It is recommended to run this in cleanly cloned edk2 repo without submodules to prevent submodule files
+     (such as Brotli files in MdeModulePkg) from getting included in the file list (if you are running against all
+     files). Including all files will significantly increase the amount of time Uncrustify takes to run.
+
+   * Sample Powershell command to recursively write all .c and .h files in a given package to a text file:
+
+     ```powershell
+     Get-ChildItem -Path .\MdePkg\* -Include *.c, *.h -Recurse -Force | %{$_.fullname} | Out-File
+     -FilePath .\MdePkgFiles.txt -Encoding utf8
+     ```
+
+   > **WARNING** Powershell will put the UTF-8 BOM at the beginning of the output file. Uncrustify does not recognize the BOM
+   and it should be removed before passing the file as input to Uncrustify. If it is not removed, Uncrustify will
+   not read the first file path in the text file properly which will cause the file to not be formatted.
+
+2. Run Uncrustify using the generated text file as input
+
+   The following assume you move the EDK II Uncrustify configuration file to the directory .uncrustify in your edk2
+   workspace.
+
+   ```shell
+   uncrustify.exe -c .\.pytool\Plugin\UncrustifyCheck\uncrustify.cfg -F MdePkgFiles.txt --replace --no-backup --if-changed
+   ```
+
+   > *Note:* When testing a configuration change, it is sometimes useful to run Uncrustify against a particular file
+     and check the debug output to understand what rule was applied and why it was applied. The command shows an
+     example of how to run the configuration file `uncrustify.cfg` against the source file `VariableSmm.c` where the
+     file is forced to be treated as C, the debug output is written to `uncrustify_debug.txt` and the log severity level
+     is set to "all".
+
+   ```shell
+   uncrustify.exe -c .\.pytool\Plugin\UncrustifyCheck\uncrustify.cfg -f .\MdeModulePkg\Universal\Variable\RuntimeDxe\VariableSmm.c -o output.c -l C -p uncrustify_debug.txt -L A 2>verbose_debug.txt
+   ```
+
+Uncrustify will update the source files in-place (with the commands given). This allows you to diff the results with
+git. From here, you can iteratively tweak the configuration file and check the results until your satisfied with the
+outcome.
+
+## Uncrustify in CI
+
+The UncrustifyCheck CI plugin that will verify formatting on the server can be run locally. It is recommended to run
+local CI to verify the patch submission will pass CI on the server.
+
+This can be done using the `stuart_ci_build` command.
+
+> Tip: To quickly only run UncrustifyCheck, remove the other plugin directories from your local `.pytool` directory
+  and, of course, add them back when you're done.
+
+Here's an example of running UncrustifyCheck against MdeModulePkg from the root of an edk2 workspace:
+
+`stuart_ci_build -c .pytool/CISettings.py -p MdeModulePkg`
+
+If a file has a formatting error, it will be noted in the output from `stuart_ci_build`. This is visible in the
+terminal output in local CI and the build output log in server CI.
+
+Read the [UncrustifyCheck Readme.md](https://github.com/tianocore/edk2/blob/master/.pytool/Plugin/UncrustifyCheck/Readme.md)
+to understand more about how the plugin can be configured for CI.
+
+## Extra Reading: Tracing History Across the Uncrustify Changes
+
+It might be helpful to view the entire history rewritten with Uncrustify formatting on every commit. For example, an
+alternate version of the edk2 repository that serves as "documentation" with the entire history re-written.
+
+A tool called git-filter-repo can be used to perform this transformation and runs in a reasonable period of time
+(a few hours):
+
+* <https://github.com/newren/git-filter-repo>
+* <https://github.com/newren/git-filter-repo/blob/main/contrib/filter-repo-demos/lint-history>
+
+The following steps can be used to perform this transformation. This is the Windows process. A Linux process will be
+added in the future.
+
+  > **WARNING**  This operation modifies (rewrites) all the commits in the local  copy of the repo.  Do not perform
+                 these steps on a local repo you are using for active development.
+
+1. Clone edk2 into a new directory (see **WARNING**)
+
+   ```shell
+   git clone https://github.com/tianocore/edk2.git edk2-uncrustified
+   cd edk2-uncrustified
+   ```
+
+2. Setup python virtual env, install pytools, and run stuart commands to setup build environment which includes
+   installing uncrustify tools. See [Running CI Locally](https://github.com/tianocore/edk2/tree/master/.pytool#running-ci-locally).
+
+3. Make a backup copy of the plugin UncrustifyCheck outside WORKSPACE. (e.g. C:\Temp\UncrustifyCheck) so the
+   Uncrustify executable and EDK II specific Uncrustify configuration file are available when working with a branch
+   that does not have those tools in its scope.
+
+   ```shell
+   xcopy .pytool\Plugin\UncrustifyCheck C:\Temp\UncrustifyCheck
+   ```
+
+4. Use lint-history.py from git-filter-repo examples
+
+   * [git-filter-repo](https://github.com/newren/git-filter-repo)
+   * [lint-history](https://github.com/newren/git-filter-repo/blob/main/contrib/filter-repo-demos/lint-history)
+
+Line #127 - Add try except around subprocess.check_call() with except being pass. This is required because there are a
+few commits of C files in the edk2 repo that have incorrect C syntax and do not build with a C compiler and break the
+Uncrustify parser. Skip reformat of C files that can not be parsed by uncrustify. These rare instances are addressed in
+the commit that fixes the C syntax error.
+
+Run this slightly modified version of lint-history. Include only .c/.h files and exclude directories that start with
+`Tools` or `BaseTools`. This step took about 2.2 hours on a laptop.
+
+```shell
+lint-history.py
+    --relevant "return (not filename.startswith(b'Tools') and not filename.startswith(b'BaseTools') and (filename.endswith(b'.c') or filename.endswith(b'.h')))"
+    c:\\work\\GitHub\\tianocore\\foo\\UncrustifyCheck\\mu-uncrustify-release_extdep\\Windows-x86\\uncrustify.exe -c c:\\work\\GitHub\\tianocore\\foo\\UncrustifyCheck\\uncrustify.cfg --replace --no-backup --if-changed
+```
-- 
2.28.0.windows.1