Re: Progress on getting Uncrustify working for EDK2?

Re: Progress on getting Uncrustify working for EDK2?

[Rebecca Cran]

[-- Attachment #1: Type: text/plain, Size: 745 bytes --]

cc devel@ .

On 8/16/21 1:33 PM, Rebecca Cran wrote:
>
> I noticed a message on Twitter about an idea of using Uncrustify for 
> EDK2 instead of the ECC tool, and came across 
> https://www.mail-archive.com/search?l=devel@edk2.groups.io&q=subject:%22Re%5C%3A+%5C%5Bedk2%5C-devel%5C%5D+TianoCore+Community+Meeting+Minutes+%5C-+2%5C%2F4%22&o=newest&f=1 
> .
>
> I was wondering if there's been any progress on it that I could check out?
>
>
> Michael Kubacki: in that message, you said:
>
> "I'm planning to put up a branch that we can use as a reference for a 
> conversation around uncrustify in the next couple of weeks."
>
>
> Did you end up creating that branch, and if so could you provide a 
> link to it please?
>
>
> -- 
> Rebecca Cran
>

[-- Attachment #2: Type: text/html, Size: 2045 bytes --]

Re: [edk2-devel] Progress on getting Uncrustify working for EDK2?

[Marvin Häuser]

Hey Rebecca,

I think even Uncrustify has issues with the EDK II indentation style. 
You might want to check the UEFI Talkbox Discord server, I had a brief 
chat with Michael about it there. I don't think realistically any tool 
supports EDK II's indentation style however, so I'd propose it is 
changed. This could be for new submissions only, or actually the entire 
codebase could be reformatted at once with a good tool setup. While this 
screws with git blame, the (to my understanding) decided on CRLF -> LF 
change does that anyway, so at least two evils could be dealt with in 
one go really.

Best regards,
Marvin

On 16/08/2021 21:34, Rebecca Cran wrote:
>
> cc devel@ .
>
> On 8/16/21 1:33 PM, Rebecca Cran wrote:
>>
>> I noticed a message on Twitter about an idea of using Uncrustify for 
>> EDK2 instead of the ECC tool, and came across 
>> https://www.mail-archive.com/search?l=devel@edk2.groups.io&q=subject:%22Re%5C%3A+%5C%5Bedk2%5C-devel%5C%5D+TianoCore+Community+Meeting+Minutes+%5C-+2%5C%2F4%22&o=newest&f=1 
>> .
>>
>> I was wondering if there's been any progress on it that I could check 
>> out?
>>
>>
>> Michael Kubacki: in that message, you said:
>>
>> "I'm planning to put up a branch that we can use as a reference for a 
>> conversation around uncrustify in the next couple of weeks."
>>
>>
>> Did you end up creating that branch, and if so could you provide a 
>> link to it please?
>>
>>
>> -- 
>> Rebecca Cran
>>
> 

Re: [edk2-devel] Progress on getting Uncrustify working for EDK2?

[Michael Kubacki]

The edk2 branch was created here: 
https://github.com/makubacki/edk2/tree/uncrustify_poc_2

We have a Project Mu fork with custom changes to the Uncrustify tool to 
help comply with EDK II formatting here: 
https://dev.azure.com/projectmu/_git/Uncrustify

The latest information about the status and how to experiment with the 
configuration file and the tool are in that fork: 
https://dev.azure.com/projectmu/Uncrustify/_wiki/wikis/Uncrustify.wiki/1/Project-Mu-(EDK-II)-Fork-Readme

That said, I have also finished a CI plugin to run Uncrustify that 
should be ready soon to initially deploy in Project Mu. Before doing so, 
I am trying to settle on an initial configuration file that less 
strictly but more reliably formats the code than in the examples in 
those branches. For example, remove heuristics that when run against the 
same set of code multiple times can produce different results. An 
example would be a rule that reformats code because it exceeds a 
specified column width on one run but on the next run that reformatted 
code triggers a different rule to further align the code and so on. At 
least initially, some rules might be tweaked in a more conservative 
approach that can be tightened in the future. Once this configuration 
file is ready, we will baseline Project Mu code as an example and turn 
on the plugin. The CI plugin runs Uncrustify against modified files and 
if there's any changes, indicating a formatting deviation, the diff 
chunks are saved in a log so they can be viewed as a build artifact.

I am making progress on the updated config file and I should be able to 
post a "uncrustify_poc_3" branch soon with the results.

Regarding indentation, Marvin is right that Uncrustify cannot support 
edk2 indentation style out-of-box. Some changes are made in that fork to 
handle the formatting. At this point, it can handle the indentation in 
the cases I've seen. Uncrustify does potentially give us the ability to 
massively deploy changes across the codebase in case a decision were 
made to change the style.

Thanks,
Michael

On 8/16/2021 3:39 PM, Marvin Häuser wrote:
> Hey Rebecca,
>
> I think even Uncrustify has issues with the EDK II indentation style. 
> You might want to check the UEFI Talkbox Discord server, I had a brief 
> chat with Michael about it there. I don't think realistically any tool 
> supports EDK II's indentation style however, so I'd propose it is 
> changed. This could be for new submissions only, or actually the 
> entire codebase could be reformatted at once with a good tool setup. 
> While this screws with git blame, the (to my understanding) decided on 
> CRLF -> LF change does that anyway, so at least two evils could be 
> dealt with in one go really.
>
> Best regards,
> Marvin
>
> On 16/08/2021 21:34, Rebecca Cran wrote:
>>
>> cc devel@ .
>>
>> On 8/16/21 1:33 PM, Rebecca Cran wrote:
>>>
>>> I noticed a message on Twitter about an idea of using Uncrustify for 
>>> EDK2 instead of the ECC tool, and came across 
>>> https://www.mail-archive.com/search?l=devel@edk2.groups.io&q=subject:%22Re%5C%3A+%5C%5Bedk2%5C-devel%5C%5D+TianoCore+Community+Meeting+Minutes+%5C-+2%5C%2F4%22&o=newest&f=1 
>>> .
>>>
>>> I was wondering if there's been any progress on it that I could 
>>> check out?
>>>
>>>
>>> Michael Kubacki: in that message, you said:
>>>
>>> "I'm planning to put up a branch that we can use as a reference for 
>>> a conversation around uncrustify in the next couple of weeks."
>>>
>>>
>>> Did you end up creating that branch, and if so could you provide a 
>>> link to it please?
>>>
>>>
>>> -- 
>>> Rebecca Cran
>>>
>>
>
>
>
> 
>

Re: [edk2-devel] Progress on getting Uncrustify working for EDK2?

[Leif Lindholm]

Hi Michael,

Apologies, I've owed you a response (promised off-list) for a while
now.

First, let me say I hugely appreciate this effort. Apart from aligning
the codebase(s), this will reduce manual reviewing effort
substantially, as well as cutting down on number of rework cycles for
developers.

Looking at the changes to (well, the comments in) uncrustify, this
seems to be constrained to:
- Newline after '(' for multi-line function calls.
- Dealing with "(("/"))" for DEBUG macros.
- Function pointer typedefs:
  - typedef\nEFIAPI
  - closing parentheses indentation

I don't think I've made any secret over the years that I am not a
massive fan of the EDK2 coding style in general. So I think for any
of its quirks that are substantial enough that they require not just
custom configuration but actual new function added to existing code
conformance tools, this would be an excellent point to sanitise the
coding style instead.

Taking these in order:

Newline after '('
-----------------
I think we already reached a level of flexibility around this, where
we don't actually enforce this (or single argument per
line). Personally, I'd be happy to update the coding style as
required instead.

DEBUG macro parentheses
-----------------------
How does uncrustify treat DEBUG macros without this modification?
Do we start getting everything turned into multi-level indented
multi-line statements without this change?

Function pointer typedefs:
--------------------------
I don't see that function pointer typedefs need to rigidly follow the
same pattern as the declaration of functions implementing them. Could
we update the coding style (if needed) instead?

Best Regards,

Leif

On Mon, Aug 16, 2021 at 16:00:38 -0400, Michael Kubacki wrote:
> The edk2 branch was created here:
> https://github.com/makubacki/edk2/tree/uncrustify_poc_2
> 
> We have a Project Mu fork with custom changes to the Uncrustify tool to help
> comply with EDK II formatting here:
> https://dev.azure.com/projectmu/_git/Uncrustify
> 
> The latest information about the status and how to experiment with the
> configuration file and the tool are in that fork: https://dev.azure.com/projectmu/Uncrustify/_wiki/wikis/Uncrustify.wiki/1/Project-Mu-(EDK-II)-Fork-Readme
> 
> That said, I have also finished a CI plugin to run Uncrustify that should be
> ready soon to initially deploy in Project Mu. Before doing so, I am trying
> to settle on an initial configuration file that less strictly but more
> reliably formats the code than in the examples in those branches. For
> example, remove heuristics that when run against the same set of code
> multiple times can produce different results. An example would be a rule
> that reformats code because it exceeds a specified column width on one run
> but on the next run that reformatted code triggers a different rule to
> further align the code and so on. At least initially, some rules might be
> tweaked in a more conservative approach that can be tightened in the future.
> Once this configuration file is ready, we will baseline Project Mu code as
> an example and turn on the plugin. The CI plugin runs Uncrustify against
> modified files and if there's any changes, indicating a formatting
> deviation, the diff chunks are saved in a log so they can be viewed as a
> build artifact.
> 
> I am making progress on the updated config file and I should be able to post
> a "uncrustify_poc_3" branch soon with the results.
> 
> Regarding indentation, Marvin is right that Uncrustify cannot support edk2
> indentation style out-of-box. Some changes are made in that fork to handle
> the formatting. At this point, it can handle the indentation in the cases
> I've seen. Uncrustify does potentially give us the ability to massively
> deploy changes across the codebase in case a decision were made to change
> the style.
> 
> Thanks,
> Michael
> 
> On 8/16/2021 3:39 PM, Marvin Häuser wrote:
> > Hey Rebecca,
> > 
> > I think even Uncrustify has issues with the EDK II indentation style.
> > You might want to check the UEFI Talkbox Discord server, I had a brief
> > chat with Michael about it there. I don't think realistically any tool
> > supports EDK II's indentation style however, so I'd propose it is
> > changed. This could be for new submissions only, or actually the entire
> > codebase could be reformatted at once with a good tool setup. While this
> > screws with git blame, the (to my understanding) decided on CRLF -> LF
> > change does that anyway, so at least two evils could be dealt with in
> > one go really.
> > 
> > Best regards,
> > Marvin
> > 
> > On 16/08/2021 21:34, Rebecca Cran wrote:
> > > 
> > > cc devel@ .
> > > 
> > > On 8/16/21 1:33 PM, Rebecca Cran wrote:
> > > > 
> > > > I noticed a message on Twitter about an idea of using Uncrustify
> > > > for EDK2 instead of the ECC tool, and came across https://www.mail-archive.com/search?l=devel@edk2.groups.io&q=subject:%22Re%5C%3A+%5C%5Bedk2%5C-devel%5C%5D+TianoCore+Community+Meeting+Minutes+%5C-+2%5C%2F4%22&o=newest&f=1
> > > > .
> > > > 
> > > > I was wondering if there's been any progress on it that I could
> > > > check out?
> > > > 
> > > > 
> > > > Michael Kubacki: in that message, you said:
> > > > 
> > > > "I'm planning to put up a branch that we can use as a reference
> > > > for a conversation around uncrustify in the next couple of
> > > > weeks."
> > > > 
> > > > 
> > > > Did you end up creating that branch, and if so could you provide
> > > > a link to it please?
> > > > 
> > > > 
> > > > -- 
> > > > Rebecca Cran
> > > > 
> > > 
> > 
> > 
> > 
> > 
> > 
> 
> 
> 
> 
> 
> 

Re: [edk2-devel] Progress on getting Uncrustify working for EDK2?

[Andrew Fish]

[-- Attachment #1: Type: text/plain, Size: 6129 bytes --]



> On Oct 7, 2021, at 6:48 AM, Leif Lindholm <leif@nuviainc.com> wrote:
> 
> Hi Michael,
> 
> Apologies, I've owed you a response (promised off-list) for a while
> now.
> 
> First, let me say I hugely appreciate this effort. Apart from aligning
> the codebase(s), this will reduce manual reviewing effort
> substantially, as well as cutting down on number of rework cycles for
> developers.
> 
> Looking at the changes to (well, the comments in) uncrustify, this
> seems to be constrained to:
> - Newline after '(' for multi-line function calls.
> - Dealing with "(("/"))" for DEBUG macros.
> - Function pointer typedefs:
>  - typedef\nEFIAPI
>  - closing parentheses indentation
> 
> I don't think I've made any secret over the years that I am not a
> massive fan of the EDK2 coding style in general. So I think for any
> of its quirks that are substantial enough that they require not just
> custom configuration but actual new function added to existing code
> conformance tools, this would be an excellent point to sanitise the
> coding style instead.
> 
> Taking these in order:
> 
> Newline after '('
> -----------------
> I think we already reached a level of flexibility around this, where
> we don't actually enforce this (or single argument per
> line). Personally, I'd be happy to update the coding style as
> required instead.
> 
> DEBUG macro parentheses
> -----------------------
> How does uncrustify treat DEBUG macros without this modification?
> Do we start getting everything turned into multi-level indented
> multi-line statements without this change?

Can we disable the rule that does not like the DEBUG macro? I seem to remember clang nags about some strange () usage, so maybe we would not lose that much?

Thanks,

Andrew Fish

> 
> Function pointer typedefs:
> --------------------------
> I don't see that function pointer typedefs need to rigidly follow the
> same pattern as the declaration of functions implementing them. Could
> we update the coding style (if needed) instead?
> 
> Best Regards,
> 
> Leif
> 
> On Mon, Aug 16, 2021 at 16:00:38 -0400, Michael Kubacki wrote:
>> The edk2 branch was created here:
>> https://github.com/makubacki/edk2/tree/uncrustify_poc_2
>> 
>> We have a Project Mu fork with custom changes to the Uncrustify tool to help
>> comply with EDK II formatting here:
>> https://dev.azure.com/projectmu/_git/Uncrustify
>> 
>> The latest information about the status and how to experiment with the
>> configuration file and the tool are in that fork: https://dev.azure.com/projectmu/Uncrustify/_wiki/wikis/Uncrustify.wiki/1/Project-Mu-(EDK-II)-Fork-Readme
>> 
>> That said, I have also finished a CI plugin to run Uncrustify that should be
>> ready soon to initially deploy in Project Mu. Before doing so, I am trying
>> to settle on an initial configuration file that less strictly but more
>> reliably formats the code than in the examples in those branches. For
>> example, remove heuristics that when run against the same set of code
>> multiple times can produce different results. An example would be a rule
>> that reformats code because it exceeds a specified column width on one run
>> but on the next run that reformatted code triggers a different rule to
>> further align the code and so on. At least initially, some rules might be
>> tweaked in a more conservative approach that can be tightened in the future.
>> Once this configuration file is ready, we will baseline Project Mu code as
>> an example and turn on the plugin. The CI plugin runs Uncrustify against
>> modified files and if there's any changes, indicating a formatting
>> deviation, the diff chunks are saved in a log so they can be viewed as a
>> build artifact.
>> 
>> I am making progress on the updated config file and I should be able to post
>> a "uncrustify_poc_3" branch soon with the results.
>> 
>> Regarding indentation, Marvin is right that Uncrustify cannot support edk2
>> indentation style out-of-box. Some changes are made in that fork to handle
>> the formatting. At this point, it can handle the indentation in the cases
>> I've seen. Uncrustify does potentially give us the ability to massively
>> deploy changes across the codebase in case a decision were made to change
>> the style.
>> 
>> Thanks,
>> Michael
>> 
>> On 8/16/2021 3:39 PM, Marvin Häuser wrote:
>>> Hey Rebecca,
>>> 
>>> I think even Uncrustify has issues with the EDK II indentation style.
>>> You might want to check the UEFI Talkbox Discord server, I had a brief
>>> chat with Michael about it there. I don't think realistically any tool
>>> supports EDK II's indentation style however, so I'd propose it is
>>> changed. This could be for new submissions only, or actually the entire
>>> codebase could be reformatted at once with a good tool setup. While this
>>> screws with git blame, the (to my understanding) decided on CRLF -> LF
>>> change does that anyway, so at least two evils could be dealt with in
>>> one go really.
>>> 
>>> Best regards,
>>> Marvin
>>> 
>>> On 16/08/2021 21:34, Rebecca Cran wrote:
>>>> 
>>>> cc devel@ .
>>>> 
>>>> On 8/16/21 1:33 PM, Rebecca Cran wrote:
>>>>> 
>>>>> I noticed a message on Twitter about an idea of using Uncrustify
>>>>> for EDK2 instead of the ECC tool, and came across https://www.mail-archive.com/search?l=devel@edk2.groups.io&q=subject:%22Re%5C%3A+%5C%5Bedk2%5C-devel%5C%5D+TianoCore+Community+Meeting+Minutes+%5C-+2%5C%2F4%22&o=newest&f=1
>>>>> .
>>>>> 
>>>>> I was wondering if there's been any progress on it that I could
>>>>> check out?
>>>>> 
>>>>> 
>>>>> Michael Kubacki: in that message, you said:
>>>>> 
>>>>> "I'm planning to put up a branch that we can use as a reference
>>>>> for a conversation around uncrustify in the next couple of
>>>>> weeks."
>>>>> 
>>>>> 
>>>>> Did you end up creating that branch, and if so could you provide
>>>>> a link to it please?
>>>>> 
>>>>> 
>>>>> -- 
>>>>> Rebecca Cran
>>>>> 
>>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>> 
>> 
>> 
>> 
>> 
>> 
> 
> 
> 


[-- Attachment #2: Type: text/html, Size: 41425 bytes --]

Re: [edk2-devel] Progress on getting Uncrustify working for EDK2?

[Andrew Fish]

[-- Attachment #1: Type: text/plain, Size: 7204 bytes --]

On the git-blame front we might be able to build a recipe that walks past the code style and line ending commits that would could add to an FAQ or Wiki.

For bonus points we could add a git command that does this. You can add a Python git command to the repo by adding a git-<commandName> script in the path. So we could add a Python git command that skips blame of any of the known Uncrustify or line ending type commits. Basically `git eblame` that works just like `git blame` but skips formatting changes. 

I’m working on a git-pgrep that only greps files used by a given build, hopeful I’ll be able to contribute that soon. My co-works keep finding bugs so I’m not quite done.

Thanks,

Andrew Fish

> On Oct 7, 2021, at 11:30 AM, Andrew Fish via groups.io <afish=apple.com@groups.io> wrote:
> 
> 
> 
>> On Oct 7, 2021, at 6:48 AM, Leif Lindholm <leif@nuviainc.com <mailto:leif@nuviainc.com>> wrote:
>> 
>> Hi Michael,
>> 
>> Apologies, I've owed you a response (promised off-list) for a while
>> now.
>> 
>> First, let me say I hugely appreciate this effort. Apart from aligning
>> the codebase(s), this will reduce manual reviewing effort
>> substantially, as well as cutting down on number of rework cycles for
>> developers.
>> 
>> Looking at the changes to (well, the comments in) uncrustify, this
>> seems to be constrained to:
>> - Newline after '(' for multi-line function calls.
>> - Dealing with "(("/"))" for DEBUG macros.
>> - Function pointer typedefs:
>>  - typedef\nEFIAPI
>>  - closing parentheses indentation
>> 
>> I don't think I've made any secret over the years that I am not a
>> massive fan of the EDK2 coding style in general. So I think for any
>> of its quirks that are substantial enough that they require not just
>> custom configuration but actual new function added to existing code
>> conformance tools, this would be an excellent point to sanitise the
>> coding style instead.
>> 
>> Taking these in order:
>> 
>> Newline after '('
>> -----------------
>> I think we already reached a level of flexibility around this, where
>> we don't actually enforce this (or single argument per
>> line). Personally, I'd be happy to update the coding style as
>> required instead.
>> 
>> DEBUG macro parentheses
>> -----------------------
>> How does uncrustify treat DEBUG macros without this modification?
>> Do we start getting everything turned into multi-level indented
>> multi-line statements without this change?
> 
> Can we disable the rule that does not like the DEBUG macro? I seem to remember clang nags about some strange () usage, so maybe we would not lose that much?
> 
> Thanks,
> 
> Andrew Fish
> 
>> 
>> Function pointer typedefs:
>> --------------------------
>> I don't see that function pointer typedefs need to rigidly follow the
>> same pattern as the declaration of functions implementing them. Could
>> we update the coding style (if needed) instead?
>> 
>> Best Regards,
>> 
>> Leif
>> 
>> On Mon, Aug 16, 2021 at 16:00:38 -0400, Michael Kubacki wrote:
>>> The edk2 branch was created here:
>>> https://github.com/makubacki/edk2/tree/uncrustify_poc_2 <https://github.com/makubacki/edk2/tree/uncrustify_poc_2>
>>> 
>>> We have a Project Mu fork with custom changes to the Uncrustify tool to help
>>> comply with EDK II formatting here:
>>> https://dev.azure.com/projectmu/_git/Uncrustify
>>> 
>>> The latest information about the status and how to experiment with the
>>> configuration file and the tool are in that fork: https://dev.azure.com/projectmu/Uncrustify/_wiki/wikis/Uncrustify.wiki/1/Project-Mu-(EDK-II)-Fork-Readme
>>> 
>>> That said, I have also finished a CI plugin to run Uncrustify that should be
>>> ready soon to initially deploy in Project Mu. Before doing so, I am trying
>>> to settle on an initial configuration file that less strictly but more
>>> reliably formats the code than in the examples in those branches. For
>>> example, remove heuristics that when run against the same set of code
>>> multiple times can produce different results. An example would be a rule
>>> that reformats code because it exceeds a specified column width on one run
>>> but on the next run that reformatted code triggers a different rule to
>>> further align the code and so on. At least initially, some rules might be
>>> tweaked in a more conservative approach that can be tightened in the future.
>>> Once this configuration file is ready, we will baseline Project Mu code as
>>> an example and turn on the plugin. The CI plugin runs Uncrustify against
>>> modified files and if there's any changes, indicating a formatting
>>> deviation, the diff chunks are saved in a log so they can be viewed as a
>>> build artifact.
>>> 
>>> I am making progress on the updated config file and I should be able to post
>>> a "uncrustify_poc_3" branch soon with the results.
>>> 
>>> Regarding indentation, Marvin is right that Uncrustify cannot support edk2
>>> indentation style out-of-box. Some changes are made in that fork to handle
>>> the formatting. At this point, it can handle the indentation in the cases
>>> I've seen. Uncrustify does potentially give us the ability to massively
>>> deploy changes across the codebase in case a decision were made to change
>>> the style.
>>> 
>>> Thanks,
>>> Michael
>>> 
>>> On 8/16/2021 3:39 PM, Marvin Häuser wrote:
>>>> Hey Rebecca,
>>>> 
>>>> I think even Uncrustify has issues with the EDK II indentation style.
>>>> You might want to check the UEFI Talkbox Discord server, I had a brief
>>>> chat with Michael about it there. I don't think realistically any tool
>>>> supports EDK II's indentation style however, so I'd propose it is
>>>> changed. This could be for new submissions only, or actually the entire
>>>> codebase could be reformatted at once with a good tool setup. While this
>>>> screws with git blame, the (to my understanding) decided on CRLF -> LF
>>>> change does that anyway, so at least two evils could be dealt with in
>>>> one go really.
>>>> 
>>>> Best regards,
>>>> Marvin
>>>> 
>>>> On 16/08/2021 21:34, Rebecca Cran wrote:
>>>>> 
>>>>> cc devel@ .
>>>>> 
>>>>> On 8/16/21 1:33 PM, Rebecca Cran wrote:
>>>>>> 
>>>>>> I noticed a message on Twitter about an idea of using Uncrustify
>>>>>> for EDK2 instead of the ECC tool, and came across https://www.mail-archive.com/search?l=devel@edk2.groups.io&q=subject:%22Re%5C%3A+%5C%5Bedk2%5C-devel%5C%5D+TianoCore+Community+Meeting+Minutes+%5C-+2%5C%2F4%22&o=newest&f=1
>>>>>> .
>>>>>> 
>>>>>> I was wondering if there's been any progress on it that I could
>>>>>> check out?
>>>>>> 
>>>>>> 
>>>>>> Michael Kubacki: in that message, you said:
>>>>>> 
>>>>>> "I'm planning to put up a branch that we can use as a reference
>>>>>> for a conversation around uncrustify in the next couple of
>>>>>> weeks."
>>>>>> 
>>>>>> 
>>>>>> Did you end up creating that branch, and if so could you provide
>>>>>> a link to it please?
>>>>>> 
>>>>>> 
>>>>>> -- 
>>>>>> Rebecca Cran
>>>>>> 
>>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>> 
>> 
> 
> 


[-- Attachment #2: Type: text/html, Size: 43456 bytes --]

Re: [edk2-devel] Progress on getting Uncrustify working for EDK2?

[Michael D Kinney]

Hi Michael,

I am looking at your uncrustify_poc_3 branch, and I see an unexpected change
related to use of IN/OUT in a function declaration.

File: MdeModulePkg\Logo\Logo.c

Before Uncrustify
=================
EFI_STATUS
EFIAPI
GetImage (
  IN     EDKII_PLATFORM_LOGO_PROTOCOL          *This,
  IN OUT UINT32                                *Instance,
     OUT EFI_IMAGE_INPUT                       *Image,
     OUT EDKII_PLATFORM_LOGO_DISPLAY_ATTRIBUTE *Attribute,
     OUT INTN                                  *OffsetX,
     OUT INTN                                  *OffsetY
  )

After Uncrustify
================
GetImage (
  IN     EDKII_PLATFORM_LOGO_PROTOCOL          *This,
  IN OUT UINT32                                *Instance,
  OUT EFI_IMAGE_INPUT                       *Image,
  OUT EDKII_PLATFORM_LOGO_DISPLAY_ATTRIBUTE *Attribute,
  OUT INTN                                  *OffsetX,
  OUT INTN                                  *OffsetY
  )

It aligned the start of each parameter line with 2 space indent, but lost the
alignment of the parameter names and did not do 2 spaces between the parameter
type and parameter name.

Best regards,

Mike

----------------------------------------------------------

From: Andrew Fish <afish@apple.com> 
Sent: Thursday, October 7, 2021 9:04 AM
To: edk2-devel-groups-io <devel@edk2.groups.io>
Cc: Leif Lindholm <leif@nuviainc.com>; mikuback@linux.microsoft.com; Marvin Häuser <mhaeuser@posteo.de>; Rebecca Cran <rebecca@nuviainc.com>; Michael Kubacki <Michael.Kubacki@microsoft.com>; Bret Barkelew <Bret.Barkelew@microsoft.com>; Kinney, Michael D <michael.d.kinney@intel.com>; Andrew Fish <afish@apple.com>
Subject: Re: [edk2-devel] Progress on getting Uncrustify working for EDK2?

On the git-blame front we might be able to build a recipe that walks past the code style and line ending commits that would could add to an FAQ or Wiki.

For bonus points we could add a git command that does this. You can add a Python git command to the repo by adding a git-<commandName> script in the path. So we could add a Python git command that skips blame of any of the known Uncrustify or line ending type commits. Basically `git eblame` that works just like `git blame` but skips formatting changes. 

I’m working on a git-pgrep that only greps files used by a given build, hopeful I’ll be able to contribute that soon. My co-works keep finding bugs so I’m not quite done.

Thanks,

Andrew Fish


On Oct 7, 2021, at 11:30 AM, Andrew Fish via http://groups.io <mailto:afish=apple.com@groups.io> wrote:




On Oct 7, 2021, at 6:48 AM, Leif Lindholm <mailto:leif@nuviainc.com> wrote:

Hi Michael,

Apologies, I've owed you a response (promised off-list) for a while
now.

First, let me say I hugely appreciate this effort. Apart from aligning
the codebase(s), this will reduce manual reviewing effort
substantially, as well as cutting down on number of rework cycles for
developers.

Looking at the changes to (well, the comments in) uncrustify, this
seems to be constrained to:
- Newline after '(' for multi-line function calls.
- Dealing with "(("/"))" for DEBUG macros.
- Function pointer typedefs:
 - typedef\nEFIAPI
 - closing parentheses indentation

I don't think I've made any secret over the years that I am not a
massive fan of the EDK2 coding style in general. So I think for any
of its quirks that are substantial enough that they require not just
custom configuration but actual new function added to existing code
conformance tools, this would be an excellent point to sanitise the
coding style instead.

Taking these in order:

Newline after '('
-----------------
I think we already reached a level of flexibility around this, where
we don't actually enforce this (or single argument per
line). Personally, I'd be happy to update the coding style as
required instead.

DEBUG macro parentheses
-----------------------
How does uncrustify treat DEBUG macros without this modification?
Do we start getting everything turned into multi-level indented
multi-line statements without this change?

Can we disable the rule that does not like the DEBUG macro? I seem to remember clang nags about some strange () usage, so maybe we would not lose that much?

Thanks,

Andrew Fish



Function pointer typedefs:
--------------------------
I don't see that function pointer typedefs need to rigidly follow the
same pattern as the declaration of functions implementing them. Could
we update the coding style (if needed) instead?

Best Regards,

Leif

On Mon, Aug 16, 2021 at 16:00:38 -0400, Michael Kubacki wrote:

The edk2 branch was created here:
https://github.com/makubacki/edk2/tree/uncrustify_poc_2

We have a Project Mu fork with custom changes to the Uncrustify tool to help
comply with EDK II formatting here:
https://dev.azure.com/projectmu/_git/Uncrustify

The latest information about the status and how to experiment with the
configuration file and the tool are in that fork: https://dev.azure.com/projectmu/Uncrustify/_wiki/wikis/Uncrustify.wiki/1/Project-Mu-(EDK-II)-Fork-Readme

That said, I have also finished a CI plugin to run Uncrustify that should be
ready soon to initially deploy in Project Mu. Before doing so, I am trying
to settle on an initial configuration file that less strictly but more
reliably formats the code than in the examples in those branches. For
example, remove heuristics that when run against the same set of code
multiple times can produce different results. An example would be a rule
that reformats code because it exceeds a specified column width on one run
but on the next run that reformatted code triggers a different rule to
further align the code and so on. At least initially, some rules might be
tweaked in a more conservative approach that can be tightened in the future.
Once this configuration file is ready, we will baseline Project Mu code as
an example and turn on the plugin. The CI plugin runs Uncrustify against
modified files and if there's any changes, indicating a formatting
deviation, the diff chunks are saved in a log so they can be viewed as a
build artifact.

I am making progress on the updated config file and I should be able to post
a "uncrustify_poc_3" branch soon with the results.

Regarding indentation, Marvin is right that Uncrustify cannot support edk2
indentation style out-of-box. Some changes are made in that fork to handle
the formatting. At this point, it can handle the indentation in the cases
I've seen. Uncrustify does potentially give us the ability to massively
deploy changes across the codebase in case a decision were made to change
the style.

Thanks,
Michael

On 8/16/2021 3:39 PM, Marvin Häuser wrote:

Hey Rebecca,

I think even Uncrustify has issues with the EDK II indentation style.
You might want to check the UEFI Talkbox Discord server, I had a brief
chat with Michael about it there. I don't think realistically any tool
supports EDK II's indentation style however, so I'd propose it is
changed. This could be for new submissions only, or actually the entire
codebase could be reformatted at once with a good tool setup. While this
screws with git blame, the (to my understanding) decided on CRLF -> LF
change does that anyway, so at least two evils could be dealt with in
one go really.

Best regards,
Marvin

On 16/08/2021 21:34, Rebecca Cran wrote:


cc devel@ .

On 8/16/21 1:33 PM, Rebecca Cran wrote:


I noticed a message on Twitter about an idea of using Uncrustify
for EDK2 instead of the ECC tool, and came across https://www.mail-archive.com/search?l=devel@edk2.groups.io&q=subject:%22Re%5C%3A+%5C%5Bedk2%5C-devel%5C%5D+TianoCore+Community+Meeting+Minutes+%5C-+2%5C%2F4%22&o=newest&f=1
.

I was wondering if there's been any progress on it that I could
check out?


Michael Kubacki: in that message, you said:

"I'm planning to put up a branch that we can use as a reference
for a conversation around uncrustify in the next couple of
weeks."


Did you end up creating that branch, and if so could you provide
a link to it please?


-- 
Rebecca Cran

Re: [edk2-devel] Progress on getting Uncrustify working for EDK2?

[Michael Kubacki]

Thanks Mike, I will look into that.

For others, I recently published an "uncrustify_poc_3" branch on my edk2 
fork that demonstrates the latest set of results I've obtained with 
Uncrustify tool changes and the configuration file. Most of the 
edk2-specific issues have been mitigated and I was able to reach a 
steady state of formatting results after two runs against the edk2 code 
base.

The results are starting to trend toward what might be "acceptable".

I would note that the edk2 repository and, in particular, certain 
packages like MdePkg and MdeModulePkg will have less substantial changes 
than others as the coding style tends to be followed more consistently 
there relative to many other repos.

Here's a link to the branch:

https://github.com/makubacki/edk2/tree/uncrustify_poc_3

There's a version here that attempts to add missing file and function 
headers (same branch just with that commit at the top):

https://github.com/makubacki/edk2/tree/uncrustify_poc_3_with_headers

Here's an Uncrustify application PR that I'm currently using the 
executable from for testing (it is in the PR build artifacts):

https://dev.azure.com/projectmu/Uncrustify/_git/Uncrustify/pullrequest/24

I'm interested in the discussion around style changes though we might be 
able to get away with less of those now. I am working on this in between 
other work so I also appreciate anyone willing to help out in any way.

Thanks,
Michael

On 10/7/2021 12:34 PM, Kinney, Michael D wrote:
> Hi Michael,
> 
> I am looking at your uncrustify_poc_3 branch, and I see an unexpected change
> related to use of IN/OUT in a function declaration.
> 
> File: MdeModulePkg\Logo\Logo.c
> 
> Before Uncrustify
> =================
> EFI_STATUS
> EFIAPI
> GetImage (
>    IN     EDKII_PLATFORM_LOGO_PROTOCOL          *This,
>    IN OUT UINT32                                *Instance,
>       OUT EFI_IMAGE_INPUT                       *Image,
>       OUT EDKII_PLATFORM_LOGO_DISPLAY_ATTRIBUTE *Attribute,
>       OUT INTN                                  *OffsetX,
>       OUT INTN                                  *OffsetY
>    )
> 
> After Uncrustify
> ================
> GetImage (
>    IN     EDKII_PLATFORM_LOGO_PROTOCOL          *This,
>    IN OUT UINT32                                *Instance,
>    OUT EFI_IMAGE_INPUT                       *Image,
>    OUT EDKII_PLATFORM_LOGO_DISPLAY_ATTRIBUTE *Attribute,
>    OUT INTN                                  *OffsetX,
>    OUT INTN                                  *OffsetY
>    )
> 
> It aligned the start of each parameter line with 2 space indent, but lost the
> alignment of the parameter names and did not do 2 spaces between the parameter
> type and parameter name.
> 
> Best regards,
> 
> Mike
> 
> ----------------------------------------------------------
> 
> From: Andrew Fish <afish@apple.com>
> Sent: Thursday, October 7, 2021 9:04 AM
> To: edk2-devel-groups-io <devel@edk2.groups.io>
> Cc: Leif Lindholm <leif@nuviainc.com>; mikuback@linux.microsoft.com; Marvin Häuser <mhaeuser@posteo.de>; Rebecca Cran <rebecca@nuviainc.com>; Michael Kubacki <Michael.Kubacki@microsoft.com>; Bret Barkelew <Bret.Barkelew@microsoft.com>; Kinney, Michael D <michael.d.kinney@intel.com>; Andrew Fish <afish@apple.com>
> Subject: Re: [edk2-devel] Progress on getting Uncrustify working for EDK2?
> 
> On the git-blame front we might be able to build a recipe that walks past the code style and line ending commits that would could add to an FAQ or Wiki.
> 
> For bonus points we could add a git command that does this. You can add a Python git command to the repo by adding a git-<commandName> script in the path. So we could add a Python git command that skips blame of any of the known Uncrustify or line ending type commits. Basically `git eblame` that works just like `git blame` but skips formatting changes.
> 
> I’m working on a git-pgrep that only greps files used by a given build, hopeful I’ll be able to contribute that soon. My co-works keep finding bugs so I’m not quite done.
> 
> Thanks,
> 
> Andrew Fish
> 
> 
> On Oct 7, 2021, at 11:30 AM, Andrew Fish via http://groups.io <mailto:afish=apple.com@groups.io> wrote:
> 
> 
> 
> 
> On Oct 7, 2021, at 6:48 AM, Leif Lindholm <mailto:leif@nuviainc.com> wrote:
> 
> Hi Michael,
> 
> Apologies, I've owed you a response (promised off-list) for a while
> now.
> 
> First, let me say I hugely appreciate this effort. Apart from aligning
> the codebase(s), this will reduce manual reviewing effort
> substantially, as well as cutting down on number of rework cycles for
> developers.
> 
> Looking at the changes to (well, the comments in) uncrustify, this
> seems to be constrained to:
> - Newline after '(' for multi-line function calls.
> - Dealing with "(("/"))" for DEBUG macros.
> - Function pointer typedefs:
>   - typedef\nEFIAPI
>   - closing parentheses indentation
> 
> I don't think I've made any secret over the years that I am not a
> massive fan of the EDK2 coding style in general. So I think for any
> of its quirks that are substantial enough that they require not just
> custom configuration but actual new function added to existing code
> conformance tools, this would be an excellent point to sanitise the
> coding style instead.
> 
> Taking these in order:
> 
> Newline after '('
> -----------------
> I think we already reached a level of flexibility around this, where
> we don't actually enforce this (or single argument per
> line). Personally, I'd be happy to update the coding style as
> required instead.
> 
> DEBUG macro parentheses
> -----------------------
> How does uncrustify treat DEBUG macros without this modification?
> Do we start getting everything turned into multi-level indented
> multi-line statements without this change?
> 
> Can we disable the rule that does not like the DEBUG macro? I seem to remember clang nags about some strange () usage, so maybe we would not lose that much?
> 
> Thanks,
> 
> Andrew Fish
> 
> 
> 
> Function pointer typedefs:
> --------------------------
> I don't see that function pointer typedefs need to rigidly follow the
> same pattern as the declaration of functions implementing them. Could
> we update the coding style (if needed) instead?
> 
> Best Regards,
> 
> Leif
> 
> On Mon, Aug 16, 2021 at 16:00:38 -0400, Michael Kubacki wrote:
> 
> The edk2 branch was created here:
> https://github.com/makubacki/edk2/tree/uncrustify_poc_2
> 
> We have a Project Mu fork with custom changes to the Uncrustify tool to help
> comply with EDK II formatting here:
> https://dev.azure.com/projectmu/_git/Uncrustify
> 
> The latest information about the status and how to experiment with the
> configuration file and the tool are in that fork: https://dev.azure.com/projectmu/Uncrustify/_wiki/wikis/Uncrustify.wiki/1/Project-Mu-(EDK-II)-Fork-Readme
> 
> That said, I have also finished a CI plugin to run Uncrustify that should be
> ready soon to initially deploy in Project Mu. Before doing so, I am trying
> to settle on an initial configuration file that less strictly but more
> reliably formats the code than in the examples in those branches. For
> example, remove heuristics that when run against the same set of code
> multiple times can produce different results. An example would be a rule
> that reformats code because it exceeds a specified column width on one run
> but on the next run that reformatted code triggers a different rule to
> further align the code and so on. At least initially, some rules might be
> tweaked in a more conservative approach that can be tightened in the future.
> Once this configuration file is ready, we will baseline Project Mu code as
> an example and turn on the plugin. The CI plugin runs Uncrustify against
> modified files and if there's any changes, indicating a formatting
> deviation, the diff chunks are saved in a log so they can be viewed as a
> build artifact.
> 
> I am making progress on the updated config file and I should be able to post
> a "uncrustify_poc_3" branch soon with the results.
> 
> Regarding indentation, Marvin is right that Uncrustify cannot support edk2
> indentation style out-of-box. Some changes are made in that fork to handle
> the formatting. At this point, it can handle the indentation in the cases
> I've seen. Uncrustify does potentially give us the ability to massively
> deploy changes across the codebase in case a decision were made to change
> the style.
> 
> Thanks,
> Michael
> 
> On 8/16/2021 3:39 PM, Marvin Häuser wrote:
> 
> Hey Rebecca,
> 
> I think even Uncrustify has issues with the EDK II indentation style.
> You might want to check the UEFI Talkbox Discord server, I had a brief
> chat with Michael about it there. I don't think realistically any tool
> supports EDK II's indentation style however, so I'd propose it is
> changed. This could be for new submissions only, or actually the entire
> codebase could be reformatted at once with a good tool setup. While this
> screws with git blame, the (to my understanding) decided on CRLF -> LF
> change does that anyway, so at least two evils could be dealt with in
> one go really.
> 
> Best regards,
> Marvin
> 
> On 16/08/2021 21:34, Rebecca Cran wrote:
> 
> 
> cc devel@ .
> 
> On 8/16/21 1:33 PM, Rebecca Cran wrote:
> 
> 
> I noticed a message on Twitter about an idea of using Uncrustify
> for EDK2 instead of the ECC tool, and came across https://www.mail-archive.com/search?l=devel@edk2.groups.io&q=subject:%22Re%5C%3A+%5C%5Bedk2%5C-devel%5C%5D+TianoCore+Community+Meeting+Minutes+%5C-+2%5C%2F4%22&o=newest&f=1
> .
> 
> I was wondering if there's been any progress on it that I could
> check out?
> 
> 
> Michael Kubacki: in that message, you said:
> 
> "I'm planning to put up a branch that we can use as a reference
> for a conversation around uncrustify in the next couple of
> weeks."
> 
> 
> Did you end up creating that branch, and if so could you provide
> a link to it please?
> 
> 
> -- 
> Rebecca Cran
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 

Re: [edk2-devel] Progress on getting Uncrustify working for EDK2?

[Michael D Kinney]

Hi Leif,

Thanks for the feedback.  Adjusting the EDK II C Coding Style should absolutely
be part of this discussion.  We need to work towards specific proposals.  Here are
the current open bugs against the EDK II C Coding Style.  Please update or add new 
for the changes you are suggesting.

https://bugzilla.tianocore.org/buglist.cgi?bug_status=UNCONFIRMED&bug_status=CONFIRMED&bug_status=IN_PROGRESS&columnlist=product%2Ccomponent%2Cshort_desc%2Creporter&component=Documentation&f1=cf_documents&list_id=23944&o1=equals&product=EDK2&query_format=advanced&resolution=---&v1=EDK%20II%20C%20Coding%20Standards

  2664  Discrepancies/inconsistencies in coding standards, style and examples 
  1698  Spurious rule about comment style in CCS 6.2.3 
  713   Update EDK II C Coding standard to state a stronger preference for 80 column line widths 
  714   Update EDK II C Coding Standards to allow multiple arguments per line in a function call 
  1766  Remove use of STATIC macros from EDK II C Coding Standard Specification 
  2391  Dangling word at end of Section 5.1.9 In-line assembler shall not be used
  723   5.1.8 Typo provided should be proven 
  884   The cover.jpg file for the EDK II specs has the old style logo 
  839   Minor typographical errors 

Additional comments below.

Thanks,

Mike

> -----Original Message-----
> From: Leif Lindholm <leif@nuviainc.com>
> Sent: Thursday, October 7, 2021 3:48 AM
> To: devel@edk2.groups.io; mikuback@linux.microsoft.com
> Cc: mhaeuser@posteo.de; rebecca@nuviainc.com; Michael Kubacki <Michael.Kubacki@microsoft.com>; Bret Barkelew
> <Bret.Barkelew@microsoft.com>; Kinney, Michael D <michael.d.kinney@intel.com>
> Subject: Re: [edk2-devel] Progress on getting Uncrustify working for EDK2?
> 
> Hi Michael,
> 
> Apologies, I've owed you a response (promised off-list) for a while
> now.
> 
> First, let me say I hugely appreciate this effort. Apart from aligning
> the codebase(s), this will reduce manual reviewing effort
> substantially, as well as cutting down on number of rework cycles for
> developers.
> 
> Looking at the changes to (well, the comments in) uncrustify, this
> seems to be constrained to:
> - Newline after '(' for multi-line function calls.
> - Dealing with "(("/"))" for DEBUG macros.
> - Function pointer typedefs:
>   - typedef\nEFIAPI
>   - closing parentheses indentation
> 
> I don't think I've made any secret over the years that I am not a
> massive fan of the EDK2 coding style in general. So I think for any
> of its quirks that are substantial enough that they require not just
> custom configuration but actual new function added to existing code
> conformance tools, this would be an excellent point to sanitise the
> coding style instead.
> 
> Taking these in order:
> 
> Newline after '('
> -----------------
> I think we already reached a level of flexibility around this, where
> we don't actually enforce this (or single argument per
> line). Personally, I'd be happy to update the coding style as
> required instead.


Here is an example of a change that uncrustify did that I think you
are suggesting is not required with support for multiple params
per line.  Correct?

Before Uncristify
=================
  AsmCpuidEx (
    CPUID_EXTENDED_STATE, CPUID_EXTENDED_STATE_SUB_LEAF,
    &Eax.Uint32, &Ebx, &Ecx.Uint32, &Edx
    );

After Uncristify
=================
  AsmCpuidEx (
    CPUID_EXTENDED_STATE,
    CPUID_EXTENDED_STATE_SUB_LEAF,
    &Eax.Uint32,
    &Ebx,
    &Ecx.Uint32,
    &Edx
    );

Here is the BZ on the topic.

https://bugzilla.tianocore.org/show_bug.cgi?id=714

I think you are suggesting that a 4th style may be allowed
which would be to support arguments in the same line as
the function name and additional arguments on additional
lines.  Do you want to update this BZ with your additional
thoughts?

> 
> DEBUG macro parentheses
> -----------------------
> How does uncrustify treat DEBUG macros without this modification?
> Do we start getting everything turned into multi-level indented
> multi-line statements without this change?

I think this is related to the DEBUG() macro using double (( and double ))
parens.

> 
> Function pointer typedefs:
> --------------------------
> I don't see that function pointer typedefs need to rigidly follow the
> same pattern as the declaration of functions implementing them. Could
> we update the coding style (if needed) instead?

I know one issue with function pointer typedefs is making sure the format
is compatible with Doxygen document generation.  

    https://github.com/tianocore/edk2/tree/master/BaseTools/Scripts/PackageDocumentTools

> 
> Best Regards,
> 
> Leif
> 
> On Mon, Aug 16, 2021 at 16:00:38 -0400, Michael Kubacki wrote:
> > The edk2 branch was created here:
> > https://github.com/makubacki/edk2/tree/uncrustify_poc_2
> >
> > We have a Project Mu fork with custom changes to the Uncrustify tool to help
> > comply with EDK II formatting here:
> > https://dev.azure.com/projectmu/_git/Uncrustify
> >
> > The latest information about the status and how to experiment with the
> > configuration file and the tool are in that fork:
> https://dev.azure.com/projectmu/Uncrustify/_wiki/wikis/Uncrustify.wiki/1/Project-Mu-(EDK-II)-Fork-Readme
> >
> > That said, I have also finished a CI plugin to run Uncrustify that should be
> > ready soon to initially deploy in Project Mu. Before doing so, I am trying
> > to settle on an initial configuration file that less strictly but more
> > reliably formats the code than in the examples in those branches. For
> > example, remove heuristics that when run against the same set of code
> > multiple times can produce different results. An example would be a rule
> > that reformats code because it exceeds a specified column width on one run
> > but on the next run that reformatted code triggers a different rule to
> > further align the code and so on. At least initially, some rules might be
> > tweaked in a more conservative approach that can be tightened in the future.
> > Once this configuration file is ready, we will baseline Project Mu code as
> > an example and turn on the plugin. The CI plugin runs Uncrustify against
> > modified files and if there's any changes, indicating a formatting
> > deviation, the diff chunks are saved in a log so they can be viewed as a
> > build artifact.
> >
> > I am making progress on the updated config file and I should be able to post
> > a "uncrustify_poc_3" branch soon with the results.
> >
> > Regarding indentation, Marvin is right that Uncrustify cannot support edk2
> > indentation style out-of-box. Some changes are made in that fork to handle
> > the formatting. At this point, it can handle the indentation in the cases
> > I've seen. Uncrustify does potentially give us the ability to massively
> > deploy changes across the codebase in case a decision were made to change
> > the style.
> >
> > Thanks,
> > Michael
> >
> > On 8/16/2021 3:39 PM, Marvin Häuser wrote:
> > > Hey Rebecca,
> > >
> > > I think even Uncrustify has issues with the EDK II indentation style.
> > > You might want to check the UEFI Talkbox Discord server, I had a brief
> > > chat with Michael about it there. I don't think realistically any tool
> > > supports EDK II's indentation style however, so I'd propose it is
> > > changed. This could be for new submissions only, or actually the entire
> > > codebase could be reformatted at once with a good tool setup. While this
> > > screws with git blame, the (to my understanding) decided on CRLF -> LF
> > > change does that anyway, so at least two evils could be dealt with in
> > > one go really.
> > >
> > > Best regards,
> > > Marvin
> > >
> > > On 16/08/2021 21:34, Rebecca Cran wrote:
> > > >
> > > > cc devel@ .
> > > >
> > > > On 8/16/21 1:33 PM, Rebecca Cran wrote:
> > > > >
> > > > > I noticed a message on Twitter about an idea of using Uncrustify
> > > > > for EDK2 instead of the ECC tool, and came across https://www.mail-
> archive.com/search?l=devel@edk2.groups.io&q=subject:%22Re%5C%3A+%5C%5Bedk2%5C-
> devel%5C%5D+TianoCore+Community+Meeting+Minutes+%5C-+2%5C%2F4%22&o=newest&f=1
> > > > > .
> > > > >
> > > > > I was wondering if there's been any progress on it that I could
> > > > > check out?
> > > > >
> > > > >
> > > > > Michael Kubacki: in that message, you said:
> > > > >
> > > > > "I'm planning to put up a branch that we can use as a reference
> > > > > for a conversation around uncrustify in the next couple of
> > > > > weeks."
> > > > >
> > > > >
> > > > > Did you end up creating that branch, and if so could you provide
> > > > > a link to it please?
> > > > >
> > > > >
> > > > > --
> > > > > Rebecca Cran
> > > > >
> > > >
> > >
> > >
> > >
> > >
> > >
> >
> >
> >
> > 
> >
> >

Re: [edk2-devel] Progress on getting Uncrustify working for EDK2?

[Michael D Kinney]

Hi Marvin,

Some comments below.

Mike

> -----Original Message-----
> From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Marvin Häuser
> Sent: Thursday, October 7, 2021 11:31 AM
> To: Leif Lindholm <leif@nuviainc.com>; devel@edk2.groups.io; mikuback@linux.microsoft.com
> Cc: rebecca@nuviainc.com; Michael Kubacki <Michael.Kubacki@microsoft.com>; Bret Barkelew <Bret.Barkelew@microsoft.com>;
> Kinney, Michael D <michael.d.kinney@intel.com>
> Subject: Re: [edk2-devel] Progress on getting Uncrustify working for EDK2?
> 
> Good day,
> 
> +1, but while you're at it, can we have arguments not align to the
> function name...
> 
>    Status = Test (
>               a
>               );
> 
> ... but to the next natural indentation level?
> 
>    Status = Test (
>      a
>      );
> 
> Basically no IDE I have seen supports EDK II's style, and I wouldn't be
> keen on writing known-broken style to then rely on Uncrustify to fix it.
> 
> I also have heard some controversy regarding casts off-list, where some
> prefer no spaces after casts to stress the evaluation order, and some
> prefer spaces to have clearer visuals (as a cast *ideally* would be
> something rare that requires good justification). Just throwing that out
> there.
> 
> 
> For things unrelated to autoformat (so semi-offtopic) but still relevant
> to the coding spec:
> 
> 1. Allow STATIC functions (if the debugging concerns are still relevant,
> there could be another level of indirection, like RELEASE_STATIC)?

Debugging concerns are no longer relevant.  The suggestion in the BZ below 
is to remove the STATIC macro and allow EDK II sources to add 'static'
to any functions it makes sense to use on.

    https://bugzilla.tianocore.org/show_bug.cgi?id=1766

> 
> 2. Allow variable assignments on definition (basically non-static CONST
> variables are banned...)?

Are referring to use of pre-initialized CONST variables declared within
a function?  I think Bret brought this topic up when implementing some
unit tests and the suggestion to pass ECCC was to promote them to 
pre-initialized CONST global variables.

The challenges we have seen in the past with pre-initialized variables within
a function is that they can cause compilers to inject use of memcpy() calls,
especially if the variable being initialized on the stack is a structure.
These cause build breaks today.

> 
> 3. Allow variable declarations at any scope (I had some nasty shadowing
> bugs before, probably prohibit shadowing with warnings)?

By shadowing do you mean the declaration of the same variable name in
multiple scoped within the same function?

> 
> 4. Require that exactly all function declarations and all function
> definitions with no prior declaration must be documented (first
> direction is enforcing docs, second is prohibiting doc duplication, I've
> seen them go out-of-sync plenty of times)?

I agree that this can reduce duplication and sync issues.  The uncrustify
tool being discussed here could not help clean this up or enforce this
type of rule.  It is a good topic, but may need to be split out into its
own thread.

> 
> The latter bunch would not require any autoformat rules or reformatation
> of existing code, but would be target only new submissions in my
> opinion. Thoughts?
> 
> 
> Thanks for your efforts!
> 
> Best regards,
> Marvin
> 
> 
> Am 07.10.2021 um 12:48 schrieb Leif Lindholm:
> > Hi Michael,
> >
> > Apologies, I've owed you a response (promised off-list) for a while
> > now.
> >
> > First, let me say I hugely appreciate this effort. Apart from aligning
> > the codebase(s), this will reduce manual reviewing effort
> > substantially, as well as cutting down on number of rework cycles for
> > developers.
> >
> > Looking at the changes to (well, the comments in) uncrustify, this
> > seems to be constrained to:
> > - Newline after '(' for multi-line function calls.
> > - Dealing with "(("/"))" for DEBUG macros.
> > - Function pointer typedefs:
> >    - typedef\nEFIAPI
> >    - closing parentheses indentation
> >
> > I don't think I've made any secret over the years that I am not a
> > massive fan of the EDK2 coding style in general. So I think for any
> > of its quirks that are substantial enough that they require not just
> > custom configuration but actual new function added to existing code
> > conformance tools, this would be an excellent point to sanitise the
> > coding style instead.
> >
> > Taking these in order:
> >
> > Newline after '('
> > -----------------
> > I think we already reached a level of flexibility around this, where
> > we don't actually enforce this (or single argument per
> > line). Personally, I'd be happy to update the coding style as
> > required instead.
> >
> > DEBUG macro parentheses
> > -----------------------
> > How does uncrustify treat DEBUG macros without this modification?
> > Do we start getting everything turned into multi-level indented
> > multi-line statements without this change?
> >
> > Function pointer typedefs:
> > --------------------------
> > I don't see that function pointer typedefs need to rigidly follow the
> > same pattern as the declaration of functions implementing them. Could
> > we update the coding style (if needed) instead?
> >
> > Best Regards,
> >
> > Leif
> >
> > On Mon, Aug 16, 2021 at 16:00:38 -0400, Michael Kubacki wrote:
> >> The edk2 branch was created here:
> >> https://github.com/makubacki/edk2/tree/uncrustify_poc_2
> >>
> >> We have a Project Mu fork with custom changes to the Uncrustify tool to help
> >> comply with EDK II formatting here:
> >> https://dev.azure.com/projectmu/_git/Uncrustify
> >>
> >> The latest information about the status and how to experiment with the
> >> configuration file and the tool are in that fork:
> https://dev.azure.com/projectmu/Uncrustify/_wiki/wikis/Uncrustify.wiki/1/Project-Mu-(EDK-II)-Fork-Readme
> >>
> >> That said, I have also finished a CI plugin to run Uncrustify that should be
> >> ready soon to initially deploy in Project Mu. Before doing so, I am trying
> >> to settle on an initial configuration file that less strictly but more
> >> reliably formats the code than in the examples in those branches. For
> >> example, remove heuristics that when run against the same set of code
> >> multiple times can produce different results. An example would be a rule
> >> that reformats code because it exceeds a specified column width on one run
> >> but on the next run that reformatted code triggers a different rule to
> >> further align the code and so on. At least initially, some rules might be
> >> tweaked in a more conservative approach that can be tightened in the future.
> >> Once this configuration file is ready, we will baseline Project Mu code as
> >> an example and turn on the plugin. The CI plugin runs Uncrustify against
> >> modified files and if there's any changes, indicating a formatting
> >> deviation, the diff chunks are saved in a log so they can be viewed as a
> >> build artifact.
> >>
> >> I am making progress on the updated config file and I should be able to post
> >> a "uncrustify_poc_3" branch soon with the results.
> >>
> >> Regarding indentation, Marvin is right that Uncrustify cannot support edk2
> >> indentation style out-of-box. Some changes are made in that fork to handle
> >> the formatting. At this point, it can handle the indentation in the cases
> >> I've seen. Uncrustify does potentially give us the ability to massively
> >> deploy changes across the codebase in case a decision were made to change
> >> the style.
> >>
> >> Thanks,
> >> Michael
> >>
> >> On 8/16/2021 3:39 PM, Marvin Häuser wrote:
> >>> Hey Rebecca,
> >>>
> >>> I think even Uncrustify has issues with the EDK II indentation style.
> >>> You might want to check the UEFI Talkbox Discord server, I had a brief
> >>> chat with Michael about it there. I don't think realistically any tool
> >>> supports EDK II's indentation style however, so I'd propose it is
> >>> changed. This could be for new submissions only, or actually the entire
> >>> codebase could be reformatted at once with a good tool setup. While this
> >>> screws with git blame, the (to my understanding) decided on CRLF -> LF
> >>> change does that anyway, so at least two evils could be dealt with in
> >>> one go really.
> >>>
> >>> Best regards,
> >>> Marvin
> >>>
> >>> On 16/08/2021 21:34, Rebecca Cran wrote:
> >>>>
> >>>> cc devel@ .
> >>>>
> >>>> On 8/16/21 1:33 PM, Rebecca Cran wrote:
> >>>>>
> >>>>> I noticed a message on Twitter about an idea of using Uncrustify
> >>>>> for EDK2 instead of the ECC tool, and came across https://www.mail-
> archive.com/search?l=devel@edk2.groups.io&q=subject:%22Re%5C%3A+%5C%5Bedk2%5C-
> devel%5C%5D+TianoCore+Community+Meeting+Minutes+%5C-+2%5C%2F4%22&o=newest&f=1
> >>>>> .
> >>>>>
> >>>>> I was wondering if there's been any progress on it that I could
> >>>>> check out?
> >>>>>
> >>>>>
> >>>>> Michael Kubacki: in that message, you said:
> >>>>>
> >>>>> "I'm planning to put up a branch that we can use as a reference
> >>>>> for a conversation around uncrustify in the next couple of
> >>>>> weeks."
> >>>>>
> >>>>>
> >>>>> Did you end up creating that branch, and if so could you provide
> >>>>> a link to it please?
> >>>>>
> >>>>>
> >>>>> --
> >>>>> Rebecca Cran
> >>>>>
> >>>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>
> >>
> >>
> >>
> >>
> >>
> 
> 
> 
> 

Re: [edk2-devel] Progress on getting Uncrustify working for EDK2?

[Andrew Fish]

[-- Attachment #1: Type: text/plain, Size: 11126 bytes --]



> On Oct 7, 2021, at 1:19 PM, Michael D Kinney <michael.d.kinney@intel.com> wrote:
> 
> Hi Marvin,
> 
> Some comments below.
> 
> Mike
> 
>> -----Original Message-----
>> From: devel@edk2.groups.io <mailto:devel@edk2.groups.io> <devel@edk2.groups.io <mailto:devel@edk2.groups.io>> On Behalf Of Marvin Häuser
>> Sent: Thursday, October 7, 2021 11:31 AM
>> To: Leif Lindholm <leif@nuviainc.com <mailto:leif@nuviainc.com>>; devel@edk2.groups.io <mailto:devel@edk2.groups.io>; mikuback@linux.microsoft.com <mailto:mikuback@linux.microsoft.com>
>> Cc: rebecca@nuviainc.com <mailto:rebecca@nuviainc.com>; Michael Kubacki <Michael.Kubacki@microsoft.com <mailto:Michael.Kubacki@microsoft.com>>; Bret Barkelew <Bret.Barkelew@microsoft.com <mailto:Bret.Barkelew@microsoft.com>>;
>> Kinney, Michael D <michael.d.kinney@intel.com <mailto:michael.d.kinney@intel.com>>
>> Subject: Re: [edk2-devel] Progress on getting Uncrustify working for EDK2?
>> 
>> Good day,
>> 
>> +1, but while you're at it, can we have arguments not align to the
>> function name...
>> 
>>   Status = Test (
>>              a
>>              );
>> 
>> ... but to the next natural indentation level?
>> 
>>   Status = Test (
>>     a
>>     );
>> 
>> Basically no IDE I have seen supports EDK II's style, and I wouldn't be
>> keen on writing known-broken style to then rely on Uncrustify to fix it.
>> 
>> I also have heard some controversy regarding casts off-list, where some
>> prefer no spaces after casts to stress the evaluation order, and some
>> prefer spaces to have clearer visuals (as a cast *ideally* would be
>> something rare that requires good justification). Just throwing that out
>> there.
>> 
>> 
>> For things unrelated to autoformat (so semi-offtopic) but still relevant
>> to the coding spec:
>> 
>> 1. Allow STATIC functions (if the debugging concerns are still relevant,
>> there could be another level of indirection, like RELEASE_STATIC)?
> 
> Debugging concerns are no longer relevant.  The suggestion in the BZ below 
> is to remove the STATIC macro and allow EDK II sources to add 'static'
> to any functions it makes sense to use on.
> 
>    https://bugzilla.tianocore.org/show_bug.cgi?id=1766 <https://bugzilla.tianocore.org/show_bug.cgi?id=1766>
> 
>> 
>> 2. Allow variable assignments on definition (basically non-static CONST
>> variables are banned...)?
> 
> Are referring to use of pre-initialized CONST variables declared within
> a function?  I think Bret brought this topic up when implementing some
> unit tests and the suggestion to pass ECCC was to promote them to 
> pre-initialized CONST global variables.
> 
> The challenges we have seen in the past with pre-initialized variables within
> a function is that they can cause compilers to inject use of memcpy() calls,
> especially if the variable being initialized on the stack is a structure.
> These cause build breaks today.

This issue is independent of CONST. I’m not sure a coding style tool is smart enough to catch this generically? You need an understanding of C types to know if the local variable assignment is going to trigger a memcpy(). 

What I’ve seen in the real world is the firmware compiles with -Os or LTO to fit int he ROM for DEBUG and RELEASE, and the optimizer optimizes away the call to memcpy. Then if you try to build NOOPT (or over ride the compiler flags on an individual driver/lib) you fail to link as only the NOOPT build injects the memcpy. 

Thus I think the best way to enforce this rule is to compile a project NOOPT. I’m trying to remember are there flags to built to tell it to compile and skip the FD construction? Maybe we should advocate platforms add a NOOPT build target that just compiles the code, but does not create the FD?

Thanks,

Andrew Fish

> 
>> 
>> 3. Allow variable declarations at any scope (I had some nasty shadowing
>> bugs before, probably prohibit shadowing with warnings)?
> 
> By shadowing do you mean the declaration of the same variable name in
> multiple scoped within the same function?
> 
>> 
>> 4. Require that exactly all function declarations and all function
>> definitions with no prior declaration must be documented (first
>> direction is enforcing docs, second is prohibiting doc duplication, I've
>> seen them go out-of-sync plenty of times)?
> 
> I agree that this can reduce duplication and sync issues.  The uncrustify
> tool being discussed here could not help clean this up or enforce this
> type of rule.  It is a good topic, but may need to be split out into its
> own thread.
> 
>> 
>> The latter bunch would not require any autoformat rules or reformatation
>> of existing code, but would be target only new submissions in my
>> opinion. Thoughts?
>> 
>> 
>> Thanks for your efforts!
>> 
>> Best regards,
>> Marvin
>> 
>> 
>> Am 07.10.2021 um 12:48 schrieb Leif Lindholm:
>>> Hi Michael,
>>> 
>>> Apologies, I've owed you a response (promised off-list) for a while
>>> now.
>>> 
>>> First, let me say I hugely appreciate this effort. Apart from aligning
>>> the codebase(s), this will reduce manual reviewing effort
>>> substantially, as well as cutting down on number of rework cycles for
>>> developers.
>>> 
>>> Looking at the changes to (well, the comments in) uncrustify, this
>>> seems to be constrained to:
>>> - Newline after '(' for multi-line function calls.
>>> - Dealing with "(("/"))" for DEBUG macros.
>>> - Function pointer typedefs:
>>>   - typedef\nEFIAPI
>>>   - closing parentheses indentation
>>> 
>>> I don't think I've made any secret over the years that I am not a
>>> massive fan of the EDK2 coding style in general. So I think for any
>>> of its quirks that are substantial enough that they require not just
>>> custom configuration but actual new function added to existing code
>>> conformance tools, this would be an excellent point to sanitise the
>>> coding style instead.
>>> 
>>> Taking these in order:
>>> 
>>> Newline after '('
>>> -----------------
>>> I think we already reached a level of flexibility around this, where
>>> we don't actually enforce this (or single argument per
>>> line). Personally, I'd be happy to update the coding style as
>>> required instead.
>>> 
>>> DEBUG macro parentheses
>>> -----------------------
>>> How does uncrustify treat DEBUG macros without this modification?
>>> Do we start getting everything turned into multi-level indented
>>> multi-line statements without this change?
>>> 
>>> Function pointer typedefs:
>>> --------------------------
>>> I don't see that function pointer typedefs need to rigidly follow the
>>> same pattern as the declaration of functions implementing them. Could
>>> we update the coding style (if needed) instead?
>>> 
>>> Best Regards,
>>> 
>>> Leif
>>> 
>>> On Mon, Aug 16, 2021 at 16:00:38 -0400, Michael Kubacki wrote:
>>>> The edk2 branch was created here:
>>>> https://github.com/makubacki/edk2/tree/uncrustify_poc_2
>>>> 
>>>> We have a Project Mu fork with custom changes to the Uncrustify tool to help
>>>> comply with EDK II formatting here:
>>>> https://dev.azure.com/projectmu/_git/Uncrustify
>>>> 
>>>> The latest information about the status and how to experiment with the
>>>> configuration file and the tool are in that fork:
>> https://dev.azure.com/projectmu/Uncrustify/_wiki/wikis/Uncrustify.wiki/1/Project-Mu-(EDK-II)-Fork-Readme
>>>> 
>>>> That said, I have also finished a CI plugin to run Uncrustify that should be
>>>> ready soon to initially deploy in Project Mu. Before doing so, I am trying
>>>> to settle on an initial configuration file that less strictly but more
>>>> reliably formats the code than in the examples in those branches. For
>>>> example, remove heuristics that when run against the same set of code
>>>> multiple times can produce different results. An example would be a rule
>>>> that reformats code because it exceeds a specified column width on one run
>>>> but on the next run that reformatted code triggers a different rule to
>>>> further align the code and so on. At least initially, some rules might be
>>>> tweaked in a more conservative approach that can be tightened in the future.
>>>> Once this configuration file is ready, we will baseline Project Mu code as
>>>> an example and turn on the plugin. The CI plugin runs Uncrustify against
>>>> modified files and if there's any changes, indicating a formatting
>>>> deviation, the diff chunks are saved in a log so they can be viewed as a
>>>> build artifact.
>>>> 
>>>> I am making progress on the updated config file and I should be able to post
>>>> a "uncrustify_poc_3" branch soon with the results.
>>>> 
>>>> Regarding indentation, Marvin is right that Uncrustify cannot support edk2
>>>> indentation style out-of-box. Some changes are made in that fork to handle
>>>> the formatting. At this point, it can handle the indentation in the cases
>>>> I've seen. Uncrustify does potentially give us the ability to massively
>>>> deploy changes across the codebase in case a decision were made to change
>>>> the style.
>>>> 
>>>> Thanks,
>>>> Michael
>>>> 
>>>> On 8/16/2021 3:39 PM, Marvin Häuser wrote:
>>>>> Hey Rebecca,
>>>>> 
>>>>> I think even Uncrustify has issues with the EDK II indentation style.
>>>>> You might want to check the UEFI Talkbox Discord server, I had a brief
>>>>> chat with Michael about it there. I don't think realistically any tool
>>>>> supports EDK II's indentation style however, so I'd propose it is
>>>>> changed. This could be for new submissions only, or actually the entire
>>>>> codebase could be reformatted at once with a good tool setup. While this
>>>>> screws with git blame, the (to my understanding) decided on CRLF -> LF
>>>>> change does that anyway, so at least two evils could be dealt with in
>>>>> one go really.
>>>>> 
>>>>> Best regards,
>>>>> Marvin
>>>>> 
>>>>> On 16/08/2021 21:34, Rebecca Cran wrote:
>>>>>> 
>>>>>> cc devel@ .
>>>>>> 
>>>>>> On 8/16/21 1:33 PM, Rebecca Cran wrote:
>>>>>>> 
>>>>>>> I noticed a message on Twitter about an idea of using Uncrustify
>>>>>>> for EDK2 instead of the ECC tool, and came across https://www.mail-
>> archive.com/search?l=devel@edk2.groups.io&q=subject:%22Re%5C%3A+%5C%5Bedk2%5C-
>> devel%5C%5D+TianoCore+Community+Meeting+Minutes+%5C-+2%5C%2F4%22&o=newest&f=1
>>>>>>> .
>>>>>>> 
>>>>>>> I was wondering if there's been any progress on it that I could
>>>>>>> check out?
>>>>>>> 
>>>>>>> 
>>>>>>> Michael Kubacki: in that message, you said:
>>>>>>> 
>>>>>>> "I'm planning to put up a branch that we can use as a reference
>>>>>>> for a conversation around uncrustify in the next couple of
>>>>>>> weeks."
>>>>>>> 
>>>>>>> 
>>>>>>> Did you end up creating that branch, and if so could you provide
>>>>>>> a link to it please?
>>>>>>> 
>>>>>>> 
>>>>>>> --
>>>>>>> Rebecca Cran
>>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>> 
>> 
>> 
>> 
> 
> 
> 
> 


[-- Attachment #2: Type: text/html, Size: 37857 bytes --]

Re: [edk2-devel] Progress on getting Uncrustify working for EDK2?

[Marvin Häuser]

Hey Mike,
Hey Andrew,

I'll just reply to both mails at once :)

On 07/10/2021 19:36, Andrew Fish wrote:
>
>
>> On Oct 7, 2021, at 1:19 PM, Michael D Kinney 
>> <michael.d.kinney@intel.com> wrote:
>>
>> Hi Marvin,
>>
>> Some comments below.
>>
>> Mike
>>
>>> -----Original Message-----
>>> From:devel@edk2.groups.io<devel@edk2.groups.io> On Behalf Of Marvin 
>>> Häuser
>>> Sent: Thursday, October 7, 2021 11:31 AM
>>> To: Leif Lindholm 
>>> <leif@nuviainc.com>;devel@edk2.groups.io;mikuback@linux.microsoft.com
>>> Cc:rebecca@nuviainc.com; Michael Kubacki 
>>> <Michael.Kubacki@microsoft.com>; Bret Barkelew 
>>> <Bret.Barkelew@microsoft.com>;
>>> Kinney, Michael D <michael.d.kinney@intel.com>
>>> Subject: Re: [edk2-devel] Progress on getting Uncrustify working for 
>>> EDK2?
>>>
>>> Good day,
>>>
>>> +1, but while you're at it, can we have arguments not align to the
>>> function name...
>>>
>>>   Status = Test (
>>>              a
>>>              );
>>>
>>> ... but to the next natural indentation level?
>>>
>>>   Status = Test (
>>>     a
>>>     );
>>>
>>> Basically no IDE I have seen supports EDK II's style, and I wouldn't be
>>> keen on writing known-broken style to then rely on Uncrustify to fix it.
>>>
>>> I also have heard some controversy regarding casts off-list, where some
>>> prefer no spaces after casts to stress the evaluation order, and some
>>> prefer spaces to have clearer visuals (as a cast *ideally* would be
>>> something rare that requires good justification). Just throwing that out
>>> there.
>>>
>>>
>>> For things unrelated to autoformat (so semi-offtopic) but still relevant
>>> to the coding spec:
>>>
>>> 1. Allow STATIC functions (if the debugging concerns are still relevant,
>>> there could be another level of indirection, like RELEASE_STATIC)?
>>
>> Debugging concerns are no longer relevant.  The suggestion in the BZ 
>> below
>> is to remove the STATIC macro and allow EDK II sources to add 'static'
>> to any functions it makes sense to use on.
>>
>> https://bugzilla.tianocore.org/show_bug.cgi?id=1766

Thanks! I'd keep STATIC actually just for the sake of not doing no-op 
changes that do not really do anything and for consistency with CONST, 
but whatever works really.

>>
>>>
>>> 2. Allow variable assignments on definition (basically non-static CONST
>>> variables are banned...)?
>>
>> Are referring to use of pre-initialized CONST variables declared within
>> a function?  I think Bret brought this topic up when implementing some
>> unit tests and the suggestion to pass ECCC was to promote them to
>> pre-initialized CONST global variables.

Yes.

>>
>> The challenges we have seen in the past with pre-initialized 
>> variables within
>> a function is that they can cause compilers to inject use of memcpy() 
>> calls,
>> especially if the variable being initialized on the stack is a structure.
>> These cause build breaks today.
>
> This issue is independent of CONST. I’m not sure a coding style tool 
> is smart enough to catch this generically? You need an understanding 
> of C types to know if the local variable assignment is going to 
> trigger a memcpy().
>
> What I’ve seen in the real world is the firmware compiles with -Os or 
> LTO to fit int he ROM for DEBUG and RELEASE, and the optimizer 
> optimizes away the call to memcpy. Then if you try to build NOOPT (or 
> over ride the compiler flags on an individual driver/lib) you fail to 
> link as only the NOOPT build injects the memcpy.

+1

>
> Thus I think the best way to enforce this rule is to compile a project 
> NOOPT. I’m trying to remember are there flags to built to tell it to 
> compile and skip the FD construction? Maybe we should advocate 
> platforms add a NOOPT build target that just compiles the code, but 
> does not create the FD?

I know there were stability concerns with intrinsics in the past, but 
memcpy() is in the standard, and the rest remained stable to my 
knowledge. Maybe it's time to fix the issues at the root? Works for us:
https://github.com/acidanthera/OpenCorePkg/tree/master/Library/OcCompilerIntrinsicsLib

Best regards,
Marvin

>
> Thanks,
>
> Andrew Fish
>
>>
>>>
>>> 3. Allow variable declarations at any scope (I had some nasty shadowing
>>> bugs before, probably prohibit shadowing with warnings)?
>>
>> By shadowing do you mean the declaration of the same variable name in
>> multiple scoped within the same function?
>>
>>>
>>> 4. Require that exactly all function declarations and all function
>>> definitions with no prior declaration must be documented (first
>>> direction is enforcing docs, second is prohibiting doc duplication, I've
>>> seen them go out-of-sync plenty of times)?
>>
>> I agree that this can reduce duplication and sync issues.  The uncrustify
>> tool being discussed here could not help clean this up or enforce this
>> type of rule.  It is a good topic, but may need to be split out into its
>> own thread.
>>
>>>
>>> The latter bunch would not require any autoformat rules or reformatation
>>> of existing code, but would be target only new submissions in my
>>> opinion. Thoughts?
>>>
>>>
>>> Thanks for your efforts!
>>>
>>> Best regards,
>>> Marvin
>>>
>>>
>>> Am 07.10.2021 um 12:48 schrieb Leif Lindholm:
>>>> Hi Michael,
>>>>
>>>> Apologies, I've owed you a response (promised off-list) for a while
>>>> now.
>>>>
>>>> First, let me say I hugely appreciate this effort. Apart from aligning
>>>> the codebase(s), this will reduce manual reviewing effort
>>>> substantially, as well as cutting down on number of rework cycles for
>>>> developers.
>>>>
>>>> Looking at the changes to (well, the comments in) uncrustify, this
>>>> seems to be constrained to:
>>>> - Newline after '(' for multi-line function calls.
>>>> - Dealing with "(("/"))" for DEBUG macros.
>>>> - Function pointer typedefs:
>>>>   - typedef\nEFIAPI
>>>>   - closing parentheses indentation
>>>>
>>>> I don't think I've made any secret over the years that I am not a
>>>> massive fan of the EDK2 coding style in general. So I think for any
>>>> of its quirks that are substantial enough that they require not just
>>>> custom configuration but actual new function added to existing code
>>>> conformance tools, this would be an excellent point to sanitise the
>>>> coding style instead.
>>>>
>>>> Taking these in order:
>>>>
>>>> Newline after '('
>>>> -----------------
>>>> I think we already reached a level of flexibility around this, where
>>>> we don't actually enforce this (or single argument per
>>>> line). Personally, I'd be happy to update the coding style as
>>>> required instead.
>>>>
>>>> DEBUG macro parentheses
>>>> -----------------------
>>>> How does uncrustify treat DEBUG macros without this modification?
>>>> Do we start getting everything turned into multi-level indented
>>>> multi-line statements without this change?
>>>>
>>>> Function pointer typedefs:
>>>> --------------------------
>>>> I don't see that function pointer typedefs need to rigidly follow the
>>>> same pattern as the declaration of functions implementing them. Could
>>>> we update the coding style (if needed) instead?
>>>>
>>>> Best Regards,
>>>>
>>>> Leif
>>>>
>>>> On Mon, Aug 16, 2021 at 16:00:38 -0400, Michael Kubacki wrote:
>>>>> The edk2 branch was created here:
>>>>> https://github.com/makubacki/edk2/tree/uncrustify_poc_2
>>>>>
>>>>> We have a Project Mu fork with custom changes to the Uncrustify 
>>>>> tool to help
>>>>> comply with EDK II formatting here:
>>>>> https://dev.azure.com/projectmu/_git/Uncrustify
>>>>>
>>>>> The latest information about the status and how to experiment with the
>>>>> configuration file and the tool are in that fork:
>>> https://dev.azure.com/projectmu/Uncrustify/_wiki/wikis/Uncrustify.wiki/1/Project-Mu-(EDK-II)-Fork-Readme
>>>>>
>>>>> That said, I have also finished a CI plugin to run Uncrustify that 
>>>>> should be
>>>>> ready soon to initially deploy in Project Mu. Before doing so, I 
>>>>> am trying
>>>>> to settle on an initial configuration file that less strictly but more
>>>>> reliably formats the code than in the examples in those branches. For
>>>>> example, remove heuristics that when run against the same set of code
>>>>> multiple times can produce different results. An example would be 
>>>>> a rule
>>>>> that reformats code because it exceeds a specified column width on 
>>>>> one run
>>>>> but on the next run that reformatted code triggers a different rule to
>>>>> further align the code and so on. At least initially, some rules 
>>>>> might be
>>>>> tweaked in a more conservative approach that can be tightened in 
>>>>> the future.
>>>>> Once this configuration file is ready, we will baseline Project Mu 
>>>>> code as
>>>>> an example and turn on the plugin. The CI plugin runs Uncrustify 
>>>>> against
>>>>> modified files and if there's any changes, indicating a formatting
>>>>> deviation, the diff chunks are saved in a log so they can be 
>>>>> viewed as a
>>>>> build artifact.
>>>>>
>>>>> I am making progress on the updated config file and I should be 
>>>>> able to post
>>>>> a "uncrustify_poc_3" branch soon with the results.
>>>>>
>>>>> Regarding indentation, Marvin is right that Uncrustify cannot 
>>>>> support edk2
>>>>> indentation style out-of-box. Some changes are made in that fork 
>>>>> to handle
>>>>> the formatting. At this point, it can handle the indentation in 
>>>>> the cases
>>>>> I've seen. Uncrustify does potentially give us the ability to 
>>>>> massively
>>>>> deploy changes across the codebase in case a decision were made to 
>>>>> change
>>>>> the style.
>>>>>
>>>>> Thanks,
>>>>> Michael
>>>>>
>>>>> On 8/16/2021 3:39 PM, Marvin Häuser wrote:
>>>>>> Hey Rebecca,
>>>>>>
>>>>>> I think even Uncrustify has issues with the EDK II indentation style.
>>>>>> You might want to check the UEFI Talkbox Discord server, I had a 
>>>>>> brief
>>>>>> chat with Michael about it there. I don't think realistically any 
>>>>>> tool
>>>>>> supports EDK II's indentation style however, so I'd propose it is
>>>>>> changed. This could be for new submissions only, or actually the 
>>>>>> entire
>>>>>> codebase could be reformatted at once with a good tool setup. 
>>>>>> While this
>>>>>> screws with git blame, the (to my understanding) decided on CRLF 
>>>>>> -> LF
>>>>>> change does that anyway, so at least two evils could be dealt with in
>>>>>> one go really.
>>>>>>
>>>>>> Best regards,
>>>>>> Marvin
>>>>>>
>>>>>> On 16/08/2021 21:34, Rebecca Cran wrote:
>>>>>>>
>>>>>>> cc devel@ .
>>>>>>>
>>>>>>> On 8/16/21 1:33 PM, Rebecca Cran wrote:
>>>>>>>>
>>>>>>>> I noticed a message on Twitter about an idea of using Uncrustify
>>>>>>>> for EDK2 instead of the ECC tool, and came across https://www.mail-
>>> archive.com/search?l=devel@edk2.groups.io&q=subject:%22Re%5C%3A+%5C%5Bedk2%5C-
>>> devel%5C%5D+TianoCore+Community+Meeting+Minutes+%5C-+2%5C%2F4%22&o=newest&f=1
>>>>>>>> .
>>>>>>>>
>>>>>>>> I was wondering if there's been any progress on it that I could
>>>>>>>> check out?
>>>>>>>>
>>>>>>>>
>>>>>>>> Michael Kubacki: in that message, you said:
>>>>>>>>
>>>>>>>> "I'm planning to put up a branch that we can use as a reference
>>>>>>>> for a conversation around uncrustify in the next couple of
>>>>>>>> weeks."
>>>>>>>>
>>>>>>>>
>>>>>>>> Did you end up creating that branch, and if so could you provide
>>>>>>>> a link to it please?
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Rebecca Cran
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>
>>>
>>>
>>>
>>
>>
>>
>> 
>

Re: [edk2-devel] Progress on getting Uncrustify working for EDK2?

[Marvin Häuser]

Good day,

+1, but while you're at it, can we have arguments not align to the 
function name...

   Status = Test (
              a
              );

... but to the next natural indentation level?

   Status = Test (
     a
     );

Basically no IDE I have seen supports EDK II's style, and I wouldn't be 
keen on writing known-broken style to then rely on Uncrustify to fix it.

I also have heard some controversy regarding casts off-list, where some 
prefer no spaces after casts to stress the evaluation order, and some 
prefer spaces to have clearer visuals (as a cast *ideally* would be 
something rare that requires good justification). Just throwing that out 
there.


For things unrelated to autoformat (so semi-offtopic) but still relevant 
to the coding spec:

1. Allow STATIC functions (if the debugging concerns are still relevant, 
there could be another level of indirection, like RELEASE_STATIC)?

2. Allow variable assignments on definition (basically non-static CONST 
variables are banned...)?

3. Allow variable declarations at any scope (I had some nasty shadowing 
bugs before, probably prohibit shadowing with warnings)?

4. Require that exactly all function declarations and all function 
definitions with no prior declaration must be documented (first 
direction is enforcing docs, second is prohibiting doc duplication, I've 
seen them go out-of-sync plenty of times)?

The latter bunch would not require any autoformat rules or reformatation 
of existing code, but would be target only new submissions in my 
opinion. Thoughts?


Thanks for your efforts!

Best regards,
Marvin


Am 07.10.2021 um 12:48 schrieb Leif Lindholm:
> Hi Michael,
> 
> Apologies, I've owed you a response (promised off-list) for a while
> now.
> 
> First, let me say I hugely appreciate this effort. Apart from aligning
> the codebase(s), this will reduce manual reviewing effort
> substantially, as well as cutting down on number of rework cycles for
> developers.
> 
> Looking at the changes to (well, the comments in) uncrustify, this
> seems to be constrained to:
> - Newline after '(' for multi-line function calls.
> - Dealing with "(("/"))" for DEBUG macros.
> - Function pointer typedefs:
>    - typedef\nEFIAPI
>    - closing parentheses indentation
> 
> I don't think I've made any secret over the years that I am not a
> massive fan of the EDK2 coding style in general. So I think for any
> of its quirks that are substantial enough that they require not just
> custom configuration but actual new function added to existing code
> conformance tools, this would be an excellent point to sanitise the
> coding style instead.
> 
> Taking these in order:
> 
> Newline after '('
> -----------------
> I think we already reached a level of flexibility around this, where
> we don't actually enforce this (or single argument per
> line). Personally, I'd be happy to update the coding style as
> required instead.
> 
> DEBUG macro parentheses
> -----------------------
> How does uncrustify treat DEBUG macros without this modification?
> Do we start getting everything turned into multi-level indented
> multi-line statements without this change?
> 
> Function pointer typedefs:
> --------------------------
> I don't see that function pointer typedefs need to rigidly follow the
> same pattern as the declaration of functions implementing them. Could
> we update the coding style (if needed) instead?
> 
> Best Regards,
> 
> Leif
> 
> On Mon, Aug 16, 2021 at 16:00:38 -0400, Michael Kubacki wrote:
>> The edk2 branch was created here:
>> https://github.com/makubacki/edk2/tree/uncrustify_poc_2
>>
>> We have a Project Mu fork with custom changes to the Uncrustify tool to help
>> comply with EDK II formatting here:
>> https://dev.azure.com/projectmu/_git/Uncrustify
>>
>> The latest information about the status and how to experiment with the
>> configuration file and the tool are in that fork: https://dev.azure.com/projectmu/Uncrustify/_wiki/wikis/Uncrustify.wiki/1/Project-Mu-(EDK-II)-Fork-Readme
>>
>> That said, I have also finished a CI plugin to run Uncrustify that should be
>> ready soon to initially deploy in Project Mu. Before doing so, I am trying
>> to settle on an initial configuration file that less strictly but more
>> reliably formats the code than in the examples in those branches. For
>> example, remove heuristics that when run against the same set of code
>> multiple times can produce different results. An example would be a rule
>> that reformats code because it exceeds a specified column width on one run
>> but on the next run that reformatted code triggers a different rule to
>> further align the code and so on. At least initially, some rules might be
>> tweaked in a more conservative approach that can be tightened in the future.
>> Once this configuration file is ready, we will baseline Project Mu code as
>> an example and turn on the plugin. The CI plugin runs Uncrustify against
>> modified files and if there's any changes, indicating a formatting
>> deviation, the diff chunks are saved in a log so they can be viewed as a
>> build artifact.
>>
>> I am making progress on the updated config file and I should be able to post
>> a "uncrustify_poc_3" branch soon with the results.
>>
>> Regarding indentation, Marvin is right that Uncrustify cannot support edk2
>> indentation style out-of-box. Some changes are made in that fork to handle
>> the formatting. At this point, it can handle the indentation in the cases
>> I've seen. Uncrustify does potentially give us the ability to massively
>> deploy changes across the codebase in case a decision were made to change
>> the style.
>>
>> Thanks,
>> Michael
>>
>> On 8/16/2021 3:39 PM, Marvin Häuser wrote:
>>> Hey Rebecca,
>>>
>>> I think even Uncrustify has issues with the EDK II indentation style.
>>> You might want to check the UEFI Talkbox Discord server, I had a brief
>>> chat with Michael about it there. I don't think realistically any tool
>>> supports EDK II's indentation style however, so I'd propose it is
>>> changed. This could be for new submissions only, or actually the entire
>>> codebase could be reformatted at once with a good tool setup. While this
>>> screws with git blame, the (to my understanding) decided on CRLF -> LF
>>> change does that anyway, so at least two evils could be dealt with in
>>> one go really.
>>>
>>> Best regards,
>>> Marvin
>>>
>>> On 16/08/2021 21:34, Rebecca Cran wrote:
>>>>
>>>> cc devel@ .
>>>>
>>>> On 8/16/21 1:33 PM, Rebecca Cran wrote:
>>>>>
>>>>> I noticed a message on Twitter about an idea of using Uncrustify
>>>>> for EDK2 instead of the ECC tool, and came across https://www.mail-archive.com/search?l=devel@edk2.groups.io&q=subject:%22Re%5C%3A+%5C%5Bedk2%5C-devel%5C%5D+TianoCore+Community+Meeting+Minutes+%5C-+2%5C%2F4%22&o=newest&f=1
>>>>> .
>>>>>
>>>>> I was wondering if there's been any progress on it that I could
>>>>> check out?
>>>>>
>>>>>
>>>>> Michael Kubacki: in that message, you said:
>>>>>
>>>>> "I'm planning to put up a branch that we can use as a reference
>>>>> for a conversation around uncrustify in the next couple of
>>>>> weeks."
>>>>>
>>>>>
>>>>> Did you end up creating that branch, and if so could you provide
>>>>> a link to it please?
>>>>>
>>>>>
>>>>> -- 
>>>>> Rebecca Cran
>>>>>
>>>>
>>>
>>>
>>>
>>>
>>>
>>
>>
>>
>> 
>>
>>

Re: [edk2-devel] Progress on getting Uncrustify working for EDK2?

[Andrew Fish]

[-- Attachment #1: Type: text/plain, Size: 12466 bytes --]



> On Oct 7, 2021, at 1:43 PM, Marvin Häuser <mhaeuser@posteo.de> wrote:
> 
> Hey Mike,
> Hey Andrew,
> 
> I'll just reply to both mails at once :)
> 
> On 07/10/2021 19:36, Andrew Fish wrote:
>> 
>> 
>>> On Oct 7, 2021, at 1:19 PM, Michael D Kinney <michael.d.kinney@intel.com> wrote:
>>> 
>>> Hi Marvin,
>>> 
>>> Some comments below.
>>> 
>>> Mike
>>> 
>>>> -----Original Message-----
>>>> From:devel@edk2.groups.io<devel@edk2.groups.io> On Behalf Of Marvin Häuser
>>>> Sent: Thursday, October 7, 2021 11:31 AM
>>>> To: Leif Lindholm <leif@nuviainc.com>;devel@edk2.groups.io;mikuback@linux.microsoft.com
>>>> Cc:rebecca@nuviainc.com; Michael Kubacki <Michael.Kubacki@microsoft.com>; Bret Barkelew <Bret.Barkelew@microsoft.com>;
>>>> Kinney, Michael D <michael.d.kinney@intel.com>
>>>> Subject: Re: [edk2-devel] Progress on getting Uncrustify working for EDK2?
>>>> 
>>>> Good day,
>>>> 
>>>> +1, but while you're at it, can we have arguments not align to the
>>>> function name...
>>>> 
>>>>   Status = Test (
>>>>              a
>>>>              );
>>>> 
>>>> ... but to the next natural indentation level?
>>>> 
>>>>   Status = Test (
>>>>     a
>>>>     );
>>>> 
>>>> Basically no IDE I have seen supports EDK II's style, and I wouldn't be
>>>> keen on writing known-broken style to then rely on Uncrustify to fix it.
>>>> 
>>>> I also have heard some controversy regarding casts off-list, where some
>>>> prefer no spaces after casts to stress the evaluation order, and some
>>>> prefer spaces to have clearer visuals (as a cast *ideally* would be
>>>> something rare that requires good justification). Just throwing that out
>>>> there.
>>>> 
>>>> 
>>>> For things unrelated to autoformat (so semi-offtopic) but still relevant
>>>> to the coding spec:
>>>> 
>>>> 1. Allow STATIC functions (if the debugging concerns are still relevant,
>>>> there could be another level of indirection, like RELEASE_STATIC)?
>>> 
>>> Debugging concerns are no longer relevant.  The suggestion in the BZ below
>>> is to remove the STATIC macro and allow EDK II sources to add 'static'
>>> to any functions it makes sense to use on.
>>> 
>>> https://bugzilla.tianocore.org/show_bug.cgi?id=1766
> 
> Thanks! I'd keep STATIC actually just for the sake of not doing no-op changes that do not really do anything and for consistency with CONST, but whatever works really.
> 
>>> 
>>>> 
>>>> 2. Allow variable assignments on definition (basically non-static CONST
>>>> variables are banned...)?
>>> 
>>> Are referring to use of pre-initialized CONST variables declared within
>>> a function?  I think Bret brought this topic up when implementing some
>>> unit tests and the suggestion to pass ECCC was to promote them to
>>> pre-initialized CONST global variables.
> 
> Yes.
> 
>>> 
>>> The challenges we have seen in the past with pre-initialized variables within
>>> a function is that they can cause compilers to inject use of memcpy() calls,
>>> especially if the variable being initialized on the stack is a structure.
>>> These cause build breaks today.
>> 
>> This issue is independent of CONST. I’m not sure a coding style tool is smart enough to catch this generically? You need an understanding of C types to know if the local variable assignment is going to trigger a memcpy().
>> 
>> What I’ve seen in the real world is the firmware compiles with -Os or LTO to fit int he ROM for DEBUG and RELEASE, and the optimizer optimizes away the call to memcpy. Then if you try to build NOOPT (or over ride the compiler flags on an individual driver/lib) you fail to link as only the NOOPT build injects the memcpy.
> 
> +1
> 
>> 
>> Thus I think the best way to enforce this rule is to compile a project NOOPT. I’m trying to remember are there flags to built to tell it to compile and skip the FD construction? Maybe we should advocate platforms add a NOOPT build target that just compiles the code, but does not create the FD?
> 
> I know there were stability concerns with intrinsics in the past, but memcpy() is in the standard, and the rest remained stable to my knowledge. Maybe it's time to fix the issues at the root? Works for us:
> https://github.com/acidanthera/OpenCorePkg/tree/master/Library/OcCompilerIntrinsicsLib <https://github.com/acidanthera/OpenCorePkg/tree/master/Library/OcCompilerIntrinsicsLib>
> 

Marvin,

Good point. This would make the rule moot. So maybe just removing the requirement would be the easiest long term fix. 

Other embedded projects I know of do this too, and as you point out the compilers keep these APIs standard for folks the provide their own runtimes.

Thanks,

Andrew Fish

> Best regards,
> Marvin
> 
>> 
>> Thanks,
>> 
>> Andrew Fish
>> 
>>> 
>>>> 
>>>> 3. Allow variable declarations at any scope (I had some nasty shadowing
>>>> bugs before, probably prohibit shadowing with warnings)?
>>> 
>>> By shadowing do you mean the declaration of the same variable name in
>>> multiple scoped within the same function?
>>> 
>>>> 
>>>> 4. Require that exactly all function declarations and all function
>>>> definitions with no prior declaration must be documented (first
>>>> direction is enforcing docs, second is prohibiting doc duplication, I've
>>>> seen them go out-of-sync plenty of times)?
>>> 
>>> I agree that this can reduce duplication and sync issues.  The uncrustify
>>> tool being discussed here could not help clean this up or enforce this
>>> type of rule.  It is a good topic, but may need to be split out into its
>>> own thread.
>>> 
>>>> 
>>>> The latter bunch would not require any autoformat rules or reformatation
>>>> of existing code, but would be target only new submissions in my
>>>> opinion. Thoughts?
>>>> 
>>>> 
>>>> Thanks for your efforts!
>>>> 
>>>> Best regards,
>>>> Marvin
>>>> 
>>>> 
>>>> Am 07.10.2021 um 12:48 schrieb Leif Lindholm:
>>>>> Hi Michael,
>>>>> 
>>>>> Apologies, I've owed you a response (promised off-list) for a while
>>>>> now.
>>>>> 
>>>>> First, let me say I hugely appreciate this effort. Apart from aligning
>>>>> the codebase(s), this will reduce manual reviewing effort
>>>>> substantially, as well as cutting down on number of rework cycles for
>>>>> developers.
>>>>> 
>>>>> Looking at the changes to (well, the comments in) uncrustify, this
>>>>> seems to be constrained to:
>>>>> - Newline after '(' for multi-line function calls.
>>>>> - Dealing with "(("/"))" for DEBUG macros.
>>>>> - Function pointer typedefs:
>>>>>   - typedef\nEFIAPI
>>>>>   - closing parentheses indentation
>>>>> 
>>>>> I don't think I've made any secret over the years that I am not a
>>>>> massive fan of the EDK2 coding style in general. So I think for any
>>>>> of its quirks that are substantial enough that they require not just
>>>>> custom configuration but actual new function added to existing code
>>>>> conformance tools, this would be an excellent point to sanitise the
>>>>> coding style instead.
>>>>> 
>>>>> Taking these in order:
>>>>> 
>>>>> Newline after '('
>>>>> -----------------
>>>>> I think we already reached a level of flexibility around this, where
>>>>> we don't actually enforce this (or single argument per
>>>>> line). Personally, I'd be happy to update the coding style as
>>>>> required instead.
>>>>> 
>>>>> DEBUG macro parentheses
>>>>> -----------------------
>>>>> How does uncrustify treat DEBUG macros without this modification?
>>>>> Do we start getting everything turned into multi-level indented
>>>>> multi-line statements without this change?
>>>>> 
>>>>> Function pointer typedefs:
>>>>> --------------------------
>>>>> I don't see that function pointer typedefs need to rigidly follow the
>>>>> same pattern as the declaration of functions implementing them. Could
>>>>> we update the coding style (if needed) instead?
>>>>> 
>>>>> Best Regards,
>>>>> 
>>>>> Leif
>>>>> 
>>>>> On Mon, Aug 16, 2021 at 16:00:38 -0400, Michael Kubacki wrote:
>>>>>> The edk2 branch was created here:
>>>>>> https://github.com/makubacki/edk2/tree/uncrustify_poc_2
>>>>>> 
>>>>>> We have a Project Mu fork with custom changes to the Uncrustify tool to help
>>>>>> comply with EDK II formatting here:
>>>>>> https://dev.azure.com/projectmu/_git/Uncrustify
>>>>>> 
>>>>>> The latest information about the status and how to experiment with the
>>>>>> configuration file and the tool are in that fork:
>>>> https://dev.azure.com/projectmu/Uncrustify/_wiki/wikis/Uncrustify.wiki/1/Project-Mu-(EDK-II)-Fork-Readme
>>>>>> 
>>>>>> That said, I have also finished a CI plugin to run Uncrustify that should be
>>>>>> ready soon to initially deploy in Project Mu. Before doing so, I am trying
>>>>>> to settle on an initial configuration file that less strictly but more
>>>>>> reliably formats the code than in the examples in those branches. For
>>>>>> example, remove heuristics that when run against the same set of code
>>>>>> multiple times can produce different results. An example would be a rule
>>>>>> that reformats code because it exceeds a specified column width on one run
>>>>>> but on the next run that reformatted code triggers a different rule to
>>>>>> further align the code and so on. At least initially, some rules might be
>>>>>> tweaked in a more conservative approach that can be tightened in the future.
>>>>>> Once this configuration file is ready, we will baseline Project Mu code as
>>>>>> an example and turn on the plugin. The CI plugin runs Uncrustify against
>>>>>> modified files and if there's any changes, indicating a formatting
>>>>>> deviation, the diff chunks are saved in a log so they can be viewed as a
>>>>>> build artifact.
>>>>>> 
>>>>>> I am making progress on the updated config file and I should be able to post
>>>>>> a "uncrustify_poc_3" branch soon with the results.
>>>>>> 
>>>>>> Regarding indentation, Marvin is right that Uncrustify cannot support edk2
>>>>>> indentation style out-of-box. Some changes are made in that fork to handle
>>>>>> the formatting. At this point, it can handle the indentation in the cases
>>>>>> I've seen. Uncrustify does potentially give us the ability to massively
>>>>>> deploy changes across the codebase in case a decision were made to change
>>>>>> the style.
>>>>>> 
>>>>>> Thanks,
>>>>>> Michael
>>>>>> 
>>>>>> On 8/16/2021 3:39 PM, Marvin Häuser wrote:
>>>>>>> Hey Rebecca,
>>>>>>> 
>>>>>>> I think even Uncrustify has issues with the EDK II indentation style.
>>>>>>> You might want to check the UEFI Talkbox Discord server, I had a brief
>>>>>>> chat with Michael about it there. I don't think realistically any tool
>>>>>>> supports EDK II's indentation style however, so I'd propose it is
>>>>>>> changed. This could be for new submissions only, or actually the entire
>>>>>>> codebase could be reformatted at once with a good tool setup. While this
>>>>>>> screws with git blame, the (to my understanding) decided on CRLF -> LF
>>>>>>> change does that anyway, so at least two evils could be dealt with in
>>>>>>> one go really.
>>>>>>> 
>>>>>>> Best regards,
>>>>>>> Marvin
>>>>>>> 
>>>>>>> On 16/08/2021 21:34, Rebecca Cran wrote:
>>>>>>>> 
>>>>>>>> cc devel@ .
>>>>>>>> 
>>>>>>>> On 8/16/21 1:33 PM, Rebecca Cran wrote:
>>>>>>>>> 
>>>>>>>>> I noticed a message on Twitter about an idea of using Uncrustify
>>>>>>>>> for EDK2 instead of the ECC tool, and came across https://www.mail-
>>>> archive.com/search?l=devel@edk2.groups.io&q=subject:%22Re%5C%3A+%5C%5Bedk2%5C-
>>>> devel%5C%5D+TianoCore+Community+Meeting+Minutes+%5C-+2%5C%2F4%22&o=newest&f=1
>>>>>>>>> .
>>>>>>>>> 
>>>>>>>>> I was wondering if there's been any progress on it that I could
>>>>>>>>> check out?
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> Michael Kubacki: in that message, you said:
>>>>>>>>> 
>>>>>>>>> "I'm planning to put up a branch that we can use as a reference
>>>>>>>>> for a conversation around uncrustify in the next couple of
>>>>>>>>> weeks."
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> Did you end up creating that branch, and if so could you provide
>>>>>>>>> a link to it please?
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> --
>>>>>>>>> Rebecca Cran
>>>>>>>>> 
>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>> 
>>> 
>>> 
>>> 


[-- Attachment #2: Type: text/html, Size: 29720 bytes --]

Re: [edk2-devel] Progress on getting Uncrustify working for EDK2?

[Michael D Kinney]

[-- Attachment #1: Type: text/plain, Size: 13243 bytes --]

Hello,

Good information in this thread on code style.

Some of the topics apply to uncrustify and some are out of scope for what uncrustify can fix on its own.

I would like to focus on a date to convert all source code in edk2 repo using the uncrustify tool and to capture the other code style topics into their own thread and bugzillas.

I would like to propose a conversion date for uncrustify immediately after the edk2-stable202111 release on 2021-11-26.

I have been working with Michael Kubacki on a build comparison tool that verifies that the build generate the same obj/lib/dll/efi/fv/fd files before and after the uncrustify changes.  We would run and publish the results from this tool before committing the changes.

We need TianoCore community approval of the following:


  1.  Approve format of C source generated by the uncrustify.
  2.  Approve uncrustify changes right after edk2-stable-202111 release.
     *   Extend code freeze until these changes are committed.
  3.  Require use of uncrustify tool before submitting patch review emails or PRs.
     *   The required version would be a formally released version  from the fork maintained by Michael Kubacki until the changes can be upstreamed.
     *   https://dev.azure.com/projectmu/Uncrustify
  4.  Add EDK II CI check to verify that all PRs submitted exactly match uncrustified version.  Reject PRs that do not match exactly.
  5.  Implement a git hook available that would automatically run uncristufy before committing changes to a local branch of an edk2 repo.

Thanks,

Mike

From: Andrew Fish <afish@apple.com>
Sent: Thursday, October 7, 2021 2:09 PM
To: Marvin Häuser <mhaeuser@posteo.de>
Cc: edk2-devel-groups-io <devel@edk2.groups.io>; Kinney, Michael D <michael.d.kinney@intel.com>; Leif Lindholm <leif@nuviainc.com>; mikuback@linux.microsoft.com; rebecca@nuviainc.com; Michael Kubacki <Michael.Kubacki@microsoft.com>; Bret Barkelew <Bret.Barkelew@microsoft.com>
Subject: Re: [edk2-devel] Progress on getting Uncrustify working for EDK2?




On Oct 7, 2021, at 1:43 PM, Marvin Häuser <mhaeuser@posteo.de<mailto:mhaeuser@posteo.de>> wrote:

Hey Mike,
Hey Andrew,

I'll just reply to both mails at once :)

On 07/10/2021 19:36, Andrew Fish wrote:




On Oct 7, 2021, at 1:19 PM, Michael D Kinney <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>> wrote:

Hi Marvin,

Some comments below.

Mike


-----Original Message-----
From:devel@edk2.groups.io<mailto:devel@edk2.groups.io><devel@edk2.groups.io<mailto:devel@edk2.groups.io>> On Behalf Of Marvin Häuser
Sent: Thursday, October 7, 2021 11:31 AM
To: Leif Lindholm <leif@nuviainc.com<mailto:leif@nuviainc.com>>;devel@edk2.groups.io<mailto:devel@edk2.groups.io>;mikuback@linux.microsoft.com<mailto:mikuback@linux.microsoft.com>
Cc:rebecca@nuviainc.com<mailto:rebecca@nuviainc.com>; Michael Kubacki <Michael.Kubacki@microsoft.com<mailto:Michael.Kubacki@microsoft.com>>; Bret Barkelew <Bret.Barkelew@microsoft.com<mailto:Bret.Barkelew@microsoft.com>>;
Kinney, Michael D <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>>
Subject: Re: [edk2-devel] Progress on getting Uncrustify working for EDK2?

Good day,

+1, but while you're at it, can we have arguments not align to the
function name...

  Status = Test (
             a
             );

... but to the next natural indentation level?

  Status = Test (
    a
    );

Basically no IDE I have seen supports EDK II's style, and I wouldn't be
keen on writing known-broken style to then rely on Uncrustify to fix it.

I also have heard some controversy regarding casts off-list, where some
prefer no spaces after casts to stress the evaluation order, and some
prefer spaces to have clearer visuals (as a cast *ideally* would be
something rare that requires good justification). Just throwing that out
there.


For things unrelated to autoformat (so semi-offtopic) but still relevant
to the coding spec:

1. Allow STATIC functions (if the debugging concerns are still relevant,
there could be another level of indirection, like RELEASE_STATIC)?

Debugging concerns are no longer relevant.  The suggestion in the BZ below
is to remove the STATIC macro and allow EDK II sources to add 'static'
to any functions it makes sense to use on.

https://bugzilla.tianocore.org/show_bug.cgi?id=1766

Thanks! I'd keep STATIC actually just for the sake of not doing no-op changes that do not really do anything and for consistency with CONST, but whatever works really.





2. Allow variable assignments on definition (basically non-static CONST
variables are banned...)?

Are referring to use of pre-initialized CONST variables declared within
a function?  I think Bret brought this topic up when implementing some
unit tests and the suggestion to pass ECCC was to promote them to
pre-initialized CONST global variables.

Yes.



The challenges we have seen in the past with pre-initialized variables within
a function is that they can cause compilers to inject use of memcpy() calls,
especially if the variable being initialized on the stack is a structure.
These cause build breaks today.

This issue is independent of CONST. I’m not sure a coding style tool is smart enough to catch this generically? You need an understanding of C types to know if the local variable assignment is going to trigger a memcpy().

What I’ve seen in the real world is the firmware compiles with -Os or LTO to fit int he ROM for DEBUG and RELEASE, and the optimizer optimizes away the call to memcpy. Then if you try to build NOOPT (or over ride the compiler flags on an individual driver/lib) you fail to link as only the NOOPT build injects the memcpy.

+1



Thus I think the best way to enforce this rule is to compile a project NOOPT. I’m trying to remember are there flags to built to tell it to compile and skip the FD construction? Maybe we should advocate platforms add a NOOPT build target that just compiles the code, but does not create the FD?

I know there were stability concerns with intrinsics in the past, but memcpy() is in the standard, and the rest remained stable to my knowledge. Maybe it's time to fix the issues at the root? Works for us:
https://github.com/acidanthera/OpenCorePkg/tree/master/Library/OcCompilerIntrinsicsLib

Marvin,

Good point. This would make the rule moot. So maybe just removing the requirement would be the easiest long term fix.

Other embedded projects I know of do this too, and as you point out the compilers keep these APIs standard for folks the provide their own runtimes.

Thanks,

Andrew Fish


Best regards,
Marvin



Thanks,

Andrew Fish





3. Allow variable declarations at any scope (I had some nasty shadowing
bugs before, probably prohibit shadowing with warnings)?

By shadowing do you mean the declaration of the same variable name in
multiple scoped within the same function?



4. Require that exactly all function declarations and all function
definitions with no prior declaration must be documented (first
direction is enforcing docs, second is prohibiting doc duplication, I've
seen them go out-of-sync plenty of times)?

I agree that this can reduce duplication and sync issues.  The uncrustify
tool being discussed here could not help clean this up or enforce this
type of rule.  It is a good topic, but may need to be split out into its
own thread.



The latter bunch would not require any autoformat rules or reformatation
of existing code, but would be target only new submissions in my
opinion. Thoughts?


Thanks for your efforts!

Best regards,
Marvin


Am 07.10.2021 um 12:48 schrieb Leif Lindholm:

Hi Michael,

Apologies, I've owed you a response (promised off-list) for a while
now.

First, let me say I hugely appreciate this effort. Apart from aligning
the codebase(s), this will reduce manual reviewing effort
substantially, as well as cutting down on number of rework cycles for
developers.

Looking at the changes to (well, the comments in) uncrustify, this
seems to be constrained to:
- Newline after '(' for multi-line function calls.
- Dealing with "(("/"))" for DEBUG macros.
- Function pointer typedefs:
  - typedef\nEFIAPI
  - closing parentheses indentation

I don't think I've made any secret over the years that I am not a
massive fan of the EDK2 coding style in general. So I think for any
of its quirks that are substantial enough that they require not just
custom configuration but actual new function added to existing code
conformance tools, this would be an excellent point to sanitise the
coding style instead.

Taking these in order:

Newline after '('
-----------------
I think we already reached a level of flexibility around this, where
we don't actually enforce this (or single argument per
line). Personally, I'd be happy to update the coding style as
required instead.

DEBUG macro parentheses
-----------------------
How does uncrustify treat DEBUG macros without this modification?
Do we start getting everything turned into multi-level indented
multi-line statements without this change?

Function pointer typedefs:
--------------------------
I don't see that function pointer typedefs need to rigidly follow the
same pattern as the declaration of functions implementing them. Could
we update the coding style (if needed) instead?

Best Regards,

Leif

On Mon, Aug 16, 2021 at 16:00:38 -0400, Michael Kubacki wrote:

The edk2 branch was created here:
https://github.com/makubacki/edk2/tree/uncrustify_poc_2

We have a Project Mu fork with custom changes to the Uncrustify tool to help
comply with EDK II formatting here:
https://dev.azure.com/projectmu/_git/Uncrustify

The latest information about the status and how to experiment with the
configuration file and the tool are in that fork:
https://dev.azure.com/projectmu/Uncrustify/_wiki/wikis/Uncrustify.wiki/1/Project-Mu-(EDK-II)-Fork-Readme


That said, I have also finished a CI plugin to run Uncrustify that should be
ready soon to initially deploy in Project Mu. Before doing so, I am trying
to settle on an initial configuration file that less strictly but more
reliably formats the code than in the examples in those branches. For
example, remove heuristics that when run against the same set of code
multiple times can produce different results. An example would be a rule
that reformats code because it exceeds a specified column width on one run
but on the next run that reformatted code triggers a different rule to
further align the code and so on. At least initially, some rules might be
tweaked in a more conservative approach that can be tightened in the future.
Once this configuration file is ready, we will baseline Project Mu code as
an example and turn on the plugin. The CI plugin runs Uncrustify against
modified files and if there's any changes, indicating a formatting
deviation, the diff chunks are saved in a log so they can be viewed as a
build artifact.

I am making progress on the updated config file and I should be able to post
a "uncrustify_poc_3" branch soon with the results.

Regarding indentation, Marvin is right that Uncrustify cannot support edk2
indentation style out-of-box. Some changes are made in that fork to handle
the formatting. At this point, it can handle the indentation in the cases
I've seen. Uncrustify does potentially give us the ability to massively
deploy changes across the codebase in case a decision were made to change
the style.

Thanks,
Michael

On 8/16/2021 3:39 PM, Marvin Häuser wrote:

Hey Rebecca,

I think even Uncrustify has issues with the EDK II indentation style.
You might want to check the UEFI Talkbox Discord server, I had a brief
chat with Michael about it there. I don't think realistically any tool
supports EDK II's indentation style however, so I'd propose it is
changed. This could be for new submissions only, or actually the entire
codebase could be reformatted at once with a good tool setup. While this
screws with git blame, the (to my understanding) decided on CRLF -> LF
change does that anyway, so at least two evils could be dealt with in
one go really.

Best regards,
Marvin

On 16/08/2021 21:34, Rebecca Cran wrote:


cc devel@ .

On 8/16/21 1:33 PM, Rebecca Cran wrote:


I noticed a message on Twitter about an idea of using Uncrustify
for EDK2 instead of the ECC tool, and came across https://www.mail-
archive.com/search?l=devel@edk2.groups.io&q=subject:%22Re%5C%3A+%5C%5Bedk2%5C-<mailto:archive.com/search?l=devel@edk2.groups.io&q=subject:%22Re%5C%3A+%5C%5Bedk2%5C->
devel%5C%5D+TianoCore+Community+Meeting+Minutes+%5C-+2%5C%2F4%22&o=newest&f=1

.

I was wondering if there's been any progress on it that I could
check out?


Michael Kubacki: in that message, you said:

"I'm planning to put up a branch that we can use as a reference
for a conversation around uncrustify in the next couple of
weeks."


Did you end up creating that branch, and if so could you provide
a link to it please?


--
Rebecca Cran



















[-- Attachment #2: Type: text/html, Size: 75455 bytes --]

Re: [edk2-devel] Progress on getting Uncrustify working for EDK2?

[Andrew Fish]

[-- Attachment #1: Type: text/plain, Size: 15222 bytes --]

MIke,

I could not figure out how to download uncrustify tool from the provided link. 99% of the people are just going to want to install the tool, not be a developer of the fork. We should have some simple instructions on how to download the tool. 

The link points to something git web view looking Azure DevOps page and talks about this Nuget thing I know nothing about. I ran out of time and had to give up trying to download the tool. 

Thanks,

Andrew Fish

> On Nov 8, 2021, at 4:23 PM, Michael D Kinney <michael.d.kinney@intel.com> wrote:
> 
> Hello,
>  
> Good information in this thread on code style.
>  
> Some of the topics apply to uncrustify and some are out of scope for what uncrustify can fix on its own.
>  
> I would like to focus on a date to convert all source code in edk2 repo using the uncrustify tool and to capture the other code style topics into their own thread andbugzillas.
>  
> I would like to propose a conversion date for uncrustify immediately after the edk2-stable202111 release on 2021-11-26.
>  
> I have been working with Michael Kubacki on a build comparison tool that verifies that the build generate the same obj/lib/dll/efi/fv/fd files before and after the uncrustify changes.  We would run and publish the results from this tool before committing the changes.
>  
> We need TianoCore community approval of the following:
>  
> Approve format of C source generated by the uncrustify.
> Approve uncrustify changes right after edk2-stable-202111 release. 
> Extend code freeze until these changes are committed.
> Require use of uncrustify tool before submitting patch review emails or PRs.
> The required version would be a formally released version  from the fork maintained by Michael Kubacki until the changes can be upstreamed.
> https://dev.azure.com/projectmu/Uncrustify <https://dev.azure.com/projectmu/Uncrustify>
> Add EDK II CI check to verify that all PRs submitted exactly match uncrustified version.  Reject PRs that do not match exactly.
> Implement a git hook available that would automatically run uncristufy before committing changes to a local branch of an edk2 repo.
>  
> Thanks,
>  
> Mike
>  
> From: Andrew Fish <afish@apple.com <mailto:afish@apple.com>> 
> Sent: Thursday, October 7, 2021 2:09 PM
> To: Marvin Häuser <mhaeuser@posteo.de <mailto:mhaeuser@posteo.de>>
> Cc: edk2-devel-groups-io <devel@edk2.groups.io <mailto:devel@edk2.groups.io>>; Kinney, Michael D <michael.d.kinney@intel.com <mailto:michael.d.kinney@intel.com>>; Leif Lindholm <leif@nuviainc.com <mailto:leif@nuviainc.com>>; mikuback@linux.microsoft.com <mailto:mikuback@linux.microsoft.com>; rebecca@nuviainc.com <mailto:rebecca@nuviainc.com>; Michael Kubacki <Michael.Kubacki@microsoft.com <mailto:Michael.Kubacki@microsoft.com>>; Bret Barkelew <Bret.Barkelew@microsoft.com <mailto:Bret.Barkelew@microsoft.com>>
> Subject: Re: [edk2-devel] Progress on getting Uncrustify working for EDK2?
>  
>  
> 
> 
> On Oct 7, 2021, at 1:43 PM, Marvin Häuser <mhaeuser@posteo.de <mailto:mhaeuser@posteo.de>> wrote:
>  
> Hey Mike,
> Hey Andrew,
> 
> I'll just reply to both mails at once :)
> 
> On 07/10/2021 19:36, Andrew Fish wrote:
> 
> 
> 
> 
> On Oct 7, 2021, at 1:19 PM, Michael D Kinney <michael.d.kinney@intel.com <mailto:michael.d.kinney@intel.com>> wrote:
> 
> Hi Marvin,
> 
> Some comments below.
> 
> Mike
> 
> 
> -----Original Message-----
> From:devel@edk2.groups.io <mailto:devel@edk2.groups.io><devel@edk2.groups.io <mailto:devel@edk2.groups.io>> On Behalf Of Marvin Häuser
> Sent: Thursday, October 7, 2021 11:31 AM
> To: Leif Lindholm <leif@nuviainc.com <mailto:leif@nuviainc.com>>;devel@edk2.groups.io <mailto:devel@edk2.groups.io>;mikuback@linux.microsoft.com <mailto:mikuback@linux.microsoft.com>
> Cc:rebecca@nuviainc.com <mailto:rebecca@nuviainc.com>; Michael Kubacki <Michael.Kubacki@microsoft.com <mailto:Michael.Kubacki@microsoft.com>>; Bret Barkelew <Bret.Barkelew@microsoft.com <mailto:Bret.Barkelew@microsoft.com>>;
> Kinney, Michael D <michael.d.kinney@intel.com <mailto:michael.d.kinney@intel.com>>
> Subject: Re: [edk2-devel] Progress on getting Uncrustify working for EDK2?
> 
> Good day,
> 
> +1, but while you're at it, can we have arguments not align to the
> function name...
> 
>   Status = Test (
>              a
>              );
> 
> ... but to the next natural indentation level?
> 
>   Status = Test (
>     a
>     );
> 
> Basically no IDE I have seen supports EDK II's style, and I wouldn't be
> keen on writing known-broken style to then rely on Uncrustify to fix it.
> 
> I also have heard some controversy regarding casts off-list, where some
> prefer no spaces after casts to stress the evaluation order, and some
> prefer spaces to have clearer visuals (as a cast *ideally* would be
> something rare that requires good justification). Just throwing that out
> there.
> 
> 
> For things unrelated to autoformat (so semi-offtopic) but still relevant
> to the coding spec:
> 
> 1. Allow STATIC functions (if the debugging concerns are still relevant,
> there could be another level of indirection, like RELEASE_STATIC)?
> 
> Debugging concerns are no longer relevant.  The suggestion in the BZ below
> is to remove the STATIC macro and allow EDK II sources to add 'static'
> to any functions it makes sense to use on.
> 
> https://bugzilla.tianocore.org/show_bug.cgi?id=1766 <https://bugzilla.tianocore.org/show_bug.cgi?id=1766>
> 
> Thanks! I'd keep STATIC actually just for the sake of not doing no-op changes that do not really do anything and for consistency with CONST, but whatever works really.
> 
> 
> 
> 
> 
> 2. Allow variable assignments on definition (basically non-static CONST
> variables are banned...)?
> 
> Are referring to use of pre-initialized CONST variables declared within
> a function?  I think Bret brought this topic up when implementing some
> unit tests and the suggestion to pass ECCC was to promote them to
> pre-initialized CONST global variables.
> 
> Yes.
> 
> 
> 
> The challenges we have seen in the past with pre-initialized variables within
> a function is that they can cause compilers to inject use of memcpy() calls,
> especially if the variable being initialized on the stack is a structure.
> These cause build breaks today.
> 
> This issue is independent of CONST. I’m not sure a coding style tool is smart enough to catch this generically? You need an understanding of C types to know if the local variable assignment is going to trigger a memcpy().
> 
> What I’ve seen in the real world is the firmware compiles with -Os or LTO to fit int he ROM for DEBUG and RELEASE, and the optimizer optimizes away the call to memcpy. Then if you try to build NOOPT (or over ride the compiler flags on an individual driver/lib) you fail to link as only the NOOPT build injects the memcpy.
> 
> +1
> 
> 
> 
> Thus I think the best way to enforce this rule is to compile a project NOOPT. I’m trying to remember are there flags to built to tell it to compile and skip the FD construction? Maybe we should advocate platforms add a NOOPT build target that just compiles the code, but does not create the FD?
> 
> I know there were stability concerns with intrinsics in the past, but memcpy() is in the standard, and the rest remained stable to my knowledge. Maybe it's time to fix the issues at the root? Works for us:
> https://github.com/acidanthera/OpenCorePkg/tree/master/Library/OcCompilerIntrinsicsLib <https://github.com/acidanthera/OpenCorePkg/tree/master/Library/OcCompilerIntrinsicsLib>
>  
> Marvin,
>  
> Good point. This would make the rule moot. So maybe just removing the requirement would be the easiest long term fix. 
>  
> Other embedded projects I know of do this too, and as you point out the compilers keep these APIs standard for folks the provide their own runtimes.
>  
> Thanks,
>  
> Andrew Fish
> 
> 
> Best regards,
> Marvin
> 
> 
> 
> Thanks,
> 
> Andrew Fish
> 
> 
> 
> 
> 
> 3. Allow variable declarations at any scope (I had some nasty shadowing
> bugs before, probably prohibit shadowing with warnings)?
> 
> By shadowing do you mean the declaration of the same variable name in
> multiple scoped within the same function?
> 
> 
> 
> 4. Require that exactly all function declarations and all function
> definitions with no prior declaration must be documented (first
> direction is enforcing docs, second is prohibiting doc duplication, I've
> seen them go out-of-sync plenty of times)?
> 
> I agree that this can reduce duplication and sync issues.  The uncrustify
> tool being discussed here could not help clean this up or enforce this
> type of rule.  It is a good topic, but may need to be split out into its
> own thread.
> 
> 
> 
> The latter bunch would not require any autoformat rules or reformatation
> of existing code, but would be target only new submissions in my
> opinion. Thoughts?
> 
> 
> Thanks for your efforts!
> 
> Best regards,
> Marvin
> 
> 
> Am 07.10.2021 um 12:48 schrieb Leif Lindholm:
> 
> Hi Michael,
> 
> Apologies, I've owed you a response (promised off-list) for a while
> now.
> 
> First, let me say I hugely appreciate this effort. Apart from aligning
> the codebase(s), this will reduce manual reviewing effort
> substantially, as well as cutting down on number of rework cycles for
> developers.
> 
> Looking at the changes to (well, the comments in) uncrustify, this
> seems to be constrained to:
> - Newline after '(' for multi-line function calls.
> - Dealing with "(("/"))" for DEBUG macros.
> - Function pointer typedefs:
>   - typedef\nEFIAPI
>   - closing parentheses indentation
> 
> I don't think I've made any secret over the years that I am not a
> massive fan of the EDK2 coding style in general. So I think for any
> of its quirks that are substantial enough that they require not just
> custom configuration but actual new function added to existing code
> conformance tools, this would be an excellent point to sanitise the
> coding style instead.
> 
> Taking these in order:
> 
> Newline after '('
> -----------------
> I think we already reached a level of flexibility around this, where
> we don't actually enforce this (or single argument per
> line). Personally, I'd be happy to update the coding style as
> required instead.
> 
> DEBUG macro parentheses
> -----------------------
> How does uncrustify treat DEBUG macros without this modification?
> Do we start getting everything turned into multi-level indented
> multi-line statements without this change?
> 
> Function pointer typedefs:
> --------------------------
> I don't see that function pointer typedefs need to rigidly follow the
> same pattern as the declaration of functions implementing them. Could
> we update the coding style (if needed) instead?
> 
> Best Regards,
> 
> Leif
> 
> On Mon, Aug 16, 2021 at 16:00:38 -0400, Michael Kubacki wrote:
> 
> The edk2 branch was created here:
> https://github.com/makubacki/edk2/tree/uncrustify_poc_2 <https://github.com/makubacki/edk2/tree/uncrustify_poc_2>
> 
> We have a Project Mu fork with custom changes to the Uncrustify tool to help
> comply with EDK II formatting here:
> https://dev.azure.com/projectmu/_git/Uncrustify <https://dev.azure.com/projectmu/_git/Uncrustify>
> 
> The latest information about the status and how to experiment with the
> configuration file and the tool are in that fork:
> https://dev.azure.com/projectmu/Uncrustify/_wiki/wikis/Uncrustify.wiki/1/Project-Mu-(EDK-II)-Fork-Readme <https://dev.azure.com/projectmu/Uncrustify/_wiki/wikis/Uncrustify.wiki/1/Project-Mu-(EDK-II)-Fork-Readme>
> 
> 
> That said, I have also finished a CI plugin to run Uncrustify that should be
> ready soon to initially deploy in Project Mu. Before doing so, I am trying
> to settle on an initial configuration file that less strictly but more
> reliably formats the code than in the examples in those branches. For
> example, remove heuristics that when run against the same set of code
> multiple times can produce different results. An example would be a rule
> that reformats code because it exceeds a specified column width on one run
> but on the next run that reformatted code triggers a different rule to
> further align the code and so on. At least initially, some rules might be
> tweaked in a more conservative approach that can be tightened in the future.
> Once this configuration file is ready, we will baseline Project Mu code as
> an example and turn on the plugin. The CI plugin runs Uncrustify against
> modified files and if there's any changes, indicating a formatting
> deviation, the diff chunks are saved in a log so they can be viewed as a
> build artifact.
> 
> I am making progress on the updated config file and I should be able to post
> a "uncrustify_poc_3" branch soon with the results.
> 
> Regarding indentation, Marvin is right that Uncrustify cannot support edk2
> indentation style out-of-box. Some changes are made in that fork to handle
> the formatting. At this point, it can handle the indentation in the cases
> I've seen. Uncrustify does potentially give us the ability to massively
> deploy changes across the codebase in case a decision were made to change
> the style.
> 
> Thanks,
> Michael
> 
> On 8/16/2021 3:39 PM, Marvin Häuser wrote:
> 
> Hey Rebecca,
> 
> I think even Uncrustify has issues with the EDK II indentation style.
> You might want to check the UEFI Talkbox Discord server, I had a brief
> chat with Michael about it there. I don't think realistically any tool
> supports EDK II's indentation style however, so I'd propose it is
> changed. This could be for new submissions only, or actually the entire
> codebase could be reformatted at once with a good tool setup. While this
> screws with git blame, the (to my understanding) decided on CRLF -> LF
> change does that anyway, so at least two evils could be dealt with in
> one go really.
> 
> Best regards,
> Marvin
> 
> On 16/08/2021 21:34, Rebecca Cran wrote:
> 
> 
> cc devel@ .
> 
> On 8/16/21 1:33 PM, Rebecca Cran wrote:
> 
> 
> I noticed a message on Twitter about an idea of using Uncrustify
> for EDK2 instead of the ECC tool, and came across https://www.mail- <https://www.mail-/>
> archive.com/search?l=devel@edk2.groups.io&q=subject:%22Re%5C%3A+%5C%5Bedk2%5C- <mailto:archive.com/search?l=devel@edk2.groups.io&q=subject:%22Re%5C%3A+%5C%5Bedk2%5C->
> devel%5C%5D+TianoCore+Community+Meeting+Minutes+%5C-+2%5C%2F4%22&o=newest&f=1
> 
> .
> 
> I was wondering if there's been any progress on it that I could
> check out?
> 
> 
> Michael Kubacki: in that message, you said:
> 
> "I'm planning to put up a branch that we can use as a reference
> for a conversation around uncrustify in the next couple of
> weeks."
> 
> 
> Did you end up creating that branch, and if so could you provide
> a link to it please?
> 
> 
> --
> Rebecca Cran
> 
>  
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
>  
> 


[-- Attachment #2: Type: text/html, Size: 40789 bytes --]

回复: [edk2-devel] Progress on getting Uncrustify working for EDK2?

[gaoliming]

[-- Attachment #1: Type: text/plain, Size: 14804 bytes --]

Mike： 

  I add my comments below. 

 

发件人: devel@edk2.groups.io <devel@edk2.groups.io> 代表 Michael D Kinney
发送时间: 2021年11月9日 8:24
收件人: Andrew Fish <afish@apple.com>; Marvin Häuser <mhaeuser@posteo.de>; Kinney, Michael D <michael.d.kinney@intel.com>; Michael Kubacki <Michael.Kubacki@microsoft.com>
抄送: edk2-devel-groups-io <devel@edk2.groups.io>; Leif Lindholm <leif@nuviainc.com>; mikuback@linux.microsoft.com; rebecca@nuviainc.com; Bret Barkelew <Bret.Barkelew@microsoft.com>
主题: Re: [edk2-devel] Progress on getting Uncrustify working for EDK2?

 

Hello,

 

Good information in this thread on code style.

 

Some of the topics apply to uncrustify and some are out of scope for what uncrustify can fix on its own.

 

I would like to focus on a date to convert all source code in edk2 repo using the uncrustify tool and to capture the other code style topics into their own thread and bugzillas.

 

I would like to propose a conversion date for uncrustify immediately after the edk2-stable202111 release on 2021-11-26.

 

I have been working with Michael Kubacki on a build comparison tool that verifies that the build generate the same obj/lib/dll/efi/fv/fd files before and after the uncrustify changes.  We would run and publish the results from this tool before committing the changes.

 

We need TianoCore community approval of the following:

 

1.	Approve format of C source generated by the uncrustify.

     [Liming] This format is for C source in Edk2 repo. Dose it also include BaseTools C source files? Submodule git repo will not cover. Rigth?

2.	Approve uncrustify changes right after edk2-stable-202111 release.  

a.	Extend code freeze until these changes are committed.

      [Liming] Which stable tag code freeze will be extended? Do you mean next stable tag 202202?

3.	Require use of uncrustify tool before submitting patch review emails or PRs.

a.	The required version would be a formally released version  from the fork maintained by Michael Kubacki until the changes can be upstreamed.
b.	https://dev.azure.com/projectmu/Uncrustify

4.	Add EDK II CI check to verify that all PRs submitted exactly match uncrustified version.  Reject PRs that do not match exactly.
5.	Implement a git hook available that would automatically run uncristufy before committing changes to a local branch of an edk2 repo.

[Liming] Is there the test version git hook for the developer? 

 

Thanks

Liming

 

Thanks,

 

Mike

 

From: Andrew Fish <afish@apple.com <mailto:afish@apple.com> > 
Sent: Thursday, October 7, 2021 2:09 PM
To: Marvin Häuser <mhaeuser@posteo.de <mailto:mhaeuser@posteo.de> >
Cc: edk2-devel-groups-io <devel@edk2.groups.io <mailto:devel@edk2.groups.io> >; Kinney, Michael D <michael.d.kinney@intel.com <mailto:michael.d.kinney@intel.com> >; Leif Lindholm <leif@nuviainc.com <mailto:leif@nuviainc.com> >; mikuback@linux.microsoft.com <mailto:mikuback@linux.microsoft.com> ; rebecca@nuviainc.com <mailto:rebecca@nuviainc.com> ; Michael Kubacki <Michael.Kubacki@microsoft.com <mailto:Michael.Kubacki@microsoft.com> >; Bret Barkelew <Bret.Barkelew@microsoft.com <mailto:Bret.Barkelew@microsoft.com> >
Subject: Re: [edk2-devel] Progress on getting Uncrustify working for EDK2?

 

 





On Oct 7, 2021, at 1:43 PM, Marvin Häuser <mhaeuser@posteo.de <mailto:mhaeuser@posteo.de> > wrote:

 

Hey Mike,
Hey Andrew,

I'll just reply to both mails at once :)

On 07/10/2021 19:36, Andrew Fish wrote:








On Oct 7, 2021, at 1:19 PM, Michael D Kinney <michael.d.kinney@intel.com <mailto:michael.d.kinney@intel.com> > wrote:

Hi Marvin,

Some comments below.

Mike




-----Original Message-----
From:devel@edk2.groups.io <mailto:devel@edk2.groups.io> <devel@edk2.groups.io <mailto:devel@edk2.groups.io> > On Behalf Of Marvin Häuser
Sent: Thursday, October 7, 2021 11:31 AM
To: Leif Lindholm <leif@nuviainc.com <mailto:leif@nuviainc.com> >;devel@edk2.groups.io <mailto:devel@edk2.groups.io> ;mikuback@linux.microsoft.com <mailto:mikuback@linux.microsoft.com> 
Cc:rebecca@nuviainc.com <mailto:rebecca@nuviainc.com> ; Michael Kubacki <Michael.Kubacki@microsoft.com <mailto:Michael.Kubacki@microsoft.com> >; Bret Barkelew <Bret.Barkelew@microsoft.com <mailto:Bret.Barkelew@microsoft.com> >;
Kinney, Michael D <michael.d.kinney@intel.com <mailto:michael.d.kinney@intel.com> >
Subject: Re: [edk2-devel] Progress on getting Uncrustify working for EDK2?

Good day,

+1, but while you're at it, can we have arguments not align to the
function name...

  Status = Test (
             a
             );

... but to the next natural indentation level?

  Status = Test (
    a
    );

Basically no IDE I have seen supports EDK II's style, and I wouldn't be
keen on writing known-broken style to then rely on Uncrustify to fix it.

I also have heard some controversy regarding casts off-list, where some
prefer no spaces after casts to stress the evaluation order, and some
prefer spaces to have clearer visuals (as a cast *ideally* would be
something rare that requires good justification). Just throwing that out
there.


For things unrelated to autoformat (so semi-offtopic) but still relevant
to the coding spec:

1. Allow STATIC functions (if the debugging concerns are still relevant,
there could be another level of indirection, like RELEASE_STATIC)?


Debugging concerns are no longer relevant.  The suggestion in the BZ below
is to remove the STATIC macro and allow EDK II sources to add 'static'
to any functions it makes sense to use on.

https://bugzilla.tianocore.org/show_bug.cgi?id=1766


Thanks! I'd keep STATIC actually just for the sake of not doing no-op changes that do not really do anything and for consistency with CONST, but whatever works really.









2. Allow variable assignments on definition (basically non-static CONST
variables are banned...)?


Are referring to use of pre-initialized CONST variables declared within
a function?  I think Bret brought this topic up when implementing some
unit tests and the suggestion to pass ECCC was to promote them to
pre-initialized CONST global variables.


Yes.





The challenges we have seen in the past with pre-initialized variables within
a function is that they can cause compilers to inject use of memcpy() calls,
especially if the variable being initialized on the stack is a structure.
These cause build breaks today.


This issue is independent of CONST. I’m not sure a coding style tool is smart enough to catch this generically? You need an understanding of C types to know if the local variable assignment is going to trigger a memcpy().

What I’ve seen in the real world is the firmware compiles with -Os or LTO to fit int he ROM for DEBUG and RELEASE, and the optimizer optimizes away the call to memcpy. Then if you try to build NOOPT (or over ride the compiler flags on an individual driver/lib) you fail to link as only the NOOPT build injects the memcpy.


+1





Thus I think the best way to enforce this rule is to compile a project NOOPT. I’m trying to remember are there flags to built to tell it to compile and skip the FD construction? Maybe we should advocate platforms add a NOOPT build target that just compiles the code, but does not create the FD?


I know there were stability concerns with intrinsics in the past, but memcpy() is in the standard, and the rest remained stable to my knowledge. Maybe it's time to fix the issues at the root? Works for us:
 <https://github.com/acidanthera/OpenCorePkg/tree/master/Library/OcCompilerIntrinsicsLib> https://github.com/acidanthera/OpenCorePkg/tree/master/Library/OcCompilerIntrinsicsLib

 

Marvin,

 

Good point. This would make the rule moot. So maybe just removing the requirement would be the easiest long term fix. 

 

Other embedded projects I know of do this too, and as you point out the compilers keep these APIs standard for folks the provide their own runtimes.

 

Thanks,

 

Andrew Fish





Best regards,
Marvin





Thanks,

Andrew Fish









3. Allow variable declarations at any scope (I had some nasty shadowing
bugs before, probably prohibit shadowing with warnings)?


By shadowing do you mean the declaration of the same variable name in
multiple scoped within the same function?





4. Require that exactly all function declarations and all function
definitions with no prior declaration must be documented (first
direction is enforcing docs, second is prohibiting doc duplication, I've
seen them go out-of-sync plenty of times)?


I agree that this can reduce duplication and sync issues.  The uncrustify
tool being discussed here could not help clean this up or enforce this
type of rule.  It is a good topic, but may need to be split out into its
own thread.





The latter bunch would not require any autoformat rules or reformatation
of existing code, but would be target only new submissions in my
opinion. Thoughts?


Thanks for your efforts!

Best regards,
Marvin


Am 07.10.2021 um 12:48 schrieb Leif Lindholm:



Hi Michael,

Apologies, I've owed you a response (promised off-list) for a while
now.

First, let me say I hugely appreciate this effort. Apart from aligning
the codebase(s), this will reduce manual reviewing effort
substantially, as well as cutting down on number of rework cycles for
developers.

Looking at the changes to (well, the comments in) uncrustify, this
seems to be constrained to:
- Newline after '(' for multi-line function calls.
- Dealing with "(("/"))" for DEBUG macros.
- Function pointer typedefs:
  - typedef\nEFIAPI
  - closing parentheses indentation

I don't think I've made any secret over the years that I am not a
massive fan of the EDK2 coding style in general. So I think for any
of its quirks that are substantial enough that they require not just
custom configuration but actual new function added to existing code
conformance tools, this would be an excellent point to sanitise the
coding style instead.

Taking these in order:

Newline after '('
-----------------
I think we already reached a level of flexibility around this, where
we don't actually enforce this (or single argument per
line). Personally, I'd be happy to update the coding style as
required instead.

DEBUG macro parentheses
-----------------------
How does uncrustify treat DEBUG macros without this modification?
Do we start getting everything turned into multi-level indented
multi-line statements without this change?

Function pointer typedefs:
--------------------------
I don't see that function pointer typedefs need to rigidly follow the
same pattern as the declaration of functions implementing them. Could
we update the coding style (if needed) instead?

Best Regards,

Leif

On Mon, Aug 16, 2021 at 16:00:38 -0400, Michael Kubacki wrote:



The edk2 branch was created here:
https://github.com/makubacki/edk2/tree/uncrustify_poc_2

We have a Project Mu fork with custom changes to the Uncrustify tool to help
comply with EDK II formatting here:
https://dev.azure.com/projectmu/_git/Uncrustify

The latest information about the status and how to experiment with the
configuration file and the tool are in that fork:

https://dev.azure.com/projectmu/Uncrustify/_wiki/wikis/Uncrustify.wiki/1/Project-Mu-(EDK-II)-Fork-Readme




That said, I have also finished a CI plugin to run Uncrustify that should be
ready soon to initially deploy in Project Mu. Before doing so, I am trying
to settle on an initial configuration file that less strictly but more
reliably formats the code than in the examples in those branches. For
example, remove heuristics that when run against the same set of code
multiple times can produce different results. An example would be a rule
that reformats code because it exceeds a specified column width on one run
but on the next run that reformatted code triggers a different rule to
further align the code and so on. At least initially, some rules might be
tweaked in a more conservative approach that can be tightened in the future.
Once this configuration file is ready, we will baseline Project Mu code as
an example and turn on the plugin. The CI plugin runs Uncrustify against
modified files and if there's any changes, indicating a formatting
deviation, the diff chunks are saved in a log so they can be viewed as a
build artifact.

I am making progress on the updated config file and I should be able to post
a "uncrustify_poc_3" branch soon with the results.

Regarding indentation, Marvin is right that Uncrustify cannot support edk2
indentation style out-of-box. Some changes are made in that fork to handle
the formatting. At this point, it can handle the indentation in the cases
I've seen. Uncrustify does potentially give us the ability to massively
deploy changes across the codebase in case a decision were made to change
the style.

Thanks,
Michael

On 8/16/2021 3:39 PM, Marvin Häuser wrote:



Hey Rebecca,

I think even Uncrustify has issues with the EDK II indentation style.
You might want to check the UEFI Talkbox Discord server, I had a brief
chat with Michael about it there. I don't think realistically any tool
supports EDK II's indentation style however, so I'd propose it is
changed. This could be for new submissions only, or actually the entire
codebase could be reformatted at once with a good tool setup. While this
screws with git blame, the (to my understanding) decided on CRLF -> LF
change does that anyway, so at least two evils could be dealt with in
one go really.

Best regards,
Marvin

On 16/08/2021 21:34, Rebecca Cran wrote:




cc devel@ .

On 8/16/21 1:33 PM, Rebecca Cran wrote:




I noticed a message on Twitter about an idea of using Uncrustify
for EDK2 instead of the ECC tool, and came across https://www.mail-

archive.com/search?l=devel@edk2.groups.io <mailto:archive.com/search?l=devel@edk2.groups.io&q=subject:%22Re%5C%3A+%5C%5Bedk2%5C-> &q=subject:%22Re%5C%3A+%5C%5Bedk2%5C-
devel%5C%5D+TianoCore+Community+Meeting+Minutes+%5C-+2%5C%2F4%22&o=newest&f=1



.

I was wondering if there's been any progress on it that I could
check out?


Michael Kubacki: in that message, you said:

"I'm planning to put up a branch that we can use as a reference
for a conversation around uncrustify in the next couple of
weeks."


Did you end up creating that branch, and if so could you provide
a link to it please?


--
Rebecca Cran

 























 




[-- Attachment #2: Type: text/html, Size: 32974 bytes --]

Re: [edk2-devel] Progress on getting Uncrustify working for EDK2?

[Michael D Kinney]

[-- Attachment #1: Type: text/plain, Size: 14870 bytes --]

HI Andrew,

Great feedback.

What your preferred way to install a tool like this?  If we collect that data from the community, we can make sure the pipeline generates the right type of installers.

Thanks,

Mike

From: Andrew Fish <afish@apple.com>
Sent: Monday, November 8, 2021 5:09 PM
To: devel@edk2.groups.io; Kinney, Michael D <michael.d.kinney@intel.com>
Cc: Marvin Häuser <mhaeuser@posteo.de>; Michael Kubacki <Michael.Kubacki@microsoft.com>; Leif Lindholm <leif@nuviainc.com>; mikuback@linux.microsoft.com; rebecca@nuviainc.com; Bret Barkelew <Bret.Barkelew@microsoft.com>
Subject: Re: [edk2-devel] Progress on getting Uncrustify working for EDK2?

MIke,

I could not figure out how to download uncrustify tool from the provided link. 99% of the people are just going to want to install the tool, not be a developer of the fork. We should have some simple instructions on how to download the tool.

The link points to something git web view looking Azure DevOps page and talks about this Nuget thing I know nothing about. I ran out of time and had to give up trying to download the tool.

Thanks,

Andrew Fish


On Nov 8, 2021, at 4:23 PM, Michael D Kinney <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>> wrote:

Hello,

Good information in this thread on code style.

Some of the topics apply to uncrustify and some are out of scope for what uncrustify can fix on its own.

I would like to focus on a date to convert all source code in edk2 repo using the uncrustify tool and to capture the other code style topics into their own thread andbugzillas.

I would like to propose a conversion date for uncrustify immediately after the edk2-stable202111 release on 2021-11-26.

I have been working with Michael Kubacki on a build comparison tool that verifies that the build generate the same obj/lib/dll/efi/fv/fd files before and after the uncrustify changes.  We would run and publish the results from this tool before committing the changes.

We need TianoCore community approval of the following:


  1.  Approve format of C source generated by the uncrustify.
  2.  Approve uncrustify changes right after edk2-stable-202111 release.

     *   Extend code freeze until these changes are committed.

  1.  Require use of uncrustify tool before submitting patch review emails or PRs.

     *   The required version would be a formally released version  from the fork maintained by Michael Kubacki until the changes can be upstreamed.
     *   https://dev.azure.com/projectmu/Uncrustify

  1.  Add EDK II CI check to verify that all PRs submitted exactly match uncrustified version.  Reject PRs that do not match exactly.
  2.  Implement a git hook available that would automatically run uncristufy before committing changes to a local branch of an edk2 repo.

Thanks,

Mike

From: Andrew Fish <afish@apple.com<mailto:afish@apple.com>>
Sent: Thursday, October 7, 2021 2:09 PM
To: Marvin Häuser <mhaeuser@posteo.de<mailto:mhaeuser@posteo.de>>
Cc: edk2-devel-groups-io <devel@edk2.groups.io<mailto:devel@edk2.groups.io>>; Kinney, Michael D <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>>; Leif Lindholm <leif@nuviainc.com<mailto:leif@nuviainc.com>>; mikuback@linux.microsoft.com<mailto:mikuback@linux.microsoft.com>; rebecca@nuviainc.com<mailto:rebecca@nuviainc.com>; Michael Kubacki <Michael.Kubacki@microsoft.com<mailto:Michael.Kubacki@microsoft.com>>; Bret Barkelew <Bret.Barkelew@microsoft.com<mailto:Bret.Barkelew@microsoft.com>>
Subject: Re: [edk2-devel] Progress on getting Uncrustify working for EDK2?





On Oct 7, 2021, at 1:43 PM, Marvin Häuser <mhaeuser@posteo.de<mailto:mhaeuser@posteo.de>> wrote:

Hey Mike,
Hey Andrew,

I'll just reply to both mails at once :)

On 07/10/2021 19:36, Andrew Fish wrote:






On Oct 7, 2021, at 1:19 PM, Michael D Kinney <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>> wrote:

Hi Marvin,

Some comments below.

Mike



-----Original Message-----
From:devel@edk2.groups.io<mailto:devel@edk2.groups.io><devel@edk2.groups.io<mailto:devel@edk2.groups.io>> On Behalf Of Marvin Häuser
Sent: Thursday, October 7, 2021 11:31 AM
To: Leif Lindholm <leif@nuviainc.com<mailto:leif@nuviainc.com>>;devel@edk2.groups.io<mailto:devel@edk2.groups.io>;mikuback@linux.microsoft.com<mailto:mikuback@linux.microsoft.com>
Cc:rebecca@nuviainc.com<mailto:rebecca@nuviainc.com>; Michael Kubacki <Michael.Kubacki@microsoft.com<mailto:Michael.Kubacki@microsoft.com>>; Bret Barkelew <Bret.Barkelew@microsoft.com<mailto:Bret.Barkelew@microsoft.com>>;
Kinney, Michael D <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>>
Subject: Re: [edk2-devel] Progress on getting Uncrustify working for EDK2?

Good day,

+1, but while you're at it, can we have arguments not align to the
function name...

  Status = Test (
             a
             );

... but to the next natural indentation level?

  Status = Test (
    a
    );

Basically no IDE I have seen supports EDK II's style, and I wouldn't be
keen on writing known-broken style to then rely on Uncrustify to fix it.

I also have heard some controversy regarding casts off-list, where some
prefer no spaces after casts to stress the evaluation order, and some
prefer spaces to have clearer visuals (as a cast *ideally* would be
something rare that requires good justification). Just throwing that out
there.


For things unrelated to autoformat (so semi-offtopic) but still relevant
to the coding spec:

1. Allow STATIC functions (if the debugging concerns are still relevant,
there could be another level of indirection, like RELEASE_STATIC)?

Debugging concerns are no longer relevant.  The suggestion in the BZ below
is to remove the STATIC macro and allow EDK II sources to add 'static'
to any functions it makes sense to use on.

https://bugzilla.tianocore.org/show_bug.cgi?id=1766

Thanks! I'd keep STATIC actually just for the sake of not doing no-op changes that do not really do anything and for consistency with CONST, but whatever works really.







2. Allow variable assignments on definition (basically non-static CONST
variables are banned...)?

Are referring to use of pre-initialized CONST variables declared within
a function?  I think Bret brought this topic up when implementing some
unit tests and the suggestion to pass ECCC was to promote them to
pre-initialized CONST global variables.

Yes.




The challenges we have seen in the past with pre-initialized variables within
a function is that they can cause compilers to inject use of memcpy() calls,
especially if the variable being initialized on the stack is a structure.
These cause build breaks today.

This issue is independent of CONST. I’m not sure a coding style tool is smart enough to catch this generically? You need an understanding of C types to know if the local variable assignment is going to trigger a memcpy().

What I’ve seen in the real world is the firmware compiles with -Os or LTO to fit int he ROM for DEBUG and RELEASE, and the optimizer optimizes away the call to memcpy. Then if you try to build NOOPT (or over ride the compiler flags on an individual driver/lib) you fail to link as only the NOOPT build injects the memcpy.

+1




Thus I think the best way to enforce this rule is to compile a project NOOPT. I’m trying to remember are there flags to built to tell it to compile and skip the FD construction? Maybe we should advocate platforms add a NOOPT build target that just compiles the code, but does not create the FD?

I know there were stability concerns with intrinsics in the past, but memcpy() is in the standard, and the rest remained stable to my knowledge. Maybe it's time to fix the issues at the root? Works for us:
https://github.com/acidanthera/OpenCorePkg/tree/master/Library/OcCompilerIntrinsicsLib

Marvin,

Good point. This would make the rule moot. So maybe just removing the requirement would be the easiest long term fix.

Other embedded projects I know of do this too, and as you point out the compilers keep these APIs standard for folks the provide their own runtimes.

Thanks,

Andrew Fish



Best regards,
Marvin




Thanks,

Andrew Fish







3. Allow variable declarations at any scope (I had some nasty shadowing
bugs before, probably prohibit shadowing with warnings)?

By shadowing do you mean the declaration of the same variable name in
multiple scoped within the same function?




4. Require that exactly all function declarations and all function
definitions with no prior declaration must be documented (first
direction is enforcing docs, second is prohibiting doc duplication, I've
seen them go out-of-sync plenty of times)?

I agree that this can reduce duplication and sync issues.  The uncrustify
tool being discussed here could not help clean this up or enforce this
type of rule.  It is a good topic, but may need to be split out into its
own thread.




The latter bunch would not require any autoformat rules or reformatation
of existing code, but would be target only new submissions in my
opinion. Thoughts?


Thanks for your efforts!

Best regards,
Marvin


Am 07.10.2021 um 12:48 schrieb Leif Lindholm:


Hi Michael,

Apologies, I've owed you a response (promised off-list) for a while
now.

First, let me say I hugely appreciate this effort. Apart from aligning
the codebase(s), this will reduce manual reviewing effort
substantially, as well as cutting down on number of rework cycles for
developers.

Looking at the changes to (well, the comments in) uncrustify, this
seems to be constrained to:
- Newline after '(' for multi-line function calls.
- Dealing with "(("/"))" for DEBUG macros.
- Function pointer typedefs:
  - typedef\nEFIAPI
  - closing parentheses indentation

I don't think I've made any secret over the years that I am not a
massive fan of the EDK2 coding style in general. So I think for any
of its quirks that are substantial enough that they require not just
custom configuration but actual new function added to existing code
conformance tools, this would be an excellent point to sanitise the
coding style instead.

Taking these in order:

Newline after '('
-----------------
I think we already reached a level of flexibility around this, where
we don't actually enforce this (or single argument per
line). Personally, I'd be happy to update the coding style as
required instead.

DEBUG macro parentheses
-----------------------
How does uncrustify treat DEBUG macros without this modification?
Do we start getting everything turned into multi-level indented
multi-line statements without this change?

Function pointer typedefs:
--------------------------
I don't see that function pointer typedefs need to rigidly follow the
same pattern as the declaration of functions implementing them. Could
we update the coding style (if needed) instead?

Best Regards,

Leif

On Mon, Aug 16, 2021 at 16:00:38 -0400, Michael Kubacki wrote:


The edk2 branch was created here:
https://github.com/makubacki/edk2/tree/uncrustify_poc_2

We have a Project Mu fork with custom changes to the Uncrustify tool to help
comply with EDK II formatting here:
https://dev.azure.com/projectmu/_git/Uncrustify

The latest information about the status and how to experiment with the
configuration file and the tool are in that fork:
https://dev.azure.com/projectmu/Uncrustify/_wiki/wikis/Uncrustify.wiki/1/Project-Mu-(EDK-II)-Fork-Readme



That said, I have also finished a CI plugin to run Uncrustify that should be
ready soon to initially deploy in Project Mu. Before doing so, I am trying
to settle on an initial configuration file that less strictly but more
reliably formats the code than in the examples in those branches. For
example, remove heuristics that when run against the same set of code
multiple times can produce different results. An example would be a rule
that reformats code because it exceeds a specified column width on one run
but on the next run that reformatted code triggers a different rule to
further align the code and so on. At least initially, some rules might be
tweaked in a more conservative approach that can be tightened in the future.
Once this configuration file is ready, we will baseline Project Mu code as
an example and turn on the plugin. The CI plugin runs Uncrustify against
modified files and if there's any changes, indicating a formatting
deviation, the diff chunks are saved in a log so they can be viewed as a
build artifact.

I am making progress on the updated config file and I should be able to post
a "uncrustify_poc_3" branch soon with the results.

Regarding indentation, Marvin is right that Uncrustify cannot support edk2
indentation style out-of-box. Some changes are made in that fork to handle
the formatting. At this point, it can handle the indentation in the cases
I've seen. Uncrustify does potentially give us the ability to massively
deploy changes across the codebase in case a decision were made to change
the style.

Thanks,
Michael

On 8/16/2021 3:39 PM, Marvin Häuser wrote:


Hey Rebecca,

I think even Uncrustify has issues with the EDK II indentation style.
You might want to check the UEFI Talkbox Discord server, I had a brief
chat with Michael about it there. I don't think realistically any tool
supports EDK II's indentation style however, so I'd propose it is
changed. This could be for new submissions only, or actually the entire
codebase could be reformatted at once with a good tool setup. While this
screws with git blame, the (to my understanding) decided on CRLF -> LF
change does that anyway, so at least two evils could be dealt with in
one go really.

Best regards,
Marvin

On 16/08/2021 21:34, Rebecca Cran wrote:



cc devel@ .

On 8/16/21 1:33 PM, Rebecca Cran wrote:



I noticed a message on Twitter about an idea of using Uncrustify
for EDK2 instead of the ECC tool, and came across https://www.mail-<https://www.mail-/>
archive.com/search?l=devel@edk2.groups.io&q=subject:%22Re%5C%3A+%5C%5Bedk2%5C-<mailto:archive.com/search?l=devel@edk2.groups.io&q=subject:%22Re%5C%3A+%5C%5Bedk2%5C->
devel%5C%5D+TianoCore+Community+Meeting+Minutes+%5C-+2%5C%2F4%22&o=newest&f=1


.

I was wondering if there's been any progress on it that I could
check out?


Michael Kubacki: in that message, you said:

"I'm planning to put up a branch that we can use as a reference
for a conversation around uncrustify in the next couple of
weeks."


Did you end up creating that branch, and if so could you provide
a link to it please?


--
Rebecca Cran
























[-- Attachment #2: Type: text/html, Size: 84422 bytes --]

Re: [edk2-devel] Progress on getting Uncrustify working for EDK2?

[Andrew Fish]

[-- Attachment #1: Type: text/plain, Size: 16905 bytes --]



> On Nov 8, 2021, at 5:13 PM, Kinney, Michael D <michael.d.kinney@intel.com> wrote:
> 
> HI Andrew,
>  
> Great feedback.
>  
> What your preferred way to install a tool like this?  If we collect that data from the community, we can make sure the pipeline generates the right type of installers.
>  

I could not figure out how to download an installer from the links. 

If I go to http://uncrustify.sourceforge.net <http://uncrustify.sourceforge.net/> I see https://sourceforge.net/projects/uncrustify/files/ <https://sourceforge.net/projects/uncrustify/files/> and a button to download binaries. Not ideal that it is only for Windows but at least the workflow was obvious. Looks like I need to build my own version for macOS, not ideal but I can at least figure that out. 

Worst case we can have a Tianocore.org <http://tianocore.org/> page to describe a simple recipe to install the tool. With step by step instructions some one can just type in without thinking too much. 

Thanks,

Andrew Fish

> Thanks,
>  
> Mike
>  
> From: Andrew Fish <afish@apple.com> 
> Sent: Monday, November 8, 2021 5:09 PM
> To: devel@edk2.groups.io; Kinney, Michael D <michael.d.kinney@intel.com>
> Cc: Marvin Häuser <mhaeuser@posteo.de>; Michael Kubacki <Michael.Kubacki@microsoft.com>; Leif Lindholm <leif@nuviainc.com>; mikuback@linux.microsoft.com; rebecca@nuviainc.com; Bret Barkelew <Bret.Barkelew@microsoft.com>
> Subject: Re: [edk2-devel] Progress on getting Uncrustify working for EDK2?
>  
> MIke,
>  
> I could not figure out how to download uncrustify tool from the provided link. 99% of the people are just going to want to install the tool, not be a developer of the fork. We should have some simple instructions on how to download the tool. 
>  
> The link points to something git web view looking Azure DevOps page and talks about this Nuget thing I know nothing about. I ran out of time and had to give up trying to download the tool. 
>  
> Thanks,
>  
> Andrew Fish
> 
> 
> On Nov 8, 2021, at 4:23 PM, Michael D Kinney <michael.d.kinney@intel.com <mailto:michael.d.kinney@intel.com>> wrote:
>  
> Hello,
>  
> Good information in this thread on code style.
>  
> Some of the topics apply to uncrustify and some are out of scope for what uncrustify can fix on its own.
>  
> I would like to focus on a date to convert all source code in edk2 repo using the uncrustify tool and to capture the other code style topics into their own thread andbugzillas.
>  
> I would like to propose a conversion date for uncrustify immediately after the edk2-stable202111 release on 2021-11-26.
>  
> I have been working with Michael Kubacki on a build comparison tool that verifies that the build generate the same obj/lib/dll/efi/fv/fd files before and after the uncrustify changes.  We would run and publish the results from this tool before committing the changes.
>  
> We need TianoCore community approval of the following:
>  
> Approve format of C source generated by the uncrustify.
> Approve uncrustify changes right after edk2-stable-202111 release. 
> Extend code freeze until these changes are committed.
> Require use of uncrustify tool before submitting patch review emails or PRs.
> The required version would be a formally released version  from the fork maintained by Michael Kubacki until the changes can be upstreamed.
> https://dev.azure.com/projectmu/Uncrustify <https://dev.azure.com/projectmu/Uncrustify>
> Add EDK II CI check to verify that all PRs submitted exactly match uncrustified version.  Reject PRs that do not match exactly.
> Implement a git hook available that would automatically run uncristufy before committing changes to a local branch of an edk2 repo.
>  
> Thanks,
>  
> Mike
>  
> From: Andrew Fish <afish@apple.com <mailto:afish@apple.com>> 
> Sent: Thursday, October 7, 2021 2:09 PM
> To: Marvin Häuser <mhaeuser@posteo.de <mailto:mhaeuser@posteo.de>>
> Cc: edk2-devel-groups-io <devel@edk2.groups.io <mailto:devel@edk2.groups.io>>; Kinney, Michael D <michael.d.kinney@intel.com <mailto:michael.d.kinney@intel.com>>; Leif Lindholm <leif@nuviainc.com <mailto:leif@nuviainc.com>>; mikuback@linux.microsoft.com <mailto:mikuback@linux.microsoft.com>; rebecca@nuviainc.com <mailto:rebecca@nuviainc.com>; Michael Kubacki <Michael.Kubacki@microsoft.com <mailto:Michael.Kubacki@microsoft.com>>; Bret Barkelew <Bret.Barkelew@microsoft.com <mailto:Bret.Barkelew@microsoft.com>>
> Subject: Re: [edk2-devel] Progress on getting Uncrustify working for EDK2?
>  
>  
> 
> 
> 
> On Oct 7, 2021, at 1:43 PM, Marvin Häuser <mhaeuser@posteo.de <mailto:mhaeuser@posteo.de>> wrote:
>  
> Hey Mike,
> Hey Andrew,
> 
> I'll just reply to both mails at once :)
> 
> On 07/10/2021 19:36, Andrew Fish wrote:
> 
> 
> 
> 
> 
> 
> On Oct 7, 2021, at 1:19 PM, Michael D Kinney <michael.d.kinney@intel.com <mailto:michael.d.kinney@intel.com>> wrote:
> 
> Hi Marvin,
> 
> Some comments below.
> 
> Mike
> 
> 
> 
> -----Original Message-----
> From:devel@edk2.groups.io <mailto:devel@edk2.groups.io><devel@edk2.groups.io <mailto:devel@edk2.groups.io>> On Behalf Of Marvin Häuser
> Sent: Thursday, October 7, 2021 11:31 AM
> To: Leif Lindholm <leif@nuviainc.com <mailto:leif@nuviainc.com>>;devel@edk2.groups.io <mailto:devel@edk2.groups.io>;mikuback@linux.microsoft.com <mailto:mikuback@linux.microsoft.com>
> Cc:rebecca@nuviainc.com <mailto:rebecca@nuviainc.com>; Michael Kubacki <Michael.Kubacki@microsoft.com <mailto:Michael.Kubacki@microsoft.com>>; Bret Barkelew <Bret.Barkelew@microsoft.com <mailto:Bret.Barkelew@microsoft.com>>;
> Kinney, Michael D <michael.d.kinney@intel.com <mailto:michael.d.kinney@intel.com>>
> Subject: Re: [edk2-devel] Progress on getting Uncrustify working for EDK2?
> 
> Good day,
> 
> +1, but while you're at it, can we have arguments not align to the
> function name...
> 
>   Status = Test (
>              a
>              );
> 
> ... but to the next natural indentation level?
> 
>   Status = Test (
>     a
>     );
> 
> Basically no IDE I have seen supports EDK II's style, and I wouldn't be
> keen on writing known-broken style to then rely on Uncrustify to fix it.
> 
> I also have heard some controversy regarding casts off-list, where some
> prefer no spaces after casts to stress the evaluation order, and some
> prefer spaces to have clearer visuals (as a cast *ideally* would be
> something rare that requires good justification). Just throwing that out
> there.
> 
> 
> For things unrelated to autoformat (so semi-offtopic) but still relevant
> to the coding spec:
> 
> 1. Allow STATIC functions (if the debugging concerns are still relevant,
> there could be another level of indirection, like RELEASE_STATIC)?
> 
> Debugging concerns are no longer relevant.  The suggestion in the BZ below
> is to remove the STATIC macro and allow EDK II sources to add 'static'
> to any functions it makes sense to use on.
> 
> https://bugzilla.tianocore.org/show_bug.cgi?id=1766 <https://bugzilla.tianocore.org/show_bug.cgi?id=1766>
> 
> Thanks! I'd keep STATIC actually just for the sake of not doing no-op changes that do not really do anything and for consistency with CONST, but whatever works really.
> 
> 
> 
> 
> 
> 
> 
> 2. Allow variable assignments on definition (basically non-static CONST
> variables are banned...)?
> 
> Are referring to use of pre-initialized CONST variables declared within
> a function?  I think Bret brought this topic up when implementing some
> unit tests and the suggestion to pass ECCC was to promote them to
> pre-initialized CONST global variables.
> 
> Yes.
> 
> 
> 
> 
> The challenges we have seen in the past with pre-initialized variables within
> a function is that they can cause compilers to inject use of memcpy() calls,
> especially if the variable being initialized on the stack is a structure.
> These cause build breaks today.
> 
> This issue is independent of CONST. I’m not sure a coding style tool is smart enough to catch this generically? You need an understanding of C types to know if the local variable assignment is going to trigger a memcpy().
> 
> What I’ve seen in the real world is the firmware compiles with -Os or LTO to fit int he ROM for DEBUG and RELEASE, and the optimizer optimizes away the call to memcpy. Then if you try to build NOOPT (or over ride the compiler flags on an individual driver/lib) you fail to link as only the NOOPT build injects the memcpy.
> 
> +1
> 
> 
> 
> 
> Thus I think the best way to enforce this rule is to compile a project NOOPT. I’m trying to remember are there flags to built to tell it to compile and skip the FD construction? Maybe we should advocate platforms add a NOOPT build target that just compiles the code, but does not create the FD?
> 
> I know there were stability concerns with intrinsics in the past, but memcpy() is in the standard, and the rest remained stable to my knowledge. Maybe it's time to fix the issues at the root? Works for us:
> https://github.com/acidanthera/OpenCorePkg/tree/master/Library/OcCompilerIntrinsicsLib <https://github.com/acidanthera/OpenCorePkg/tree/master/Library/OcCompilerIntrinsicsLib>
>  
> Marvin,
>  
> Good point. This would make the rule moot. So maybe just removing the requirement would be the easiest long term fix. 
>  
> Other embedded projects I know of do this too, and as you point out the compilers keep these APIs standard for folks the provide their own runtimes.
>  
> Thanks,
>  
> Andrew Fish
> 
> 
> 
> Best regards,
> Marvin
> 
> 
> 
> 
> Thanks,
> 
> Andrew Fish
> 
> 
> 
> 
> 
> 
> 
> 3. Allow variable declarations at any scope (I had some nasty shadowing
> bugs before, probably prohibit shadowing with warnings)?
> 
> By shadowing do you mean the declaration of the same variable name in
> multiple scoped within the same function?
> 
> 
> 
> 
> 4. Require that exactly all function declarations and all function
> definitions with no prior declaration must be documented (first
> direction is enforcing docs, second is prohibiting doc duplication, I've
> seen them go out-of-sync plenty of times)?
> 
> I agree that this can reduce duplication and sync issues.  The uncrustify
> tool being discussed here could not help clean this up or enforce this
> type of rule.  It is a good topic, but may need to be split out into its
> own thread.
> 
> 
> 
> 
> The latter bunch would not require any autoformat rules or reformatation
> of existing code, but would be target only new submissions in my
> opinion. Thoughts?
> 
> 
> Thanks for your efforts!
> 
> Best regards,
> Marvin
> 
> 
> Am 07.10.2021 um 12:48 schrieb Leif Lindholm:
> 
> 
> Hi Michael,
> 
> Apologies, I've owed you a response (promised off-list) for a while
> now.
> 
> First, let me say I hugely appreciate this effort. Apart from aligning
> the codebase(s), this will reduce manual reviewing effort
> substantially, as well as cutting down on number of rework cycles for
> developers.
> 
> Looking at the changes to (well, the comments in) uncrustify, this
> seems to be constrained to:
> - Newline after '(' for multi-line function calls.
> - Dealing with "(("/"))" for DEBUG macros.
> - Function pointer typedefs:
>   - typedef\nEFIAPI
>   - closing parentheses indentation
> 
> I don't think I've made any secret over the years that I am not a
> massive fan of the EDK2 coding style in general. So I think for any
> of its quirks that are substantial enough that they require not just
> custom configuration but actual new function added to existing code
> conformance tools, this would be an excellent point to sanitise the
> coding style instead.
> 
> Taking these in order:
> 
> Newline after '('
> -----------------
> I think we already reached a level of flexibility around this, where
> we don't actually enforce this (or single argument per
> line). Personally, I'd be happy to update the coding style as
> required instead.
> 
> DEBUG macro parentheses
> -----------------------
> How does uncrustify treat DEBUG macros without this modification?
> Do we start getting everything turned into multi-level indented
> multi-line statements without this change?
> 
> Function pointer typedefs:
> --------------------------
> I don't see that function pointer typedefs need to rigidly follow the
> same pattern as the declaration of functions implementing them. Could
> we update the coding style (if needed) instead?
> 
> Best Regards,
> 
> Leif
> 
> On Mon, Aug 16, 2021 at 16:00:38 -0400, Michael Kubacki wrote:
> 
> 
> The edk2 branch was created here:
> https://github.com/makubacki/edk2/tree/uncrustify_poc_2 <https://github.com/makubacki/edk2/tree/uncrustify_poc_2>
> 
> We have a Project Mu fork with custom changes to the Uncrustify tool to help
> comply with EDK II formatting here:
> https://dev.azure.com/projectmu/_git/Uncrustify <https://dev.azure.com/projectmu/_git/Uncrustify>
> 
> The latest information about the status and how to experiment with the
> configuration file and the tool are in that fork:
> https://dev.azure.com/projectmu/Uncrustify/_wiki/wikis/Uncrustify.wiki/1/Project-Mu-(EDK-II)-Fork-Readme <https://dev.azure.com/projectmu/Uncrustify/_wiki/wikis/Uncrustify.wiki/1/Project-Mu-(EDK-II)-Fork-Readme>
> 
> 
> 
> That said, I have also finished a CI plugin to run Uncrustify that should be
> ready soon to initially deploy in Project Mu. Before doing so, I am trying
> to settle on an initial configuration file that less strictly but more
> reliably formats the code than in the examples in those branches. For
> example, remove heuristics that when run against the same set of code
> multiple times can produce different results. An example would be a rule
> that reformats code because it exceeds a specified column width on one run
> but on the next run that reformatted code triggers a different rule to
> further align the code and so on. At least initially, some rules might be
> tweaked in a more conservative approach that can be tightened in the future.
> Once this configuration file is ready, we will baseline Project Mu code as
> an example and turn on the plugin. The CI plugin runs Uncrustify against
> modified files and if there's any changes, indicating a formatting
> deviation, the diff chunks are saved in a log so they can be viewed as a
> build artifact.
> 
> I am making progress on the updated config file and I should be able to post
> a "uncrustify_poc_3" branch soon with the results.
> 
> Regarding indentation, Marvin is right that Uncrustify cannot support edk2
> indentation style out-of-box. Some changes are made in that fork to handle
> the formatting. At this point, it can handle the indentation in the cases
> I've seen. Uncrustify does potentially give us the ability to massively
> deploy changes across the codebase in case a decision were made to change
> the style.
> 
> Thanks,
> Michael
> 
> On 8/16/2021 3:39 PM, Marvin Häuser wrote:
> 
> 
> Hey Rebecca,
> 
> I think even Uncrustify has issues with the EDK II indentation style.
> You might want to check the UEFI Talkbox Discord server, I had a brief
> chat with Michael about it there. I don't think realistically any tool
> supports EDK II's indentation style however, so I'd propose it is
> changed. This could be for new submissions only, or actually the entire
> codebase could be reformatted at once with a good tool setup. While this
> screws with git blame, the (to my understanding) decided on CRLF -> LF
> change does that anyway, so at least two evils could be dealt with in
> one go really.
> 
> Best regards,
> Marvin
> 
> On 16/08/2021 21:34, Rebecca Cran wrote:
> 
> 
> 
> cc devel@ .
> 
> On 8/16/21 1:33 PM, Rebecca Cran wrote:
> 
> 
> 
> I noticed a message on Twitter about an idea of using Uncrustify
> for EDK2 instead of the ECC tool, and came across https://www.mail- <https://www.mail-/>
> archive.com/search?l=devel@edk2.groups.io&q=subject:%22Re%5C%3A+%5C%5Bedk2%5C- <mailto:archive.com/search?l=devel@edk2.groups.io&q=subject:%22Re%5C%3A+%5C%5Bedk2%5C->
> devel%5C%5D+TianoCore+Community+Meeting+Minutes+%5C-+2%5C%2F4%22&o=newest&f=1
> 
> 
> .
> 
> I was wondering if there's been any progress on it that I could
> check out?
> 
> 
> Michael Kubacki: in that message, you said:
> 
> "I'm planning to put up a branch that we can use as a reference
> for a conversation around uncrustify in the next couple of
> weeks."
> 
> 
> Did you end up creating that branch, and if so could you provide
> a link to it please?
> 
> 
> --
> Rebecca Cran
> 
>  
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
>  
> 


[-- Attachment #2: Type: text/html, Size: 50997 bytes --]

Re: [edk2-devel] Progress on getting Uncrustify working for EDK2?

[Michael D Kinney]

[-- Attachment #1: Type: text/plain, Size: 16993 bytes --]

Andrew,

I think Michael Kubacki started with a nuget feed because that can be easily used by EDK II CI agents.

However, that does not work as easily for all development environments using Windows, Linux, and MacOS.  Creating releases that can be easily installed by developers is critical for success.

For MacOS, there is a homebrew recipe.  You should just have to update the URL to the uncrustify fork.

https://macappstore.org/uncrustify/

Adding the installation information to the EDK II Getting Started page would be a good place to capture the developer install.

Mike

From: Andrew Fish <afish@apple.com>
Sent: Monday, November 8, 2021 6:47 PM
To: Kinney, Michael D <michael.d.kinney@intel.com>
Cc: devel@edk2.groups.io; Marvin Häuser <mhaeuser@posteo.de>; Michael Kubacki <Michael.Kubacki@microsoft.com>; Leif Lindholm <leif@nuviainc.com>; mikuback@linux.microsoft.com; rebecca@nuviainc.com; Bret Barkelew <Bret.Barkelew@microsoft.com>
Subject: Re: [edk2-devel] Progress on getting Uncrustify working for EDK2?




On Nov 8, 2021, at 5:13 PM, Kinney, Michael D <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>> wrote:

HI Andrew,

Great feedback.

What your preferred way to install a tool like this?  If we collect that data from the community, we can make sure the pipeline generates the right type of installers.


I could not figure out how to download an installer from the links.

If I go to http://uncrustify.sourceforge.net I see https://sourceforge.net/projects/uncrustify/files/ and a button to download binaries. Not ideal that it is only for Windows but at least the workflow was obvious. Looks like I need to build my own version for macOS, not ideal but I can at least figure that out.

Worst case we can have a Tianocore.org<http://Tianocore.org> page to describe a simple recipe to install the tool. With step by step instructions some one can just type in without thinking too much.

Thanks,

Andrew Fish


Thanks,

Mike

From: Andrew Fish <afish@apple.com<mailto:afish@apple.com>>
Sent: Monday, November 8, 2021 5:09 PM
To: devel@edk2.groups.io<mailto:devel@edk2.groups.io>; Kinney, Michael D <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>>
Cc: Marvin Häuser <mhaeuser@posteo.de<mailto:mhaeuser@posteo.de>>; Michael Kubacki <Michael.Kubacki@microsoft.com<mailto:Michael.Kubacki@microsoft.com>>; Leif Lindholm <leif@nuviainc.com<mailto:leif@nuviainc.com>>; mikuback@linux.microsoft.com<mailto:mikuback@linux.microsoft.com>; rebecca@nuviainc.com<mailto:rebecca@nuviainc.com>; Bret Barkelew <Bret.Barkelew@microsoft.com<mailto:Bret.Barkelew@microsoft.com>>
Subject: Re: [edk2-devel] Progress on getting Uncrustify working for EDK2?

MIke,

I could not figure out how to download uncrustify tool from the provided link. 99% of the people are just going to want to install the tool, not be a developer of the fork. We should have some simple instructions on how to download the tool.

The link points to something git web view looking Azure DevOps page and talks about this Nuget thing I know nothing about. I ran out of time and had to give up trying to download the tool.

Thanks,

Andrew Fish



On Nov 8, 2021, at 4:23 PM, Michael D Kinney <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>> wrote:

Hello,

Good information in this thread on code style.

Some of the topics apply to uncrustify and some are out of scope for what uncrustify can fix on its own.

I would like to focus on a date to convert all source code in edk2 repo using the uncrustify tool and to capture the other code style topics into their own thread andbugzillas.

I would like to propose a conversion date for uncrustify immediately after the edk2-stable202111 release on 2021-11-26.

I have been working with Michael Kubacki on a build comparison tool that verifies that the build generate the same obj/lib/dll/efi/fv/fd files before and after the uncrustify changes.  We would run and publish the results from this tool before committing the changes.

We need TianoCore community approval of the following:


  1.  Approve format of C source generated by the uncrustify.
  2.  Approve uncrustify changes right after edk2-stable-202111 release.

     *   Extend code freeze until these changes are committed.

  1.  Require use of uncrustify tool before submitting patch review emails or PRs.

     *   The required version would be a formally released version  from the fork maintained by Michael Kubacki until the changes can be upstreamed.
     *   https://dev.azure.com/projectmu/Uncrustify

  1.  Add EDK II CI check to verify that all PRs submitted exactly match uncrustified version.  Reject PRs that do not match exactly.
  2.  Implement a git hook available that would automatically run uncristufy before committing changes to a local branch of an edk2 repo.

Thanks,

Mike

From: Andrew Fish <afish@apple.com<mailto:afish@apple.com>>
Sent: Thursday, October 7, 2021 2:09 PM
To: Marvin Häuser <mhaeuser@posteo.de<mailto:mhaeuser@posteo.de>>
Cc: edk2-devel-groups-io <devel@edk2.groups.io<mailto:devel@edk2.groups.io>>; Kinney, Michael D <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>>; Leif Lindholm <leif@nuviainc.com<mailto:leif@nuviainc.com>>; mikuback@linux.microsoft.com<mailto:mikuback@linux.microsoft.com>; rebecca@nuviainc.com<mailto:rebecca@nuviainc.com>; Michael Kubacki <Michael.Kubacki@microsoft.com<mailto:Michael.Kubacki@microsoft.com>>; Bret Barkelew <Bret.Barkelew@microsoft.com<mailto:Bret.Barkelew@microsoft.com>>
Subject: Re: [edk2-devel] Progress on getting Uncrustify working for EDK2?






On Oct 7, 2021, at 1:43 PM, Marvin Häuser <mhaeuser@posteo.de<mailto:mhaeuser@posteo.de>> wrote:

Hey Mike,
Hey Andrew,

I'll just reply to both mails at once :)

On 07/10/2021 19:36, Andrew Fish wrote:








On Oct 7, 2021, at 1:19 PM, Michael D Kinney <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>> wrote:

Hi Marvin,

Some comments below.

Mike




-----Original Message-----
From:devel@edk2.groups.io<mailto:devel@edk2.groups.io><devel@edk2.groups.io<mailto:devel@edk2.groups.io>> On Behalf Of Marvin Häuser
Sent: Thursday, October 7, 2021 11:31 AM
To: Leif Lindholm <leif@nuviainc.com<mailto:leif@nuviainc.com>>;devel@edk2.groups.io<mailto:devel@edk2.groups.io>;mikuback@linux.microsoft.com<mailto:mikuback@linux.microsoft.com>
Cc:rebecca@nuviainc.com<mailto:rebecca@nuviainc.com>; Michael Kubacki <Michael.Kubacki@microsoft.com<mailto:Michael.Kubacki@microsoft.com>>; Bret Barkelew <Bret.Barkelew@microsoft.com<mailto:Bret.Barkelew@microsoft.com>>;
Kinney, Michael D <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>>
Subject: Re: [edk2-devel] Progress on getting Uncrustify working for EDK2?

Good day,

+1, but while you're at it, can we have arguments not align to the
function name...

  Status = Test (
             a
             );

... but to the next natural indentation level?

  Status = Test (
    a
    );

Basically no IDE I have seen supports EDK II's style, and I wouldn't be
keen on writing known-broken style to then rely on Uncrustify to fix it.

I also have heard some controversy regarding casts off-list, where some
prefer no spaces after casts to stress the evaluation order, and some
prefer spaces to have clearer visuals (as a cast *ideally* would be
something rare that requires good justification). Just throwing that out
there.


For things unrelated to autoformat (so semi-offtopic) but still relevant
to the coding spec:

1. Allow STATIC functions (if the debugging concerns are still relevant,
there could be another level of indirection, like RELEASE_STATIC)?

Debugging concerns are no longer relevant.  The suggestion in the BZ below
is to remove the STATIC macro and allow EDK II sources to add 'static'
to any functions it makes sense to use on.

https://bugzilla.tianocore.org/show_bug.cgi?id=1766

Thanks! I'd keep STATIC actually just for the sake of not doing no-op changes that do not really do anything and for consistency with CONST, but whatever works really.









2. Allow variable assignments on definition (basically non-static CONST
variables are banned...)?

Are referring to use of pre-initialized CONST variables declared within
a function?  I think Bret brought this topic up when implementing some
unit tests and the suggestion to pass ECCC was to promote them to
pre-initialized CONST global variables.

Yes.





The challenges we have seen in the past with pre-initialized variables within
a function is that they can cause compilers to inject use of memcpy() calls,
especially if the variable being initialized on the stack is a structure.
These cause build breaks today.

This issue is independent of CONST. I’m not sure a coding style tool is smart enough to catch this generically? You need an understanding of C types to know if the local variable assignment is going to trigger a memcpy().

What I’ve seen in the real world is the firmware compiles with -Os or LTO to fit int he ROM for DEBUG and RELEASE, and the optimizer optimizes away the call to memcpy. Then if you try to build NOOPT (or over ride the compiler flags on an individual driver/lib) you fail to link as only the NOOPT build injects the memcpy.

+1





Thus I think the best way to enforce this rule is to compile a project NOOPT. I’m trying to remember are there flags to built to tell it to compile and skip the FD construction? Maybe we should advocate platforms add a NOOPT build target that just compiles the code, but does not create the FD?

I know there were stability concerns with intrinsics in the past, but memcpy() is in the standard, and the rest remained stable to my knowledge. Maybe it's time to fix the issues at the root? Works for us:
https://github.com/acidanthera/OpenCorePkg/tree/master/Library/OcCompilerIntrinsicsLib

Marvin,

Good point. This would make the rule moot. So maybe just removing the requirement would be the easiest long term fix.

Other embedded projects I know of do this too, and as you point out the compilers keep these APIs standard for folks the provide their own runtimes.

Thanks,

Andrew Fish




Best regards,
Marvin





Thanks,

Andrew Fish









3. Allow variable declarations at any scope (I had some nasty shadowing
bugs before, probably prohibit shadowing with warnings)?

By shadowing do you mean the declaration of the same variable name in
multiple scoped within the same function?





4. Require that exactly all function declarations and all function
definitions with no prior declaration must be documented (first
direction is enforcing docs, second is prohibiting doc duplication, I've
seen them go out-of-sync plenty of times)?

I agree that this can reduce duplication and sync issues.  The uncrustify
tool being discussed here could not help clean this up or enforce this
type of rule.  It is a good topic, but may need to be split out into its
own thread.





The latter bunch would not require any autoformat rules or reformatation
of existing code, but would be target only new submissions in my
opinion. Thoughts?


Thanks for your efforts!

Best regards,
Marvin


Am 07.10.2021 um 12:48 schrieb Leif Lindholm:



Hi Michael,

Apologies, I've owed you a response (promised off-list) for a while
now.

First, let me say I hugely appreciate this effort. Apart from aligning
the codebase(s), this will reduce manual reviewing effort
substantially, as well as cutting down on number of rework cycles for
developers.

Looking at the changes to (well, the comments in) uncrustify, this
seems to be constrained to:
- Newline after '(' for multi-line function calls.
- Dealing with "(("/"))" for DEBUG macros.
- Function pointer typedefs:
  - typedef\nEFIAPI
  - closing parentheses indentation

I don't think I've made any secret over the years that I am not a
massive fan of the EDK2 coding style in general. So I think for any
of its quirks that are substantial enough that they require not just
custom configuration but actual new function added to existing code
conformance tools, this would be an excellent point to sanitise the
coding style instead.

Taking these in order:

Newline after '('
-----------------
I think we already reached a level of flexibility around this, where
we don't actually enforce this (or single argument per
line). Personally, I'd be happy to update the coding style as
required instead.

DEBUG macro parentheses
-----------------------
How does uncrustify treat DEBUG macros without this modification?
Do we start getting everything turned into multi-level indented
multi-line statements without this change?

Function pointer typedefs:
--------------------------
I don't see that function pointer typedefs need to rigidly follow the
same pattern as the declaration of functions implementing them. Could
we update the coding style (if needed) instead?

Best Regards,

Leif

On Mon, Aug 16, 2021 at 16:00:38 -0400, Michael Kubacki wrote:



The edk2 branch was created here:
https://github.com/makubacki/edk2/tree/uncrustify_poc_2

We have a Project Mu fork with custom changes to the Uncrustify tool to help
comply with EDK II formatting here:
https://dev.azure.com/projectmu/_git/Uncrustify

The latest information about the status and how to experiment with the
configuration file and the tool are in that fork:
https://dev.azure.com/projectmu/Uncrustify/_wiki/wikis/Uncrustify.wiki/1/Project-Mu-(EDK-II)-Fork-Readme




That said, I have also finished a CI plugin to run Uncrustify that should be
ready soon to initially deploy in Project Mu. Before doing so, I am trying
to settle on an initial configuration file that less strictly but more
reliably formats the code than in the examples in those branches. For
example, remove heuristics that when run against the same set of code
multiple times can produce different results. An example would be a rule
that reformats code because it exceeds a specified column width on one run
but on the next run that reformatted code triggers a different rule to
further align the code and so on. At least initially, some rules might be
tweaked in a more conservative approach that can be tightened in the future.
Once this configuration file is ready, we will baseline Project Mu code as
an example and turn on the plugin. The CI plugin runs Uncrustify against
modified files and if there's any changes, indicating a formatting
deviation, the diff chunks are saved in a log so they can be viewed as a
build artifact.

I am making progress on the updated config file and I should be able to post
a "uncrustify_poc_3" branch soon with the results.

Regarding indentation, Marvin is right that Uncrustify cannot support edk2
indentation style out-of-box. Some changes are made in that fork to handle
the formatting. At this point, it can handle the indentation in the cases
I've seen. Uncrustify does potentially give us the ability to massively
deploy changes across the codebase in case a decision were made to change
the style.

Thanks,
Michael

On 8/16/2021 3:39 PM, Marvin Häuser wrote:



Hey Rebecca,

I think even Uncrustify has issues with the EDK II indentation style.
You might want to check the UEFI Talkbox Discord server, I had a brief
chat with Michael about it there. I don't think realistically any tool
supports EDK II's indentation style however, so I'd propose it is
changed. This could be for new submissions only, or actually the entire
codebase could be reformatted at once with a good tool setup. While this
screws with git blame, the (to my understanding) decided on CRLF -> LF
change does that anyway, so at least two evils could be dealt with in
one go really.

Best regards,
Marvin

On 16/08/2021 21:34, Rebecca Cran wrote:




cc devel@ .

On 8/16/21 1:33 PM, Rebecca Cran wrote:




I noticed a message on Twitter about an idea of using Uncrustify
for EDK2 instead of the ECC tool, and came across https://www.mail-<https://www.mail-/>
archive.com/search?l=devel@edk2.groups.io&q=subject:%22Re%5C%3A+%5C%5Bedk2%5C-<mailto:archive.com/search?l=devel@edk2.groups.io&q=subject:%22Re%5C%3A+%5C%5Bedk2%5C->
devel%5C%5D+TianoCore+Community+Meeting+Minutes+%5C-+2%5C%2F4%22&o=newest&f=1



.

I was wondering if there's been any progress on it that I could
check out?


Michael Kubacki: in that message, you said:

"I'm planning to put up a branch that we can use as a reference
for a conversation around uncrustify in the next couple of
weeks."


Did you end up creating that branch, and if so could you provide
a link to it please?


--
Rebecca Cran




























[-- Attachment #2: Type: text/html, Size: 95948 bytes --]

Re: [edk2-devel] Progress on getting Uncrustify working for EDK2?

[Marvin Häuser]

Hey all,

Thanks for the effort!

1. If virtually everyone will need Uncrustify, why cannot it be built along with BaseTools from a submodule? Especially with the fork that makes sense, after that it depends on the upstream (it does not look too nice to me).

2. Does this cover CRLF->LF?

3. Will there be a list of implicit changes from the current code style?

4. I feel like if the other code style changes are not tackled now, they will never be, because who wants to deal with continuous cosmetic commits. But as long as there is a tool now to enforce the current one, that's not a dealbreaker at all. :)

Best regards,
Marvin

09.11.2021 04:03:06 Kinney, Michael D <michael.d.kinney@intel.com>:

> Andrew,
>  
> I think Michael Kubacki started with a nuget feed because that can be easily used by EDK II CI agents.
>  
> However, that does not work as easily for all development environments using Windows, Linux, and MacOS.  Creating releases that can be easily installed by developers is critical for success.
>  
> For MacOS, there is a homebrew recipe.  You should just have to update the URL to the uncrustify fork.
>  
> https://macappstore.org/uncrustify/
>  
> Adding the installation information to the EDK II Getting Started page would be a good place to capture the developer install.
>  
> Mike
>  
> *From:* Andrew Fish <afish@apple.com>
> *Sent:* Monday, November 8, 2021 6:47 PM
> *To:* Kinney, Michael D <michael.d.kinney@intel.com>
> *Cc:* devel@edk2.groups.io; Marvin Häuser <mhaeuser@posteo.de>; Michael Kubacki <Michael.Kubacki@microsoft.com>; Leif Lindholm <leif@nuviainc.com>; mikuback@linux.microsoft.com; rebecca@nuviainc.com; Bret Barkelew <Bret.Barkelew@microsoft.com>
> *Subject:* Re: [edk2-devel] Progress on getting Uncrustify working for EDK2?
>  
>  
> 
> 
> On Nov 8, 2021, at 5:13 PM, Kinney, Michael D <michael.d.kinney@intel.com> wrote:
>  
> HI Andrew,
>  
> Great feedback.
>  
> What your preferred way to install a tool like this?  If we collect that data from the community, we can make sure the pipeline generates the right type of installers.
>  
>  
> I could not figure out how to download an installer from the links. 
>  
> If I go to http://uncrustify.sourceforge.net I see https://sourceforge.net/projects/uncrustify/files/ and a button to download binaries. Not ideal that it is only for Windows but at least the workflow was obvious. Looks like I need to build my own version for macOS, not ideal but I can at least figure that out. 
>  
> Worst case we can have a Tianocore.org[http://Tianocore.org] page to describe a simple recipe to install the tool. With step by step instructions some one can just type in without thinking too much. 
>  
> Thanks,
>  
> Andrew Fish
> 
> 
> Thanks,
>  
> Mike
>  
> *From:* Andrew Fish <afish@apple.com> 
> *Sent:* Monday, November 8, 2021 5:09 PM
> *To:* devel@edk2.groups.io; Kinney, Michael D <michael.d.kinney@intel.com>
> *Cc:* Marvin Häuser <mhaeuser@posteo.de>; Michael Kubacki <Michael.Kubacki@microsoft.com>; Leif Lindholm <leif@nuviainc.com>; mikuback@linux.microsoft.com; rebecca@nuviainc.com; Bret Barkelew <Bret.Barkelew@microsoft.com>
> *Subject:* Re: [edk2-devel] Progress on getting Uncrustify working for EDK2?
>  
> MIke,
>  
> I could not figure out how to download uncrustify tool from the provided link. 99% of the people are just going to want to install the tool, not be a developer of the fork. We should have some simple instructions on how to download the tool. 
>  
> The link points to something git web view looking Azure DevOps page and talks about this Nuget thing I know nothing about. I ran out of time and had to give up trying to download the tool. 
>  
> Thanks,
>  
> Andrew Fish
> 
> 
> 
> On Nov 8, 2021, at 4:23 PM, Michael D Kinney <michael.d.kinney@intel.com> wrote:
>  
> Hello,
>  
> Good information in this thread on code style.
>  
> Some of the topics apply to uncrustify and some are out of scope for what uncrustify can fix on its own.
>  
> I would like to focus on a date to convert all source code in edk2 repo using the uncrustify tool and to capture the other code style topics into their own thread andbugzillas.
>  
> I would like to propose a conversion date for uncrustify immediately after the edk2-stable202111 release on 2021-11-26.
>  
> I have been working with Michael Kubacki on a build comparison tool that verifies that the build generate the same obj/lib/dll/efi/fv/fd files before and after the uncrustify changes.  We would run and publish the results from this tool before committing the changes.
>  
> We need TianoCore community approval of the following:
>  
1. > Approve format of C source generated by the uncrustify.
2. > Approve uncrustify changes right after edk2-stable-202111 release. 
  1. > Extend code freeze until these changes are committed.
3. > Require use of uncrustify tool before submitting patch review emails or PRs.
  1. > The required version would be a formally released version  from the fork maintained by Michael Kubacki until the changes can be upstreamed.
  2. > https://dev.azure.com/projectmu/Uncrustify
4. > Add EDK II CI check to verify that all PRs submitted exactly match uncrustified version.  Reject PRs that do not match exactly.
5. > Implement a git hook available that would automatically run uncristufy before committing changes to a local branch of an edk2 repo.
>  
> Thanks,
>  
> Mike
>  
> *From:* Andrew Fish <afish@apple.com> 
> *Sent:* Thursday, October 7, 2021 2:09 PM
> *To:* Marvin Häuser <mhaeuser@posteo.de>
> *Cc:* edk2-devel-groups-io <devel@edk2.groups.io>; Kinney, Michael D <michael.d.kinney@intel.com>; Leif Lindholm <leif@nuviainc.com>; mikuback@linux.microsoft.com; rebecca@nuviainc.com; Michael Kubacki <Michael.Kubacki@microsoft.com>; Bret Barkelew <Bret.Barkelew@microsoft.com>
> *Subject:* Re: [edk2-devel] Progress on getting Uncrustify working for EDK2?
>  
>  
> 
> 
> 
> 
> On Oct 7, 2021, at 1:43 PM, Marvin Häuser <mhaeuser@posteo.de> wrote:
>  
> Hey Mike,
> Hey Andrew,
> 
> I'll just reply to both mails at once :)
> 
> On 07/10/2021 19:36, Andrew Fish wrote:
> 
> 
> 
> 
> 
> 
> 
> 
> …
> 
> Thanks! I'd keep STATIC actually just for the sake of not doing no-op changes that do not really do anything and for consistency with CONST, but whatever works really.
> 
> 
> 
> 
> …
> 
> Yes.
> 
> 
> 
> 
> …
> 
> This issue is independent of CONST. I’m not sure a coding style tool is smart enough to catch this generically? You need an understanding of C types to know if the local variable assignment is going to trigger a memcpy().
> 
> What I’ve seen in the real world is the firmware compiles with -Os or LTO to fit int he ROM for DEBUG and RELEASE, and the optimizer optimizes away the call to memcpy. Then if you try to build NOOPT (or over ride the compiler flags on an individual driver/lib) you fail to link as only the NOOPT build injects the memcpy.
> 
> +1
> 
> 
> 
> 
> 
> Thus I think the best way to enforce this rule is to compile a project NOOPT. I’m trying to remember are there flags to built to tell it to compile and skip the FD construction? Maybe we should advocate platforms add a NOOPT build target that just compiles the code, but does not create the FD?
> 
> I know there were stability concerns with intrinsics in the past, but memcpy() is in the standard, and the rest remained stable to my knowledge. Maybe it's time to fix the issues at the root? Works for us:
> https://github.com/acidanthera/OpenCorePkg/tree/master/Library/OcCompilerIntrinsicsLib
> 
> 
>  
> Marvin,
>  
> Good point. This would make the rule moot. So maybe just removing the requirement would be the easiest long term fix. 
>  
> Other embedded projects I know of do this too, and as you point out the compilers keep these APIs standard for folks the provide their own runtimes.
>  
> Thanks,
>  
> Andrew Fish
> 
> 
> 
> 
> Best regards,
> Marvin
> 
> 
> 
> 
> 
> Thanks,
> 
> Andrew Fish
> 
> 
> 
> 
> …
>  
> 
>  

Re: [edk2-devel] Progress on getting Uncrustify working for EDK2?

[Gerd Hoffmann]

  Hi,

>   3.  Require use of uncrustify tool before submitting patch review emails or PRs.
>      *   The required version would be a formally released version  from the fork maintained by Michael Kubacki until the changes can be upstreamed.
>      *   https://dev.azure.com/projectmu/Uncrustify

Can we please *first* get the changes merged to upstream uncrustify?

That'll make the whole process much less painful because the usual
software repositories (linux distro packages, macos homebrew, ...)
can be used to install uncrustify then, and it's also less confusing if
developers don't have to juggle with different uncrustify variants
(upstream vs. edk2).

thanks,
  Gerd

Re: [edk2-devel] Progress on getting Uncrustify working for EDK2?

[Michael Brown]

On 09/11/2021 08:40, Gerd Hoffmann wrote:
>>    3.  Require use of uncrustify tool before submitting patch review emails or PRs.
>>       *   The required version would be a formally released version  from the fork maintained by Michael Kubacki until the changes can be upstreamed.
>>       *   https://dev.azure.com/projectmu/Uncrustify
> 
> Can we please *first* get the changes merged to upstream uncrustify?
> 
> That'll make the whole process much less painful because the usual
> software repositories (linux distro packages, macos homebrew, ...)
> can be used to install uncrustify then, and it's also less confusing if
> developers don't have to juggle with different uncrustify variants
> (upstream vs. edk2).

I very strongly agree with this.  It's always a bad sign when a project 
requires installation of a custom version of a widely used tool.

Michael

Re: [edk2-devel] Progress on getting Uncrustify working for EDK2?

[Leif Lindholm]

On Tue, Nov 09, 2021 at 09:40:02 +0100, Gerd Hoffmann wrote:
>   Hi,
> 
> >   3.  Require use of uncrustify tool before submitting patch review emails or PRs.
> >      *   The required version would be a formally released version  from the fork maintained by Michael Kubacki until the changes can be upstreamed.
> >      *   https://dev.azure.com/projectmu/Uncrustify
> 
> Can we please *first* get the changes merged to upstream uncrustify?
> 
> That'll make the whole process much less painful because the usual
> software repositories (linux distro packages, macos homebrew, ...)
> can be used to install uncrustify then, and it's also less confusing if
> developers don't have to juggle with different uncrustify variants
> (upstream vs. edk2).

Whilst I agree in principle...

This means postponing automated coding style changes until 2023
(Debian stable), 2025 (Ubuntu LTS), ??? (RHEL10), or even later
... and I'd rather not.

I like Marvin's suggestion of a submodule. Which we could drop once
no longer needed.

/
    Leif

Re: [edk2-devel] Progress on getting Uncrustify working for EDK2?

[Gerd Hoffmann]

On Tue, Nov 09, 2021 at 12:36:30PM +0000, Leif Lindholm wrote:
> On Tue, Nov 09, 2021 at 09:40:02 +0100, Gerd Hoffmann wrote:
> >   Hi,
> > 
> > >   3.  Require use of uncrustify tool before submitting patch review emails or PRs.
> > >      *   The required version would be a formally released version  from the fork maintained by Michael Kubacki until the changes can be upstreamed.
> > >      *   https://dev.azure.com/projectmu/Uncrustify
> > 
> > Can we please *first* get the changes merged to upstream uncrustify?
> > 
> > That'll make the whole process much less painful because the usual
> > software repositories (linux distro packages, macos homebrew, ...)
> > can be used to install uncrustify then, and it's also less confusing if
> > developers don't have to juggle with different uncrustify variants
> > (upstream vs. edk2).
> 
> Whilst I agree in principle...
> 
> This means postponing automated coding style changes until 2023
> (Debian stable), 2025 (Ubuntu LTS), ??? (RHEL10), or even later
> ... and I'd rather not.
> 
> I like Marvin's suggestion of a submodule. Which we could drop once
> no longer needed.

I'm fine with a submodule (temporary & optional) so we don't have to
wait until all distros (especially the LTS ones which are slower) have
picked up the latest uncrustify release.

But I still think upstreaming the uncrustify changes should be the first
step.

take care,
  Gerd

Re: [edk2-devel] Progress on getting Uncrustify working for EDK2?

[Michael D Kinney]

Submodule within which repo?  What would be the proposed location?

Would a fork of uncrustify maintained as a repo under TianoCore work as well?

There are CI checks (including extensive unit tests) and release generation that are built into uncrustify repo and I do not know if those will be functional if it is maintained as a submodule.

Thanks,

Mike

> -----Original Message-----
> From: Leif Lindholm <leif@nuviainc.com>
> Sent: Tuesday, November 9, 2021 4:37 AM
> To: Gerd Hoffmann <kraxel@redhat.com>
> Cc: devel@edk2.groups.io; Kinney, Michael D <michael.d.kinney@intel.com>; Andrew Fish <afish@apple.com>; Marvin Häuser
> <mhaeuser@posteo.de>; Michael Kubacki <Michael.Kubacki@microsoft.com>; mikuback@linux.microsoft.com; rebecca@nuviainc.com;
> Bret Barkelew <Bret.Barkelew@microsoft.com>
> Subject: Re: [edk2-devel] Progress on getting Uncrustify working for EDK2?
> 
> On Tue, Nov 09, 2021 at 09:40:02 +0100, Gerd Hoffmann wrote:
> >   Hi,
> >
> > >   3.  Require use of uncrustify tool before submitting patch review emails or PRs.
> > >      *   The required version would be a formally released version  from the fork maintained by Michael Kubacki until
> the changes can be upstreamed.
> > >      *   https://dev.azure.com/projectmu/Uncrustify
> >
> > Can we please *first* get the changes merged to upstream uncrustify?
> >
> > That'll make the whole process much less painful because the usual
> > software repositories (linux distro packages, macos homebrew, ...)
> > can be used to install uncrustify then, and it's also less confusing if
> > developers don't have to juggle with different uncrustify variants
> > (upstream vs. edk2).
> 
> Whilst I agree in principle...
> 
> This means postponing automated coding style changes until 2023
> (Debian stable), 2025 (Ubuntu LTS), ??? (RHEL10), or even later
> ... and I'd rather not.
> 
> I like Marvin's suggestion of a submodule. Which we could drop once
> no longer needed.
> 
> /
>     Leif

Re: [edk2-devel] Progress on getting Uncrustify working for EDK2?

[Michael D Kinney]

[-- Attachment #1: Type: text/plain, Size: 15799 bytes --]

Liming,


  *   Uncrustify will not be run on any submodules.
  *   Uncrustify will not be run on BaseTools.
     *   The focus is firmware sources.
     *   We would like to look for opportunities to convert BaseTools C components to Python.  We already have specs/tools to format Python sources.
  *   My proposal is to finish the edk2-stable202111 as planned on Nov 26.  Then extend the hard freeze after Nov 26 to apply uncrustify code style changes.  The target date range would be Nov 27-Dec 3.

Thanks,

Mike

From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of gaoliming
Sent: Monday, November 8, 2021 5:12 PM
To: devel@edk2.groups.io; Kinney, Michael D <michael.d.kinney@intel.com>; 'Andrew Fish' <afish@apple.com>; 'Marvin Häuser' <mhaeuser@posteo.de>; 'Michael Kubacki' <Michael.Kubacki@microsoft.com>
Cc: 'Leif Lindholm' <leif@nuviainc.com>; mikuback@linux.microsoft.com; rebecca@nuviainc.com; 'Bret Barkelew' <Bret.Barkelew@microsoft.com>
Subject: 回复: [edk2-devel] Progress on getting Uncrustify working for EDK2?

Mike：
  I add my comments below.

发件人: devel@edk2.groups.io<mailto:devel@edk2.groups.io> <devel@edk2.groups.io<mailto:devel@edk2.groups.io>> 代表 Michael D Kinney
发送时间: 2021年11月9日 8:24
收件人: Andrew Fish <afish@apple.com<mailto:afish@apple.com>>; Marvin Häuser <mhaeuser@posteo.de<mailto:mhaeuser@posteo.de>>; Kinney, Michael D <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>>; Michael Kubacki <Michael.Kubacki@microsoft.com<mailto:Michael.Kubacki@microsoft.com>>
抄送: edk2-devel-groups-io <devel@edk2.groups.io<mailto:devel@edk2.groups.io>>; Leif Lindholm <leif@nuviainc.com<mailto:leif@nuviainc.com>>; mikuback@linux.microsoft.com<mailto:mikuback@linux.microsoft.com>; rebecca@nuviainc.com<mailto:rebecca@nuviainc.com>; Bret Barkelew <Bret.Barkelew@microsoft.com<mailto:Bret.Barkelew@microsoft.com>>
主题: Re: [edk2-devel] Progress on getting Uncrustify working for EDK2?

Hello,

Good information in this thread on code style.

Some of the topics apply to uncrustify and some are out of scope for what uncrustify can fix on its own.

I would like to focus on a date to convert all source code in edk2 repo using the uncrustify tool and to capture the other code style topics into their own thread and bugzillas.

I would like to propose a conversion date for uncrustify immediately after the edk2-stable202111 release on 2021-11-26.

I have been working with Michael Kubacki on a build comparison tool that verifies that the build generate the same obj/lib/dll/efi/fv/fd files before and after the uncrustify changes.  We would run and publish the results from this tool before committing the changes.

We need TianoCore community approval of the following:


  1.  Approve format of C source generated by the uncrustify.
     [Liming] This format is for C source in Edk2 repo. Dose it also include BaseTools C source files? Submodule git repo will not cover. Rigth?

  1.  Approve uncrustify changes right after edk2-stable-202111 release.
     *   Extend code freeze until these changes are committed.
      [Liming] Which stable tag code freeze will be extended? Do you mean next stable tag 202202?

  1.  Require use of uncrustify tool before submitting patch review emails or PRs.
     *   The required version would be a formally released version  from the fork maintained by Michael Kubacki until the changes can be upstreamed.
     *   https://dev.azure.com/projectmu/Uncrustify
  2.  Add EDK II CI check to verify that all PRs submitted exactly match uncrustified version.  Reject PRs that do not match exactly.
  3.  Implement a git hook available that would automatically run uncristufy before committing changes to a local branch of an edk2 repo.
[Liming] Is there the test version git hook for the developer?

Thanks
Liming

Thanks,

Mike

From: Andrew Fish <afish@apple.com<mailto:afish@apple.com>>
Sent: Thursday, October 7, 2021 2:09 PM
To: Marvin Häuser <mhaeuser@posteo.de<mailto:mhaeuser@posteo.de>>
Cc: edk2-devel-groups-io <devel@edk2.groups.io<mailto:devel@edk2.groups.io>>; Kinney, Michael D <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>>; Leif Lindholm <leif@nuviainc.com<mailto:leif@nuviainc.com>>; mikuback@linux.microsoft.com<mailto:mikuback@linux.microsoft.com>; rebecca@nuviainc.com<mailto:rebecca@nuviainc.com>; Michael Kubacki <Michael.Kubacki@microsoft.com<mailto:Michael.Kubacki@microsoft.com>>; Bret Barkelew <Bret.Barkelew@microsoft.com<mailto:Bret.Barkelew@microsoft.com>>
Subject: Re: [edk2-devel] Progress on getting Uncrustify working for EDK2?



On Oct 7, 2021, at 1:43 PM, Marvin Häuser <mhaeuser@posteo.de<mailto:mhaeuser@posteo.de>> wrote:

Hey Mike,
Hey Andrew,

I'll just reply to both mails at once :)

On 07/10/2021 19:36, Andrew Fish wrote:


On Oct 7, 2021, at 1:19 PM, Michael D Kinney <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>> wrote:

Hi Marvin,

Some comments below.

Mike

-----Original Message-----
From:devel@edk2.groups.io<mailto:devel@edk2.groups.io><devel@edk2.groups.io<mailto:devel@edk2.groups.io>> On Behalf Of Marvin Häuser
Sent: Thursday, October 7, 2021 11:31 AM
To: Leif Lindholm <leif@nuviainc.com<mailto:leif@nuviainc.com>>;devel@edk2.groups.io<mailto:devel@edk2.groups.io>;mikuback@linux.microsoft.com<mailto:mikuback@linux.microsoft.com>
Cc:rebecca@nuviainc.com<mailto:rebecca@nuviainc.com>; Michael Kubacki <Michael.Kubacki@microsoft.com<mailto:Michael.Kubacki@microsoft.com>>; Bret Barkelew <Bret.Barkelew@microsoft.com<mailto:Bret.Barkelew@microsoft.com>>;
Kinney, Michael D <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>>
Subject: Re: [edk2-devel] Progress on getting Uncrustify working for EDK2?

Good day,

+1, but while you're at it, can we have arguments not align to the
function name...

  Status = Test (
             a
             );

... but to the next natural indentation level?

  Status = Test (
    a
    );

Basically no IDE I have seen supports EDK II's style, and I wouldn't be
keen on writing known-broken style to then rely on Uncrustify to fix it.

I also have heard some controversy regarding casts off-list, where some
prefer no spaces after casts to stress the evaluation order, and some
prefer spaces to have clearer visuals (as a cast *ideally* would be
something rare that requires good justification). Just throwing that out
there.


For things unrelated to autoformat (so semi-offtopic) but still relevant
to the coding spec:

1. Allow STATIC functions (if the debugging concerns are still relevant,
there could be another level of indirection, like RELEASE_STATIC)?

Debugging concerns are no longer relevant.  The suggestion in the BZ below
is to remove the STATIC macro and allow EDK II sources to add 'static'
to any functions it makes sense to use on.

https://bugzilla.tianocore.org/show_bug.cgi?id=1766

Thanks! I'd keep STATIC actually just for the sake of not doing no-op changes that do not really do anything and for consistency with CONST, but whatever works really.



2. Allow variable assignments on definition (basically non-static CONST
variables are banned...)?

Are referring to use of pre-initialized CONST variables declared within
a function?  I think Bret brought this topic up when implementing some
unit tests and the suggestion to pass ECCC was to promote them to
pre-initialized CONST global variables.

Yes.


The challenges we have seen in the past with pre-initialized variables within
a function is that they can cause compilers to inject use of memcpy() calls,
especially if the variable being initialized on the stack is a structure.
These cause build breaks today.

This issue is independent of CONST. I’m not sure a coding style tool is smart enough to catch this generically? You need an understanding of C types to know if the local variable assignment is going to trigger a memcpy().

What I’ve seen in the real world is the firmware compiles with -Os or LTO to fit int he ROM for DEBUG and RELEASE, and the optimizer optimizes away the call to memcpy. Then if you try to build NOOPT (or over ride the compiler flags on an individual driver/lib) you fail to link as only the NOOPT build injects the memcpy.

+1


Thus I think the best way to enforce this rule is to compile a project NOOPT. I’m trying to remember are there flags to built to tell it to compile and skip the FD construction? Maybe we should advocate platforms add a NOOPT build target that just compiles the code, but does not create the FD?

I know there were stability concerns with intrinsics in the past, but memcpy() is in the standard, and the rest remained stable to my knowledge. Maybe it's time to fix the issues at the root? Works for us:
https://github.com/acidanthera/OpenCorePkg/tree/master/Library/OcCompilerIntrinsicsLib

Marvin,

Good point. This would make the rule moot. So maybe just removing the requirement would be the easiest long term fix.

Other embedded projects I know of do this too, and as you point out the compilers keep these APIs standard for folks the provide their own runtimes.

Thanks,

Andrew Fish

Best regards,
Marvin


Thanks,

Andrew Fish



3. Allow variable declarations at any scope (I had some nasty shadowing
bugs before, probably prohibit shadowing with warnings)?

By shadowing do you mean the declaration of the same variable name in
multiple scoped within the same function?


4. Require that exactly all function declarations and all function
definitions with no prior declaration must be documented (first
direction is enforcing docs, second is prohibiting doc duplication, I've
seen them go out-of-sync plenty of times)?

I agree that this can reduce duplication and sync issues.  The uncrustify
tool being discussed here could not help clean this up or enforce this
type of rule.  It is a good topic, but may need to be split out into its
own thread.


The latter bunch would not require any autoformat rules or reformatation
of existing code, but would be target only new submissions in my
opinion. Thoughts?


Thanks for your efforts!

Best regards,
Marvin


Am 07.10.2021 um 12:48 schrieb Leif Lindholm:
Hi Michael,

Apologies, I've owed you a response (promised off-list) for a while
now.

First, let me say I hugely appreciate this effort. Apart from aligning
the codebase(s), this will reduce manual reviewing effort
substantially, as well as cutting down on number of rework cycles for
developers.

Looking at the changes to (well, the comments in) uncrustify, this
seems to be constrained to:
- Newline after '(' for multi-line function calls.
- Dealing with "(("/"))" for DEBUG macros.
- Function pointer typedefs:
  - typedef\nEFIAPI
  - closing parentheses indentation

I don't think I've made any secret over the years that I am not a
massive fan of the EDK2 coding style in general. So I think for any
of its quirks that are substantial enough that they require not just
custom configuration but actual new function added to existing code
conformance tools, this would be an excellent point to sanitise the
coding style instead.

Taking these in order:

Newline after '('
-----------------
I think we already reached a level of flexibility around this, where
we don't actually enforce this (or single argument per
line). Personally, I'd be happy to update the coding style as
required instead.

DEBUG macro parentheses
-----------------------
How does uncrustify treat DEBUG macros without this modification?
Do we start getting everything turned into multi-level indented
multi-line statements without this change?

Function pointer typedefs:
--------------------------
I don't see that function pointer typedefs need to rigidly follow the
same pattern as the declaration of functions implementing them. Could
we update the coding style (if needed) instead?

Best Regards,

Leif

On Mon, Aug 16, 2021 at 16:00:38 -0400, Michael Kubacki wrote:
The edk2 branch was created here:
https://github.com/makubacki/edk2/tree/uncrustify_poc_2

We have a Project Mu fork with custom changes to the Uncrustify tool to help
comply with EDK II formatting here:
https://dev.azure.com/projectmu/_git/Uncrustify

The latest information about the status and how to experiment with the
configuration file and the tool are in that fork:
https://dev.azure.com/projectmu/Uncrustify/_wiki/wikis/Uncrustify.wiki/1/Project-Mu-(EDK-II)-Fork-Readme

That said, I have also finished a CI plugin to run Uncrustify that should be
ready soon to initially deploy in Project Mu. Before doing so, I am trying
to settle on an initial configuration file that less strictly but more
reliably formats the code than in the examples in those branches. For
example, remove heuristics that when run against the same set of code
multiple times can produce different results. An example would be a rule
that reformats code because it exceeds a specified column width on one run
but on the next run that reformatted code triggers a different rule to
further align the code and so on. At least initially, some rules might be
tweaked in a more conservative approach that can be tightened in the future.
Once this configuration file is ready, we will baseline Project Mu code as
an example and turn on the plugin. The CI plugin runs Uncrustify against
modified files and if there's any changes, indicating a formatting
deviation, the diff chunks are saved in a log so they can be viewed as a
build artifact.

I am making progress on the updated config file and I should be able to post
a "uncrustify_poc_3" branch soon with the results.

Regarding indentation, Marvin is right that Uncrustify cannot support edk2
indentation style out-of-box. Some changes are made in that fork to handle
the formatting. At this point, it can handle the indentation in the cases
I've seen. Uncrustify does potentially give us the ability to massively
deploy changes across the codebase in case a decision were made to change
the style.

Thanks,
Michael

On 8/16/2021 3:39 PM, Marvin Häuser wrote:
Hey Rebecca,

I think even Uncrustify has issues with the EDK II indentation style.
You might want to check the UEFI Talkbox Discord server, I had a brief
chat with Michael about it there. I don't think realistically any tool
supports EDK II's indentation style however, so I'd propose it is
changed. This could be for new submissions only, or actually the entire
codebase could be reformatted at once with a good tool setup. While this
screws with git blame, the (to my understanding) decided on CRLF -> LF
change does that anyway, so at least two evils could be dealt with in
one go really.

Best regards,
Marvin

On 16/08/2021 21:34, Rebecca Cran wrote:

cc devel@ .

On 8/16/21 1:33 PM, Rebecca Cran wrote:

I noticed a message on Twitter about an idea of using Uncrustify
for EDK2 instead of the ECC tool, and came across https://www.mail-
archive.com/search?l=devel@edk2.groups.io&q=subject:%22Re%5C%3A+%5C%5Bedk2%5C-<mailto:archive.com/search?l=devel@edk2.groups.io&q=subject:%22Re%5C%3A+%5C%5Bedk2%5C->
devel%5C%5D+TianoCore+Community+Meeting+Minutes+%5C-+2%5C%2F4%22&o=newest&f=1
.

I was wondering if there's been any progress on it that I could
check out?


Michael Kubacki: in that message, you said:

"I'm planning to put up a branch that we can use as a reference
for a conversation around uncrustify in the next couple of
weeks."


Did you end up creating that branch, and if so could you provide
a link to it please?


--
Rebecca Cran














[-- Attachment #2: Type: text/html, Size: 83405 bytes --]

Re: [edk2-devel] Progress on getting Uncrustify working for EDK2?

[Michael D Kinney]

Hi Marvin,

Comments below.

Mike

> -----Original Message-----
> From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Marvin Häuser
> Sent: Monday, November 8, 2021 11:33 PM
> To: Kinney, Michael D <michael.d.kinney@intel.com>
> Cc: Andrew Fish <afish@apple.com>; devel@edk2.groups.io; Michael Kubacki <Michael.Kubacki@microsoft.com>; Leif Lindholm
> <leif@nuviainc.com>; mikuback@linux.microsoft.com; rebecca@nuviainc.com; Bret Barkelew <Bret.Barkelew@microsoft.com>
> Subject: Re: [edk2-devel] Progress on getting Uncrustify working for EDK2?
> 
> Hey all,
> 
> Thanks for the effort!
> 
> 1. If virtually everyone will need Uncrustify, why cannot it be built along with BaseTools from a submodule? Especially
> with the fork that makes sense, after that it depends on the upstream (it does not look too nice to me).

No matter where uncrustify sources are hosted, developers can always choose to build the uncrustify tool locally.
Providing release binaries for the tool may be simpler for some customers.
Using release binaries from EDK II CI agents will reduce CI execution time.

The goal is to upstream all changes to uncrustify project.  However, we need to get the EDK II community to review
and accept the style of the source code produced by this fork of uncrustify.  We do not want to do this type of
change more than once, so getting everyone to agree to a specific format is critical.

Once the content is upstreamed, then the EDK II communities maintenance of a fork can end and the EDK II 
community can work with the uncrustify community for any future changes and developers and CI agents can
use the release binaries and/or sources from the uncrustify project.

> 
> 2. Does this cover CRLF->LF?

Are you referring to updating the GitHub repo line endings on the GitHub server side?  That is a different
topic and will require a different tool and developer process changes.

> 
> 3. Will there be a list of implicit changes from the current code style?


I believe the uncrustify tool being discussed here is conformant with the current EDK II C Coding Style Specification.
I am not aware of any changes that are required to that spec.

> 
> 4. I feel like if the other code style changes are not tackled now, they will never be, because who wants to deal with
> continuous cosmetic commits. But as long as there is a tool now to enforce the current one, that's not a dealbreaker at
> all. :)

The additional discussion topics I have seen that are not addressed by source format tools:

1) Allowing local variables to be declared in scope of if/while/case.  
   * Current style only allows locals to be declared at top of function.
2) Allow local variables to be assigned to a value in their declaration.
3) Allow local variables that are struct/array to be assigned a value in their declaration
   * Depends on support for memcpy()/memset() intrinsics for all supported tool chains.
4) Use of STATIC vs static.

Please let me know if I missed any of the additional topics.

> 
> Best regards,
> Marvin
> 
> 09.11.2021 04:03:06 Kinney, Michael D <michael.d.kinney@intel.com>:
> 
> > Andrew,
> >
> > I think Michael Kubacki started with a nuget feed because that can be easily used by EDK II CI agents.
> >
> > However, that does not work as easily for all development environments using Windows, Linux, and MacOS.  Creating
> releases that can be easily installed by developers is critical for success.
> >
> > For MacOS, there is a homebrew recipe.  You should just have to update the URL to the uncrustify fork.
> >
> > https://macappstore.org/uncrustify/
> >
> > Adding the installation information to the EDK II Getting Started page would be a good place to capture the developer
> install.
> >
> > Mike
> >
> > *From:* Andrew Fish <afish@apple.com>
> > *Sent:* Monday, November 8, 2021 6:47 PM
> > *To:* Kinney, Michael D <michael.d.kinney@intel.com>
> > *Cc:* devel@edk2.groups.io; Marvin Häuser <mhaeuser@posteo.de>; Michael Kubacki <Michael.Kubacki@microsoft.com>; Leif
> Lindholm <leif@nuviainc.com>; mikuback@linux.microsoft.com; rebecca@nuviainc.com; Bret Barkelew
> <Bret.Barkelew@microsoft.com>
> > *Subject:* Re: [edk2-devel] Progress on getting Uncrustify working for EDK2?
> >
> >
> >
> >
> > On Nov 8, 2021, at 5:13 PM, Kinney, Michael D <michael.d.kinney@intel.com> wrote:
> >
> > HI Andrew,
> >
> > Great feedback.
> >
> > What your preferred way to install a tool like this?  If we collect that data from the community, we can make sure the
> pipeline generates the right type of installers.
> >
> >
> > I could not figure out how to download an installer from the links.
> >
> > If I go to http://uncrustify.sourceforge.net I see https://sourceforge.net/projects/uncrustify/files/ and a button to
> download binaries. Not ideal that it is only for Windows but at least the workflow was obvious. Looks like I need to build
> my own version for macOS, not ideal but I can at least figure that out.
> >
> > Worst case we can have a Tianocore.org[http://Tianocore.org] page to describe a simple recipe to install the tool. With
> step by step instructions some one can just type in without thinking too much.
> >
> > Thanks,
> >
> > Andrew Fish
> >
> >
> > Thanks,
> >
> > Mike
> >
> > *From:* Andrew Fish <afish@apple.com>
> > *Sent:* Monday, November 8, 2021 5:09 PM
> > *To:* devel@edk2.groups.io; Kinney, Michael D <michael.d.kinney@intel.com>
> > *Cc:* Marvin Häuser <mhaeuser@posteo.de>; Michael Kubacki <Michael.Kubacki@microsoft.com>; Leif Lindholm
> <leif@nuviainc.com>; mikuback@linux.microsoft.com; rebecca@nuviainc.com; Bret Barkelew <Bret.Barkelew@microsoft.com>
> > *Subject:* Re: [edk2-devel] Progress on getting Uncrustify working for EDK2?
> >
> > MIke,
> >
> > I could not figure out how to download uncrustify tool from the provided link. 99% of the people are just going to want
> to install the tool, not be a developer of the fork. We should have some simple instructions on how to download the tool.
> >
> > The link points to something git web view looking Azure DevOps page and talks about this Nuget thing I know nothing
> about. I ran out of time and had to give up trying to download the tool.
> >
> > Thanks,
> >
> > Andrew Fish
> >
> >
> >
> > On Nov 8, 2021, at 4:23 PM, Michael D Kinney <michael.d.kinney@intel.com> wrote:
> >
> > Hello,
> >
> > Good information in this thread on code style.
> >
> > Some of the topics apply to uncrustify and some are out of scope for what uncrustify can fix on its own.
> >
> > I would like to focus on a date to convert all source code in edk2 repo using the uncrustify tool and to capture the
> other code style topics into their own thread andbugzillas.
> >
> > I would like to propose a conversion date for uncrustify immediately after the edk2-stable202111 release on 2021-11-26.
> >
> > I have been working with Michael Kubacki on a build comparison tool that verifies that the build generate the same
> obj/lib/dll/efi/fv/fd files before and after the uncrustify changes.  We would run and publish the results from this tool
> before committing the changes.
> >
> > We need TianoCore community approval of the following:
> >
> 1. > Approve format of C source generated by the uncrustify.
> 2. > Approve uncrustify changes right after edk2-stable-202111 release.
>   1. > Extend code freeze until these changes are committed.
> 3. > Require use of uncrustify tool before submitting patch review emails or PRs.
>   1. > The required version would be a formally released version  from the fork maintained by Michael Kubacki until the
> changes can be upstreamed.
>   2. > https://dev.azure.com/projectmu/Uncrustify
> 4. > Add EDK II CI check to verify that all PRs submitted exactly match uncrustified version.  Reject PRs that do not
> match exactly.
> 5. > Implement a git hook available that would automatically run uncristufy before committing changes to a local branch of
> an edk2 repo.
> >
> > Thanks,
> >
> > Mike
> >
> > *From:* Andrew Fish <afish@apple.com>
> > *Sent:* Thursday, October 7, 2021 2:09 PM
> > *To:* Marvin Häuser <mhaeuser@posteo.de>
> > *Cc:* edk2-devel-groups-io <devel@edk2.groups.io>; Kinney, Michael D <michael.d.kinney@intel.com>; Leif Lindholm
> <leif@nuviainc.com>; mikuback@linux.microsoft.com; rebecca@nuviainc.com; Michael Kubacki <Michael.Kubacki@microsoft.com>;
> Bret Barkelew <Bret.Barkelew@microsoft.com>
> > *Subject:* Re: [edk2-devel] Progress on getting Uncrustify working for EDK2?
> >
> >
> >
> >
> >
> >
> > On Oct 7, 2021, at 1:43 PM, Marvin Häuser <mhaeuser@posteo.de> wrote:
> >
> > Hey Mike,
> > Hey Andrew,
> >
> > I'll just reply to both mails at once :)
> >
> > On 07/10/2021 19:36, Andrew Fish wrote:
> >
> >
> >
> >
> >
> >
> >
> >
> > …
> >
> > Thanks! I'd keep STATIC actually just for the sake of not doing no-op changes that do not really do anything and for
> consistency with CONST, but whatever works really.
> >
> >
> >
> >
> > …
> >
> > Yes.
> >
> >
> >
> >
> > …
> >
> > This issue is independent of CONST. I’m not sure a coding style tool is smart enough to catch this generically? You need
> an understanding of C types to know if the local variable assignment is going to trigger a memcpy().
> >
> > What I’ve seen in the real world is the firmware compiles with -Os or LTO to fit int he ROM for DEBUG and RELEASE, and
> the optimizer optimizes away the call to memcpy. Then if you try to build NOOPT (or over ride the compiler flags on an
> individual driver/lib) you fail to link as only the NOOPT build injects the memcpy.
> >
> > +1
> >
> >
> >
> >
> >
> > Thus I think the best way to enforce this rule is to compile a project NOOPT. I’m trying to remember are there flags to
> built to tell it to compile and skip the FD construction? Maybe we should advocate platforms add a NOOPT build target that
> just compiles the code, but does not create the FD?
> >
> > I know there were stability concerns with intrinsics in the past, but memcpy() is in the standard, and the rest remained
> stable to my knowledge. Maybe it's time to fix the issues at the root? Works for us:
> > https://github.com/acidanthera/OpenCorePkg/tree/master/Library/OcCompilerIntrinsicsLib
> >
> >
> >
> > Marvin,
> >
> > Good point. This would make the rule moot. So maybe just removing the requirement would be the easiest long term fix.
> >
> > Other embedded projects I know of do this too, and as you point out the compilers keep these APIs standard for folks the
> provide their own runtimes.
> >
> > Thanks,
> >
> > Andrew Fish
> >
> >
> >
> >
> > Best regards,
> > Marvin
> >
> >
> >
> >
> >
> > Thanks,
> >
> > Andrew Fish
> >
> >
> >
> >
> > …
> >
> >
> >
> 
> 
> 
> 

Re: [edk2-devel] Progress on getting Uncrustify working for EDK2?

[Michael D Kinney]

> -----Original Message-----
> From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Gerd Hoffmann
> Sent: Tuesday, November 9, 2021 6:10 AM
> To: Leif Lindholm <leif@nuviainc.com>
> Cc: devel@edk2.groups.io; Kinney, Michael D <michael.d.kinney@intel.com>; Andrew Fish <afish@apple.com>; Marvin Häuser
> <mhaeuser@posteo.de>; Michael Kubacki <Michael.Kubacki@microsoft.com>; mikuback@linux.microsoft.com; rebecca@nuviainc.com;
> Bret Barkelew <Bret.Barkelew@microsoft.com>
> Subject: Re: [edk2-devel] Progress on getting Uncrustify working for EDK2?
> 
> On Tue, Nov 09, 2021 at 12:36:30PM +0000, Leif Lindholm wrote:
> > On Tue, Nov 09, 2021 at 09:40:02 +0100, Gerd Hoffmann wrote:
> > >   Hi,
> > >
> > > >   3.  Require use of uncrustify tool before submitting patch review emails or PRs.
> > > >      *   The required version would be a formally released version  from the fork maintained by Michael Kubacki
> until the changes can be upstreamed.
> > > >      *   https://dev.azure.com/projectmu/Uncrustify
> > >
> > > Can we please *first* get the changes merged to upstream uncrustify?
> > >
> > > That'll make the whole process much less painful because the usual
> > > software repositories (linux distro packages, macos homebrew, ...)
> > > can be used to install uncrustify then, and it's also less confusing if
> > > developers don't have to juggle with different uncrustify variants
> > > (upstream vs. edk2).
> >
> > Whilst I agree in principle...
> >
> > This means postponing automated coding style changes until 2023
> > (Debian stable), 2025 (Ubuntu LTS), ??? (RHEL10), or even later
> > ... and I'd rather not.
> >
> > I like Marvin's suggestion of a submodule. Which we could drop once
> > no longer needed.
> 
> I'm fine with a submodule (temporary & optional) so we don't have to
> wait until all distros (especially the LTS ones which are slower) have
> picked up the latest uncrustify release.
> 
> But I still think upstreaming the uncrustify changes should be the first
> step.

The first step is to review and accept the style of code generated by 
the current fork.

> 
> take care,
>   Gerd
> 
> 
> 
> 
> 

Re: [edk2-devel] Progress on getting Uncrustify working for EDK2?

[Marvin Häuser]

Hey Mike,

On 09.11.21 16:43, Kinney, Michael D wrote:
> Hi Marvin,
>
> Comments below.
>
> Mike
>
>> -----Original Message-----
>> From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Marvin Häuser
>> Sent: Monday, November 8, 2021 11:33 PM
>> To: Kinney, Michael D <michael.d.kinney@intel.com>
>> Cc: Andrew Fish <afish@apple.com>; devel@edk2.groups.io; Michael Kubacki <Michael.Kubacki@microsoft.com>; Leif Lindholm
>> <leif@nuviainc.com>; mikuback@linux.microsoft.com; rebecca@nuviainc.com; Bret Barkelew <Bret.Barkelew@microsoft.com>
>> Subject: Re: [edk2-devel] Progress on getting Uncrustify working for EDK2?
>>
>> Hey all,
>>
>> Thanks for the effort!
>>
>> 1. If virtually everyone will need Uncrustify, why cannot it be built along with BaseTools from a submodule? Especially
>> with the fork that makes sense, after that it depends on the upstream (it does not look too nice to me).
> No matter where uncrustify sources are hosted, developers can always choose to build the uncrustify tool locally.
> Providing release binaries for the tool may be simpler for some customers.
> Using release binaries from EDK II CI agents will reduce CI execution time.

My point was it'd be nice if it (optionally) "just worked", so 
Uncrustify could be be built as part of the edk2 BaseTools build 
process, or release binaries could be downloaded by some script, or 
whatever really. I guess it could be the same logic as for the CI?

> The goal is to upstream all changes to uncrustify project.  However, we need to get the EDK II community to review
> and accept the style of the source code produced by this fork of uncrustify.  We do not want to do this type of
> change more than once, so getting everyone to agree to a specific format is critical.
>
> Once the content is upstreamed, then the EDK II communities maintenance of a fork can end and the EDK II
> community can work with the uncrustify community for any future changes and developers and CI agents can
> use the release binaries and/or sources from the uncrustify project.

Thanks!

>> 2. Does this cover CRLF->LF?
> Are you referring to updating the GitHub repo line endings on the GitHub server side?  That is a different
> topic and will require a different tool and developer process changes.

OK, thanks.

>> 3. Will there be a list of implicit changes from the current code style?
>
> I believe the uncrustify tool being discussed here is conformant with the current EDK II C Coding Style Specification.
> I am not aware of any changes that are required to that spec.

Nice, thanks!

>> 4. I feel like if the other code style changes are not tackled now, they will never be, because who wants to deal with
>> continuous cosmetic commits. But as long as there is a tool now to enforce the current one, that's not a dealbreaker at
>> all. :)
> The additional discussion topics I have seen that are not addressed by source format tools:
>
> 1) Allowing local variables to be declared in scope of if/while/case.
>     * Current style only allows locals to be declared at top of function.
> 2) Allow local variables to be assigned to a value in their declaration.
> 3) Allow local variables that are struct/array to be assigned a value in their declaration
>     * Depends on support for memcpy()/memset() intrinsics for all supported tool chains.
> 4) Use of STATIC vs static.

There were more proposals:

5) Indent relative to the last indentation, not to symbol names, i.e.

   Status = Function (
     A
     );

instead of

   Status = Function (
                    A
                    );

Uncrustify might support this now, but I have never seen even a single 
IDE that does. It'd be kind of awkward to always write "style-broken" 
code and rely on the external tool to fix it.

6) Allow static function declarations.

Best regards,
Marvin

>
> Please let me know if I missed any of the additional topics.
>
>> Best regards,
>> Marvin
>>
>> 09.11.2021 04:03:06 Kinney, Michael D <michael.d.kinney@intel.com>:
>>
>>> Andrew,
>>>
>>> I think Michael Kubacki started with a nuget feed because that can be easily used by EDK II CI agents.
>>>
>>> However, that does not work as easily for all development environments using Windows, Linux, and MacOS.  Creating
>> releases that can be easily installed by developers is critical for success.
>>> For MacOS, there is a homebrew recipe.  You should just have to update the URL to the uncrustify fork.
>>>
>>> https://macappstore.org/uncrustify/
>>>
>>> Adding the installation information to the EDK II Getting Started page would be a good place to capture the developer
>> install.
>>> Mike
>>>
>>> *From:* Andrew Fish <afish@apple.com>
>>> *Sent:* Monday, November 8, 2021 6:47 PM
>>> *To:* Kinney, Michael D <michael.d.kinney@intel.com>
>>> *Cc:* devel@edk2.groups.io; Marvin Häuser <mhaeuser@posteo.de>; Michael Kubacki <Michael.Kubacki@microsoft.com>; Leif
>> Lindholm <leif@nuviainc.com>; mikuback@linux.microsoft.com; rebecca@nuviainc.com; Bret Barkelew
>> <Bret.Barkelew@microsoft.com>
>>> *Subject:* Re: [edk2-devel] Progress on getting Uncrustify working for EDK2?
>>>
>>>
>>>
>>>
>>> On Nov 8, 2021, at 5:13 PM, Kinney, Michael D <michael.d.kinney@intel.com> wrote:
>>>
>>> HI Andrew,
>>>
>>> Great feedback.
>>>
>>> What your preferred way to install a tool like this?  If we collect that data from the community, we can make sure the
>> pipeline generates the right type of installers.
>>>
>>> I could not figure out how to download an installer from the links.
>>>
>>> If I go to http://uncrustify.sourceforge.net I see https://sourceforge.net/projects/uncrustify/files/ and a button to
>> download binaries. Not ideal that it is only for Windows but at least the workflow was obvious. Looks like I need to build
>> my own version for macOS, not ideal but I can at least figure that out.
>>> Worst case we can have a Tianocore.org[http://Tianocore.org] page to describe a simple recipe to install the tool. With
>> step by step instructions some one can just type in without thinking too much.
>>> Thanks,
>>>
>>> Andrew Fish
>>>
>>>
>>> Thanks,
>>>
>>> Mike
>>>
>>> *From:* Andrew Fish <afish@apple.com>
>>> *Sent:* Monday, November 8, 2021 5:09 PM
>>> *To:* devel@edk2.groups.io; Kinney, Michael D <michael.d.kinney@intel.com>
>>> *Cc:* Marvin Häuser <mhaeuser@posteo.de>; Michael Kubacki <Michael.Kubacki@microsoft.com>; Leif Lindholm
>> <leif@nuviainc.com>; mikuback@linux.microsoft.com; rebecca@nuviainc.com; Bret Barkelew <Bret.Barkelew@microsoft.com>
>>> *Subject:* Re: [edk2-devel] Progress on getting Uncrustify working for EDK2?
>>>
>>> MIke,
>>>
>>> I could not figure out how to download uncrustify tool from the provided link. 99% of the people are just going to want
>> to install the tool, not be a developer of the fork. We should have some simple instructions on how to download the tool.
>>> The link points to something git web view looking Azure DevOps page and talks about this Nuget thing I know nothing
>> about. I ran out of time and had to give up trying to download the tool.
>>> Thanks,
>>>
>>> Andrew Fish
>>>
>>>
>>>
>>> On Nov 8, 2021, at 4:23 PM, Michael D Kinney <michael.d.kinney@intel.com> wrote:
>>>
>>> Hello,
>>>
>>> Good information in this thread on code style.
>>>
>>> Some of the topics apply to uncrustify and some are out of scope for what uncrustify can fix on its own.
>>>
>>> I would like to focus on a date to convert all source code in edk2 repo using the uncrustify tool and to capture the
>> other code style topics into their own thread andbugzillas.
>>> I would like to propose a conversion date for uncrustify immediately after the edk2-stable202111 release on 2021-11-26.
>>>
>>> I have been working with Michael Kubacki on a build comparison tool that verifies that the build generate the same
>> obj/lib/dll/efi/fv/fd files before and after the uncrustify changes.  We would run and publish the results from this tool
>> before committing the changes.
>>> We need TianoCore community approval of the following:
>>>
>> 1. > Approve format of C source generated by the uncrustify.
>> 2. > Approve uncrustify changes right after edk2-stable-202111 release.
>>    1. > Extend code freeze until these changes are committed.
>> 3. > Require use of uncrustify tool before submitting patch review emails or PRs.
>>    1. > The required version would be a formally released version  from the fork maintained by Michael Kubacki until the
>> changes can be upstreamed.
>>    2. > https://dev.azure.com/projectmu/Uncrustify
>> 4. > Add EDK II CI check to verify that all PRs submitted exactly match uncrustified version.  Reject PRs that do not
>> match exactly.
>> 5. > Implement a git hook available that would automatically run uncristufy before committing changes to a local branch of
>> an edk2 repo.
>>> Thanks,
>>>
>>> Mike
>>>
>>> *From:* Andrew Fish <afish@apple.com>
>>> *Sent:* Thursday, October 7, 2021 2:09 PM
>>> *To:* Marvin Häuser <mhaeuser@posteo.de>
>>> *Cc:* edk2-devel-groups-io <devel@edk2.groups.io>; Kinney, Michael D <michael.d.kinney@intel.com>; Leif Lindholm
>> <leif@nuviainc.com>; mikuback@linux.microsoft.com; rebecca@nuviainc.com; Michael Kubacki <Michael.Kubacki@microsoft.com>;
>> Bret Barkelew <Bret.Barkelew@microsoft.com>
>>> *Subject:* Re: [edk2-devel] Progress on getting Uncrustify working for EDK2?
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Oct 7, 2021, at 1:43 PM, Marvin Häuser <mhaeuser@posteo.de> wrote:
>>>
>>> Hey Mike,
>>> Hey Andrew,
>>>
>>> I'll just reply to both mails at once :)
>>>
>>> On 07/10/2021 19:36, Andrew Fish wrote:
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> …
>>>
>>> Thanks! I'd keep STATIC actually just for the sake of not doing no-op changes that do not really do anything and for
>> consistency with CONST, but whatever works really.
>>>
>>>
>>>
>>> …
>>>
>>> Yes.
>>>
>>>
>>>
>>>
>>> …
>>>
>>> This issue is independent of CONST. I’m not sure a coding style tool is smart enough to catch this generically? You need
>> an understanding of C types to know if the local variable assignment is going to trigger a memcpy().
>>> What I’ve seen in the real world is the firmware compiles with -Os or LTO to fit int he ROM for DEBUG and RELEASE, and
>> the optimizer optimizes away the call to memcpy. Then if you try to build NOOPT (or over ride the compiler flags on an
>> individual driver/lib) you fail to link as only the NOOPT build injects the memcpy.
>>> +1
>>>
>>>
>>>
>>>
>>>
>>> Thus I think the best way to enforce this rule is to compile a project NOOPT. I’m trying to remember are there flags to
>> built to tell it to compile and skip the FD construction? Maybe we should advocate platforms add a NOOPT build target that
>> just compiles the code, but does not create the FD?
>>> I know there were stability concerns with intrinsics in the past, but memcpy() is in the standard, and the rest remained
>> stable to my knowledge. Maybe it's time to fix the issues at the root? Works for us:
>>> https://github.com/acidanthera/OpenCorePkg/tree/master/Library/OcCompilerIntrinsicsLib
>>>
>>>
>>>
>>> Marvin,
>>>
>>> Good point. This would make the rule moot. So maybe just removing the requirement would be the easiest long term fix.
>>>
>>> Other embedded projects I know of do this too, and as you point out the compilers keep these APIs standard for folks the
>> provide their own runtimes.
>>> Thanks,
>>>
>>> Andrew Fish
>>>
>>>
>>>
>>>
>>> Best regards,
>>> Marvin
>>>
>>>
>>>
>>>
>>>
>>> Thanks,
>>>
>>> Andrew Fish
>>>
>>>
>>>
>>>
>>> …
>>>
>>>
>>>
>>
>> 
>>

Re: [edk2-devel] Progress on getting Uncrustify working for EDK2?

[Michael D Kinney]

Hi Marvin,

> -----Original Message-----
> From: Marvin Häuser <mhaeuser@posteo.de>
> Sent: Tuesday, November 9, 2021 8:13 AM
> To: Kinney, Michael D <michael.d.kinney@intel.com>; devel@edk2.groups.io
> Cc: Andrew Fish <afish@apple.com>; Michael Kubacki <Michael.Kubacki@microsoft.com>; Leif Lindholm <leif@nuviainc.com>;
> mikuback@linux.microsoft.com; rebecca@nuviainc.com; Bret Barkelew <Bret.Barkelew@microsoft.com>
> Subject: Re: [edk2-devel] Progress on getting Uncrustify working for EDK2?
> 
> Hey Mike,
> 
> On 09.11.21 16:43, Kinney, Michael D wrote:
> > Hi Marvin,
> >
> > Comments below.
> >
> > Mike
> >
> >> -----Original Message-----
> >> From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Marvin Häuser
> >> Sent: Monday, November 8, 2021 11:33 PM
> >> To: Kinney, Michael D <michael.d.kinney@intel.com>
> >> Cc: Andrew Fish <afish@apple.com>; devel@edk2.groups.io; Michael Kubacki <Michael.Kubacki@microsoft.com>; Leif Lindholm
> >> <leif@nuviainc.com>; mikuback@linux.microsoft.com; rebecca@nuviainc.com; Bret Barkelew <Bret.Barkelew@microsoft.com>
> >> Subject: Re: [edk2-devel] Progress on getting Uncrustify working for EDK2?
> >>
> >> Hey all,
> >>
> >> Thanks for the effort!
> >>
> >> 1. If virtually everyone will need Uncrustify, why cannot it be built along with BaseTools from a submodule? Especially
> >> with the fork that makes sense, after that it depends on the upstream (it does not look too nice to me).
> > No matter where uncrustify sources are hosted, developers can always choose to build the uncrustify tool locally.
> > Providing release binaries for the tool may be simpler for some customers.
> > Using release binaries from EDK II CI agents will reduce CI execution time.
> 
> My point was it'd be nice if it (optionally) "just worked", so
> Uncrustify could be be built as part of the edk2 BaseTools build
> process, or release binaries could be downloaded by some script, or
> whatever really. I guess it could be the same logic as for the CI?

Setting up a development work environment also requires the installation of tools such as 
compilers, NASM, IASL, etc.  These are not handled as part of BaseTools today.

I agree we want to make setting of the EDK II development environment as easy as possible.

> 
> > The goal is to upstream all changes to uncrustify project.  However, we need to get the EDK II community to review
> > and accept the style of the source code produced by this fork of uncrustify.  We do not want to do this type of
> > change more than once, so getting everyone to agree to a specific format is critical.
> >
> > Once the content is upstreamed, then the EDK II communities maintenance of a fork can end and the EDK II
> > community can work with the uncrustify community for any future changes and developers and CI agents can
> > use the release binaries and/or sources from the uncrustify project.
> 
> Thanks!
> 
> >> 2. Does this cover CRLF->LF?
> > Are you referring to updating the GitHub repo line endings on the GitHub server side?  That is a different
> > topic and will require a different tool and developer process changes.
> 
> OK, thanks.
> 
> >> 3. Will there be a list of implicit changes from the current code style?
> >
> > I believe the uncrustify tool being discussed here is conformant with the current EDK II C Coding Style Specification.
> > I am not aware of any changes that are required to that spec.
> 
> Nice, thanks!
> 
> >> 4. I feel like if the other code style changes are not tackled now, they will never be, because who wants to deal with
> >> continuous cosmetic commits. But as long as there is a tool now to enforce the current one, that's not a dealbreaker at
> >> all. :)
> > The additional discussion topics I have seen that are not addressed by source format tools:
> >
> > 1) Allowing local variables to be declared in scope of if/while/case.
> >     * Current style only allows locals to be declared at top of function.
> > 2) Allow local variables to be assigned to a value in their declaration.
> > 3) Allow local variables that are struct/array to be assigned a value in their declaration
> >     * Depends on support for memcpy()/memset() intrinsics for all supported tool chains.
> > 4) Use of STATIC vs static.
> 
> There were more proposals:
> 
> 5) Indent relative to the last indentation, not to symbol names, i.e.
> 
>    Status = Function (
>      A
>      );
> 
> instead of
> 
>    Status = Function (
>                     A
>                     );> 
> Uncrustify might support this now, but I have never seen even a single
> IDE that does. It'd be kind of awkward to always write "style-broken"
> code and rely on the external tool to fix it.


The EDK II C Coding Style Spec and the fork of uncrustify are aligned
to support indent of 2 spaces from the start column of the function name.
This would be in the following format from your example:

    Status = Function (
               A
               );

Since many of the APIs in EDK II are called through a services table, PPI,
or Protocol, the indent is 2 spaces from the start of the member function.
Examples:

  Status = gBS->OpenProtocol (
                  ControllerHandle,
                  &gEfiBlockIoProtocolGuid,
                  (VOID **)&BlockIo,
                  This->DriverBindingHandle,
                  ControllerHandle,
                  EFI_OPEN_PROTOCOL_BY_DRIVER
                  );

        Status = BlockIo2->ReadBlocksEx (
                             BlockIo2,
                             MediaId,
                             Subtask->Lba,
                             &Subtask->BlockIo2Token,
                             (Subtask->Length % Media->BlockSize == 0) ? Subtask->Length : Media->BlockSize,
                             (Subtask->WorkingBuffer != NULL) ? Subtask->WorkingBuffer : Subtask->Buffer
                             );

Uncrustify can be installed as a plugin to Visual Studio Code, so the
style the EDK II Community agrees to can be supported by that IDE.

Supporting this indent style is one of the enhancements in the fork.  If we want to align
to one of the indent styles supported by a wider array of source editors/IDEs, then that
would require a change to the EDK II Coding Style Specification and approval from the
EDK II community.

> 
> 6) Allow static function declarations.

I agree that static functions should be allowed.  Please add any comments you have to 
the following Bugzilla.  If you have ideas on the specific spec updates required, then
please provide a patch against the spec markdown.

    https://bugzilla.tianocore.org/show_bug.cgi?id=1766

> 
> Best regards,
> Marvin
> 
> >
> > Please let me know if I missed any of the additional topics.
> >
> >> Best regards,
> >> Marvin
> >>
> >> 09.11.2021 04:03:06 Kinney, Michael D <michael.d.kinney@intel.com>:
> >>
> >>> Andrew,
> >>>
> >>> I think Michael Kubacki started with a nuget feed because that can be easily used by EDK II CI agents.
> >>>
> >>> However, that does not work as easily for all development environments using Windows, Linux, and MacOS.  Creating
> >> releases that can be easily installed by developers is critical for success.
> >>> For MacOS, there is a homebrew recipe.  You should just have to update the URL to the uncrustify fork.
> >>>
> >>> https://macappstore.org/uncrustify/
> >>>
> >>> Adding the installation information to the EDK II Getting Started page would be a good place to capture the developer
> >> install.
> >>> Mike
> >>>
> >>> *From:* Andrew Fish <afish@apple.com>
> >>> *Sent:* Monday, November 8, 2021 6:47 PM
> >>> *To:* Kinney, Michael D <michael.d.kinney@intel.com>
> >>> *Cc:* devel@edk2.groups.io; Marvin Häuser <mhaeuser@posteo.de>; Michael Kubacki <Michael.Kubacki@microsoft.com>; Leif
> >> Lindholm <leif@nuviainc.com>; mikuback@linux.microsoft.com; rebecca@nuviainc.com; Bret Barkelew
> >> <Bret.Barkelew@microsoft.com>
> >>> *Subject:* Re: [edk2-devel] Progress on getting Uncrustify working for EDK2?
> >>>
> >>>
> >>>
> >>>
> >>> On Nov 8, 2021, at 5:13 PM, Kinney, Michael D <michael.d.kinney@intel.com> wrote:
> >>>
> >>> HI Andrew,
> >>>
> >>> Great feedback.
> >>>
> >>> What your preferred way to install a tool like this?  If we collect that data from the community, we can make sure the
> >> pipeline generates the right type of installers.
> >>>
> >>> I could not figure out how to download an installer from the links.
> >>>
> >>> If I go to http://uncrustify.sourceforge.net I see https://sourceforge.net/projects/uncrustify/files/ and a button to
> >> download binaries. Not ideal that it is only for Windows but at least the workflow was obvious. Looks like I need to
> build
> >> my own version for macOS, not ideal but I can at least figure that out.
> >>> Worst case we can have a Tianocore.org[http://Tianocore.org] page to describe a simple recipe to install the tool.
> With
> >> step by step instructions some one can just type in without thinking too much.
> >>> Thanks,
> >>>
> >>> Andrew Fish
> >>>
> >>>
> >>> Thanks,
> >>>
> >>> Mike
> >>>
> >>> *From:* Andrew Fish <afish@apple.com>
> >>> *Sent:* Monday, November 8, 2021 5:09 PM
> >>> *To:* devel@edk2.groups.io; Kinney, Michael D <michael.d.kinney@intel.com>
> >>> *Cc:* Marvin Häuser <mhaeuser@posteo.de>; Michael Kubacki <Michael.Kubacki@microsoft.com>; Leif Lindholm
> >> <leif@nuviainc.com>; mikuback@linux.microsoft.com; rebecca@nuviainc.com; Bret Barkelew <Bret.Barkelew@microsoft.com>
> >>> *Subject:* Re: [edk2-devel] Progress on getting Uncrustify working for EDK2?
> >>>
> >>> MIke,
> >>>
> >>> I could not figure out how to download uncrustify tool from the provided link. 99% of the people are just going to
> want
> >> to install the tool, not be a developer of the fork. We should have some simple instructions on how to download the
> tool.
> >>> The link points to something git web view looking Azure DevOps page and talks about this Nuget thing I know nothing
> >> about. I ran out of time and had to give up trying to download the tool.
> >>> Thanks,
> >>>
> >>> Andrew Fish
> >>>
> >>>
> >>>
> >>> On Nov 8, 2021, at 4:23 PM, Michael D Kinney <michael.d.kinney@intel.com> wrote:
> >>>
> >>> Hello,
> >>>
> >>> Good information in this thread on code style.
> >>>
> >>> Some of the topics apply to uncrustify and some are out of scope for what uncrustify can fix on its own.
> >>>
> >>> I would like to focus on a date to convert all source code in edk2 repo using the uncrustify tool and to capture the
> >> other code style topics into their own thread andbugzillas.
> >>> I would like to propose a conversion date for uncrustify immediately after the edk2-stable202111 release on 2021-11-
> 26.
> >>>
> >>> I have been working with Michael Kubacki on a build comparison tool that verifies that the build generate the same
> >> obj/lib/dll/efi/fv/fd files before and after the uncrustify changes.  We would run and publish the results from this
> tool
> >> before committing the changes.
> >>> We need TianoCore community approval of the following:
> >>>
> >> 1. > Approve format of C source generated by the uncrustify.
> >> 2. > Approve uncrustify changes right after edk2-stable-202111 release.
> >>    1. > Extend code freeze until these changes are committed.
> >> 3. > Require use of uncrustify tool before submitting patch review emails or PRs.
> >>    1. > The required version would be a formally released version  from the fork maintained by Michael Kubacki until
> the
> >> changes can be upstreamed.
> >>    2. > https://dev.azure.com/projectmu/Uncrustify
> >> 4. > Add EDK II CI check to verify that all PRs submitted exactly match uncrustified version.  Reject PRs that do not
> >> match exactly.
> >> 5. > Implement a git hook available that would automatically run uncristufy before committing changes to a local branch
> of
> >> an edk2 repo.
> >>> Thanks,
> >>>
> >>> Mike
> >>>
> >>> *From:* Andrew Fish <afish@apple.com>
> >>> *Sent:* Thursday, October 7, 2021 2:09 PM
> >>> *To:* Marvin Häuser <mhaeuser@posteo.de>
> >>> *Cc:* edk2-devel-groups-io <devel@edk2.groups.io>; Kinney, Michael D <michael.d.kinney@intel.com>; Leif Lindholm
> >> <leif@nuviainc.com>; mikuback@linux.microsoft.com; rebecca@nuviainc.com; Michael Kubacki
> <Michael.Kubacki@microsoft.com>;
> >> Bret Barkelew <Bret.Barkelew@microsoft.com>
> >>> *Subject:* Re: [edk2-devel] Progress on getting Uncrustify working for EDK2?
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> On Oct 7, 2021, at 1:43 PM, Marvin Häuser <mhaeuser@posteo.de> wrote:
> >>>
> >>> Hey Mike,
> >>> Hey Andrew,
> >>>
> >>> I'll just reply to both mails at once :)
> >>>
> >>> On 07/10/2021 19:36, Andrew Fish wrote:
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> …
> >>>
> >>> Thanks! I'd keep STATIC actually just for the sake of not doing no-op changes that do not really do anything and for
> >> consistency with CONST, but whatever works really.
> >>>
> >>>
> >>>
> >>> …
> >>>
> >>> Yes.
> >>>
> >>>
> >>>
> >>>
> >>> …
> >>>
> >>> This issue is independent of CONST. I’m not sure a coding style tool is smart enough to catch this generically? You
> need
> >> an understanding of C types to know if the local variable assignment is going to trigger a memcpy().
> >>> What I’ve seen in the real world is the firmware compiles with -Os or LTO to fit int he ROM for DEBUG and RELEASE, and
> >> the optimizer optimizes away the call to memcpy. Then if you try to build NOOPT (or over ride the compiler flags on an
> >> individual driver/lib) you fail to link as only the NOOPT build injects the memcpy.
> >>> +1
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> Thus I think the best way to enforce this rule is to compile a project NOOPT. I’m trying to remember are there flags
> to
> >> built to tell it to compile and skip the FD construction? Maybe we should advocate platforms add a NOOPT build target
> that
> >> just compiles the code, but does not create the FD?
> >>> I know there were stability concerns with intrinsics in the past, but memcpy() is in the standard, and the rest
> remained
> >> stable to my knowledge. Maybe it's time to fix the issues at the root? Works for us:
> >>> https://github.com/acidanthera/OpenCorePkg/tree/master/Library/OcCompilerIntrinsicsLib
> >>>
> >>>
> >>>
> >>> Marvin,
> >>>
> >>> Good point. This would make the rule moot. So maybe just removing the requirement would be the easiest long term fix.
> >>>
> >>> Other embedded projects I know of do this too, and as you point out the compilers keep these APIs standard for folks
> the
> >> provide their own runtimes.
> >>> Thanks,
> >>>
> >>> Andrew Fish
> >>>
> >>>
> >>>
> >>>
> >>> Best regards,
> >>> Marvin
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> Thanks,
> >>>
> >>> Andrew Fish
> >>>
> >>>
> >>>
> >>>
> >>> …
> >>>
> >>>
> >>>
> >>
> >> 
> >>

Re: [edk2-devel] Progress on getting Uncrustify working for EDK2?

[Andrew Fish]

[-- Attachment #1: Type: text/plain, Size: 3471 bytes --]

Mike,

My main concern is there is an easy path for the developers. I don’t want to have developers have to learn new development technologies just to run a “required” tool. I’m not as concerned on how we do it. Hosting binaries from a webpage that auto picks what to download would be fine with me. 

I think the important thing is we make this walkup an use, and we write up the instructions not assuming “other knowledge”. 

My use case is probably uncommon. We don’t use brew, but there is an internal brew we can use. So we generally just install build tools into our repo. We  are more flexible on personal productivity tools so I’d probably consider switching to Visual Studio Code, especially if there was an Extension in the Marketplace I could just click on to configure Code for edk2 development. On the other hand nothing inspires hate like asking people to switch their editor, so I don’t think we can depend on Code as the solution. 

Thanks,

Andrew Fish

> On Nov 9, 2021, at 7:08 AM, Michael D Kinney <michael.d.kinney@intel.com> wrote:
> 
> Submodule within which repo?  What would be the proposed location?
> 
> Would a fork of uncrustify maintained as a repo under TianoCore work as well?
> 
> There are CI checks (including extensive unit tests) and release generation that are built into uncrustify repo and I do not know if those will be functional if it is maintained as a submodule.
> 
> Thanks,
> 
> Mike
> 
>> -----Original Message-----
>> From: Leif Lindholm <leif@nuviainc.com <mailto:leif@nuviainc.com>>
>> Sent: Tuesday, November 9, 2021 4:37 AM
>> To: Gerd Hoffmann <kraxel@redhat.com <mailto:kraxel@redhat.com>>
>> Cc: devel@edk2.groups.io <mailto:devel@edk2.groups.io>; Kinney, Michael D <michael.d.kinney@intel.com <mailto:michael.d.kinney@intel.com>>; Andrew Fish <afish@apple.com <mailto:afish@apple.com>>; Marvin Häuser
>> <mhaeuser@posteo.de <mailto:mhaeuser@posteo.de>>; Michael Kubacki <Michael.Kubacki@microsoft.com <mailto:Michael.Kubacki@microsoft.com>>; mikuback@linux.microsoft.com <mailto:mikuback@linux.microsoft.com>; rebecca@nuviainc.com <mailto:rebecca@nuviainc.com>;
>> Bret Barkelew <Bret.Barkelew@microsoft.com <mailto:Bret.Barkelew@microsoft.com>>
>> Subject: Re: [edk2-devel] Progress on getting Uncrustify working for EDK2?
>> 
>> On Tue, Nov 09, 2021 at 09:40:02 +0100, Gerd Hoffmann wrote:
>>>  Hi,
>>> 
>>>>  3.  Require use of uncrustify tool before submitting patch review emails or PRs.
>>>>     *   The required version would be a formally released version  from the fork maintained by Michael Kubacki until
>> the changes can be upstreamed.
>>>>     *   https://dev.azure.com/projectmu/Uncrustify
>>> 
>>> Can we please *first* get the changes merged to upstream uncrustify?
>>> 
>>> That'll make the whole process much less painful because the usual
>>> software repositories (linux distro packages, macos homebrew, ...)
>>> can be used to install uncrustify then, and it's also less confusing if
>>> developers don't have to juggle with different uncrustify variants
>>> (upstream vs. edk2).
>> 
>> Whilst I agree in principle...
>> 
>> This means postponing automated coding style changes until 2023
>> (Debian stable), 2025 (Ubuntu LTS), ??? (RHEL10), or even later
>> ... and I'd rather not.
>> 
>> I like Marvin's suggestion of a submodule. Which we could drop once
>> no longer needed.
>> 
>> /
>>    Leif
> 
> 
> 


[-- Attachment #2: Type: text/html, Size: 11893 bytes --]

Re: [edk2-devel] Progress on getting Uncrustify working for EDK2?

[Michael D Kinney]

[-- Attachment #1: Type: text/plain, Size: 4665 bytes --]

Andrew,

Agree that we want it to be very simple and walk up and use.

There is no requirement to use Visual Studio Code.

There are multiple ways available for use by developers

  1.  Visual Studio Code plugin.
  2.  Run uncrustify tool from command line.
  3.  Implement a githook that automatically runs uncrustify as part of local commit.
  4.  If you look at bottom of this readme it describe how to integrate with Vim and IntelliJ

https://github.com/uncrustify/uncrustify

What is critical at this stage is identifying specific work items to complete the source style conversion.  Defining exactly “how we do it” is important for success.  Looks like your suggestion for the developer use case is a download page that auto picks the right installer.

Best regards,

Mike

From: Andrew Fish <afish@apple.com>
Sent: Tuesday, November 9, 2021 10:14 AM
To: edk2-devel-groups-io <devel@edk2.groups.io>; Kinney, Michael D <michael.d.kinney@intel.com>
Cc: Leif Lindholm <leif@nuviainc.com>; Gerd Hoffmann <kraxel@redhat.com>; Marvin Häuser <mhaeuser@posteo.de>; Michael Kubacki <Michael.Kubacki@microsoft.com>; mikuback@linux.microsoft.com; rebecca@nuviainc.com; Bret Barkelew <Bret.Barkelew@microsoft.com>
Subject: Re: [edk2-devel] Progress on getting Uncrustify working for EDK2?

Mike,

My main concern is there is an easy path for the developers. I don’t want to have developers have to learn new development technologies just to run a “required” tool. I’m not as concerned on how we do it. Hosting binaries from a webpage that auto picks what to download would be fine with me.

I think the important thing is we make this walkup an use, and we write up the instructions not assuming “other knowledge”.

My use case is probably uncommon. We don’t use brew, but there is an internal brew we can use. So we generally just install build tools into our repo. We  are more flexible on personal productivity tools so I’d probably consider switching to Visual Studio Code, especially if there was an Extension in the Marketplace I could just click on to configure Code for edk2 development. On the other hand nothing inspires hate like asking people to switch their editor, so I don’t think we can depend on Code as the solution.

Thanks,

Andrew Fish


On Nov 9, 2021, at 7:08 AM, Michael D Kinney <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>> wrote:

Submodule within which repo?  What would be the proposed location?

Would a fork of uncrustify maintained as a repo under TianoCore work as well?

There are CI checks (including extensive unit tests) and release generation that are built into uncrustify repo and I do not know if those will be functional if it is maintained as a submodule.

Thanks,

Mike


-----Original Message-----
From: Leif Lindholm <leif@nuviainc.com<mailto:leif@nuviainc.com>>
Sent: Tuesday, November 9, 2021 4:37 AM
To: Gerd Hoffmann <kraxel@redhat.com<mailto:kraxel@redhat.com>>
Cc: devel@edk2.groups.io<mailto:devel@edk2.groups.io>; Kinney, Michael D <michael.d.kinney@intel.com<mailto:michael.d.kinney@intel.com>>; Andrew Fish <afish@apple.com<mailto:afish@apple.com>>; Marvin Häuser
<mhaeuser@posteo.de<mailto:mhaeuser@posteo.de>>; Michael Kubacki <Michael.Kubacki@microsoft.com<mailto:Michael.Kubacki@microsoft.com>>; mikuback@linux.microsoft.com<mailto:mikuback@linux.microsoft.com>; rebecca@nuviainc.com<mailto:rebecca@nuviainc.com>;
Bret Barkelew <Bret.Barkelew@microsoft.com<mailto:Bret.Barkelew@microsoft.com>>
Subject: Re: [edk2-devel] Progress on getting Uncrustify working for EDK2?

On Tue, Nov 09, 2021 at 09:40:02 +0100, Gerd Hoffmann wrote:

 Hi,


 3.  Require use of uncrustify tool before submitting patch review emails or PRs.
    *   The required version would be a formally released version  from the fork maintained by Michael Kubacki until
the changes can be upstreamed.

    *   https://dev.azure.com/projectmu/Uncrustify

Can we please *first* get the changes merged to upstream uncrustify?

That'll make the whole process much less painful because the usual
software repositories (linux distro packages, macos homebrew, ...)
can be used to install uncrustify then, and it's also less confusing if
developers don't have to juggle with different uncrustify variants
(upstream vs. edk2).

Whilst I agree in principle...

This means postponing automated coding style changes until 2023
(Debian stable), 2025 (Ubuntu LTS), ??? (RHEL10), or even later
... and I'd rather not.

I like Marvin's suggestion of a submodule. Which we could drop once
no longer needed.

/
   Leif





[-- Attachment #2: Type: text/html, Size: 53474 bytes --]

Re: [edk2-devel] Progress on getting Uncrustify working for EDK2?

[Andrew Fish]

[-- Attachment #1: Type: text/plain, Size: 5170 bytes --]



> On Nov 9, 2021, at 10:26 AM, Kinney, Michael D <michael.d.kinney@intel.com> wrote:
> 
> Andrew,
>  
> Agree that we want it to be very simple and walk up and use.
>  
> There is no requirement to use Visual Studio Code.
>  
> There are multiple ways available for use by developers
> Visual Studio Code plugin.
> Run uncrustify tool from command line.
> Implement a githook that automatically runs uncrustify as part of local commit.
> If you look at bottom of this readme it describe how to integrate with Vim and IntelliJ
> https://github.com/uncrustify/uncrustify <https://github.com/uncrustify/uncrustify>
>  
> What is critical at this stage is identifying specific work items to complete the source style conversion.  Defining exactly “how we do it” is important for success. Looks like your suggestion for the developer use case is a download page that auto picks the right installer.
>  

I’d be OK if to start it just listed the various options, if making the button smart is too much work. 

Thanks,

Andrew Fish

> Best regards,
>  
> Mike
>  
> From: Andrew Fish <afish@apple.com> 
> Sent: Tuesday, November 9, 2021 10:14 AM
> To: edk2-devel-groups-io <devel@edk2.groups.io>; Kinney, Michael D <michael.d.kinney@intel.com>
> Cc: Leif Lindholm <leif@nuviainc.com>; Gerd Hoffmann <kraxel@redhat.com>; Marvin Häuser <mhaeuser@posteo.de>; Michael Kubacki <Michael.Kubacki@microsoft.com>; mikuback@linux.microsoft.com; rebecca@nuviainc.com; Bret Barkelew <Bret.Barkelew@microsoft.com>
> Subject: Re: [edk2-devel] Progress on getting Uncrustify working for EDK2?
>  
> Mike,
>  
> My main concern is there is an easy path for the developers. I don’t want to have developers have to learn new development technologies just to run a “required” tool. I’m not as concerned on how we do it. Hosting binaries from a webpage that auto picks what to download would be fine with me. 
>  
> I think the important thing is we make this walkup an use, and we write up the instructions not assuming “other knowledge”. 
>  
> My use case is probably uncommon. We don’t use brew, but there is an internal brew we can use. So we generally just install build tools into our repo. We  are more flexible on personal productivity tools so I’d probably consider switching to Visual Studio Code, especially if there was an Extension in the Marketplace I could just click on to configure Code for edk2 development. On the other hand nothing inspires hate like asking people to switch their editor, so I don’t think we can depend on Code as the solution. 
>  
> Thanks,
>  
> Andrew Fish
> 
> 
> On Nov 9, 2021, at 7:08 AM, Michael D Kinney <michael.d.kinney@intel.com <mailto:michael.d.kinney@intel.com>> wrote:
>  
> Submodule within which repo?  What would be the proposed location?
> 
> Would a fork of uncrustify maintained as a repo under TianoCore work as well?
> 
> There are CI checks (including extensive unit tests) and release generation that are built into uncrustify repo and I do not know if those will be functional if it is maintained as a submodule.
> 
> Thanks,
> 
> Mike
> 
> 
> -----Original Message-----
> From: Leif Lindholm <leif@nuviainc.com <mailto:leif@nuviainc.com>>
> Sent: Tuesday, November 9, 2021 4:37 AM
> To: Gerd Hoffmann <kraxel@redhat.com <mailto:kraxel@redhat.com>>
> Cc: devel@edk2.groups.io <mailto:devel@edk2.groups.io>; Kinney, Michael D <michael.d.kinney@intel.com <mailto:michael.d.kinney@intel.com>>; Andrew Fish <afish@apple.com <mailto:afish@apple.com>>; Marvin Häuser
> <mhaeuser@posteo.de <mailto:mhaeuser@posteo.de>>; Michael Kubacki <Michael.Kubacki@microsoft.com <mailto:Michael.Kubacki@microsoft.com>>; mikuback@linux.microsoft.com <mailto:mikuback@linux.microsoft.com>; rebecca@nuviainc.com <mailto:rebecca@nuviainc.com>;
> Bret Barkelew <Bret.Barkelew@microsoft.com <mailto:Bret.Barkelew@microsoft.com>>
> Subject: Re: [edk2-devel] Progress on getting Uncrustify working for EDK2?
> 
> On Tue, Nov 09, 2021 at 09:40:02 +0100, Gerd Hoffmann wrote:
> 
>  Hi,
> 
> 
>  3.  Require use of uncrustify tool before submitting patch review emails or PRs.
>     *   The required version would be a formally released version  from the fork maintained by Michael Kubacki until
> the changes can be upstreamed.
> 
>     *   https://dev.azure.com/projectmu/Uncrustify <https://dev.azure.com/projectmu/Uncrustify>
> 
> Can we please *first* get the changes merged to upstream uncrustify?
> 
> That'll make the whole process much less painful because the usual
> software repositories (linux distro packages, macos homebrew, ...)
> can be used to install uncrustify then, and it's also less confusing if
> developers don't have to juggle with different uncrustify variants
> (upstream vs. edk2).
> 
> Whilst I agree in principle...
> 
> This means postponing automated coding style changes until 2023
> (Debian stable), 2025 (Ubuntu LTS), ??? (RHEL10), or even later
> ... and I'd rather not.
> 
> I like Marvin's suggestion of a submodule. Which we could drop once
> no longer needed.
> 
> /
>    Leif
> 
> 
> 


[-- Attachment #2: Type: text/html, Size: 17396 bytes --]

Re: [edk2-devel] Progress on getting Uncrustify working for EDK2?

[Leif Lindholm]

Hi Mike,

On Tue, Nov 09, 2021 at 16:33:31 +0000, Michael D Kinney wrote:
> > >> 1. If virtually everyone will need Uncrustify, why cannot it be built along with BaseTools from a submodule? Especially
> > >> with the fork that makes sense, after that it depends on the upstream (it does not look too nice to me).
> > > No matter where uncrustify sources are hosted, developers can always choose to build the uncrustify tool locally.
> > > Providing release binaries for the tool may be simpler for some customers.
> > > Using release binaries from EDK II CI agents will reduce CI execution time.
> > 
> > My point was it'd be nice if it (optionally) "just worked", so
> > Uncrustify could be be built as part of the edk2 BaseTools build
> > process, or release binaries could be downloaded by some script, or
> > whatever really. I guess it could be the same logic as for the CI?
> 
> Setting up a development work environment also requires the installation of tools such as 
> compilers, NASM, IASL, etc.  These are not handled as part of BaseTools today.

No, but users are not required to install a specific, not-yet-upstream
version of a tool. This is the fundamental problem here.

Once the edk2 support is available in upstream uncrustify, there is a
substantial lag in how the updated version trickles down through
packaging systems.

> Supporting this indent style is one of the enhancements in the fork.  If we want to align
> to one of the indent styles supported by a wider array of source editors/IDEs, then that
> would require a change to the EDK II Coding Style Specification and approval from the
> EDK II community.

Yes. The above problem of getting the tool at the appropriate version
is why I previously stated my preference as being to change to coding
style so it matched what was already supported in upstream uncrustify.

Adding a version of the tool as a submodule of BaseTools would be  a
mitigation for the 5-year window introduced by *depending* on
not-yet-upstream support in a commonly used tool.

> > 6) Allow static function declarations.
> 
> I agree that static functions should be allowed.  Please add any comments you have to 
> the following Bugzilla.  If you have ideas on the specific spec updates required, then
> please provide a patch against the spec markdown.
> 
>     https://bugzilla.tianocore.org/show_bug.cgi?id=1766

I think that's a different issue, but otherwise agreed.

/
    Leif

Re: [edk2-devel] Progress on getting Uncrustify working for EDK2?

[Michael D Kinney]

Leif,

Comments below.

Mike

> -----Original Message-----
> From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Leif Lindholm
> Sent: Tuesday, November 9, 2021 12:56 PM
> To: devel@edk2.groups.io; Kinney, Michael D <michael.d.kinney@intel.com>
> Cc: Marvin Häuser <mhaeuser@posteo.de>; Andrew Fish <afish@apple.com>; Michael Kubacki <Michael.Kubacki@microsoft.com>;
> mikuback@linux.microsoft.com; rebecca@nuviainc.com; Bret Barkelew <Bret.Barkelew@microsoft.com>
> Subject: Re: [edk2-devel] Progress on getting Uncrustify working for EDK2?
> 
> Hi Mike,
> 
> On Tue, Nov 09, 2021 at 16:33:31 +0000, Michael D Kinney wrote:
> > > >> 1. If virtually everyone will need Uncrustify, why cannot it be built along with BaseTools from a submodule?
> Especially
> > > >> with the fork that makes sense, after that it depends on the upstream (it does not look too nice to me).
> > > > No matter where uncrustify sources are hosted, developers can always choose to build the uncrustify tool locally.
> > > > Providing release binaries for the tool may be simpler for some customers.
> > > > Using release binaries from EDK II CI agents will reduce CI execution time.
> > >
> > > My point was it'd be nice if it (optionally) "just worked", so
> > > Uncrustify could be be built as part of the edk2 BaseTools build
> > > process, or release binaries could be downloaded by some script, or
> > > whatever really. I guess it could be the same logic as for the CI?
> >
> > Setting up a development work environment also requires the installation of tools such as
> > compilers, NASM, IASL, etc.  These are not handled as part of BaseTools today.
> 
> No, but users are not required to install a specific, not-yet-upstream
> version of a tool. This is the fundamental problem here.
> 
> Once the edk2 support is available in upstream uncrustify, there is a
> substantial lag in how the updated version trickles down through
> packaging systems.

There is an RFC with no negative feedback to move to NASM 2.15.05.

I do not believe that is the version supported in all the distros
packaging systems yet.

So I expect there will likely be many windows of time where the 
EDK II projects have a dependency on a tool in that state.

If this was a significant concern, why was it not raised with the
NASM 2.15.05?

> 
> > Supporting this indent style is one of the enhancements in the fork.  If we want to align
> > to one of the indent styles supported by a wider array of source editors/IDEs, then that
> > would require a change to the EDK II Coding Style Specification and approval from the
> > EDK II community.
> 
> Yes. The above problem of getting the tool at the appropriate version
> is why I previously stated my preference as being to change to coding
> style so it matched what was already supported in upstream uncrustify.

Stating a preference without specific details on the exact style that is
actionable by developers and verifiable does not help close on this topic.

Can you provide the specific details and update the BZs provided in an
earlier version of this thread:

	https://edk2.groups.io/g/devel/message/81609

If you are not able to provide details, but are rejecting this proposal all
together, then please be clear that you are rejecting the proposal so those
that are working on it can move on to other priorities.

> 
> Adding a version of the tool as a submodule of BaseTools would be  a
> mitigation for the 5-year window introduced by *depending* on
> not-yet-upstream support in a commonly used tool.
> 
> > > 6) Allow static function declarations.
> >
> > I agree that static functions should be allowed.  Please add any comments you have to
> > the following Bugzilla.  If you have ideas on the specific spec updates required, then
> > please provide a patch against the spec markdown.
> >
> >     https://bugzilla.tianocore.org/show_bug.cgi?id=1766
> 
> I think that's a different issue, but otherwise agreed.
> 
> /
>     Leif
> 
> 
> 
> 

Re: [edk2-devel] Progress on getting Uncrustify working for EDK2?

[Leif Lindholm]

On Tue, Nov 09, 2021 at 21:20:51 +0000, Michael D Kinney wrote:
> > > Setting up a development work environment also requires the installation of tools such as
> > > compilers, NASM, IASL, etc.  These are not handled as part of BaseTools today.
> > 
> > No, but users are not required to install a specific, not-yet-upstream
> > version of a tool. This is the fundamental problem here.
> > 
> > Once the edk2 support is available in upstream uncrustify, there is a
> > substantial lag in how the updated version trickles down through
> > packaging systems.
> 
> There is an RFC with no negative feedback to move to NASM 2.15.05.
> 
> I do not believe that is the version supported in all the distros
> packaging systems yet.

It is in current Debian stable. It is *not* in Ubuntu LTS.

> So I expect there will likely be many windows of time where the 
> EDK II projects have a dependency on a tool in that state.
> 
> If this was a significant concern, why was it not raised with the
> NASM 2.15.05?

I might have raised it if I had been actively tracking NASM usage, but
I'll confess I don't. But even then - that is talking about NASM
2.15.05, released August 2020, not NASM HEAD today with some
additional patches.

> > > Supporting this indent style is one of the enhancements in the fork.  If we want to align
> > > to one of the indent styles supported by a wider array of source editors/IDEs, then that
> > > would require a change to the EDK II Coding Style Specification and approval from the
> > > EDK II community.
> > 
> > Yes. The above problem of getting the tool at the appropriate version
> > is why I previously stated my preference as being to change to coding
> > style so it matched what was already supported in upstream uncrustify.
> 
> Stating a preference without specific details on the exact style that is
> actionable by developers and verifiable does not help close on this
> topic.

The preference was raised here only as the alternative to requiring
uncrustify modifications.

I stated the delta (with specific details) in my email on 7 October,
and yes, you responeded to that, but no one else seemed interested in
that path, so I didn't follow up.

> Can you provide the specific details and update the BZs provided in an
> earlier version of this thread:
> 
> 	https://edk2.groups.io/g/devel/message/81609
> 
> If you are not able to provide details, but are rejecting this proposal all
> together, then please be clear that you are rejecting the proposal so those
> that are working on it can move on to other priorities.

As per above.

/
    Leif

Re: [edk2-devel] Progress on getting Uncrustify working for EDK2?

[Gerd Hoffmann]

> 5) Indent relative to the last indentation, not to symbol names, i.e.
> 
>   Status = Function (
>     A
>     );
> 
> instead of
> 
>   Status = Function (
>                    A
>                    );
> 
> Uncrustify might support this now, but I have never seen even a single IDE
> that does. It'd be kind of awkward to always write "style-broken" code and
> rely on the external tool to fix it.

IDEs typically use the parentheses for that, so this style ...

    Status = Function
        (A,
         B,
         C,
         );

... might work better in IDEs.

take care,
  Gerd

Re: [edk2-devel] Progress on getting Uncrustify working for EDK2?

[Michael Kubacki]

On 11/9/2021 10:44 AM, Michael D Kinney wrote:
> 
> 
>> -----Original Message-----
>> From: devel@edk2.groups.io <devel@edk2.groups.io> On Behalf Of Gerd Hoffmann
>> Sent: Tuesday, November 9, 2021 6:10 AM
>> To: Leif Lindholm <leif@nuviainc.com>
>> Cc: devel@edk2.groups.io; Kinney, Michael D <michael.d.kinney@intel.com>; Andrew Fish <afish@apple.com>; Marvin Häuser
>> <mhaeuser@posteo.de>; Michael Kubacki <Michael.Kubacki@microsoft.com>; mikuback@linux.microsoft.com; rebecca@nuviainc.com;
>> Bret Barkelew <Bret.Barkelew@microsoft.com>
>> Subject: Re: [edk2-devel] Progress on getting Uncrustify working for EDK2?
>>
>> On Tue, Nov 09, 2021 at 12:36:30PM +0000, Leif Lindholm wrote:
>>> On Tue, Nov 09, 2021 at 09:40:02 +0100, Gerd Hoffmann wrote:
>>>>    Hi,
>>>>
>>>>>    3.  Require use of uncrustify tool before submitting patch review emails or PRs.
>>>>>       *   The required version would be a formally released version  from the fork maintained by Michael Kubacki
>> until the changes can be upstreamed.
>>>>>       *   https://dev.azure.com/projectmu/Uncrustify
>>>>
>>>> Can we please *first* get the changes merged to upstream uncrustify?
>>>>
>>>> That'll make the whole process much less painful because the usual
>>>> software repositories (linux distro packages, macos homebrew, ...)
>>>> can be used to install uncrustify then, and it's also less confusing if
>>>> developers don't have to juggle with different uncrustify variants
>>>> (upstream vs. edk2).
>>>
>>> Whilst I agree in principle...
>>>
>>> This means postponing automated coding style changes until 2023
>>> (Debian stable), 2025 (Ubuntu LTS), ??? (RHEL10), or even later
>>> ... and I'd rather not.
>>>
>>> I like Marvin's suggestion of a submodule. Which we could drop once
>>> no longer needed.
>>
>> I'm fine with a submodule (temporary & optional) so we don't have to
>> wait until all distros (especially the LTS ones which are slower) have
>> picked up the latest uncrustify release.
>>
>> But I still think upstreaming the uncrustify changes should be the first
>> step.
> 
> The first step is to review and accept the style of code generated by
> the current fork.
> 
I agree. I'm happy to prepare the changes for upstream but I would like 
to make sure the changes are needed.
>>
>> take care,
>>    Gerd
>>
>>
>>
>>
>>
> 
> 
> 
> 
> 
>