TianoCore Community Meeting Minutes

TianoCore Community Meeting Minutes

[Jeremiah Cox]

Finally finding time to respond to an older thread...
>>> The was a concern raised over potential lock-in to Github's, specifically in
>>> regards to history retention.
>>> Several Github users brought up that this shouldn't be an issue.
>>
>> Hopefully they said more than that.
>> What does "shouldn't be an issue" mean.
>> Were these users from multiple organisations?
>
>I will ask for more details in the next meeting, but the comments were
>that the API is quite robust and lends itself to readily accessible data.

I'd like to better understand assertions of GitHub "lock in".  GitHub provides a comprehensive REST API that you can easily drive via Python ("pip install PyGithub").
I recall one assertion that GitHub holds PR comments hostage, but it is trivial to dump out all Comments on all PRs in a repo.


   g = Github( <access_token> )
   repo = g.get_repo("chipsec/chipsec")
   pulls = repo.get_pulls(sort='created', base='master')
   for pr in pulls:
      for comment in pr.get_review_comments():
         print(pr.number)
         print(comment.body)
         ...

Can folks help me better understand examples of GitHub "lock in"?  I think it would be trivial to author a daemon that listens for PRs and comments, and forward those to a mailing list, if that is preferred for archival.


>>
>>> Shawn mentioned some benefits to stock Github such as
>>> it is always up to date, it includes APIs to extract data, pull
>>> requests
>>
>> Since we are discussing multiple different development systems, can we
>> try to be a bit more explicit? This is referring to github's web-based
>> branch-based ticketing system, yes?
>
>Yes that's correct.

With respect to web-based, we believe it facilitates convenient, multi-platform access, and GitHub's REST APIs can be leveraged to provide a command-line driven experience.  I believe Sean's point was that GitHub provides a 1-stop shop for code reviews and issue tracking such that there is no need to spend TianoCore resources set up, maintain, and update separate Bugzilla and Gerrit services, VMs, OSs, & machines.  The REST APIs could be used to maintain a mailing list or other mirror.  I am not asserting that GitHub provides the best code review and issue tracker experience, our team finds Azure Dev Ops is superior, but GitHub is sufficient for most use cases while handling the infrastructure so that we can focus on getting things done.

I think it would be helpful to construct a PROs/CONs table for each of the proposed end-to-end solution (source control, code review, gates, CI/CD, issue tracking, testing, ...).  Perhaps this table is on a wiki to enable faster iteration than emails on the DL.


>> I know language drift and all, but
>> <LINK>
>> is what pull request means to at least 3 participants in this conversation.
>
>I think this is an important distinction to make and I'm sure it will
>come up in future discussions. I'll be sure to bring this up.

We think of "Pull Request" as a contribution & review process akin to GitHub, GitLab, or Azure DevOps.  Feedback, responses, automated check-in gates, and signoff can be handled within each PR ticket.  Policies can be assigned, automated tests that must pass, enforcement of community wide rules (# of reviewers, reviewer list, all comments "resolved").  Tests could start as simple as successful build and over time could grow to a more comprehensive set of boot, functional, static analysis, ...
By leveraging a popular hosting provider's built-in workflow we enjoy significant efficiencies.  Its fully documented and known to most.  New capabilities are being added daily.  No TianoCore resources would be required to manage systems, OS patches, network issues, etc and can instead be focused on TianoCore code, building tests, and driving process improvements.


>>
>> It seems somewhat less than ideal to me that all of the github
>> proponents were on the opposite call to me and Laszlo (and Ard). Were
>> any of them Asia-based or could we try to get them on the call with
>> Europe next time? I'm sure me and Laszlo could be somewhat more
>> accommodating than your 7AM, but we're not going to stay up for a 3/4AM
>> meeting about source control.
>
>That's understandable. I'm not sure where Shawn and Nate are located,
>but I believe it is in the US. We could certainly move the AM call a bit
>later if it makes it easier to get more folks from the US on that call.

Agreed, pushing the US/EU call out a couple hours (8 or 9am Pacific) would likely increase Pacific participation.


Thanks,
Jeremiah

Re: TianoCore Community Meeting Minutes

[Laszlo Ersek]

Hello Jeremiah,

On 10/19/18 18:09, Jeremiah Cox wrote:

> I'd like to better understand assertions of GitHub "lock in".  GitHub
> provides a comprehensive REST API that you can easily drive via Python
> ("pip install PyGithub").
> I recall one assertion that GitHub holds PR comments hostage, but it
> is trivial to dump out all Comments on all PRs in a repo.
>
>
>    g = Github( <access_token> )
>    repo = g.get_repo("chipsec/chipsec")
>    pulls = repo.get_pulls(sort='created', base='master')
>    for pr in pulls:
>       for comment in pr.get_review_comments():
>          print(pr.number)
>          print(comment.body)
>          ...
>
> Can folks help me better understand examples of GitHub "lock in"?  I
> think it would be trivial to author a daemon that listens for PRs and
> comments, and forward those to a mailing list, if that is preferred
> for archival.

> With respect to web-based, we believe it facilitates convenient,
> multi-platform access, and GitHub's REST APIs can be leveraged to
> provide a command-line driven experience.  I believe Sean's point was
> that GitHub provides a 1-stop shop for code reviews and issue tracking
> such that there is no need to spend TianoCore resources set up,
> maintain, and update separate Bugzilla and Gerrit services, VMs, OSs,
> & machines.  The REST APIs could be used to maintain a mailing list or
> other mirror.  I am not asserting that GitHub provides the best code
> review and issue tracker experience, our team finds Azure Dev Ops is
> superior, but GitHub is sufficient for most use cases while handling
> the infrastructure so that we can focus on getting things done.
>
> I think it would be helpful to construct a PROs/CONs table for each of
> the proposed end-to-end solution (source control, code review, gates,
> CI/CD, issue tracking, testing, ...).  Perhaps this table is on a wiki
> to enable faster iteration than emails on the DL.

> We think of "Pull Request" as a contribution & review process akin to
> GitHub, GitLab, or Azure DevOps.  Feedback, responses, automated
> check-in gates, and signoff can be handled within each PR ticket.
> Policies can be assigned, automated tests that must pass, enforcement
> of community wide rules (# of reviewers, reviewer list, all comments
> "resolved").  Tests could start as simple as successful build and over
> time could grow to a more comprehensive set of boot, functional,
> static analysis, ...
> By leveraging a popular hosting provider's built-in workflow we enjoy
> significant efficiencies.  Its fully documented and known to most. New
> capabilities are being added daily.  No TianoCore resources would be
> required to manage systems, OS patches, network issues, etc and can
> instead be focused on TianoCore code, building tests, and driving
> process improvements.

I don't have anything against GitHub specifically. I also don't have any
"favorite" WebUI (beyond Bugzilla, of course :) ). I'm concerned about a
number of general anti-patterns that might not be obvious, and might not
be easy to prevent with any given modern web application. Perhaps I can
word this as a personal wish list:

* Full email audit trail (all changes mailed out, even own ones), about
  patch submission, PRs, patch review comments, and bug tracker actions
  alike.

  One should be able to establish the full history of a bug or a pull
  request just from the emails. (This would be for human readers, not
  for machines.) Review comments made on the web, in reference to
  specific parts of a patch, should preserve that context in email form.

  The messages generated for a given bug or PR should form a thread on
  the mailing list. (Proper use of In-Reply-To and References headers.)

  The generated emails should be readable without HTML support (i.e. as
  plain text). As a counter-example for this, I'll mention JIRA --
  JIRA's emails are practically unreadable without HTML rendering
  support in the MUA.

  If a WebUI can offer such an email gateway as a feature, that's
  awesome. It is fine if the gateway is one-way only (web to email). The
  service should not be an external one however, in order to minimize
  email lossage when the WebUI works but the gateway goes offline for
  some reason. The gateway should be tightly integrated.

* The comment editing interface on the WebUI should support, in addition
  to the default rich editing interface, a plaintext mode that uses
  monospace font, interprets no markdown sequences, and preserves the
  exact layout of the plaintext comment (including indentation and line
  wrapping).

  Counter-examples abound. Perhaps somewhat unexpectedly, I'll call out
  Launchpad. Among other things:

  (a) Launchpad truncates comments it thinks are "too long", in the full
      bug display -- and then readers have to open the bug comment
      separately, to see it in its full length;

  (b) Launchpad inserts invisible breaks into words it thinks are "too
      long" -- and then, if someone double clicks a 40-nibble commit
      hash in the middle, they will not see it wholly selected; only a
      middle section will be selected between the hidden word breaks.
      The invisible word breaks will also allow commit hashes to be
      wrapped.

* A similar "stick to basics" criterion exists for commit message
  authoring, and display.

  Commit messages are to be authored with monospace font in mind, and 74
  character line length.

  Commit messages should not be reflowed or rewrapped automatically on
  display. BZ numbers and commit hash references may be highlighted and
  turned into actual links, but they should not be shortened or
  otherwise reformatted. The original formatting of a commit message is
  not a mindless process, and any WebUI should respect that, and
  preserve the original author's work, for display too.

* Old versions of patch series (pull requests) and their associated
  review comments must remain publicly available forever. No historical
  review comments must ever turn into dangling references on the WebUI,
  nor must old (rejected) pull requests (topic branches) ever be
  possible to delete (or to overwrite by force-pushing), even for the
  original author.

* No "squash on merge". Structuring a patch series is a first class
  development activity, and the git history is a first class aspect of
  the codebase.

* More generally, admins should have fine-grained control to disable
  unwanted WebUI features.

* "Edit by reviewer on merge" should be restricted to minimally invasive
  changes, and participants must agree on such actions up-front.

* I don't know how a WebUI can support "rebase by maintainer". On one
  hand, such a feature would have to be used very carefully (see the
  previous point); on the other hand, adding Reviewed-by and similar
  tags without it is not possible for the maintainer.

* Support for displaying and perhaps editing "git notes" (see
  git-notes(1)) would be a plus. They permit developers to attach
  patch-level changelogs (incremental change descriptions) to individual
  patches, such that these changelogs never make it to the final git
  history. Such notes are useful for repeated reviews, but not really
  useful for the long-term git history (which does not capture the
  discussion context / previous versions of the pull requests anyway).

Thanks!
Laszlo

TianoCore Community Meeting Minutes

[stephano]

Thank you all for a great set of meetings!

This is an overview of the topics discussed and the tasks that were 
assigned. Please feel free to send me any questions or comments.

Licensing
---------
Licensing was discussed and no questions or comments came up. Please 
feel free to contact me directly if you would like more info.

Community Meetings
------------------
Community meetings will be held regularly. We will plan public bug 
scrubs and design meetings. Meeting formats will use zoom and be held 2 
times per day to accommodate all time zones. Invites will be posted to 
the mailing list and the wiki. Since the mailing list does not allow 
attachments, I will work on getting an "add to calendar" link attached 
to the wiki page.

I will send out a poll to see if the community would be interested in a 
Slack channel. I'm happy to admin it if there is interest.

I will also be setting up a specific meeting, probably next month, to 
discuss general code and commit message standards.


Patch Workflow Improvement
--------------------------
We would like to find a more modern, user friendly patch workflow that 
fits (as best as it can) most company's workflow. It is recommended that 
the community familiarize themselves with some of the options that are 
out there so that we can pose useful questions / comments at the next 
meeting. Please be sure to research up-to-date information and not rely 
on old info. Options and tools discussed included:
-Github
-Gitlab
-Phabricator
-Gerrit

The was a concern raised over potential lock-in to Github's, 
specifically in regards to history retention. Several Github users 
brought up that this shouldn't be an issue. Shawn mentioned some 
benefits to stock Github such as it is always up to date, it includes 
APIs to extract data, pull requests lend themselves to simple(r) 
automation, and it has a much more modern web UI than Gerrit.

Several community members including Marcin and Rebecca have used 
Phabricator and can recommend it. Intel has used Gerrit internally and 
can recommend its use.

No one voiced any issues with Bugzilla and several from the community 
thought that we should continue to use it.

Improvements to the Build Process
--------------------------------
We would like to gather a list of concrete specific proposals. Nate 
mentioned that there will always be some specialized tooling because of 
the nature of BIOS. Shawn mentioned that we need to keep in mind that 
making development easier for current developers should be a priority 
over making the processes easier for newcomers. I will send out emails 
asking for specific feedback on some of these topics:

-Which toolchains are being used, which are validated, and which are 
known to create reproducible builds.
-When toolchains are known to be orphaned, should they be archived or 
simply removed.
-Could we add some kconfig-like tool that allows introspection into what 
type of builds are available.
-How can we better track the code quality of BaseTools and the current 
build system on the whole. Should we add a "requires documentation 
change" flag to BZ so that it will be easier to compile a list of 
required doc changes.

Changing to LF instead of CRLF
------------------------------
It was agreed that this is a good idea but that we should create a test 
repo to verify this change before implementing it. It was noted that 
this will cause some overhead with tasks like git blame, however the 
benefits outweigh the costs.

Switching to Standard C Types
-----------------------------
Both Shawn and Nate mentioned that the current system has been in place 
for a long time and some people prefer the current setup. I can start an 
email discussion around this issue specifically if anyone feels strongly 
that we should be using standard types.

Using Git Submodules (like we do with OpenSSL)
--------------------
Many in the group liked this idea. I will engage with the communities to 
gauge their interest in taking patches or helping with our integration 
of their software.

Public CI
---------
This subject was not discussed at length but there seems to be quite a 
bit of community interest. We will discuss this more in the next 
meeting. Please familiarize yourself with some possible solutions, with 
a focus on supporting Windows + Linux + Mac. Some possible solutions 
include:

https://azure.microsoft.com/en-us/blog/introducing-azure-devops/
https://cirrus-ci.org/

Mike has tried out Cirrus with some success. Some positive points to 
Cirrus include:
-Build time of ~ 7 Minutes (clone edk2, apt-get all dependent tools, 
build BaseTools, and build 3 versions of OVMF)
-Caching of VM contents so reduce build times of subsequent builds
-We could use platforms like OVMF to boot FW and provide test results 
from tools like MicroPythonTestFramework
-Cirrus includes integration with docker


Cheers,
Stephano

Stephano Cetola
TianoCore Community Manager
stephano.cetola@linux.intel.com

Re: TianoCore Community Meeting Minutes

[Leif Lindholm]

Hi Stephano,

Thanks for pulling this together.
Some comments below.

On Thu, Oct 11, 2018 at 10:43:57AM -0700, stephano wrote:
> Thank you all for a great set of meetings!
> 
> This is an overview of the topics discussed and the tasks that were
> assigned. Please feel free to send me any questions or comments.
> 
> Licensing
> ---------
> Licensing was discussed and no questions or comments came up. Please feel
> free to contact me directly if you would like more info.
> 
> Community Meetings
> ------------------
> Community meetings will be held regularly. We will plan public bug scrubs
> and design meetings. Meeting formats will use zoom and be held 2 times per
> day to accommodate all time zones.

So I didn't comment on the call (which I dialled in to, using a
phone, because that was the only thing that wasn't going to delay me
10 minutes), but I would prefer a conferencing system that does not require
binary downloads to participate in.

I know at least one organisation that would be prevented from using it
until they had explicit approval (which could take a while), and it
puts me in the position of having to keep a spare x86 machine on my
desktop in order to participate.

Solutions I know would work in this scenario (i.e. work without
plugins or downloads in any modern browser) are:
- google hangouts (requires accounts, not ideal for China)
- Skype (not SFB though, only plain old Skype)
- Bluejeans (requires organiser account)

Sure, I could attend via the Android app for zoom, but it would make
it tedious to paste links to/from chat and put me on a tiny screen.

> Invites will be posted to the mailing
> list and the wiki. Since the mailing list does not allow attachments, I will
> work on getting an "add to calendar" link attached to the wiki page.
> 
> I will send out a poll to see if the community would be interested in a
> Slack channel. I'm happy to admin it if there is interest.
> 
> I will also be setting up a specific meeting, probably next month, to
> discuss general code and commit message standards.
> 
> 
> Patch Workflow Improvement
> --------------------------
> We would like to find a more modern, user friendly patch workflow that fits
> (as best as it can) most company's workflow.
> It is recommended that the
> community familiarize themselves with some of the options that are out there
> so that we can pose useful questions / comments at the next meeting. Please
> be sure to research up-to-date information and not rely on old info. Options
> and tools discussed included:
> -Github
> -Gitlab
> -Phabricator
> -Gerrit
> 
> The was a concern raised over potential lock-in to Github's, specifically in
> regards to history retention.
> Several Github users brought up that this shouldn't be an issue.

Hopefully they said more than that.
What does "shouldn't be an issue" mean.
Were these users from multiple organisations?

> Shawn mentioned some benefits to stock Github such as
> it is always up to date, it includes APIs to extract data, pull
> requests

Since we are discussing multiple different development systems, can we
try to be a bit more explicit? This is referring to github's web-based
branch-based ticketing system, yes?

I know language drift and all, but
https://www.kernel.org/doc/html/latest/maintainer/pull-requests.html
is what pull request means to at least 3 participants in this conversation.

> lend themselves to simple(r) automation, and it has a much more modern web
> UI than Gerrit.

It seems somewhat less than ideal to me that all of the github
proponents were on the opposite call to me and Laszlo (and Ard). Were
any of them Asia-based or could we try to get them on the call with
Europe next time? I'm sure me and Laszlo could be somewhat more
accommodating than your 7AM, but we're not going to stay up for a 3/4AM
meeting about source control.

> Several community members including Marcin and Rebecca have used Phabricator
> and can recommend it. Intel has used Gerrit internally and can recommend its
> use.

I don't think we had anyone from ARM on either of the calls, but I
know they use gerrit internally.

For those who weren't on the call I was on, I am _not_ a fan of gerrit
- but if the deciding factor (as suggested above) is to best align
with the internal processes of contributing organisations, then
gerrit wins hands down.

> No one voiced any issues with Bugzilla and several from the community
> thought that we should continue to use it.
> 
> Improvements to the Build Process
> --------------------------------
> We would like to gather a list of concrete specific proposals. Nate
> mentioned that there will always be some specialized tooling because of the
> nature of BIOS. Shawn mentioned that we need to keep in mind that making
> development easier for current developers should be a priority over making
> the processes easier for newcomers. I will send out emails asking for
> specific feedback on some of these topics:
> 
> -Which toolchains are being used, which are validated, and which are known
> to create reproducible builds.
> -When toolchains are known to be orphaned, should they be archived or simply
> removed.
> -Could we add some kconfig-like tool that allows introspection into what
> type of builds are available.

Well, in our call this was more about the ability to enable larger
groups of functionality without having to explicitly list every single
module included.

> -How can we better track the code quality of BaseTools and the current build
> system on the whole. Should we add a "requires documentation change" flag to
> BZ so that it will be easier to compile a list of required doc changes.

Could we also have a thread on someting that you sort of mention in
the text, but not the bullet points?:
- Suggestions for improvements to BaseTools.

> Changing to LF instead of CRLF
> ------------------------------
> It was agreed that this is a good idea but that we should create a test repo
> to verify this change before implementing it. It was noted that this will
> cause some overhead with tasks like git blame, however the benefits outweigh
> the costs.
> 
> Switching to Standard C Types
> -----------------------------
> Both Shawn and Nate mentioned that the current system has been in place for
> a long time and some people prefer the current setup. I can start an email
> discussion around this issue specifically if anyone feels strongly that we
> should be using standard types.

So, I don't think we made it this far down the agenda on the US-EU
call.

One way would be to simply explicitly permit it, possibly with the
constraint that every module needs to pick one and stick with it,
unless people object.

I think we'll want to discuss this in a US-EU call as well.

> Using Git Submodules (like we do with OpenSSL)
> --------------------

We didn't make it here either. What would we use it _for_?
I think the openssl case makes a lot of sense, but what else?

> Many in the group liked this idea. I will engage with the communities to
> gauge their interest in taking patches or helping with our integration of
> their software.

I think we do need to discuss it on the US-EU call as well before we
do anything else.

> Public CI
> ---------
> This subject was not discussed at length but there seems to be quite a bit
> of community interest. We will discuss this more in the next meeting. Please
> familiarize yourself with some possible solutions, with a focus on
> supporting Windows + Linux + Mac. Some possible solutions include:
> 
> https://azure.microsoft.com/en-us/blog/introducing-azure-devops/
> https://cirrus-ci.org/
> 
> Mike has tried out Cirrus with some success. Some positive points to Cirrus
> include:
> -Build time of ~ 7 Minutes (clone edk2, apt-get all dependent tools, build
> BaseTools, and build 3 versions of OVMF)
> -Caching of VM contents so reduce build times of subsequent builds
> -We could use platforms like OVMF to boot FW and provide test results from
> tools like MicroPythonTestFramework
> -Cirrus includes integration with docker

I wouldn't have anything to add on this topic in a US-EU call, but
thanks for the links.

Regards,

Leif

Re: TianoCore Community Meeting Minutes

[Laszlo Ersek]

On 10/12/18 15:27, Leif Lindholm wrote:
> On Thu, Oct 11, 2018 at 10:43:57AM -0700, stephano wrote:

>> Switching to Standard C Types
>> -----------------------------
>> Both Shawn and Nate mentioned that the current system has been in place for
>> a long time and some people prefer the current setup. I can start an email
>> discussion around this issue specifically if anyone feels strongly that we
>> should be using standard types.
> 
> So, I don't think we made it this far down the agenda on the US-EU
> call.
> 
> One way would be to simply explicitly permit it, possibly with the
> constraint that every module needs to pick one and stick with it,
> unless people object.
> 
> I think we'll want to discuss this in a US-EU call as well.

I'm playing devil's advocate here -- because, in general, I'm a fan of
sticking with standard C as much as possible --, but I see a big
obstacle in the way.

That obstacle is "Table 5. Common UEFI Data Types", in the UEFI spec.
Until a good portion of that table is expressed in terms of standard C
types as well (expanding upon the current definitions), possibly in an
edk2-level spec (i.e. not necessarily in the UEFI spec itself), I think
there's no chance to enable standard C types in edk2 *meaningfully*.

Because, as soon as you have to call a PI or UEFI interface, you'll have
to stick with the PI/UEFI spec types anyway.


>> Using Git Submodules (like we do with OpenSSL)
>> --------------------
> 
> We didn't make it here either. What would we use it _for_?
> I think the openssl case makes a lot of sense, but what else?

We embed a bunch of other projects (libraries, mainly):
- Oniguruma
- Brotli
- fdt
- LZMA SDK
- ...

Thanks
Laszlo

Re: TianoCore Community Meeting Minutes

[Leif Lindholm]

On Fri, Oct 12, 2018 at 06:07:01PM +0200, Laszlo Ersek wrote:
> On 10/12/18 15:27, Leif Lindholm wrote:
> > On Thu, Oct 11, 2018 at 10:43:57AM -0700, stephano wrote:
> 
> >> Switching to Standard C Types
> >> -----------------------------
> >> Both Shawn and Nate mentioned that the current system has been in place for
> >> a long time and some people prefer the current setup. I can start an email
> >> discussion around this issue specifically if anyone feels strongly that we
> >> should be using standard types.
> > 
> > So, I don't think we made it this far down the agenda on the US-EU
> > call.
> > 
> > One way would be to simply explicitly permit it, possibly with the
> > constraint that every module needs to pick one and stick with it,
> > unless people object.
> > 
> > I think we'll want to discuss this in a US-EU call as well.
> 
> I'm playing devil's advocate here -- because, in general, I'm a fan of
> sticking with standard C as much as possible --, but I see a big
> obstacle in the way.
> 
> That obstacle is "Table 5. Common UEFI Data Types", in the UEFI spec.
> Until a good portion of that table is expressed in terms of standard C
> types as well (expanding upon the current definitions), possibly in an
> edk2-level spec (i.e. not necessarily in the UEFI spec itself), I think
> there's no chance to enable standard C types in edk2 *meaningfully*.
> 
> Because, as soon as you have to call a PI or UEFI interface, you'll have
> to stick with the PI/UEFI spec types anyway.

I don't necessarily see that as an issue. But it is a good point that
it can't just be the codebase changing.

Since we are however extremely specificly not looking to change the
underlying storage types, all it would take would be to make a
2-column table into a 3-column table in both specs. Or just add a
separate table for the mapping. Then edk2 could adopt the "permitted"
rule as soon as the specs were out. Arguably (because we're not
changing underlying types) we could do it before, but we _are_
supposed to be the reference implementation, so it would be poor form.

> >> Using Git Submodules (like we do with OpenSSL)
> >> --------------------
> > 
> > We didn't make it here either. What would we use it _for_?
> > I think the openssl case makes a lot of sense, but what else?
> 
> We embed a bunch of other projects (libraries, mainly):
> - Oniguruma
> - Brotli
> - fdt
> - LZMA SDK
> - ...

Sure. But do we know that is what was meant?

I certainly recall the "each package should be a submodule" idea from
a (much) earlier conversation, and would like to ensure we're not
resurrecting that.

Regards,

Leif

Re: TianoCore Community Meeting Minutes

[Kinney, Michael D]

> -----Original Message-----
> From: edk2-devel [mailto:edk2-devel-bounces@lists.01.org]
> On Behalf Of Leif Lindholm
> Sent: Friday, October 12, 2018 11:06 AM
> To: Laszlo Ersek <lersek@redhat.com>
> Cc: edk2-devel@lists.01.org
> Subject: Re: [edk2] TianoCore Community Meeting Minutes
> 
> On Fri, Oct 12, 2018 at 06:07:01PM +0200, Laszlo Ersek
> wrote:
> > On 10/12/18 15:27, Leif Lindholm wrote:
> > > On Thu, Oct 11, 2018 at 10:43:57AM -0700, stephano
> wrote:
> >
> > >> Switching to Standard C Types
> > >> -----------------------------
> > >> Both Shawn and Nate mentioned that the current
> system has been in place for
> > >> a long time and some people prefer the current
> setup. I can start an email
> > >> discussion around this issue specifically if anyone
> feels strongly that we
> > >> should be using standard types.
> > >
> > > So, I don't think we made it this far down the agenda
> on the US-EU
> > > call.
> > >
> > > One way would be to simply explicitly permit it,
> possibly with the
> > > constraint that every module needs to pick one and
> stick with it,
> > > unless people object.
> > >
> > > I think we'll want to discuss this in a US-EU call as
> well.
> >
> > I'm playing devil's advocate here -- because, in
> general, I'm a fan of
> > sticking with standard C as much as possible --, but I
> see a big
> > obstacle in the way.
> >
> > That obstacle is "Table 5. Common UEFI Data Types", in
> the UEFI spec.
> > Until a good portion of that table is expressed in
> terms of standard C
> > types as well (expanding upon the current definitions),
> possibly in an
> > edk2-level spec (i.e. not necessarily in the UEFI spec
> itself), I think
> > there's no chance to enable standard C types in edk2
> *meaningfully*.
> >
> > Because, as soon as you have to call a PI or UEFI
> interface, you'll have
> > to stick with the PI/UEFI spec types anyway.
> 
> I don't necessarily see that as an issue. But it is a
> good point that
> it can't just be the codebase changing.
> 
> Since we are however extremely specificly not looking to
> change the
> underlying storage types, all it would take would be to
> make a
> 2-column table into a 3-column table in both specs. Or
> just add a
> separate table for the mapping. Then edk2 could adopt the
> "permitted"
> rule as soon as the specs were out. Arguably (because
> we're not
> changing underlying types) we could do it before, but we
> _are_
> supposed to be the reference implementation, so it would
> be poor form.

I agree that it would be best if the specs list synonymous
type names.  We would have to guarantee in the edk2 implementation
that they types are identical.  One potential issue is support
for really old compilers.  If we can decide to only support
compilers that fully support the synonymous types, then that
would be clean.  Not sure what the ANSI C equivalents are for
INTN/UINTN on all compilers.

> 
> > >> Using Git Submodules (like we do with OpenSSL)
> > >> --------------------
> > >
> > > We didn't make it here either. What would we use it
> _for_?
> > > I think the openssl case makes a lot of sense, but
> what else?
> >
> > We embed a bunch of other projects (libraries, mainly):
> > - Oniguruma
> > - Brotli
> > - fdt
> > - LZMA SDK
> > - ...
> 
> Sure. But do we know that is what was meant?
> 
> I certainly recall the "each package should be a
> submodule" idea from
> a (much) earlier conversation, and would like to ensure
> we're not
> resurrecting that.

Yes.  Those other projects was the brief discussion.
Not submodule per package.

> 
> Regards,
> 
> Leif
> _______________________________________________
> edk2-devel mailing list
> edk2-devel@lists.01.org
> https://lists.01.org/mailman/listinfo/edk2-devel

Re: TianoCore Community Meeting Minutes

[Andrew Fish]

> On Oct 12, 2018, at 11:30 AM, Kinney, Michael D <michael.d.kinney@intel.com> wrote:
> 
>> -----Original Message-----
>> From: edk2-devel [mailto:edk2-devel-bounces@lists.01.org <mailto:edk2-devel-bounces@lists.01.org>]
>> On Behalf Of Leif Lindholm
>> Sent: Friday, October 12, 2018 11:06 AM
>> To: Laszlo Ersek <lersek@redhat.com <mailto:lersek@redhat.com>>
>> Cc: edk2-devel@lists.01.org <mailto:edk2-devel@lists.01.org>
>> Subject: Re: [edk2] TianoCore Community Meeting Minutes
>> 
>> On Fri, Oct 12, 2018 at 06:07:01PM +0200, Laszlo Ersek
>> wrote:
>>> On 10/12/18 15:27, Leif Lindholm wrote:
>>>> On Thu, Oct 11, 2018 at 10:43:57AM -0700, stephano
>> wrote:
>>> 
>>>>> Switching to Standard C Types
>>>>> -----------------------------
>>>>> Both Shawn and Nate mentioned that the current
>> system has been in place for
>>>>> a long time and some people prefer the current
>> setup. I can start an email
>>>>> discussion around this issue specifically if anyone
>> feels strongly that we
>>>>> should be using standard types.
>>>> 
>>>> So, I don't think we made it this far down the agenda
>> on the US-EU
>>>> call.
>>>> 
>>>> One way would be to simply explicitly permit it,
>> possibly with the
>>>> constraint that every module needs to pick one and
>> stick with it,
>>>> unless people object.
>>>> 
>>>> I think we'll want to discuss this in a US-EU call as
>> well.
>>> 
>>> I'm playing devil's advocate here -- because, in
>> general, I'm a fan of
>>> sticking with standard C as much as possible --, but I
>> see a big
>>> obstacle in the way.
>>> 
>>> That obstacle is "Table 5. Common UEFI Data Types", in
>> the UEFI spec.
>>> Until a good portion of that table is expressed in
>> terms of standard C
>>> types as well (expanding upon the current definitions),
>> possibly in an
>>> edk2-level spec (i.e. not necessarily in the UEFI spec
>> itself), I think
>>> there's no chance to enable standard C types in edk2
>> *meaningfully*.
>>> 
>>> Because, as soon as you have to call a PI or UEFI
>> interface, you'll have
>>> to stick with the PI/UEFI spec types anyway.
>> 
>> I don't necessarily see that as an issue. But it is a
>> good point that
>> it can't just be the codebase changing.
>> 
>> Since we are however extremely specificly not looking to
>> change the
>> underlying storage types, all it would take would be to
>> make a
>> 2-column table into a 3-column table in both specs. Or
>> just add a
>> separate table for the mapping. Then edk2 could adopt the
>> "permitted"
>> rule as soon as the specs were out. Arguably (because
>> we're not
>> changing underlying types) we could do it before, but we
>> _are_
>> supposed to be the reference implementation, so it would
>> be poor form.
> 
> I agree that it would be best if the specs list synonymous
> type names.  We would have to guarantee in the edk2 implementation
> that they types are identical.  One potential issue is support
> for really old compilers.  If we can decide to only support
> compilers that fully support the synonymous types, then that
> would be clean.  Not sure what the ANSI C equivalents are for
> INTN/UINTN on all compilers.
> 

Mike,

INTN -> intptr_t
UINTN -> uintptr_t

If I understand correctly the types we are talking about are defined via stdint.h. Thus if we are a freestanding (__STDC_HOSTED__ == 0) project and we did not include stdint.h we would need to define them in the edk2 headers? 

If the edk2 App/Driver is also including stdint.h we would need some way to avoid conflicts. Likely including stdint.h from ProcessorBind.h in place of defining the values. 


Thanks,

Andrew Fish

>> 
>>>>> Using Git Submodules (like we do with OpenSSL)
>>>>> --------------------
>>>> 
>>>> We didn't make it here either. What would we use it
>> _for_?
>>>> I think the openssl case makes a lot of sense, but
>> what else?
>>> 
>>> We embed a bunch of other projects (libraries, mainly):
>>> - Oniguruma
>>> - Brotli
>>> - fdt
>>> - LZMA SDK
>>> - ...
>> 
>> Sure. But do we know that is what was meant?
>> 
>> I certainly recall the "each package should be a
>> submodule" idea from
>> a (much) earlier conversation, and would like to ensure
>> we're not
>> resurrecting that.
> 
> Yes.  Those other projects was the brief discussion.
> Not submodule per package.
> 
>> 
>> Regards,
>> 
>> Leif
>> _______________________________________________
>> edk2-devel mailing list
>> edk2-devel@lists.01.org <mailto:edk2-devel@lists.01.org>
>> https://lists.01.org/mailman/listinfo/edk2-devel <https://lists.01.org/mailman/listinfo/edk2-devel>
> _______________________________________________
> edk2-devel mailing list
> edk2-devel@lists.01.org <mailto:edk2-devel@lists.01.org>
> https://lists.01.org/mailman/listinfo/edk2-devel <https://lists.01.org/mailman/listinfo/edk2-devel>

Re: TianoCore Community Meeting Minutes

[Andrew Fish]

> On Oct 12, 2018, at 9:07 AM, Laszlo Ersek <lersek@redhat.com> wrote:
> 
> On 10/12/18 15:27, Leif Lindholm wrote:
>> On Thu, Oct 11, 2018 at 10:43:57AM -0700, stephano wrote:
> 
>>> Switching to Standard C Types
>>> -----------------------------
>>> Both Shawn and Nate mentioned that the current system has been in place for
>>> a long time and some people prefer the current setup. I can start an email
>>> discussion around this issue specifically if anyone feels strongly that we
>>> should be using standard types.
>> 
>> So, I don't think we made it this far down the agenda on the US-EU
>> call.
>> 
>> One way would be to simply explicitly permit it, possibly with the
>> constraint that every module needs to pick one and stick with it,
>> unless people object.
>> 
>> I think we'll want to discuss this in a US-EU call as well.
> 
> I'm playing devil's advocate here -- because, in general, I'm a fan of
> sticking with standard C as much as possible --, but I see a big
> obstacle in the way.
> 
> That obstacle is "Table 5. Common UEFI Data Types", in the UEFI spec.
> Until a good portion of that table is expressed in terms of standard C
> types as well (expanding upon the current definitions), possibly in an
> edk2-level spec (i.e. not necessarily in the UEFI spec itself), I think
> there's no chance to enable standard C types in edk2 *meaningfully*.
> 
> Because, as soon as you have to call a PI or UEFI interface, you'll have
> to stick with the PI/UEFI spec types anyway.
> 

Lazlo,

Given there is also a C ABI for each supported processor architecture in the UEFI spec it should be possible to define the EFI types in terms of C types. The only potential issue is I'm not sure BOOLEAN maps to bool in all cases? 

https://github.com/tianocore/edk2/blob/master/MdePkg/Include/X64/ProcessorBind.h#L188
So
typedef __int64    UINT64; 
or
typedef long long INT64;

becomes:
typedef int64_t    INT64;

Thus we could move the definition of UINT64, INT64, UINT32, INT32, UINT16, INT16, UINT8, INT8, CHAR, and CHAR16 to Base.h and have ProcessorBind.h just define the C standard types for a given architecture. 

The tricky part is when there are different answers to the question are you using a standard C lib. Basically the definitions in MdePkg/Include/X64/ProcessorBind.h could conflict with stdint.h. Almost seems like you would need a build flag to control if you use stdint.h. 


I think the bigger question is what problem are we trying to solve? If some one is programming with uint32_t are they going to expect printf(), memcpy(), etc. We solve that problem today StdLib? I guess we only have an option of the full pedantic C lib (I can't remember if StdLib depends on the shell?), or nothing. Are we trying to define a light weight C lib that works in the firmware code? How much of the C lib do we need to make that useful? 

Thanks,

Andrew Fish

PS Speaking of printf() != Print() or DEBUG()... I see a lot of people botching Print() in EFI. It would nice if we could get compiler warnings for miss use.

You can add printf warnings to any functing in 
int 
my_printf (void *my_object, const char *my_format, ...)   __attribute__ ((format (printf, 2, 3)));

It would be awesome if we could add edk2_print to at least gcc and clang. Given they are open source projects anything is possible. Not sure how this works on VC++?

> 
>>> Using Git Submodules (like we do with OpenSSL)
>>> --------------------
>> 
>> We didn't make it here either. What would we use it _for_?
>> I think the openssl case makes a lot of sense, but what else?
> 
> We embed a bunch of other projects (libraries, mainly):
> - Oniguruma
> - Brotli
> - fdt
> - LZMA SDK
> - ...
> 
> Thanks
> Laszlo
> _______________________________________________
> edk2-devel mailing list
> edk2-devel@lists.01.org
> https://lists.01.org/mailman/listinfo/edk2-devel

Re: TianoCore Community Meeting Minutes

[stephano]

On 10/12/2018 6:27 AM, Leif Lindholm wrote:> I know at least one 
organisation that would be prevented from using it
> until they had explicit approval (which could take a while), and it
> puts me in the position of having to keep a spare x86 machine on my
> desktop in order to participate.
> 
> Solutions I know would work in this scenario (i.e. work without
> plugins or downloads in any modern browser) are:
> - google hangouts (requires accounts, not ideal for China)
> - Skype (not SFB though, only plain old Skype)
> - Bluejeans (requires organiser account)

We needed an organizer account for Zoom since our meeting was 1 hour 
long. So switching to something paid is not an issue.

I've used Bluejeans with success in the past. I'll look into switching 
to this platform if no one else has any objections. I'd like to stay 
away from my other preferred google hangouts because so many of our 
members are in PRC.

>>
>> The was a concern raised over potential lock-in to Github's, specifically in
>> regards to history retention.
>> Several Github users brought up that this shouldn't be an issue.
> 
> Hopefully they said more than that.
> What does "shouldn't be an issue" mean.
> Were these users from multiple organisations?

I will ask for more details in the next meeting, but the comments were 
that the API is quite robust and lends itself to readily accessible data.

> 
>> Shawn mentioned some benefits to stock Github such as
>> it is always up to date, it includes APIs to extract data, pull
>> requests
> 
> Since we are discussing multiple different development systems, can we
> try to be a bit more explicit? This is referring to github's web-based
> branch-based ticketing system, yes?

Yes that's correct.

> 
> I know language drift and all, but
> https://www.kernel.org/doc/html/latest/maintainer/pull-requests.html
> is what pull request means to at least 3 participants in this conversation.

I think this is an important distinction to make and I'm sure it will 
come up in future discussions. I'll be sure to bring this up.

> 
>> lend themselves to simple(r) automation, and it has a much more modern web
>> UI than Gerrit.
> 
> It seems somewhat less than ideal to me that all of the github
> proponents were on the opposite call to me and Laszlo (and Ard). Were
> any of them Asia-based or could we try to get them on the call with
> Europe next time? I'm sure me and Laszlo could be somewhat more
> accommodating than your 7AM, but we're not going to stay up for a 3/4AM
> meeting about source control.

That's understandable. I'm not sure where Shawn and Nate are located, 
but I believe it is in the US. We could certainly move the AM call a bit 
later if it makes it easier to get more folks from the US on that call.

>> -How can we better track the code quality of BaseTools and the current build
>> system on the whole. Should we add a "requires documentation change" flag to
>> BZ so that it will be easier to compile a list of required doc changes.
> 
> Could we also have a thread on someting that you sort of mention in
> the text, but not the bullet points?:
> - Suggestions for improvements to BaseTools.

Of course. I'll add this to the list of "community conversations" that I 
will start up on the list this week.

Cheers,
Stephano

Re: TianoCore Community Meeting Minutes

[Andrew Fish]

> On Oct 11, 2018, at 10:43 AM, stephano <stephano.cetola@linux.intel.com> wrote:
> 
> Thank you all for a great set of meetings!
> 
> This is an overview of the topics discussed and the tasks that were assigned. Please feel free to send me any questions or comments.
> 

...

> Improvements to the Build Process
> --------------------------------
> We would like to gather a list of concrete specific proposals. Nate mentioned that there will always be some specialized tooling because of the nature of BIOS. Shawn mentioned that we need to keep in mind that making development easier for current developers should be a priority over making the processes easier for newcomers. I will send out emails asking for specific feedback on some of these topics:
> 
> -Which toolchains are being used, which are validated, and which are known to create reproducible builds.
> -When toolchains are known to be orphaned, should they be archived or simply removed.
> -Could we add some kconfig-like tool that allows introspection into what type of builds are available.
> -How can we better track the code quality of BaseTools and the current build system on the whole. Should we add a "requires documentation change" flag to BZ so that it will be easier to compile a list of required doc changes.
> 


Stephano,

I'll help start the conversation on the build. 

1) Making the process easier for newcomers. 

a) Getting it to build the 1st time
I think we could aspire to a more Unix like flow:
git clone
./configure
make ovmf
make ovmf.run

The ./config step would automate resolving dependencies. Things like figuring out what compiler is installed, what extra tools need to be installed etc. It could be interactive?

make <target> would build all the build tools, and do what ever local config is required to get the build working. Then build the requested target. 

make <target>.run is more of an optional step. It could do things like launch Ovmf with your new ROM you just constructed. 

b) Debugging failures. 
Better quality error messages. 
Making the build.log from the build command a default setting(or have an abstracted defined database that can be queried), and add documentation on how to use it. Maybe make the output easier to consume?

c) Making changes. 
That can be a phase 2.

2) Build performance, complexity, and maintainability. 
I'm about to complain about the build system, but I was around when it was first invented so I'm to blame as much as anyone....

We have way too much custom Python that is very complex, hard to maintain, and slow. One thing we did better on the EDK was break up the build tools into smaller chunks (maybe too small in some cases) and I always found I could figure things out, but then again all the code was C. Another interesting data point is we added parallel build, via makefiles, to the EDK. When we moved from the EDK to the edk2 we noticed the builds slowed down a lot. I did some profiling on the edk2 build and I noticed 1,000,000 calls to regex (part of that long  ..... sequence on a build of a complex platform). It turns out we have custom Python code that is generating the makefile dependencies so it was scanning all the C code. The other issue I see is build parallelism is controlled from the Python code. It also seems the edk2 build system is not a complete solution and a lot of full platform implementations have shell scripts or batch files, that wrap the calls to build. 

So I'd like to see a build system:
1) Start with a top level makefile in the root of the tree. 
2) Replaces custom batch or shell scripts with makefiles to handle pre and post build tasks. 
3) Use a tool to construct a parser for the ekd2 build files (INF, DSC, DEC). 
	Use this data to construct the makefiles and build a database for other build components to use.
	Chris Roberts has messed around and built an Earley parser using the EBNF in the build documents. There are some problematic issues in the grammar, but some grammar streamlining might make it possible to use tooling to construct a parser vs. writing one from scratch in Python. 
4) Move to using the compiler to generate dependencies. gcc/clang support this via -M*. I think it may be a little more complex for NMAKE but there is a /showIncludes compiler flag. 
5) Have the build tool terminate after parallel makefile generation
6) Top level makefile can invoke parallel build on generated makefiles. 
7) Invoke build tool to generate FVs and FDs from makefile. 
8) Invoke custom post processing steps in the makefile. 

Thanks,

Andrew Fish