Re: [Patch v4 03/22] requirements.txt: Add python pip requirements file

["Michael D Kinney" <michael.d.kinney@intel.com>]

Laszlo,

Thank you for the extra effort on this evaluation.

I will change to pip-requirements.txt

Mike

> -----Original Message-----
> From: Laszlo Ersek <lersek@redhat.com>
> Sent: Friday, November 8, 2019 5:13 AM
> To: Kinney, Michael D <michael.d.kinney@intel.com>;
> devel@edk2.groups.io
> Cc: Sean Brogan <sean.brogan@microsoft.com>; Andrew Fish
> <afish@apple.com>; Leif Lindholm
> <leif.lindholm@linaro.org>
> Subject: Re: [Patch v4 03/22] requirements.txt: Add
> python pip requirements file
> 
> Hi Mike,
> 
> On 11/07/19 18:44, Kinney, Michael D wrote:
> > Hi Laszlo,
> >
> > requirements.txt is not only for CI.
> >
> > If a developer wants to run the same tests that CI
> runs locally that
> > also need to install using pip and need  this file.
> >
> > Sean responded to this feedback earlier and pointed to
> some features
> > that may depend on this specific filename or a
> specific filename
> > pattern.
> >
> > https://edk2.groups.io/g/devel/message/49620
> >
> > I agree that the pip command supports using a
> different filename.  I
> > considered several options:
> >
> > 1) Keep current requirements.txt in root
> > 2) Change to pip_requirements.txt in root
> > 3) Change to requirements.txt in the .pytool directory
> >
> > I set (3) aside because the use of the python
> extensions installed
> > using pip are not limited to content in the .pytool
> directory.  There
> > is new content in BaseTools as well that depends on
> the pip install
> > components.  The  root directory is the only common
> parent directory.
> >
> > Given the feedback that there may be some services
> that look for
> > requirements.txt, I thought it would be better to
> leave the filename
> > alone and add the file header comment block with a
> clear description
> > of the file.
> >
> > With this additional context, if there is still
> feedback that the
> > filename must be changed, then I would recommend a
> filename change
> > that also follows camel case in the root.
> >
> >     PipRequirements.txt
> 
> Thanks for all this information.
> 
> It looks like "requirements.txt" is a *github.com*
> feature. It is something that the (closed source,
> server-side) <github.com> software looks at, to
> establish inter-repo (inter-project) dependencies. With
> these dependencies parsed automatically, <github.com>
> can offer various features. Such as:
> 
>   [1] https://github.blog/2018-07-12-security-
> vulnerability-alerts-for-python/
>   [2] https://help.github.com/en/github/visualizing-
> repository-data-with-graphs/listing-the-packages-that-a-
> repository-depends-on
> 
> (This is at least what I gather from the links inside
> Sean's message that you reference above.)
> 
> Therefore, in my opinion, we should look for a solution
> (filename) that satisfies both goals below:
> 
> - keep the github.com integration happy and functional
> (with regard to
>   the above-linked features),
> 
> - use a filename that does not imply "requirements" for
> the strictly
>   defined edk2 project itself.
> 
> While I agree "PipRequirements.txt" looks native to
> edk2, I think that would break the first goal --
> github.com would likely not recognize it.
> 
> However, there are signs that "pip-requirements.txt" is
> recognized by github. We can test this theory as
> follows:
> 
> - with google or another search engine, look for some
> repositories --
>   any repositories really -- on <github.com> that use
>   "pip-requirements.txt" rather than "requirements.txt",
> 
> - check whether the <github.com> feature marked with [2]
> above *works*
>   for those projects.
> 
> Now, here are three -- basically randomly chosen --
> repositories on <github.com> that contain "pip-
> requirements.txt" and *no*
> "requirements.txt":
> 
> - https://github.com/datagovuk/ckanext-dgu        @
> cb17b9e
> - https://github.com/ClearingHouse/clearinghoused @
> f85881f
> - https://github.com/jalajthanaki/NLPython        @
> 47e6861
> 
> In order to subject them to the
> 
>   does <github.com> recognize "pip-requirements.txt"?
> 
> test, we should substitute their organization names and
> project names in the following URL pattern, taken from
> Sean's email:
> 
>   https://github.com/tianocore/edk2-pytool-
> extensions/network/dependencies
>                      ^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^
>                      ORG_NAME  PROJECT_NAME
> 
> That makes for:
> 
> - https://github.com/datagovuk/ckanext-
> dgu/network/dependencies
> -
> https://github.com/ClearingHouse/clearinghoused/network/
> dependencies
> -
> https://github.com/jalajthanaki/NLPython/network/depende
> ncies
> 
> All three links work; and in each page, there is a
> section called
> 
>   Dependencies defined in pip-requirements.txt
> 
> Therefore, I claim that <github.com> recognizes "pip-
> requirements.txt"
> too, not just "requirements.txt".
> 
> ----*----
> 
> So how does that apply to us? It seems like the
> "dependencies" insight, using the URL format
> 
> 
> https://github.com/ORG_NAME/PROJECT_NAME/network/depende
> ncies
> 
> *only* considers the master branch (more precisely, the
> "default"
> branch) of a repository.
> 
> Therefore, in order to test the viability of "pip-
> requirements.txt":
> 
> - We need to create two new forks of the edk2-staging
> project on
>   <github.com>.
> 
> - In one of the forks (let's call it F1), we need to
> push the current
>   "edk2-staging/edk2-ci" branch as F1's "master" (or
> maybe "about")
>   branch, containing "requirements.txt".
> 
> - In the other fork (let's call it F2), we need to push
> the current
>   "edk2-staging/edk2-ci" branch as F2's "master" (or
> perhaps "about")
>   branch, containing "pip-requirements.txt".
> 
> - Finally, we must compare (visually) the following two
> links:
> 
>   https://github.com/ORG_NAME/F1/network/dependencies
>   https://github.com/ORG_NAME/F2/network/dependencies
> 
> and ascertain the dependencies that are parsed (from
> "requirements.txt"
> vs. "pip-requirements.txt") are identical.
> 
> Now... Obviously, I wanted to perform this test myself.
> For creating "F1" and "F2", I wanted to fork edk2-
> staging twice more, under my account.
> 
> Unfortunately, in its *infinite wisdom*, <github.com>
> does not allow me to fork the edk2-staging project at
> all, at this point (not even under different names),
> because I already have a fork of edk2-staging. Sigh.
> 
> So here's what I managed to do. In my current edk2-
> staging fork, I force-pushed the "edk2-ci" branch *as*
> both the "master" and "about"
> branches, at commit 5f901d3f96e5 ("Readme.md: Fix link
> to pytool Readme.md", 2019-11-02). Then I went to
> 
>   https://github.com/lersek/edk2-
> staging/network/dependencies
> 
> and took a screenshot.
> 
> Then, I renamed "requirements.txt" to "pip-
> requirements.txt":
> 
> > commit a55d524ab200593f2a907662dce2260df86810fa (HEAD
> -> req-txt)
> > Author: Laszlo Ersek <lersek@redhat.com>
> > Date:   Fri Nov 8 13:50:20 2019 +0100
> >
> >     rename "requirements.txt" to "pip-
> requirements.txt"
> >
> >     Signed-off-by: Laszlo Ersek <lersek@redhat.com>
> >
> > diff --git a/requirements.txt b/pip-requirements.txt
> similarity index
> > 100% rename from requirements.txt rename to pip-
> requirements.txt
> 
> pushed the change (again to both the "master" branch and
> the "about"
> branch), and took another screenshot, after reloading
> the URL
> 
>   https://github.com/lersek/edk2-
> staging/network/dependencies
> 
> Please find both screenshots attached.
> 
> Therefore I (again) claim that <github.com> honors "pip-
> requirements.txt" equally, and therefore we should use
> the "pip-requirements.txt" filename.
> 
> Thanks!
> Laszlo