Re: [Patch v4 03/22] requirements.txt: Add python pip requirements file

["Laszlo Ersek" <lersek@redhat.com>]

[-- Attachment #1: Type: text/plain, Size: 6853 bytes --]

Hi Mike,

On 11/07/19 18:44, Kinney, Michael D wrote:
> Hi Laszlo,
>
> requirements.txt is not only for CI.
>
> If a developer wants to run the same tests that CI runs locally that
> also need to install using pip and need  this file.
>
> Sean responded to this feedback earlier and pointed to  some features
> that may depend on this specific filename or a specific filename
> pattern.
>
> https://edk2.groups.io/g/devel/message/49620
>
> I agree that the pip command supports using a different filename.  I
> considered several options:
>
> 1) Keep current requirements.txt in root
> 2) Change to pip_requirements.txt in root
> 3) Change to requirements.txt in the .pytool directory
>
> I set (3) aside because the use of the python extensions installed
> using pip are not limited to content in the .pytool directory.  There
> is new content in BaseTools as well that depends on the pip install
> components.  The  root directory is the only common parent directory.
>
> Given the feedback that there may be some services that look for
> requirements.txt, I thought it would be better to leave the filename
> alone and add the file header comment block with a clear description
> of the file.
>
> With this additional context, if there is still feedback that the
> filename must be changed, then I would recommend a filename change
> that also follows camel case in the root.
>
>     PipRequirements.txt

Thanks for all this information.

It looks like "requirements.txt" is a *github.com* feature. It is
something that the (closed source, server-side) <github.com> software
looks at, to establish inter-repo (inter-project) dependencies. With
these dependencies parsed automatically, <github.com> can offer various
features. Such as:

  [1] https://github.blog/2018-07-12-security-vulnerability-alerts-for-python/
  [2] https://help.github.com/en/github/visualizing-repository-data-with-graphs/listing-the-packages-that-a-repository-depends-on

(This is at least what I gather from the links inside Sean's message
that you reference above.)

Therefore, in my opinion, we should look for a solution (filename) that
satisfies both goals below:

- keep the github.com integration happy and functional (with regard to
  the above-linked features),

- use a filename that does not imply "requirements" for the strictly
  defined edk2 project itself.

While I agree "PipRequirements.txt" looks native to edk2, I think that
would break the first goal -- github.com would likely not recognize it.

However, there are signs that "pip-requirements.txt" is recognized by
github. We can test this theory as follows:

- with google or another search engine, look for some repositories --
  any repositories really -- on <github.com> that use
  "pip-requirements.txt" rather than "requirements.txt",

- check whether the <github.com> feature marked with [2] above *works*
  for those projects.

Now, here are three -- basically randomly chosen -- repositories on
<github.com> that contain "pip-requirements.txt" and *no*
"requirements.txt":

- https://github.com/datagovuk/ckanext-dgu        @ cb17b9e
- https://github.com/ClearingHouse/clearinghoused @ f85881f
- https://github.com/jalajthanaki/NLPython        @ 47e6861

In order to subject them to the

  does <github.com> recognize "pip-requirements.txt"?

test, we should substitute their organization names and project names in
the following URL pattern, taken from Sean's email:

  https://github.com/tianocore/edk2-pytool-extensions/network/dependencies
                     ^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^
                     ORG_NAME  PROJECT_NAME

That makes for:

- https://github.com/datagovuk/ckanext-dgu/network/dependencies
- https://github.com/ClearingHouse/clearinghoused/network/dependencies
- https://github.com/jalajthanaki/NLPython/network/dependencies

All three links work; and in each page, there is a section called

  Dependencies defined in pip-requirements.txt

Therefore, I claim that <github.com> recognizes "pip-requirements.txt"
too, not just "requirements.txt".

----*----

So how does that apply to us? It seems like the "dependencies" insight,
using the URL format

  https://github.com/ORG_NAME/PROJECT_NAME/network/dependencies

*only* considers the master branch (more precisely, the "default"
branch) of a repository.

Therefore, in order to test the viability of "pip-requirements.txt":

- We need to create two new forks of the edk2-staging project on
  <github.com>.

- In one of the forks (let's call it F1), we need to push the current
  "edk2-staging/edk2-ci" branch as F1's "master" (or maybe "about")
  branch, containing "requirements.txt".

- In the other fork (let's call it F2), we need to push the current
  "edk2-staging/edk2-ci" branch as F2's "master" (or perhaps "about")
  branch, containing "pip-requirements.txt".

- Finally, we must compare (visually) the following two links:

  https://github.com/ORG_NAME/F1/network/dependencies
  https://github.com/ORG_NAME/F2/network/dependencies

and ascertain the dependencies that are parsed (from "requirements.txt"
vs. "pip-requirements.txt") are identical.

Now... Obviously, I wanted to perform this test myself. For creating
"F1" and "F2", I wanted to fork edk2-staging twice more, under my
account.

Unfortunately, in its *infinite wisdom*, <github.com> does not allow me
to fork the edk2-staging project at all, at this point (not even under
different names), because I already have a fork of edk2-staging. Sigh.

So here's what I managed to do. In my current edk2-staging fork, I
force-pushed the "edk2-ci" branch *as* both the "master" and "about"
branches, at commit 5f901d3f96e5 ("Readme.md: Fix link to pytool
Readme.md", 2019-11-02). Then I went to

  https://github.com/lersek/edk2-staging/network/dependencies

and took a screenshot.

Then, I renamed "requirements.txt" to "pip-requirements.txt":

> commit a55d524ab200593f2a907662dce2260df86810fa (HEAD -> req-txt)
> Author: Laszlo Ersek <lersek@redhat.com>
> Date:   Fri Nov 8 13:50:20 2019 +0100
>
>     rename "requirements.txt" to "pip-requirements.txt"
>
>     Signed-off-by: Laszlo Ersek <lersek@redhat.com>
>
> diff --git a/requirements.txt b/pip-requirements.txt
> similarity index 100%
> rename from requirements.txt
> rename to pip-requirements.txt

pushed the change (again to both the "master" branch and the "about"
branch), and took another screenshot, after reloading the URL

  https://github.com/lersek/edk2-staging/network/dependencies

Please find both screenshots attached.

Therefore I (again) claim that <github.com> honors
"pip-requirements.txt" equally, and therefore we should use the
"pip-requirements.txt" filename.

Thanks!
Laszlo

[-- Attachment #2: s1.png --]
[-- Type: image/png, Size: 55806 bytes --]

[-- Attachment #3: s2.png --]
[-- Type: image/png, Size: 56133 bytes --]