Re: [edk2-devel] FW: Discussion: Basetools a separate repo

["Laszlo Ersek" <lersek@redhat.com>]

On 04/20/20 20:23, Sean Brogan wrote:
> Laszlo,
> 
> PIP vs Submodules:
> The issue with submodule and not using python packages (pip is really just a super convenient helper to install that consistently) is that it requires the paradigm of file system python path management.  Think of this as edk1 vs edk2 public includes (if you remember back that far), basically you just figure out the path from yourself to the file you want and add that.  It gives you no ability to apply consistent ways to access modules and in the end means you just have a folder with python scripts/modules in it.  It causes major pain for downstream consumers when/if there is ever refactoring (and we know there needs to be refactoring).  It leads to others not being able to extend or leverage these modules without introducing significant fragility.  
> 
> What Pip does is gives you a way to consistently install and access the python modules.  You can pip install from a local copy of the git repo or you can pip install a "release" using something like pypi.  Either way it doesn't change how consuming code accesses the python modules.  This is the value of pip in this case.   
> 
> If there is a strong desire from the developer community to have a workflow that avoids pip I believe a little documentation could handle that.  Pip is just a helper.  Python packages can be built from the source and installed.  I see no reason to recommend this as it requires numerous more commands but if avoiding pip is your goal, it is possible.  
> 
> More value for python package management (PIP):
> As we start looking at refactoring and potentially moving tools from C to python we can take advantage of package management to lower our maintenance burden.  For example Brotli has a pypi release and thus we wouldn't need to carry that code.  https://pypi.org/project/Brotli/
> 
> 
> VERSION / DEPENDENCY: 
> To minimize the dependency challenges and "bisectability" I would suggest we leverage the versioning capabilities within pip and repo tagging.  With versioning you have lots of options as you can lock to a specific version which requires an update each time or you can use some sort of floating version within the tuple of version (xx.yy.zz).  It also needs to be discussed if a release would be generated for every change in the python or only on an important change.  These two tools can make this pretty flexible.  
> 
> In your scenario of DEC or INF syntax change.  My first pass on workflow would be. 
> 1. Create the issue for basetools
> 2. Update basetools python
> 3. Write the unit test that shows it works as expected
> 4. Check in and make a release
> 5. Update edk2 pip-requirements.txt to require at least this new version.  This gives you the tracking necessary to align with the tools.
> 6. Use this new feature in the edk2 fw code. 
> 
> 
> TOOLS DEVELOPER WORKFLOWS:
> You mentioned you wanted to make sure a developer could modify source locally and test against local code.  This is covered by workflow C.  It is very easy to manage and is one of the reasons we are proposing this change.  

Please provide a step by step guide for the following use case:

- Edk2 at commit C1 uses a particular new BaseTools feature. However,
there is a regression in BaseTools, and an independent MdeModulePkg
module (unrelated to the new BaseTools feature) no longer builds.

- Edk2 at commit C0 works fine. At this point in time (i.e. when C0 was
made), the BaseTools feature / regression didn't exist yet.

- The edk2 clone is checked out at C1, and the breakage is witnessed.
The user wants to return to state C0, and get a functional build. The
user wants to revert BaseTools too, to the same version that was
"BaseTools master" at the time edk2 commit C0 was made.

- The user is not allowed to run commands as root, and they also don't
want to confuse other python programs they may have installed under
their $HOME.

- How *exactly* does the user determine the exact BaseTools version, in
retrospect, for when commit C0 was the HEAD of edk2 master? For example,
is it

  git show C0:pip-requirements.txt

?

- What are the *precise* commands the user needs to run, for moving from
state C1 to C0, covering both edk2 and BaseTools?

I know I can run "git checkout C0" in the edk2 tree, to return to that
point in the past. What are the exact commands I need to run for
synching BaseTools to that state?


If the C1->C0 transition requires (re)installing BaseTools to some
private directory under $HOME, using a "virtual environment" or some
such with pip, that's entirely fine. I'm happy to do that. I've just
never done it before, and I want to make it absolutely sure that "moving
(back) through time" like this is doable without interfering with either
system Python state, or the user's general ($HOME) Python state.

For reference, with a git submodule, the BaseTools "sync" command would be

  git submodule update --init --force

Thanks
Laszlo