edk2setup.sh shortcomings

edk2setup.sh shortcomings

[tlaronde]

edk2setup.sh has shortcomings. To list some:

	- The functions return a status but it is not tested; hence the
	  script goes to the end with a final "return $?" that simply
	  returns the status of the last command that is "unset" which
	  always successfully unsets, even a not set variable. Hence a
	  script can not catch a failure by testing the end status that is
	  always 0;
	- If WORKSPACE is set, --reconfig does nothing;
	- If EDK_TOOLS_PATH and PACKAGES_PATH are set, even to incorrect
	  values, the script succeeds even if BaseTools/ is not found
	  anywhere;
	- The comments are obsolete (1): bash(1) is required because the syntax
	  is not POSIX.2 sh(1) compliant and because some Makefile recipes
	  have "bash'isms" (indeed, a GMAKE variable should be exported
	  with a definition of "/path/to/gnu/make SHELL=/path/to/bash" and
	  a canonical call should be "$GMAKE ...");
	- The comments are obsolete (2): CYGWIN is not treated in anyway
	  specifically and, on the contrary, the regexp translation of ':'
	  in spaces for PACKAGES_PATH would be sure to create a mess with
	  a MS Windows like path;
 	- The settings have obviously evolved and the help message does not
	  list all the variables that can be set and that do modify the
	  way the setting is done;
	- Some commands (notably whereis(1)) are not standard utilities, not
	  to be found on all Unix like systems and, even if found, have
	  greatly diverging behaviors.

What is the preferred procedure? Should I file BZ to list all the
problems so that someone authorized may address them? Or can I propose
a patch to address these (keeping it backward compatible with a present
correct use) with a reasonable hope that, as an exception that will not
become a rule, it will not be ignored?

Best,
-- 
        Thierry Laronde <tlaronde +AT+ polynum +dot+ com>
                     http://www.kergis.com/
                    http://kertex.kergis.com/
Key fingerprint = 0FF7 E906 FBAF FE95 FD89  250D 52B1 AE95 6006 F40C

Re: [edk2-devel] edk2setup.sh shortcomings

[Gerd Hoffmann]

On Thu, Feb 02, 2023 at 12:29:32PM +0100, tlaronde@polynum.com wrote:
> edk2setup.sh has shortcomings. To list some:
> 
> 	- The functions return a status but it is not tested; hence the
> 	  script goes to the end with a final "return $?" that simply
> 	  returns the status of the last command that is "unset" which
> 	  always successfully unsets, even a not set variable. Hence a
> 	  script can not catch a failure by testing the end status that is
> 	  always 0;
> 	- If WORKSPACE is set, --reconfig does nothing;
> 	- If EDK_TOOLS_PATH and PACKAGES_PATH are set, even to incorrect
> 	  values, the script succeeds even if BaseTools/ is not found
> 	  anywhere;
> 	- The comments are obsolete (1): bash(1) is required because the syntax
> 	  is not POSIX.2 sh(1) compliant and because some Makefile recipes
> 	  have "bash'isms" (indeed, a GMAKE variable should be exported
> 	  with a definition of "/path/to/gnu/make SHELL=/path/to/bash" and
> 	  a canonical call should be "$GMAKE ...");
> 	- The comments are obsolete (2): CYGWIN is not treated in anyway
> 	  specifically and, on the contrary, the regexp translation of ':'
> 	  in spaces for PACKAGES_PATH would be sure to create a mess with
> 	  a MS Windows like path;
>  	- The settings have obviously evolved and the help message does not
> 	  list all the variables that can be set and that do modify the
> 	  way the setting is done;
> 	- Some commands (notably whereis(1)) are not standard utilities, not
> 	  to be found on all Unix like systems and, even if found, have
> 	  greatly diverging behaviors.
> 
> What is the preferred procedure?

Ignore it and to just use BaseTools/BuildEnv directly?
I'm not fully sure what value it adds ...

> Should I file BZ to list all the
> problems so that someone authorized may address them? Or can I propose
> a patch to address these (keeping it backward compatible with a present
> correct use) with a reasonable hope that, as an exception that will not
> become a rule, it will not be ignored?

Sending patches has a much higher chance to succeed, although there is
no guarantee unfortunately.

I'd start with removing code:  The python handling it adds should be
obsolete, python2 is EOL and I think meanwhile the python tools require
python3 anyway.  So all that can probably replaced with "export
PYTHON_COMMAND=python3".

Leaves less code which needs actual fixing ;)

take care,
  Gerd

Re: [edk2-devel] edk2setup.sh shortcomings

[tlaronde]

Le Thu, Feb 02, 2023 at 05:50:32PM +0100, Gerd Hoffmann a écrit :
> On Thu, Feb 02, 2023 at 12:29:32PM +0100, tlaronde@polynum.com wrote:
> > edk2setup.sh has shortcomings. To list some:
> > 
> > 	- The functions return a status but it is not tested; hence the
> > 	  script goes to the end with a final "return $?" that simply
> > 	  returns the status of the last command that is "unset" which
> > 	  always successfully unsets, even a not set variable. Hence a
> > 	  script can not catch a failure by testing the end status that is
> > 	  always 0;
> > 	- If WORKSPACE is set, --reconfig does nothing;
> > 	- If EDK_TOOLS_PATH and PACKAGES_PATH are set, even to incorrect
> > 	  values, the script succeeds even if BaseTools/ is not found
> > 	  anywhere;
> > 	- The comments are obsolete (1): bash(1) is required because the syntax
> > 	  is not POSIX.2 sh(1) compliant and because some Makefile recipes
> > 	  have "bash'isms" (indeed, a GMAKE variable should be exported
> > 	  with a definition of "/path/to/gnu/make SHELL=/path/to/bash" and
> > 	  a canonical call should be "$GMAKE ...");
> > 	- The comments are obsolete (2): CYGWIN is not treated in anyway
> > 	  specifically and, on the contrary, the regexp translation of ':'
> > 	  in spaces for PACKAGES_PATH would be sure to create a mess with
> > 	  a MS Windows like path;
> >  	- The settings have obviously evolved and the help message does not
> > 	  list all the variables that can be set and that do modify the
> > 	  way the setting is done;
> > 	- Some commands (notably whereis(1)) are not standard utilities, not
> > 	  to be found on all Unix like systems and, even if found, have
> > 	  greatly diverging behaviors.
> > 
> > What is the preferred procedure?
> 
> Ignore it and to just use BaseTools/BuildEnv directly?
> I'm not fully sure what value it adds ...

That's the problem: it does not add much, but it adds some things and
the problem (for me) is to know if these are used or not:

1) It exports PYTHONHASHSEED=1 (needed?);
2) Undocumented features allow to use something else instead of
BaseTools/BuildEnv...

I also think that it would be far better to state clearly what is
needed (including with BaseTools/BuildEnv and the
like) and let builders simply set the correct environment with
defined variables being used afterwards (MAKE, SHELL, PYTHON and
so on).

To state, for example the Python major version required and that
everything depends on GNU make and bash(1) (hence the
/path/to/gnu/make SHELL=/path/to/bash invocation) and that
would be it.

But: who uses it (edk2setup.sh) and with what? This question, I can not
answer, obviously...

> 
> > Should I file BZ to list all the
> > problems so that someone authorized may address them? Or can I propose
> > a patch to address these (keeping it backward compatible with a present
> > correct use) with a reasonable hope that, as an exception that will not
> > become a rule, it will not be ignored?
> 
> Sending patches has a much higher chance to succeed, although there is
> no guarantee unfortunately.

Hum... There is a very lethal weapon actually in use: the pillow. I
already sent various patches and they are silently ignored...

If my contribution will be ignored as others have been till now, honesty
should be to clearly state: "we don't care and we won't care" so that
I can use my time trying to solve the problems I want to address at a 
different level than UEFI...

Best,
-- 
        Thierry Laronde <tlaronde +AT+ polynum +dot+ com>
                     http://www.kergis.com/
                    http://kertex.kergis.com/
Key fingerprint = 0FF7 E906 FBAF FE95 FD89  250D 52B1 AE95 6006 F40C

Re: [edk2-devel] edk2setup.sh shortcomings

[Brian J. Johnson]

 > 1) It exports PYTHONHASHSEED=1 (needed?);

Setting PYTHONHASHSEED causes python hashes to be iterated in a 
deterministic order.  The autogen tools use hashes internally, so 
setting PYTHONHASHSEED to a fixed value causes them to produce identical 
output each time they are run with identical input.  That helps avoid 
unnecessary rebuilds of code which hasn't actually changed.  It affects 
items like the order of dependencies in the generated makefiles.

Setting PYTHONHASHSEED knocked several minutes off of the rebuild time 
for one of our platforms.  Recommended.

Brian J. Johnson
HPE

-------- Original Message --------
From: tlaronde@polynum.com [mailto:tlaronde@polynum.com]
Sent: Thursday, February 2, 2023 at 11:59 AM
To: Gerd Hoffmann <kraxel@redhat.com>
Cc: devel@edk2.groups.io, Andrew Fish <afish@apple.com>, Leif Lindholm 
<quic_llindhol@quicinc.com>, Michael D Kinney <michael.d.kinney@intel.com>
Subject: [edk2-devel] edk2setup.sh shortcomings

Le Thu, Feb 02, 2023 at 05:50:32PM +0100, Gerd Hoffmann a écrit :
> On Thu, Feb 02, 2023 at 12:29:32PM +0100, tlaronde@polynum.com wrote:
>> edk2setup.sh has shortcomings. To list some:
>>
>> 	- The functions return a status but it is not tested; hence the
>> 	  script goes to the end with a final "return $?" that simply
>> 	  returns the status of the last command that is "unset" which
>> 	  always successfully unsets, even a not set variable. Hence a
>> 	  script can not catch a failure by testing the end status that is
>> 	  always 0;
>> 	- If WORKSPACE is set, --reconfig does nothing;
>> 	- If EDK_TOOLS_PATH and PACKAGES_PATH are set, even to incorrect
>> 	  values, the script succeeds even if BaseTools/ is not found
>> 	  anywhere;
>> 	- The comments are obsolete (1): bash(1) is required because the syntax
>> 	  is not POSIX.2 sh(1) compliant and because some Makefile recipes
>> 	  have "bash'isms" (indeed, a GMAKE variable should be exported
>> 	  with a definition of "/path/to/gnu/make SHELL=/path/to/bash" and
>> 	  a canonical call should be "$GMAKE ...");
>> 	- The comments are obsolete (2): CYGWIN is not treated in anyway
>> 	  specifically and, on the contrary, the regexp translation of ':'
>> 	  in spaces for PACKAGES_PATH would be sure to create a mess with
>> 	  a MS Windows like path;
>>   	- The settings have obviously evolved and the help message does not
>> 	  list all the variables that can be set and that do modify the
>> 	  way the setting is done;
>> 	- Some commands (notably whereis(1)) are not standard utilities, not
>> 	  to be found on all Unix like systems and, even if found, have
>> 	  greatly diverging behaviors.
>>
>> What is the preferred procedure?
>
> Ignore it and to just use BaseTools/BuildEnv directly?
> I'm not fully sure what value it adds ...

That's the problem: it does not add much, but it adds some things and
the problem (for me) is to know if these are used or not:

1) It exports PYTHONHASHSEED=1 (needed?);
2) Undocumented features allow to use something else instead of
BaseTools/BuildEnv...

I also think that it would be far better to state clearly what is
needed (including with BaseTools/BuildEnv and the
like) and let builders simply set the correct environment with
defined variables being used afterwards (MAKE, SHELL, PYTHON and
so on).

To state, for example the Python major version required and that
everything depends on GNU make and bash(1) (hence the
/path/to/gnu/make SHELL=/path/to/bash invocation) and that
would be it.

But: who uses it (edk2setup.sh) and with what? This question, I can not
answer, obviously...

>
>> Should I file BZ to list all the
>> problems so that someone authorized may address them? Or can I propose
>> a patch to address these (keeping it backward compatible with a present
>> correct use) with a reasonable hope that, as an exception that will not
>> become a rule, it will not be ignored?
>
> Sending patches has a much higher chance to succeed, although there is
> no guarantee unfortunately.

Hum... There is a very lethal weapon actually in use: the pillow. I
already sent various patches and they are silently ignored...

If my contribution will be ignored as others have been till now, honesty
should be to clearly state: "we don't care and we won't care" so that
I can use my time trying to solve the problems I want to address at a
different level than UEFI...

Best,

Re: [edk2-devel] edk2setup.sh shortcomings

[Rebecca Cran]

On 2/2/23 14:49, Brian J. Johnson wrote:

> Hum... There is a very lethal weapon actually in use: the pillow. I
> already sent various patches and they are silently ignored...
> 
> If my contribution will be ignored as others have been till now, honesty
> should be to clearly state: "we don't care and we won't care" so that
> I can use my time trying to solve the problems I want to address at a
> different level than UEFI...

I prefer to think that patches I've submitted are _missed_ rather than 
ignored. It's still frustrating to realize that patches I've sent 6 
months ago were never committed, even if they were reviewed.

I suspect part of the problem is that many people working on edk2 are in 
companies which use Outlook, which can make keeping track of high 
traffic mailing lists difficult.

And even in Thunderbird, if all edk2-devel emails go into a single 
folder it can be difficult to keep track of patches.

I'm hoping that moving to Github PRs will improve things, with queries 
like "is:open is:pr review-requested:@me" letting people easily find PRs 
that need attention - though the UI looks like it could use some work to 
allow people to save their queries etc.

-- 
Rebecca Cran

Re: [edk2-devel] edk2setup.sh shortcomings

[Marvin Häuser]

[-- Attachment #1: Type: text/plain, Size: 585 bytes --]

Getting a patch reviewed by one maintainer and having the rest not respond even after multiple(!) pings is not „missed“. Let‘s be realistic here.

When I was still actively subscribed to the list, I had all mails that CC‘d me go to a separate folder. If I had more patch traffic, I would have had two (patch and non-patch mails) or more folders. At least in theory, everyone can organise their inbox as they please.

The edk2 build system stack is pretty inconvenient at times. Voluntary contributions such as this should be proactively endorsed.

Best regards,
Marvin

[-- Attachment #2: Type: text/html, Size: 641 bytes --]

Re: [edk2-devel] edk2setup.sh shortcomings

[tlaronde]

Le Thu, Feb 02, 2023 at 04:06:13PM -0700, Rebecca Cran a écrit :
> On 2/2/23 14:49, Brian J. Johnson wrote:
> 

This above is misleading: this is not Brian J. Johnson who wrote what is
below: this is me (Thierry Laronde):

> > Hum... There is a very lethal weapon actually in use: the pillow. I
> > already sent various patches and they are silently ignored...
> > 
> > If my contribution will be ignored as others have been till now, honesty
> > should be to clearly state: "we don't care and we won't care" so that
> > I can use my time trying to solve the problems I want to address at a
> > different level than UEFI...
> 

And you wrote (Rebecca Cran):

> I prefer to think that patches I've submitted are _missed_ rather than
> ignored. It's still frustrating to realize that patches I've sent 6 months
> ago were never committed, even if they were reviewed.
> 
> I suspect part of the problem is that many people working on edk2 are in
> companies which use Outlook, which can make keeping track of high traffic
> mailing lists difficult.
> 
> And even in Thunderbird, if all edk2-devel emails go into a single folder it
> can be difficult to keep track of patches.
> 
> I'm hoping that moving to Github PRs will improve things, with queries like
> "is:open is:pr review-requested:@me" letting people easily find PRs that
> need attention - though the UI looks like it could use some work to allow
> people to save their queries etc.

Not to mention the fact that to have the obligation to send endlessly the same
patches does nothing to decrease the trafic on the mailing list.

I ask only for honesty: if this is open source but not open development,
or if this is restricted open development i.e. only to big firms providing either
hardware or OSes, this is no problem with me as long as it is stated.
-- 
        Thierry Laronde <tlaronde +AT+ polynum +dot+ com>
                     http://www.kergis.com/
                    http://kertex.kergis.com/
Key fingerprint = 0FF7 E906 FBAF FE95 FD89  250D 52B1 AE95 6006 F40C