edksetup.sh: fix for non POSIX whereis(1)

edksetup.sh: fix for non POSIX whereis(1)

[tlaronde]

diff --git a/edksetup.sh b/edksetup.sh
index 06d2f041e6..46b295c430 100755
--- a/edksetup.sh
+++ b/edksetup.sh
@@ -105,6 +105,19 @@ function SetupEnv()
   fi
 }
 
+# whereis(1) is not a POSIX utility and, for example, its implementation
+# in NetBSD is different form the Linux one.
+#
+function whereis()
+{
+  (
+    IFS=:
+    for dir in $PATH; do
+      eval ls $dir/${1}* 2>/dev/null || true 
+    done
+  )
+}
+
 function SetupPython3()
 {
   if [ $origin_version ];then

signed-off-by: Thierry LARONDE <tlaronde@polynum.com>
-- 
        Thierry Laronde <tlaronde +AT+ polynum +dot+ com>
                     http://www.kergis.com/
                    http://kertex.kergis.com/
Key fingerprint = 0FF7 E906 FBAF FE95 FD89  250D 52B1 AE95 6006 F40C

Re: [edk2-devel] edksetup.sh: fix for non POSIX whereis(1)

[Pedro Falcato]

[-- Attachment #1: Type: text/plain, Size: 1131 bytes --]

On Mon, Nov 21, 2022 at 9:21 PM <tlaronde@polynum.com> wrote:

> diff --git a/edksetup.sh b/edksetup.sh
> index 06d2f041e6..46b295c430 100755
> --- a/edksetup.sh
> +++ b/edksetup.sh
> @@ -105,6 +105,19 @@ function SetupEnv()
>    fi
>  }
>
> +# whereis(1) is not a POSIX utility and, for example, its implementation
> +# in NetBSD is different form the Linux one.
> +#
> +function whereis()
> +{
> +  (
> +    IFS=:
> +    for dir in $PATH; do
> +      eval ls $dir/${1}* 2>/dev/null || true
> +    done
> +  )
> +}
>
Hi Thierry,

First of all, thanks for the patch! I had noticed this problem when running
edksetup.sh on a POSIX but not quite Linux system before.

I kind of dislike your solution. Does NetBSD ship /bin/which by default? I
think replacing whereis with "which -a" would be a lot better.
I don't think there's a 100% standard way to do this in POSIX, as which
isn't POSIX either, and your solution seems... hacky?

Also, please send patches in the standard git format (git commit -s + git
format-patch + git send-email with the proper CCs to the maintainers, see
the proper guides for more details).

Thanks,
Pedro

[-- Attachment #2: Type: text/html, Size: 1612 bytes --]

Re: [edk2-devel] edksetup.sh: fix for non POSIX whereis(1)

[tlaronde]

Hello Pedro,

Le Mon, Nov 21, 2022 at 10:22:21PM +0000, Pedro Falcato a écrit :
> On Mon, Nov 21, 2022 at 9:21 PM <tlaronde@polynum.com> wrote:
> 
> > diff --git a/edksetup.sh b/edksetup.sh
> > index 06d2f041e6..46b295c430 100755
> > --- a/edksetup.sh
> > +++ b/edksetup.sh
> > @@ -105,6 +105,19 @@ function SetupEnv()
> >    fi
> >  }
> >
> > +# whereis(1) is not a POSIX utility and, for example, its implementation
> > +# in NetBSD is different form the Linux one.
> > +#
> > +function whereis()
> > +{
> > +  (
> > +    IFS=:
> > +    for dir in $PATH; do
> > +      eval ls $dir/${1}* 2>/dev/null || true
> > +    done
> > +  )
> > +}
> >
> Hi Thierry,
> 
> First of all, thanks for the patch! I had noticed this problem when running
> edksetup.sh on a POSIX but not quite Linux system before.
> 
> I kind of dislike your solution. Does NetBSD ship /bin/which by default? I
> think replacing whereis with "which -a" would be a lot better.
> I don't think there's a 100% standard way to do this in POSIX, as which
> isn't POSIX either, and your solution seems... hacky?

The problem is that which -a does not do what is required by the usage
of whereis(1) in the script and what is indeed implemented in Linux whereis(1) (and not in the obsolescent
NetBSD whereis(1)): look for "string*". "which -a python" or "which -a
'python*'" will return nothing.

This is why I ended up with this solution that uses only POSIX sh(1)
features and that is, if I'm not mistaken, an implementation of what is
used.

Best,
-- 
        Thierry Laronde <tlaronde +AT+ polynum +dot+ com>
                     http://www.kergis.com/
                    http://kertex.kergis.com/
Key fingerprint = 0FF7 E906 FBAF FE95 FD89  250D 52B1 AE95 6006 F40C

Re: [edk2-devel] edksetup.sh: fix for non POSIX whereis(1)

[Rebecca Cran]

On 11/21/22 15:22, Pedro Falcato wrote:

> I kind of dislike your solution. Does NetBSD ship /bin/which by default? 
> I think replacing whereis with "which -a" would be a lot better.
> I don't think there's a 100% standard way to do this in POSIX, as which 
> isn't POSIX either, and your solution seems... hacky?


"command" seems to be the POSIX way to do this?
https://pubs.opengroup.org/onlinepubs/9699919799/

Though "whereis python3" shows the following on my system:

python3: /usr/bin/python3.9-config /usr/bin/python3.9 /usr/bin/python3 
/usr/lib/python3.9 /usr/lib/python3 /etc/python3.9 /etc/python3 
/usr/local/lib/python3.9 /usr/include/python3.9 /usr/share/python3 
/usr/share/man/man1/python3.1.gz

"which -a python3" returns:

/usr/bin/python3
/bin/python3

And "command -p -v" returns:

/bin/python3

I don't know if we need all the results from "whereis"?

-- 
Rebecca Cran

Re: [edk2-devel] edksetup.sh: fix for non POSIX whereis(1)

[Pedro Falcato]

[-- Attachment #1: Type: text/plain, Size: 1433 bytes --]

On Tue, Nov 22, 2022 at 3:40 PM Rebecca Cran <quic_rcran@quicinc.com> wrote:

> On 11/21/22 15:22, Pedro Falcato wrote:
>
> > I kind of dislike your solution. Does NetBSD ship /bin/which by default?
> > I think replacing whereis with "which -a" would be a lot better.
> > I don't think there's a 100% standard way to do this in POSIX, as which
> > isn't POSIX either, and your solution seems... hacky?
>
>
> "command" seems to be the POSIX way to do this?
> https://pubs.opengroup.org/onlinepubs/9699919799/
>
> Though "whereis python3" shows the following on my system:
>
> python3: /usr/bin/python3.9-config /usr/bin/python3.9 /usr/bin/python3
> /usr/lib/python3.9 /usr/lib/python3 /etc/python3.9 /etc/python3
> /usr/local/lib/python3.9 /usr/include/python3.9 /usr/share/python3
> /usr/share/man/man1/python3.1.gz
>
> "which -a python3" returns:
>
> /usr/bin/python3
> /bin/python3
>
> And "command -p -v" returns:
>
> /bin/python3
>
> I don't know if we need all the results from "whereis"?
>
> --
> Rebecca Cran
>

I guess we could just use /bin/python3 (as in command -v python3) and
readlink to find the actual version?

In my local system "command -v python3" returns /usr/bin/python3
"readlink /usr/bin/python3" returns python3.10, which we could promptly
parse into a version?

I don't know how portable this is, but it's an idea. I definitely don't see
the reason to iterate through every possible python3* in PATH.

Pedro

[-- Attachment #2: Type: text/html, Size: 2180 bytes --]

Re: [edk2-devel] edksetup.sh: fix for non POSIX whereis(1)

[tlaronde]

Le Tue, Nov 22, 2022 at 08:40:30AM -0700, Rebecca Cran a écrit :
> On 11/21/22 15:22, Pedro Falcato wrote:
> 
> > I kind of dislike your solution. Does NetBSD ship /bin/which by default?
> > I think replacing whereis with "which -a" would be a lot better.
> > I don't think there's a 100% standard way to do this in POSIX, as which
> > isn't POSIX either, and your solution seems... hacky?
> 
> 
> "command" seems to be the POSIX way to do this?
> https://pubs.opengroup.org/onlinepubs/9699919799/
> 
> Though "whereis python3" shows the following on my system:
> 
> python3: /usr/bin/python3.9-config /usr/bin/python3.9 /usr/bin/python3
> /usr/lib/python3.9 /usr/lib/python3 /etc/python3.9 /etc/python3
> /usr/local/lib/python3.9 /usr/include/python3.9 /usr/share/python3
> /usr/share/man/man1/python3.1.gz
> 
> "which -a python3" returns:
> 
> /usr/bin/python3
> /bin/python3
> 
> And "command -p -v" returns:
> 
> /bin/python3
> 
> I don't know if we need all the results from "whereis"?

The problem is when one does not know which exact version of python
is here and how, exactly, the command is named.

whereis(1) returns whatever command with python3 as prefix, while
command or "which -a" will return only exactly python3. If, on the
system, python is fully version qualified (as is the case with
pkgsrc, the opt packages framework, used on NetBSD):

which -a python3

will return nothing, since, it is python3.9 for example on the OS.


There is a way to manipulate the namespace, since edk2setup.sh sets
environment and looks, by default, in some directories, but I have
respected the logics here: since whereis(1) is used to find a python3
related command in the PATH, even if it is named python3.x, the simplest
is to provide a basic whereis implementation in the script itself. Or,
changing the behavior, will perhaps break existing installation handling
the builds.
-- 
        Thierry Laronde <tlaronde +AT+ polynum +dot+ com>
                     http://www.kergis.com/
                    http://kertex.kergis.com/
Key fingerprint = 0FF7 E906 FBAF FE95 FD89  250D 52B1 AE95 6006 F40C

Re: [edk2-devel] edksetup.sh: fix for non POSIX whereis(1)

[Pedro Falcato]

[-- Attachment #1: Type: text/plain, Size: 2074 bytes --]

On Tue, Nov 22, 2022 at 4:26 PM <tlaronde@polynum.com> wrote:

> Le Tue, Nov 22, 2022 at 08:40:30AM -0700, Rebecca Cran a écrit :
> > On 11/21/22 15:22, Pedro Falcato wrote:
> >
> > > I kind of dislike your solution. Does NetBSD ship /bin/which by
> default?
> > > I think replacing whereis with "which -a" would be a lot better.
> > > I don't think there's a 100% standard way to do this in POSIX, as which
> > > isn't POSIX either, and your solution seems... hacky?
> >
> >
> > "command" seems to be the POSIX way to do this?
> > https://pubs.opengroup.org/onlinepubs/9699919799/
> >
> > Though "whereis python3" shows the following on my system:
> >
> > python3: /usr/bin/python3.9-config /usr/bin/python3.9 /usr/bin/python3
> > /usr/lib/python3.9 /usr/lib/python3 /etc/python3.9 /etc/python3
> > /usr/local/lib/python3.9 /usr/include/python3.9 /usr/share/python3
> > /usr/share/man/man1/python3.1.gz
> >
> > "which -a python3" returns:
> >
> > /usr/bin/python3
> > /bin/python3
> >
> > And "command -p -v" returns:
> >
> > /bin/python3
> >
> > I don't know if we need all the results from "whereis"?
>
> The problem is when one does not know which exact version of python
> is here and how, exactly, the command is named.
>
> whereis(1) returns whatever command with python3 as prefix, while
> command or "which -a" will return only exactly python3. If, on the
> system, python is fully version qualified (as is the case with
> pkgsrc, the opt packages framework, used on NetBSD):
>
> which -a python3
>
> will return nothing, since, it is python3.9 for example on the OS.
>

Sorry, what? This sounds so broken. How can a script shebang ever work?

For Sane(tm) systems, I propose command -v python3 + $(command -v python3)
-c 'import sys; print(sys.version.split(" ")[0])',
which gives us the path to the Python 3 interpreter + the path in a
relatively easy, simple way.
I think this could work mostly everywhere but apparently NetBSD since you
don't provide python3? Which makes little sense in my head.

Pedro

[-- Attachment #2: Type: text/html, Size: 2861 bytes --]

Re: [edk2-devel] edksetup.sh: fix for non POSIX whereis(1)

[tlaronde]

Le Tue, Nov 22, 2022 at 04:31:14PM +0000, Pedro Falcato a écrit :
> On Tue, Nov 22, 2022 at 4:26 PM <tlaronde@polynum.com> wrote:
> 
> > Le Tue, Nov 22, 2022 at 08:40:30AM -0700, Rebecca Cran a écrit :
> > > On 11/21/22 15:22, Pedro Falcato wrote:
> > >
> > > > I kind of dislike your solution. Does NetBSD ship /bin/which by
> > default?
> > > > I think replacing whereis with "which -a" would be a lot better.
> > > > I don't think there's a 100% standard way to do this in POSIX, as which
> > > > isn't POSIX either, and your solution seems... hacky?
> > >
> > >
> > > "command" seems to be the POSIX way to do this?
> > > https://pubs.opengroup.org/onlinepubs/9699919799/
> > >
> > > Though "whereis python3" shows the following on my system:
> > >
> > > python3: /usr/bin/python3.9-config /usr/bin/python3.9 /usr/bin/python3
> > > /usr/lib/python3.9 /usr/lib/python3 /etc/python3.9 /etc/python3
> > > /usr/local/lib/python3.9 /usr/include/python3.9 /usr/share/python3
> > > /usr/share/man/man1/python3.1.gz
> > >
> > > "which -a python3" returns:
> > >
> > > /usr/bin/python3
> > > /bin/python3
> > >
> > > And "command -p -v" returns:
> > >
> > > /bin/python3
> > >
> > > I don't know if we need all the results from "whereis"?
> >
> > The problem is when one does not know which exact version of python
> > is here and how, exactly, the command is named.
> >
> > whereis(1) returns whatever command with python3 as prefix, while
> > command or "which -a" will return only exactly python3. If, on the
> > system, python is fully version qualified (as is the case with
> > pkgsrc, the opt packages framework, used on NetBSD):
> >
> > which -a python3
> >
> > will return nothing, since, it is python3.9 for example on the OS.
> >
> 
> Sorry, what? This sounds so broken. How can a script shebang ever work?
> 
> For Sane(tm) systems, I propose command -v python3 + $(command -v python3)
> -c 'import sys; print(sys.version.split(" ")[0])',
> which gives us the path to the Python 3 interpreter + the path in a
> relatively easy, simple way.
> I think this could work mostly everywhere but apparently NetBSD since you
> don't provide python3? Which makes little sense in my head.
> 

NetBSD (as other OSes) does not provide python in base. It is available
as an add-on. And the add-on framework on NetBSD allows to install
several versions of Python (since there seems to be so many of them...)
and in order to do that, the binary is not called "python", neither
"python3" but "python3.9" or "python3.10". And since one can
concurrently install different versions, there is no symbolic or
hardlink python3 to a preferred python 3 series binary.

As I have already said, edk2setup.sh allows to manipulate the
environment and, for example, allows to set explicitely the env var
PYTHON_COMMAND. So I know how to circumvent the problem. 

But, since the script uses whereis(1), the Linux version, that does
take "python3" as a prefix and searches for "python3*" (in more than the
directories in PATH in fact), I have simply made so that the current
behavior, intended(?), work on NetBSD too or on any system without
whereis(1) or with a whereis(1) that does not behave as the Linux one.

If you want to nuke whereis and use a POSIX2 utility instead, I'm
perfectly fine: I will have to make other adjustements for NetBSD (not
in edk2setup.sh) and I'm writing documentation on how-to, so I can
circumvent this elsewhere.
-- 
        Thierry Laronde <tlaronde +AT+ polynum +dot+ com>
                     http://www.kergis.com/
                    http://kertex.kergis.com/
Key fingerprint = 0FF7 E906 FBAF FE95 FD89  250D 52B1 AE95 6006 F40C