public inbox for devel@edk2.groups.io
 help / color / mirror / Atom feed
From: aiden.park@intel.com
To: "Feng, Bob C" <bob.c.feng@intel.com>,
	"devel@edk2.groups.io" <devel@edk2.groups.io>
Subject: Re: [PATCH] [edk2/BaseTools] edksetup.bat stuck on unicode locale Windows
Date: Fri, 20 Dec 2019 00:23:19 +0000	[thread overview]
Message-ID: <BN8PR11MB36197F67CD19D68E2FF2AD698E2D0@BN8PR11MB3619.namprd11.prod.outlook.com> (raw)
In-Reply-To: <08650203BA1BD64D8AD9B6D5D74A85D16158434D@SHSMSX104.ccr.corp.intel.com>

Hi Bob,

> -----Original Message-----
> From: Feng, Bob C <bob.c.feng@intel.com>
> Sent: Tuesday, December 17, 2019 9:23 PM
> To: devel@edk2.groups.io; Park, Aiden <aiden.park@intel.com>
> Cc: Feng, Bob C <bob.c.feng@intel.com>
> Subject: RE: [PATCH] [edk2/BaseTools] edksetup.bat stuck on unicode locale
> Windows
> 
> Hi Aiden,
> 
> If set kwargs["stdout"] = sys.stdout, then stdout and stderr messages from
> sub process will directly write to sys.stdout.
> I think this patch works because sys.stdout.encoding is the correct encoding.
> So what do you think if we fix this Non-Ascii issue by Changing the line of
> message = stdout.decode(encoding='utf-8', errors='ignore') #for
> compatibility in python 2 and 3 to message =
> stdout.decode(encoding=sys.stdout.encoding , errors='ignore') #for
> compatibility in python 2 and 3
> 
> Otherwise, I think this patch may need to take care of the code after this line
> kwargs["stdout"] = sys.stdout, I mean
> p.communicate() will return empty string after your change, and the
> following code would be useless.
>
Thanks for your comment. You are right. This change makes the following code useless. It should use PIPE.

I did double-check the environment to make sure reproduce steps, and it turns out that it's reproducible on Unicode locale Windows + Python 2. Python 3 does not have the issue.

There seems to be 2 locations which make deadlock.
1. subprocess.Popen()
It looks there is a race condition when creating a child process with PIPE. Therefore, a lock is put for Popen().
+    popen_lock.acquire(True)
     p = subprocess.Popen(Args, cwd=WorkDir, stderr=kwargs["stderr"], stdout=kwargs["stdout"])
+    popen_lock.release()

2. stdout.decode()
stdout variable is decoded to 'unicode' type message variable. But, it's stuck if stdout has Unicode string even if errors='ignore'.
Converting The 'unicode' type message to 'str' type message resolves the issue.
-        message = stdout.decode(encoding='utf-8', errors='ignore') #for compatibility in python 2 and 3
+        message = stdout.decode(encoding='utf-8', errors='ignore').encode('utf-8') #for compatibility in python 2 and 3
FYI, your recommendation encoding=sys.stdout.encoding does not help to resolve this issue.

I look forward to your feedback. Thanks.

> Thanks,
> Bob
> 
> -----Original Message-----
> From: devel@edk2.groups.io [mailto:devel@edk2.groups.io] On Behalf Of
> Park, Aiden
> Sent: Thursday, December 12, 2019 6:43 AM
> To: devel@edk2.groups.io
> Subject: [edk2-devel] [PATCH] [edk2/BaseTools] edksetup.bat stuck on
> unicode locale Windows
> 
> This issue happens under two conditions.
>   1. Unicode language environment in Windows
>   2. Call 'edksetup.bat forcerebuild' with python subprocess.call()
> 
> Steps to reproduce
>   C:\edk2>python
>   Python 2.7.12 (v2.7.12:d33e0cf91556, Jun 27 2016, 15:24:40)
>   Type help, copyright, credits or license for more information.
>   >>> import subprocess
>   >>> subprocess.call(['edksetup.bat', 'forcerebuild'])
> 
>   The edksetup.bat stuck at 'nmake cleanall'.
> 
> One of multi-threads is on deadlock when python handles stdout and stderr
> in a subprocess pipe only if the outputs include unicode chars.
> Only stderr will be handled in the pipe same as a single thread call.
> 
> Reported in Slim Bootloader.
>   https://github.com/slimbootloader/slimbootloader/issues/478
> Local fix has been made in Slim Bootloader.
>   https://github.com/slimbootloader/slimbootloader/pull/490
> 
> Signed-off-by: Aiden Park <aiden.park@intel.com>
> ---
>  BaseTools/Source/C/Makefiles/NmakeSubdirs.py | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/BaseTools/Source/C/Makefiles/NmakeSubdirs.py
> b/BaseTools/Source/C/Makefiles/NmakeSubdirs.py
> index 356f5ac..c77bfb0 100644
> --- a/BaseTools/Source/C/Makefiles/NmakeSubdirs.py
> +++ b/BaseTools/Source/C/Makefiles/NmakeSubdirs.py
> @@ -33,7 +33,7 @@ def RunCommand(WorkDir=None, *Args, **kwargs):
>      if "stderr" not in kwargs:
>          kwargs["stderr"] = subprocess.STDOUT
>      if "stdout" not in kwargs:
> -        kwargs["stdout"] = subprocess.PIPE
> +        kwargs["stdout"] = sys.stdout
>      p = subprocess.Popen(Args, cwd=WorkDir, stderr=kwargs["stderr"],
> stdout=kwargs["stdout"])
>      stdout, stderr = p.communicate()
>      message = ""
> --
> 2.10.2.windows.1
> 
> 
> 

Best Regards,
Aiden

  reply	other threads:[~2019-12-20  0:23 UTC|newest]

Thread overview: 7+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-12-11 22:42 [PATCH] [edk2/BaseTools] edksetup.bat stuck on unicode locale Windows Park, Aiden
2019-12-18  5:23 ` Bob Feng
2019-12-20  0:23   ` aiden.park [this message]
2019-12-20  6:12     ` Bob Feng
2019-12-20  7:03       ` Park, Aiden
2019-12-20  8:19         ` Bob Feng
2019-12-20 18:08           ` Park, Aiden

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-list from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=BN8PR11MB36197F67CD19D68E2FF2AD698E2D0@BN8PR11MB3619.namprd11.prod.outlook.com \
    --to=devel@edk2.groups.io \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox