From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by mx.groups.io with SMTP id smtpd.web10.6647.1576829950015855176 for ; Fri, 20 Dec 2019 00:19:10 -0800 Authentication-Results: mx.groups.io; dkim=missing; spf=pass (domain: intel.com, ip: 192.55.52.115, mailfrom: bob.c.feng@intel.com) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga006.fm.intel.com ([10.253.24.20]) by fmsmga103.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 20 Dec 2019 00:19:09 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.69,335,1571727600"; d="scan'208";a="417886777" Received: from fmsmsx104.amr.corp.intel.com ([10.18.124.202]) by fmsmga006.fm.intel.com with ESMTP; 20 Dec 2019 00:19:09 -0800 Received: from fmsmsx120.amr.corp.intel.com (10.18.124.208) by fmsmsx104.amr.corp.intel.com (10.18.124.202) with Microsoft SMTP Server (TLS) id 14.3.439.0; Fri, 20 Dec 2019 00:19:09 -0800 Received: from shsmsx106.ccr.corp.intel.com (10.239.4.159) by fmsmsx120.amr.corp.intel.com (10.18.124.208) with Microsoft SMTP Server (TLS) id 14.3.439.0; Fri, 20 Dec 2019 00:19:09 -0800 Received: from shsmsx104.ccr.corp.intel.com ([169.254.5.90]) by SHSMSX106.ccr.corp.intel.com ([169.254.10.236]) with mapi id 14.03.0439.000; Fri, 20 Dec 2019 16:19:07 +0800 From: "Bob Feng" To: "Park, Aiden" , "devel@edk2.groups.io" Subject: Re: [PATCH] [edk2/BaseTools] edksetup.bat stuck on unicode locale Windows Thread-Topic: [PATCH] [edk2/BaseTools] edksetup.bat stuck on unicode locale Windows Thread-Index: AdWwdDLVp9xBmfjoQ6mo1EN+7xsRzgE3xbnAAFrZBwAADwZSYAAAwTkAAALx4yA= Date: Fri, 20 Dec 2019 08:19:06 +0000 Message-ID: <08650203BA1BD64D8AD9B6D5D74A85D161585B2F@SHSMSX104.ccr.corp.intel.com> References: <08650203BA1BD64D8AD9B6D5D74A85D16158434D@SHSMSX104.ccr.corp.intel.com> <08650203BA1BD64D8AD9B6D5D74A85D16158584A@SHSMSX104.ccr.corp.intel.com> In-Reply-To: Accept-Language: zh-CN, en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.239.127.40] MIME-Version: 1.0 Return-Path: bob.c.feng@intel.com Content-Language: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hi Aiden, Thanks for clarifying this issue. Since I have no Unicode locales windows = environment, I can't help to do the verification.=20 For the following 2# message =3D stdout.decode(encoding=3D'utf-8', errors= =3D'ignore').encode('utf-8'), after encode(), on python3, the message will= not a string, it's bytes type data. BTW, would you help check whether "message =3D stdout.decode(errors=3D'ign= ore')" works? Thanks, Bob -----Original Message----- From: Park, Aiden=20 Sent: Friday, December 20, 2019 3:03 PM To: Feng, Bob C ; devel@edk2.groups.io Subject: RE: [PATCH] [edk2/BaseTools] edksetup.bat stuck on unicode locale= Windows Hi Bob, > -----Original Message----- > From: Feng, Bob C > Sent: Thursday, December 19, 2019 10:12 PM > To: Park, Aiden ; devel@edk2.groups.io > Subject: RE: [PATCH] [edk2/BaseTools] edksetup.bat stuck on unicode=20 > locale Windows >=20 > Hi Aiden, >=20 > I'd like to know why need to call 'edksetup.bat forcerebuild' with=20 > python subprocess.call(). > Is there other way to execute edksetup.bat? >=20 I forgot to mention one more thing in reproduce steps. This is also reprod= ucible by just calling 'edksetup.bat forcerebuild' in Windows command promp= t w/o python subprocess.call(). I have tested on 'Chinese - Traditional, Taiwan' and 'Korean' locale Windo= ws and it's reproducible on both Unicode locales. And below change works ar= ound the issue on both locales. Some experiment. Not sure if this is python2 bug. message =3D "" <=3D type 'str' temp =3D stdout.decode(encoding=3D'utf-8', errors=3D'ignore') <=3D type 'u= nicode' which has unicode string message =3D temp.encode('utf-8') <=3D pass= message =3D temp <=3D deadlock > Thanks, > Bob > -----Original Message----- > From: Park, Aiden > Sent: Friday, December 20, 2019 8:23 AM > To: Feng, Bob C ; devel@edk2.groups.io > Subject: RE: [PATCH] [edk2/BaseTools] edksetup.bat stuck on unicode=20 > locale Windows >=20 > Hi Bob, >=20 > > -----Original Message----- > > From: Feng, Bob C > > Sent: Tuesday, December 17, 2019 9:23 PM > > To: devel@edk2.groups.io; Park, Aiden > > Cc: Feng, Bob C > > Subject: RE: [PATCH] [edk2/BaseTools] edksetup.bat stuck on unicode=20 > > locale Windows > > > > Hi Aiden, > > > > If set kwargs["stdout"] =3D sys.stdout, then stdout and stderr=20 > > messages from sub process will directly write to sys.stdout. > > I think this patch works because sys.stdout.encoding is the correct > encoding. > > So what do you think if we fix this Non-Ascii issue by Changing the=20 > > line of message =3D stdout.decode(encoding=3D'utf-8', errors=3D'ignore= ')=20 > > #for compatibility in python 2 and 3 to message =3D=20 > > stdout.decode(encoding=3Dsys.stdout.encoding , errors=3D'ignore') #for= = =20 > > compatibility in python 2 and 3 > > > > Otherwise, I think this patch may need to take care of the code=20 > > after this line kwargs["stdout"] =3D sys.stdout, I mean > > p.communicate() will return empty string after your change, and the=20 > > following code would be useless. > > > Thanks for your comment. You are right. This change makes the=20 > following code useless. It should use PIPE. >=20 > I did double-check the environment to make sure reproduce steps, and=20 > it turns out that it's reproducible on Unicode locale Windows + Python 2= . > Python 3 does not have the issue. >=20 > There seems to be 2 locations which make deadlock. > 1. subprocess.Popen() > It looks there is a race condition when creating a child process with PI= PE. > Therefore, a lock is put for Popen(). > + popen_lock.acquire(True) > p =3D subprocess.Popen(Args, cwd=3DWorkDir, stderr=3Dkwargs["stderr= "], > stdout=3Dkwargs["stdout"]) > + popen_lock.release() >=20 > 2. stdout.decode() > stdout variable is decoded to 'unicode' type message variable. But,=20 > it's stuck if stdout has Unicode string even if errors=3D'ignore'. > Converting The 'unicode' type message to 'str' type message resolves=20 > the issue. > - message =3D stdout.decode(encoding=3D'utf-8', errors=3D'ignore'= ) #for > compatibility in python 2 and 3 > + message =3D stdout.decode(encoding=3D'utf-8', > + errors=3D'ignore').encode('utf-8') #for compatibility in python 2 and= =20 > + 3 >=20 > FYI, your recommendation encoding=3Dsys.stdout.encoding does not help to= = =20 > resolve this issue. >=20 > I look forward to your feedback. Thanks. >=20 > > Thanks, > > Bob > > > > -----Original Message----- > > From: devel@edk2.groups.io [mailto:devel@edk2.groups.io] On Behalf=20 > > Of Park, Aiden > > Sent: Thursday, December 12, 2019 6:43 AM > > To: devel@edk2.groups.io > > Subject: [edk2-devel] [PATCH] [edk2/BaseTools] edksetup.bat stuck on= =20 > > unicode locale Windows > > > > This issue happens under two conditions. > > 1. Unicode language environment in Windows > > 2. Call 'edksetup.bat forcerebuild' with python subprocess.call() > > > > Steps to reproduce > > C:\edk2>python > > Python 2.7.12 (v2.7.12:d33e0cf91556, Jun 27 2016, 15:24:40) > > Type help, copyright, credits or license for more information. > > >>> import subprocess > > >>> subprocess.call(['edksetup.bat', 'forcerebuild']) > > > > The edksetup.bat stuck at 'nmake cleanall'. > > > > One of multi-threads is on deadlock when python handles stdout and=20 > > stderr in a subprocess pipe only if the outputs include unicode chars. > > Only stderr will be handled in the pipe same as a single thread call. > > > > Reported in Slim Bootloader. > > https://github.com/slimbootloader/slimbootloader/issues/478 > > Local fix has been made in Slim Bootloader. > > https://github.com/slimbootloader/slimbootloader/pull/490 > > > > Signed-off-by: Aiden Park > > --- > > BaseTools/Source/C/Makefiles/NmakeSubdirs.py | 2 +- > > 1 file changed, 1 insertion(+), 1 deletion(-) > > > > diff --git a/BaseTools/Source/C/Makefiles/NmakeSubdirs.py > > b/BaseTools/Source/C/Makefiles/NmakeSubdirs.py > > index 356f5ac..c77bfb0 100644 > > --- a/BaseTools/Source/C/Makefiles/NmakeSubdirs.py > > +++ b/BaseTools/Source/C/Makefiles/NmakeSubdirs.py > > @@ -33,7 +33,7 @@ def RunCommand(WorkDir=3DNone, *Args, **kwargs): > > if "stderr" not in kwargs: > > kwargs["stderr"] =3D subprocess.STDOUT > > if "stdout" not in kwargs: > > - kwargs["stdout"] =3D subprocess.PIPE > > + kwargs["stdout"] =3D sys.stdout > > p =3D subprocess.Popen(Args, cwd=3DWorkDir,=20 > > stderr=3Dkwargs["stderr"], > > stdout=3Dkwargs["stdout"]) > > stdout, stderr =3D p.communicate() > > message =3D "" > > -- > > 2.10.2.windows.1 > > > >=20 > > >=20 > Best Regards, > Aiden >=20 Best Regards, Aiden