From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mga12.intel.com (mga12.intel.com [192.55.52.136]) by mx.groups.io with SMTP id smtpd.web12.5785.1576822338489228614 for ; Thu, 19 Dec 2019 22:12:20 -0800 Authentication-Results: mx.groups.io; dkim=missing; spf=pass (domain: intel.com, ip: 192.55.52.136, mailfrom: bob.c.feng@intel.com) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by fmsmga106.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 19 Dec 2019 22:12:16 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.69,334,1571727600"; d="scan'208";a="222435413" Received: from fmsmsx107.amr.corp.intel.com ([10.18.124.205]) by fmsmga001.fm.intel.com with ESMTP; 19 Dec 2019 22:12:16 -0800 Received: from fmsmsx111.amr.corp.intel.com (10.18.116.5) by fmsmsx107.amr.corp.intel.com (10.18.124.205) with Microsoft SMTP Server (TLS) id 14.3.439.0; Thu, 19 Dec 2019 22:12:16 -0800 Received: from shsmsx152.ccr.corp.intel.com (10.239.6.52) by fmsmsx111.amr.corp.intel.com (10.18.116.5) with Microsoft SMTP Server (TLS) id 14.3.439.0; Thu, 19 Dec 2019 22:12:16 -0800 Received: from shsmsx104.ccr.corp.intel.com ([169.254.5.90]) by SHSMSX152.ccr.corp.intel.com ([169.254.6.222]) with mapi id 14.03.0439.000; Fri, 20 Dec 2019 14:12:13 +0800 From: "Bob Feng" To: "Park, Aiden" , "devel@edk2.groups.io" Subject: Re: [PATCH] [edk2/BaseTools] edksetup.bat stuck on unicode locale Windows Thread-Topic: [PATCH] [edk2/BaseTools] edksetup.bat stuck on unicode locale Windows Thread-Index: AdWwdDLVp9xBmfjoQ6mo1EN+7xsRzgE3xbnAAFrZBwAADwZSYA== Date: Fri, 20 Dec 2019 06:12:13 +0000 Message-ID: <08650203BA1BD64D8AD9B6D5D74A85D16158584A@SHSMSX104.ccr.corp.intel.com> References: <08650203BA1BD64D8AD9B6D5D74A85D16158434D@SHSMSX104.ccr.corp.intel.com> In-Reply-To: Accept-Language: zh-CN, en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.239.127.40] MIME-Version: 1.0 Return-Path: bob.c.feng@intel.com Content-Language: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hi Aiden, I'd like to know why need to call 'edksetup.bat forcerebuild' with python = subprocess.call(). Is there other way to execute edksetup.bat? Thanks, Bob -----Original Message----- From: Park, Aiden=20 Sent: Friday, December 20, 2019 8:23 AM To: Feng, Bob C ; devel@edk2.groups.io Subject: RE: [PATCH] [edk2/BaseTools] edksetup.bat stuck on unicode locale= Windows Hi Bob, > -----Original Message----- > From: Feng, Bob C > Sent: Tuesday, December 17, 2019 9:23 PM > To: devel@edk2.groups.io; Park, Aiden > Cc: Feng, Bob C > Subject: RE: [PATCH] [edk2/BaseTools] edksetup.bat stuck on unicode=20 > locale Windows >=20 > Hi Aiden, >=20 > If set kwargs["stdout"] =3D sys.stdout, then stdout and stderr messages= =20 > from sub process will directly write to sys.stdout. > I think this patch works because sys.stdout.encoding is the correct enco= ding. > So what do you think if we fix this Non-Ascii issue by Changing the=20 > line of message =3D stdout.decode(encoding=3D'utf-8', errors=3D'ignore')= = =20 > #for compatibility in python 2 and 3 to message =3D=20 > stdout.decode(encoding=3Dsys.stdout.encoding , errors=3D'ignore') #for= =20 > compatibility in python 2 and 3 >=20 > Otherwise, I think this patch may need to take care of the code after=20 > this line kwargs["stdout"] =3D sys.stdout, I mean > p.communicate() will return empty string after your change, and the=20 > following code would be useless. > Thanks for your comment. You are right. This change makes the following co= de useless. It should use PIPE. I did double-check the environment to make sure reproduce steps, and it tu= rns out that it's reproducible on Unicode locale Windows + Python 2. Python= 3 does not have the issue. There seems to be 2 locations which make deadlock. 1. subprocess.Popen() It looks there is a race condition when creating a child process with PIPE= . Therefore, a lock is put for Popen(). + popen_lock.acquire(True) p =3D subprocess.Popen(Args, cwd=3DWorkDir, stderr=3Dkwargs["stderr"]= , stdout=3Dkwargs["stdout"]) + popen_lock.release() 2. stdout.decode() stdout variable is decoded to 'unicode' type message variable. But, it's s= tuck if stdout has Unicode string even if errors=3D'ignore'. Converting The 'unicode' type message to 'str' type message resolves the i= ssue. - message =3D stdout.decode(encoding=3D'utf-8', errors=3D'ignore') = #for compatibility in python 2 and 3 + message =3D stdout.decode(encoding=3D'utf-8',=20 + errors=3D'ignore').encode('utf-8') #for compatibility in python 2 and 3 FYI, your recommendation encoding=3Dsys.stdout.encoding does not help to r= esolve this issue. I look forward to your feedback. Thanks. > Thanks, > Bob >=20 > -----Original Message----- > From: devel@edk2.groups.io [mailto:devel@edk2.groups.io] On Behalf Of=20 > Park, Aiden > Sent: Thursday, December 12, 2019 6:43 AM > To: devel@edk2.groups.io > Subject: [edk2-devel] [PATCH] [edk2/BaseTools] edksetup.bat stuck on=20 > unicode locale Windows >=20 > This issue happens under two conditions. > 1. Unicode language environment in Windows > 2. Call 'edksetup.bat forcerebuild' with python subprocess.call() >=20 > Steps to reproduce > C:\edk2>python > Python 2.7.12 (v2.7.12:d33e0cf91556, Jun 27 2016, 15:24:40) > Type help, copyright, credits or license for more information. > >>> import subprocess > >>> subprocess.call(['edksetup.bat', 'forcerebuild']) >=20 > The edksetup.bat stuck at 'nmake cleanall'. >=20 > One of multi-threads is on deadlock when python handles stdout and=20 > stderr in a subprocess pipe only if the outputs include unicode chars. > Only stderr will be handled in the pipe same as a single thread call. >=20 > Reported in Slim Bootloader. > https://github.com/slimbootloader/slimbootloader/issues/478 > Local fix has been made in Slim Bootloader. > https://github.com/slimbootloader/slimbootloader/pull/490 >=20 > Signed-off-by: Aiden Park > --- > BaseTools/Source/C/Makefiles/NmakeSubdirs.py | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) >=20 > diff --git a/BaseTools/Source/C/Makefiles/NmakeSubdirs.py > b/BaseTools/Source/C/Makefiles/NmakeSubdirs.py > index 356f5ac..c77bfb0 100644 > --- a/BaseTools/Source/C/Makefiles/NmakeSubdirs.py > +++ b/BaseTools/Source/C/Makefiles/NmakeSubdirs.py > @@ -33,7 +33,7 @@ def RunCommand(WorkDir=3DNone, *Args, **kwargs): > if "stderr" not in kwargs: > kwargs["stderr"] =3D subprocess.STDOUT > if "stdout" not in kwargs: > - kwargs["stdout"] =3D subprocess.PIPE > + kwargs["stdout"] =3D sys.stdout > p =3D subprocess.Popen(Args, cwd=3DWorkDir, stderr=3Dkwargs["stderr= "], > stdout=3Dkwargs["stdout"]) > stdout, stderr =3D p.communicate() > message =3D "" > -- > 2.10.2.windows.1 >=20 >=20 >=20 Best Regards, Aiden