From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by mx.groups.io with SMTP id smtpd.web11.6035.1576825413053121072 for ; Thu, 19 Dec 2019 23:03:33 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@intel.onmicrosoft.com header.s=selector2-intel-onmicrosoft-com header.b=PZUq3QCv; spf=pass (domain: intel.com, ip: 134.134.136.24, mailfrom: aiden.park@intel.com) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga007.jf.intel.com ([10.7.209.58]) by orsmga102.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 19 Dec 2019 23:03:32 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.69,334,1571727600"; d="scan'208";a="206462200" Received: from fmsmsx106.amr.corp.intel.com ([10.18.124.204]) by orsmga007.jf.intel.com with ESMTP; 19 Dec 2019 23:03:32 -0800 Received: from FMSEDG002.ED.cps.intel.com (10.1.192.134) by FMSMSX106.amr.corp.intel.com (10.18.124.204) with Microsoft SMTP Server (TLS) id 14.3.439.0; Thu, 19 Dec 2019 23:03:32 -0800 Received: from NAM11-CO1-obe.outbound.protection.outlook.com (104.47.56.172) by edgegateway.intel.com (192.55.55.69) with Microsoft SMTP Server (TLS) id 14.3.439.0; Thu, 19 Dec 2019 23:03:31 -0800 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=II15EsfnxJp1czh0pmAU7gvXUtSDbe1ZoTFwc336LixXXbTF/0co1obeoGso7C0cFdyxe7ECuaAnpxPWGVvl0AssvAPAOwvSctD/WzR44uMgpf35yi6YMoiC/HMLyOlxXuvBQLEsGY9iB4o46CzJgOIQDnoJcWLiegfdzMA3Mr3mw8ogVVVL6Kelprx1SKuKwRd86pRFzSa0zTCN7xMrWyHsDgJ+nHuXOHj1rC7SmKPLiyRU9Lem3nBjcx+NJVP0oVIeOt0aOq202sipmn1vUrd/ygy9cSdAy066cB92zeZuAMaXCIAvbzPcJ63d82K8p0GokQhUbvEviHOd22XxBQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=4qq5eyJDa3XJqz0xTL3rFayd6h/VPaAQQMF9x8i9tjk=; b=HDQFoFWLmeSLB5FaNFiSQa/gvGkdMI0SRTGTPfrXgyBmQMfKisYYlSgS5PnVDHkfcflaonqeyZcbIIKesVj9UvBaPMO4IHN+McVnX4OxWQGhTcBrN+gPPb6oB0IoItQU2enD30Uf7ZukLcDFJiEU15Cn33ssd50DiSmELsm7G9KGA2no6t2twQ4IhWf7Od1T4/Iy8pos37SGn0isOdb8GncsTJoIJmDPel9u57ePZg+fCfEatkuZ6Z0cwjYPTxCB7uQVSKLDcqDdBkKC9M+8ljI3Ef3GYm+pH/cKZDxOJg6zwllQf3ttnLCwMV8cozIMh7AQOiZTHgT0xCPcxxhVsw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=intel.onmicrosoft.com; s=selector2-intel-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=4qq5eyJDa3XJqz0xTL3rFayd6h/VPaAQQMF9x8i9tjk=; b=PZUq3QCvXrMcYt47s8W/8IW0pNHhimhxEr1975jAD2MjKYCEtpaGW5QlMPOnp3zpWU1EkdLDHRt/aQv72XbNceidsqm8l+j+U4tjhsvxz3Rt2R8WpsbbIp+JO7Mtp/dWhoivQn2iAzJWdPK654aC+O0e2WgRrqtr1OzMMDEWjNM= Received: from BN8PR11MB3619.namprd11.prod.outlook.com (20.178.219.86) by BN8PR11MB3809.namprd11.prod.outlook.com (20.178.221.91) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2559.14; Fri, 20 Dec 2019 07:03:16 +0000 Received: from BN8PR11MB3619.namprd11.prod.outlook.com ([fe80::ed50:624b:38b9:e703]) by BN8PR11MB3619.namprd11.prod.outlook.com ([fe80::ed50:624b:38b9:e703%4]) with mapi id 15.20.2559.015; Fri, 20 Dec 2019 07:03:16 +0000 From: "Park, Aiden" To: "Feng, Bob C" , "devel@edk2.groups.io" Subject: Re: [PATCH] [edk2/BaseTools] edksetup.bat stuck on unicode locale Windows Thread-Topic: [PATCH] [edk2/BaseTools] edksetup.bat stuck on unicode locale Windows Thread-Index: AdWwdDLVp9xBmfjoQ6mo1EN+7xsRzgE3xbnAAFrZBwAADwZSYAAAwTkA Date: Fri, 20 Dec 2019 07:03:15 +0000 Message-ID: References: <08650203BA1BD64D8AD9B6D5D74A85D16158434D@SHSMSX104.ccr.corp.intel.com> <08650203BA1BD64D8AD9B6D5D74A85D16158584A@SHSMSX104.ccr.corp.intel.com> In-Reply-To: <08650203BA1BD64D8AD9B6D5D74A85D16158584A@SHSMSX104.ccr.corp.intel.com> Accept-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: dlp-reaction: no-action dlp-version: 11.2.0.6 dlp-product: dlpe-windows authentication-results: spf=none (sender IP is ) smtp.mailfrom=aiden.park@intel.com; x-originating-ip: [134.134.136.214] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: a04dd85a-e186-4079-2b48-08d7851aafff x-ms-traffictypediagnostic: BN8PR11MB3809: x-ld-processed: 46c98d88-e344-4ed4-8496-4ed7712e255d,ExtAddr x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:7219; x-forefront-prvs: 025796F161 x-forefront-antispam-report: SFV:NSPM;SFS:(10019020)(376002)(366004)(346002)(136003)(396003)(39860400002)(199004)(189003)(43544003)(13464003)(71200400001)(55016002)(966005)(110136005)(8676002)(7696005)(81156014)(8936002)(26005)(186003)(81166006)(53546011)(9686003)(33656002)(66476007)(66946007)(76116006)(5660300002)(52536014)(2906002)(66556008)(64756008)(478600001)(6506007)(66446008)(316002)(86362001);DIR:OUT;SFP:1102;SCL:1;SRVR:BN8PR11MB3809;H:BN8PR11MB3619.namprd11.prod.outlook.com;FPR:;SPF:None;LANG:en;PTR:InfoNoRecords;A:1;MX:1; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: 2gPWfwruUvTHvYq6pdstONkBUkZGPAVva0Ov4Q6bUtPSi4eaJ1QuO78VDkIshquCl4Ecxkjca+TQtBXv7RLcqkLWjTCQigkHROkoGifEZu60habNmtVHiMalbX1CcFOiDzee/Dap2HrhCMidmA8ZCOzXIvpzTEOAkF1h1iL1Z3FDqFoNlDwYkqVQcy395+8FDIWDHcYNzE8ljxHCLfAICPjVbQ4G5/f3hQFhIKApDxb1oQRbhHzve+aS+pdjsFakaRhiCPUXBUNLAA4R2injKBnnJmYCEtu484Gjja2UfOuVp91yYKlaeJbSVT0L0vSGvWvUkn1V/LtGfF1tLawRtTZfYgpNjrw8D51lpo4yy1Rs3eJ9wFmHmonguC3X6OOgrOaGGhCpSxAt26Mc4iA16d9Y1tHpWtpReadaLJUZsWBq2KJ8QyfzVglwl7Hl7yCyvDMGc97lEEBDAivBVzU6oCCxbZsOghyet4nDxl0G4Uh7RDKZjEWPx9Rwmcl61/MUwIrAUvNuoq4xntfz5d1IRHgDVq2ZxPqCQJ4LUapvAC966ca2XyY7RBhuLQaZ1RbH MIME-Version: 1.0 X-MS-Exchange-CrossTenant-Network-Message-Id: a04dd85a-e186-4079-2b48-08d7851aafff X-MS-Exchange-CrossTenant-originalarrivaltime: 20 Dec 2019 07:03:15.9157 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: iLwA9dHgAb2TDZ7na5/IK6adoI7diphIr6PB4pZRIBwH37Aw3OX31T1YQnoLDoh15WEOninFyIpC/0SXWVO57A== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN8PR11MB3809 Return-Path: aiden.park@intel.com X-OriginatorOrg: intel.com Content-Language: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hi Bob, > -----Original Message----- > From: Feng, Bob C > Sent: Thursday, December 19, 2019 10:12 PM > To: Park, Aiden ; devel@edk2.groups.io > Subject: RE: [PATCH] [edk2/BaseTools] edksetup.bat stuck on unicode loca= le > Windows >=20 > Hi Aiden, >=20 > I'd like to know why need to call 'edksetup.bat forcerebuild' with pytho= n > subprocess.call(). > Is there other way to execute edksetup.bat? >=20 I forgot to mention one more thing in reproduce steps. This is also reprod= ucible by just calling 'edksetup.bat forcerebuild' in Windows command promp= t w/o python subprocess.call(). I have tested on 'Chinese - Traditional, Taiwan' and 'Korean' locale Windo= ws and it's reproducible on both Unicode locales. And below change works ar= ound the issue on both locales. Some experiment. Not sure if this is python2 bug. message =3D "" <=3D type 'str' temp =3D stdout.decode(encoding=3D'utf-8', errors=3D'ignore') <=3D type 'u= nicode' which has unicode string message =3D temp.encode('utf-8') <=3D pass message =3D temp <=3D deadlock > Thanks, > Bob > -----Original Message----- > From: Park, Aiden > Sent: Friday, December 20, 2019 8:23 AM > To: Feng, Bob C ; devel@edk2.groups.io > Subject: RE: [PATCH] [edk2/BaseTools] edksetup.bat stuck on unicode loca= le > Windows >=20 > Hi Bob, >=20 > > -----Original Message----- > > From: Feng, Bob C > > Sent: Tuesday, December 17, 2019 9:23 PM > > To: devel@edk2.groups.io; Park, Aiden > > Cc: Feng, Bob C > > Subject: RE: [PATCH] [edk2/BaseTools] edksetup.bat stuck on unicode > > locale Windows > > > > Hi Aiden, > > > > If set kwargs["stdout"] =3D sys.stdout, then stdout and stderr message= s > > from sub process will directly write to sys.stdout. > > I think this patch works because sys.stdout.encoding is the correct > encoding. > > So what do you think if we fix this Non-Ascii issue by Changing the > > line of message =3D stdout.decode(encoding=3D'utf-8', errors=3D'ignore= ') > > #for compatibility in python 2 and 3 to message =3D > > stdout.decode(encoding=3Dsys.stdout.encoding , errors=3D'ignore') #for > > compatibility in python 2 and 3 > > > > Otherwise, I think this patch may need to take care of the code after > > this line kwargs["stdout"] =3D sys.stdout, I mean > > p.communicate() will return empty string after your change, and the > > following code would be useless. > > > Thanks for your comment. You are right. This change makes the following > code useless. It should use PIPE. >=20 > I did double-check the environment to make sure reproduce steps, and it > turns out that it's reproducible on Unicode locale Windows + Python 2. > Python 3 does not have the issue. >=20 > There seems to be 2 locations which make deadlock. > 1. subprocess.Popen() > It looks there is a race condition when creating a child process with PI= PE. > Therefore, a lock is put for Popen(). > + popen_lock.acquire(True) > p =3D subprocess.Popen(Args, cwd=3DWorkDir, stderr=3Dkwargs["stderr= "], > stdout=3Dkwargs["stdout"]) > + popen_lock.release() >=20 > 2. stdout.decode() > stdout variable is decoded to 'unicode' type message variable. But, it's= stuck > if stdout has Unicode string even if errors=3D'ignore'. > Converting The 'unicode' type message to 'str' type message resolves the > issue. > - message =3D stdout.decode(encoding=3D'utf-8', errors=3D'ignore'= ) #for > compatibility in python 2 and 3 > + message =3D stdout.decode(encoding=3D'utf-8', > + errors=3D'ignore').encode('utf-8') #for compatibility in python 2 and = 3 >=20 > FYI, your recommendation encoding=3Dsys.stdout.encoding does not help to > resolve this issue. >=20 > I look forward to your feedback. Thanks. >=20 > > Thanks, > > Bob > > > > -----Original Message----- > > From: devel@edk2.groups.io [mailto:devel@edk2.groups.io] On Behalf Of > > Park, Aiden > > Sent: Thursday, December 12, 2019 6:43 AM > > To: devel@edk2.groups.io > > Subject: [edk2-devel] [PATCH] [edk2/BaseTools] edksetup.bat stuck on > > unicode locale Windows > > > > This issue happens under two conditions. > > 1. Unicode language environment in Windows > > 2. Call 'edksetup.bat forcerebuild' with python subprocess.call() > > > > Steps to reproduce > > C:\edk2>python > > Python 2.7.12 (v2.7.12:d33e0cf91556, Jun 27 2016, 15:24:40) > > Type help, copyright, credits or license for more information. > > >>> import subprocess > > >>> subprocess.call(['edksetup.bat', 'forcerebuild']) > > > > The edksetup.bat stuck at 'nmake cleanall'. > > > > One of multi-threads is on deadlock when python handles stdout and > > stderr in a subprocess pipe only if the outputs include unicode chars. > > Only stderr will be handled in the pipe same as a single thread call. > > > > Reported in Slim Bootloader. > > https://github.com/slimbootloader/slimbootloader/issues/478 > > Local fix has been made in Slim Bootloader. > > https://github.com/slimbootloader/slimbootloader/pull/490 > > > > Signed-off-by: Aiden Park > > --- > > BaseTools/Source/C/Makefiles/NmakeSubdirs.py | 2 +- > > 1 file changed, 1 insertion(+), 1 deletion(-) > > > > diff --git a/BaseTools/Source/C/Makefiles/NmakeSubdirs.py > > b/BaseTools/Source/C/Makefiles/NmakeSubdirs.py > > index 356f5ac..c77bfb0 100644 > > --- a/BaseTools/Source/C/Makefiles/NmakeSubdirs.py > > +++ b/BaseTools/Source/C/Makefiles/NmakeSubdirs.py > > @@ -33,7 +33,7 @@ def RunCommand(WorkDir=3DNone, *Args, **kwargs): > > if "stderr" not in kwargs: > > kwargs["stderr"] =3D subprocess.STDOUT > > if "stdout" not in kwargs: > > - kwargs["stdout"] =3D subprocess.PIPE > > + kwargs["stdout"] =3D sys.stdout > > p =3D subprocess.Popen(Args, cwd=3DWorkDir, stderr=3Dkwargs["stde= rr"], > > stdout=3Dkwargs["stdout"]) > > stdout, stderr =3D p.communicate() > > message =3D "" > > -- > > 2.10.2.windows.1 > > > >=20 > > >=20 > Best Regards, > Aiden >=20 Best Regards, Aiden