From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by mx.groups.io with SMTP id smtpd.web09.12453.1576865329815005003 for ; Fri, 20 Dec 2019 10:08:50 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@intel.onmicrosoft.com header.s=selector2-intel-onmicrosoft-com header.b=LqIgjzKm; spf=pass (domain: intel.com, ip: 134.134.136.65, mailfrom: aiden.park@intel.com) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga002.jf.intel.com ([10.7.209.21]) by orsmga103.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 20 Dec 2019 10:08:49 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.69,336,1571727600"; d="scan'208";a="228673440" Received: from fmsmsx105.amr.corp.intel.com ([10.18.124.203]) by orsmga002.jf.intel.com with ESMTP; 20 Dec 2019 10:08:48 -0800 Received: from fmsmsx102.amr.corp.intel.com (10.18.124.200) by FMSMSX105.amr.corp.intel.com (10.18.124.203) with Microsoft SMTP Server (TLS) id 14.3.439.0; Fri, 20 Dec 2019 10:08:48 -0800 Received: from FMSEDG001.ED.cps.intel.com (10.1.192.133) by FMSMSX102.amr.corp.intel.com (10.18.124.200) with Microsoft SMTP Server (TLS) id 14.3.439.0; Fri, 20 Dec 2019 10:08:48 -0800 Received: from NAM02-BL2-obe.outbound.protection.outlook.com (104.47.38.58) by edgegateway.intel.com (192.55.55.68) with Microsoft SMTP Server (TLS) id 14.3.439.0; Fri, 20 Dec 2019 10:08:47 -0800 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=mlGwyXgSTg72nA+pg+5gk2Q/+KrXMWduNL5mDAIJe0tRY54MP1CkQpSvUz+RHZ7KAI/96YIqi0pT/GuMoucBY2zF2C9Spfz4FVIz9YgzLhgNlMZndkhYkxGSJQCNPke83Na8unzV4nADR7ZBciF9lNDL8qNVxjbiJ19meTQaEnBNfrc7ZJOGHOZVyI/SVz4AT7K69WNrce8cTNPBu3VHlon+Ri1fjvo1OUvpOdqFEP01PW512sZRvLOTKCtM4QVLOksmfxBLO0FZpva0UU0rUpzH3kz762+4op4/p7GFGmKPIGCVL3TjX1Z+42VIV0BWVTyhCT/hygu/tLLpHRVaLw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=kHEaJmQh/BGRStMF5F+Jzmf2agDKdrV2jFv7PCMkNiw=; b=G5PNbB3XPDmvSbhtT1zz9vZbv8eMQ9FTf5InKO83J6ExvFoOJV1rwekO6J0HkjCMQ+V0RoraK1+EFAvYIx0duzdN960sMAhtE74JknJhv2znqPZBOWl973t7iy445HpPlQFO6/60nLMjSxJBIEarySrKNhpMIg9T17LKcuQDvYCmMZNGyg1omHwYI6EDKiaNb/GQcFviD/mrmdQNyi0snkBcJFMRRCXx0hWtzExsnUfm9CX8U1P6mX3SrwIlyX0RAmJsWVDLr2/zA46VcnRkeuC0RUOwgyHjjg3xSIjk+rGCI4rStT9U/pVotpoHYdnMTzwxOQz97iIAC2KX77hCWA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=intel.onmicrosoft.com; s=selector2-intel-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=kHEaJmQh/BGRStMF5F+Jzmf2agDKdrV2jFv7PCMkNiw=; b=LqIgjzKmlFnrnH48mVFTiG8f97olIIPrgW3gWizl4/Ku1TLfKbo+h4iw3ZUWDXopUcEEqcBKb2CBDgSuj44LtwkCVQcD4fCyJS4RwfEF8CJzWi9Fz++sI8AWOhXOGvH1Tb63uJlWnmjzLlnDfBTrYidAhnJvxNWmpf/rE8e187A= Received: from BN8PR11MB3619.namprd11.prod.outlook.com (20.178.219.86) by BN8PR11MB3652.namprd11.prod.outlook.com (20.178.219.211) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2559.16; Fri, 20 Dec 2019 18:08:46 +0000 Received: from BN8PR11MB3619.namprd11.prod.outlook.com ([fe80::ed50:624b:38b9:e703]) by BN8PR11MB3619.namprd11.prod.outlook.com ([fe80::ed50:624b:38b9:e703%4]) with mapi id 15.20.2559.015; Fri, 20 Dec 2019 18:08:45 +0000 From: "Park, Aiden" To: "Feng, Bob C" , "devel@edk2.groups.io" Subject: Re: [PATCH] [edk2/BaseTools] edksetup.bat stuck on unicode locale Windows Thread-Topic: [PATCH] [edk2/BaseTools] edksetup.bat stuck on unicode locale Windows Thread-Index: AdWwdDLVp9xBmfjoQ6mo1EN+7xsRzgE3xbnAAFrZBwAADwZSYAAAwTkAAALx4yAAFYBVUA== Date: Fri, 20 Dec 2019 18:08:45 +0000 Message-ID: References: <08650203BA1BD64D8AD9B6D5D74A85D16158434D@SHSMSX104.ccr.corp.intel.com> <08650203BA1BD64D8AD9B6D5D74A85D16158584A@SHSMSX104.ccr.corp.intel.com> <08650203BA1BD64D8AD9B6D5D74A85D161585B2F@SHSMSX104.ccr.corp.intel.com> In-Reply-To: <08650203BA1BD64D8AD9B6D5D74A85D161585B2F@SHSMSX104.ccr.corp.intel.com> Accept-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: dlp-reaction: no-action dlp-version: 11.2.0.6 dlp-product: dlpe-windows authentication-results: spf=none (sender IP is ) smtp.mailfrom=aiden.park@intel.com; x-originating-ip: [134.134.136.205] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: e63ee559-d1fb-40c2-ed21-08d78577a7c2 x-ms-traffictypediagnostic: BN8PR11MB3652: x-ld-processed: 46c98d88-e344-4ed4-8496-4ed7712e255d,ExtAddr x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:6430; x-forefront-prvs: 025796F161 x-forefront-antispam-report: SFV:NSPM;SFS:(10019020)(39860400002)(346002)(396003)(366004)(136003)(376002)(13464003)(199004)(189003)(43544003)(26005)(8676002)(52536014)(186003)(53546011)(6506007)(81156014)(81166006)(8936002)(86362001)(966005)(55016002)(9686003)(5660300002)(110136005)(66556008)(66446008)(64756008)(7696005)(316002)(66476007)(2906002)(71200400001)(33656002)(66946007)(76116006)(478600001);DIR:OUT;SFP:1102;SCL:1;SRVR:BN8PR11MB3652;H:BN8PR11MB3619.namprd11.prod.outlook.com;FPR:;SPF:None;LANG:en;PTR:InfoNoRecords;A:1;MX:1; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: FtYJFFvJauyDHk5FiuawkBJwBQN9HJpYIxqw0U+rblzYWwFQqEArv3gbTuT0YeTdspzyyQmobYypWJ3jJiKw9ijXm/qzZU+iawZLbbuIRfk/JocgJ+vi5Cd7n2QhCocs/42B6sngtJkKxIsxo5sxfS8N08xBsP2uHauhQtdNYKbPN3R0HdHRIpDscnFcyM5/E7EL0FMk1Hca6qQfiNbT42QXlTh+YZ+WGLmA6kRWikvwY/tTc6FYijZoSr3Xh4r38UeoMXKyrn3njDDOt7n+MHQkBHcLMvAF/0dV6ueR+w/7yP/bVCdBKsg6KH3oJdHDf/HIiQ/RWiep7gV34/A1Tqf0CLICrlK/kal3df56FcFLQO+MDkKcz++qCfvwe9sdSozCSgFiRImz0BKZr0S4c1/ozCfYogwVIFp2NAVuXudLiqPCs9sg0/q3xydtWDktP92GyaDjrkkGGWOX+o2c4e066m8u5unvifnEIvTRBg9OGNeH/sU/68+jwJsK6h/giLMUGGst41qXkOmEHyyyS2VOcgpJCF/YnhagVvwXdTa8FS47Jpr1hHhRWbTBKVvF MIME-Version: 1.0 X-MS-Exchange-CrossTenant-Network-Message-Id: e63ee559-d1fb-40c2-ed21-08d78577a7c2 X-MS-Exchange-CrossTenant-originalarrivaltime: 20 Dec 2019 18:08:45.7197 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: xF93oP5PultdSLO0QUitihqaN3veyODvt90dH1wowRchvg4ess04h6o3AQh6JNmHQePBKqcD6s+WP/fe7yopVQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN8PR11MB3652 Return-Path: aiden.park@intel.com X-OriginatorOrg: intel.com Content-Language: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hi Bob, > -----Original Message----- > From: Feng, Bob C > Sent: Friday, December 20, 2019 12:19 AM > To: Park, Aiden ; devel@edk2.groups.io > Subject: RE: [PATCH] [edk2/BaseTools] edksetup.bat stuck on unicode loca= le > Windows >=20 > Hi Aiden, >=20 > Thanks for clarifying this issue. Since I have no Unicode locales window= s > environment, I can't help to do the verification. > For the following 2# message =3D stdout.decode(encoding=3D'utf-8', > errors=3D'ignore').encode('utf-8'), after encode(), on python3, the mess= age > will not a string, it's bytes type data. >=20 > BTW, would you help check whether "message =3D > stdout.decode(errors=3D'ignore')" works? Your recommendation works as expected. The issue is not reproducible anymo= re with this change. I have also dry-run on https://github.com/tianocore/edk2/pull/255 and all = checks passed. Thanks for giving better recommendation. >=20 > Thanks, > Bob >=20 > -----Original Message----- > From: Park, Aiden > Sent: Friday, December 20, 2019 3:03 PM > To: Feng, Bob C ; devel@edk2.groups.io > Subject: RE: [PATCH] [edk2/BaseTools] edksetup.bat stuck on unicode loca= le > Windows >=20 > Hi Bob, >=20 > > -----Original Message----- > > From: Feng, Bob C > > Sent: Thursday, December 19, 2019 10:12 PM > > To: Park, Aiden ; devel@edk2.groups.io > > Subject: RE: [PATCH] [edk2/BaseTools] edksetup.bat stuck on unicode > > locale Windows > > > > Hi Aiden, > > > > I'd like to know why need to call 'edksetup.bat forcerebuild' with > > python subprocess.call(). > > Is there other way to execute edksetup.bat? > > > I forgot to mention one more thing in reproduce steps. This is also > reproducible by just calling 'edksetup.bat forcerebuild' in Windows comm= and > prompt w/o python subprocess.call(). > I have tested on 'Chinese - Traditional, Taiwan' and 'Korean' locale Win= dows > and it's reproducible on both Unicode locales. And below change works > around the issue on both locales. >=20 > Some experiment. Not sure if this is python2 bug. > message =3D "" <=3D type 'str' > temp =3D stdout.decode(encoding=3D'utf-8', errors=3D'ignore') <=3D type = 'unicode' > which has unicode string message =3D temp.encode('utf-8') <=3D pass mess= age =3D > temp <=3D deadlock >=20 > > Thanks, > > Bob > > -----Original Message----- > > From: Park, Aiden > > Sent: Friday, December 20, 2019 8:23 AM > > To: Feng, Bob C ; devel@edk2.groups.io > > Subject: RE: [PATCH] [edk2/BaseTools] edksetup.bat stuck on unicode > > locale Windows > > > > Hi Bob, > > > > > -----Original Message----- > > > From: Feng, Bob C > > > Sent: Tuesday, December 17, 2019 9:23 PM > > > To: devel@edk2.groups.io; Park, Aiden > > > Cc: Feng, Bob C > > > Subject: RE: [PATCH] [edk2/BaseTools] edksetup.bat stuck on unicode > > > locale Windows > > > > > > Hi Aiden, > > > > > > If set kwargs["stdout"] =3D sys.stdout, then stdout and stderr > > > messages from sub process will directly write to sys.stdout. > > > I think this patch works because sys.stdout.encoding is the correct > > encoding. > > > So what do you think if we fix this Non-Ascii issue by Changing the > > > line of message =3D stdout.decode(encoding=3D'utf-8', errors=3D'igno= re') > > > #for compatibility in python 2 and 3 to message =3D > > > stdout.decode(encoding=3Dsys.stdout.encoding , errors=3D'ignore') #f= or > > > compatibility in python 2 and 3 > > > > > > Otherwise, I think this patch may need to take care of the code > > > after this line kwargs["stdout"] =3D sys.stdout, I mean > > > p.communicate() will return empty string after your change, and the > > > following code would be useless. > > > > > Thanks for your comment. You are right. This change makes the > > following code useless. It should use PIPE. > > > > I did double-check the environment to make sure reproduce steps, and > > it turns out that it's reproducible on Unicode locale Windows + Python= 2. > > Python 3 does not have the issue. > > > > There seems to be 2 locations which make deadlock. > > 1. subprocess.Popen() > > It looks there is a race condition when creating a child process with = PIPE. > > Therefore, a lock is put for Popen(). > > + popen_lock.acquire(True) > > p =3D subprocess.Popen(Args, cwd=3DWorkDir, stderr=3Dkwargs["stde= rr"], > > stdout=3Dkwargs["stdout"]) > > + popen_lock.release() > > > > 2. stdout.decode() > > stdout variable is decoded to 'unicode' type message variable. But, > > it's stuck if stdout has Unicode string even if errors=3D'ignore'. > > Converting The 'unicode' type message to 'str' type message resolves > > the issue. > > - message =3D stdout.decode(encoding=3D'utf-8', errors=3D'ignor= e') #for > > compatibility in python 2 and 3 > > + message =3D stdout.decode(encoding=3D'utf-8', > > + errors=3D'ignore').encode('utf-8') #for compatibility in python 2 an= d > > + 3 > > > > FYI, your recommendation encoding=3Dsys.stdout.encoding does not help = to > > resolve this issue. > > > > I look forward to your feedback. Thanks. > > > > > Thanks, > > > Bob > > > > > > -----Original Message----- > > > From: devel@edk2.groups.io [mailto:devel@edk2.groups.io] On Behalf > > > Of Park, Aiden > > > Sent: Thursday, December 12, 2019 6:43 AM > > > To: devel@edk2.groups.io > > > Subject: [edk2-devel] [PATCH] [edk2/BaseTools] edksetup.bat stuck on > > > unicode locale Windows > > > > > > This issue happens under two conditions. > > > 1. Unicode language environment in Windows > > > 2. Call 'edksetup.bat forcerebuild' with python subprocess.call() > > > > > > Steps to reproduce > > > C:\edk2>python > > > Python 2.7.12 (v2.7.12:d33e0cf91556, Jun 27 2016, 15:24:40) > > > Type help, copyright, credits or license for more information. > > > >>> import subprocess > > > >>> subprocess.call(['edksetup.bat', 'forcerebuild']) > > > > > > The edksetup.bat stuck at 'nmake cleanall'. > > > > > > One of multi-threads is on deadlock when python handles stdout and > > > stderr in a subprocess pipe only if the outputs include unicode char= s. > > > Only stderr will be handled in the pipe same as a single thread call= . > > > > > > Reported in Slim Bootloader. > > > https://github.com/slimbootloader/slimbootloader/issues/478 > > > Local fix has been made in Slim Bootloader. > > > https://github.com/slimbootloader/slimbootloader/pull/490 > > > > > > Signed-off-by: Aiden Park > > > --- > > > BaseTools/Source/C/Makefiles/NmakeSubdirs.py | 2 +- > > > 1 file changed, 1 insertion(+), 1 deletion(-) > > > > > > diff --git a/BaseTools/Source/C/Makefiles/NmakeSubdirs.py > > > b/BaseTools/Source/C/Makefiles/NmakeSubdirs.py > > > index 356f5ac..c77bfb0 100644 > > > --- a/BaseTools/Source/C/Makefiles/NmakeSubdirs.py > > > +++ b/BaseTools/Source/C/Makefiles/NmakeSubdirs.py > > > @@ -33,7 +33,7 @@ def RunCommand(WorkDir=3DNone, *Args, **kwargs): > > > if "stderr" not in kwargs: > > > kwargs["stderr"] =3D subprocess.STDOUT > > > if "stdout" not in kwargs: > > > - kwargs["stdout"] =3D subprocess.PIPE > > > + kwargs["stdout"] =3D sys.stdout > > > p =3D subprocess.Popen(Args, cwd=3DWorkDir, > > > stderr=3Dkwargs["stderr"], > > > stdout=3Dkwargs["stdout"]) > > > stdout, stderr =3D p.communicate() > > > message =3D "" > > > -- > > > 2.10.2.windows.1 > > > > > >=20 > > > > > > > Best Regards, > > Aiden > > >=20 > Best Regards, > Aiden >=20 Best Regards, Aiden