From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mga01.intel.com (mga01.intel.com [192.55.52.88]) by mx.groups.io with SMTP id smtpd.web10.3454.1576801402947022637 for ; Thu, 19 Dec 2019 16:23:23 -0800 Authentication-Results: mx.groups.io; dkim=pass header.i=@intel.onmicrosoft.com header.s=selector2-intel-onmicrosoft-com header.b=UI8F3+2O; spf=pass (domain: intel.com, ip: 192.55.52.88, mailfrom: aiden.park@intel.com) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by fmsmga101.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 19 Dec 2019 16:23:22 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.69,333,1571727600"; d="scan'208";a="241342859" Received: from fmsmsx104.amr.corp.intel.com ([10.18.124.202]) by fmsmga004.fm.intel.com with ESMTP; 19 Dec 2019 16:23:22 -0800 Received: from fmsmsx602.amr.corp.intel.com (10.18.126.82) by fmsmsx104.amr.corp.intel.com (10.18.124.202) with Microsoft SMTP Server (TLS) id 14.3.439.0; Thu, 19 Dec 2019 16:23:22 -0800 Received: from fmsmsx602.amr.corp.intel.com (10.18.126.82) by fmsmsx602.amr.corp.intel.com (10.18.126.82) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Thu, 19 Dec 2019 16:23:21 -0800 Received: from FMSEDG002.ED.cps.intel.com (10.1.192.134) by fmsmsx602.amr.corp.intel.com (10.18.126.82) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256) id 15.1.1713.5 via Frontend Transport; Thu, 19 Dec 2019 16:23:21 -0800 Received: from NAM04-CO1-obe.outbound.protection.outlook.com (104.47.45.50) by edgegateway.intel.com (192.55.55.69) with Microsoft SMTP Server (TLS) id 14.3.439.0; Thu, 19 Dec 2019 16:23:21 -0800 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=j4UPqVB4TrSP5QgWgXQYjLhL0is0F/n2YJLABvSNMC0PMPWBwNlPjGbHGIoei/20Fzs5HUTYPYgKgCGt1p3OLW72QGH4oV/1BFnu9lSWtb9/aO+GBhzGNcfLpD15FUrxXC8pFsP2UnhZ5OkZy/4V0Yqs5loqjvpb4zmwHjEs04ctwQh3b1/iKEs/zo+HzTPW73XcZk2o/H46RG9nXY6eglYNBXcpa7pbnm2NeDLe9ufpjauDi9zn3vOgNNKo2EU1c4HxoR4XkSnN5zPfKPIcgE1BxtmD48vYj87BzFm4uzKk7WtvC3hP8cb+EppPTvfjiOC8mrUYB4SftiEYGZxvyQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=l0HtMmKg4oPLNxdPdtJDEaVNdsmZfVmIZB100H82GNA=; b=TC63zzE/yYRm3NZdJItJ1HOfsh4aB4VCFIW9KhWueLmZs1nK92EjALc+AWfMxDply4AYXv7di9LBF+hzXYJF3hBagZfppv6dbECDvsg3o1vP3R08XTJGaV8ZazEctMphaLCVai3z0oUKV43JtWGelJBlezmRraJnvvIEQuZB2NJ6wRixQCsMuP7oxVywESKdMmDp+XuJBJQep+4iWvyw+FEUwYjyG1fRNqQbwn8m6jdXWvZUjdZl0VsjjRccvxDFtoup+DWNZkisqLZS1qbv0H7NwnohPz4B68RkJcVAf9UCEAngZVTvhrm6JJZj0E1A96wou1HzHzfBx+15dGoLqw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=intel.com; dmarc=pass action=none header.from=intel.com; dkim=pass header.d=intel.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=intel.onmicrosoft.com; s=selector2-intel-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=l0HtMmKg4oPLNxdPdtJDEaVNdsmZfVmIZB100H82GNA=; b=UI8F3+2OUDLm9LT0+EfjzSWX9Ckbzf/I8czi73yk6KqXapT232BKD2+gNSSicRVVRxcShFMWExYDNxRBtWrzS03BbLutgSvW1MkBk1ANcBpT/HXZPHnq3jaWkqaYS9rpUCnidkg5u+7paHjhxCl7Tq3e1VIr3c0FfCnboDPIDkI= Received: from BN8PR11MB3619.namprd11.prod.outlook.com (20.178.219.86) by BN8PR11MB3809.namprd11.prod.outlook.com (20.178.221.91) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2559.14; Fri, 20 Dec 2019 00:23:19 +0000 Received: from BN8PR11MB3619.namprd11.prod.outlook.com ([fe80::ed50:624b:38b9:e703]) by BN8PR11MB3619.namprd11.prod.outlook.com ([fe80::ed50:624b:38b9:e703%4]) with mapi id 15.20.2559.015; Fri, 20 Dec 2019 00:23:19 +0000 From: aiden.park@intel.com To: "Feng, Bob C" , "devel@edk2.groups.io" Subject: Re: [PATCH] [edk2/BaseTools] edksetup.bat stuck on unicode locale Windows Thread-Topic: [PATCH] [edk2/BaseTools] edksetup.bat stuck on unicode locale Windows Thread-Index: AdWwdDLVp9xBmfjoQ6mo1EN+7xsRzgE3xbnAAFrZBwA= Date: Fri, 20 Dec 2019 00:23:19 +0000 Message-ID: References: <08650203BA1BD64D8AD9B6D5D74A85D16158434D@SHSMSX104.ccr.corp.intel.com> In-Reply-To: <08650203BA1BD64D8AD9B6D5D74A85D16158434D@SHSMSX104.ccr.corp.intel.com> Accept-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: dlp-reaction: no-action dlp-version: 11.2.0.6 dlp-product: dlpe-windows authentication-results: spf=none (sender IP is ) smtp.mailfrom=aiden.park@intel.com; x-originating-ip: [134.134.136.221] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 68387c6f-0b24-409e-1171-08d784e2d0b3 x-ms-traffictypediagnostic: BN8PR11MB3809: x-ld-processed: 46c98d88-e344-4ed4-8496-4ed7712e255d,ExtAddr x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:7219; x-forefront-prvs: 025796F161 x-forefront-antispam-report: SFV:NSPM;SFS:(10019020)(376002)(366004)(346002)(136003)(396003)(39860400002)(189003)(199004)(43544003)(13464003)(71200400001)(55016002)(8676002)(110136005)(7696005)(81156014)(8936002)(26005)(186003)(81166006)(53546011)(9686003)(66476007)(33656002)(66946007)(76116006)(5660300002)(52536014)(2906002)(64756008)(66556008)(478600001)(6506007)(66446008)(316002)(86362001)(966005);DIR:OUT;SFP:1102;SCL:1;SRVR:BN8PR11MB3809;H:BN8PR11MB3619.namprd11.prod.outlook.com;FPR:;SPF:None;LANG:en;PTR:InfoNoRecords;MX:1;A:1; x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: oWUtCHOVEnhvifV0yHcK8B6CHrmkOTHGWShVoEeY0dDc7Msrzm8KgJZ6zf8A4l3SNS3RL4ZkqxWn7uGdLaBmlfhm2JLh0PlkrbYmfuvOkfNPDCvKe+kLCAYxev24JJPGd07P7BRn6gsoJtzXlUSwchT8MLO8Lr/Vegjhn25TdVwMpFkmisUCJ39kKBiDnBV3xxcQ8l6IATGeq3UeCSfFhik4wX55SOfhPoWS3LRZDO9LH0n+bJMCOQAo6+YVMSalHOhn7uoA16zCpwHr/GRKeM7Uurh4DkzLVJIHcsaZ9prl7H6aLFj8MxSknGshc5C0aTEP1fw1E9fHWS5BYlVh0DVPMiGfjdSwULpKc7OJvf0yzH5ReAciaZZ3I9q4ny3ux1FRa3M0enpdvnt5EbsCoPITWw+f1+Miua+aEvhzXmVKAXyptvUMyZfkFZNKt3Oea8YHsYOMLkbXbFfxJ2G9t8I+UpxaWBnIy/l5p3GLfllv0tW1DhoppdgaKbHMdIvGuwM3D1FGFJ1TiNrllnKTDGdgOFPIiXEF2DtTHn1p1Fc/vCHDDLNrGqYIfrtNLCn0 MIME-Version: 1.0 X-MS-Exchange-CrossTenant-Network-Message-Id: 68387c6f-0b24-409e-1171-08d784e2d0b3 X-MS-Exchange-CrossTenant-originalarrivaltime: 20 Dec 2019 00:23:19.3087 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 46c98d88-e344-4ed4-8496-4ed7712e255d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: KZVcO3Ne18QEKjPzU61tMxhHDJ0ADIJOzmXazp/wYlEQUSMo3NTqXtAN3RG2ThnRCToITYES3qMg+1FizAXddg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN8PR11MB3809 Return-Path: aiden.park@intel.com X-OriginatorOrg: intel.com Content-Language: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hi Bob, > -----Original Message----- > From: Feng, Bob C > Sent: Tuesday, December 17, 2019 9:23 PM > To: devel@edk2.groups.io; Park, Aiden > Cc: Feng, Bob C > Subject: RE: [PATCH] [edk2/BaseTools] edksetup.bat stuck on unicode loca= le > Windows >=20 > Hi Aiden, >=20 > If set kwargs["stdout"] =3D sys.stdout, then stdout and stderr messages = from > sub process will directly write to sys.stdout. > I think this patch works because sys.stdout.encoding is the correct enco= ding. > So what do you think if we fix this Non-Ascii issue by Changing the line= of > message =3D stdout.decode(encoding=3D'utf-8', errors=3D'ignore') #for > compatibility in python 2 and 3 to message =3D > stdout.decode(encoding=3Dsys.stdout.encoding , errors=3D'ignore') #for > compatibility in python 2 and 3 >=20 > Otherwise, I think this patch may need to take care of the code after th= is line > kwargs["stdout"] =3D sys.stdout, I mean > p.communicate() will return empty string after your change, and the > following code would be useless. > Thanks for your comment. You are right. This change makes the following co= de useless. It should use PIPE. I did double-check the environment to make sure reproduce steps, and it tu= rns out that it's reproducible on Unicode locale Windows + Python 2. Python= 3 does not have the issue. There seems to be 2 locations which make deadlock. 1. subprocess.Popen() It looks there is a race condition when creating a child process with PIPE= . Therefore, a lock is put for Popen(). + popen_lock.acquire(True) p =3D subprocess.Popen(Args, cwd=3DWorkDir, stderr=3Dkwargs["stderr"]= , stdout=3Dkwargs["stdout"]) + popen_lock.release() 2. stdout.decode() stdout variable is decoded to 'unicode' type message variable. But, it's s= tuck if stdout has Unicode string even if errors=3D'ignore'. Converting The 'unicode' type message to 'str' type message resolves the i= ssue. - message =3D stdout.decode(encoding=3D'utf-8', errors=3D'ignore') = #for compatibility in python 2 and 3 + message =3D stdout.decode(encoding=3D'utf-8', errors=3D'ignore').= encode('utf-8') #for compatibility in python 2 and 3 FYI, your recommendation encoding=3Dsys.stdout.encoding does not help to r= esolve this issue. I look forward to your feedback. Thanks. > Thanks, > Bob >=20 > -----Original Message----- > From: devel@edk2.groups.io [mailto:devel@edk2.groups.io] On Behalf Of > Park, Aiden > Sent: Thursday, December 12, 2019 6:43 AM > To: devel@edk2.groups.io > Subject: [edk2-devel] [PATCH] [edk2/BaseTools] edksetup.bat stuck on > unicode locale Windows >=20 > This issue happens under two conditions. > 1. Unicode language environment in Windows > 2. Call 'edksetup.bat forcerebuild' with python subprocess.call() >=20 > Steps to reproduce > C:\edk2>python > Python 2.7.12 (v2.7.12:d33e0cf91556, Jun 27 2016, 15:24:40) > Type help, copyright, credits or license for more information. > >>> import subprocess > >>> subprocess.call(['edksetup.bat', 'forcerebuild']) >=20 > The edksetup.bat stuck at 'nmake cleanall'. >=20 > One of multi-threads is on deadlock when python handles stdout and stder= r > in a subprocess pipe only if the outputs include unicode chars. > Only stderr will be handled in the pipe same as a single thread call. >=20 > Reported in Slim Bootloader. > https://github.com/slimbootloader/slimbootloader/issues/478 > Local fix has been made in Slim Bootloader. > https://github.com/slimbootloader/slimbootloader/pull/490 >=20 > Signed-off-by: Aiden Park > --- > BaseTools/Source/C/Makefiles/NmakeSubdirs.py | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) >=20 > diff --git a/BaseTools/Source/C/Makefiles/NmakeSubdirs.py > b/BaseTools/Source/C/Makefiles/NmakeSubdirs.py > index 356f5ac..c77bfb0 100644 > --- a/BaseTools/Source/C/Makefiles/NmakeSubdirs.py > +++ b/BaseTools/Source/C/Makefiles/NmakeSubdirs.py > @@ -33,7 +33,7 @@ def RunCommand(WorkDir=3DNone, *Args, **kwargs): > if "stderr" not in kwargs: > kwargs["stderr"] =3D subprocess.STDOUT > if "stdout" not in kwargs: > - kwargs["stdout"] =3D subprocess.PIPE > + kwargs["stdout"] =3D sys.stdout > p =3D subprocess.Popen(Args, cwd=3DWorkDir, stderr=3Dkwargs["stderr= "], > stdout=3Dkwargs["stdout"]) > stdout, stderr =3D p.communicate() > message =3D "" > -- > 2.10.2.windows.1 >=20 >=20 >=20 Best Regards, Aiden