From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received-SPF: Pass (sender SPF authorized) identity=mailfrom; client-ip=192.55.52.115; helo=mga14.intel.com; envelope-from=liming.gao@intel.com; receiver=edk2-devel@lists.01.org Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ml01.01.org (Postfix) with ESMTPS id C5033211E00E6 for ; Fri, 29 Mar 2019 05:30:29 -0700 (PDT) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by fmsmga103.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 29 Mar 2019 05:30:29 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.60,284,1549958400"; d="scan'208";a="156942275" Received: from fmsmsx105.amr.corp.intel.com ([10.18.124.203]) by fmsmga004.fm.intel.com with ESMTP; 29 Mar 2019 05:30:29 -0700 Received: from fmsmsx115.amr.corp.intel.com (10.18.116.19) by FMSMSX105.amr.corp.intel.com (10.18.124.203) with Microsoft SMTP Server (TLS) id 14.3.408.0; Fri, 29 Mar 2019 05:30:29 -0700 Received: from shsmsx105.ccr.corp.intel.com (10.239.4.158) by fmsmsx115.amr.corp.intel.com (10.18.116.19) with Microsoft SMTP Server (TLS) id 14.3.408.0; Fri, 29 Mar 2019 05:30:28 -0700 Received: from shsmsx104.ccr.corp.intel.com ([169.254.5.74]) by SHSMSX105.ccr.corp.intel.com ([169.254.11.113]) with mapi id 14.03.0415.000; Fri, 29 Mar 2019 20:30:26 +0800 From: "Gao, Liming" To: "Fan, ZhijuX" , "edk2-devel@lists.01.org" Thread-Topic: [edk2][PATCH V3] BaseTools:Trim will trig exception when input asl UTF8 format file Thread-Index: AdTlFSvnOgCPcRRUQeOgkgNlPm2ssABFf/iA Date: Fri, 29 Mar 2019 12:30:26 +0000 Message-ID: <4A89E2EF3DFEDB4C8BFDE51014F606A14E40ECAD@SHSMSX104.ccr.corp.intel.com> References: In-Reply-To: Accept-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ctpclassification: CTP_NT x-titus-metadata-40: eyJDYXRlZ29yeUxhYmVscyI6IiIsIk1ldGFkYXRhIjp7Im5zIjoiaHR0cDpcL1wvd3d3LnRpdHVzLmNvbVwvbnNcL0ludGVsMyIsImlkIjoiOTk5NzE0Y2ItZDViNS00ZWI2LWFlOGUtZjRiNTI1Y2QxNmFmIiwicHJvcHMiOlt7Im4iOiJDVFBDbGFzc2lmaWNhdGlvbiIsInZhbHMiOlt7InZhbHVlIjoiQ1RQX05UIn1dfV19LCJTdWJqZWN0TGFiZWxzIjpbXSwiVE1DVmVyc2lvbiI6IjE3LjEwLjE4MDQuNDkiLCJUcnVzdGVkTGFiZWxIYXNoIjoiTXNyaDJnR2hMRUVhajVkcWxqRlorVm9Nd1Y5Q09MNlN0STc4SURERVFpZ0xEU3grWERxZFRvd1wvU0h6UjdRaEEifQ== dlp-product: dlpe-windows dlp-version: 11.0.400.15 dlp-reaction: no-action x-originating-ip: [10.239.127.40] MIME-Version: 1.0 Subject: Re: [PATCH V3] BaseTools:Trim will trig exception when input asl UTF8 format file X-BeenThere: edk2-devel@lists.01.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: EDK II Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 29 Mar 2019 12:30:30 -0000 Content-Language: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Reviewed-by: Liming Gao > -----Original Message----- > From: Fan, ZhijuX > Sent: Thursday, March 28, 2019 11:21 AM > To: edk2-devel@lists.01.org > Cc: Gao, Liming ; Feng, Bob C > Subject: [edk2][PATCH V3] BaseTools:Trim will trig exception when input a= sl UTF8 format file >=20 > BZ:https://bugzilla.tianocore.org/show_bug.cgi?id=3D1641 >=20 > The command trim --asl-file -o test.i UTF8.asl will trig the exception. > There's a problem with the encoding of the file,it only appears in python= 3. > I changed the way I opened it to support reading this file >=20 > Cc: Bob Feng > Cc: Liming Gao > Contributed-under: TianoCore Contribution Agreement 1.1 > Signed-off-by: Zhiju.Fan > --- > BaseTools/Source/Python/Trim/Trim.py | 44 +++++++++++++++++-------------= ------ > 1 file changed, 21 insertions(+), 23 deletions(-) >=20 > diff --git a/BaseTools/Source/Python/Trim/Trim.py b/BaseTools/Source/Pyth= on/Trim/Trim.py > index 228779b5a9..05feab6abe 100644 > --- a/BaseTools/Source/Python/Trim/Trim.py > +++ b/BaseTools/Source/Python/Trim/Trim.py > @@ -18,7 +18,7 @@ import Common.LongFilePathOs as os > import sys > import re > from io import BytesIO > - > +import codecs > from optparse import OptionParser > from optparse import make_option > from Common.BuildToolError import * > @@ -77,14 +77,11 @@ gIncludedAslFile =3D [] > def TrimPreprocessedFile(Source, Target, ConvertHex, TrimLong): > CreateDirectory(os.path.dirname(Target)) > try: > - f =3D open (Source, 'r') > + with open(Source, "r") as File: > + Lines =3D File.readlines() > except: > EdkLogger.error("Trim", FILE_OPEN_FAILURE, ExtraData=3DSource) >=20 > - # read whole file > - Lines =3D f.readlines() > - f.close() > - > PreprocessedFile =3D "" > InjectedFile =3D "" > LineIndexOfOriginalFile =3D None > @@ -181,11 +178,10 @@ def TrimPreprocessedFile(Source, Target, ConvertHex= , TrimLong): >=20 > # save to file > try: > - f =3D open (Target, 'w') > + with open(Target, 'w') as File: > + File.writelines(NewLines) > except: > EdkLogger.error("Trim", FILE_OPEN_FAILURE, ExtraData=3DTarget) > - f.writelines(NewLines) > - f.close() >=20 > ## Trim preprocessed VFR file > # > @@ -199,12 +195,11 @@ def TrimPreprocessedVfr(Source, Target): > CreateDirectory(os.path.dirname(Target)) >=20 > try: > - f =3D open (Source, 'r') > + with open(Source, "r") as File: > + Lines =3D File.readlines() > except: > EdkLogger.error("Trim", FILE_OPEN_FAILURE, ExtraData=3DSource) > # read whole file > - Lines =3D f.readlines() > - f.close() >=20 > FoundTypedef =3D False > Brace =3D 0 > @@ -248,11 +243,10 @@ def TrimPreprocessedVfr(Source, Target): >=20 > # save all lines trimmed > try: > - f =3D open (Target, 'w') > + with open(Target, 'w') as File: > + File.writelines(Lines) > except: > EdkLogger.error("Trim", FILE_OPEN_FAILURE, ExtraData=3DTarget) > - f.writelines(Lines) > - f.close() >=20 > ## Read the content ASL file, including ASL included, recursively > # > @@ -278,7 +272,12 @@ def DoInclude(Source, Indent=3D'', IncludePathList= =3D[], LocalSearchPath=3DNone): > for IncludePath in SearchPathList: > IncludeFile =3D os.path.join(IncludePath, Source) > if os.path.isfile(IncludeFile): > - F =3D open(IncludeFile, "r") > + try: > + with open(IncludeFile, "r") as File: > + F =3D File.readlines() > + except: > + with codecs.open(IncludeFile, "r", encoding=3D'utf-8= ') as File: > + F =3D File.readlines() > break > else: > EdkLogger.error("Trim", "Failed to find include file %s" % S= ource) > @@ -313,7 +312,6 @@ def DoInclude(Source, Indent=3D'', IncludePathList=3D= [], LocalSearchPath=3DNone): > NewFileContent.append("\n") >=20 > gIncludedAslFile.pop() > - F.close() >=20 > return NewFileContent >=20 > @@ -345,7 +343,9 @@ def TrimAslFile(Source, Target, IncludePathFile): > if IncludePathFile: > try: > LineNum =3D 0 > - for Line in open(IncludePathFile, 'r'): > + with open(IncludePathFile, 'r') as File: > + FileLines =3D File.readlines() > + for Line in FileLines: > LineNum +=3D 1 > if Line.startswith("/I") or Line.startswith ("-I"): > IncludePathList.append(Line[2:].strip()) > @@ -363,13 +363,11 @@ def TrimAslFile(Source, Target, IncludePathFile): >=20 > # save all lines trimmed > try: > - f =3D open (Target, 'w') > + with open(Target, 'w') as File: > + File.writelines(Lines) > except: > EdkLogger.error("Trim", FILE_OPEN_FAILURE, ExtraData=3DTarget) >=20 > - f.writelines(Lines) > - f.close() > - > def GenerateVfrBinSec(ModuleName, DebugDir, OutputFile): > VfrNameList =3D [] > if os.path.isdir(DebugDir): > @@ -389,7 +387,7 @@ def GenerateVfrBinSec(ModuleName, DebugDir, OutputFil= e): > return >=20 > try: > - fInputfile =3D open(OutputFile, "wb+", 0) > + fInputfile =3D open(OutputFile, "wb+") > except: > EdkLogger.error("Trim", FILE_OPEN_FAILURE, "File open failed for= %s" %OutputFile, None) >=20 > -- > 2.14.1.windows.1