From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received-SPF: Pass (sender SPF authorized) identity=mailfrom; client-ip=104.47.37.133; helo=nam02-cy1-obe.outbound.protection.outlook.com; envelope-from=christopher.co@microsoft.com; receiver=edk2-devel@lists.01.org Received: from NAM02-CY1-obe.outbound.protection.outlook.com (mail-cys01nam02on0133.outbound.protection.outlook.com [104.47.37.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (No client certificate requested) by ml01.01.org (Postfix) with ESMTPS id B4FED21B02845 for ; Wed, 27 Jun 2018 11:23:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=famWI2Wfp9ILstjQO8YSSWvhnY0MhGC4nSOa7k2PCpw=; b=Dtq/KttlT3+N6nx45VHnjVNmapnxtMFz4+0tWXO4T8cWlX2iFrzlvQx1HcXTgJk/io1uF2E81AQtA+dL0IUo9LMXvS5t1VREAGwnlSo2G751PgHASntsYwez38GCygwAv0oGoo2AIZK53dDVu4R4QpR3rs7nHBiJnxhKIC1hVLg= Received: from DM5PR2101MB1128.namprd21.prod.outlook.com (52.132.133.20) by DM5PR2101MB0805.namprd21.prod.outlook.com (10.167.105.149) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.930.4; Wed, 27 Jun 2018 18:23:11 +0000 Received: from DM5PR2101MB1128.namprd21.prod.outlook.com ([fe80::28cb:9380:49e8:e0d0]) by DM5PR2101MB1128.namprd21.prod.outlook.com ([fe80::28cb:9380:49e8:e0d0%2]) with mapi id 15.20.0930.005; Wed, 27 Jun 2018 18:23:11 +0000 From: Chris Co To: Evan Lloyd , "edk2-devel@lists.01.org" , Liming Gao Thread-Topic: [PATCH v1 1/1] BaseTools/Trim: Canonicalize filepaths to fix comparison Thread-Index: AQHUDcsNZUVBqOxBWkWwqrDKFPQJj6Rz8L2wgABoEeCAABL1AA== Date: Wed, 27 Jun 2018 18:23:11 +0000 Message-ID: References: <20180627035753.49848-1-christopher.co@microsoft.com> <20180627035753.49848-2-christopher.co@microsoft.com> In-Reply-To: Accept-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: msip_labels: MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Enabled=True; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_SiteId=72f988bf-86f1-41af-91ab-2d7cd011db47; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Owner=chrco@microsoft.com; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_SetDate=2018-06-27T18:13:24.3213231Z; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Name=General; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Application=Microsoft Azure Information Protection; MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Extended_MSFT_Method=Automatic; Sensitivity=General x-originating-ip: [2001:4898:80e8:a:ac56:341b:8c69:467b] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; DM5PR2101MB0805; 7:aXyXRLTY5Oo9vjME8JIRHrHcPDuI+8gFR77NoXSy94fXt1G4/i47lJvKgNDa52nfmMr80n55ozP+hbbKNa6A+Y3SnTOOsCtm1VDDtuFDiYYDmvz9FYj55pmL7md6V3WMP8yJV0axHfW/2eR8TsP8+B72Kr8yJ2SoX0SoZg4V09YooPovHiv7gt7YM57qMkMQ0wq3gL9BfKfuKLS6BTlA8ffI4CnQcC5QAprvfGjhJHhrI08g2SYxBODG0clLY/5V x-ms-exchange-antispam-srfa-diagnostics: SOS; x-ms-office365-filtering-correlation-id: f259e65e-444b-454c-89dd-08d5dc5b0a34 x-ms-office365-filtering-ht: Tenant x-microsoft-antispam: UriScan:(223705240517415); BCL:0; PCL:0; RULEID:(7020095)(4652020)(8989117)(4534165)(4627221)(201703031133081)(201702281549075)(8990107)(5600026)(711020)(48565401081)(2017052603328)(7193020); SRVR:DM5PR2101MB0805; x-ms-traffictypediagnostic: DM5PR2101MB0805: x-ld-processed: 72f988bf-86f1-41af-91ab-2d7cd011db47,ExtAddr x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:(28532068793085)(180628864354917)(89211679590171)(166708455590820)(189930954265078)(223705240517415)(788757137089)(162533806227266)(219752817060721)(228905959029699); x-ms-exchange-senderadcheck: 1 x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(8211001083)(6040522)(2401047)(8121501046)(5005006)(3002001)(93006095)(93001095)(10201501046)(3231254)(2018427008)(944501410)(52105095)(6055026)(149027)(150027)(6041310)(20161123564045)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(20161123560045)(20161123558120)(20161123562045)(6072148)(201708071742011)(7699016); SRVR:DM5PR2101MB0805; BCL:0; PCL:0; RULEID:; SRVR:DM5PR2101MB0805; x-forefront-prvs: 0716E70AB6 x-forefront-antispam-report: SFV:NSPM; SFS:(10019020)(376002)(366004)(346002)(136003)(396003)(39860400002)(13464003)(189003)(199004)(4326008)(106356001)(11346002)(486006)(105586002)(446003)(305945005)(476003)(7736002)(86612001)(25786009)(6306002)(86362001)(9686003)(74316002)(68736007)(72206003)(14454004)(93886005)(6246003)(53936002)(966005)(2900100001)(575784001)(6116002)(10290500003)(54906003)(99286004)(316002)(110136005)(22452003)(478600001)(33656002)(7696005)(5250100002)(2906002)(10090500001)(8990500004)(186003)(5024004)(14444005)(2940100002)(46003)(6506007)(53546011)(8936002)(2501003)(256004)(76176011)(97736004)(102836004)(6346003)(81156014)(6436002)(8676002)(5660300001)(229853002)(55016002)(81166006)(460985005); DIR:OUT; SFP:1102; SCL:1; SRVR:DM5PR2101MB0805; H:DM5PR2101MB1128.namprd21.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; MX:1; A:1; received-spf: None (protection.outlook.com: microsoft.com does not designate permitted sender hosts) authentication-results: spf=none (sender IP is ) smtp.mailfrom=Christopher.Co@microsoft.com; x-microsoft-antispam-message-info: EI+xHMs5hdRsaXwfr1+8tKxJE1tTkLiJAdciPpd68KY07xuONb44/Bb+HMR1xBvB+Iw9LUkXJPKe8DkKoZNKM5L/a5pXqa0UKK3w7ShTBPmAM+X6ocM17ECF8z1McNF+CwllNDQ1zA1adT6Jmynf0UP5/aZN83+w0iZs5Si1zr68sJfSe4ZkMAZQm/V0uuF7N0sVdspazjhXDeOrfzwmaMurZ8i5GtmFtxhF79iPidiCBzN0xzlrEzaR7XM5bj88d6dw4hJcYU7V0NPbNk+kMmqF6FZe+xJPu9L3YCXt48o5cAvxpj70GZekzHK9Bp4NbRlBAptVhLtmbz8MQpcvlsXtgq/JcmzoQnRSDDMDXAc= spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM MIME-Version: 1.0 X-OriginatorOrg: microsoft.com X-MS-Exchange-CrossTenant-Network-Message-Id: f259e65e-444b-454c-89dd-08d5dc5b0a34 X-MS-Exchange-CrossTenant-originalarrivaltime: 27 Jun 2018 18:23:11.2962 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 72f988bf-86f1-41af-91ab-2d7cd011db47 X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5PR2101MB0805 Subject: Re: [PATCH v1 1/1] BaseTools/Trim: Canonicalize filepaths to fix comparison X-BeenThere: edk2-devel@lists.01.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: EDK II Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Jun 2018 18:23:14 -0000 Content-Language: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Forgot to add that I'll try out the os.path.normpath/normcase to see if thi= s will work. > -----Original Message----- > From: Chris Co > Sent: Wednesday, June 27, 2018 11:13 AM > To: 'Evan Lloyd' ; edk2-devel@lists.01.org; Liming > Gao > Cc: Leif Lindholm ; Sami Mujawar > > Subject: RE: [PATCH v1 1/1] BaseTools/Trim: Canonicalize filepaths to fix > comparison >=20 > Hi Liming, Evan, >=20 > I have attached an example of a generated DSDT.iii file which gets proces= sed > using Trim. It will give us a concrete example of what these changes do. >=20 > The first line has its #line directive stripped and the filepath saved as= the > PreProcessedFile. In GCC49, the filepath was: > "e:\\rebase\\Build\\HUMMINGBOARD_EDGE_IMX6Q_2GB\\RELEASE_GCC4 > 9xASL\\ARM\\Platform\\SolidRun\\HUMMINGBOARD_EDGE_IMX6Q_2GB\\ > AcpiTables\\AcpiTables\\OUTPUT\\.\\DSDT.i" >=20 > while in GCC5+ it is: > "e:\\rebase\\build\\hummingboard_edge_imx6q_2gb\\release_gcc5\\arm\ > \platform\\solidrun\\hummingboard_edge_imx6q_2gb\\acpitables\\acpitab > les\\output\\dsdt.i" >=20 > So there is a difference in both lowercase as well as \\.\\ before dsdt.i= . >=20 > As the Trim function goes through each line, it strips the #line directiv= e and > saves the filepath as InjectedFile, which is then compared against > PreProcessedFile > (https://github.com/tianocore/edk2/blob/975478f6bb22668efae311eb3f740 > 6e1f18411c2/BaseTools/Source/Python/Trim/Trim.py#L174) > If it is not a match, skip to next line. When using GCC5+, DSDT.iii's #l= ine > directives after the first keeps the old GCC49 filepath syntax so the str= ing > comparison always fails. An example of a line which should match line 60= 2 in > DSDT.iii. >=20 > So my changes target these two cases directly, but admittedly I'm more of= a > kernel dev than a python dev so if there is a better way to address this,= I'm all > for it. >=20 > > -----Original Message----- > > From: Evan Lloyd > > Sent: Wednesday, June 27, 2018 4:35 AM > > To: Chris Co ; edk2-devel@lists.01.org > > Cc: Liming Gao ; Leif Lindholm > > ; Sami Mujawar > > Subject: RE: [PATCH v1 1/1] BaseTools/Trim: Canonicalize filepaths to > > fix comparison > > > > Hi Chris. > > Firstly, thank you: this is a useful, pragmatic solution to an major > annoyance. > > I personally think it unfortunate that the GCC guys can't be bothered > > to fix their compiler, but ... > > > > > -----Original Message----- > > > From: edk2-devel On Behalf Of > > > Chris Co > > > Sent: 27 June 2018 04:58 > > > To: edk2-devel@lists.01.org > > > Cc: Liming Gao > > > Subject: [edk2] [PATCH v1 1/1] BaseTools/Trim: Canonicalize > > > filepaths to fix comparison > > > > > > When using Linaro GCC5+ arm-eabi toolchain on Windows, the generated > > > DSDT.iii contains a canonicalized ("\.\" removed and lower case) > > > filepath for the preprocessed DSDT.i file in the first line. > > > Due to this, when Trim.exe is called to generate DSDT.iiii, future > > > filepath comparisons against this canonicalized filepath, which > > > should match, actually fail the comparison which results in an empty > DSDT.iiii. > > > > > > Issue was first reported to Linaro here: > > > > > > https://na01.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2Fbugs. > > > > > > linaro.org%2Fshow_bug.cgi%3Fid%3D2909&data=3D02%7C01%7CChristop > > her.C > > > > > > o%40microsoft.com%7C2abbef5e0163411f08e708d5dc220678%7C72f988bf86 > > f141a > > > > > > f91ab2d7cd011db47%7C1%7C0%7C636656961074375810&sdata=3D%2FvFv > > maBD0ul > > > z89CY%2BMaOpwvpTvK9%2FxXjeZeSiXUCYH0%3D&reserved=3D0 > > > where the recommendation was to address the issue in Trim.exe. > > > > > > This patch canonicalizes and lower cases all file paths encountered > > > during trim execution on preprocessed files. Since file paths are > > > standarized, the comparison succeeds for files that should match > > > regardless of the presence of upper case or "\.\" characters in the f= ile > path. > > > > > > Contributed-under: TianoCore Contribution Agreement 1.1 > > > Signed-off-by: Christopher Co > > > Cc: Leif Lindholm > > > Cc: Yonghong Zhu > > > Cc: Liming Gao > > > --- > > > BaseTools/Source/Python/Trim/Trim.py | 2 ++ > > > 1 file changed, 2 insertions(+) > > > > > > diff --git a/BaseTools/Source/Python/Trim/Trim.py > > > b/BaseTools/Source/Python/Trim/Trim.py > > > index a74075859148..cca4e5c9694a 100644 > > > --- a/BaseTools/Source/Python/Trim/Trim.py > > > +++ b/BaseTools/Source/Python/Trim/Trim.py > > > @@ -166,6 +166,8 @@ def TrimPreprocessedFile(Source, Target, > > > ConvertHex, > > > TrimLong): > > > if len(MatchList) =3D=3D 2: > > > LineNumber =3D int(MatchList[0], 0) > > > InjectedFile =3D MatchList[1] > > > + InjectedFile =3D InjectedFile.replace("\\.\\","") > > > > [[Evan Lloyd]] I've not actually tried this yet, but it looks > > surprizing. I'd have expected InjectedFile.replace("\\.\\","\\"). > > Can I ask how it works, please? Have I missed something? > > [[Evan Lloyd]] Would it be possible to achieve the same effect with > > os.path.normpath (see Common/LongFilePathOsPath.py imported via line > > 17)? > > > > > + InjectedFile =3D InjectedFile.lower() > > [[Evan Lloyd]] Similarly, there is "os.path.normcase" - but that may > > not get the comparison working (because it converts '/' to '\' on Windo= ws). > > > > > # The first injetcted file must be the preprocessed = file itself > > > if PreprocessedFile =3D=3D "": > > > PreprocessedFile =3D InjectedFile > > > -- > > > 2.16.2.gvfs.1.33.gf5370f1 > > > > > > _______________________________________________ > > > edk2-devel mailing list > > > edk2-devel@lists.01.org > > > https://na01.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2Fli= s > > > ts > > > .01.org%2Fmailman%2Flistinfo%2Fedk2- > > devel&data=3D02%7C01%7CChristoph > > > > > > er.Co%40microsoft.com%7C2abbef5e0163411f08e708d5dc220678%7C72f988 > > bf86f > > > > > > 141af91ab2d7cd011db47%7C1%7C0%7C636656961074385815&sdata=3Duu > > S9M90lN > > > ewzqS6ScNwbSm8xC9wpdzAFVZhjEu8w2kk%3D&reserved=3D0 > > IMPORTANT NOTICE: The contents of this email and any attachments are > > confidential and may also be privileged. If you are not the intended > > recipient, please notify the sender immediately and do not disclose > > the contents to any other person, use it for any purpose, or store or > > copy the information in any medium. Thank you.