[PATCH v2 0/1] BaseTools/Trim: Normalize filepaths to fix comparisons on Windows

[PATCH v2 0/1] BaseTools/Trim: Normalize filepaths to fix comparisons on Windows

[Chris Co]

REF: https://github.com/christopherco/edk2/tree/trim_gcc_v2

v1: https://lists.01.org/pipermail/edk2-devel/2018-June/026562.html
 
new in v2: Use os module to normalize filepaths instead of direct string manipulation

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Christopher Co <christopher.co@microsoft.com>
Cc: Leif Lindholm <leif.lindholm@linaro.org>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Evan Lloyd <Evan.Lloyd@arm.com>

Christopher Co (1):
  BaseTools/Trim: Normalize filepaths to fix comparisons on Windows

 BaseTools/Source/Python/Trim/Trim.py | 2 ++
 1 file changed, 2 insertions(+)

-- 
2.16.2.gvfs.1.33.gf5370f1

[PATCH v2 1/1] BaseTools/Trim: Normalize filepaths to fix comparisons on Windows

[Chris Co]

When using Linaro GCC5+ arm-eabi toolchain on Windows, the generated
DSDT.iii contains a canonicalized ("\.\" removed and lower case)
filepath for the preprocessed DSDT.i file in the first line.
Trim.exe is called on DSDT.iii to generate DSDT.iiii, which does a
line for line comparison of filepaths encountered to the preprocessed
DSDT.i filepath found in the first line to determine what lines to
place in DSDT.iiii. Since the DSDT.i filepath is canonicalized and
all later filepaths in DSDT.iii are not canonicalized, all comparisons
fail and the result is in an empty DSDT.iiii.

Issue was first reported to Linaro here:
https://bugs.linaro.org/show_bug.cgi?id=2909
where the recommendation was to address the issue in Trim.exe.

This patch normalizes the case and pathname of all filepaths
encountered during Trim.exe execution on preprocessed files.  This
fixes comparisons of filepaths that contain mismatching case on
case-insensitive filesystems, redundant separators, and uplevel
references.

Contributed-under: TianoCore Contribution Agreement 1.1
Signed-off-by: Christopher Co <christopher.co@microsoft.com>
Cc: Leif Lindholm <leif.lindholm@linaro.org>
Cc: Yonghong Zhu <yonghong.zhu@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
---
 BaseTools/Source/Python/Trim/Trim.py | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/BaseTools/Source/Python/Trim/Trim.py b/BaseTools/Source/Python/Trim/Trim.py
index 76944c0e25b3..b46d507b4e55 100644
--- a/BaseTools/Source/Python/Trim/Trim.py
+++ b/BaseTools/Source/Python/Trim/Trim.py
@@ -166,6 +166,8 @@ def TrimPreprocessedFile(Source, Target, ConvertHex, TrimLong):
             if len(MatchList) == 2:
                 LineNumber = int(MatchList[0], 0)
                 InjectedFile = MatchList[1]
+                InjectedFile = os.path.normpath(InjectedFile)
+                InjectedFile = os.path.normcase(InjectedFile)
                 # The first injetcted file must be the preprocessed file itself
                 if PreprocessedFile == "":
                     PreprocessedFile = InjectedFile
-- 
2.16.2.gvfs.1.33.gf5370f1

Re: [PATCH v2 1/1] BaseTools/Trim: Normalize filepaths to fix comparisons on Windows

[Gao, Liming]

Reviewed-by: Liming Gao <liming.gao@intel.com>

If no other comment, I will help push this change. 

>-----Original Message-----
>From: Chris Co [mailto:Christopher.Co@microsoft.com]
>Sent: Friday, June 29, 2018 3:31 AM
>To: edk2-devel@lists.01.org
>Cc: Leif Lindholm <leif.lindholm@linaro.org>; Zhu, Yonghong
><yonghong.zhu@intel.com>; Gao, Liming <liming.gao@intel.com>
>Subject: [PATCH v2 1/1] BaseTools/Trim: Normalize filepaths to fix
>comparisons on Windows
>
>When using Linaro GCC5+ arm-eabi toolchain on Windows, the generated
>DSDT.iii contains a canonicalized ("\.\" removed and lower case)
>filepath for the preprocessed DSDT.i file in the first line.
>Trim.exe is called on DSDT.iii to generate DSDT.iiii, which does a
>line for line comparison of filepaths encountered to the preprocessed
>DSDT.i filepath found in the first line to determine what lines to
>place in DSDT.iiii. Since the DSDT.i filepath is canonicalized and
>all later filepaths in DSDT.iii are not canonicalized, all comparisons
>fail and the result is in an empty DSDT.iiii.
>
>Issue was first reported to Linaro here:
>https://bugs.linaro.org/show_bug.cgi?id=2909
>where the recommendation was to address the issue in Trim.exe.
>
>This patch normalizes the case and pathname of all filepaths
>encountered during Trim.exe execution on preprocessed files.  This
>fixes comparisons of filepaths that contain mismatching case on
>case-insensitive filesystems, redundant separators, and uplevel
>references.
>
>Contributed-under: TianoCore Contribution Agreement 1.1
>Signed-off-by: Christopher Co <christopher.co@microsoft.com>
>Cc: Leif Lindholm <leif.lindholm@linaro.org>
>Cc: Yonghong Zhu <yonghong.zhu@intel.com>
>Cc: Liming Gao <liming.gao@intel.com>
>---
> BaseTools/Source/Python/Trim/Trim.py | 2 ++
> 1 file changed, 2 insertions(+)
>
>diff --git a/BaseTools/Source/Python/Trim/Trim.py
>b/BaseTools/Source/Python/Trim/Trim.py
>index 76944c0e25b3..b46d507b4e55 100644
>--- a/BaseTools/Source/Python/Trim/Trim.py
>+++ b/BaseTools/Source/Python/Trim/Trim.py
>@@ -166,6 +166,8 @@ def TrimPreprocessedFile(Source, Target, ConvertHex,
>TrimLong):
>             if len(MatchList) == 2:
>                 LineNumber = int(MatchList[0], 0)
>                 InjectedFile = MatchList[1]
>+                InjectedFile = os.path.normpath(InjectedFile)
>+                InjectedFile = os.path.normcase(InjectedFile)
>                 # The first injetcted file must be the preprocessed file itself
>                 if PreprocessedFile == "":
>                     PreprocessedFile = InjectedFile
>--
>2.16.2.gvfs.1.33.gf5370f1