Re: [PATCH v1 1/1] BaseTools: AutoGen: Expand macros in INF files

Re: [PATCH v1 1/1] BaseTools: AutoGen: Expand macros in INF files

[Gao, Liming]

Hi, 
  This usage will break UDP (UEFI Platform Initialization Distribution Packaging Specification).  It can be downloaded from http://www.uefi.org/specifications. This spec defines the packaging format. It defines SourceFiles.Filename as the path (relative to the Module "root" directory) and filename of the file (as specified in the Distribution Content File.) That means source files must be in the module directory. 
  And, EDK2 BaseTools UPT tool follows UDP to create Package.dist based on edk2 source Package, and install Package.dist to edk2 source package. If source package has no change, UPT should always create the same package.dist file. But, this feature will break it. So, I don't suggest to add this support. 

Thanks
Liming
>-----Original Message-----
>From: kyletpugh@gmail.com [mailto:kyletpugh@gmail.com]
>Sent: Wednesday, January 25, 2017 3:16 AM
>To: edk2-devel@lists.01.org
>Cc: Laszlo Ersek <lersek@redhat.com>; Zhu, Yonghong
><yonghong.zhu@intel.com>; Gao, Liming <liming.gao@intel.com>
>Subject: [PATCH v1 1/1] BaseTools: AutoGen: Expand macros in INF files
>
>From: apianti <apianti@users.sourceforge.net>
>
>Expand environment macros used for paths and names in INF files
>
>Cc: Laszlo Ersek <lersek@redhat.com>
>Cc: Yonghong Zhu <yonghong.zhu@intel.com>
>Cc: Liming Gao <liming.gao@intel.com>
>Contributed-under: TianoCore Contribution Agreement 1.0
>Signed-off-by: apianti <apianti@users.sourceforge.net>
>---
> BaseTools/Source/Python/AutoGen/GenC.py    | 4 ++--
> BaseTools/Source/Python/AutoGen/GenMake.py | 2 +-
> 2 files changed, 3 insertions(+), 3 deletions(-)
>
>diff --git a/BaseTools/Source/Python/AutoGen/GenC.py
>b/BaseTools/Source/Python/AutoGen/GenC.py
>index 63cfe0422bbc..866e906f2f89 100644
>--- a/BaseTools/Source/Python/AutoGen/GenC.py
>+++ b/BaseTools/Source/Python/AutoGen/GenC.py
>@@ -21,7 +21,7 @@ from Common import EdkLogger
> from Common.BuildToolError import *
> from Common.DataType import *
> from Common.Misc import *
>-from Common.String import StringToArray
>+from Common.String import StringToArray, ReplaceMacro
> from StrGather import *
> from GenPcdDb import CreatePcdDatabaseCode
> from IdfClassObject import *
>@@ -1911,7 +1911,7 @@ def CreateHeaderCode(Info, AutoGenC, AutoGenH):
>         # Publish the CallerId Guid
>         #
>         AutoGenC.Append('\nGLOBAL_REMOVE_IF_UNREFERENCED GUID
>gEfiCallerIdGuid = %s;\n' % GuidStringToGuidStructureString(Info.Guid))
>-        AutoGenC.Append('\nGLOBAL_REMOVE_IF_UNREFERENCED CHAR8
>*gEfiCallerBaseName = "%s";\n' % Info.Name)
>+        AutoGenC.Append('\nGLOBAL_REMOVE_IF_UNREFERENCED CHAR8
>*gEfiCallerBaseName = "%s";\n' % ReplaceMacro(Info.Name, os.environ))
>
> ## Create common code for header file
> #
>diff --git a/BaseTools/Source/Python/AutoGen/GenMake.py
>b/BaseTools/Source/Python/AutoGen/GenMake.py
>index 51c5238fd17e..7ec59fca5271 100644
>--- a/BaseTools/Source/Python/AutoGen/GenMake.py
>+++ b/BaseTools/Source/Python/AutoGen/GenMake.py
>@@ -587,7 +587,7 @@ cleanlib:
>             FileMacroList.append("%s = %s" % (ListFileMacro, ListFileName))
>             SaveFileOnChange(
>                 ListFileName,
>-                "\n".join(self.ListFileMacros[ListFileMacro]),
>+                "\n".join(ReplaceMacros(self.ListFileMacros[ListFileMacro],
>os.environ)),
>                 False
>                 )
>
>--
>2.11.0.windows.1

Re: [PATCH v1 1/1] BaseTools: AutoGen: Expand macros in INF files

[Gao, Liming]

Pugh:
  Environment macros are allowed in DSC/FDF, but not in DEC/INF. DEC and INF belongs to Packages. They are required to be the fixed information, like the definition in C header file. I understand your usage is to change BASE_NAME defined in module INF. Why need to the different BASE_NAME?

Thanks
Liming
From: Kyle Pugh [mailto:kyletpugh@gmail.com]
Sent: Wednesday, January 25, 2017 6:58 PM
To: Gao, Liming <liming.gao@intel.com>
Cc: edk2-devel@lists.01.org; Laszlo Ersek <lersek@redhat.com>; Zhu, Yonghong <yonghong.zhu@intel.com>
Subject: Re: [PATCH v1 1/1] BaseTools: AutoGen: Expand macros in INF files

Hello,
I'm not sure I understand, the macros are expanded elsewhere I was just fixing a few places where they are not. I don't see anywhere in that document where it says anything about expanding macros. Also there is already PACKAGES_PATH, so I'm not sure what you mean. I am using macros to generate the base name of modules, there is no way around this and the modules need to built with a script since it generates extra stuff that's needed. I ran into an issue where the name was correct in all but two places, one caused failed link because it didn't expand the name macro correctly but it did expand the ARCH macro right next to it. The other was the gEfiCallerBaseName string global, it didn't expand the macros either. I have no problem still building any package and now it correctly names the binaries.

Thanks

On Tue, Jan 24, 2017 at 10:21 PM, Gao, Liming <liming.gao@intel.com<mailto:liming.gao@intel.com>> wrote:
Hi,
  This usage will break UDP (UEFI Platform Initialization Distribution Packaging Specification).  It can be downloaded from http://www.uefi.org/specifications. This spec defines the packaging format. It defines SourceFiles.Filename as the path (relative to the Module "root" directory) and filename of the file (as specified in the Distribution Content File.) That means source files must be in the module directory.
  And, EDK2 BaseTools UPT tool follows UDP to create Package.dist based on edk2 source Package, and install Package.dist to edk2 source package. If source package has no change, UPT should always create the same package.dist file. But, this feature will break it. So, I don't suggest to add this support.

Thanks
Liming
>-----Original Message-----
>From: kyletpugh@gmail.com<mailto:kyletpugh@gmail.com> [mailto:kyletpugh@gmail.com<mailto:kyletpugh@gmail.com>]
>Sent: Wednesday, January 25, 2017 3:16 AM
>To: edk2-devel@lists.01.org<mailto:edk2-devel@lists.01.org>
>Cc: Laszlo Ersek <lersek@redhat.com<mailto:lersek@redhat.com>>; Zhu, Yonghong
><yonghong.zhu@intel.com<mailto:yonghong.zhu@intel.com>>; Gao, Liming <liming.gao@intel.com<mailto:liming.gao@intel.com>>
>Subject: [PATCH v1 1/1] BaseTools: AutoGen: Expand macros in INF files
>
>From: apianti <apianti@users.sourceforge.net<mailto:apianti@users.sourceforge.net>>
>
>Expand environment macros used for paths and names in INF files
>
>Cc: Laszlo Ersek <lersek@redhat.com<mailto:lersek@redhat.com>>
>Cc: Yonghong Zhu <yonghong.zhu@intel.com<mailto:yonghong.zhu@intel.com>>
>Cc: Liming Gao <liming.gao@intel.com<mailto:liming.gao@intel.com>>
>Contributed-under: TianoCore Contribution Agreement 1.0
>Signed-off-by: apianti <apianti@users.sourceforge.net<mailto:apianti@users.sourceforge.net>>
>---
> BaseTools/Source/Python/AutoGen/GenC.py    | 4 ++--
> BaseTools/Source/Python/AutoGen/GenMake.py | 2 +-
> 2 files changed, 3 insertions(+), 3 deletions(-)
>
>diff --git a/BaseTools/Source/Python/AutoGen/GenC.py
>b/BaseTools/Source/Python/AutoGen/GenC.py
>index 63cfe0422bbc..866e906f2f89 100644
>--- a/BaseTools/Source/Python/AutoGen/GenC.py
>+++ b/BaseTools/Source/Python/AutoGen/GenC.py
>@@ -21,7 +21,7 @@ from Common import EdkLogger
> from Common.BuildToolError import *
> from Common.DataType import *
> from Common.Misc import *
>-from Common.String import StringToArray
>+from Common.String import StringToArray, ReplaceMacro
> from StrGather import *
> from GenPcdDb import CreatePcdDatabaseCode
> from IdfClassObject import *
>@@ -1911,7 +1911,7 @@ def CreateHeaderCode(Info, AutoGenC, AutoGenH):
>         # Publish the CallerId Guid
>         #
>         AutoGenC.Append('\nGLOBAL_REMOVE_IF_UNREFERENCED GUID
>gEfiCallerIdGuid = %s;\n' % GuidStringToGuidStructureString(Info.Guid))
>-        AutoGenC.Append('\nGLOBAL_REMOVE_IF_UNREFERENCED CHAR8
>*gEfiCallerBaseName = "%s";\n' % Info.Name)
>+        AutoGenC.Append('\nGLOBAL_REMOVE_IF_UNREFERENCED CHAR8
>*gEfiCallerBaseName = "%s";\n' % ReplaceMacro(Info.Name, os.environ))
>
> ## Create common code for header file
> #
>diff --git a/BaseTools/Source/Python/AutoGen/GenMake.py
>b/BaseTools/Source/Python/AutoGen/GenMake.py
>index 51c5238fd17e..7ec59fca5271 100644
>--- a/BaseTools/Source/Python/AutoGen/GenMake.py
>+++ b/BaseTools/Source/Python/AutoGen/GenMake.py
>@@ -587,7 +587,7 @@ cleanlib:
>             FileMacroList.append("%s = %s" % (ListFileMacro, ListFileName))
>             SaveFileOnChange(
>                 ListFileName,
>-                "\n".join(self.ListFileMacros[ListFileMacro]),
>+                "\n".join(ReplaceMacros(self.ListFileMacros[ListFileMacro],
>os.environ)),
>                 False
>                 )
>
>--
>2.11.0.windows.1

Re: [PATCH v1 1/1] BaseTools: AutoGen: Expand macros in INF files

[Gao, Liming]

Kyle:
  I agree the inconsistent behavior causes the confuse. Per INF spec 2.2.6 Macro Statements, Use of MACRO statements in the EDK II INF files is limited to local usage only; global or external macros are not permitted. So, BASE_NAME=Test$(ARCH) is a wrong usage. We will update BaseTools report error.

Thanks
Liming
From: Kyle Pugh [mailto:kyletpugh@gmail.com]
Sent: Saturday, February 04, 2017 12:21 AM
To: Gao, Liming <liming.gao@intel.com>
Cc: edk2-devel@lists.01.org; Laszlo Ersek <lersek@redhat.com>; Zhu, Yonghong <yonghong.zhu@intel.com>
Subject: Re: [PATCH v1 1/1] BaseTools: AutoGen: Expand macros in INF files

Liming,

The thing is that macros are expanded in DEC/INF files but only in some cases, and not even consistently. I wanted to name all my files appropriately so I don't have to create a map to copy the original name to the actual name it needs to be and instead can just copy the *.efi to the destination. Like, if you write BASE_NAME=Test$(ARCH) for a driver or application, it creates the file TestX64.efi, etc, but however if you write that for a base library then it fails to link because it creates the correct library named TestX64.lib but it can't link because its looking for Test$(ARCH).lib. This appears to happen if you use any macro, like I don't want to hard code the project name since it could change, or the version, so these are both macros. Setting $(PROJECT_NAME)$(ARCH) works exactly the same way as above. If you check out https://bugzilla.tianocore.org/show_bug.cgi?id=334, there's more explanation. It's not a super priority or anything, I can make it work without it's just more work.

Thanks,
Kyle


On Fri, Feb 3, 2017 at 12:31 AM, Gao, Liming <liming.gao@intel.com<mailto:liming.gao@intel.com>> wrote:
Pugh:
  Environment macros are allowed in DSC/FDF, but not in DEC/INF. DEC and INF belongs to Packages. They are required to be the fixed information, like the definition in C header file. I understand your usage is to change BASE_NAME defined in module INF. Why need to the different BASE_NAME?

Thanks
Liming
From: Kyle Pugh [mailto:kyletpugh@gmail.com<mailto:kyletpugh@gmail.com>]
Sent: Wednesday, January 25, 2017 6:58 PM
To: Gao, Liming <liming.gao@intel.com<mailto:liming.gao@intel.com>>
Cc: edk2-devel@lists.01.org<mailto:edk2-devel@lists.01.org>; Laszlo Ersek <lersek@redhat.com<mailto:lersek@redhat.com>>; Zhu, Yonghong <yonghong.zhu@intel.com<mailto:yonghong.zhu@intel.com>>
Subject: Re: [PATCH v1 1/1] BaseTools: AutoGen: Expand macros in INF files

Hello,
I'm not sure I understand, the macros are expanded elsewhere I was just fixing a few places where they are not. I don't see anywhere in that document where it says anything about expanding macros. Also there is already PACKAGES_PATH, so I'm not sure what you mean. I am using macros to generate the base name of modules, there is no way around this and the modules need to built with a script since it generates extra stuff that's needed. I ran into an issue where the name was correct in all but two places, one caused failed link because it didn't expand the name macro correctly but it did expand the ARCH macro right next to it. The other was the gEfiCallerBaseName string global, it didn't expand the macros either. I have no problem still building any package and now it correctly names the binaries.

Thanks

On Tue, Jan 24, 2017 at 10:21 PM, Gao, Liming <liming.gao@intel.com<mailto:liming.gao@intel.com>> wrote:
Hi,
  This usage will break UDP (UEFI Platform Initialization Distribution Packaging Specification).  It can be downloaded from http://www.uefi.org/specifications. This spec defines the packaging format. It defines SourceFiles.Filename as the path (relative to the Module "root" directory) and filename of the file (as specified in the Distribution Content File.) That means source files must be in the module directory.
  And, EDK2 BaseTools UPT tool follows UDP to create Package.dist based on edk2 source Package, and install Package.dist to edk2 source package. If source package has no change, UPT should always create the same package.dist file. But, this feature will break it. So, I don't suggest to add this support.

Thanks
Liming
>-----Original Message-----
>From: kyletpugh@gmail.com<mailto:kyletpugh@gmail.com> [mailto:kyletpugh@gmail.com<mailto:kyletpugh@gmail.com>]
>Sent: Wednesday, January 25, 2017 3:16 AM
>To: edk2-devel@lists.01.org<mailto:edk2-devel@lists.01.org>
>Cc: Laszlo Ersek <lersek@redhat.com<mailto:lersek@redhat.com>>; Zhu, Yonghong
><yonghong.zhu@intel.com<mailto:yonghong.zhu@intel.com>>; Gao, Liming <liming.gao@intel.com<mailto:liming.gao@intel.com>>
>Subject: [PATCH v1 1/1] BaseTools: AutoGen: Expand macros in INF files
>
>From: apianti <apianti@users.sourceforge.net<mailto:apianti@users.sourceforge.net>>
>
>Expand environment macros used for paths and names in INF files
>
>Cc: Laszlo Ersek <lersek@redhat.com<mailto:lersek@redhat.com>>
>Cc: Yonghong Zhu <yonghong.zhu@intel.com<mailto:yonghong.zhu@intel.com>>
>Cc: Liming Gao <liming.gao@intel.com<mailto:liming.gao@intel.com>>
>Contributed-under: TianoCore Contribution Agreement 1.0
>Signed-off-by: apianti <apianti@users.sourceforge.net<mailto:apianti@users.sourceforge.net>>
>---
> BaseTools/Source/Python/AutoGen/GenC.py    | 4 ++--
> BaseTools/Source/Python/AutoGen/GenMake.py | 2 +-
> 2 files changed, 3 insertions(+), 3 deletions(-)
>
>diff --git a/BaseTools/Source/Python/AutoGen/GenC.py
>b/BaseTools/Source/Python/AutoGen/GenC.py
>index 63cfe0422bbc..866e906f2f89 100644
>--- a/BaseTools/Source/Python/AutoGen/GenC.py
>+++ b/BaseTools/Source/Python/AutoGen/GenC.py
>@@ -21,7 +21,7 @@ from Common import EdkLogger
> from Common.BuildToolError import *
> from Common.DataType import *
> from Common.Misc import *
>-from Common.String import StringToArray
>+from Common.String import StringToArray, ReplaceMacro
> from StrGather import *
> from GenPcdDb import CreatePcdDatabaseCode
> from IdfClassObject import *
>@@ -1911,7 +1911,7 @@ def CreateHeaderCode(Info, AutoGenC, AutoGenH):
>         # Publish the CallerId Guid
>         #
>         AutoGenC.Append('\nGLOBAL_REMOVE_IF_UNREFERENCED GUID
>gEfiCallerIdGuid = %s;\n' % GuidStringToGuidStructureString(Info.Guid))
>-        AutoGenC.Append('\nGLOBAL_REMOVE_IF_UNREFERENCED CHAR8
>*gEfiCallerBaseName = "%s";\n' % Info.Name)
>+        AutoGenC.Append('\nGLOBAL_REMOVE_IF_UNREFERENCED CHAR8
>*gEfiCallerBaseName = "%s";\n' % ReplaceMacro(Info.Name, os.environ))
>
> ## Create common code for header file
> #
>diff --git a/BaseTools/Source/Python/AutoGen/GenMake.py
>b/BaseTools/Source/Python/AutoGen/GenMake.py
>index 51c5238fd17e..7ec59fca5271 100644
>--- a/BaseTools/Source/Python/AutoGen/GenMake.py
>+++ b/BaseTools/Source/Python/AutoGen/GenMake.py
>@@ -587,7 +587,7 @@ cleanlib:
>             FileMacroList.append("%s = %s" % (ListFileMacro, ListFileName))
>             SaveFileOnChange(
>                 ListFileName,
>-                "\n".join(self.ListFileMacros[ListFileMacro]),
>+                "\n".join(ReplaceMacros(self.ListFileMacros[ListFileMacro],
>os.environ)),
>                 False
>                 )
>
>--
>2.11.0.windows.1