[Patch][edk2-stable201908] BaseTools: fix an incremental build issue.

[Patch][edk2-stable201908] BaseTools: fix an incremental build issue.

[Bob Feng]

BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=2137

This patch is going to fix a regression issue of incremental
build which was introduced by the commit 94459080c.
The changing on INF file will not trigger module rebuild.

Signed-off-by: Bob Feng <bob.c.feng@intel.com>
Cc: Liming Gao <liming.gao@intel.com>
Cc: Steven Shi <steven.shi@intel.com>
---
 BaseTools/Source/Python/AutoGen/ModuleAutoGen.py | 8 ++------
 1 file changed, 2 insertions(+), 6 deletions(-)

diff --git a/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py b/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py
index 2cd0d3859e..f610365f6d 100755
--- a/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py
+++ b/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py
@@ -1764,16 +1764,12 @@ class ModuleAutoGen(AutoGen):
             for f in self.AutoGenDepSet:
                 FileSet.add (f.Path)
 
             if os.path.exists (self.TimeStampPath):
                 os.remove (self.TimeStampPath)
-            with tempfile.NamedTemporaryFile('w+', dir=os.path.dirname(self.TimeStampPath), delete=False) as tf:
-                for f in FileSet:
-                    tf.write(f)
-                    tf.write("\n")
-                tempname = tf.name
-            SaveFileOnChange(self.TimeStampPath, tempname, False)
+
+            SaveFileOnChange(self.TimeStampPath, "\n".join(FileSet), False)
 
         # Ignore generating makefile when it is a binary module
         if self.IsBinaryModule:
             return
 
-- 
2.20.1.windows.1

Re: [Patch][edk2-stable201908] BaseTools: fix an incremental build issue.

[Liming Gao]

Bob:
  This is the critical regression issue for the incremental build. I request to catch it in this stable tag 201908. 

  This patch corrects the behavior. Reviewed-by: Liming Gao <liming.gao@intel.com>

Thanks
Liming
> -----Original Message-----
> From: Feng, Bob C
> Sent: Wednesday, August 28, 2019 4:52 PM
> To: devel@edk2.groups.io
> Cc: Feng, Bob C <bob.c.feng@intel.com>; Gao, Liming <liming.gao@intel.com>; Shi, Steven <steven.shi@intel.com>
> Subject: [Patch][edk2-stable201908] BaseTools: fix an incremental build issue.
> 
> BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=2137
> 
> This patch is going to fix a regression issue of incremental
> build which was introduced by the commit 94459080c.
> The changing on INF file will not trigger module rebuild.
> 
> Signed-off-by: Bob Feng <bob.c.feng@intel.com>
> Cc: Liming Gao <liming.gao@intel.com>
> Cc: Steven Shi <steven.shi@intel.com>
> ---
>  BaseTools/Source/Python/AutoGen/ModuleAutoGen.py | 8 ++------
>  1 file changed, 2 insertions(+), 6 deletions(-)
> 
> diff --git a/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py b/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py
> index 2cd0d3859e..f610365f6d 100755
> --- a/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py
> +++ b/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py
> @@ -1764,16 +1764,12 @@ class ModuleAutoGen(AutoGen):
>              for f in self.AutoGenDepSet:
>                  FileSet.add (f.Path)
> 
>              if os.path.exists (self.TimeStampPath):
>                  os.remove (self.TimeStampPath)
> -            with tempfile.NamedTemporaryFile('w+', dir=os.path.dirname(self.TimeStampPath), delete=False) as tf:
> -                for f in FileSet:
> -                    tf.write(f)
> -                    tf.write("\n")
> -                tempname = tf.name
> -            SaveFileOnChange(self.TimeStampPath, tempname, False)
> +
> +            SaveFileOnChange(self.TimeStampPath, "\n".join(FileSet), False)
> 
>          # Ignore generating makefile when it is a binary module
>          if self.IsBinaryModule:
>              return
> 
> --
> 2.20.1.windows.1

Re: [Patch][edk2-stable201908] BaseTools: fix an incremental build issue.

[Michael D Kinney]

Bob and Liming,

What is an example test case that fails?  Please add to BZ.

We need this information to test that this fix resolves the
issue.

Mike

> -----Original Message-----
> From: Gao, Liming
> Sent: Wednesday, August 28, 2019 8:26 AM
> To: Feng, Bob C <bob.c.feng@intel.com>;
> devel@edk2.groups.io
> Cc: Shi, Steven <steven.shi@intel.com>; Kinney, Michael D
> <michael.d.kinney@intel.com>; afish@apple.com; Laszlo
> Ersek (lersek@redhat.com) <lersek@redhat.com>;
> leif.lindholm@linaro.org
> Subject: RE: [Patch][edk2-stable201908] BaseTools: fix an
> incremental build issue.
> 
> Bob:
>   This is the critical regression issue for the
> incremental build. I request to catch it in this stable
> tag 201908.
> 
>   This patch corrects the behavior. Reviewed-by: Liming
> Gao <liming.gao@intel.com>
> 
> Thanks
> Liming
> > -----Original Message-----
> > From: Feng, Bob C
> > Sent: Wednesday, August 28, 2019 4:52 PM
> > To: devel@edk2.groups.io
> > Cc: Feng, Bob C <bob.c.feng@intel.com>; Gao, Liming
> > <liming.gao@intel.com>; Shi, Steven
> <steven.shi@intel.com>
> > Subject: [Patch][edk2-stable201908] BaseTools: fix an
> incremental build issue.
> >
> > BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=2137
> >
> > This patch is going to fix a regression issue of
> incremental build
> > which was introduced by the commit 94459080c.
> > The changing on INF file will not trigger module
> rebuild.
> >
> > Signed-off-by: Bob Feng <bob.c.feng@intel.com>
> > Cc: Liming Gao <liming.gao@intel.com>
> > Cc: Steven Shi <steven.shi@intel.com>
> > ---
> >  BaseTools/Source/Python/AutoGen/ModuleAutoGen.py | 8
> ++------
> >  1 file changed, 2 insertions(+), 6 deletions(-)
> >
> > diff --git
> a/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py
> > b/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py
> > index 2cd0d3859e..f610365f6d 100755
> > --- a/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py
> > +++ b/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py
> > @@ -1764,16 +1764,12 @@ class ModuleAutoGen(AutoGen):
> >              for f in self.AutoGenDepSet:
> >                  FileSet.add (f.Path)
> >
> >              if os.path.exists (self.TimeStampPath):
> >                  os.remove (self.TimeStampPath)
> > -            with tempfile.NamedTemporaryFile('w+',
> dir=os.path.dirname(self.TimeStampPath), delete=False) as
> tf:
> > -                for f in FileSet:
> > -                    tf.write(f)
> > -                    tf.write("\n")
> > -                tempname = tf.name
> > -            SaveFileOnChange(self.TimeStampPath,
> tempname, False)
> > +
> > +            SaveFileOnChange(self.TimeStampPath,
> "\n".join(FileSet),
> > + False)
> >
> >          # Ignore generating makefile when it is a
> binary module
> >          if self.IsBinaryModule:
> >              return
> >
> > --
> > 2.20.1.windows.1

Re: [Patch][edk2-stable201908] BaseTools: fix an incremental build issue.

[Liming Gao]

Mike:
  I add notes in BZ. Current problem is that the change in INF file will not trig rebuild. For example:

1st build:
build -p MdeModulePkg\MdeModulePkg.dsc -a IA32 -m MdeModulePkg\Application\HelloWorld\HelloWorld.inf

2st build with below change 
update MdeModulePkg\Application\HelloWorld\HelloWorld.inf
and remove MdeModulePkg/MdeModulePkg.dec from [Packages] section

2st build should update Makefile and trig build. But now, there is no build trig. 

So, this is the critical incremental build issue.


Thanks
Liming
> -----Original Message-----
> From: Kinney, Michael D
> Sent: Wednesday, August 28, 2019 11:56 PM
> To: Gao, Liming <liming.gao@intel.com>; Feng, Bob C <bob.c.feng@intel.com>; devel@edk2.groups.io; Kinney, Michael D
> <michael.d.kinney@intel.com>
> Cc: Shi, Steven <steven.shi@intel.com>; afish@apple.com; Laszlo Ersek (lersek@redhat.com) <lersek@redhat.com>;
> leif.lindholm@linaro.org
> Subject: RE: [Patch][edk2-stable201908] BaseTools: fix an incremental build issue.
> 
> Bob and Liming,
> 
> What is an example test case that fails?  Please add to BZ.
> 
> We need this information to test that this fix resolves the
> issue.
> 
> Mike
> 
> > -----Original Message-----
> > From: Gao, Liming
> > Sent: Wednesday, August 28, 2019 8:26 AM
> > To: Feng, Bob C <bob.c.feng@intel.com>;
> > devel@edk2.groups.io
> > Cc: Shi, Steven <steven.shi@intel.com>; Kinney, Michael D
> > <michael.d.kinney@intel.com>; afish@apple.com; Laszlo
> > Ersek (lersek@redhat.com) <lersek@redhat.com>;
> > leif.lindholm@linaro.org
> > Subject: RE: [Patch][edk2-stable201908] BaseTools: fix an
> > incremental build issue.
> >
> > Bob:
> >   This is the critical regression issue for the
> > incremental build. I request to catch it in this stable
> > tag 201908.
> >
> >   This patch corrects the behavior. Reviewed-by: Liming
> > Gao <liming.gao@intel.com>
> >
> > Thanks
> > Liming
> > > -----Original Message-----
> > > From: Feng, Bob C
> > > Sent: Wednesday, August 28, 2019 4:52 PM
> > > To: devel@edk2.groups.io
> > > Cc: Feng, Bob C <bob.c.feng@intel.com>; Gao, Liming
> > > <liming.gao@intel.com>; Shi, Steven
> > <steven.shi@intel.com>
> > > Subject: [Patch][edk2-stable201908] BaseTools: fix an
> > incremental build issue.
> > >
> > > BZ: https://bugzilla.tianocore.org/show_bug.cgi?id=2137
> > >
> > > This patch is going to fix a regression issue of
> > incremental build
> > > which was introduced by the commit 94459080c.
> > > The changing on INF file will not trigger module
> > rebuild.
> > >
> > > Signed-off-by: Bob Feng <bob.c.feng@intel.com>
> > > Cc: Liming Gao <liming.gao@intel.com>
> > > Cc: Steven Shi <steven.shi@intel.com>
> > > ---
> > >  BaseTools/Source/Python/AutoGen/ModuleAutoGen.py | 8
> > ++------
> > >  1 file changed, 2 insertions(+), 6 deletions(-)
> > >
> > > diff --git
> > a/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py
> > > b/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py
> > > index 2cd0d3859e..f610365f6d 100755
> > > --- a/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py
> > > +++ b/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py
> > > @@ -1764,16 +1764,12 @@ class ModuleAutoGen(AutoGen):
> > >              for f in self.AutoGenDepSet:
> > >                  FileSet.add (f.Path)
> > >
> > >              if os.path.exists (self.TimeStampPath):
> > >                  os.remove (self.TimeStampPath)
> > > -            with tempfile.NamedTemporaryFile('w+',
> > dir=os.path.dirname(self.TimeStampPath), delete=False) as
> > tf:
> > > -                for f in FileSet:
> > > -                    tf.write(f)
> > > -                    tf.write("\n")
> > > -                tempname = tf.name
> > > -            SaveFileOnChange(self.TimeStampPath,
> > tempname, False)
> > > +
> > > +            SaveFileOnChange(self.TimeStampPath,
> > "\n".join(FileSet),
> > > + False)
> > >
> > >          # Ignore generating makefile when it is a
> > binary module
> > >          if self.IsBinaryModule:
> > >              return
> > >
> > > --
> > > 2.20.1.windows.1

Re: [Patch][edk2-stable201908] BaseTools: fix an incremental build issue.

[Michael D Kinney]

Liming,

This fix does not look complete to me.

I applied this patch and used FatPkg as a test.

I modified FatPkg/EnhancedFatDxe/Fat.inf removing
Defines, Source files, Library Classes, GUID, Protocols, 
and PCDs and it never rebuilds the module.

If I remove the BASE_NAME define, I get a stack trace...which does
not appear to be a regression, but a new issue.

build.py...
 : error C0DE: Unknown fatal error when processing [c:\work\github\tianocore\edk2\FatPkg\EnhancedFatDxe\Fat.inf]

(Please send email to devel@edk2.groups.io for help, attaching following call stack trace!)

(Python 3.7.3 on win32) Traceback (most recent call last):
  File "c:\work\GitHub\tianocore\edk2\BaseTools\Source\Python\build\build.py", line 2493, in Main
    MyBuild.Launch()
  File "c:\work\GitHub\tianocore\edk2\BaseTools\Source\Python\build\build.py", line 2289, in Launch
    self._MultiThreadBuildPlatform()
  File "c:\work\GitHub\tianocore\edk2\BaseTools\Source\Python\build\build.py", line 2021, in _MultiThreadBuildPlatform
    self.Progress
  File "c:\work\GitHub\tianocore\edk2\BaseTools\Source\Python\AutoGen\WorkspaceAutoGen.py", line 42, in __init__
    self._InitWorker(Workspace, MetaFile, Target, Toolchain, Arch, *args, **kwargs)
  File "c:\work\GitHub\tianocore\edk2\BaseTools\Source\Python\AutoGen\WorkspaceAutoGen.py", line 112, in _InitWorker
    self.ProcessPcdType()
  File "c:\work\GitHub\tianocore\edk2\BaseTools\Source\Python\AutoGen\WorkspaceAutoGen.py", line 240, in ProcessPcdType
    Platform.Pcds
  File "c:\work\GitHub\tianocore\edk2\BaseTools\Source\Python\Workspace\DscBuildData.py", line 1194, in Pcds
    self._Pcds = self.UpdateStructuredPcds(MODEL_PCD_TYPE_LIST, self._Pcds)
  File "c:\work\GitHub\tianocore\edk2\BaseTools\Source\Python\Workspace\DscBuildData.py", line 1506, in UpdateStructuredPcds
    for Pcd in self.DecPcds:
  File "c:\work\GitHub\tianocore\edk2\BaseTools\Source\Python\Workspace\DscBuildData.py", line 3323, in DecPcds
    self._DecPcds, self._GuidDict = GetDeclaredPcd(self, self._Bdb, self._Arch, self._Target, self._Toolchain, PkgSet)
  File "c:\work\GitHub\tianocore\edk2\BaseTools\Source\Python\Workspace\WorkspaceCommon.py", line 58, in GetDeclaredPcd
    PkgList = GetPackageList(Platform, BuildDatabase, Arch, Target, Toolchain)
  File "c:\work\GitHub\tianocore\edk2\BaseTools\Source\Python\Workspace\WorkspaceCommon.py", line 42, in GetPackageList
    PkgSet.update(Data.Packages)
  File "c:\work\GitHub\tianocore\edk2\BaseTools\Source\Python\Common\caching.py", line 28, in __get__
    Value = obj.__dict__[self._function.__name__] = self._function(obj)
  File "c:\work\GitHub\tianocore\edk2\BaseTools\Source\Python\Workspace\InfBuildData.py", line 696, in Packages
    RecordList = self._RawData[MODEL_META_DATA_PACKAGE, self._Arch, self._Platform]
  File "c:\work\GitHub\tianocore\edk2\BaseTools\Source\Python\Workspace\MetaFileParser.py", line 241, in __getitem__
    self.StartParse()
  File "c:\work\GitHub\tianocore\edk2\BaseTools\Source\Python\Workspace\MetaFileParser.py", line 260, in StartParse
    self.Start()
  File "c:\work\GitHub\tianocore\edk2\BaseTools\Source\Python\Workspace\MetaFileParser.py", line 665, in Start
    self._SectionParser[self._SectionType](self)
  File "c:\work\GitHub\tianocore\edk2\BaseTools\Source\Python\Workspace\MetaFileParser.py", line 46, in MacroParser
    Parser(self)
  File "c:\work\GitHub\tianocore\edk2\BaseTools\Source\Python\Workspace\MetaFileParser.py", line 736, in _SourceFileParser
    if self._Defines['BASE_NAME'] == 'Microcode':
KeyError: 'BASE_NAME'

If I go back to commit e8449e1d8e, the changes I tried above to remove lines
from the INF file always trigger a rebuild of the module and fail as expected
for the content I removed.

I appears that the change to reduce autogen time has many issues that are just
now being discovered.

Do we need to consider a revert of the entire feature?

Mike

> -----Original Message-----
> From: Gao, Liming
> Sent: Wednesday, August 28, 2019 9:11 AM
> To: Kinney, Michael D <michael.d.kinney@intel.com>; Feng,
> Bob C <bob.c.feng@intel.com>; devel@edk2.groups.io
> Cc: Shi, Steven <steven.shi@intel.com>; afish@apple.com;
> Laszlo Ersek (lersek@redhat.com) <lersek@redhat.com>;
> leif.lindholm@linaro.org
> Subject: RE: [Patch][edk2-stable201908] BaseTools: fix an
> incremental build issue.
> 
> Mike:
>   I add notes in BZ. Current problem is that the change
> in INF file will not trig rebuild. For example:
> 
> 1st build:
> build -p MdeModulePkg\MdeModulePkg.dsc -a IA32 -m
> MdeModulePkg\Application\HelloWorld\HelloWorld.inf
> 
> 2st build with below change
> update MdeModulePkg\Application\HelloWorld\HelloWorld.inf
> and remove MdeModulePkg/MdeModulePkg.dec from [Packages]
> section
> 
> 2st build should update Makefile and trig build. But now,
> there is no build trig.
> 
> So, this is the critical incremental build issue.
> 
> 
> Thanks
> Liming
> > -----Original Message-----
> > From: Kinney, Michael D
> > Sent: Wednesday, August 28, 2019 11:56 PM
> > To: Gao, Liming <liming.gao@intel.com>; Feng, Bob C
> > <bob.c.feng@intel.com>; devel@edk2.groups.io; Kinney,
> Michael D
> > <michael.d.kinney@intel.com>
> > Cc: Shi, Steven <steven.shi@intel.com>;
> afish@apple.com; Laszlo Ersek
> > (lersek@redhat.com) <lersek@redhat.com>;
> leif.lindholm@linaro.org
> > Subject: RE: [Patch][edk2-stable201908] BaseTools: fix
> an incremental build issue.
> >
> > Bob and Liming,
> >
> > What is an example test case that fails?  Please add to
> BZ.
> >
> > We need this information to test that this fix resolves
> the issue.
> >
> > Mike
> >
> > > -----Original Message-----
> > > From: Gao, Liming
> > > Sent: Wednesday, August 28, 2019 8:26 AM
> > > To: Feng, Bob C <bob.c.feng@intel.com>;
> devel@edk2.groups.io
> > > Cc: Shi, Steven <steven.shi@intel.com>; Kinney,
> Michael D
> > > <michael.d.kinney@intel.com>; afish@apple.com; Laszlo
> Ersek
> > > (lersek@redhat.com) <lersek@redhat.com>;
> leif.lindholm@linaro.org
> > > Subject: RE: [Patch][edk2-stable201908] BaseTools:
> fix an
> > > incremental build issue.
> > >
> > > Bob:
> > >   This is the critical regression issue for the
> incremental build. I
> > > request to catch it in this stable tag 201908.
> > >
> > >   This patch corrects the behavior. Reviewed-by:
> Liming Gao
> > > <liming.gao@intel.com>
> > >
> > > Thanks
> > > Liming
> > > > -----Original Message-----
> > > > From: Feng, Bob C
> > > > Sent: Wednesday, August 28, 2019 4:52 PM
> > > > To: devel@edk2.groups.io
> > > > Cc: Feng, Bob C <bob.c.feng@intel.com>; Gao, Liming
> > > > <liming.gao@intel.com>; Shi, Steven
> > > <steven.shi@intel.com>
> > > > Subject: [Patch][edk2-stable201908] BaseTools: fix
> an
> > > incremental build issue.
> > > >
> > > > BZ:
> https://bugzilla.tianocore.org/show_bug.cgi?id=2137
> > > >
> > > > This patch is going to fix a regression issue of
> > > incremental build
> > > > which was introduced by the commit 94459080c.
> > > > The changing on INF file will not trigger module
> > > rebuild.
> > > >
> > > > Signed-off-by: Bob Feng <bob.c.feng@intel.com>
> > > > Cc: Liming Gao <liming.gao@intel.com>
> > > > Cc: Steven Shi <steven.shi@intel.com>
> > > > ---
> > > >  BaseTools/Source/Python/AutoGen/ModuleAutoGen.py |
> 8
> > > ++------
> > > >  1 file changed, 2 insertions(+), 6 deletions(-)
> > > >
> > > > diff --git
> > > a/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py
> > > > b/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py
> > > > index 2cd0d3859e..f610365f6d 100755
> > > > ---
> a/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py
> > > > +++
> b/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py
> > > > @@ -1764,16 +1764,12 @@ class
> ModuleAutoGen(AutoGen):
> > > >              for f in self.AutoGenDepSet:
> > > >                  FileSet.add (f.Path)
> > > >
> > > >              if os.path.exists
> (self.TimeStampPath):
> > > >                  os.remove (self.TimeStampPath)
> > > > -            with tempfile.NamedTemporaryFile('w+',
> > > dir=os.path.dirname(self.TimeStampPath),
> delete=False) as
> > > tf:
> > > > -                for f in FileSet:
> > > > -                    tf.write(f)
> > > > -                    tf.write("\n")
> > > > -                tempname = tf.name
> > > > -            SaveFileOnChange(self.TimeStampPath,
> > > tempname, False)
> > > > +
> > > > +            SaveFileOnChange(self.TimeStampPath,
> > > "\n".join(FileSet),
> > > > + False)
> > > >
> > > >          # Ignore generating makefile when it is a
> > > binary module
> > > >          if self.IsBinaryModule:
> > > >              return
> > > >
> > > > --
> > > > 2.20.1.windows.1

Re: [edk2-devel] [Patch][edk2-stable201908] BaseTools: fix an incremental build issue.

[Michael D Kinney]

[-- Attachment #1: Type: text/plain, Size: 265 bytes --]

Liming,

I set up a fresh environment and I now see the expected behavior when
modifying INF files with this patch applied.  I can not reproduce the issues
I saw earlier today.

I will continue to do some more testing before I provide a Tested-by.

Mike


[-- Attachment #2: Type: text/html, Size: 291 bytes --]

Re: [Patch][edk2-stable201908] BaseTools: fix an incremental build issue.

[Michael D Kinney]

Liming,

I can no longer reproduce the issues I noted below.

I agree this is a critical regression issue that must
be fixed for the edk2-stable201908 release.

Tested-by: Michael D Kinney <michael.d.kinney@intel.com>
Acked-by: Michael D Kinney <michael.d.kinney@intel.com>

Thanks,

Mike

> -----Original Message-----
> From: Kinney, Michael D
> Sent: Wednesday, August 28, 2019 10:32 AM
> To: Gao, Liming <liming.gao@intel.com>; Feng, Bob C
> <bob.c.feng@intel.com>; devel@edk2.groups.io; Kinney,
> Michael D <michael.d.kinney@intel.com>
> Cc: Shi, Steven <steven.shi@intel.com>;
> afish@apple.com; Laszlo Ersek (lersek@redhat.com)
> <lersek@redhat.com>; leif.lindholm@linaro.org
> Subject: RE: [Patch][edk2-stable201908] BaseTools: fix
> an incremental build issue.
> 
> Liming,
> 
> This fix does not look complete to me.
> 
> I applied this patch and used FatPkg as a test.
> 
> I modified FatPkg/EnhancedFatDxe/Fat.inf removing
> Defines, Source files, Library Classes, GUID,
> Protocols, and PCDs and it never rebuilds the module.
> 
> If I remove the BASE_NAME define, I get a stack
> trace...which does not appear to be a regression, but a
> new issue.
> 
> build.py...
>  : error C0DE: Unknown fatal error when processing
> [c:\work\github\tianocore\edk2\FatPkg\EnhancedFatDxe\Fa
> t.inf]
> 
> (Please send email to devel@edk2.groups.io for help,
> attaching following call stack trace!)
> 
> (Python 3.7.3 on win32) Traceback (most recent call
> last):
>   File
> "c:\work\GitHub\tianocore\edk2\BaseTools\Source\Python\
> build\build.py", line 2493, in Main
>     MyBuild.Launch()
>   File
> "c:\work\GitHub\tianocore\edk2\BaseTools\Source\Python\
> build\build.py", line 2289, in Launch
>     self._MultiThreadBuildPlatform()
>   File
> "c:\work\GitHub\tianocore\edk2\BaseTools\Source\Python\
> build\build.py", line 2021, in
> _MultiThreadBuildPlatform
>     self.Progress
>   File
> "c:\work\GitHub\tianocore\edk2\BaseTools\Source\Python\
> AutoGen\WorkspaceAutoGen.py", line 42, in __init__
>     self._InitWorker(Workspace, MetaFile, Target,
> Toolchain, Arch, *args, **kwargs)
>   File
> "c:\work\GitHub\tianocore\edk2\BaseTools\Source\Python\
> AutoGen\WorkspaceAutoGen.py", line 112, in _InitWorker
>     self.ProcessPcdType()
>   File
> "c:\work\GitHub\tianocore\edk2\BaseTools\Source\Python\
> AutoGen\WorkspaceAutoGen.py", line 240, in
> ProcessPcdType
>     Platform.Pcds
>   File
> "c:\work\GitHub\tianocore\edk2\BaseTools\Source\Python\
> Workspace\DscBuildData.py", line 1194, in Pcds
>     self._Pcds =
> self.UpdateStructuredPcds(MODEL_PCD_TYPE_LIST,
> self._Pcds)
>   File
> "c:\work\GitHub\tianocore\edk2\BaseTools\Source\Python\
> Workspace\DscBuildData.py", line 1506, in
> UpdateStructuredPcds
>     for Pcd in self.DecPcds:
>   File
> "c:\work\GitHub\tianocore\edk2\BaseTools\Source\Python\
> Workspace\DscBuildData.py", line 3323, in DecPcds
>     self._DecPcds, self._GuidDict =
> GetDeclaredPcd(self, self._Bdb, self._Arch,
> self._Target, self._Toolchain, PkgSet)
>   File
> "c:\work\GitHub\tianocore\edk2\BaseTools\Source\Python\
> Workspace\WorkspaceCommon.py", line 58, in
> GetDeclaredPcd
>     PkgList = GetPackageList(Platform, BuildDatabase,
> Arch, Target, Toolchain)
>   File
> "c:\work\GitHub\tianocore\edk2\BaseTools\Source\Python\
> Workspace\WorkspaceCommon.py", line 42, in
> GetPackageList
>     PkgSet.update(Data.Packages)
>   File
> "c:\work\GitHub\tianocore\edk2\BaseTools\Source\Python\
> Common\caching.py", line 28, in __get__
>     Value = obj.__dict__[self._function.__name__] =
> self._function(obj)
>   File
> "c:\work\GitHub\tianocore\edk2\BaseTools\Source\Python\
> Workspace\InfBuildData.py", line 696, in Packages
>     RecordList = self._RawData[MODEL_META_DATA_PACKAGE,
> self._Arch, self._Platform]
>   File
> "c:\work\GitHub\tianocore\edk2\BaseTools\Source\Python\
> Workspace\MetaFileParser.py", line 241, in __getitem__
>     self.StartParse()
>   File
> "c:\work\GitHub\tianocore\edk2\BaseTools\Source\Python\
> Workspace\MetaFileParser.py", line 260, in StartParse
>     self.Start()
>   File
> "c:\work\GitHub\tianocore\edk2\BaseTools\Source\Python\
> Workspace\MetaFileParser.py", line 665, in Start
>     self._SectionParser[self._SectionType](self)
>   File
> "c:\work\GitHub\tianocore\edk2\BaseTools\Source\Python\
> Workspace\MetaFileParser.py", line 46, in MacroParser
>     Parser(self)
>   File
> "c:\work\GitHub\tianocore\edk2\BaseTools\Source\Python\
> Workspace\MetaFileParser.py", line 736, in
> _SourceFileParser
>     if self._Defines['BASE_NAME'] == 'Microcode':
> KeyError: 'BASE_NAME'
> 
> If I go back to commit e8449e1d8e, the changes I tried
> above to remove lines from the INF file always trigger
> a rebuild of the module and fail as expected for the
> content I removed.
> 
> I appears that the change to reduce autogen time has
> many issues that are just now being discovered.
> 
> Do we need to consider a revert of the entire feature?
> 
> Mike
> 
> > -----Original Message-----
> > From: Gao, Liming
> > Sent: Wednesday, August 28, 2019 9:11 AM
> > To: Kinney, Michael D <michael.d.kinney@intel.com>;
> Feng, Bob C
> > <bob.c.feng@intel.com>; devel@edk2.groups.io
> > Cc: Shi, Steven <steven.shi@intel.com>;
> afish@apple.com; Laszlo Ersek
> > (lersek@redhat.com) <lersek@redhat.com>;
> leif.lindholm@linaro.org
> > Subject: RE: [Patch][edk2-stable201908] BaseTools:
> fix an incremental
> > build issue.
> >
> > Mike:
> >   I add notes in BZ. Current problem is that the
> change in INF file
> > will not trig rebuild. For example:
> >
> > 1st build:
> > build -p MdeModulePkg\MdeModulePkg.dsc -a IA32 -m
> > MdeModulePkg\Application\HelloWorld\HelloWorld.inf
> >
> > 2st build with below change
> > update
> MdeModulePkg\Application\HelloWorld\HelloWorld.inf
> > and remove MdeModulePkg/MdeModulePkg.dec from
> [Packages] section
> >
> > 2st build should update Makefile and trig build. But
> now, there is no
> > build trig.
> >
> > So, this is the critical incremental build issue.
> >
> >
> > Thanks
> > Liming
> > > -----Original Message-----
> > > From: Kinney, Michael D
> > > Sent: Wednesday, August 28, 2019 11:56 PM
> > > To: Gao, Liming <liming.gao@intel.com>; Feng, Bob C
> > > <bob.c.feng@intel.com>; devel@edk2.groups.io;
> Kinney,
> > Michael D
> > > <michael.d.kinney@intel.com>
> > > Cc: Shi, Steven <steven.shi@intel.com>;
> > afish@apple.com; Laszlo Ersek
> > > (lersek@redhat.com) <lersek@redhat.com>;
> > leif.lindholm@linaro.org
> > > Subject: RE: [Patch][edk2-stable201908] BaseTools:
> fix
> > an incremental build issue.
> > >
> > > Bob and Liming,
> > >
> > > What is an example test case that fails?  Please
> add to
> > BZ.
> > >
> > > We need this information to test that this fix
> resolves
> > the issue.
> > >
> > > Mike
> > >
> > > > -----Original Message-----
> > > > From: Gao, Liming
> > > > Sent: Wednesday, August 28, 2019 8:26 AM
> > > > To: Feng, Bob C <bob.c.feng@intel.com>;
> > devel@edk2.groups.io
> > > > Cc: Shi, Steven <steven.shi@intel.com>; Kinney,
> > Michael D
> > > > <michael.d.kinney@intel.com>; afish@apple.com;
> Laszlo
> > Ersek
> > > > (lersek@redhat.com) <lersek@redhat.com>;
> > leif.lindholm@linaro.org
> > > > Subject: RE: [Patch][edk2-stable201908]
> BaseTools:
> > fix an
> > > > incremental build issue.
> > > >
> > > > Bob:
> > > >   This is the critical regression issue for the
> > incremental build. I
> > > > request to catch it in this stable tag 201908.
> > > >
> > > >   This patch corrects the behavior. Reviewed-by:
> > Liming Gao
> > > > <liming.gao@intel.com>
> > > >
> > > > Thanks
> > > > Liming
> > > > > -----Original Message-----
> > > > > From: Feng, Bob C
> > > > > Sent: Wednesday, August 28, 2019 4:52 PM
> > > > > To: devel@edk2.groups.io
> > > > > Cc: Feng, Bob C <bob.c.feng@intel.com>; Gao,
> Liming
> > > > > <liming.gao@intel.com>; Shi, Steven
> > > > <steven.shi@intel.com>
> > > > > Subject: [Patch][edk2-stable201908] BaseTools:
> fix
> > an
> > > > incremental build issue.
> > > > >
> > > > > BZ:
> > https://bugzilla.tianocore.org/show_bug.cgi?id=2137
> > > > >
> > > > > This patch is going to fix a regression issue
> of
> > > > incremental build
> > > > > which was introduced by the commit 94459080c.
> > > > > The changing on INF file will not trigger
> module
> > > > rebuild.
> > > > >
> > > > > Signed-off-by: Bob Feng <bob.c.feng@intel.com>
> > > > > Cc: Liming Gao <liming.gao@intel.com>
> > > > > Cc: Steven Shi <steven.shi@intel.com>
> > > > > ---
> > > > >
> BaseTools/Source/Python/AutoGen/ModuleAutoGen.py |
> > 8
> > > > ++------
> > > > >  1 file changed, 2 insertions(+), 6 deletions(-
> )
> > > > >
> > > > > diff --git
> > > >
> a/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py
> > > > >
> b/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py
> > > > > index 2cd0d3859e..f610365f6d 100755
> > > > > ---
> > a/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py
> > > > > +++
> > b/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py
> > > > > @@ -1764,16 +1764,12 @@ class
> > ModuleAutoGen(AutoGen):
> > > > >              for f in self.AutoGenDepSet:
> > > > >                  FileSet.add (f.Path)
> > > > >
> > > > >              if os.path.exists
> > (self.TimeStampPath):
> > > > >                  os.remove (self.TimeStampPath)
> > > > > -            with
> tempfile.NamedTemporaryFile('w+',
> > > > dir=os.path.dirname(self.TimeStampPath),
> > delete=False) as
> > > > tf:
> > > > > -                for f in FileSet:
> > > > > -                    tf.write(f)
> > > > > -                    tf.write("\n")
> > > > > -                tempname = tf.name
> > > > > -
> SaveFileOnChange(self.TimeStampPath,
> > > > tempname, False)
> > > > > +
> > > > > +
> SaveFileOnChange(self.TimeStampPath,
> > > > "\n".join(FileSet),
> > > > > + False)
> > > > >
> > > > >          # Ignore generating makefile when it
> is a
> > > > binary module
> > > > >          if self.IsBinaryModule:
> > > > >              return
> > > > >
> > > > > --
> > > > > 2.20.1.windows.1

Re: [Patch][edk2-stable201908] BaseTools: fix an incremental build issue.

[Bob Feng]

Pushed at df43ea6c08100487776bb0962c657d214e345ae7

Thanks,
Bob

-----Original Message-----
From: Kinney, Michael D 
Sent: Thursday, August 29, 2019 2:40 AM
To: Gao, Liming <liming.gao@intel.com>; Feng, Bob C <bob.c.feng@intel.com>; devel@edk2.groups.io; Kinney, Michael D <michael.d.kinney@intel.com>
Cc: Shi, Steven <steven.shi@intel.com>; afish@apple.com; Laszlo Ersek (lersek@redhat.com) <lersek@redhat.com>; leif.lindholm@linaro.org
Subject: RE: [Patch][edk2-stable201908] BaseTools: fix an incremental build issue.

Liming,

I can no longer reproduce the issues I noted below.

I agree this is a critical regression issue that must be fixed for the edk2-stable201908 release.

Tested-by: Michael D Kinney <michael.d.kinney@intel.com>
Acked-by: Michael D Kinney <michael.d.kinney@intel.com>

Thanks,

Mike

> -----Original Message-----
> From: Kinney, Michael D
> Sent: Wednesday, August 28, 2019 10:32 AM
> To: Gao, Liming <liming.gao@intel.com>; Feng, Bob C 
> <bob.c.feng@intel.com>; devel@edk2.groups.io; Kinney, Michael D 
> <michael.d.kinney@intel.com>
> Cc: Shi, Steven <steven.shi@intel.com>; afish@apple.com; Laszlo Ersek 
> (lersek@redhat.com) <lersek@redhat.com>; leif.lindholm@linaro.org
> Subject: RE: [Patch][edk2-stable201908] BaseTools: fix an incremental 
> build issue.
> 
> Liming,
> 
> This fix does not look complete to me.
> 
> I applied this patch and used FatPkg as a test.
> 
> I modified FatPkg/EnhancedFatDxe/Fat.inf removing Defines, Source 
> files, Library Classes, GUID, Protocols, and PCDs and it never 
> rebuilds the module.
> 
> If I remove the BASE_NAME define, I get a stack trace...which does not 
> appear to be a regression, but a new issue.
> 
> build.py...
>  : error C0DE: Unknown fatal error when processing 
> [c:\work\github\tianocore\edk2\FatPkg\EnhancedFatDxe\Fa
> t.inf]
> 
> (Please send email to devel@edk2.groups.io for help, attaching 
> following call stack trace!)
> 
> (Python 3.7.3 on win32) Traceback (most recent call
> last):
>   File
> "c:\work\GitHub\tianocore\edk2\BaseTools\Source\Python\
> build\build.py", line 2493, in Main
>     MyBuild.Launch()
>   File
> "c:\work\GitHub\tianocore\edk2\BaseTools\Source\Python\
> build\build.py", line 2289, in Launch
>     self._MultiThreadBuildPlatform()
>   File
> "c:\work\GitHub\tianocore\edk2\BaseTools\Source\Python\
> build\build.py", line 2021, in
> _MultiThreadBuildPlatform
>     self.Progress
>   File
> "c:\work\GitHub\tianocore\edk2\BaseTools\Source\Python\
> AutoGen\WorkspaceAutoGen.py", line 42, in __init__
>     self._InitWorker(Workspace, MetaFile, Target, Toolchain, Arch, 
> *args, **kwargs)
>   File
> "c:\work\GitHub\tianocore\edk2\BaseTools\Source\Python\
> AutoGen\WorkspaceAutoGen.py", line 112, in _InitWorker
>     self.ProcessPcdType()
>   File
> "c:\work\GitHub\tianocore\edk2\BaseTools\Source\Python\
> AutoGen\WorkspaceAutoGen.py", line 240, in ProcessPcdType
>     Platform.Pcds
>   File
> "c:\work\GitHub\tianocore\edk2\BaseTools\Source\Python\
> Workspace\DscBuildData.py", line 1194, in Pcds
>     self._Pcds =
> self.UpdateStructuredPcds(MODEL_PCD_TYPE_LIST,
> self._Pcds)
>   File
> "c:\work\GitHub\tianocore\edk2\BaseTools\Source\Python\
> Workspace\DscBuildData.py", line 1506, in UpdateStructuredPcds
>     for Pcd in self.DecPcds:
>   File
> "c:\work\GitHub\tianocore\edk2\BaseTools\Source\Python\
> Workspace\DscBuildData.py", line 3323, in DecPcds
>     self._DecPcds, self._GuidDict =
> GetDeclaredPcd(self, self._Bdb, self._Arch, self._Target, 
> self._Toolchain, PkgSet)
>   File
> "c:\work\GitHub\tianocore\edk2\BaseTools\Source\Python\
> Workspace\WorkspaceCommon.py", line 58, in GetDeclaredPcd
>     PkgList = GetPackageList(Platform, BuildDatabase, Arch, Target, 
> Toolchain)
>   File
> "c:\work\GitHub\tianocore\edk2\BaseTools\Source\Python\
> Workspace\WorkspaceCommon.py", line 42, in GetPackageList
>     PkgSet.update(Data.Packages)
>   File
> "c:\work\GitHub\tianocore\edk2\BaseTools\Source\Python\
> Common\caching.py", line 28, in __get__
>     Value = obj.__dict__[self._function.__name__] =
> self._function(obj)
>   File
> "c:\work\GitHub\tianocore\edk2\BaseTools\Source\Python\
> Workspace\InfBuildData.py", line 696, in Packages
>     RecordList = self._RawData[MODEL_META_DATA_PACKAGE,
> self._Arch, self._Platform]
>   File
> "c:\work\GitHub\tianocore\edk2\BaseTools\Source\Python\
> Workspace\MetaFileParser.py", line 241, in __getitem__
>     self.StartParse()
>   File
> "c:\work\GitHub\tianocore\edk2\BaseTools\Source\Python\
> Workspace\MetaFileParser.py", line 260, in StartParse
>     self.Start()
>   File
> "c:\work\GitHub\tianocore\edk2\BaseTools\Source\Python\
> Workspace\MetaFileParser.py", line 665, in Start
>     self._SectionParser[self._SectionType](self)
>   File
> "c:\work\GitHub\tianocore\edk2\BaseTools\Source\Python\
> Workspace\MetaFileParser.py", line 46, in MacroParser
>     Parser(self)
>   File
> "c:\work\GitHub\tianocore\edk2\BaseTools\Source\Python\
> Workspace\MetaFileParser.py", line 736, in _SourceFileParser
>     if self._Defines['BASE_NAME'] == 'Microcode':
> KeyError: 'BASE_NAME'
> 
> If I go back to commit e8449e1d8e, the changes I tried above to remove 
> lines from the INF file always trigger a rebuild of the module and 
> fail as expected for the content I removed.
> 
> I appears that the change to reduce autogen time has many issues that 
> are just now being discovered.
> 
> Do we need to consider a revert of the entire feature?
> 
> Mike
> 
> > -----Original Message-----
> > From: Gao, Liming
> > Sent: Wednesday, August 28, 2019 9:11 AM
> > To: Kinney, Michael D <michael.d.kinney@intel.com>;
> Feng, Bob C
> > <bob.c.feng@intel.com>; devel@edk2.groups.io
> > Cc: Shi, Steven <steven.shi@intel.com>;
> afish@apple.com; Laszlo Ersek
> > (lersek@redhat.com) <lersek@redhat.com>;
> leif.lindholm@linaro.org
> > Subject: RE: [Patch][edk2-stable201908] BaseTools:
> fix an incremental
> > build issue.
> >
> > Mike:
> >   I add notes in BZ. Current problem is that the
> change in INF file
> > will not trig rebuild. For example:
> >
> > 1st build:
> > build -p MdeModulePkg\MdeModulePkg.dsc -a IA32 -m 
> > MdeModulePkg\Application\HelloWorld\HelloWorld.inf
> >
> > 2st build with below change
> > update
> MdeModulePkg\Application\HelloWorld\HelloWorld.inf
> > and remove MdeModulePkg/MdeModulePkg.dec from
> [Packages] section
> >
> > 2st build should update Makefile and trig build. But
> now, there is no
> > build trig.
> >
> > So, this is the critical incremental build issue.
> >
> >
> > Thanks
> > Liming
> > > -----Original Message-----
> > > From: Kinney, Michael D
> > > Sent: Wednesday, August 28, 2019 11:56 PM
> > > To: Gao, Liming <liming.gao@intel.com>; Feng, Bob C 
> > > <bob.c.feng@intel.com>; devel@edk2.groups.io;
> Kinney,
> > Michael D
> > > <michael.d.kinney@intel.com>
> > > Cc: Shi, Steven <steven.shi@intel.com>;
> > afish@apple.com; Laszlo Ersek
> > > (lersek@redhat.com) <lersek@redhat.com>;
> > leif.lindholm@linaro.org
> > > Subject: RE: [Patch][edk2-stable201908] BaseTools:
> fix
> > an incremental build issue.
> > >
> > > Bob and Liming,
> > >
> > > What is an example test case that fails?  Please
> add to
> > BZ.
> > >
> > > We need this information to test that this fix
> resolves
> > the issue.
> > >
> > > Mike
> > >
> > > > -----Original Message-----
> > > > From: Gao, Liming
> > > > Sent: Wednesday, August 28, 2019 8:26 AM
> > > > To: Feng, Bob C <bob.c.feng@intel.com>;
> > devel@edk2.groups.io
> > > > Cc: Shi, Steven <steven.shi@intel.com>; Kinney,
> > Michael D
> > > > <michael.d.kinney@intel.com>; afish@apple.com;
> Laszlo
> > Ersek
> > > > (lersek@redhat.com) <lersek@redhat.com>;
> > leif.lindholm@linaro.org
> > > > Subject: RE: [Patch][edk2-stable201908]
> BaseTools:
> > fix an
> > > > incremental build issue.
> > > >
> > > > Bob:
> > > >   This is the critical regression issue for the
> > incremental build. I
> > > > request to catch it in this stable tag 201908.
> > > >
> > > >   This patch corrects the behavior. Reviewed-by:
> > Liming Gao
> > > > <liming.gao@intel.com>
> > > >
> > > > Thanks
> > > > Liming
> > > > > -----Original Message-----
> > > > > From: Feng, Bob C
> > > > > Sent: Wednesday, August 28, 2019 4:52 PM
> > > > > To: devel@edk2.groups.io
> > > > > Cc: Feng, Bob C <bob.c.feng@intel.com>; Gao,
> Liming
> > > > > <liming.gao@intel.com>; Shi, Steven
> > > > <steven.shi@intel.com>
> > > > > Subject: [Patch][edk2-stable201908] BaseTools:
> fix
> > an
> > > > incremental build issue.
> > > > >
> > > > > BZ:
> > https://bugzilla.tianocore.org/show_bug.cgi?id=2137
> > > > >
> > > > > This patch is going to fix a regression issue
> of
> > > > incremental build
> > > > > which was introduced by the commit 94459080c.
> > > > > The changing on INF file will not trigger
> module
> > > > rebuild.
> > > > >
> > > > > Signed-off-by: Bob Feng <bob.c.feng@intel.com>
> > > > > Cc: Liming Gao <liming.gao@intel.com>
> > > > > Cc: Steven Shi <steven.shi@intel.com>
> > > > > ---
> > > > >
> BaseTools/Source/Python/AutoGen/ModuleAutoGen.py |
> > 8
> > > > ++------
> > > > >  1 file changed, 2 insertions(+), 6 deletions(-
> )
> > > > >
> > > > > diff --git
> > > >
> a/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py
> > > > >
> b/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py
> > > > > index 2cd0d3859e..f610365f6d 100755
> > > > > ---
> > a/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py
> > > > > +++
> > b/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py
> > > > > @@ -1764,16 +1764,12 @@ class
> > ModuleAutoGen(AutoGen):
> > > > >              for f in self.AutoGenDepSet:
> > > > >                  FileSet.add (f.Path)
> > > > >
> > > > >              if os.path.exists
> > (self.TimeStampPath):
> > > > >                  os.remove (self.TimeStampPath)
> > > > > -            with
> tempfile.NamedTemporaryFile('w+',
> > > > dir=os.path.dirname(self.TimeStampPath),
> > delete=False) as
> > > > tf:
> > > > > -                for f in FileSet:
> > > > > -                    tf.write(f)
> > > > > -                    tf.write("\n")
> > > > > -                tempname = tf.name
> > > > > -
> SaveFileOnChange(self.TimeStampPath,
> > > > tempname, False)
> > > > > +
> > > > > +
> SaveFileOnChange(self.TimeStampPath,
> > > > "\n".join(FileSet),
> > > > > + False)
> > > > >
> > > > >          # Ignore generating makefile when it
> is a
> > > > binary module
> > > > >          if self.IsBinaryModule:
> > > > >              return
> > > > >
> > > > > --
> > > > > 2.20.1.windows.1