Re: [Patch][edk2-stable201908] BaseTools: fix an incremental build issue.

["Michael D Kinney" <michael.d.kinney@intel.com>]

Liming,

This fix does not look complete to me.

I applied this patch and used FatPkg as a test.

I modified FatPkg/EnhancedFatDxe/Fat.inf removing
Defines, Source files, Library Classes, GUID, Protocols, 
and PCDs and it never rebuilds the module.

If I remove the BASE_NAME define, I get a stack trace...which does
not appear to be a regression, but a new issue.

build.py...
 : error C0DE: Unknown fatal error when processing [c:\work\github\tianocore\edk2\FatPkg\EnhancedFatDxe\Fat.inf]

(Please send email to devel@edk2.groups.io for help, attaching following call stack trace!)

(Python 3.7.3 on win32) Traceback (most recent call last):
  File "c:\work\GitHub\tianocore\edk2\BaseTools\Source\Python\build\build.py", line 2493, in Main
    MyBuild.Launch()
  File "c:\work\GitHub\tianocore\edk2\BaseTools\Source\Python\build\build.py", line 2289, in Launch
    self._MultiThreadBuildPlatform()
  File "c:\work\GitHub\tianocore\edk2\BaseTools\Source\Python\build\build.py", line 2021, in _MultiThreadBuildPlatform
    self.Progress
  File "c:\work\GitHub\tianocore\edk2\BaseTools\Source\Python\AutoGen\WorkspaceAutoGen.py", line 42, in __init__
    self._InitWorker(Workspace, MetaFile, Target, Toolchain, Arch, *args, **kwargs)
  File "c:\work\GitHub\tianocore\edk2\BaseTools\Source\Python\AutoGen\WorkspaceAutoGen.py", line 112, in _InitWorker
    self.ProcessPcdType()
  File "c:\work\GitHub\tianocore\edk2\BaseTools\Source\Python\AutoGen\WorkspaceAutoGen.py", line 240, in ProcessPcdType
    Platform.Pcds
  File "c:\work\GitHub\tianocore\edk2\BaseTools\Source\Python\Workspace\DscBuildData.py", line 1194, in Pcds
    self._Pcds = self.UpdateStructuredPcds(MODEL_PCD_TYPE_LIST, self._Pcds)
  File "c:\work\GitHub\tianocore\edk2\BaseTools\Source\Python\Workspace\DscBuildData.py", line 1506, in UpdateStructuredPcds
    for Pcd in self.DecPcds:
  File "c:\work\GitHub\tianocore\edk2\BaseTools\Source\Python\Workspace\DscBuildData.py", line 3323, in DecPcds
    self._DecPcds, self._GuidDict = GetDeclaredPcd(self, self._Bdb, self._Arch, self._Target, self._Toolchain, PkgSet)
  File "c:\work\GitHub\tianocore\edk2\BaseTools\Source\Python\Workspace\WorkspaceCommon.py", line 58, in GetDeclaredPcd
    PkgList = GetPackageList(Platform, BuildDatabase, Arch, Target, Toolchain)
  File "c:\work\GitHub\tianocore\edk2\BaseTools\Source\Python\Workspace\WorkspaceCommon.py", line 42, in GetPackageList
    PkgSet.update(Data.Packages)
  File "c:\work\GitHub\tianocore\edk2\BaseTools\Source\Python\Common\caching.py", line 28, in __get__
    Value = obj.__dict__[self._function.__name__] = self._function(obj)
  File "c:\work\GitHub\tianocore\edk2\BaseTools\Source\Python\Workspace\InfBuildData.py", line 696, in Packages
    RecordList = self._RawData[MODEL_META_DATA_PACKAGE, self._Arch, self._Platform]
  File "c:\work\GitHub\tianocore\edk2\BaseTools\Source\Python\Workspace\MetaFileParser.py", line 241, in __getitem__
    self.StartParse()
  File "c:\work\GitHub\tianocore\edk2\BaseTools\Source\Python\Workspace\MetaFileParser.py", line 260, in StartParse
    self.Start()
  File "c:\work\GitHub\tianocore\edk2\BaseTools\Source\Python\Workspace\MetaFileParser.py", line 665, in Start
    self._SectionParser[self._SectionType](self)
  File "c:\work\GitHub\tianocore\edk2\BaseTools\Source\Python\Workspace\MetaFileParser.py", line 46, in MacroParser
    Parser(self)
  File "c:\work\GitHub\tianocore\edk2\BaseTools\Source\Python\Workspace\MetaFileParser.py", line 736, in _SourceFileParser
    if self._Defines['BASE_NAME'] == 'Microcode':
KeyError: 'BASE_NAME'

If I go back to commit e8449e1d8e, the changes I tried above to remove lines
from the INF file always trigger a rebuild of the module and fail as expected
for the content I removed.

I appears that the change to reduce autogen time has many issues that are just
now being discovered.

Do we need to consider a revert of the entire feature?

Mike

> -----Original Message-----
> From: Gao, Liming
> Sent: Wednesday, August 28, 2019 9:11 AM
> To: Kinney, Michael D <michael.d.kinney@intel.com>; Feng,
> Bob C <bob.c.feng@intel.com>; devel@edk2.groups.io
> Cc: Shi, Steven <steven.shi@intel.com>; afish@apple.com;
> Laszlo Ersek (lersek@redhat.com) <lersek@redhat.com>;
> leif.lindholm@linaro.org
> Subject: RE: [Patch][edk2-stable201908] BaseTools: fix an
> incremental build issue.
> 
> Mike:
>   I add notes in BZ. Current problem is that the change
> in INF file will not trig rebuild. For example:
> 
> 1st build:
> build -p MdeModulePkg\MdeModulePkg.dsc -a IA32 -m
> MdeModulePkg\Application\HelloWorld\HelloWorld.inf
> 
> 2st build with below change
> update MdeModulePkg\Application\HelloWorld\HelloWorld.inf
> and remove MdeModulePkg/MdeModulePkg.dec from [Packages]
> section
> 
> 2st build should update Makefile and trig build. But now,
> there is no build trig.
> 
> So, this is the critical incremental build issue.
> 
> 
> Thanks
> Liming
> > -----Original Message-----
> > From: Kinney, Michael D
> > Sent: Wednesday, August 28, 2019 11:56 PM
> > To: Gao, Liming <liming.gao@intel.com>; Feng, Bob C
> > <bob.c.feng@intel.com>; devel@edk2.groups.io; Kinney,
> Michael D
> > <michael.d.kinney@intel.com>
> > Cc: Shi, Steven <steven.shi@intel.com>;
> afish@apple.com; Laszlo Ersek
> > (lersek@redhat.com) <lersek@redhat.com>;
> leif.lindholm@linaro.org
> > Subject: RE: [Patch][edk2-stable201908] BaseTools: fix
> an incremental build issue.
> >
> > Bob and Liming,
> >
> > What is an example test case that fails?  Please add to
> BZ.
> >
> > We need this information to test that this fix resolves
> the issue.
> >
> > Mike
> >
> > > -----Original Message-----
> > > From: Gao, Liming
> > > Sent: Wednesday, August 28, 2019 8:26 AM
> > > To: Feng, Bob C <bob.c.feng@intel.com>;
> devel@edk2.groups.io
> > > Cc: Shi, Steven <steven.shi@intel.com>; Kinney,
> Michael D
> > > <michael.d.kinney@intel.com>; afish@apple.com; Laszlo
> Ersek
> > > (lersek@redhat.com) <lersek@redhat.com>;
> leif.lindholm@linaro.org
> > > Subject: RE: [Patch][edk2-stable201908] BaseTools:
> fix an
> > > incremental build issue.
> > >
> > > Bob:
> > >   This is the critical regression issue for the
> incremental build. I
> > > request to catch it in this stable tag 201908.
> > >
> > >   This patch corrects the behavior. Reviewed-by:
> Liming Gao
> > > <liming.gao@intel.com>
> > >
> > > Thanks
> > > Liming
> > > > -----Original Message-----
> > > > From: Feng, Bob C
> > > > Sent: Wednesday, August 28, 2019 4:52 PM
> > > > To: devel@edk2.groups.io
> > > > Cc: Feng, Bob C <bob.c.feng@intel.com>; Gao, Liming
> > > > <liming.gao@intel.com>; Shi, Steven
> > > <steven.shi@intel.com>
> > > > Subject: [Patch][edk2-stable201908] BaseTools: fix
> an
> > > incremental build issue.
> > > >
> > > > BZ:
> https://bugzilla.tianocore.org/show_bug.cgi?id=2137
> > > >
> > > > This patch is going to fix a regression issue of
> > > incremental build
> > > > which was introduced by the commit 94459080c.
> > > > The changing on INF file will not trigger module
> > > rebuild.
> > > >
> > > > Signed-off-by: Bob Feng <bob.c.feng@intel.com>
> > > > Cc: Liming Gao <liming.gao@intel.com>
> > > > Cc: Steven Shi <steven.shi@intel.com>
> > > > ---
> > > >  BaseTools/Source/Python/AutoGen/ModuleAutoGen.py |
> 8
> > > ++------
> > > >  1 file changed, 2 insertions(+), 6 deletions(-)
> > > >
> > > > diff --git
> > > a/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py
> > > > b/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py
> > > > index 2cd0d3859e..f610365f6d 100755
> > > > ---
> a/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py
> > > > +++
> b/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py
> > > > @@ -1764,16 +1764,12 @@ class
> ModuleAutoGen(AutoGen):
> > > >              for f in self.AutoGenDepSet:
> > > >                  FileSet.add (f.Path)
> > > >
> > > >              if os.path.exists
> (self.TimeStampPath):
> > > >                  os.remove (self.TimeStampPath)
> > > > -            with tempfile.NamedTemporaryFile('w+',
> > > dir=os.path.dirname(self.TimeStampPath),
> delete=False) as
> > > tf:
> > > > -                for f in FileSet:
> > > > -                    tf.write(f)
> > > > -                    tf.write("\n")
> > > > -                tempname = tf.name
> > > > -            SaveFileOnChange(self.TimeStampPath,
> > > tempname, False)
> > > > +
> > > > +            SaveFileOnChange(self.TimeStampPath,
> > > "\n".join(FileSet),
> > > > + False)
> > > >
> > > >          # Ignore generating makefile when it is a
> > > binary module
> > > >          if self.IsBinaryModule:
> > > >              return
> > > >
> > > > --
> > > > 2.20.1.windows.1