Re: [Patch][edk2-stable201908] BaseTools: fix an incremental build issue.

["Michael D Kinney" <michael.d.kinney@intel.com>]

Liming,

I can no longer reproduce the issues I noted below.

I agree this is a critical regression issue that must
be fixed for the edk2-stable201908 release.

Tested-by: Michael D Kinney <michael.d.kinney@intel.com>
Acked-by: Michael D Kinney <michael.d.kinney@intel.com>

Thanks,

Mike

> -----Original Message-----
> From: Kinney, Michael D
> Sent: Wednesday, August 28, 2019 10:32 AM
> To: Gao, Liming <liming.gao@intel.com>; Feng, Bob C
> <bob.c.feng@intel.com>; devel@edk2.groups.io; Kinney,
> Michael D <michael.d.kinney@intel.com>
> Cc: Shi, Steven <steven.shi@intel.com>;
> afish@apple.com; Laszlo Ersek (lersek@redhat.com)
> <lersek@redhat.com>; leif.lindholm@linaro.org
> Subject: RE: [Patch][edk2-stable201908] BaseTools: fix
> an incremental build issue.
> 
> Liming,
> 
> This fix does not look complete to me.
> 
> I applied this patch and used FatPkg as a test.
> 
> I modified FatPkg/EnhancedFatDxe/Fat.inf removing
> Defines, Source files, Library Classes, GUID,
> Protocols, and PCDs and it never rebuilds the module.
> 
> If I remove the BASE_NAME define, I get a stack
> trace...which does not appear to be a regression, but a
> new issue.
> 
> build.py...
>  : error C0DE: Unknown fatal error when processing
> [c:\work\github\tianocore\edk2\FatPkg\EnhancedFatDxe\Fa
> t.inf]
> 
> (Please send email to devel@edk2.groups.io for help,
> attaching following call stack trace!)
> 
> (Python 3.7.3 on win32) Traceback (most recent call
> last):
>   File
> "c:\work\GitHub\tianocore\edk2\BaseTools\Source\Python\
> build\build.py", line 2493, in Main
>     MyBuild.Launch()
>   File
> "c:\work\GitHub\tianocore\edk2\BaseTools\Source\Python\
> build\build.py", line 2289, in Launch
>     self._MultiThreadBuildPlatform()
>   File
> "c:\work\GitHub\tianocore\edk2\BaseTools\Source\Python\
> build\build.py", line 2021, in
> _MultiThreadBuildPlatform
>     self.Progress
>   File
> "c:\work\GitHub\tianocore\edk2\BaseTools\Source\Python\
> AutoGen\WorkspaceAutoGen.py", line 42, in __init__
>     self._InitWorker(Workspace, MetaFile, Target,
> Toolchain, Arch, *args, **kwargs)
>   File
> "c:\work\GitHub\tianocore\edk2\BaseTools\Source\Python\
> AutoGen\WorkspaceAutoGen.py", line 112, in _InitWorker
>     self.ProcessPcdType()
>   File
> "c:\work\GitHub\tianocore\edk2\BaseTools\Source\Python\
> AutoGen\WorkspaceAutoGen.py", line 240, in
> ProcessPcdType
>     Platform.Pcds
>   File
> "c:\work\GitHub\tianocore\edk2\BaseTools\Source\Python\
> Workspace\DscBuildData.py", line 1194, in Pcds
>     self._Pcds =
> self.UpdateStructuredPcds(MODEL_PCD_TYPE_LIST,
> self._Pcds)
>   File
> "c:\work\GitHub\tianocore\edk2\BaseTools\Source\Python\
> Workspace\DscBuildData.py", line 1506, in
> UpdateStructuredPcds
>     for Pcd in self.DecPcds:
>   File
> "c:\work\GitHub\tianocore\edk2\BaseTools\Source\Python\
> Workspace\DscBuildData.py", line 3323, in DecPcds
>     self._DecPcds, self._GuidDict =
> GetDeclaredPcd(self, self._Bdb, self._Arch,
> self._Target, self._Toolchain, PkgSet)
>   File
> "c:\work\GitHub\tianocore\edk2\BaseTools\Source\Python\
> Workspace\WorkspaceCommon.py", line 58, in
> GetDeclaredPcd
>     PkgList = GetPackageList(Platform, BuildDatabase,
> Arch, Target, Toolchain)
>   File
> "c:\work\GitHub\tianocore\edk2\BaseTools\Source\Python\
> Workspace\WorkspaceCommon.py", line 42, in
> GetPackageList
>     PkgSet.update(Data.Packages)
>   File
> "c:\work\GitHub\tianocore\edk2\BaseTools\Source\Python\
> Common\caching.py", line 28, in __get__
>     Value = obj.__dict__[self._function.__name__] =
> self._function(obj)
>   File
> "c:\work\GitHub\tianocore\edk2\BaseTools\Source\Python\
> Workspace\InfBuildData.py", line 696, in Packages
>     RecordList = self._RawData[MODEL_META_DATA_PACKAGE,
> self._Arch, self._Platform]
>   File
> "c:\work\GitHub\tianocore\edk2\BaseTools\Source\Python\
> Workspace\MetaFileParser.py", line 241, in __getitem__
>     self.StartParse()
>   File
> "c:\work\GitHub\tianocore\edk2\BaseTools\Source\Python\
> Workspace\MetaFileParser.py", line 260, in StartParse
>     self.Start()
>   File
> "c:\work\GitHub\tianocore\edk2\BaseTools\Source\Python\
> Workspace\MetaFileParser.py", line 665, in Start
>     self._SectionParser[self._SectionType](self)
>   File
> "c:\work\GitHub\tianocore\edk2\BaseTools\Source\Python\
> Workspace\MetaFileParser.py", line 46, in MacroParser
>     Parser(self)
>   File
> "c:\work\GitHub\tianocore\edk2\BaseTools\Source\Python\
> Workspace\MetaFileParser.py", line 736, in
> _SourceFileParser
>     if self._Defines['BASE_NAME'] == 'Microcode':
> KeyError: 'BASE_NAME'
> 
> If I go back to commit e8449e1d8e, the changes I tried
> above to remove lines from the INF file always trigger
> a rebuild of the module and fail as expected for the
> content I removed.
> 
> I appears that the change to reduce autogen time has
> many issues that are just now being discovered.
> 
> Do we need to consider a revert of the entire feature?
> 
> Mike
> 
> > -----Original Message-----
> > From: Gao, Liming
> > Sent: Wednesday, August 28, 2019 9:11 AM
> > To: Kinney, Michael D <michael.d.kinney@intel.com>;
> Feng, Bob C
> > <bob.c.feng@intel.com>; devel@edk2.groups.io
> > Cc: Shi, Steven <steven.shi@intel.com>;
> afish@apple.com; Laszlo Ersek
> > (lersek@redhat.com) <lersek@redhat.com>;
> leif.lindholm@linaro.org
> > Subject: RE: [Patch][edk2-stable201908] BaseTools:
> fix an incremental
> > build issue.
> >
> > Mike:
> >   I add notes in BZ. Current problem is that the
> change in INF file
> > will not trig rebuild. For example:
> >
> > 1st build:
> > build -p MdeModulePkg\MdeModulePkg.dsc -a IA32 -m
> > MdeModulePkg\Application\HelloWorld\HelloWorld.inf
> >
> > 2st build with below change
> > update
> MdeModulePkg\Application\HelloWorld\HelloWorld.inf
> > and remove MdeModulePkg/MdeModulePkg.dec from
> [Packages] section
> >
> > 2st build should update Makefile and trig build. But
> now, there is no
> > build trig.
> >
> > So, this is the critical incremental build issue.
> >
> >
> > Thanks
> > Liming
> > > -----Original Message-----
> > > From: Kinney, Michael D
> > > Sent: Wednesday, August 28, 2019 11:56 PM
> > > To: Gao, Liming <liming.gao@intel.com>; Feng, Bob C
> > > <bob.c.feng@intel.com>; devel@edk2.groups.io;
> Kinney,
> > Michael D
> > > <michael.d.kinney@intel.com>
> > > Cc: Shi, Steven <steven.shi@intel.com>;
> > afish@apple.com; Laszlo Ersek
> > > (lersek@redhat.com) <lersek@redhat.com>;
> > leif.lindholm@linaro.org
> > > Subject: RE: [Patch][edk2-stable201908] BaseTools:
> fix
> > an incremental build issue.
> > >
> > > Bob and Liming,
> > >
> > > What is an example test case that fails?  Please
> add to
> > BZ.
> > >
> > > We need this information to test that this fix
> resolves
> > the issue.
> > >
> > > Mike
> > >
> > > > -----Original Message-----
> > > > From: Gao, Liming
> > > > Sent: Wednesday, August 28, 2019 8:26 AM
> > > > To: Feng, Bob C <bob.c.feng@intel.com>;
> > devel@edk2.groups.io
> > > > Cc: Shi, Steven <steven.shi@intel.com>; Kinney,
> > Michael D
> > > > <michael.d.kinney@intel.com>; afish@apple.com;
> Laszlo
> > Ersek
> > > > (lersek@redhat.com) <lersek@redhat.com>;
> > leif.lindholm@linaro.org
> > > > Subject: RE: [Patch][edk2-stable201908]
> BaseTools:
> > fix an
> > > > incremental build issue.
> > > >
> > > > Bob:
> > > >   This is the critical regression issue for the
> > incremental build. I
> > > > request to catch it in this stable tag 201908.
> > > >
> > > >   This patch corrects the behavior. Reviewed-by:
> > Liming Gao
> > > > <liming.gao@intel.com>
> > > >
> > > > Thanks
> > > > Liming
> > > > > -----Original Message-----
> > > > > From: Feng, Bob C
> > > > > Sent: Wednesday, August 28, 2019 4:52 PM
> > > > > To: devel@edk2.groups.io
> > > > > Cc: Feng, Bob C <bob.c.feng@intel.com>; Gao,
> Liming
> > > > > <liming.gao@intel.com>; Shi, Steven
> > > > <steven.shi@intel.com>
> > > > > Subject: [Patch][edk2-stable201908] BaseTools:
> fix
> > an
> > > > incremental build issue.
> > > > >
> > > > > BZ:
> > https://bugzilla.tianocore.org/show_bug.cgi?id=2137
> > > > >
> > > > > This patch is going to fix a regression issue
> of
> > > > incremental build
> > > > > which was introduced by the commit 94459080c.
> > > > > The changing on INF file will not trigger
> module
> > > > rebuild.
> > > > >
> > > > > Signed-off-by: Bob Feng <bob.c.feng@intel.com>
> > > > > Cc: Liming Gao <liming.gao@intel.com>
> > > > > Cc: Steven Shi <steven.shi@intel.com>
> > > > > ---
> > > > >
> BaseTools/Source/Python/AutoGen/ModuleAutoGen.py |
> > 8
> > > > ++------
> > > > >  1 file changed, 2 insertions(+), 6 deletions(-
> )
> > > > >
> > > > > diff --git
> > > >
> a/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py
> > > > >
> b/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py
> > > > > index 2cd0d3859e..f610365f6d 100755
> > > > > ---
> > a/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py
> > > > > +++
> > b/BaseTools/Source/Python/AutoGen/ModuleAutoGen.py
> > > > > @@ -1764,16 +1764,12 @@ class
> > ModuleAutoGen(AutoGen):
> > > > >              for f in self.AutoGenDepSet:
> > > > >                  FileSet.add (f.Path)
> > > > >
> > > > >              if os.path.exists
> > (self.TimeStampPath):
> > > > >                  os.remove (self.TimeStampPath)
> > > > > -            with
> tempfile.NamedTemporaryFile('w+',
> > > > dir=os.path.dirname(self.TimeStampPath),
> > delete=False) as
> > > > tf:
> > > > > -                for f in FileSet:
> > > > > -                    tf.write(f)
> > > > > -                    tf.write("\n")
> > > > > -                tempname = tf.name
> > > > > -
> SaveFileOnChange(self.TimeStampPath,
> > > > tempname, False)
> > > > > +
> > > > > +
> SaveFileOnChange(self.TimeStampPath,
> > > > "\n".join(FileSet),
> > > > > + False)
> > > > >
> > > > >          # Ignore generating makefile when it
> is a
> > > > binary module
> > > > >          if self.IsBinaryModule:
> > > > >              return
> > > > >
> > > > > --
> > > > > 2.20.1.windows.1