File in the build question. I'm working on a smart git grep

["Andrew Fish" <afish@apple.com>]

[-- Attachment #1: Type: text/plain, Size: 8796 bytes --]

I think I mentioned before on the list that I think a smart git grep that only scopes to a give target would be useful. For our internal repo we have lots of targets and lots of generations of platforms so lots of code that hits in a git grep that is not in the platform. 

In thinking about this it looks to me like the Build/**/AutoGenTimeStamp (module build files) files and the Build/**/AutoGen (global build files) contain a list of all the files for a project.

I wrote some python that takes the build target (OvmfX64 for example) parses all the  AutoGenTimeStamp and AutoGen files merges then together and removes the workspace path. All the other arguments are passed to `git grep` and only lines that match the file set are displayed. 

My questions are:
1) Did I extract all the files?
2) Are people interested in this feature?
3) Any ideas on how to hook it in? I just dropped in the root of the git tree for development. 
4) Any other cool ideas of what we could do with a file list of the project?

Example (to give people a feel for how it works):
$ time ./ggrep.py OvmfX64 CC_FLAGS | wc -l
      42

real	0m0.318s
user	0m0.243s
sys	0m0.576s
$ time git grep CC_FLAGS | wc -l
     479

real	0m0.110s
user	0m0.148s
sys	0m0.525s
$ ./ggrep.py OvmfX64 CC_FLAGS
OvmfX64 only: git grep CC_FLAGS
CryptoPkg/Library/BaseCryptLib/BaseCryptLib.inf:96:  MSFT:*_*_*_CC_FLAGS = /wd4090
CryptoPkg/Library/BaseCryptLib/BaseCryptLib.inf:100:  RVCT:*_*_ARM_CC_FLAGS = -JCryptoPkg/Include --diag_remark=1
CryptoPkg/Library/BaseCryptLib/BaseCryptLib.inf:102:  GCC:*_CLANG35_*_CC_FLAGS = -std=c99
CryptoPkg/Library/BaseCryptLib/BaseCryptLib.inf:103:  GCC:*_CLANG38_*_CC_FLAGS = -std=c99
CryptoPkg/Library/BaseCryptLib/BaseCryptLib.inf:104:  GCC:*_CLANGPDB_*_CC_FLAGS = -std=c99 -Wno-error=incompatible-pointer-types
CryptoPkg/Library/BaseCryptLib/BaseCryptLib.inf:106:  XCODE:*_*_*_CC_FLAGS = -std=c99
CryptoPkg/Library/IntrinsicLib/IntrinsicLib.inf:55:   MSFT:DEBUG_VS2003_IA32_CC_FLAGS        == /nologo /c /WX /W4 /Gs32768 /Gy /D UNICODE /O1b2 /FIAutoGen.h /EHs-c- /GR- /GF /GX- /Zi /Gm
CryptoPkg/Library/IntrinsicLib/IntrinsicLib.inf:56:   MSFT:RELEASE_VS2003_IA32_CC_FLAGS      == /nologo /c /WX /W4 /Gs32768 /Gy /D UNICODE /O1b2 /FIAutoGen.h /EHs-c- /GR- /GF /GX-
CryptoPkg/Library/IntrinsicLib/IntrinsicLib.inf:57:   MSFT:DEBUG_VS2003xASL_IA32_CC_FLAGS    == /nologo /c /WX /W4 /Gs32768 /Gy /D UNICODE /O1b2 /FIAutoGen.h /EHs-c- /GR- /GF /GX- /Zi /Gm
CryptoPkg/Library/IntrinsicLib/IntrinsicLib.inf:58:   MSFT:RELEASE_VS2003xASL_IA32_CC_FLAGS  == /nologo /c /WX /W4 /Gs32768 /Gy /D UNICODE /O1b2 /FIAutoGen.h /EHs-c- /GR- /GF /GX-
CryptoPkg/Library/IntrinsicLib/IntrinsicLib.inf:59:   MSFT:DEBUG_DDK3790_IA32_CC_FLAGS       == /nologo /c /WX /Gy /Gs32768 /D UNICODE /O1b2 /FIAutoGen.h /EHs-c- /GR- /GF /Zi /Gm
CryptoPkg/Library/IntrinsicLib/IntrinsicLib.inf:60:   MSFT:RELEASE_DDK3790_IA32_CC_FLAGS     == /nologo /c /WX /Gy /Gs32768 /D UNICODE /O1b2 /FIAutoGen.h /EHs-c- /GR- /GF
CryptoPkg/Library/IntrinsicLib/IntrinsicLib.inf:61:   MSFT:DEBUG_DDK3790xASL_IA32_CC_FLAGS   == /nologo /c /WX /Gy /Gs32768 /D UNICODE /O1b2 /FIAutoGen.h /EHs-c- /GR- /GF /Zi /Gm
CryptoPkg/Library/IntrinsicLib/IntrinsicLib.inf:62:   MSFT:RELEASE_DDK3790xASL_IA32_CC_FLAGS == /nologo /c /WX /Gy /Gs32768 /D UNICODE /O1b2 /FIAutoGen.h /EHs-c- /GR- /GF
CryptoPkg/Library/IntrinsicLib/IntrinsicLib.inf:63:   MSFT:DEBUG_*_IA32_CC_FLAGS             == /nologo /c /WX /GS- /W4 /Gs32768 /D UNICODE /O1b2 /FIAutoGen.h /EHs-c- /GR- /GF /Gy /Zi /Gm
CryptoPkg/Library/IntrinsicLib/IntrinsicLib.inf:64:   MSFT:RELEASE_*_IA32_CC_FLAGS           == /nologo /c /WX /GS- /W4 /Gs32768 /D UNICODE /O1b2 /FIAutoGen.h /EHs-c- /GR- /GF
CryptoPkg/Library/IntrinsicLib/IntrinsicLib.inf:65:   MSFT:DEBUG_*_X64_CC_FLAGS              == /nologo /c /WX /GS- /X /W4 /Gs32768 /D UNICODE /O1b2s /Gy /FIAutoGen.h /EHs-c- /GR- /GF /Zi /Gm
CryptoPkg/Library/IntrinsicLib/IntrinsicLib.inf:66:   MSFT:RELEASE_*_X64_CC_FLAGS            == /nologo /c /WX /GS- /X /W4 /Gs32768 /D UNICODE /O1b2s /Gy /FIAutoGen.h /EHs-c- /GR- /GF
CryptoPkg/Library/IntrinsicLib/IntrinsicLib.inf:67:  INTEL:*_*_*_CC_FLAGS                    =  /Oi-
CryptoPkg/Library/OpensslLib/OpensslLibCrypto.inf:569:  MSFT:*_*_IA32_CC_FLAGS   = -U_WIN32 -U_WIN64 -U_MSC_VER $(OPENSSL_FLAGS) /wd4090 /wd4132 /wd4244 /wd4245 /wd4267 /wd4310 /wd4389 /wd4700 /wd4702 /wd4706 /wd4819
CryptoPkg/Library/OpensslLib/OpensslLibCrypto.inf:570:  MSFT:*_*_X64_CC_FLAGS    = -U_WIN32 -U_WIN64 -U_MSC_VER $(OPENSSL_FLAGS) /wd4090 /wd4132 /wd4244 /wd4245 /wd4267 /wd4306 /wd4310 /wd4700 /wd4389 /wd4702 /wd4706 /wd4819
CryptoPkg/Library/OpensslLib/OpensslLibCrypto.inf:572:  INTEL:*_*_IA32_CC_FLAGS  = -U_WIN32 -U_WIN64 -U_MSC_VER -U__ICC $(OPENSSL_FLAGS) /w
CryptoPkg/Library/OpensslLib/OpensslLibCrypto.inf:573:  INTEL:*_*_X64_CC_FLAGS   = -U_WIN32 -U_WIN64 -U_MSC_VER -U__ICC $(OPENSSL_FLAGS) /w
CryptoPkg/Library/OpensslLib/OpensslLibCrypto.inf:582:  GCC:*_*_IA32_CC_FLAGS    = -U_WIN32 -U_WIN64 $(OPENSSL_FLAGS) -Wno-error=maybe-uninitialized -Wno-error=unused-but-set-variable
CryptoPkg/Library/OpensslLib/OpensslLibCrypto.inf:583:  GCC:*_*_X64_CC_FLAGS     = -U_WIN32 -U_WIN64 $(OPENSSL_FLAGS) -Wno-error=maybe-uninitialized -Wno-error=format -Wno-format -Wno-error=unused-but-set-variable -DNO_MSABI_VA_FUNCS
CryptoPkg/Library/OpensslLib/OpensslLibCrypto.inf:584:  GCC:*_*_ARM_CC_FLAGS     = $(OPENSSL_FLAGS) -Wno-error=maybe-uninitialized -Wno-error=unused-but-set-variable
CryptoPkg/Library/OpensslLib/OpensslLibCrypto.inf:585:  GCC:*_*_AARCH64_CC_FLAGS = $(OPENSSL_FLAGS) -Wno-error=maybe-uninitialized -Wno-format -Wno-error=unused-but-set-variable
CryptoPkg/Library/OpensslLib/OpensslLibCrypto.inf:586:  GCC:*_*_RISCV64_CC_FLAGS = $(OPENSSL_FLAGS) -Wno-error=maybe-uninitialized -Wno-format -Wno-error=unused-but-set-variable
CryptoPkg/Library/OpensslLib/OpensslLibCrypto.inf:587:  GCC:*_CLANG35_*_CC_FLAGS = -std=c99 -Wno-error=uninitialized
CryptoPkg/Library/OpensslLib/OpensslLibCrypto.inf:588:  GCC:*_CLANG38_*_CC_FLAGS = -std=c99 -Wno-error=uninitialized
CryptoPkg/Library/OpensslLib/OpensslLibCrypto.inf:589:  GCC:*_CLANGPDB_*_CC_FLAGS = -std=c99 -Wno-error=uninitialized -Wno-error=incompatible-pointer-types -Wno-error=pointer-sign -Wno-error=implicit-function-declaration -Wno-error=ignored-pragma-optimize
CryptoPkg/Library/OpensslLib/OpensslLibCrypto.inf:610:  RVCT:*_*_ARM_CC_FLAGS     = $(OPENSSL_FLAGS) --library_interface=aeabi_clib99 --diag_suppress=1296,1295,550,1293,111,68,177,223,144,513,188,128,546,1,3017 -JCryptoPkg/Include
CryptoPkg/Library/OpensslLib/OpensslLibCrypto.inf:611:  XCODE:*_*_IA32_CC_FLAGS   = -mmmx -msse -U_WIN32 -U_WIN64 $(OPENSSL_FLAGS) -w -std=c99 -Wno-error=uninitialized
CryptoPkg/Library/OpensslLib/OpensslLibCrypto.inf:612:  XCODE:*_*_X64_CC_FLAGS    = -mmmx -msse -U_WIN32 -U_WIN64 $(OPENSSL_FLAGS) -w -std=c99 -Wno-error=uninitialized
OvmfPkg/OvmfPkgX64.dsc:73:  GCC:RELEASE_*_*_CC_FLAGS             = -DMDEPKG_NDEBUG
OvmfPkg/OvmfPkgX64.dsc:74:  INTEL:RELEASE_*_*_CC_FLAGS           = /D MDEPKG_NDEBUG
OvmfPkg/OvmfPkgX64.dsc:75:  MSFT:RELEASE_*_*_CC_FLAGS            = /D MDEPKG_NDEBUG
OvmfPkg/OvmfPkgX64.dsc:77:  GCC:*_*_*_CC_FLAGS                   = -mno-mmx -mno-sse
OvmfPkg/OvmfPkgX64.dsc:88:  MSFT:*_*_*_CC_FLAGS = /D DISABLE_NEW_DEPRECATED_INTERFACES
OvmfPkg/OvmfPkgX64.dsc:89:  INTEL:*_*_*_CC_FLAGS = /D DISABLE_NEW_DEPRECATED_INTERFACES
OvmfPkg/OvmfPkgX64.dsc:90:  GCC:*_*_*_CC_FLAGS = -D DISABLE_NEW_DEPRECATED_INTERFACES
$./ggrep.py OvmfX64 CC_FLAG -- *.dsc
OvmfX64 only: git grep CC_FLAG -- *.dsc
OvmfPkg/OvmfPkgX64.dsc:73:  GCC:RELEASE_*_*_CC_FLAGS             = -DMDEPKG_NDEBUG
OvmfPkg/OvmfPkgX64.dsc:74:  INTEL:RELEASE_*_*_CC_FLAGS           = /D MDEPKG_NDEBUG
OvmfPkg/OvmfPkgX64.dsc:75:  MSFT:RELEASE_*_*_CC_FLAGS            = /D MDEPKG_NDEBUG
OvmfPkg/OvmfPkgX64.dsc:77:  GCC:*_*_*_CC_FLAGS                   = -mno-mmx -mno-sse
OvmfPkg/OvmfPkgX64.dsc:88:  MSFT:*_*_*_CC_FLAGS = /D DISABLE_NEW_DEPRECATED_INTERFACES
OvmfPkg/OvmfPkgX64.dsc:89:  INTEL:*_*_*_CC_FLAGS = /D DISABLE_NEW_DEPRECATED_INTERFACES
OvmfPkg/OvmfPkgX64.dsc:90:  GCC:*_*_*_CC_FLAGS = -D DISABLE_NEW_DEPRECATED_INTERFACES
$

Note if we cache the combined AutoGenTimeStamp and AutoGen files into a single file in the Build direction and use that file on the next run the time goes down to:
real	0m0.177s
user	0m0.185s
sys	0m0.522s

If you don’t pass any `git grep` commands you get the file list back:
$./ggrep.py OvmfX64
Conf/build_rule.txt
Conf/tools_def.txt
CryptoPkg/CryptoPkg.dec
CryptoPkg/Library/BaseCryptLib/BaseCryptLib.inf
CryptoPkg/Library/BaseCryptLib/Cipher/CryptAes.c
…


My prototype is around 135 lines of Python.

Thanks,

Andrew Fish


[-- Attachment #2: Type: text/html, Size: 27282 bytes --]