> On Jul 16, 2020, at 6:18 AM, Laszlo Ersek wrote: > > On 07/16/20 03:41, Andrew Fish via groups.io wrote: >> I think I mentioned before on the list that I think a smart git grep that only scopes to a give target would be useful. For our internal repo we have lots of targets and lots of generations of platforms so lots of code that hits in a git grep that is not in the platform. >> >> In thinking about this it looks to me like the Build/**/AutoGenTimeStamp (module build files) files and the Build/**/AutoGen (global build files) contain a list of all the files for a project. >> >> I wrote some python that takes the build target (OvmfX64 for example) parses all the AutoGenTimeStamp and AutoGen files merges then together and removes the workspace path. All the other arguments are passed to `git grep` and only lines that match the file set are displayed. >> >> My questions are: >> 1) Did I extract all the files? > > I don't know enough of BaseTools to answer the question :) > >> 2) Are people interested in this feature? > > It seems useful. :) > >> 3) Any ideas on how to hook it in? I just dropped in the root of the git tree for development. > > We usually place such utilities under BaseTools/Scripts/, I think (e.g. > PatchCheck.py, SetupGit.py, GetMaintainer.py). I figured out if I rename my script to git-pgrep and move it to BaseTools/BinWrappers/PosixLike it becomes a new git command. $ git pgrep OvmfX64 CC_FLAGS -- *.dsc OvmfX64 only: git grep CC_FLAGS -- *.dsc OvmfPkg/OvmfPkgX64.dsc:73: GCC:RELEASE_*_*_CC_FLAGS = -DMDEPKG_NDEBUG OvmfPkg/OvmfPkgX64.dsc:74: INTEL:RELEASE_*_*_CC_FLAGS = /D MDEPKG_NDEBUG OvmfPkg/OvmfPkgX64.dsc:75: MSFT:RELEASE_*_*_CC_FLAGS = /D MDEPKG_NDEBUG OvmfPkg/OvmfPkgX64.dsc:77: GCC:*_*_*_CC_FLAGS = -mno-mmx -mno-sse OvmfPkg/OvmfPkgX64.dsc:88: MSFT:*_*_*_CC_FLAGS = /D DISABLE_NEW_DEPRECATED_INTERFACES OvmfPkg/OvmfPkgX64.dsc:89: INTEL:*_*_*_CC_FLAGS = /D DISABLE_NEW_DEPRECATED_INTERFACES OvmfPkg/OvmfPkgX64.dsc:90: GCC:*_*_*_CC_FLAGS = -D DISABLE_NEW_DEPRECATED_INTERFACES I picked pgrep for platform grep. I guess we could add $EDK_TOOLS_PATH/Scripts to the $PATH and then custom git commands could get added in $EDK_TOOLS_PATH/Scripts > >> 4) Any other cool ideas of what we could do with a file list of the project? > > Perhaps introduce a more focused utility that only produces the file > list (on stdout) so people could pass the list to whatever commands they > desire? > I added that feature. If you don’t pass any git-grep arguments you get the file list for the passed in platform. $ git pgrep OvmfX64 Conf/build_rule.txt Conf/tools_def.txt CryptoPkg/CryptoPkg.dec CryptoPkg/Library/BaseCryptLib/BaseCryptLib.inf … Thanks, Andrew Fish > Thanks! > Laszlo > >> >> Example (to give people a feel for how it works): >> $ time ./ggrep.py OvmfX64 CC_FLAGS | wc -l >> 42 >> >> real 0m0.318s >> user 0m0.243s >> sys 0m0.576s >> $ time git grep CC_FLAGS | wc -l >> 479 >> >> real 0m0.110s >> user 0m0.148s >> sys 0m0.525s >> $ ./ggrep.py OvmfX64 CC_FLAGS >> OvmfX64 only: git grep CC_FLAGS >> CryptoPkg/Library/BaseCryptLib/BaseCryptLib.inf:96: MSFT:*_*_*_CC_FLAGS = /wd4090 >> CryptoPkg/Library/BaseCryptLib/BaseCryptLib.inf:100: RVCT:*_*_ARM_CC_FLAGS = -JCryptoPkg/Include --diag_remark=1 >> CryptoPkg/Library/BaseCryptLib/BaseCryptLib.inf:102: GCC:*_CLANG35_*_CC_FLAGS = -std=c99 >> CryptoPkg/Library/BaseCryptLib/BaseCryptLib.inf:103: GCC:*_CLANG38_*_CC_FLAGS = -std=c99 >> CryptoPkg/Library/BaseCryptLib/BaseCryptLib.inf:104: GCC:*_CLANGPDB_*_CC_FLAGS = -std=c99 -Wno-error=incompatible-pointer-types >> CryptoPkg/Library/BaseCryptLib/BaseCryptLib.inf:106: XCODE:*_*_*_CC_FLAGS = -std=c99 >> CryptoPkg/Library/IntrinsicLib/IntrinsicLib.inf:55: MSFT:DEBUG_VS2003_IA32_CC_FLAGS == /nologo /c /WX /W4 /Gs32768 /Gy /D UNICODE /O1b2 /FIAutoGen.h /EHs-c- /GR- /GF /GX- /Zi /Gm >> CryptoPkg/Library/IntrinsicLib/IntrinsicLib.inf:56: MSFT:RELEASE_VS2003_IA32_CC_FLAGS == /nologo /c /WX /W4 /Gs32768 /Gy /D UNICODE /O1b2 /FIAutoGen.h /EHs-c- /GR- /GF /GX- >> CryptoPkg/Library/IntrinsicLib/IntrinsicLib.inf:57: MSFT:DEBUG_VS2003xASL_IA32_CC_FLAGS == /nologo /c /WX /W4 /Gs32768 /Gy /D UNICODE /O1b2 /FIAutoGen.h /EHs-c- /GR- /GF /GX- /Zi /Gm >> CryptoPkg/Library/IntrinsicLib/IntrinsicLib.inf:58: MSFT:RELEASE_VS2003xASL_IA32_CC_FLAGS == /nologo /c /WX /W4 /Gs32768 /Gy /D UNICODE /O1b2 /FIAutoGen.h /EHs-c- /GR- /GF /GX- >> CryptoPkg/Library/IntrinsicLib/IntrinsicLib.inf:59: MSFT:DEBUG_DDK3790_IA32_CC_FLAGS == /nologo /c /WX /Gy /Gs32768 /D UNICODE /O1b2 /FIAutoGen.h /EHs-c- /GR- /GF /Zi /Gm >> CryptoPkg/Library/IntrinsicLib/IntrinsicLib.inf:60: MSFT:RELEASE_DDK3790_IA32_CC_FLAGS == /nologo /c /WX /Gy /Gs32768 /D UNICODE /O1b2 /FIAutoGen.h /EHs-c- /GR- /GF >> CryptoPkg/Library/IntrinsicLib/IntrinsicLib.inf:61: MSFT:DEBUG_DDK3790xASL_IA32_CC_FLAGS == /nologo /c /WX /Gy /Gs32768 /D UNICODE /O1b2 /FIAutoGen.h /EHs-c- /GR- /GF /Zi /Gm >> CryptoPkg/Library/IntrinsicLib/IntrinsicLib.inf:62: MSFT:RELEASE_DDK3790xASL_IA32_CC_FLAGS == /nologo /c /WX /Gy /Gs32768 /D UNICODE /O1b2 /FIAutoGen.h /EHs-c- /GR- /GF >> CryptoPkg/Library/IntrinsicLib/IntrinsicLib.inf:63: MSFT:DEBUG_*_IA32_CC_FLAGS == /nologo /c /WX /GS- /W4 /Gs32768 /D UNICODE /O1b2 /FIAutoGen.h /EHs-c- /GR- /GF /Gy /Zi /Gm >> CryptoPkg/Library/IntrinsicLib/IntrinsicLib.inf:64: MSFT:RELEASE_*_IA32_CC_FLAGS == /nologo /c /WX /GS- /W4 /Gs32768 /D UNICODE /O1b2 /FIAutoGen.h /EHs-c- /GR- /GF >> CryptoPkg/Library/IntrinsicLib/IntrinsicLib.inf:65: MSFT:DEBUG_*_X64_CC_FLAGS == /nologo /c /WX /GS- /X /W4 /Gs32768 /D UNICODE /O1b2s /Gy /FIAutoGen.h /EHs-c- /GR- /GF /Zi /Gm >> CryptoPkg/Library/IntrinsicLib/IntrinsicLib.inf:66: MSFT:RELEASE_*_X64_CC_FLAGS == /nologo /c /WX /GS- /X /W4 /Gs32768 /D UNICODE /O1b2s /Gy /FIAutoGen.h /EHs-c- /GR- /GF >> CryptoPkg/Library/IntrinsicLib/IntrinsicLib.inf:67: INTEL:*_*_*_CC_FLAGS = /Oi- >> CryptoPkg/Library/OpensslLib/OpensslLibCrypto.inf:569: MSFT:*_*_IA32_CC_FLAGS = -U_WIN32 -U_WIN64 -U_MSC_VER $(OPENSSL_FLAGS) /wd4090 /wd4132 /wd4244 /wd4245 /wd4267 /wd4310 /wd4389 /wd4700 /wd4702 /wd4706 /wd4819 >> CryptoPkg/Library/OpensslLib/OpensslLibCrypto.inf:570: MSFT:*_*_X64_CC_FLAGS = -U_WIN32 -U_WIN64 -U_MSC_VER $(OPENSSL_FLAGS) /wd4090 /wd4132 /wd4244 /wd4245 /wd4267 /wd4306 /wd4310 /wd4700 /wd4389 /wd4702 /wd4706 /wd4819 >> CryptoPkg/Library/OpensslLib/OpensslLibCrypto.inf:572: INTEL:*_*_IA32_CC_FLAGS = -U_WIN32 -U_WIN64 -U_MSC_VER -U__ICC $(OPENSSL_FLAGS) /w >> CryptoPkg/Library/OpensslLib/OpensslLibCrypto.inf:573: INTEL:*_*_X64_CC_FLAGS = -U_WIN32 -U_WIN64 -U_MSC_VER -U__ICC $(OPENSSL_FLAGS) /w >> CryptoPkg/Library/OpensslLib/OpensslLibCrypto.inf:582: GCC:*_*_IA32_CC_FLAGS = -U_WIN32 -U_WIN64 $(OPENSSL_FLAGS) -Wno-error=maybe-uninitialized -Wno-error=unused-but-set-variable >> CryptoPkg/Library/OpensslLib/OpensslLibCrypto.inf:583: GCC:*_*_X64_CC_FLAGS = -U_WIN32 -U_WIN64 $(OPENSSL_FLAGS) -Wno-error=maybe-uninitialized -Wno-error=format -Wno-format -Wno-error=unused-but-set-variable -DNO_MSABI_VA_FUNCS >> CryptoPkg/Library/OpensslLib/OpensslLibCrypto.inf:584: GCC:*_*_ARM_CC_FLAGS = $(OPENSSL_FLAGS) -Wno-error=maybe-uninitialized -Wno-error=unused-but-set-variable >> CryptoPkg/Library/OpensslLib/OpensslLibCrypto.inf:585: GCC:*_*_AARCH64_CC_FLAGS = $(OPENSSL_FLAGS) -Wno-error=maybe-uninitialized -Wno-format -Wno-error=unused-but-set-variable >> CryptoPkg/Library/OpensslLib/OpensslLibCrypto.inf:586: GCC:*_*_RISCV64_CC_FLAGS = $(OPENSSL_FLAGS) -Wno-error=maybe-uninitialized -Wno-format -Wno-error=unused-but-set-variable >> CryptoPkg/Library/OpensslLib/OpensslLibCrypto.inf:587: GCC:*_CLANG35_*_CC_FLAGS = -std=c99 -Wno-error=uninitialized >> CryptoPkg/Library/OpensslLib/OpensslLibCrypto.inf:588: GCC:*_CLANG38_*_CC_FLAGS = -std=c99 -Wno-error=uninitialized >> CryptoPkg/Library/OpensslLib/OpensslLibCrypto.inf:589: GCC:*_CLANGPDB_*_CC_FLAGS = -std=c99 -Wno-error=uninitialized -Wno-error=incompatible-pointer-types -Wno-error=pointer-sign -Wno-error=implicit-function-declaration -Wno-error=ignored-pragma-optimize >> CryptoPkg/Library/OpensslLib/OpensslLibCrypto.inf:610: RVCT:*_*_ARM_CC_FLAGS = $(OPENSSL_FLAGS) --library_interface=aeabi_clib99 --diag_suppress=1296,1295,550,1293,111,68,177,223,144,513,188,128,546,1,3017 -JCryptoPkg/Include >> CryptoPkg/Library/OpensslLib/OpensslLibCrypto.inf:611: XCODE:*_*_IA32_CC_FLAGS = -mmmx -msse -U_WIN32 -U_WIN64 $(OPENSSL_FLAGS) -w -std=c99 -Wno-error=uninitialized >> CryptoPkg/Library/OpensslLib/OpensslLibCrypto.inf:612: XCODE:*_*_X64_CC_FLAGS = -mmmx -msse -U_WIN32 -U_WIN64 $(OPENSSL_FLAGS) -w -std=c99 -Wno-error=uninitialized >> OvmfPkg/OvmfPkgX64.dsc:73: GCC:RELEASE_*_*_CC_FLAGS = -DMDEPKG_NDEBUG >> OvmfPkg/OvmfPkgX64.dsc:74: INTEL:RELEASE_*_*_CC_FLAGS = /D MDEPKG_NDEBUG >> OvmfPkg/OvmfPkgX64.dsc:75: MSFT:RELEASE_*_*_CC_FLAGS = /D MDEPKG_NDEBUG >> OvmfPkg/OvmfPkgX64.dsc:77: GCC:*_*_*_CC_FLAGS = -mno-mmx -mno-sse >> OvmfPkg/OvmfPkgX64.dsc:88: MSFT:*_*_*_CC_FLAGS = /D DISABLE_NEW_DEPRECATED_INTERFACES >> OvmfPkg/OvmfPkgX64.dsc:89: INTEL:*_*_*_CC_FLAGS = /D DISABLE_NEW_DEPRECATED_INTERFACES >> OvmfPkg/OvmfPkgX64.dsc:90: GCC:*_*_*_CC_FLAGS = -D DISABLE_NEW_DEPRECATED_INTERFACES >> $./ggrep.py OvmfX64 CC_FLAG -- *.dsc >> OvmfX64 only: git grep CC_FLAG -- *.dsc >> OvmfPkg/OvmfPkgX64.dsc:73: GCC:RELEASE_*_*_CC_FLAGS = -DMDEPKG_NDEBUG >> OvmfPkg/OvmfPkgX64.dsc:74: INTEL:RELEASE_*_*_CC_FLAGS = /D MDEPKG_NDEBUG >> OvmfPkg/OvmfPkgX64.dsc:75: MSFT:RELEASE_*_*_CC_FLAGS = /D MDEPKG_NDEBUG >> OvmfPkg/OvmfPkgX64.dsc:77: GCC:*_*_*_CC_FLAGS = -mno-mmx -mno-sse >> OvmfPkg/OvmfPkgX64.dsc:88: MSFT:*_*_*_CC_FLAGS = /D DISABLE_NEW_DEPRECATED_INTERFACES >> OvmfPkg/OvmfPkgX64.dsc:89: INTEL:*_*_*_CC_FLAGS = /D DISABLE_NEW_DEPRECATED_INTERFACES >> OvmfPkg/OvmfPkgX64.dsc:90: GCC:*_*_*_CC_FLAGS = -D DISABLE_NEW_DEPRECATED_INTERFACES >> $ >> >> Note if we cache the combined AutoGenTimeStamp and AutoGen files into a single file in the Build direction and use that file on the next run the time goes down to: >> real 0m0.177s >> user 0m0.185s >> sys 0m0.522s >> >> If you don’t pass any `git grep` commands you get the file list back: >> $./ggrep.py OvmfX64 >> Conf/build_rule.txt >> Conf/tools_def.txt >> CryptoPkg/CryptoPkg.dec >> CryptoPkg/Library/BaseCryptLib/BaseCryptLib.inf >> CryptoPkg/Library/BaseCryptLib/Cipher/CryptAes.c >> … >> >> >> My prototype is around 135 lines of Python. >> >> Thanks, >> >> Andrew Fish >> >> >>