From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from ma1-aaemail-dr-lapp01.apple.com (ma1-aaemail-dr-lapp01.apple.com [17.171.2.60]) by mx.groups.io with SMTP id smtpd.web12.4673.1595103821280553872 for ; Sat, 18 Jul 2020 13:23:42 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@apple.com header.s=20180706 header.b=sV3pR2Rz; spf=pass (domain: apple.com, ip: 17.171.2.60, mailfrom: afish@apple.com) Received: from pps.filterd (ma1-aaemail-dr-lapp01.apple.com [127.0.0.1]) by ma1-aaemail-dr-lapp01.apple.com (8.16.0.42/8.16.0.42) with SMTP id 06IKNGcS038778; Sat, 18 Jul 2020 13:23:39 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=apple.com; h=from : message-id : content-type : mime-version : subject : date : in-reply-to : cc : to : references; s=20180706; bh=3c4OHwLH5RfXNqCaApXlKjJmGBgmYC9brp3qdWrYGgw=; b=sV3pR2RzFPzebTnFY2EKq1/eZWy4RN27b6q23RYK+bO1baMKQ935QffL4pHh1j27x3+C 9iactRZ3RyZmERFy2TDxK6eArXKeZMVIbb7y1hyKxTtfPpI/rvhxdHz0ItSWI+ISfuZ3 zjTG5VjQyq/PCFOwVs3lbdNafnVROqXYMijnVk9YKMZfCWf7jzxbe79plX0P2hsEIHhu UKRDaeEspPeO1sptirVwgn1X1/A3St0BwNqdyjZ/a94evActmjGm8U29SVp1bJLxN7rV uqDg4yh9VcEvOweMB+Ez/h0eRIvZLMvWJ5lqZIad5PuRB/vTBQfFmzCipxbnd7jX776d pA== Received: from rn-mailsvcp-mta-lapp03.rno.apple.com (rn-mailsvcp-mta-lapp03.rno.apple.com [10.225.203.151]) by ma1-aaemail-dr-lapp01.apple.com with ESMTP id 32byr0pqg9-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO); Sat, 18 Jul 2020 13:23:39 -0700 Received: from rn-mailsvcp-mmp-lapp03.rno.apple.com (rn-mailsvcp-mmp-lapp03.rno.apple.com [17.179.253.16]) by rn-mailsvcp-mta-lapp03.rno.apple.com (Oracle Communications Messaging Server 8.1.0.5.20200312 64bit (built Mar 12 2020)) with ESMTPS id <0QDO00B6ILZEFO60@rn-mailsvcp-mta-lapp03.rno.apple.com>; Sat, 18 Jul 2020 13:23:39 -0700 (PDT) Received: from process_milters-daemon.rn-mailsvcp-mmp-lapp03.rno.apple.com by rn-mailsvcp-mmp-lapp03.rno.apple.com (Oracle Communications Messaging Server 8.1.0.5.20200312 64bit (built Mar 12 2020)) id <0QDO00Y00LKFYK00@rn-mailsvcp-mmp-lapp03.rno.apple.com>; Sat, 18 Jul 2020 13:23:38 -0700 (PDT) X-Va-A: X-Va-T-CD: e0acb9dc03d22e4581b62f3d752335f3 X-Va-E-CD: abc689c8dbf31225341f60396bcd51b3 X-Va-R-CD: 957f4327799c979987441c5c4157b258 X-Va-CD: 0 X-Va-ID: e7c1bbec-34ee-4f6a-9a36-515bc35c0a55 X-V-A: X-V-T-CD: e0acb9dc03d22e4581b62f3d752335f3 X-V-E-CD: abc689c8dbf31225341f60396bcd51b3 X-V-R-CD: 957f4327799c979987441c5c4157b258 X-V-CD: 0 X-V-ID: 1074ae91-1980-40a8-aec6-74db1ab2d6b2 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.235,18.0.687 definitions=2020-07-18_12:2020-07-17,2020-07-18 signatures=0 Received: from [17.235.11.135] (unknown [17.235.11.135]) by rn-mailsvcp-mmp-lapp03.rno.apple.com (Oracle Communications Messaging Server 8.1.0.5.20200312 64bit (built Mar 12 2020)) with ESMTPSA id <0QDO00WBYLZDD100@rn-mailsvcp-mmp-lapp03.rno.apple.com>; Sat, 18 Jul 2020 13:23:38 -0700 (PDT) From: "Andrew Fish" Message-id: MIME-version: 1.0 (Mac OS X Mail 13.4 \(3608.80.23.2.2\)) Subject: Re: [edk2-devel] File in the build question. I'm working on a smart git grep Date: Sat, 18 Jul 2020 13:23:37 -0700 In-reply-to: <1622466A6176DB37.5128@groups.io> Cc: Laszlo Ersek To: edk2-devel-groups-io , Andrew Fish References: <39884c36-c42d-8718-f14b-d638e0cd1d69@redhat.com> <1622466A6176DB37.5128@groups.io> X-Mailer: Apple Mail (2.3608.80.23.2.2) X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.235,18.0.687 definitions=2020-07-18_12:2020-07-17,2020-07-18 signatures=0 Content-type: multipart/alternative; boundary="Apple-Mail=_1AE72ED3-E8E9-45E7-9DA8-B0F9DFA190DA" --Apple-Mail=_1AE72ED3-E8E9-45E7-9DA8-B0F9DFA190DA Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Wanted to see if people are OK with adding BaseTools/Scripts/ to the path w= hen you run `. edksetuip.sh` or edksetup.bat? If we do that and I add BaseTools/Scripts/git-pgrep then it will show up a= s an extended git command.=20 I was thinking over time we could add git-edk2* git commands to help autom= ate the process or maintainer workflow.=20 If people are OK with adding the path I can make a patch set to contribute= git-pgrep, but I need some help with how to add the new path to edksetup.b= at.=20 Thanks, Andrew Fish > On Jul 16, 2020, at 8:50 AM, Andrew Fish via groups.io wrote: >=20 >=20 >=20 >> On Jul 16, 2020, at 6:18 AM, Laszlo Ersek > wrote: >>=20 >> On 07/16/20 03:41, Andrew Fish via groups.io wrote: >>> I think I mentioned before on the list that I think a smart git grep t= hat only scopes to a give target would be useful. For our internal repo we = have lots of targets and lots of generations of platforms so lots of code t= hat hits in a git grep that is not in the platform.=20 >>>=20 >>> In thinking about this it looks to me like the Build/**/AutoGenTimeSta= mp (module build files) files and the Build/**/AutoGen (global build files)= contain a list of all the files for a project. >>>=20 >>> I wrote some python that takes the build target (OvmfX64 for example) = parses all the AutoGenTimeStamp and AutoGen files merges then together and= removes the workspace path. All the other arguments are passed to `git gre= p` and only lines that match the file set are displayed.=20 >>>=20 >>> My questions are: >>> 1) Did I extract all the files? >>=20 >> I don't know enough of BaseTools to answer the question :) >>=20 >>> 2) Are people interested in this feature? >>=20 >> It seems useful. :) >>=20 >>> 3) Any ideas on how to hook it in? I just dropped in the root of the g= it tree for development.=20 >>=20 >> We usually place such utilities under BaseTools/Scripts/, I think (e.g. >> PatchCheck.py, SetupGit.py, GetMaintainer.py). >=20 > I figured out if I rename my script to git-pgrep and move it to BaseTool= s/BinWrappers/PosixLike it becomes a new git command.=20 >=20 > $ git pgrep OvmfX64 CC_FLAGS -- *.dsc > OvmfX64 only: git grep CC_FLAGS -- *.dsc > OvmfPkg/OvmfPkgX64.dsc:73: GCC:RELEASE_*_*_CC_FLAGS =3D -DM= DEPKG_NDEBUG > OvmfPkg/OvmfPkgX64.dsc:74: INTEL:RELEASE_*_*_CC_FLAGS =3D /D = MDEPKG_NDEBUG > OvmfPkg/OvmfPkgX64.dsc:75: MSFT:RELEASE_*_*_CC_FLAGS =3D /D = MDEPKG_NDEBUG > OvmfPkg/OvmfPkgX64.dsc:77: GCC:*_*_*_CC_FLAGS =3D -mn= o-mmx -mno-sse > OvmfPkg/OvmfPkgX64.dsc:88: MSFT:*_*_*_CC_FLAGS =3D /D DISABLE_NEW_DEPRE= CATED_INTERFACES > OvmfPkg/OvmfPkgX64.dsc:89: INTEL:*_*_*_CC_FLAGS =3D /D DISABLE_NEW_DEPR= ECATED_INTERFACES > OvmfPkg/OvmfPkgX64.dsc:90: GCC:*_*_*_CC_FLAGS =3D -D DISABLE_NEW_DEPREC= ATED_INTERFACES >=20 > I picked pgrep for platform grep.=20 >=20 > I guess we could add $EDK_TOOLS_PATH/Scripts to the $PATH and then custo= m git commands could get added in $EDK_TOOLS_PATH/Scripts >=20 >>=20 >>> 4) Any other cool ideas of what we could do with a file list of the pr= oject? >>=20 >> Perhaps introduce a more focused utility that only produces the file >> list (on stdout) so people could pass the list to whatever commands the= y >> desire? >>=20 >=20 > I added that feature. If you don=E2=80=99t pass any git-grep arguments y= ou get the file list for the passed in platform.=20 >=20 > $ git pgrep OvmfX64 > Conf/build_rule.txt > Conf/tools_def.txt > CryptoPkg/CryptoPkg.dec > CryptoPkg/Library/BaseCryptLib/BaseCryptLib.inf > =E2=80=A6 >=20 > Thanks, >=20 > Andrew Fish >=20 >> Thanks! >> Laszlo >>=20 >>>=20 >>> Example (to give people a feel for how it works): >>> $ time ./ggrep.py OvmfX64 CC_FLAGS | wc -l >>> 42 >>>=20 >>> real 0m0.318s >>> user 0m0.243s >>> sys 0m0.576s >>> $ time git grep CC_FLAGS | wc -l >>> 479 >>>=20 >>> real 0m0.110s >>> user 0m0.148s >>> sys 0m0.525s >>> $ ./ggrep.py OvmfX64 CC_FLAGS >>> OvmfX64 only: git grep CC_FLAGS >>> CryptoPkg/Library/BaseCryptLib/BaseCryptLib.inf:96: MSFT:*_*_*_CC_FLA= GS =3D /wd4090 >>> CryptoPkg/Library/BaseCryptLib/BaseCryptLib.inf:100: RVCT:*_*_ARM_CC_= FLAGS =3D -JCryptoPkg/Include --diag_remark=3D1 >>> CryptoPkg/Library/BaseCryptLib/BaseCryptLib.inf:102: GCC:*_CLANG35_*_= CC_FLAGS =3D -std=3Dc99 >>> CryptoPkg/Library/BaseCryptLib/BaseCryptLib.inf:103: GCC:*_CLANG38_*_= CC_FLAGS =3D -std=3Dc99 >>> CryptoPkg/Library/BaseCryptLib/BaseCryptLib.inf:104: GCC:*_CLANGPDB_*= _CC_FLAGS =3D -std=3Dc99 -Wno-error=3Dincompatible-pointer-types >>> CryptoPkg/Library/BaseCryptLib/BaseCryptLib.inf:106: XCODE:*_*_*_CC_F= LAGS =3D -std=3Dc99 >>> CryptoPkg/Library/IntrinsicLib/IntrinsicLib.inf:55: MSFT:DEBUG_VS200= 3_IA32_CC_FLAGS =3D=3D /nologo /c /WX /W4 /Gs32768 /Gy /D UNICODE /O= 1b2 /FIAutoGen.h /EHs-c- /GR- /GF /GX- /Zi /Gm >>> CryptoPkg/Library/IntrinsicLib/IntrinsicLib.inf:56: MSFT:RELEASE_VS2= 003_IA32_CC_FLAGS =3D=3D /nologo /c /WX /W4 /Gs32768 /Gy /D UNICODE /O= 1b2 /FIAutoGen.h /EHs-c- /GR- /GF /GX- >>> CryptoPkg/Library/IntrinsicLib/IntrinsicLib.inf:57: MSFT:DEBUG_VS200= 3xASL_IA32_CC_FLAGS =3D=3D /nologo /c /WX /W4 /Gs32768 /Gy /D UNICODE /O= 1b2 /FIAutoGen.h /EHs-c- /GR- /GF /GX- /Zi /Gm >>> CryptoPkg/Library/IntrinsicLib/IntrinsicLib.inf:58: MSFT:RELEASE_VS2= 003xASL_IA32_CC_FLAGS =3D=3D /nologo /c /WX /W4 /Gs32768 /Gy /D UNICODE /O= 1b2 /FIAutoGen.h /EHs-c- /GR- /GF /GX- >>> CryptoPkg/Library/IntrinsicLib/IntrinsicLib.inf:59: MSFT:DEBUG_DDK37= 90_IA32_CC_FLAGS =3D=3D /nologo /c /WX /Gy /Gs32768 /D UNICODE /O1b2 = /FIAutoGen.h /EHs-c- /GR- /GF /Zi /Gm >>> CryptoPkg/Library/IntrinsicLib/IntrinsicLib.inf:60: MSFT:RELEASE_DDK= 3790_IA32_CC_FLAGS =3D=3D /nologo /c /WX /Gy /Gs32768 /D UNICODE /O1b2 = /FIAutoGen.h /EHs-c- /GR- /GF >>> CryptoPkg/Library/IntrinsicLib/IntrinsicLib.inf:61: MSFT:DEBUG_DDK37= 90xASL_IA32_CC_FLAGS =3D=3D /nologo /c /WX /Gy /Gs32768 /D UNICODE /O1b2 = /FIAutoGen.h /EHs-c- /GR- /GF /Zi /Gm >>> CryptoPkg/Library/IntrinsicLib/IntrinsicLib.inf:62: MSFT:RELEASE_DDK= 3790xASL_IA32_CC_FLAGS =3D=3D /nologo /c /WX /Gy /Gs32768 /D UNICODE /O1b2 = /FIAutoGen.h /EHs-c- /GR- /GF >>> CryptoPkg/Library/IntrinsicLib/IntrinsicLib.inf:63: MSFT:DEBUG_*_IA3= 2_CC_FLAGS =3D=3D /nologo /c /WX /GS- /W4 /Gs32768 /D UNICODE /= O1b2 /FIAutoGen.h /EHs-c- /GR- /GF /Gy /Zi /Gm >>> CryptoPkg/Library/IntrinsicLib/IntrinsicLib.inf:64: MSFT:RELEASE_*_I= A32_CC_FLAGS =3D=3D /nologo /c /WX /GS- /W4 /Gs32768 /D UNICODE /= O1b2 /FIAutoGen.h /EHs-c- /GR- /GF >>> CryptoPkg/Library/IntrinsicLib/IntrinsicLib.inf:65: MSFT:DEBUG_*_X64= _CC_FLAGS =3D=3D /nologo /c /WX /GS- /X /W4 /Gs32768 /D UNICOD= E /O1b2s /Gy /FIAutoGen.h /EHs-c- /GR- /GF /Zi /Gm >>> CryptoPkg/Library/IntrinsicLib/IntrinsicLib.inf:66: MSFT:RELEASE_*_X= 64_CC_FLAGS =3D=3D /nologo /c /WX /GS- /X /W4 /Gs32768 /D UNICOD= E /O1b2s /Gy /FIAutoGen.h /EHs-c- /GR- /GF >>> CryptoPkg/Library/IntrinsicLib/IntrinsicLib.inf:67: INTEL:*_*_*_CC_FL= AGS =3D /Oi- >>> CryptoPkg/Library/OpensslLib/OpensslLibCrypto.inf:569: MSFT:*_*_IA32_= CC_FLAGS =3D -U_WIN32 -U_WIN64 -U_MSC_VER $(OPENSSL_FLAGS) /wd4090 /wd413= 2 /wd4244 /wd4245 /wd4267 /wd4310 /wd4389 /wd4700 /wd4702 /wd4706 /wd4819 >>> CryptoPkg/Library/OpensslLib/OpensslLibCrypto.inf:570: MSFT:*_*_X64_C= C_FLAGS =3D -U_WIN32 -U_WIN64 -U_MSC_VER $(OPENSSL_FLAGS) /wd4090 /wd413= 2 /wd4244 /wd4245 /wd4267 /wd4306 /wd4310 /wd4700 /wd4389 /wd4702 /wd4706 /= wd4819 >>> CryptoPkg/Library/OpensslLib/OpensslLibCrypto.inf:572: INTEL:*_*_IA32= _CC_FLAGS =3D -U_WIN32 -U_WIN64 -U_MSC_VER -U__ICC $(OPENSSL_FLAGS) /w >>> CryptoPkg/Library/OpensslLib/OpensslLibCrypto.inf:573: INTEL:*_*_X64_= CC_FLAGS =3D -U_WIN32 -U_WIN64 -U_MSC_VER -U__ICC $(OPENSSL_FLAGS) /w >>> CryptoPkg/Library/OpensslLib/OpensslLibCrypto.inf:582: GCC:*_*_IA32_C= C_FLAGS =3D -U_WIN32 -U_WIN64 $(OPENSSL_FLAGS) -Wno-error=3Dmaybe-uninit= ialized -Wno-error=3Dunused-but-set-variable >>> CryptoPkg/Library/OpensslLib/OpensslLibCrypto.inf:583: GCC:*_*_X64_CC= _FLAGS =3D -U_WIN32 -U_WIN64 $(OPENSSL_FLAGS) -Wno-error=3Dmaybe-uninit= ialized -Wno-error=3Dformat -Wno-format -Wno-error=3Dunused-but-set-variabl= e -DNO_MSABI_VA_FUNCS >>> CryptoPkg/Library/OpensslLib/OpensslLibCrypto.inf:584: GCC:*_*_ARM_CC= _FLAGS =3D $(OPENSSL_FLAGS) -Wno-error=3Dmaybe-uninitialized -Wno-error= = =3Dunused-but-set-variable >>> CryptoPkg/Library/OpensslLib/OpensslLibCrypto.inf:585: GCC:*_*_AARCH6= 4_CC_FLAGS =3D $(OPENSSL_FLAGS) -Wno-error=3Dmaybe-uninitialized -Wno-forma= t -Wno-error=3Dunused-but-set-variable >>> CryptoPkg/Library/OpensslLib/OpensslLibCrypto.inf:586: GCC:*_*_RISCV6= 4_CC_FLAGS =3D $(OPENSSL_FLAGS) -Wno-error=3Dmaybe-uninitialized -Wno-forma= t -Wno-error=3Dunused-but-set-variable >>> CryptoPkg/Library/OpensslLib/OpensslLibCrypto.inf:587: GCC:*_CLANG35_= *_CC_FLAGS =3D -std=3Dc99 -Wno-error=3Duninitialized >>> CryptoPkg/Library/OpensslLib/OpensslLibCrypto.inf:588: GCC:*_CLANG38_= *_CC_FLAGS =3D -std=3Dc99 -Wno-error=3Duninitialized >>> CryptoPkg/Library/OpensslLib/OpensslLibCrypto.inf:589: GCC:*_CLANGPDB= _*_CC_FLAGS =3D -std=3Dc99 -Wno-error=3Duninitialized -Wno-error=3Dincompat= ible-pointer-types -Wno-error=3Dpointer-sign -Wno-error=3Dimplicit-function= -declaration -Wno-error=3Dignored-pragma-optimize >>> CryptoPkg/Library/OpensslLib/OpensslLibCrypto.inf:610: RVCT:*_*_ARM_C= C_FLAGS =3D $(OPENSSL_FLAGS) --library_interface=3Daeabi_clib99 --diag_= suppress=3D1296,1295,550,1293,111,68,177,223,144,513,188,128,546,1,3017 -JC= ryptoPkg/Include >>> CryptoPkg/Library/OpensslLib/OpensslLibCrypto.inf:611: XCODE:*_*_IA32= _CC_FLAGS =3D -mmmx -msse -U_WIN32 -U_WIN64 $(OPENSSL_FLAGS) -w -std=3Dc9= 9 -Wno-error=3Duninitialized >>> CryptoPkg/Library/OpensslLib/OpensslLibCrypto.inf:612: XCODE:*_*_X64_= CC_FLAGS =3D -mmmx -msse -U_WIN32 -U_WIN64 $(OPENSSL_FLAGS) -w -std=3Dc9= 9 -Wno-error=3Duninitialized >>> OvmfPkg/OvmfPkgX64.dsc:73: GCC:RELEASE_*_*_CC_FLAGS =3D -= DMDEPKG_NDEBUG >>> OvmfPkg/OvmfPkgX64.dsc:74: INTEL:RELEASE_*_*_CC_FLAGS =3D /= D MDEPKG_NDEBUG >>> OvmfPkg/OvmfPkgX64.dsc:75: MSFT:RELEASE_*_*_CC_FLAGS =3D /= D MDEPKG_NDEBUG >>> OvmfPkg/OvmfPkgX64.dsc:77: GCC:*_*_*_CC_FLAGS =3D -= mno-mmx -mno-sse >>> OvmfPkg/OvmfPkgX64.dsc:88: MSFT:*_*_*_CC_FLAGS =3D /D DISABLE_NEW_DEP= RECATED_INTERFACES >>> OvmfPkg/OvmfPkgX64.dsc:89: INTEL:*_*_*_CC_FLAGS =3D /D DISABLE_NEW_DE= PRECATED_INTERFACES >>> OvmfPkg/OvmfPkgX64.dsc:90: GCC:*_*_*_CC_FLAGS =3D -D DISABLE_NEW_DEPR= ECATED_INTERFACES >>> $./ggrep.py OvmfX64 CC_FLAG -- *.dsc >>> OvmfX64 only: git grep CC_FLAG -- *.dsc >>> OvmfPkg/OvmfPkgX64.dsc:73: GCC:RELEASE_*_*_CC_FLAGS =3D -= DMDEPKG_NDEBUG >>> OvmfPkg/OvmfPkgX64.dsc:74: INTEL:RELEASE_*_*_CC_FLAGS =3D /= D MDEPKG_NDEBUG >>> OvmfPkg/OvmfPkgX64.dsc:75: MSFT:RELEASE_*_*_CC_FLAGS =3D /= D MDEPKG_NDEBUG >>> OvmfPkg/OvmfPkgX64.dsc:77: GCC:*_*_*_CC_FLAGS =3D -= mno-mmx -mno-sse >>> OvmfPkg/OvmfPkgX64.dsc:88: MSFT:*_*_*_CC_FLAGS =3D /D DISABLE_NEW_DEP= RECATED_INTERFACES >>> OvmfPkg/OvmfPkgX64.dsc:89: INTEL:*_*_*_CC_FLAGS =3D /D DISABLE_NEW_DE= PRECATED_INTERFACES >>> OvmfPkg/OvmfPkgX64.dsc:90: GCC:*_*_*_CC_FLAGS =3D -D DISABLE_NEW_DEPR= ECATED_INTERFACES >>> $ >>>=20 >>> Note if we cache the combined AutoGenTimeStamp and AutoGen files into = a single file in the Build direction and use that file on the next run the = time goes down to: >>> real 0m0.177s >>> user 0m0.185s >>> sys 0m0.522s >>>=20 >>> If you don=E2=80=99t pass any `git grep` commands you get the file lis= t back: >>> $./ggrep.py OvmfX64 >>> Conf/build_rule.txt >>> Conf/tools_def.txt >>> CryptoPkg/CryptoPkg.dec >>> CryptoPkg/Library/BaseCryptLib/BaseCryptLib.inf >>> CryptoPkg/Library/BaseCryptLib/Cipher/CryptAes.c >>> =E2=80=A6 >>>=20 >>>=20 >>> My prototype is around 135 lines of Python. >>>=20 >>> Thanks, >>>=20 >>> Andrew Fish >>>=20 >>>=20 >=20 >=20 --Apple-Mail=_1AE72ED3-E8E9-45E7-9DA8-B0F9DFA190DA Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 Wanted to see if people a= re OK with adding BaseTools/Scripts/ to the path when you run `. edkse= tuip.sh` or edksetup.bat?

If we do that and I add BaseTools/Scripts/git-pgrep then it will sh= ow up as an extended git command. 

I was thinking over time we could add git-edk2* git = commands to help automate the process or maintainer workflow. 

If people are OK with ad= ding the path I can make a patch set to contribute git-pgrep, but I need so= me help with how to add the new path to edksetup.bat. 

Thanks,
Andrew Fish

On Jul 16, 2020= , at 8:50 AM, Andrew Fish via group= s.io <afis= h=3Dapple.com@groups.io> wrote:



On Jul 16, 2020, at 6:18 AM, Laszlo Ersek <lersek@redhat.com> wrote= :

On = 07/16/20 03:41, Andrew Fish via = groups.io wrote:
I think I mentioned before on the list = that I think a smart git grep that only scopes to a give target would be us= eful. For our internal repo we have lots of targets and lots of generations= of platforms so lots of code that hits in a git grep that is not in the pl= atform. 
In thinking about this it looks to me like the Build/**/AutoGe= nTimeStamp (module build files) files and the Build/**/AutoGen (global buil= d files) contain a list of all the files for a project.

I wrote some python that takes the build target (OvmfX64 for exam= ple) parses all the  AutoGenTimeStamp and AutoGen files merges then to= gether and removes the workspace path. All the other arguments are passed t= o `git grep` and only lines that match the file set are displayed. 

My = questions are:
1) Did I extract all the files?
=

I don't know e= nough of BaseTools to answer the question :)

2) Are people interested in this feature?

It seems useful. :)

3) Any ideas on how to hook it in?= I just dropped in the root of the git tree for development. 

We usually place such utilities under= BaseTools/Scripts/, I think (e.g.
PatchCheck.py, SetupGit.py, GetMaintainer.py).

I figured out if I rename my script to gi= t-pgrep and move it to BaseTools/BinWrappers/PosixLike it becomes a new git= command. 

<= div class=3D"" style=3D"margin: 0px; font-stretch: normal; font-size: 11px;= line-height: normal; font-family: Menlo;">= $ git pgrep OvmfX64 CC_FLAGS -- *.dsc
OvmfX64 only: git grep CC_FLAGS -- *.dsc
<= div class=3D"" style=3D"margin: 0px; font-stretch: normal; font-size: 11px;= line-height: normal; font-family: Menlo;">OvmfPkg/OvmfPkgX64.dsc:73:  GC= C:RELEASE_*_*_CC_FLAGS             =3D -DMDEP= KG_NDEBUG
OvmfPkg/Ovmf= PkgX64.dsc:74:  INTEL:RELEASE_*_*_CC_FLAGS        =   =3D /D MDEPKG_NDEBUG
OvmfPkg/OvmfPkgX64.dsc:75:  MSFT:RELEASE_*_*_CC_FLAGS  &nb= sp;         =3D /D MDEPKG_NDEBUG
OvmfPkg/OvmfPkgX64.dsc:77:  GCC:*_*_*_= CC_FLAGS                   =3D= -mno-mmx -mno-sse
= Ovm= fPkg/OvmfPkgX64.dsc:88:  MSFT:*_*_*_CC_FLAGS =3D /D DISABLE_NEW_DEPREC= ATED_INTERFACES
OvmfPk= g/OvmfPkgX64.dsc:89:  INTEL:*_*_*_CC_FLAGS =3D /D DISABLE_NEW_DEPRECAT= ED_INTERFACES
OvmfPkg/= OvmfPkgX64.dsc:90:  GCC:*_*_*_CC_FLAGS =3D -D DISABLE_NEW_DEPRECATED_I= NTERFACES

I picked pgrep for platform grep. 

I guess we could add $EDK_TOOLS_PATH/= Scripts to the $PATH and then custom git commands could get added in $EDK_T= OOLS_PATH/Scripts


4) Any other cool ideas of what we coul= d do with a file list of the project?

Perhaps introduce a more focused util= ity that only produces the file
list (on stdout) so people could pass the list to whatever command= s they
desire?=


= I added that feature. If you don=E2=80=99t pass any git-grep arguments you = get the file list for the passed in platform. 
$ git pgrep OvmfX64
Conf/build_rule.txt
Conf/tools_def.txt
CryptoPkg/CryptoPkg.dec
CryptoPkg/Library/BaseCryptLib/BaseCryptL= ib.inf
=E2=80=A6

Thanks,=

Andrew Fish

Thanks!
L= aszlo


Example (to= give people a feel for how it works):
$ time ./ggrep.py Ovmf= X64 CC_FLAGS | wc -l
     42

real 0m0.318s
user 0m0.243s
sys 0m0.576s
$ time git grep CC_FLAGS | wc -l
   &nb= sp;479

real 0m0.110s
user 0m0.148s
s= ys 0m0.52= 5s
$ ./ggrep.py OvmfX64 CC_FLAGS
OvmfX64 only: = git grep CC_FLAGS
CryptoPkg/Library/BaseCryptLib/BaseCryptLib= .inf:96:  MSFT:*_*_*_CC_FLAGS =3D /wd4090
CryptoPkg/Libr= ary/BaseCryptLib/BaseCryptLib.inf:100:  RVCT:*_*_ARM_CC_FLAGS =3D -JCr= yptoPkg/Include --diag_remark=3D1
CryptoPkg/Library/BaseCrypt= Lib/BaseCryptLib.inf:102:  GCC:*_CLANG35_*_CC_FLAGS =3D -std=3Dc99
CryptoPkg/Library/BaseCryptLib/BaseCryptLib.inf:103:  GCC:*= _CLANG38_*_CC_FLAGS =3D -std=3Dc99
CryptoPkg/Library/BaseCryp= tLib/BaseCryptLib.inf:104:  GCC:*_CLANGPDB_*_CC_FLAGS =3D -std=3Dc99 -= Wno-error=3Dincompatible-pointer-types
CryptoPkg/Library/Base= CryptLib/BaseCryptLib.inf:106:  XCODE:*_*_*_CC_FLAGS =3D -std=3Dc99CryptoPkg/Library/IntrinsicLib/IntrinsicLib.inf:55:   = ;MSFT:DEBUG_VS2003_IA32_CC_FLAGS        = = =3D=3D /nologo /c /WX /W4 /Gs32768 /Gy /D UNICODE /O1b2 /FIAutoGen.h /EHs-= c- /GR- /GF /GX- /Zi /Gm
CryptoPkg/Library/IntrinsicLib/Intri= nsicLib.inf:56:   MSFT:RELEASE_VS2003_IA32_CC_FLAGS   &= nbsp;  =3D=3D /nologo /c /WX /W4 /Gs32768 /Gy /D UNICODE /O1b2 /F= IAutoGen.h /EHs-c- /GR- /GF /GX-
CryptoPkg/Library/IntrinsicL= ib/IntrinsicLib.inf:57:   MSFT:DEBUG_VS2003xASL_IA32_CC_FLAGS &nb= sp;  =3D=3D /nologo /c /WX /W4 /Gs32768 /Gy /D UNICODE /O1b2 /FIA= utoGen.h /EHs-c- /GR- /GF /GX- /Zi /Gm
CryptoPkg/Library/Intr= insicLib/IntrinsicLib.inf:58:   MSFT:RELEASE_VS2003xASL_IA32_CC_F= LAGS  =3D=3D /nologo /c /WX /W4 /Gs32768 /Gy /D UNICODE /O1b2 /FIAutoG= en.h /EHs-c- /GR- /GF /GX-
CryptoPkg/Library/IntrinsicLib/Int= rinsicLib.inf:59:   MSFT:DEBUG_DDK3790_IA32_CC_FLAGS   =     =3D=3D /nologo /c /WX /Gy /Gs32768 /D UNICODE /O1b2= /FIAutoGen.h /EHs-c- /GR- /GF /Zi /Gm
CryptoPkg/Library/Intr= insicLib/IntrinsicLib.inf:60:   MSFT:RELEASE_DDK3790_IA32_CC_FLAG= S     =3D=3D /nologo /c /WX /Gy /Gs32768 /D UNICODE /O1= b2 /FIAutoGen.h /EHs-c- /GR- /GF
CryptoPkg/Library/IntrinsicL= ib/IntrinsicLib.inf:61:   MSFT:DEBUG_DDK3790xASL_IA32_CC_FLAGS &n= bsp; =3D=3D /nologo /c /WX /Gy /Gs32768 /D UNICODE /O1b2 /FIAutoGen.h = /EHs-c- /GR- /GF /Zi /Gm
CryptoPkg/Library/IntrinsicLib/Intri= nsicLib.inf:62:   MSFT:RELEASE_DDK3790xASL_IA32_CC_FLAGS =3D=3D /= nologo /c /WX /Gy /Gs32768 /D UNICODE /O1b2 /FIAutoGen.h /EHs-c- /GR- /GFCryptoPkg/Library/IntrinsicLib/IntrinsicLib.inf:63:  &nbs= p;MSFT:DEBUG_*_IA32_CC_FLAGS        &nbs= p;    =3D=3D /nologo /c /WX /GS- /W4 /Gs32768 /D UNICOD= E /O1b2 /FIAutoGen.h /EHs-c- /GR- /GF /Gy /Zi /Gm
CryptoPkg/L= ibrary/IntrinsicLib/IntrinsicLib.inf:64:   MSFT:RELEASE_*_IA32_CC= _FLAGS           =3D=3D /= nologo /c /WX /GS- /W4 /Gs32768 /D UNICODE /O1b2 /FIAutoGen.h /EHs-c- /GR- = /GF
CryptoPkg/Library/IntrinsicLib/IntrinsicLib.inf:65:  = ; MSFT:DEBUG_*_X64_CC_FLAGS        =       =3D=3D /nologo /c /WX /GS- /X /W4 /Gs32= 768 /D UNICODE /O1b2s /Gy /FIAutoGen.h /EHs-c- /GR- /GF /Zi /Gm
CryptoPkg/Library/IntrinsicLib/IntrinsicLib.inf:66:   MSFT:REL= EASE_*_X64_CC_FLAGS          &= nbsp; =3D=3D /nologo /c /WX /GS- /X /W4 /Gs32768 /D UNICODE /O1b2s /Gy= /FIAutoGen.h /EHs-c- /GR- /GF
CryptoPkg/Library/IntrinsicLib= /IntrinsicLib.inf:67:  INTEL:*_*_*_CC_FLAGS     &n= bsp;            = ;  =3D  /Oi-
CryptoPkg/Library/OpensslLib/Open= sslLibCrypto.inf:569:  MSFT:*_*_IA32_CC_FLAGS   =3D -U_WIN32= -U_WIN64 -U_MSC_VER $(OPENSSL_FLAGS) /wd4090 /wd4132 /wd4244 /wd4245 /wd42= 67 /wd4310 /wd4389 /wd4700 /wd4702 /wd4706 /wd4819
CryptoPkg/= Library/OpensslLib/OpensslLibCrypto.inf:570:  MSFT:*_*_X64_CC_FLAGS &n= bsp;  =3D -U_WIN32 -U_WIN64 -U_MSC_VER $(OPENSSL_FLAGS) /wd4090 /= wd4132 /wd4244 /wd4245 /wd4267 /wd4306 /wd4310 /wd4700 /wd4389 /wd4702 /wd4= 706 /wd4819
CryptoPkg/Library/OpensslLib/OpensslLibCrypto.inf= :572:  INTEL:*_*_IA32_CC_FLAGS  =3D -U_WIN32 -U_WIN64 -U_MSC_VER = -U__ICC $(OPENSSL_FLAGS) /w
CryptoPkg/Library/OpensslLib/Open= sslLibCrypto.inf:573:  INTEL:*_*_X64_CC_FLAGS   =3D -U_WIN32= -U_WIN64 -U_MSC_VER -U__ICC $(OPENSSL_FLAGS) /w
CryptoPkg/Li= brary/OpensslLib/OpensslLibCrypto.inf:582:  GCC:*_*_IA32_CC_FLAGS &nbs= p;  =3D -U_WIN32 -U_WIN64 $(OPENSSL_FLAGS) -Wno-error=3Dmaybe-uni= nitialized -Wno-error=3Dunused-but-set-variable
CryptoPkg/Lib= rary/OpensslLib/OpensslLibCrypto.inf:583:  GCC:*_*_X64_CC_FLAGS  =    =3D -U_WIN32 -U_WIN64 $(OPENSSL_FLAGS) -Wno-error=3Dmaybe= -uninitialized -Wno-error=3Dformat -Wno-format -Wno-error=3Dunused-but-set-= variable -DNO_MSABI_VA_FUNCS
CryptoPkg/Library/OpensslLib/Ope= nsslLibCrypto.inf:584:  GCC:*_*_ARM_CC_FLAGS     = =3D $(OPENSSL_FLAGS) -Wno-error=3Dmaybe-uninitialized -Wno-error=3Dunused-= but-set-variable
CryptoPkg/Library/OpensslLib/OpensslLibCrypt= o.inf:585:  GCC:*_*_AARCH64_CC_FLAGS =3D $(OPENSSL_FLAGS) -Wno-error= =3Dmaybe-uninitialized -Wno-format -Wno-error=3Dunused-but-set-variableCryptoPkg/Library/OpensslLib/OpensslLibCrypto.inf:586:  GC= C:*_*_RISCV64_CC_FLAGS =3D $(OPENSSL_FLAGS) -Wno-error=3Dmaybe-uninitialize= d -Wno-format -Wno-error=3Dunused-but-set-variable
CryptoPkg/= Library/OpensslLib/OpensslLibCrypto.inf:587:  GCC:*_CLANG35_*_CC_FLAGS= =3D -std=3Dc99 -Wno-error=3Duninitialized
CryptoPkg/Library/= OpensslLib/OpensslLibCrypto.inf:588:  GCC:*_CLANG38_*_CC_FLAGS =3D -st= d=3Dc99 -Wno-error=3Duninitialized
CryptoPkg/Library/OpensslL= ib/OpensslLibCrypto.inf:589:  GCC:*_CLANGPDB_*_CC_FLAGS =3D -std=3Dc99= -Wno-error=3Duninitialized -Wno-error=3Dincompatible-pointer-types -Wno-er= ror=3Dpointer-sign -Wno-error=3Dimplicit-function-declaration -Wno-error=3D= ignored-pragma-optimize
CryptoPkg/Library/OpensslLib/OpensslL= ibCrypto.inf:610:  RVCT:*_*_ARM_CC_FLAGS     =3D $= (OPENSSL_FLAGS) --library_interface=3Daeabi_clib99 --diag_suppress=3D1296,1= 295,550,1293,111,68,177,223,144,513,188,128,546,1,3017 -JCryptoPkg/Include<= br class=3D"">CryptoPkg/Library/OpensslLib/OpensslLibCrypto.inf:611:  = XCODE:*_*_IA32_CC_FLAGS   =3D -mmmx -msse -U_WIN32 -U_WIN64 $(OPE= NSSL_FLAGS) -w -std=3Dc99 -Wno-error=3Duninitialized
CryptoPk= g/Library/OpensslLib/OpensslLibCrypto.inf:612:  XCODE:*_*_X64_CC_FLAGS=    =3D -mmmx -msse -U_WIN32 -U_WIN64 $(OPENSSL_FLAGS) -w -s= td=3Dc99 -Wno-error=3Duninitialized
OvmfPkg/OvmfPkgX64.dsc:73= :  GCC:RELEASE_*_*_CC_FLAGS        =      =3D -DMDEPKG_NDEBUG
OvmfPkg/Ovm= fPkgX64.dsc:74:  INTEL:RELEASE_*_*_CC_FLAGS     &n= bsp;     =3D /D MDEPKG_NDEBUG
OvmfPk= g/OvmfPkgX64.dsc:75:  MSFT:RELEASE_*_*_CC_FLAGS    &nbs= p;       =3D /D MDEPKG_NDEBUG
OvmfPkg/OvmfPkgX64.dsc:77:  GCC:*_*_*_CC_FLAGS   &nbs= p;            &= nbsp;  =3D -mno-mmx -mno-sse
OvmfPkg/OvmfPkgX64.dsc= :88:  MSFT:*_*_*_CC_FLAGS =3D /D DISABLE_NEW_DEPRECATED_INTERFACES
OvmfPkg/OvmfPkgX64.dsc:89:  INTEL:*_*_*_CC_FLAGS =3D /D DIS= ABLE_NEW_DEPRECATED_INTERFACES
OvmfPkg/OvmfPkgX64.dsc:90: &nb= sp;GCC:*_*_*_CC_FLAGS =3D -D DISABLE_NEW_DEPRECATED_INTERFACES
$./ggrep.py OvmfX64 CC_FLAG -- *.dsc
OvmfX64 only: git grep= CC_FLAG -- *.dsc
OvmfPkg/OvmfPkgX64.dsc:73:  GCC:RELEAS= E_*_*_CC_FLAGS           =   =3D -DMDEPKG_NDEBUG
OvmfPkg/OvmfPkgX64.dsc:74: &n= bsp;INTEL:RELEASE_*_*_CC_FLAGS        &n= bsp;  =3D /D MDEPKG_NDEBUG
OvmfPkg/OvmfPkgX64.dsc:7= 5:  MSFT:RELEASE_*_*_CC_FLAGS       &nbs= p;    =3D /D MDEPKG_NDEBUG
OvmfPkg/OvmfPk= gX64.dsc:77:  GCC:*_*_*_CC_FLAGS       &= nbsp;           =3D = -mno-mmx -mno-sse
OvmfPkg/OvmfPkgX64.dsc:88:  MSFT:*_*_*= _CC_FLAGS =3D /D DISABLE_NEW_DEPRECATED_INTERFACES
OvmfPkg/Ov= mfPkgX64.dsc:89:  INTEL:*_*_*_CC_FLAGS =3D /D DISABLE_NEW_DEPRECATED_I= NTERFACES
OvmfPkg/OvmfPkgX64.dsc:90:  GCC:*_*_*_CC_FLAGS= =3D -D DISABLE_NEW_DEPRECATED_INTERFACES
$
Note if we cache the combined AutoGenTimeStamp and AutoGen file= s into a single file in the Build direction and use that file on the next r= un the time goes down to:
real 0m0.177s
user 0m0.185s
sys 0m0.522s

If you don=E2=80=99t pass any `git = grep` commands you get the file list back:
$./ggrep.py OvmfX6= 4
Conf/build_rule.txt
Conf/tools_def.txt
CryptoPkg/CryptoPkg.dec
CryptoPkg/Library/BaseCryptLi= b/BaseCryptLib.inf
CryptoPkg/Library/BaseCryptLib/Cipher/Cryp= tAes.c
=E2=80=A6


= My prototype is around 135 lines of Python.

Th= anks,

Andrew Fish




--Apple-Mail=_1AE72ED3-E8E9-45E7-9DA8-B0F9DFA190DA--