From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from NAM10-MW2-obe.outbound.protection.outlook.com (NAM10-MW2-obe.outbound.protection.outlook.com [40.107.94.122]) by mx.groups.io with SMTP id smtpd.web11.12404.1584742012405096081 for ; Fri, 20 Mar 2020 15:06:52 -0700 Authentication-Results: mx.groups.io; dkim=pass header.i=@microsoft.com header.s=selector2 header.b=LlxScUFX; spf=pass (domain: microsoft.com, ip: 40.107.94.122, mailfrom: bret.barkelew@microsoft.com) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=jSxRw4/w0eGndORT9Ugp1Zf/PV2qINhta7vnsPbmVM3x0mH3stp4nwgvlb9HZJ6GW2kp3z5iAWhYyvshFVXxTNbQEfSHa25VvJER2DLicsF9WSFLU6PK7mG1xwuvU2lIyd9voAj9QNgkMcBhZgZWDn4jblzyiZjIhLSGuJDW4ifrD1eg/XBUwtESDODayJXfQN2WAD/aioiL1F+yeA2DBGSoxx4IGvtjgYjlQSHOU8mAwliAvADdbQiXN+BW35Xc1h4VvQQPa/tqFhI+gwz0XNcCMLM8Ie3Vdc3l6Bj5rOANi01aWzVm8OX4WIZJvD89BoM41SVRXZx336RVYrwOUA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=3WDyuDiUzY4HKdDTWU2Z8/LaMx4Ypz4lptIi66Tbx0U=; b=OfHP3CEpI7zOw33wDY0nQAOb+PulPy+M+jT880vEnuY35qgOCp50+CPbMasWGSY4d7i91+fqDwdJ1iPej6bctCHHWGOHRhQHahzHLMVwdSITHo6C9+JBmfzYCb+XAOjRKhjvE4Vi5axXj7L73MeOdDVFMU2cZSKSpkPSFCM3o59RX9ztcpBsOvLvfp3X8auc255GxwJvcwubABBeWYtF55hcyhnSPEG3vELq4UEjic73YWFZJCpiFpcByEzZcMe62s6RdOtNxSBaASxHUGyDqxndout2JE1a4C+F7o1MrseuZKF9YJS3Sqn0p0kqb4a7u9Kmdhow3+BUMtdB07qpqA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=microsoft.com; dmarc=pass action=none header.from=microsoft.com; dkim=pass header.d=microsoft.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=3WDyuDiUzY4HKdDTWU2Z8/LaMx4Ypz4lptIi66Tbx0U=; b=LlxScUFX5Pa4paTNd1Oh8j8MlGYzASF+IcU9JRpZ78t2UNgIVzb76TeOYOiZQBVoXSAIl81ON8lSKHpbIAFTr9v86L/B1B5bhESpmv9E0JxnGQ85NvwKtlyp3ULxjTCzHLi7tycHHzSN4R8r1aPHfyveJNMNRbxdBC4jth5Sxsw= Received: from MWHPR21MB0750.namprd21.prod.outlook.com (2603:10b6:300:76::12) by MWHPR21MB0158.namprd21.prod.outlook.com (2603:10b6:300:78::16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2835.4; Fri, 20 Mar 2020 22:06:50 +0000 Received: from MWHPR21MB0750.namprd21.prod.outlook.com ([fe80::6922:cd43:9ef5:b7ce]) by MWHPR21MB0750.namprd21.prod.outlook.com ([fe80::6922:cd43:9ef5:b7ce%7]) with mapi id 15.20.2856.003; Fri, 20 Mar 2020 22:06:50 +0000 From: "Bret Barkelew" To: "Desimone, Ashley E" , "Desimone, Nathaniel L" , "devel@edk2.groups.io" CC: "Pandya, Puja" , "Bjorge, Erik C" Subject: Re: [PATCH 1/4] EdkRepo: Installer should remove obsolete dependencies Thread-Topic: [PATCH 1/4] EdkRepo: Installer should remove obsolete dependencies Thread-Index: AQHV/iUrXu8o79TTu0KXeglfeefZm6hQURDsgAAVjACAAYdoAIAAHOgl Date: Fri, 20 Mar 2020 22:06:50 +0000 Message-ID: References: <20200319031636.2269-1-nathaniel.l.desimone@intel.com> <20200319031636.2269-2-nathaniel.l.desimone@intel.com> <4CF3A9EB60ABDA47BE7821A4DA3A0A3353D5B3EA@ORSMSX116.amr.corp.intel.com> <7FD22CB6-F1DC-4F2B-9291-5BF07C4E7CAD@intel.com>,<4CF3A9EB60ABDA47BE7821A4DA3A0A3353D5D5E3@ORSMSX116.amr.corp.intel.com> In-Reply-To: <4CF3A9EB60ABDA47BE7821A4DA3A0A3353D5D5E3@ORSMSX116.amr.corp.intel.com> Accept-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: msip_labels: MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Enabled=True;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_SiteId=72f988bf-86f1-41af-91ab-2d7cd011db47;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_SetDate=2020-03-19T19:43:34.7400074Z;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_ContentBits=0;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Method=Privileged authentication-results: spf=none (sender IP is ) smtp.mailfrom=Bret.Barkelew@microsoft.com; x-originating-ip: [71.212.145.195] x-ms-publictraffictype: Email x-ms-office365-filtering-ht: Tenant x-ms-office365-filtering-correlation-id: 775f000c-9190-4869-6a1f-08d7cd1afdb7 x-ms-traffictypediagnostic: MWHPR21MB0158: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:9508; x-forefront-prvs: 03484C0ABF x-forefront-antispam-report: SFV:NSPM;SFS:(10019020)(4636009)(376002)(366004)(346002)(396003)(39860400002)(136003)(199004)(66946007)(5660300002)(52536014)(9686003)(81166006)(8990500004)(33656002)(71200400001)(8936002)(81156014)(110136005)(7696005)(66574012)(54906003)(8676002)(26005)(30864003)(64756008)(2906002)(4326008)(76116006)(478600001)(316002)(86362001)(66556008)(55016002)(66476007)(53546011)(10290500003)(186003)(66446008)(6506007);DIR:OUT;SFP:1102;SCL:1;SRVR:MWHPR21MB0158;H:MWHPR21MB0750.namprd21.prod.outlook.com;FPR:;SPF:None;LANG:en;PTR:InfoNoRecords;A:1; received-spf: None (protection.outlook.com: microsoft.com does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: gamSjbS4dqs6+aD9dUNZ9L/viV/bj+Ib6Lj8GeA9q4M1jECVLx1vVpLLcz0OAzKq4bijax6/807HrynfKja4S/xdWRI2iDd1vaeFCm6AT4MfmD+aFFS2Jk7D4mR7THNV2/l6OkMDJulIiTH79D9frArpRrugAV4TxALq6jIzpIt2itx7TY4P+SA0FMSfIWBG6aOOwYU1GMZuKFf03YMkLXQW5bENv4/C5FD/T4ZMTDeOBLYc8p4c2CgjGD40VzqqmDmnrXAXiKBMpO+Dwtjl5zaJbhRszKN4dVOo+I29bEAWPKKUw5UK/X+4kAr78GwSschP+ns1+ER2NYPp3WcOQc/mX4Fq9y3tjWWJiUYaroIYWlGpmzdsW0h97oSpOz6fVm9wsWkhklhglHYOq1v396nECF6Pl/rAiSOQeQWMpbpxCNJZKHBA/lQfl8eWZm9D x-ms-exchange-antispam-messagedata: qMkgXNkje/TxgRJyBsfLBQ5d69YJaIJ94rzibqqKpCRciElqnrr4lIXW7RfYTuYF+uRzSRijkq6iw/m7Pmh0yirsEuazDqPpAIw0DZhR7tGopoR4KBK/u/JYVp5Zqo8mSEEkaRRr5Ywwbu3E4poGog== x-ms-exchange-transport-forked: True MIME-Version: 1.0 X-OriginatorOrg: microsoft.com X-MS-Exchange-CrossTenant-Network-Message-Id: 775f000c-9190-4869-6a1f-08d7cd1afdb7 X-MS-Exchange-CrossTenant-originalarrivaltime: 20 Mar 2020 22:06:50.4144 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 72f988bf-86f1-41af-91ab-2d7cd011db47 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: qz0WEJjDf4/Rz2W6DsyOFbDt5PDMHis5qvbpVfud6wHOIQlNM01X7R94zbgU+GL9jSiYi+IfhHlyoaSc6jZP2g== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR21MB0158 Content-Language: en-US Content-Type: multipart/alternative; boundary="_000_MWHPR21MB075064C54FB19A90275F6836EFF50MWHPR21MB0750namp_" --_000_MWHPR21MB075064C54FB19A90275F6836EFF50MWHPR21MB0750namp_ Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable Spiffy. Agreed; thanks for the background. My only other concern would be the function name and if some future develop= er thought that was permission to add other modules to it. Then again, maybe I=92m just bikeshedding. Feel free to ignore. - Bret From: Desimone, Ashley E Sent: Friday, March 20, 2020 1:21 PM To: Desimone, Nathaniel L; Bret Bark= elew; devel@edk2.groups.io Cc: Pandya, Puja; Bjorge, Erik C Subject: [EXTERNAL] RE: [PATCH 1/4] EdkRepo: Installer should remove obsole= te dependencies Thanks for the explanation. Reviewed-by: Ashley DeSimone From: Desimone, Nathaniel L Sent: Thursday, March 19, 2020 2:01 PM To: Bret Barkelew ; Desimone, Ashley E ; devel@edk2.groups.io Cc: Pandya, Puja ; Bjorge, Erik C Subject: Re: [PATCH 1/4] EdkRepo: Installer should remove obsolete dependen= cies Hi Bret, I alluded to this a little bit in the cover letter, completely agreed a mor= e detailed explanation is needed. There is an odd situation with the way GitPython=92s dependencies are evolv= ed over time. Initially, they had the following dependency graph: * GitPython * gitdb * smmap At some point in the past, they decided to make backwards incompatible API = changes to gitdb and smmap, so the dependencies where updated as follows: * GitPython * gitdb2 * smmap2 However, now that Python 2.x is deprecated, any code that was using the old= API will need updates anyway. So, the GitPython maintainers decided to go = back to the original dependency tree. The issue is, gitdb2 and smmap2 wheel= s create packages in site-packages with the names gitdb and smmap, so insta= lling gitdb2 and gitdb at the same time results in the gitdb folder in site= -packages being overwritten. End result pretty crappy, whichever wheel was = most recently installed is the one that will prevail in site-packages. The way the GitPython maintainers addressed this problem is by converting g= itdb2 and smmap2 into wheels with no content other than a dependency on git= db and smmap. So when one does a pip upgrade those packages remain and you = get a dependency graph that looks like this: * GitPython * gitdb2 * smmap2 * gitdb2 * gitdb * smmap * smmap2 * smmap So, the net result for a regular pip upgrade leaves a fair amount of detrit= us sitting on the system which would not be present for a new user doing a = clean install from pip. Obviously, we can do better than that which is why = I=92ve automated the uninstallation of smmap2 and gitdb2 before proceeding = to upgrade GitPython. This also guarantees that the duplicate folder in sit= e-packages situation does not happen at the same time. The fact that the GitPython maintainers created this situation in the first= place is rather annoying. Thanks, Nate From: Bret Barkelew > Date: Thursday, March 19, 2020 at 12:44 PM To: Ashley E Desimone >, "Desimone, Nathaniel L" >, "devel@edk2.groups.io" > Cc: "Pandya, Puja" >, "= Bjorge, Erik C" > Subject: RE: [PATCH 1/4] EdkRepo: Installer should remove obsolete dependen= cies Na=EFve question: why isn=92t this upgrade handled when pip installs the ne= w wheel? Are the dependencies not matched up correctly? I would think that = pip would uninstall the old version on its own. - Bret From: Desimone, Ashley E Sent: Thursday, March 19, 2020 12:32 PM To: Desimone, Nathaniel L; devel@edk= 2.groups.io Cc: Pandya, Puja; Bjorge, Erik C; Bret Barkelew Subject: [EXTERNAL] RE: [PATCH 1/4] EdkRepo: Installer should remove obsole= te dependencies For the following section why are we singling out gitdb2 and smmap2 when de= leting obsolete dependencies? Is there an alternative to hardcoding these v= alues? + if (DeleteObsoletePackages) + { + // + // Delete obsolete dependencies + // + foreach (string PackageName in new string[] { "smmap2"= , "gitdb2" }) + { + if (InstalledPackages.Where(p =3D> p.Name =3D=3D P= ackageName).FirstOrDefault() !=3D null) + { + InstallLogger.Log(string.Format("Uninstalling = {0}", PackageName)); + PythonOperations.UninstallPythonPackage(Python= Path, PackageName); + } + } + } Thanks, Ashley -----Original Message----- From: Desimone, Nathaniel L > Sent: Wednesday, March 18, 2020 8:17 PM To: devel@edk2.groups.io Cc: Desimone, Ashley E >; Pandya, Puja >; Bjorge, Erik C >; Bret Barkelew > Subject: [PATCH 1/4] EdkRepo: Installer should remove obsolete dependencies Cc: Ashley DeSimone > Cc: Puja Pandya > Cc: Erik Bjorge > Cc: Bret Barkelew > Signed-off-by: Nate DeSimone > --- .../EdkRepoInstaller/InstallWorker.cs | 30 +++++++++- edkrepo_installer/linux-scripts/install.py | 58 ++++++++++++------- 2 files changed, 66 insertions(+), 22 deletions(-) diff --git a/edkrepo_installer/EdkRepoInstaller/InstallWorker.cs b/edkrepo_= installer/EdkRepoInstaller/InstallWorker.cs index 8d824f2..b44e8fa 100644 --- a/edkrepo_installer/EdkRepoInstaller/InstallWorker.cs +++ b/edkrepo_installer/EdkRepoInstaller/InstallWorker.cs @@ -710,6 +710,7 @@ namespace TianoCore.EdkRepoInstaller string PythonPath; bool Has32Bit; bool Has64Bit; + bool DeleteObsoletePackages =3D false; PythonOperations.GetPythonBitness(PyInstance.Version.Major= , PyInstance.Version.Minor, out Has32Bit, out Has64Bit); if (Has32Bit && Has64Bit && PyInstance.Architecture =3D=3D= CpuArchitecture.IA32) { @@ -720,14 +721,39 @@ namespace TianoCore.EdkRepoInstaller PythonPath =3D PythonOperations.FindPython(PyInstance.= Version.Major, PyInstance.Version.Minor, false); } List InstalledPackages =3D PythonOperations= .GetInstalledPythonPackages(PythonPath); - foreach(PythonWheel Wheel in PyInstance.Wheels) + foreach (PythonWheel Wheel in PyInstance.Wheels) + { + //If a package is already installed, check if we have = a newer version bundled in the installer + //If yes, the package will be upgraded, make sure obso= lete packages are uninstalled first + PythonPackage InstalledPackage =3D InstalledPackages.W= here(p =3D> p.Name =3D=3D Wheel.Package.Name.Replace('_', '-')).FirstOrDefa= ult(); + if (InstalledPackage !=3D null && InstalledPackage.Ver= sion < Wheel.Package.Version) + { + DeleteObsoletePackages =3D true; + break; + } + } + if (DeleteObsoletePackages) + { + // + // Delete obsolete dependencies + // + foreach (string PackageName in new string[] { "smmap2"= , "gitdb2" }) + { + if (InstalledPackages.Where(p =3D> p.Name =3D=3D P= ackageName).FirstOrDefault() !=3D null) + { + InstallLogger.Log(string.Format("Uninstalling = {0}", PackageName)); + PythonOperations.UninstallPythonPackage(Python= Path, PackageName); + } + } + } + foreach (PythonWheel Wheel in PyInstance.Wheels) { //PythonPackage InstalledPackage =3D (from package in = InstalledPackages // where package.Name= =3D=3D Wheel.Package.Name // select package).Fi= rstOrDefault(); //If the package is already installed, check if we hav= e a newer version bundled in the installer //If so, upgrade the package - PythonPackage InstalledPackage =3D InstalledPackages.W= here(p =3D> p.Name =3D=3D Wheel.Package.Name).FirstOrDefault(); + PythonPackage InstalledPackage =3D + InstalledPackages.Where(p =3D> p.Name =3D=3D Wheel.Package.Name.Replace('= _', + '-')).FirstOrDefault(); if (InstalledPackage !=3D null) { if (InstalledPackage.Version < Wheel.Package.Versi= on) diff --git a/edkrepo_installer/linux-scripts/install.py b/edkrepo_insta= ller/linux-scripts/install.py index 0ea486d..b2cdfed 100755 --- a/edkrepo_installer/linux-scripts/install.py +++ b/edkrepo_installer/linux-scripts/install.py @@ -3,7 +3,7 @@ ## @file # install.py # -# Copyright (c) 2018 - 2019, Intel Corporation. All rights reserved.
+# Copyright (c) 2018 - 2020, Intel Corporation. All rights +reserved.
# SPDX-License-Identifier: BSD-2-Clause-Patent # @@ -53,6 +53,26 @@ def get_args(): parser.add_argument('-v', '--verbose', action=3D'store_true', default= =3DFalse, help=3D'Enables verbose output') return parser.parse_args() +def get_installed_packages(python_command): + pip_cmd =3D [def_python, '-m', 'pip', 'list', '--legacy'] + try: + res =3D default_run(pip_cmd) + except: + pip_cmd.pop(-1) + try: + res =3D default_run(pip_cmd) + except: + return ret_val + installed_packages =3D {} + for pip_mod in res.stdout.split('\n'): + try: + name, version =3D pip_mod.split() + version =3D version.strip().strip('()') + except: + continue + installed_packages[name] =3D version + return installed_packages + def get_required_wheels(): ret_val =3D collections.OrderedDict() if platform.machine() =3D=3D 'x86_64': @@ -79,27 +99,11 @@ def get_required_wheels(): 'version':wheel.attrib['V= ersion'], 'install':True} break - pip_cmd =3D [def_python, '-m', 'pip', 'list', '--legacy'] - try: - res =3D default_run(pip_cmd) - except: - pip_cmd.pop(-1) - try: - res =3D default_run(pip_cmd) - except: - return ret_val - installed_modules =3D {} - for pip_mod in res.stdout.split('\n'): - try: - name, version =3D pip_mod.split() - version =3D version.strip().strip('()') - except: - continue - installed_modules[name] =3D version + installed_packages =3D get_installed_packages(def_python) for name in ret_val: #pip doesn't understand the difference between '_' and '-' - if name.replace('_','-') in installed_modules: - version =3D installed_modules[name.replace('_','-')] + if name.replace('_','-') in installed_packages: + version =3D installed_packages[name.replace('_','-')] if _check_version(version, ret_val[name]['version']) >=3D 0 an= d not ret_val[name]['uninstall']: ret_val[name]['install'] =3D False else: @@ -328,16 +332,30 @@ def do_install(): if not os.path.isdir(whl_src_dir): log.info('- Missing wheel file directory') return 1 + updating_edkrepo =3D False for whl_name in wheels_to_install: uninstall_whl =3D wheels_to_install[whl_name]['uninstall'] whl_name =3D whl_name.replace('_','-') #pip doesn't understan= d the difference between '_' and '-' if uninstall_whl: + updating_edkrepo =3D True try: res =3D default_run([def_python, '-m', 'pip', 'uninsta= ll', '--yes', whl_name]) except: log.info('- Failed to uninstall {}'.format(whl_name)) return 1 log.info('+ Uninstalled {}'.format(whl_name)) + + #Delete obsolete dependencies + if updating_edkrepo: + installed_packages =3D get_installed_packages(def_python) + for whl_name in ['smmap2', 'gitdb2']: + if whl_name in installed_packages: + try: + res =3D default_run([def_python, '-m', 'pip', 'uni= nstall', '--yes', whl_name]) + except: + log.info('- Failed to uninstall {}'.format(whl_nam= e)) + return 1 + log.info('+ Uninstalled {}'.format(whl_name)) for whl_name in wheels_to_install: whl =3D wheels_to_install[whl_name]['wheel'] install_whl =3D wheels_to_install[whl_name]['install'] -- 2.24.0.windows.2 --_000_MWHPR21MB075064C54FB19A90275F6836EFF50MWHPR21MB0750namp_ Content-Type: text/html; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable

Spiffy. Agreed; thanks for the background.

 

My only other concern would be the function name and= if some future developer thought that was permission to add other modules = to it.

Then again, maybe I=92m just bikeshedding. Feel free= to ignore.

 

- Bret

 

From: Desimone, Ashley E
Sent: Friday, March 20, 2020 1:21 PM
To: Desimone, Nath= aniel L; Bret Barkelew; devel@edk2.groups.io
Cc: Pandya, Puja; Bjorge, Erik C
Subject: [EXTERNAL] RE: [PATCH 1/4] EdkRepo: Installer should remove= obsolete dependencies

 

Thanks for the explanation.

 

Reviewed-by: Ashley DeSimone <ashley.e.deismone@i= ntel.com>

 

From: Desimone, Nathaniel L <nathaniel.l.d= esimone@intel.com>
Sent: Thursday, March 19, 2020 2:01 PM
To: Bret Barkelew <Bret.Barkelew@microsoft.com>; Desimone, Ash= ley E <ashley.e.desimone@intel.com>; devel@edk2.groups.io
Cc: Pandya, Puja <puja.pandya@intel.com>; Bjorge, Erik C <e= rik.c.bjorge@intel.com>
Subject: Re: [PATCH 1/4] EdkRepo: Installer should remove obsolete d= ependencies

 

Hi Bret,

 

I alluded to this a little bit in the cover letter, = completely agreed a more detailed explanation is needed.

 

There is an odd situation with the way GitPython=92s= dependencies are evolved over time. Initially, they had the following depe= ndency graph:

 

  • GitPython
    • gitdb
      • smmap

 

At some point in the past, they decided to make back= wards incompatible API changes to gitdb and smmap, so the dependencies wher= e updated as follows:

 

  • GitPython
    • gitdb2
      • smmap2

 

However, now that Python 2.x is deprecated, any code= that was using the old API will need updates anyway. So, the GitPython mai= ntainers decided to go back to the original dependency tree. The issue is, = gitdb2 and smmap2 wheels create packages in site-packages with the names gitdb and smmap, so installing gitdb2 and = gitdb at the same time results in the gitdb folder in site-packages being o= verwritten. End result pretty crappy, whichever wheel was most recently ins= talled is the one that will prevail in site-packages.

 

The way the GitPython maintainers addressed this pro= blem is by converting gitdb2 and smmap2 into wheels with no content other t= han a dependency on gitdb and smmap. So when one does a pip upgrade those p= ackages remain and you get a dependency graph that looks like this:

 

  • GitPython
    • gitdb2
      • smmap2
  • gitdb2
    • gitdb
      • smmap
  • smmap2
    • smmap

 

 

So, the net result for a regular pip upgrade leaves = a fair amount of detritus sitting on the system which would not be present = for a new user doing a clean install from pip. Obviously, we can do better = than that which is why I=92ve automated the uninstallation of smmap2 and gitdb2 before proceeding to upgrade GitPy= thon. This also guarantees that the duplicate folder in site-packages situa= tion does not happen at the same time.

 

The fact that the GitPython maintainers created this= situation in the first place is rather annoying.

 

Thanks,

Nate

 

From= : Bret Barkelew <= ;Bret.Barkelew@microsoft.com= >
Date: Thursday, March 19, 2020 at 12:44 PM
To: Ashley E Desimone <ashley.e.desimone@intel.com>, "Desimone, Nathaniel L"= <nathaniel.l.desimone= @intel.com>, "devel@edk= 2.groups.io" <devel@edk2.groups.io> Cc: "Pandya, Puja" <puja.pandya@intel.com>, "Bjorge, Erik C" <erik.c.bjorge@intel.com>
Subject: RE: [PATCH 1/4] EdkRepo: Installer should remove obsolete d= ependencies

 

Na=EFve question: why isn=92t this upgrade handled w= hen pip installs the new wheel? Are the dependencies not matched up correct= ly? I would think that pip would uninstall the old version on its own.=

 

- Bret

 

From: Desimone, Ashley E
Sent: Thursday, March 19, 2020 12:32 PM
To: Desimone, Nath= aniel L; devel@edk2.groups.io
Cc: Pandya, Puja; Bjorge, Erik C; Bret Bar= kelew
Subject: [EXTERNAL] RE: [PATCH 1/4] EdkRepo: Installer should remove= obsolete dependencies

 

For the following sec= tion why are we singling out gitdb2 and smmap2 when deleting obsolete depen= dencies? Is there an alternative to hardcoding these values?

+           &nbs= p;    if (DeleteObsoletePackages)
+           &nbs= p;    {
+           &nbs= p;        //
+           &nbs= p;        // Delete obsolete dependencie= s
+           &nbs= p;        //
+           &nbs= p;        foreach (string PackageName in= new string[] { "smmap2", "gitdb2" })
+           &nbs= p;        {
+           &nbs= p;            if (In= stalledPackages.Where(p =3D> p.Name =3D=3D PackageName).FirstOrDefault()= !=3D null)
+           &nbs= p;            {
+           &nbs= p;            &= nbsp;   InstallLogger.Log(string.Format("Uninstalling {0}&qu= ot;, PackageName));
+           &nbs= p;            &= nbsp;   PythonOperations.UninstallPythonPackage(PythonPath, Packa= geName);
+           &nbs= p;            }
+           &nbs= p;        }
+           &nbs= p;    }

Thanks,
Ashley

-----Original Message-----
From: Desimone, Nathaniel L <nathaniel.l.desimone@intel.com>
Sent: Wednesday, March 18, 2020 8:17 PM
To: devel@edk2.groups.io
Cc: Desimone, Ashley E <a= shley.e.desimone@intel.com>; Pandya, Puja <puja.pandya@intel.com>; Bjorge, Erik C <erik.c.bjorge@intel.com>; Bret Barkelew <Bret.Bark= elew@microsoft.com>
Subject: [PATCH 1/4] EdkRepo: Installer should remove obsolete dependencies=

Cc: Ashley DeSimone <ashl= ey.e.desimone@intel.com>
Cc: Puja Pandya <puja.pandya@in= tel.com>
Cc: Erik Bjorge <erik.c.bjorg= e@intel.com>
Cc: Bret Barkelew <Bret.B= arkelew@microsoft.com>
Signed-off-by: Nate DeSimone <nathaniel.l.desimone@intel.com>
---
 .../EdkRepoInstaller/InstallWorker.cs     &n= bsp;   | 30 +++++++++-
 edkrepo_installer/linux-scripts/install.py    | 58 = 3;+++++++++++-------
 2 files changed, 66 insertions(+), 22 deletions(-)

diff --git a/edkrepo_installer/EdkRepoInstaller/InstallWorker.cs b/edkrepo_= installer/EdkRepoInstaller/InstallWorker.cs
index 8d824f2..b44e8fa 100644
--- a/edkrepo_installer/EdkRepoInstaller/InstallWorker.cs
+++ b/edkrepo_installer/EdkRepoInstaller/InstallWorker.cs
@@ -710,6 +710,7 @@ namespace TianoCore.EdkRepoInstaller
            &nb= sp;    string PythonPath;
            &nb= sp;    bool Has32Bit;
            &nb= sp;    bool Has64Bit;
+           &nbs= p;    bool DeleteObsoletePackages =3D false;
            &nb= sp;    PythonOperations.GetPythonBitness(PyInstance.Version.= Major, PyInstance.Version.Minor, out Has32Bit, out Has64Bit);
            &nb= sp;    if (Has32Bit && Has64Bit && PyInstanc= e.Architecture =3D=3D CpuArchitecture.IA32)
            &nb= sp;    {
@@ -720,14 +721,39 @@ namespace TianoCore.EdkRepoInstaller
            &nb= sp;        PythonPath =3D PythonOperatio= ns.FindPython(PyInstance.Version.Major, PyInstance.Version.Minor, false);             &nb= sp;    }
            &nb= sp;    List<PythonPackage> InstalledPackages =3D Pytho= nOperations.GetInstalledPythonPackages(PythonPath);
-            &n= bsp;   foreach(PythonWheel Wheel in PyInstance.Wheels)
+           &nbs= p;    foreach (PythonWheel Wheel in PyInstance.Wheels)
+           &nbs= p;    {
+           &nbs= p;        //If a package is already inst= alled, check if we have a newer version bundled in the installer
+           &nbs= p;        //If yes, the package will be = upgraded, make sure obsolete packages are uninstalled first
+           &nbs= p;        PythonPackage InstalledPackage= =3D InstalledPackages.Where(p =3D> p.Name =3D=3D Wheel.Package.Name.Rep= lace('_', '-')).FirstOrDefault();
+           &nbs= p;        if (InstalledPackage !=3D null= && InstalledPackage.Version < Wheel.Package.Version)
+           &nbs= p;        {
+           &nbs= p;            Delete= ObsoletePackages =3D true;
+           &nbs= p;            break;=
+           &nbs= p;        }
+           &nbs= p;    }
+           &nbs= p;    if (DeleteObsoletePackages)
+           &nbs= p;    {
+           &nbs= p;        //
+           &nbs= p;        // Delete obsolete dependencie= s
+           &nbs= p;        //
+           &nbs= p;        foreach (string PackageName in= new string[] { "smmap2", "gitdb2" })
+           &nbs= p;        {
+           &nbs= p;            if (In= stalledPackages.Where(p =3D> p.Name =3D=3D PackageName).FirstOrDefault()= !=3D null)
+           &nbs= p;            {
+           &nbs= p;            &= nbsp;   InstallLogger.Log(string.Format("Uninstalling {0}&qu= ot;, PackageName));
+           &nbs= p;            &= nbsp;   PythonOperations.UninstallPythonPackage(PythonPath, Packa= geName);
+           &nbs= p;            }
+           &nbs= p;        }
+           &nbs= p;    }
+           &nbs= p;    foreach (PythonWheel Wheel in PyInstance.Wheels)
            &nb= sp;    {
            &nb= sp;        //PythonPackage InstalledPack= age =3D (from package in InstalledPackages
            &nb= sp;        //    &nb= sp;            =             &nb= sp;    where package.Name =3D=3D Wheel.Package.Name
            &nb= sp;        //    &nb= sp;            =             &nb= sp;    select package).FirstOrDefault();
            &nb= sp;        //If the package is already i= nstalled, check if we have a newer version bundled in the installer
            &nb= sp;        //If so, upgrade the package<= br> -            &n= bsp;       PythonPackage InstalledPackage =3D= InstalledPackages.Where(p =3D> p.Name =3D=3D Wheel.Package.Name).FirstO= rDefault();
+           &nbs= p;        PythonPackage InstalledPackage= =3D
+ InstalledPackages.Where(p =3D> p.Name =3D=3D Wheel.Package.Name.Re= place('_',
+ '-')).FirstOrDefault();
            &nb= sp;        if (InstalledPackage !=3D nul= l)
            &nb= sp;        {
            &nb= sp;            if (I= nstalledPackage.Version < Wheel.Package.Version) diff --git a/edkrepo_in= staller/linux-scripts/install.py b/edkrepo_installer/linux-scripts/install.= py
index 0ea486d..b2cdfed 100755
--- a/edkrepo_installer/linux-scripts/install.py
+++ b/edkrepo_installer/linux-scripts/install.py
@@ -3,7 +3,7 @@
 ## @file
 # install.py
 #
-# Copyright (c) 2018 - 2019, Intel Corporation. All rights reserved.<BR= >
+# Copyright (c) 2018 - 2020, Intel Corporation. All rights
+reserved.<BR>
 # SPDX-License-Identifier: BSD-2-Clause-Patent  #
 
@@ -53,6 +53,26 @@ def get_args():
     parser.add_argument('-v', '--verbose', action=3D's= tore_true', default=3DFalse, help=3D'Enables verbose output')
     return parser.parse_args()
 
+def get_installed_packages(python_command):
+    pip_cmd =3D [def_python, '-m', 'pip', 'list', '--le= gacy']
+    try:
+        res =3D default_run(pip_cmd= )
+    except:
+        pip_cmd.pop(-1)
+        try:
+            res= =3D default_run(pip_cmd)
+        except:
+            ret= urn ret_val
+    installed_packages =3D {}
+    for pip_mod in res.stdout.split('\n'):
+        try:
+            nam= e, version =3D pip_mod.split()
+            ver= sion =3D version.strip().strip('()')
+        except:
+            con= tinue
+        installed_packages[name] = =3D version
+    return installed_packages
+
 def get_required_wheels():
     ret_val =3D collections.OrderedDict()
     if platform.machine() =3D=3D 'x86_64':
@@ -79,27 +99,11 @@ def get_required_wheels():
            &nb= sp;            =             &nb= sp;            'vers= ion':wheel.attrib['Version'],
            &nb= sp;            =             &nb= sp;            'inst= all':True}
             br= eak
-    pip_cmd =3D [def_python, '-m', 'pip', 'list', '--legacy= ']
-    try:
-        res =3D default_run(pip_cmd) -    except:
-        pip_cmd.pop(-1)
-        try:
-            res =3D= default_run(pip_cmd)
-        except:
-            return = ret_val
-    installed_modules =3D {}
-    for pip_mod in res.stdout.split('\n'):
-        try:
-            name, v= ersion =3D pip_mod.split()
-            version= =3D version.strip().strip('()')
-        except:
-            continu= e
-        installed_modules[name] =3D ver= sion
+    installed_packages =3D get_installed_packages(def_p= ython)
     for name in ret_val:
         #pip doesn't understand th= e difference between '_' and '-'
-        if name.replace('_','-') in ins= talled_modules:
-            version= =3D installed_modules[name.replace('_','-')]
+        if name.replace('_','-') in= installed_packages:
+            ver= sion =3D installed_packages[name.replace('_','-')]
             if= _check_version(version, ret_val[name]['version']) >=3D 0 and not ret_va= l[name]['uninstall']:
            &nb= sp;    ret_val[name]['install'] =3D False
             el= se:
@@ -328,16 +332,30 @@ def do_install():
         if not os.path.isdir(whl_s= rc_dir):
             lo= g.info('- Missing wheel file directory')
             re= turn 1
+        updating_edkrepo =3D False<= br>          for whl_name in wheels_to_= install:
             un= install_whl =3D wheels_to_install[whl_name]['uninstall']
             wh= l_name =3D whl_name.replace('_','-')  #pip doesn't understand the diff= erence between '_' and '-'
             if= uninstall_whl:
+           &nbs= p;    updating_edkrepo =3D True
            &nb= sp;    try:
            &nb= sp;        res =3D default_run([def_pyth= on, '-m', 'pip', 'uninstall', '--yes', whl_name])
            &nb= sp;    except:
            &nb= sp;        log.info('- Failed to uninsta= ll {}'.format(whl_name))
            &nb= sp;        return 1
            &nb= sp;    log.info('+ Uninstalled {}'.format(whl_name))
+
+        #Delete obsolete dependenci= es
+        if updating_edkrepo:
+            ins= talled_packages =3D get_installed_packages(def_python)
+            for= whl_name in ['smmap2', 'gitdb2']:
+           &nbs= p;    if whl_name in installed_packages:
+           &nbs= p;        try:
+           &nbs= p;            res = =3D default_run([def_python, '-m', 'pip', 'uninstall', '--yes', whl_name])<= br> +           &nbs= p;        except:
+           &nbs= p;            log.in= fo('- Failed to uninstall {}'.format(whl_name))
+           &nbs= p;            return= 1
+           &nbs= p;        log.info('+ Uninstalled {}= '.format(whl_name))
         for whl_name in wheels_to_= install:
             wh= l =3D wheels_to_install[whl_name]['wheel']
             in= stall_whl =3D wheels_to_install[whl_name]['install']
--
2.24.0.windows.2

 

 

--_000_MWHPR21MB075064C54FB19A90275F6836EFF50MWHPR21MB0750namp_--