From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail02.groups.io (mail02.groups.io [66.175.222.108]) by spool.mail.gandi.net (Postfix) with ESMTPS id 5D40B941E18 for ; Tue, 13 Feb 2024 19:42:17 +0000 (UTC) DKIM-Signature: a=rsa-sha256; bh=nUPMSjAWlaM6EwmUwafp9vrwwSEoTb+2gkZmfT5uGVo=; c=relaxed/simple; d=groups.io; h=ARC-Seal:ARC-Message-Signature:ARC-Authentication-Results:From:To:Subject:Thread-Topic:Thread-Index:Date:Message-ID:References:In-Reply-To:Accept-Language:msip_labels:MIME-Version:Precedence:List-Subscribe:List-Help:Sender:List-Id:Mailing-List:Delivered-To:Reply-To:List-Unsubscribe-Post:List-Unsubscribe:Content-Language:Content-Type:Content-Transfer-Encoding; s=20140610; t=1707853336; v=1; b=vWE5wWH2Ay9LnAWXfXYVVzxtVCHvfPW4BhyTiGG5LddDAIsz+XmjtAyKwKYBtaJEAc68OKNG u3bvJpY0SUt2rMLBDV5CX98sCcm23JfB8EQkL2DkA7T4vds3owBg1/qgIBWCFFBFlEgQc8wZFOE jyz/FHvfKXpCvvHtNBrPd/Zg= X-Received: by 127.0.0.2 with SMTP id R6V2YY7687511xdYi5i2yjAN; Tue, 13 Feb 2024 11:42:16 -0800 X-Received: from BL2PR02CU003.outbound.protection.outlook.com (BL2PR02CU003.outbound.protection.outlook.com [52.101.51.11]) by mx.groups.io with SMTP id smtpd.web11.23044.1707853329873153677 for ; Tue, 13 Feb 2024 11:42:15 -0800 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=n0Y6zCQVEmKAQdUl+M+UhU07hMzyt9nQINWwvwIRI+Uqg5uCeAwChT2c8poP4dtJhSmrmPuSxD+1SGShGwtB86j8Va9yRa0/zv1mxROIfrWXEHGuW/FsaRmn1vkEcTixzaZ5FRu5Um4FB40ik4RI+P/mowtNWEyJ8ZvLOv8JSDENY8nRfoBSZwr3SP5/+zbYvkgNnx9FUrLceJjWopgEfghQXdI321oCNHX5c4Ykix5roZrnKd1L0ged025mw9hVOrO1FT2AkiqGGnEYctd9E066DEd2jRFsYbcAAYlxDTHG2+pVCe481bntmBhnhzvxRR93wxm5T7hNP11oM6QpaQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=M9TiNTOV59Y6fWpYC/G/35OCl/yiDP2Y3vXRSIQFy0I=; b=BzWpU83I75NcPC+JrXcSEIHKeuS0XMCwlNhhd5J1nbpFkGgZ4u7M6OXJ693yKQk5t/SOgKXS38t+hmylW/W7VfnHEpBFb5XStZOGgb5izh/zodX6XLeNynIhk5nVQLAK/yP6htqLRs3ATVpjUzU0lp0kdwuZ9rc5eFvdYCmmTNAfiSPmbvvNBHy/0qc0TAyhnrNFd8f/IzKA9KPzsMan5p+ekojEKkvHNvurzYqTeavW/nljriETkBdPBIfvUW8TpBnCOCp9YSQvS3EmfepHsmShdVXp19pLOA1NzNnD8BUlyyET6plEnbwv/H7Eu+WUj8cOcsXQg7aPDuKSlgFpbQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=microsoft.com; dmarc=pass action=none header.from=microsoft.com; dkim=pass header.d=microsoft.com; arc=none X-Received: from BY1PR21MB3943.namprd21.prod.outlook.com (2603:10b6:a03:52c::19) by SJ0PR21MB1998.namprd21.prod.outlook.com (2603:10b6:a03:298::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7316.8; Tue, 13 Feb 2024 19:42:03 +0000 X-Received: from BY1PR21MB3943.namprd21.prod.outlook.com ([fe80::4c42:b704:da2d:16cb]) by BY1PR21MB3943.namprd21.prod.outlook.com ([fe80::4c42:b704:da2d:16cb%3]) with mapi id 15.20.7316.006; Tue, 13 Feb 2024 19:42:03 +0000 From: "Joey Vagedes via groups.io" To: Rebecca Cran , Rebecca Cran , "devel@edk2.groups.io" , "Kinney, Michael D" , Sean , Michael Kubacki Subject: Re: [edk2-devel] Fixing edk2-basetools CI Thread-Topic: [EXTERNAL] Re: [edk2-devel] Fixing edk2-basetools CI Thread-Index: AQHaXdJ3bjlGHvswWEy4mmcLM7dG5rEG6XswgAHAYACAAABDwA== Date: Tue, 13 Feb 2024 19:42:03 +0000 Message-ID: References: <2fe83108-e19b-4c95-8c45-08f43594e8cd@bsdio.com> <2c52fbac-dbe1-423a-a9d7-0bbe473cd9e9@os.amperecomputing.com> In-Reply-To: Accept-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: msip_labels: MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_ActionId=cb52b067-7a71-4767-854f-7d8abeb06b26;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_ContentBits=0;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Enabled=true;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Method=Standard;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Name=Internal;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_SetDate=2024-02-13T19:28:54Z;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_SiteId=72f988bf-86f1-41af-91ab-2d7cd011db47; x-ms-publictraffictype: Email x-ms-traffictypediagnostic: BY1PR21MB3943:EE_|SJ0PR21MB1998:EE_ x-ms-office365-filtering-correlation-id: eaa25a1e-167a-4304-c051-08dc2ccbda84 x-ld-processed: 72f988bf-86f1-41af-91ab-2d7cd011db47,ExtAddr x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam-message-info: Fx/PpsSVgJkRgjA4PJphiC8biG1LgTe/LJaQgB0sh+h3N/ubOizG2yDtbUxfMcvjHVKOZjerHnqeSAhMGVhuEdSfG6A84eTcMNOYFWVOrvmeDZpFaGt9hlCBefIM23dl3HVKLliX+d2kb35EQouCt+d9ywdJsKHgV25TCS9AjSuVvhNU1J8TvQ2KM9HW6eAt+qCICowobp26ebo+Y28DOYs8G8uATQCQhY+/b9mrqHRZiWk+QNpi5ggR2Og6rOxn3Kaa5nbsPy8/lFAW6NhbJocHe9lKbYFOs4azc3n5cNVBb9C3gUtn5YeFa4YMUjK7NTXAEGTkbOWEG9QU4c2bgAkTwkw/GRtRS/SbTFL4GrCO6HxxrU4yfZiPli96cEndwIoxrzKTBft5t+sjr8n1QCV5gb0xNBgxoikHmh7W0Zuctm+BviIGnCZdlADY8m6dLh+mb61jb+xWG29bQhFlUb/ec3S8QpQ0LAgBCbna7YNuYLL1bWDDV/6ubA36THiHjX3vYSFLfZhcpgJ8ZsFnJoBvjUz/c1G79rQdClZ1RWdemZq7LoKwJ+bScejuKhWW2DKbLACh/yHzlOibUzl1NHIgTPBvMniArMeuzU2zFHK1ns2rX/+cAnt+QzfXdcgn6LHtXRdsQjN3Ayl7rpENug== x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?YpFjP6B9b5SRtbsiN0lidXhBKUNYnhr/RsAGiEdAyMqBoXZHcvv5lAOVc9fm?= =?us-ascii?Q?ODpBxz0U+WjgKr62qGB0pcwWIJ3e3Uve+wXph1F56kDJylY+bzweaUJADQHb?= =?us-ascii?Q?HYcORUaGT4mPcpOM/ofwLTYjZPsuFgAw4UDG/w1vmzkVOyseBt6VSNJlhXev?= =?us-ascii?Q?9EN5oDYiBJU4+OGIqYWF9pRi/7aoVZ4hK2GXl7w04y36YIOcMBydHotGAC1j?= =?us-ascii?Q?Pe+fWHkpDXI8qBq/r9apA/BKvaWNo8N5+grmNviFHoS7c+Aem1jG+jKs6tuV?= =?us-ascii?Q?WfgOlIl/NkBVKeA5v5/rO/Q/FDS9BgxOg3Bzuj6tkhhjMllYXnvpygzpepw1?= =?us-ascii?Q?4qwSVrCpGi9Ctdsp2+z1wDp1vfr/owuIMvaTcwQS0sEDWW1kEz7bzyxfyqjY?= =?us-ascii?Q?BDXOw4wB+HkWlNRgnPup2W4uKf+pNzhlLGcimPyxInj7U0fKBQzgbIIN4reo?= =?us-ascii?Q?OewTPHWHMv4EreAxmjNop+7Oeu7w0txFp7l/ZZO9k4n4FODme82IpziwnwPH?= =?us-ascii?Q?U1Me3HDXfvfvfZ3zQK4qo0eaUA5cRCWt2xBxj30p4VX8gx6EDQ5DfggUZQlS?= =?us-ascii?Q?rhZvV5z0xSFaj7LSa6D+BYcrsL6wjPqd7a8yDDU1+biK8lF/u6RI5oQUApQm?= =?us-ascii?Q?mhGVrva6yHbWs88r8MXK9bcoEQsxRgDXrcp29EdxNQ6fy82EPW4fcwWllMpe?= =?us-ascii?Q?sI77vAPRAcM6kYnlh/kJ3nmBMqt8W+OJdcpQtzwwLeqkvYRvWY1/7P5jJxPg?= =?us-ascii?Q?p1GOR9Am8PziMS60tUnF4ohQ6fOlEFFAgrO/CPc+PmMGMf4w9xXMMLKVNHsK?= =?us-ascii?Q?7GvdVY0Y3vUNPuf5veUm0Rtrt9n44zcpIUdBobTXAWWTyxDqRC5z8epXN0Fq?= =?us-ascii?Q?bvZ1OvdtB6iAr4rJ0yqfP9Hu9DRK//k7rFLhEjFHEL3dB5UYVYsO6L6KKR5J?= =?us-ascii?Q?ldoYxnh17O6CxEOhCHCxH4xLFx5+x+2nQAVY2fdVNDQyf3tiK/7+2KZf/X5A?= =?us-ascii?Q?U/v2cEGy8X9kwN7lmQa9AVGuk33xL0LRkKkeOsLWuzAUAO69+ESCWlCXgLZY?= =?us-ascii?Q?x1aty6WdhezpkkQCeX1xHqzBnBVKhp8hQ229YK7cYF7Av0AJtQqG8MQybPSE?= =?us-ascii?Q?rfLS/KywJxVMR2JcREDOlIfRYKsI6A2F+BRiC5aCOqeQARSLZmo2l9Fm1qmg?= =?us-ascii?Q?dw6C5eZCOBYst+kQsJUSZ9QY65cLHjEtxtodBMy+PdjO0xlXhRoARzWDgKcu?= =?us-ascii?Q?fTK/VZ50vbIozBtZE5HmhZVtnD8OhDwjZcDENzKMOxTmQBU7zFoOAdS738r+?= =?us-ascii?Q?9YIEkrG1S47Ek0c1VA17/Rh90ZYKWwdGdteR/WmhbxetgMJ0UN55eevkg3cZ?= =?us-ascii?Q?K298NDDMCDgDuab1lSnwQPzrAq8DSAawFq3HWKPRs5gEr3vFmUXsDDYbzmLH?= =?us-ascii?Q?G0eqIWuhtes7Z/aigfU3AI4Vh9UbTvoFjYpSMbrG84RRKPYb03M9SfJ9eHr6?= =?us-ascii?Q?futPtFmvHxLJUF8aGI5CrfCVmEmiJRE/tIq3y+R8OjctH/eZMBZdbe8uc6ZT?= =?us-ascii?Q?UKbkOMSvE1OISsuMUExScyOjZGgFawJDpuM+hxWI?= MIME-Version: 1.0 X-OriginatorOrg: microsoft.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: BY1PR21MB3943.namprd21.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: eaa25a1e-167a-4304-c051-08dc2ccbda84 X-MS-Exchange-CrossTenant-originalarrivaltime: 13 Feb 2024 19:42:03.5280 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 72f988bf-86f1-41af-91ab-2d7cd011db47 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: YXo8Wq1eTeB7IDWZAclYqhSWwlcGH0lCby3raXC4sIw8ycpDM0Qm8kCC///l3Rb+RjxdcsoEzuywqKAtqIwiGapjJVgBeKEBf2A84x1XcZ0= X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR21MB1998 Precedence: Bulk List-Subscribe: List-Help: Sender: devel@edk2.groups.io List-Id: Mailing-List: list devel@edk2.groups.io; contact devel+owner@edk2.groups.io Reply-To: devel@edk2.groups.io,joeyvagedes@microsoft.com List-Unsubscribe-Post: List-Unsubscribe=One-Click List-Unsubscribe: X-Gm-Message-State: 9ti5Xedc96tcItqghZAlxS34x7686176AA= Content-Language: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable X-GND-Status: LEGIT Authentication-Results: spool.mail.gandi.net; dkim=pass header.d=groups.io header.s=20140610 header.b=vWE5wWH2; dmarc=pass (policy=none) header.from=groups.io; spf=pass (spool.mail.gandi.net: domain of bounce@groups.io designates 66.175.222.108 as permitted sender) smtp.mailfrom=bounce@groups.io I agree - there are multiple blocking issues that will require some fixes -= One additional thing I noted is that multiple areas still perform relative= path imports - i.e. the import starts with `from .`. These will all need to be updated to import from edk2basetools as some poin= t soon as importing via relative paths can result in it importing the wrong= file if the pypath environment variable is influenced. This open issue act= ually stems from that: https://github.com/tianocore/edk2-basetools/issues/1= 10 as the end result is that edk2-basetools pip module can unexpectedly end= up using the BaseTools/* source code rather than the pip module source cod= e. Thanks, Joey -----Original Message----- From: Rebecca Cran Sent: Tuesday, February 13, 2024 11:28 AM To: Joey Vagedes ; Rebecca Cran ; devel@edk2.groups.io; Kinney, Michael D ; Sean ; Michael Kubacki Subject: Re: [EXTERNAL] Re: [edk2-devel] Fixing edk2-basetools CI [You don't often get email from rebecca@bsdio.com. Learn why this is import= ant at https://aka.ms/LearnAboutSenderIdentification ] Thanks, I've updated https://github.com/tianocore/edk2-basetools/pull/116 with the changes you s= uggested. Unfortunately several of the tools don't work with the projects.scripts sec= tion because they don't have Main etc. It turns out even Ecc is currently b= roken in the edk2-basetools PyPI package because PYTHONPATH doesn't contain= the parent directory. I'll plan to fix the issues in the near future, once= I've worked through the other PRs. -- Rebecca Cran On 2/12/24 10:06, Joey Vagedes wrote: > Hello! Regarding flake8, I see. That will take some time to work through = - I completely understand. > > When ready to work through that, I suggest switching to ruff. It can reso= lve some errors automatically for you and allows you to customize it more t= han flake8 (it is also faster). > > I just ran ruff on the project, showing about 11k errors. One nice thing = is that a large majority of it is regarding importing using *. By fixing th= ose, you can reduce the errors by almost 70% (to 3400 errors), so it is not= *quite* as bad as it seems (though 3400 is still A LOT). > > ---------------------------------------------------------------------- > ----------------------------------------------------- > > Regarding the setup tools question, the build-backend part of pytoml (L3)= is the entry point that is invoked - so we are still invoking setuptools. = However, it pulls the necessary metadata from pyproject.toml rather than se= tup.py's setup() function. The main thing that is different is the "get_ver= sion" functionality, which we replicate in the pyproject.toml with setuptoo= ls-scm[toml] which simply uses the latest tag to set the version. > > ---------------------------------------------------------------------- > ----------------------------------------------------- > > Additionally, I wanted to comment on the wrappers that you mentioned. > > Using the [projects.scripts] section (i.e. CLI Commands feature) is in my= opinion the correct way to move forward as users will be able to invoke co= mmands themselves easily (i.e. just call Bin2Pcd, and it will work). Howeve= r, the wrappers will continue to be necessary until all python source is re= moved from Basetools, so that users can toggle between pip or local. Howeve= r, once source is removed form BaseTools, removing the wrappers is the next= step forward. > > In the meantime, you will be able to update the binpipwrappers to somethi= ng akin to the following (untested, so I don't garuntee it will be exactly = this: > > @setlocal > Ecc %* > > Rather than > > @setlocal > @set ToolName=3D%~n0% > @%PYTHON_COMMAND% -m edk2basetools.%ToolName%.EccMain %* > > ---------------------------------------------------------------------- > ----------------------------------------------------- > > One thing to keep in mind once Python is completely removed from BaseTool= s, is that consumers will need to get in the habit of verifying the correct= version of basetools is installed for the commit they are on (or automate = this checking) particularly for anyone that regularly switches between stab= le branches. > > Thanks, > Joey > > -----Original Message----- > From: Rebecca Cran > Sent: Monday, February 12, 2024 8:42 AM > To: devel@edk2.groups.io; Joey Vagedes ; > Rebecca Cran ; Kinney, Michael D > ; Sean ; Michael > Kubacki > Subject: [EXTERNAL] Re: [edk2-devel] Fixing edk2-basetools CI > > [You don't often get email from rebecca@os.amperecomputing.com. Learn > why this is important at https://aka.ms/LearnAboutSenderIdentification > ] > > On 2/12/2024 9:08 AM, Joey Vagedes via groups.io wrote: >> Hello. It can be simplified - You can view the pyproject.toml for edk2-p= ytool-library and edk2-pytool-extensions: >> >> https://git/ >> h%2F&data=3D05%7C02%7Cjoeyvagedes%40microsoft.com%7Cf64b5d2ea86245c44e5 >> 208dc2cc9e93d%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6384344929 >> 36969249%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIi >> LCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=3DORN2iHQhOfzkN9F462p >> 2H6mb8Hl9ubIIeX%2BDBorzmC0%3D&reserved=3D0 >> ub.com%2Ftianocore%2Fedk2-pytool-library%2Fblob%2Fmaster%2Fpyproject. >> t >> oml&data=3D05%7C02%7Cjoeyvagedes%40microsoft.com%7Ca37eecc569d54b5dbbf3 >> 0 >> 8dc2be996f3%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638433529483 >> 1 >> 52477%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJ >> B >> TiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=3D8RSQNcjgH02JyPiGCKmyf2u >> 7 >> Ep6BC9ldwiND0ULj9Zs%3D&reserved=3D0 >> >> Also, Why disable flake8 rather than fix the changes? I also suggest swi= tching to ruff, as it does the same thing but better, and with way more rul= es / configuration options. > Thanks. I'm not sure that looks so much simpler than my pyproject.toml bu= t I'll update it to follow the same layout. Also, pyproject.toml says it's = using the setuptools build backend but you don't have a setup.py: > how does that work? > > I decided to disable flake8 for now because the BaseTools Python code is = very non-compliant: the log file has over 41,000 lines. In most places it l= ooks like people have been trying to write it as C code. > > A few of the lines: > > .\edk2basetools\Common\TargetTxtClassObject.py:44:80: E203 whitespace bef= ore ':' > .\edk2basetools\Common\TargetTxtClassObject.py:61:14: E111 indentation > is not a multiple of 4 > .\edk2basetools\Common\TargetTxtClassObject.py:61:14: E117 > over-indented > .\edk2basetools\Common\TargetTxtClassObject.py:63:50: F405 'FILE_NOT_FOUN= D' may be undefined, or defined from star imports: > .BuildToolError > .\edk2basetools\Common\TargetTxtClassObject.py:73:121: E501 line too > long (123 > 120 characters) > .\edk2basetools\Common\TargetTxtClassObject.py:84:38: F405 'FILE_OPEN_FAI= LURE' may be undefined, or defined from star imports: > .BuildToolError > .\edk2basetools\Common\TargetTxtClassObject.py:100:108: E502 the > backslash is redundant between brackets > .\edk2basetools\Common\TargetTxtClassObject.py:108:121: E501 line too > long (139 > 120 characters) > .\edk2basetools\Common\TargetTxtClassObject.py:118:121: E501 line too > long (140 > 120 characters) > .\edk2basetools\Common\TargetTxtClassObject.py:123:97: E502 the > backslash is redundant between brackets > .\edk2basetools\Common\TargetTxtClassObject.py:128:21: F841 local > variable 'V' is assigned to but never used > > > -- > Rebecca Cran -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D- Groups.io Links: You receive all messages sent to this group. View/Reply Online (#115411): https://edk2.groups.io/g/devel/message/115411 Mute This Topic: https://groups.io/mt/104306226/7686176 Group Owner: devel+owner@edk2.groups.io Unsubscribe: https://edk2.groups.io/g/devel/unsub [rebecca@openfw.io] -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-