From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail02.groups.io (mail02.groups.io [66.175.222.108]) by spool.mail.gandi.net (Postfix) with ESMTPS id 9F36074004E for ; Mon, 12 Feb 2024 17:06:34 +0000 (UTC) DKIM-Signature: a=rsa-sha256; bh=5vPR8jlRCH5hbafBLi589ofuD/JtGu2llJjILXGtVtY=; c=relaxed/simple; d=groups.io; h=ARC-Seal:ARC-Message-Signature:ARC-Authentication-Results:From:To:Subject:Thread-Topic:Thread-Index:Date:Message-ID:References:In-Reply-To:Accept-Language:msip_labels:MIME-Version:Precedence:List-Subscribe:List-Help:Sender:List-Id:Mailing-List:Delivered-To:Reply-To:List-Unsubscribe-Post:List-Unsubscribe:Content-Language:Content-Type:Content-Transfer-Encoding; s=20140610; t=1707757593; v=1; b=GHPoAx3X6pr2JUV33jZB66ABFeZXbOvZZuhQ1zr1khJo//nm+WtdXRRf7Myn1rDiS2KNeGH0 vZBPHHzts1VLlUyf7cd+Vl1vr8aMz1xopLHvbr6nl6kbmzPCvKLqQHviVV8EoPlJdKWGo7b1Oep Z42qZMcxeQJPEz5hJ7Rbhi50= X-Received: by 127.0.0.2 with SMTP id zCghYY7687511xGne7fhfsfd; Mon, 12 Feb 2024 09:06:33 -0800 X-Received: from BL0PR05CU006.outbound.protection.outlook.com (BL0PR05CU006.outbound.protection.outlook.com [52.101.51.14]) by mx.groups.io with SMTP id smtpd.web11.11935.1707757592639015526 for ; Mon, 12 Feb 2024 09:06:32 -0800 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=D8iOSIx5dgp/m8agqR28t7n17/+67YqpRAzabSCYLfvM3f2rbZQLIAlyQHWaFK2r6MVpDa3aTuIsiZ1PL3zZpoNOaXkVpxtSyrFOLjOp75lATdt3WdIxZaJVNzp81z6Am9NEttup4RaYYeFDPsVTFkb/Q9dp5WIqK/L20yjgSDbPu8rYaa69+tRGLITRWS+L5BgFoH7JA72HVmqQXr6uH4EDxCdu/TFuEEJ9l5vzCXE01ifQrHwwSMVr6U37nai413gp84m3Xn79oEDTbyCvbVf9Znlpc+8kEppzgtFZco7C8MsrFaG8a8IwMej8rQ0kmDkzyoIrMs0OaXEugwu/2Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=qI5HJeRDJtH1V4BZDH2gWEiB0Eo7pvOZdUQiuRRc180=; b=M/pKvPY8Ip7Tl+jhf11+EcLrFkCdFEuFlPO19u5lOCm1Uy/V4R14OR8JRUm+wSUY/1ZoDSDDfAssHUte1e6qyTBg2By7XJT7HUH3RcymDADxYSBMLCouMHgHp2GkslD597RrtgUeDaNQTkWPzPbYIZZW5syQL713ARlRSGdS2mIhK4Sd5D0akPEm5ygjokW6s0Nb/6df4dPy9hCBAFwQHl2jYezR7a/+12ipGr12klzcWUGVreO/jKzvMmRBMiYKr/9nMottR7ZltrqL6RH0U04//JBTmR5vy+eCcsij27f15GcG8L+rRbMCy0nVhHcsHWjX3xYk7nXG7s37PQQj7Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=microsoft.com; dmarc=pass action=none header.from=microsoft.com; dkim=pass header.d=microsoft.com; arc=none X-Received: from BY1PR21MB3943.namprd21.prod.outlook.com (2603:10b6:a03:52c::19) by IA1PR21MB3767.namprd21.prod.outlook.com (2603:10b6:208:3e0::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.7292.12; Mon, 12 Feb 2024 17:06:28 +0000 X-Received: from BY1PR21MB3943.namprd21.prod.outlook.com ([fe80::4c42:b704:da2d:16cb]) by BY1PR21MB3943.namprd21.prod.outlook.com ([fe80::4c42:b704:da2d:16cb%3]) with mapi id 15.20.7316.006; Mon, 12 Feb 2024 17:06:28 +0000 From: "Joey Vagedes via groups.io" To: Rebecca Cran , "devel@edk2.groups.io" , Rebecca Cran , "Kinney, Michael D" , Sean , Michael Kubacki Subject: Re: [edk2-devel] Fixing edk2-basetools CI Thread-Topic: [EXTERNAL] Re: [edk2-devel] Fixing edk2-basetools CI Thread-Index: AQHaXdJ3bjlGHvswWEy4mmcLM7dG5rEG6Xsw Date: Mon, 12 Feb 2024 17:06:28 +0000 Message-ID: References: <2fe83108-e19b-4c95-8c45-08f43594e8cd@bsdio.com> <2c52fbac-dbe1-423a-a9d7-0bbe473cd9e9@os.amperecomputing.com> In-Reply-To: <2c52fbac-dbe1-423a-a9d7-0bbe473cd9e9@os.amperecomputing.com> Accept-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: msip_labels: MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_ActionId=cbd64940-ec1c-4bf7-b226-2ef3c89aaa4e;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_ContentBits=0;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Enabled=true;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Method=Standard;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Name=Internal;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_SetDate=2024-02-12T16:43:10Z;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_SiteId=72f988bf-86f1-41af-91ab-2d7cd011db47; x-ms-publictraffictype: Email x-ms-traffictypediagnostic: BY1PR21MB3943:EE_|IA1PR21MB3767:EE_ x-ms-office365-filtering-correlation-id: cfd3ea63-85f4-454e-634b-08dc2becf3f3 x-ld-processed: 72f988bf-86f1-41af-91ab-2d7cd011db47,ExtAddr x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam-message-info: NaJNN3hmQI4VfmIcKNYMgMZ6jak/ZTj4y6iW0YIO+6PKpRsU7lZCOX+Qtku29Q6WKYS6Dq4c5pMLakXFXlg4wFve45y6nquU9RuSugqf++PYLRFxbPVWDlKY3sNuZ9LBcFm3V/08Mz1KG9L6sntQG6TdRQnih1Zbm4K1nT5kPKMdYirG7lzzO9kV4b1jjz7zTjWTuUuN7ZfeolzKE7UcWO70j5dB2uGeiNeUgJWftFiUJMllLqDB9SZWREHC7S7BAAvoHjThLWy5e+5icEgwyyMUfOyF/bGmEZXYLdW0wFTXKQsTqM5OoYoUMKbejugcLxqwnzjxEKWdCVt5vn48m/SK+ReSLFMm4vWw/dqVrEaMtrvx0EztM4TYNIFSRKtrungn1whivTimoS1cGJuvMEmA4961WbtME6JSzGoESBF01gCjUrcMuvJorj+WDtIq2CH1RBHtcBpdMJDvtpG/Mt+HWnVMrIn7EKS7tWqPhxJzZwLHXml0g4tib7JGfVIdm6l5rfu9/XFRBeX9tWBO9jQZj5wyvusKj9HRHplicoZ0jviD35qNurrkd90iFfc3egp2ICO7LurIWQkHQQqTxL0xoURw2X4cSCGCvoefy8gI69lKqfnugR1p/exIeWSJ x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?H8KNDakBA9fyNSndiLKnzZjHGzSkOoYvSwkNB+axp+xfptEJe6pz8gH6X6rc?= =?us-ascii?Q?+KwgTS2+ab6I2mLghjt16iVVyT6iq8kO1jtyCPgJf7vsxNGpZenKeJH4bVD6?= =?us-ascii?Q?UthlIpf0XV/Gb2Juuai5ldNYDyrRH/CMxT63VIgnOkdCzQEDg2A2V8tmvdeq?= =?us-ascii?Q?ujgMjBSQJZcd5LH13skcmqyGDGCI+BFVwmuTh+SFa7oDasIF1NCq16V74XdB?= =?us-ascii?Q?kvqv3W5QaZxTGHA6bStj7LWamdw+22sGwPJqiunahLCGvr5tT+rMGnn1hLnU?= =?us-ascii?Q?NAYT0vGTfgUW2slYfZqeYyztVu6ruIyReVPs6YPYtT0/CMW7I2NB7W6xM85B?= =?us-ascii?Q?JXiokBc3CWR/idMW/3a+OX0yu1c0KDWdwYN6xkkwAS6ywdsDUKvifGosqrA9?= =?us-ascii?Q?vupCtxR6HxC1geIEmDlbF9rxGoT5vfbJaveKgE1E0Uwpnre0osC6ydnrPtJv?= =?us-ascii?Q?I4ssqoRaJ3b7k0tEpUxrysscu9Ef/79UMZkqF6zVv/q8guf9HWLMPvoxwLqj?= =?us-ascii?Q?bJAWKESh5kqDmAhtTsXUpeoe1PPTP5JXWmygmcduEUMA57rk9w3T0dbxsq9i?= =?us-ascii?Q?J/6oihR3BdMwlmyawpw1SayEshGCBHzmD9uneNhS38XZXzKPgT3FY5Y4exsU?= =?us-ascii?Q?JmIIC0AdPRXda86VjYdCH4t6mQNu0mfYCJYm+970eDlQ19QdC2Ss+l8/Ez9x?= =?us-ascii?Q?TEcQDZuoX2Ch7kxcp+9y8NHbBZW9eZVajcKeqzCjqweI0rEJKwqFkUaFzhY0?= =?us-ascii?Q?+ZaZ+ZXM/tUYpCY/pNC6tGGeQvd1L8877mWGHyjtVEEZpJ8XpcEdI+N6gvoa?= =?us-ascii?Q?YnLTICm+rg4jTT6Q0BygglQcowcHlm7PfkC1zKtmw3XFwVPWnkE8gmO1Tbw0?= =?us-ascii?Q?WrqHSnlqFYLDwNyjPJdkMGXgpq1OSeBu+Alem/Fqz6ZF5P2biq42QqK/0jXr?= =?us-ascii?Q?diuAIdz5eIwajFqkTsXOENYjLoo9qjprwu+b8f6XQgz0Lqzn585zCr586QAI?= =?us-ascii?Q?frnd6tBMMDdhxM3NQy2U2IqONHqmDtupbHP3mm/OEA7Nv8AqVqQEvlPUnveF?= =?us-ascii?Q?gA9tQnEnbKHAgcHxZjThYQKrdQyV+aM11u4umtAun2/LlAq10/90hv//xOlv?= =?us-ascii?Q?958BevPOcKPEOcznwdkA1a8ydw51pdxagnSYERcszZ8gwmc5bK1Q15bm5F7p?= =?us-ascii?Q?l9n0EsqqPufn1GeYDCHi1elLnFvO1ChHfZQ+d35ALvTTjO2P7lAXqIaTvMAu?= =?us-ascii?Q?2fJTp8R3Ogd1UABpfmd1t9m5zoGDgFpTmfZkEP+klCuP+HMt3axGIfg1YSPb?= =?us-ascii?Q?DgfzrqS7QgEHwuWGS1MoV1yhH8qr0LyxfCV3swbEkY9Mz3bL58J5SQuitIe+?= =?us-ascii?Q?78VAI9bUmzl+SrrA5GYyVjzbgOCYbctUwe/z09Tr99JXNlehmoKQkv12GLVk?= =?us-ascii?Q?UeokPxx5NLzVoGs1lu6eHI46utasxNHsLmtDQKMdPqL9Dnt96sGsW4u35Och?= =?us-ascii?Q?LMCd+KBQcvlHh2ZQcm2a40sunlCaBO2vqMan?= MIME-Version: 1.0 X-OriginatorOrg: microsoft.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: BY1PR21MB3943.namprd21.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: cfd3ea63-85f4-454e-634b-08dc2becf3f3 X-MS-Exchange-CrossTenant-originalarrivaltime: 12 Feb 2024 17:06:28.4653 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 72f988bf-86f1-41af-91ab-2d7cd011db47 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: 1ARnGFrpuv9x6QpPSuUoOI2NK37UvXriS1MJsJwdfT/BkI+0qToFGdQOFaRZe9t2GeUPFBp0amYHtwSE/1U5NGZu3bWEH18ln3uH82EY56o= X-MS-Exchange-Transport-CrossTenantHeadersStamped: IA1PR21MB3767 Precedence: Bulk List-Subscribe: List-Help: Sender: devel@edk2.groups.io List-Id: Mailing-List: list devel@edk2.groups.io; contact devel+owner@edk2.groups.io Reply-To: devel@edk2.groups.io,joeyvagedes@microsoft.com List-Unsubscribe-Post: List-Unsubscribe=One-Click List-Unsubscribe: X-Gm-Message-State: ZtAW0zUFj7g48UoZNwAgYpzKx7686176AA= Content-Language: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable X-GND-Status: LEGIT Authentication-Results: spool.mail.gandi.net; dkim=pass header.d=groups.io header.s=20140610 header.b=GHPoAx3X; dmarc=pass (policy=none) header.from=groups.io; spf=pass (spool.mail.gandi.net: domain of bounce@groups.io designates 66.175.222.108 as permitted sender) smtp.mailfrom=bounce@groups.io; arc=reject ("signature check failed: fail, {[1] = sig:microsoft.com:reject}") Hello! Regarding flake8, I see. That will take some time to work through - = I completely understand.=20 When ready to work through that, I suggest switching to ruff. It can resolv= e some errors automatically for you and allows you to customize it more tha= n flake8 (it is also faster).=20 I just ran ruff on the project, showing about 11k errors. One nice thing is= that a large majority of it is regarding importing using *. By fixing thos= e, you can reduce the errors by almost 70% (to 3400 errors), so it is not *= quite* as bad as it seems (though 3400 is still A LOT). ---------------------------------------------------------------------------= ------------------------------------------------ Regarding the setup tools question, the build-backend part of pytoml (L3) i= s the entry point that is invoked - so we are still invoking setuptools. Ho= wever, it pulls the necessary metadata from pyproject.toml rather than setu= p.py's setup() function. The main thing that is different is the "get_versi= on" functionality, which we replicate in the pyproject.toml with setuptools= -scm[toml] which simply uses the latest tag to set the version. ---------------------------------------------------------------------------= ------------------------------------------------ Additionally, I wanted to comment on the wrappers that you mentioned. Using the [projects.scripts] section (i.e. CLI Commands feature) is in my o= pinion the correct way to move forward as users will be able to invoke comm= ands themselves easily (i.e. just call Bin2Pcd, and it will work). However,= the wrappers will continue to be necessary until all python source is remo= ved from Basetools, so that users can toggle between pip or local. However,= once source is removed form BaseTools, removing the wrappers is the next s= tep forward. In the meantime, you will be able to update the binpipwrappers to something= akin to the following (untested, so I don't garuntee it will be exactly th= is: @setlocal Ecc %* Rather than @setlocal @set ToolName=3D%~n0% @%PYTHON_COMMAND% -m edk2basetools.%ToolName%.EccMain %* ---------------------------------------------------------------------------= ------------------------------------------------ One thing to keep in mind once Python is completely removed from BaseTools,= is that consumers will need to get in the habit of verifying the correct v= ersion of basetools is installed for the commit they are on (or automate th= is checking) particularly for anyone that regularly switches between stable= branches. Thanks,=20 Joey -----Original Message----- From: Rebecca Cran =20 Sent: Monday, February 12, 2024 8:42 AM To: devel@edk2.groups.io; Joey Vagedes ; Rebecca= Cran ; Kinney, Michael D ; = Sean ; Michael Kubacki Subject: [EXTERNAL] Re: [edk2-devel] Fixing edk2-basetools CI [You don't often get email from rebecca@os.amperecomputing.com. Learn why t= his is important at https://aka.ms/LearnAboutSenderIdentification ] On 2/12/2024 9:08 AM, Joey Vagedes via groups.io wrote: > Hello. It can be simplified - You can view the pyproject.toml for edk2-py= tool-library and edk2-pytool-extensions: > > https://nam06.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2Fgith > ub.com%2Ftianocore%2Fedk2-pytool-library%2Fblob%2Fmaster%2Fpyproject.t > oml&data=3D05%7C02%7Cjoeyvagedes%40microsoft.com%7Ca37eecc569d54b5dbbf30 > 8dc2be996f3%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6384335294831 > 52477%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJB > TiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=3D8RSQNcjgH02JyPiGCKmyf2u7 > Ep6BC9ldwiND0ULj9Zs%3D&reserved=3D0 > > Also, Why disable flake8 rather than fix the changes? I also suggest swit= ching to ruff, as it does the same thing but better, and with way more rule= s / configuration options. Thanks. I'm not sure that looks so much simpler than my pyproject.toml but = I'll update it to follow the same layout. Also, pyproject.toml says it's us= ing the setuptools build backend but you don't have a setup.py: how does that work? I decided to disable flake8 for now because the BaseTools Python code is ve= ry non-compliant: the log file has over 41,000 lines. In most places it loo= ks like people have been trying to write it as C code. A few of the lines: .\edk2basetools\Common\TargetTxtClassObject.py:44:80: E203 whitespace befor= e ':' .\edk2basetools\Common\TargetTxtClassObject.py:61:14: E111 indentation is n= ot a multiple of 4 .\edk2basetools\Common\TargetTxtClassObject.py:61:14: E117 over-indented .\edk2basetools\Common\TargetTxtClassObject.py:63:50: F405 'FILE_NOT_FOUND'= may be undefined, or defined from star imports: .BuildToolError .\edk2basetools\Common\TargetTxtClassObject.py:73:121: E501 line too long (= 123 > 120 characters) .\edk2basetools\Common\TargetTxtClassObject.py:84:38: F405 'FILE_OPEN_FAILU= RE' may be undefined, or defined from star imports: .BuildToolError .\edk2basetools\Common\TargetTxtClassObject.py:100:108: E502 the backslash = is redundant between brackets .\edk2basetools\Common\TargetTxtClassObject.py:108:121: E501 line too long = (139 > 120 characters) .\edk2basetools\Common\TargetTxtClassObject.py:118:121: E501 line too long = (140 > 120 characters) .\edk2basetools\Common\TargetTxtClassObject.py:123:97: E502 the backslash i= s redundant between brackets .\edk2basetools\Common\TargetTxtClassObject.py:128:21: F841 local variable = 'V' is assigned to but never used -- Rebecca Cran -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D- Groups.io Links: You receive all messages sent to this group. View/Reply Online (#115361): https://edk2.groups.io/g/devel/message/115361 Mute This Topic: https://groups.io/mt/104306226/7686176 Group Owner: devel+owner@edk2.groups.io Unsubscribe: https://edk2.groups.io/g/devel/unsub [rebecca@openfw.io] -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-