From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail02.groups.io (mail02.groups.io [66.175.222.108]) by spool.mail.gandi.net (Postfix) with ESMTPS id 919B2D801DB for ; Fri, 16 Feb 2024 07:51:40 +0000 (UTC) DKIM-Signature: a=rsa-sha256; bh=2ChANfpHVEREzuRE593dNxbRO+GDuSJO2Nyls3MQ/14=; c=relaxed/simple; d=groups.io; h=Feedback-ID:Message-ID:Date:MIME-Version:User-Agent:Subject:To:References:From:In-Reply-To:Precedence:List-Subscribe:List-Help:Sender:List-Id:Mailing-List:Delivered-To:Reply-To:List-Unsubscribe-Post:List-Unsubscribe:Content-Language:Content-Type:Content-Transfer-Encoding; s=20140610; t=1708069899; v=1; b=bXiXbwzP6f3MwP1D0/+ULfGHdjAu62da8D9U8z+egFJNTmhUI1FJHpzPSRh9oxevQ8oahTDz cEKw3g6RGVPIP/pWMRl/RDT0SkE4Pj3Op28ijJj2JLLwlOp0jVOyDB/TlXtP8mtnXT6TH3YodAt 9P0ytqWavgjeifgFOiMYF6lU= X-Received: by 127.0.0.2 with SMTP id MDs1YY7687511x2j2cLIzU0d; Thu, 15 Feb 2024 23:51:39 -0800 X-Received: from wfhigh2-smtp.messagingengine.com (wfhigh2-smtp.messagingengine.com [64.147.123.153]) by mx.groups.io with SMTP id smtpd.web11.14212.1708069898188237855 for ; Thu, 15 Feb 2024 23:51:38 -0800 X-Received: from compute6.internal (compute6.nyi.internal [10.202.2.47]) by mailfhigh.west.internal (Postfix) with ESMTP id 93C0F1800070; Fri, 16 Feb 2024 02:51:36 -0500 (EST) X-Received: from mailfrontend1 ([10.202.2.162]) by compute6.internal (MEProxy); Fri, 16 Feb 2024 02:51:37 -0500 X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrvddugdduudehucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepkfffgggfuffvfhfhjggtgfesthejredttddvjeenucfhrhhomheptfgvsggv tggtrgcuvehrrghnuceorhgvsggvtggtrgessghsughiohdrtghomheqnecuggftrfgrth htvghrnhepffelfeevgeejgfelheeuhfehgeejtefhudduudeuleffieeuhedvfeeihfdv vddvnecuffhomhgrihhnpehgihhthhhusgdrtghomhdprghkrgdrmhhsnecuvehluhhsth gvrhfuihiivgeptdenucfrrghrrghmpehmrghilhhfrhhomheprhgvsggvtggtrgessghs ughiohdrtghomh X-ME-Proxy: Feedback-ID: i5b994698:Fastmail X-Received: by mail.messagingengine.com (Postfix) with ESMTPA; Fri, 16 Feb 2024 02:51:35 -0500 (EST) Message-ID: <307ff7a3-d851-4bdd-83c5-9d276920b798@bsdio.com> Date: Fri, 16 Feb 2024 00:51:28 -0700 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [edk2-devel] Fixing edk2-basetools CI To: Joey Vagedes , Rebecca Cran , "devel@edk2.groups.io" , "Kinney, Michael D" , Sean , Michael Kubacki References: <2fe83108-e19b-4c95-8c45-08f43594e8cd@bsdio.com> <2c52fbac-dbe1-423a-a9d7-0bbe473cd9e9@os.amperecomputing.com> From: "Rebecca Cran" In-Reply-To: Precedence: Bulk List-Subscribe: List-Help: Sender: devel@edk2.groups.io List-Id: Mailing-List: list devel@edk2.groups.io; contact devel+owner@edk2.groups.io Reply-To: devel@edk2.groups.io,rebecca@bsdio.com List-Unsubscribe-Post: List-Unsubscribe=One-Click List-Unsubscribe: X-Gm-Message-State: fxi95aZoe5yyQpx71iDjHSYTx7686176AA= Content-Language: en-US Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: quoted-printable X-GND-Status: LEGIT Authentication-Results: spool.mail.gandi.net; dkim=pass header.d=groups.io header.s=20140610 header.b=bXiXbwzP; spf=pass (spool.mail.gandi.net: domain of bounce@groups.io designates 66.175.222.108 as permitted sender) smtp.mailfrom=bounce@groups.io; dmarc=none I've updated the PR=20 https://github.com/tianocore/edk2-basetools/pull/116/ to include commits=20 from the pyflake8 disable PR and a couple more commits that cause CI to=20 start passing. I realize that disabling both flake8 and codecov isn't=20 good, but my thinking is to get builds working again then we can work on=20 fixing the issues in the background. The edk2-basetools project needs=20 lots of work. Add pyproject.toml and fix setup.py deprecation warnings Disable running Pyflakes in CI since it's blocking releases Fix comments in build-publish-whl-steps.yml Update basic-setup-steps.yml to require Python 3.10 Disable codecov to fix CI and due to very low coverage --=20 Rebecca Cran On 2/13/24 12:42, Joey Vagedes wrote: > I agree - there are multiple blocking issues that will require some fixes= - One additional thing I noted is that multiple areas still perform relati= ve path imports - i.e. the import starts with `from .`. > > These will all need to be updated to import from edk2basetools as some po= int soon as importing via relative paths can result in it importing the wro= ng file if the pypath environment variable is influenced. This open issue a= ctually stems from that: https://github.com/tianocore/edk2-basetools/issues= /110 as the end result is that edk2-basetools pip module can unexpectedly e= nd up using the BaseTools/* source code rather than the pip module source c= ode. > > Thanks, > Joey > > -----Original Message----- > From: Rebecca Cran > Sent: Tuesday, February 13, 2024 11:28 AM > To: Joey Vagedes ; Rebecca Cran ; devel@edk2.groups.io; Kinney, Michael D ; Sean ; Michael Kubacki > Subject: Re: [EXTERNAL] Re: [edk2-devel] Fixing edk2-basetools CI > > [You don't often get email from rebecca@bsdio.com. Learn why this is impo= rtant at https://aka.ms/LearnAboutSenderIdentification ] > > Thanks, I've updated > https://github.com/tianocore/edk2-basetools/pull/116 with the changes you= suggested. > > Unfortunately several of the tools don't work with the projects.scripts s= ection because they don't have Main etc. It turns out even Ecc is currently= broken in the edk2-basetools PyPI package because PYTHONPATH doesn't conta= in the parent directory. I'll plan to fix the issues in the near future, on= ce I've worked through the other PRs. > > > -- > > Rebecca Cran > > > On 2/12/24 10:06, Joey Vagedes wrote: >> Hello! Regarding flake8, I see. That will take some time to work through= - I completely understand. >> >> When ready to work through that, I suggest switching to ruff. It can res= olve some errors automatically for you and allows you to customize it more = than flake8 (it is also faster). >> >> I just ran ruff on the project, showing about 11k errors. One nice thing= is that a large majority of it is regarding importing using *. By fixing t= hose, you can reduce the errors by almost 70% (to 3400 errors), so it is no= t *quite* as bad as it seems (though 3400 is still A LOT). >> >> ---------------------------------------------------------------------- >> ----------------------------------------------------- >> >> Regarding the setup tools question, the build-backend part of pytoml (L3= ) is the entry point that is invoked - so we are still invoking setuptools.= However, it pulls the necessary metadata from pyproject.toml rather than s= etup.py's setup() function. The main thing that is different is the "get_ve= rsion" functionality, which we replicate in the pyproject.toml with setupto= ols-scm[toml] which simply uses the latest tag to set the version. >> >> ---------------------------------------------------------------------- >> ----------------------------------------------------- >> >> Additionally, I wanted to comment on the wrappers that you mentioned. >> >> Using the [projects.scripts] section (i.e. CLI Commands feature) is in m= y opinion the correct way to move forward as users will be able to invoke c= ommands themselves easily (i.e. just call Bin2Pcd, and it will work). Howev= er, the wrappers will continue to be necessary until all python source is r= emoved from Basetools, so that users can toggle between pip or local. Howev= er, once source is removed form BaseTools, removing the wrappers is the nex= t step forward. >> >> In the meantime, you will be able to update the binpipwrappers to someth= ing akin to the following (untested, so I don't garuntee it will be exactly= this: >> >> @setlocal >> Ecc %* >> >> Rather than >> >> @setlocal >> @set ToolName=3D%~n0% >> @%PYTHON_COMMAND% -m edk2basetools.%ToolName%.EccMain %* >> >> ---------------------------------------------------------------------- >> ----------------------------------------------------- >> >> One thing to keep in mind once Python is completely removed from BaseToo= ls, is that consumers will need to get in the habit of verifying the correc= t version of basetools is installed for the commit they are on (or automate= this checking) particularly for anyone that regularly switches between sta= ble branches. >> >> Thanks, >> Joey >> >> -----Original Message----- >> From: Rebecca Cran >> Sent: Monday, February 12, 2024 8:42 AM >> To: devel@edk2.groups.io; Joey Vagedes ; >> Rebecca Cran ; Kinney, Michael D >> ; Sean ; Michael >> Kubacki >> Subject: [EXTERNAL] Re: [edk2-devel] Fixing edk2-basetools CI >> >> [You don't often get email from rebecca@os.amperecomputing.com. Learn >> why this is important at https://aka.ms/LearnAboutSenderIdentification >> ] >> >> On 2/12/2024 9:08 AM, Joey Vagedes via groups.io wrote: >>> Hello. It can be simplified - You can view the pyproject.toml for edk2-= pytool-library and edk2-pytool-extensions: >>> >>> https://git/ >>> h%2F&data=3D05%7C02%7Cjoeyvagedes%40microsoft.com%7Cf64b5d2ea86245c44e5 >>> 208dc2cc9e93d%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C6384344929 >>> 36969249%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIi >>> LCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=3DORN2iHQhOfzkN9F462p >>> 2H6mb8Hl9ubIIeX%2BDBorzmC0%3D&reserved=3D0 >>> ub.com%2Ftianocore%2Fedk2-pytool-library%2Fblob%2Fmaster%2Fpyproject. >>> t >>> oml&data=3D05%7C02%7Cjoeyvagedes%40microsoft.com%7Ca37eecc569d54b5dbbf3 >>> 0 >>> 8dc2be996f3%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638433529483 >>> 1 >>> 52477%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJ >>> B >>> TiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=3D8RSQNcjgH02JyPiGCKmyf2u >>> 7 >>> Ep6BC9ldwiND0ULj9Zs%3D&reserved=3D0 >>> >>> Also, Why disable flake8 rather than fix the changes? I also suggest sw= itching to ruff, as it does the same thing but better, and with way more ru= les / configuration options. >> Thanks. I'm not sure that looks so much simpler than my pyproject.toml b= ut I'll update it to follow the same layout. Also, pyproject.toml says it's= using the setuptools build backend but you don't have a setup.py: >> how does that work? >> >> I decided to disable flake8 for now because the BaseTools Python code is= very non-compliant: the log file has over 41,000 lines. In most places it = looks like people have been trying to write it as C code. >> >> A few of the lines: >> >> .\edk2basetools\Common\TargetTxtClassObject.py:44:80: E203 whitespace be= fore ':' >> .\edk2basetools\Common\TargetTxtClassObject.py:61:14: E111 indentation >> is not a multiple of 4 >> .\edk2basetools\Common\TargetTxtClassObject.py:61:14: E117 >> over-indented >> .\edk2basetools\Common\TargetTxtClassObject.py:63:50: F405 'FILE_NOT_FOU= ND' may be undefined, or defined from star imports: >> .BuildToolError >> .\edk2basetools\Common\TargetTxtClassObject.py:73:121: E501 line too >> long (123 > 120 characters) >> .\edk2basetools\Common\TargetTxtClassObject.py:84:38: F405 'FILE_OPEN_FA= ILURE' may be undefined, or defined from star imports: >> .BuildToolError >> .\edk2basetools\Common\TargetTxtClassObject.py:100:108: E502 the >> backslash is redundant between brackets >> .\edk2basetools\Common\TargetTxtClassObject.py:108:121: E501 line too >> long (139 > 120 characters) >> .\edk2basetools\Common\TargetTxtClassObject.py:118:121: E501 line too >> long (140 > 120 characters) >> .\edk2basetools\Common\TargetTxtClassObject.py:123:97: E502 the >> backslash is redundant between brackets >> .\edk2basetools\Common\TargetTxtClassObject.py:128:21: F841 local >> variable 'V' is assigned to but never used >> >> >> -- >> Rebecca Cran -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D- Groups.io Links: You receive all messages sent to this group. View/Reply Online (#115538): https://edk2.groups.io/g/devel/message/115538 Mute This Topic: https://groups.io/mt/104306226/7686176 Group Owner: devel+owner@edk2.groups.io Unsubscribe: https://edk2.groups.io/g/devel/unsub [rebecca@openfw.io] -=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-=3D-