[NEW FEATURE] USE_PYTHON=build to support pyproject.toml based projects (PEP-517 is supported at the level of individual ports) (take 2)

[vishwin]

Summary

The patch from the original review https://reviews.freebsd.org/D36061 was committed after 14 days of no response from python@ and then reverted.
As it turned out Charlie Li (vishwin@) has a proposal for a similar feature: https://wiki.freebsd.org/Python/PEP-517 that I wasn't aware of at the time I created the patch. The commit was reverted due to the collision with Charlie Li's proposal.

Upon a careful review the proposal https://wiki.freebsd.org/Python/PEP-517 is for a combination of local PEP-517 compliance and global PEP-517 compliance. Local PEP-517 compliance means that individual ports are built with a PEP-517 compliant process. Global PEP-517 compliance means that the whole ports tree is converted to PEP-517 and distutils/setuptools aren't used any longer to build Python ports.

It appears that it is best to handle local and global PEP-517 compliance separately due to resource limitations, their different nature, and them being only loosely related. In the first phase local PEP-517 compliance can be implemented, so that individual ports would build according to PEP-517. This would open the way to a lot of new ports to enter the ports tree. Global PEP-517 compliance can be implemented during the second phase. The second phase would require a lot of Python ports to be adjusted that would likely take a lot more time.

The patch that I am suggesting implements local PEP-517 compliance (the first phase).

One difference from the https://reviews.freebsd.org/D36061 proposal is that USE_PYTHON=build is used instead of USE_PYTHON=pep517. "build" is user's intent and is also a very easy to remember word, compared to "pep517" that isn't as easy to remember.

The second difference is that USE_PYTHON=build doesn't support the autoplist feature. The involved tools (build/installer) don't support autoplist and it would be unnatural and not in line with the rest of the ports tree to have autoplist just for Python ports.

In short: the patch herein is essentially in no contradiction with https://wiki.freebsd.org/Python/PEP-517. It provides 2 above mentioned improvements, and also splits the whole project into 2 largely unconnected parts therefore simplifying the process.

I am asking for suggestions on how to improve this patch and seeking approval from python@ and others.

[vishwin]

Upon a careful review the proposal https://wiki.freebsd.org/Python/PEP-517 is for a combination of local PEP-517 compliance and global PEP-517 compliance. Local PEP-517 compliance means that individual ports are built with a PEP-517 compliant process. Global PEP-517 compliance means that the whole ports tree is converted to PEP-517 and distutils/setuptools aren't used any longer to build Python ports.

Not every Python package even supports PEP-517 yet, so global compliance is moot and out of scope. All the wiki page does is provide a blueprint for how our ports framework supports PEP-517. The only ports to be converted at the same time are those required to bootstrap the necessary tooling. Other individual packages will roll at their own pace, independent of the framework.

One difference from the https://reviews.freebsd.org/D36061 proposal is that USE_PYTHON=build is used instead of USE_PYTHON=pep517. "build" is user's intent and is also a very easy to remember word, compared to "pep517" that isn't as easy to remember.

Again, USE_PYTHON=build is misleading. USE_PYTHON=distutils was and continues to be named as such despite actually using the setuptools package because distutils was a de facto but "largely undocumented and unmaintained collection of utilities (that) defines a configuration format that describes a Python distribution and provides the tools to convert a directory of source code into a source distribution, and some forms of binary distribution." It was also not called something like USE_PYTHON=setup, since the file can be named other than setup.py. For PEP-517, this is an actual standard that does not dictate any specific combination of tooling, but rather "toward an ecosystem of tools and tool-agnostic interoperability standards." Until relatively (as of this typing) recently, the installer and build packages did not exist (pip was first to implement PEP-517 support, but that's pip), and when they did come into existence, were not in a usable enough state for operating system-level package management to use. Besides, future-proofing the ports framework also has to consider the possibilities that installer and build may fall out of vogue or even the ports framework itself can handle virtual environment-type deployments. Additionally, the rest of the Python community have adopted the PEP-517 phrasing.

The second difference is that USE_PYTHON=build doesn't support the autoplist feature. The involved tools (build/installer) don't support autoplist and it would be unnatural and not in line with the rest of the ports tree to have autoplist just for Python ports.

autoplist is fully supported by way of the RECORD file. Just need to strip the additional CSV columns and clean up the prefixes. Those ports that do not spit out a RECORD file (it is optional) or generate an inaccurate one obviously should not use autoplist.

[tcberner]

Python any opinions?

[vishwin]

Follow wiki specification more closely.

Overridable variables for the build and install drivers have been added. This is primarily for the bootstrap packages/ports but port maintainers may also use these to not have to use devel/py-build and devel/py-installer respectively and still have PEP-517 compliant packages.

There is a minimal autoplist implementation based on wheels' RECORD files. More testing is needed for those packages with files installed to various locations in hier(7), particularly when USE_PYTHON=concurrent.

[vishwin]

remove erroneous BUILD_DEPENDS when USE_PYTHON=pep517 is not defined

[fluffy]

BTW, can we follow OpenBSD and hook flit too as pep517 option?

[vishwin]

In D36290#851418, @fluffy wrote:

BTW, can we follow OpenBSD and hook flit too as pep517 option?

Not a good idea, since the whole point of this standard is not to recommend any particular build backend over any others, including those yet to exist, in the ecosystem. Build backends are to be specified as any other BUILD_DEPENDS.

[vishwin]

Further testing still needed on parsing RECORD for autoplist when it comes to packages installing/listing files outside site-packages.

[vishwin]

• do not check build dependencies, do not isolate build, force wheel
• support entry_points in USES=pep517 autoplist
• clarify purposes of PEP517_* variables

Test suite outlined in wiki now fully passes; basic docs and notices to follow.

[vishwin]

• CHANGES: add USE_PYTHON=pep517
• copyedit unhyphen {{front,back}-}end
• re-add build dependency check (to follow wiki)

[tcberner]

I'm very happy to see some action here :)

@yuri do these changes meet the requirements you have too?

[tcberner]

Mk/Uses/python.mk

857

maybe use REINPLACE_CMD?

[vishwin]

Mk/Uses/python.mk

857

${REINPLACE_CMD} is not used anywhere in this file.

[vishwin]

Mk/Uses/python.mk

857

Further, this is not even an in-place operation: RECORD is left untouched while the output goes elsewhere.