Mk/Uses/python.mk: Introduce PEP517_BUILD_DISTDIR for PEP 517 wheel output

[michaelo]

Summary

Mk/Uses/python.mk: Introduce PEP517_BUILD_DISTDIR for PEP 517 wheel output

Packages are built in a well-known, explicit location: ${WRKDIR}/pkg. Wheels,
however, are left wherever the PEP 517 backend happens to put them under
${WRKSRC} -- and that location isn't even consistent across backends (e.g.
"dist/" for build/setuptools, "target/wheels/" for maturin) -- which makes
them hard for external tooling such as poudriere to locate reliably.

Introduce PEP517_BUILD_DISTDIR, defaulting to ${WRKDIR}/whl, as the single
place this location is defined, referenced by both PEP517_BUILD_CMD and
PEP517_INSTALL_CMD instead of each hardcoding the path independently. Wheels
now follow the same explicit convention as packages. Ports that invoke a
build tool directly rather than through PEP517_BUILD_CMD (e.g. maturin)
should likewise pass ${PEP517_BUILD_DISTDIR} as their own output directory,
rather than hardcoding ${WRKDIR}/whl, so the build and install sides cannot
drift apart if this location ever changes again.

PR: 290653

[arrowd]

I don't quite get the rationale. Can you provide an example how is it going to be used?

[michaelo]

In D53433#1223067, @arrowd wrote:

I don't quite get the rationale. Can you provide an example how is it going to be used?

Yes, sure. Let's assume this lands in main and I enable in make.conf to install the wheel files to /var/cache/python-wheels. Then in pouodriere I can collect them after the build and serve a private index: https://github.com/michael-o/poudriere-python-wheels/pull/1

There are no FreeBSD wheels on PyPI and we have hefty dependencies for numpy, scipy, etc. They take ages to compile even with 24 threads. With poudriere, I have a working build system which already builds the wheels, I will simply retain them and re-use as an index. I cannot install any of the wheels as packages because uv intentionally ignores system-installed wheels:

$ git diff -U0 pyproject.toml
diff --git a/pyproject.toml b/pyproject.toml
index 673c4c1..8a95236 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -43,0 +44,3 @@ dev = [
+[[tool.uv.index]]
+name = "FREEBSD_WHEELS"
+url  = "https://dw-eng-rsc.innomotics.net/FreeBSD/python-wheels/FreeBSD:13:amd64/kona-latest/simple/"

At the end, it is about reuseability, avoid repetitive builds across CI/CD and final deployment. As soon as the index has all deps the application needs only 15 s to asseble, without 15 min.

[michaelo]

My ideal longterm goal: The Project hosts canonical poudriere builds and these could produce FreeBSD-specific wheels which could be uploaded to PyPI and would dramatically improve the situation for the Python ecosystem on FreeBSD.

[arrowd]

Then this change alone isn't really enough and I'm not even sure it is a right step forward.

The overall idea makes sense to me, though. At the moment the only artifact type the FreeBSD Ports tree can produce is a FreeBSD package. Python wheels or AppImage binaries are another examples. I'm as well planning to implement the later at some point.

But the full implementation requires work not only in Uses/python.mk, but also in Mk/bsd.port.mk and on the Poudriere side of things. I'm ready to review a complete implementation.

[michaelo]

In D53433#1223077, @arrowd wrote:

Then this change alone isn't really enough and I'm not even sure it is a right step forward.

Of course it is not enough, but that applies to the entire ports system. Without poudriere and a distribution server you can't serve packages/wheels. It is a component.

The overall idea makes sense to me, though. At the moment the only artifact type the FreeBSD Ports tree can produce is a FreeBSD package. Python wheels or AppImage binaries are another examples. I'm as well planning to implement the later at some point.

Longterm, it should be able to do.

But the full implementation requires work not only in Uses/python.mk, but also in Mk/bsd.port.mk and on the Poudriere side of things. I'm ready to review a complete implementation.

What is a full implementation for you? My poudriere change is in the works and will be a plugin in the hooks system which works protoypically. It will be open source of course. What you have in mind is a 100% solution which is not feasable right, but this is a good step to evaluate anything else. I can mark this as EXPERIMENTAL and mention this as EXPERIMENTAL in UPDATING. I'd like to get feedback from people as well how we can evolve this.

Again, this works already: Poudriere builds wheels, collects them via nullfs mount, dir2pi generates a static index and Apache HTTPd hosts this index. Exactly as for packages.

[arrowd]

Link your Poudriere changes in there, so it is possible to get an idea how this is going to work overall.

[michaelo]

In D53433#1223122, @arrowd wrote:

Link your Poudriere changes in there, so it is possible to get an idea how this is going to work overall.

I did, work in progress for poudriere: https://github.com/michael-o/poudriere-python-wheels/pull/1. This will turned into a project on GitHub soon. uv AND are already consuming it properly in a test setup.
Apache as usual:

$ cat freebsd-python-wheels.conf
Alias /FreeBSD/python-wheels/FreeBSD:13:amd64/cafe-custom-uis-latest "/var/poudriere/data/python-wheels/135-release-amd64-default-head-cafe-custom-uis"
Alias /FreeBSD/python-wheels/FreeBSD:13:amd64/kona-latest "/var/poudriere/data/python-wheels/135-release-amd64-default-head-kona"
Alias /FreeBSD/python-wheels/FreeBSD:13:amd64/cafe-custom-uis-quarterly "/var/poudriere/data/python-wheels/135-release-amd64-default-cafe-custom-uis"
Alias /FreeBSD/python-wheels/FreeBSD:13:amd64/kona-quarterly "/var/poudriere/data/python-wheels/135-release-amd64-default-kona"

<Directory "/var/poudriere/data/python-wheels">
    Options Indexes FollowSymLinks
    IndexOptions FancyIndexing FoldersFirst SuppressDescription SuppressColumnSorting NameWidth=50 VersionSort
    AllowOverride None
    Require all granted
</Directory>

[arrowd]

It should be turned into a proper pull request to the Poudriere upstream repo. One thing that already catched my eye is that wheel generation happens from the host, not within the jail. Bryan might disagree with this.

[michaelo]

In D53433#1223124, @arrowd wrote:

It should be turned into a proper pull request to the Poudriere upstream repo.

As a plugin, yes? But as long as It is not stable I don't want to have it upstream and constantly break people's stuff unless we mark it as experimental. My goal was to decouple from poduriere because people might just use portmaster.

One thing that already catched my eye is that wheel generation happens from the host, not within the jail. Bryan might disagree with this.

No*, the wheels are generated in the jail during build. What I do is to make the multiplatform because there is current no equivalent of manylinux for FreeBSD. Unfortunately, sysconfig.get_platform() is used which will contain the patch level as well. This leads to issues that a wheel isn't found if it does not matches. Unless there is a PEP for "manyfreebsd" to have a base line on major.minor one needs to rewrite the platform tags.

\* You are right, I could move that logic into the ports system, not do it in poudriere. I have never thought of that actually. That is likely even better than doing in poudriere. Shall I give it a try?

[michaelo]

@arrowd There is one issue with retagging the wheel inside of the jail. Let's say the jail has been updated from 13.5-RELEASE-p5 to 13.5-RELEASE-p7. There is a wheel for package foo. The multiplatform wheel goes only to 13.5-RELEASE-p5. Unless the port is upgraded to a newer version the multiplatform tags will never be updated and for jails running 13.5-RELEASE-p6+ unavailable. This means that I still need to perform the retagging for older wheels, but not for newer ones with a poudriere hook. So at the end it will be duplicate work. I don't know wether this is really smart to do...

[arrowd]

In D53433#1223130, @michaelo wrote:

As a plugin, yes?

If it can be completely done via a hook - yes, but not necessary. It might be as well a core feature.

But as long as It is not stable I don't want to have it upstream and constantly break people's stuff unless we mark it as experimental.

This is not a problem - we have poudriere-devel exactly for that.

My goal was to decouple from poduriere because people might just use portmaster.
You are right, I could move that logic into the ports system, not do it in poudriere. I have never thought of that actually. That is likely even better than doing in poudriere. Shall I give it a try?

I think yes. This is what I have in mind for AppImage:

• At the Ports level we introduce the make appimage target in addition to make package. This target would create an AppImage binary in ${WRKDIR} out of ${WRKSRC}. This can be used by plain make users.
• We ifdef the AppImage stuff under the APPIMAGE_BUILDING. This disables AppImage stuff by default, but allows portmaster/synth users to invoke AppImage machinery by adding APPIMAGE_BUILDING=yes into /etc/make.conf.
• At the Poudriere level we introduce poudriere appimage-bulk subcommand that works like usual bulk, but defines APPIMAGE_BUILDING and collects AppImage artifacts instead of .pkg files.

Note that my case is more complex, because a port should be built in a different way to make it possible to be converted into an AppImage. From what I gather, there are no special measures required to create .whl from a python ${WRKDIR}, so it'd be even easier for you. We just build a Python port the usual way and then invoke make wheel instead of make package.

So, I'd start with introducing the make wheel target, that is only enabled if WHEEL_BUILDING is defined. Then we can play with it on a plain make level and then we'll figure out how to properly integrate it into Poudriere.

There is one issue with retagging the wheel inside of the jail. Let's say the jail has been updated from 13.5-RELEASE-p5 to 13.5-RELEASE-p7. There is a wheel for package foo. The multiplatform wheel goes only to 13.5-RELEASE-p5. Unless the port is upgraded to a newer version the multiplatform tags will never be updated and for jails running 13.5-RELEASE-p6+ unavailable. This means that I still need to perform the retagging for older wheels, but not for newer ones with a poudriere hook.

I don't know much about Python ways of doing thing, so I'm a bit confused. Are you saying that a wheel encodes the OS version it was built on? This does not sound like a problem to me - even a boring C port may do this. Why do you want to rebuild a such a port after an OS patch update?

[michaelo]

In D53433#1223564, @arrowd wrote:

In D53433#1223130, @michaelo wrote:

As a plugin, yes?

If it can be completely done via a hook - yes, but not necessary. It might be as well a core feature.

But as long as It is not stable I don't want to have it upstream and constantly break people's stuff unless we mark it as experimental.

This is not a problem - we have poudriere-devel exactly for that.

It is a separate project how: https://github.com/michael-o/poudriere-python-wheels/pull/1

My goal was to decouple from poduriere because people might just use portmaster.
You are right, I could move that logic into the ports system, not do it in poudriere. I have never thought of that actually. That is likely even better than doing in poudriere. Shall I give it a try?

I think yes. This is what I have in mind for AppImage:

• At the Ports level we introduce the make appimage target in addition to make package. This target would create an AppImage binary in ${WRKDIR} out of ${WRKSRC}. This can be used by plain make users.
• We ifdef the AppImage stuff under the APPIMAGE_BUILDING. This disables AppImage stuff by default, but allows portmaster/synth users to invoke AppImage machinery by adding APPIMAGE_BUILDING=yes into /etc/make.conf.
• At the Poudriere level we introduce poudriere appimage-bulk subcommand that works like usual bulk, but defines APPIMAGE_BUILDING and collects AppImage artifacts instead of .pkg files.

Note that my case is more complex, because a port should be built in a different way to make it possible to be converted into an AppImage. From what I gather, there are no special measures required to create .whl from a python ${WRKDIR}, so it'd be even easier for you. We just build a Python port the usual way and then invoke make wheel instead of make package. See PEP517_INSTALL_WHEEL_CMD

So, I'd start with introducing the make wheel target, that is only enabled if WHEEL_BUILDING is defined. Then we can play with it on a plain make level and then we'll figure out how to properly integrate it into Poudriere.

There is a subtile difference where. The wheels are already there with PEP 517, distutils just requires one more step, but I expect distutils to disappear in the future. So basically either "make wheel" would be a noop or it would move/copy the *.whl to ${WRKDIR}/whl/

There is one issue with retagging the wheel inside of the jail. Let's say the jail has been updated from 13.5-RELEASE-p5 to 13.5-RELEASE-p7. There is a wheel for package foo. The multiplatform wheel goes only to 13.5-RELEASE-p5. Unless the port is upgraded to a newer version the multiplatform tags will never be updated and for jails running 13.5-RELEASE-p6+ unavailable. This means that I still need to perform the retagging for older wheels, but not for newer ones with a poudriere hook.

I don't know much about Python ways of doing thing, so I'm a bit confused. Are you saying that a wheel encodes the OS version it was built on? This does not sound like a problem to me - even a boring C port may do this. Why do you want to rebuild a such a port after an OS patch update?

Here is the definition: https://packaging.python.org/en/latest/specifications/binary-distribution-format/

There is a notion of a "platform tag" which denotes the platform for which the wheel has been built if it contains native code, by default it does "$(os}_${release}_${arch}". Run " pip debug --verbose" and see for compatible tags. If a wheel has been compiled with 13.5-RELEASE-p5 it won't be consumed by 13.5-RELEASE-p6. "wheel tags --remove --platform-tag=..." renames the file for multiplatform AND modfies the metadata (Tag: ) in WHEEL file. So I am not rebuilding, I am retagging. I first assumed a bug in poudriere: https://github.com/freebsd/poudriere/issues/1277, but then realized otherwise.

[arrowd]

In D53433#1223585, @michaelo wrote:

There is a subtile difference where. The wheels are already there with PEP 517, distutils just requires one more step, but I expect distutils to disappear in the future. So basically either "make wheel" would be a noop or it would move/copy the *.whl to ${WRKDIR}/whl/

All right, if wheels are already generated unconditionally, then this point is resolved.

Here is the definition: https://packaging.python.org/en/latest/specifications/binary-distribution-format/

There is a notion of a "platform tag" which denotes the platform for which the wheel has been built if it contains native code, by default it does "$(os}_${release}_${arch}". Run " pip debug --verbose" and see for compatible tags. If a wheel has been compiled with 13.5-RELEASE-p5 it won't be consumed by 13.5-RELEASE-p6. "wheel tags --remove --platform-tag=..." renames the file for multiplatform AND modfies the metadata (Tag: ) in WHEEL file. So I am not rebuilding, I am retagging. I first assumed a bug in poudriere: https://github.com/freebsd/poudriere/issues/1277, but then realized otherwise.

Well, to me it is the python part that should be fixed. FreeBSD guarantees ABI compatibility between minor releases, so the tag should look like py310-none-freebsd_13_amd64. This is the same how pkg handles our native packages ABI.

[michaelo]

In D53433#1223609, @arrowd wrote:

In D53433#1223585, @michaelo wrote:

There is a subtile difference where. The wheels are already there with PEP 517, distutils just requires one more step, but I expect distutils to disappear in the future. So basically either "make wheel" would be a noop or it would move/copy the *.whl to ${WRKDIR}/whl/

All right, if wheels are already generated unconditionally, then this point is resolved.

I am working on a ligher approach now which involves minimal changes to the ports system and poudriere would do the hard work.

Here is the definition: https://packaging.python.org/en/latest/specifications/binary-distribution-format/

There is a notion of a "platform tag" which denotes the platform for which the wheel has been built if it contains native code, by default it does "$(os}_${release}_${arch}". Run " pip debug --verbose" and see for compatible tags. If a wheel has been compiled with 13.5-RELEASE-p5 it won't be consumed by 13.5-RELEASE-p6. "wheel tags --remove --platform-tag=..." renames the file for multiplatform AND modfies the metadata (Tag: ) in WHEEL file. So I am not rebuilding, I am retagging. I first assumed a bug in poudriere: https://github.com/freebsd/poudriere/issues/1277, but then realized otherwise.

Well, to me it is the python part that should be fixed. FreeBSD guarantees ABI compatibility between minor releases, so the tag should look like py310-none-freebsd_13_amd64. This is the same how pkg handles our native packages ABI.

I agree with you, but that you I think require a PEP describing this (e.g., https://peps.python.org/pep-0513/). After that one needs to modify pip, setuptools, maturin, uv, and likely other code. It isn't straight forward. While I can raise the PRs for those tools, in fact I have done fixes for maturin on FreeBSD, the PEP stuff and formalization needs a formal process by someone from ?? and then I can approach to downstream. This is separate discussion I'd like to raise, but don't know with who.

[arrowd]

In D53433#1223612, @michaelo wrote:

I agree with you, but that you I think require a PEP describing this (e.g., https://peps.python.org/pep-0513/).

While this sounds complex, I think this would be just a several lines of code in a correct place. If you don't want to open a PEP for this (which also shouldn't as complex as manylinux one) we could at least patch this in our Python ports.

After that one needs to modify pip, setuptools, maturin, uv, and likely other code. It isn't straight forward.

Hmm, why so much places should be touched? Everything should start working out of the box when we get Python to use proper tags, no?

[michaelo]

In D53433#1223613, @arrowd wrote:

In D53433#1223612, @michaelo wrote:

I agree with you, but that you I think require a PEP describing this (e.g., https://peps.python.org/pep-0513/).

While this sounds complex, I think this would be just a several lines of code in a correct place. If you don't want to open a PEP for this (which also shouldn't as complex as manylinux one) we could at least patch this in our Python ports.

After that one needs to modify pip, setuptools, maturin, uv, and likely other code. It isn't straight forward.

Hmm, why so much places should be touched? Everything should start working out of the box when we get Python to use proper tags, no?

Because there are several build tools for Python wheels. Not all are written in Python, maturin is written in Rust.
https://github.com/PyO3/maturin/commit/43527f2aef4e84ab1c7fab104681e577aff4b94d this now compiles what setuptools does. Even if there would be a single place, I still would need to validate them.

[michaelo]

Simplify approach by making wheel destination explicit

[michaelo]

@arrowd I have now simplified the patch to work out of the box. It requires a subsequent patch (already prepared) for ports (seven of them) which overwrite those commands. The collection logic can fully live in poudriere: https://github.com/michael-o/poudriere-python-wheels/pull/1/files#diff-0afccccb8f3a6069f1dad4ec5c98125d10435243f7848807d2bcf65e00e88282R20-R34

This needs an exp-run of course.

[arrowd]

This looks OK to me, but I'm not a python hat wearer.

Mk/Uses/python.mk

975

I still don't think that whl building should be enabled by default for distutils.

[michaelo]

In D53433#1224026, @arrowd wrote:

This looks OK to me, but I'm a python hat wearer.

You mean you are not a Python hat wearer, don't you?

I will request an exp-run.

Mk/Uses/python.mk

975

The problem is that PEP 517 solutions always build a wheel, you can't disable it. It is the new approach. From a user's PoV it does not matter how the wheel was built: distutils or PEP 517. Adding a switch which enables/disables only a part of the ports set would, IMHO, cause more confusion than solve a problem.

I thought about a slightly different approach: Add to USES_package+=550:do-wheel which would copy the wheel from ${WRKSRC}/dist/ to ${WRKDIR}/whl, but this would result in unnecessary disk consumption.

[vishwin]

In D53433#1223076, @michaelo wrote:

My ideal longterm goal: The Project hosts canonical poudriere builds and these could produce FreeBSD-specific wheels which could be uploaded to PyPI and would dramatically improve the situation for the Python ecosystem on FreeBSD.

Not happening. We already build artefacts for ports with non-standard flags and such, which are inappropriate to share to PyPI.

USES=distutils is going away because upstream is removing support for this entirely. It had been deprecated for quite some time but they are finally pulling the switch.

[michaelo]

In D53433#1224438, @vishwin wrote:

In D53433#1223076, @michaelo wrote:

My ideal longterm goal: The Project hosts canonical poudriere builds and these could produce FreeBSD-specific wheels which could be uploaded to PyPI and would dramatically improve the situation for the Python ecosystem on FreeBSD.

Not happening. We already build artefacts for ports with non-standard flags and such, which are inappropriate to share to PyPI.

I see, alternatively, the FreeBSD project could host them as a separate index in parallel to packages. That should be feasable. But as said, this is a separate discussion outside of this review.

USES=distutils is going away because upstream is removing support for this entirely. It had been deprecated for quite some time but they are finally pulling the switch.

I know, this is why I said that with PEP 517 wheels already there, they just need to be in a distinct, explicit place for consumption.

Can you explain why this change needs revision? ATM, I don't see a way to make it even simple than this. Again, I don't expect you do upload anthing to PyPI. It is just a possible usecaes. I am happy if people can host their own indexes with this.

[vishwin]

In D53433#1224442, @michaelo wrote:

In D53433#1224438, @vishwin wrote:

USES=distutils is going away because upstream is removing support for this entirely. It had been deprecated for quite some time but they are finally pulling the switch.

I know, this is why I said that with PEP 517 wheels already there, they just need to be in a distinct, explicit place for consumption.

Can you explain why this change needs revision? ATM, I don't see a way to make it even simple than this. Again, I don't expect you do upload anthing to PyPI. It is just a possible usecaes. I am happy if people can host their own indexes with this.

From a meta-/framework perspective, building wheels from USE_PYTHON=distutils (ie direct setup.py invocation) is not reliable unless you don't pass any flags or variables at all. This stems from setuptools themselves continuing to deal with a littany of legacy code and design which is frustrating to rid for modernity and maintenance sakes. Trying to build a binary wheel from built artefacts in this case may result in those artefacts getting wrongly rebuilt. (I had some local modifications where USE_PYTHON=distutils only built binary wheels and eliminated its do-install in favour of the PEP-517 do-install.)

For those ports that still have a setup.py without a corresponding or replacement pyproject.toml that doesn't need to have flags or variables passed into configure or build, PEP-517 already provides that setuptools is implicitly the build backend. Those ports can switch to USE_PYTHON=pep517. For ports that need to pass flags and variables, setuptools still doesn't support config settings passed from a build frontend like devel/py-build.

[michaelo]

In D53433#1224443, @vishwin wrote:

In D53433#1224442, @michaelo wrote:

In D53433#1224438, @vishwin wrote:

USES=distutils is going away because upstream is removing support for this entirely. It had been deprecated for quite some time but they are finally pulling the switch.

I know, this is why I said that with PEP 517 wheels already there, they just need to be in a distinct, explicit place for consumption.

Can you explain why this change needs revision? ATM, I don't see a way to make it even simple than this. Again, I don't expect you do upload anthing to PyPI. It is just a possible usecaes. I am happy if people can host their own indexes with this.

From a meta-/framework perspective, building wheels from USE_PYTHON=distutils (ie direct setup.py invocation) is not reliable unless you don't pass any flags or variables at all. This stems from setuptools themselves continuing to deal with a littany of legacy code and design which is frustrating to rid for modernity and maintenance sakes. Trying to build a binary wheel from built artefacts in this case may result in those artefacts getting wrongly rebuilt. (I had some local modifications where USE_PYTHON=distutils only built binary wheels and eliminated its do-install in favour of the PEP-517 do-install.)

For those ports that still have a setup.py without a corresponding or replacement pyproject.toml that doesn't need to have flags or variables passed into configure or build, PEP-517 already provides that setuptools is implicitly the build backend. Those ports can switch to USE_PYTHON=pep517. For ports that need to pass flags and variables, setuptools still doesn't support config settings passed from a build frontend like devel/py-build.

That's profound. Let me process this.

[vishwin]

In D53433#1223609, @arrowd wrote:

In D53433#1223585, @michaelo wrote:

Here is the definition: https://packaging.python.org/en/latest/specifications/binary-distribution-format/

There is a notion of a "platform tag" which denotes the platform for which the wheel has been built if it contains native code, by default it does "$(os}_${release}_${arch}". Run " pip debug --verbose" and see for compatible tags. If a wheel has been compiled with 13.5-RELEASE-p5 it won't be consumed by 13.5-RELEASE-p6. "wheel tags --remove --platform-tag=..." renames the file for multiplatform AND modfies the metadata (Tag: ) in WHEEL file. So I am not rebuilding, I am retagging. I first assumed a bug in poudriere: https://github.com/freebsd/poudriere/issues/1277, but then realized otherwise.

Well, to me it is the python part that should be fixed. FreeBSD guarantees ABI compatibility between minor releases, so the tag should look like py310-none-freebsd_13_amd64. This is the same how pkg handles our native packages ABI.

Re-tagging is not necessary as they are already correct. Not every wheel will have specific platform tags, in fact most if not all ports specifying NO_ARCH will have py3-none-any which allows consumption on anything with any implementation/distribution. Patch levels are not part of specific platform tags, only minor releases, like cp312-cp312-freebsd_14_3_release_amd64 (denotes compatibility with CPython >= 3.12 on 14.3-RELEASE amd64), cp37-abi3-freebsd_16_0_current_amd64 (CPython >= 3.7 ABI 3 on -CURRENT amd64). (If we enable building ports under other implementations like lang/rustpython, that's another can of worms entirely…) Note that 3.13 and later have differing ABIs wrt free-threaded mode or not, like cp313-cp313t-freebsd_16_0_current_amd64.

Changes to this system will almost certainly require a PEP or a discussion on their forums.

[michaelo]

In D53433#1224455, @vishwin wrote:

In D53433#1223609, @arrowd wrote:

In D53433#1223585, @michaelo wrote:

Here is the definition: https://packaging.python.org/en/latest/specifications/binary-distribution-format/

There is a notion of a "platform tag" which denotes the platform for which the wheel has been built if it contains native code, by default it does "$(os}_${release}_${arch}". Run " pip debug --verbose" and see for compatible tags. If a wheel has been compiled with 13.5-RELEASE-p5 it won't be consumed by 13.5-RELEASE-p6. "wheel tags --remove --platform-tag=..." renames the file for multiplatform AND modfies the metadata (Tag: ) in WHEEL file. So I am not rebuilding, I am retagging. I first assumed a bug in poudriere: https://github.com/freebsd/poudriere/issues/1277, but then realized otherwise.

Well, to me it is the python part that should be fixed. FreeBSD guarantees ABI compatibility between minor releases, so the tag should look like py310-none-freebsd_13_amd64. This is the same how pkg handles our native packages ABI.

Re-tagging is not necessary as they are already correct. Not every wheel will have specific platform tags, in fact most if not all ports specifying NO_ARCH will have py3-none-any which allows consumption on anything with any implementation/distribution. Patch levels are not part of specific platform tags, only minor releases, like cp312-cp312-freebsd_14_3_release_amd64 (denotes compatibility with CPython >= 3.12 on 14.3-RELEASE amd64), cp37-abi3-freebsd_16_0_current_amd64 (CPython >= 3.7 ABI 3 on -CURRENT amd64). (If we enable building ports under other implementations like lang/rustpython, that's another can of worms entirely…) Note that 3.13 and later have differing ABIs wrt free-threaded mode or not, like cp313-cp313t-freebsd_16_0_current_amd64.

I am only retagging the wheels with native code, not those with any platform tag. The patch release is, unfortunately, part of the tag: sysconfig.get_platform()/pip debug --verbose and you'll see. Example:

scipy-1.15.3-cp310-cp310-freebsd_13_5_release_p5_amd64.whl

to be usable by all 13.5-RELEASE-pX I need to retag to a multiplatform: https://github.com/michael-o/poudriere-python-wheels/pull/1/files#diff-e6accb6936f8964cc7cb3f78b81bfdf7c9883fdb20a1e297e93a0cbf11a11caeR43-R72

Changes to this system will almost certainly require a PEP or a discussion on their forums.

Yes, correct. I take @arrowd's statement for face value that everything build on a major version runs on any minor and patch version of that. So the scipy wheel should only be scipy-1.15.3-cp310-cp310-freebsd_13_amd64.whl although I don't know whether it can be compiled on 13.5 and run older minor versions.

Built by uv:

$ find uv-cache/sdists* -name *.whl                                             
uv-cache/sdists-v9/index/50675900bad406b9/scipy/1.15.3/Rhv5O1fFW5kafLR8t2Qjz/scipy-1.15.3-cp310-cp310-freebsd_13_5_release_p5_amd64.whl
uv-cache/sdists-v9/index/50675900bad406b9/backports-datetime-fromisoformat/2.0.3/VYGvOgUTu1iuMOAt8OjQy/backports_datetime_fromisoformat-2.0.3-cp310-cp310-freebsd_13_5_release_p5_amd64.whl
uv-cache/sdists-v9/index/50675900bad406b9/numpy/2.2.6/fZmiN98zV_54l0Ra4mdJ9/numpy-2.2.6-cp310-cp310-freebsd_13_5_release_p5_amd64.whl

cannot be used on 13.5-RELEASE-p6.

My wheels from poudriere after retagging: https://gist.github.com/michael-o/b79c824940017632d8301de0c28aa364

I will raise this discussion on our mailing lists after we have settled this one. Rebuilding wheels for patch versions would be insane.

[arrowd]

In D53433#1224456, @michaelo wrote:

I don't know whether it can be compiled on 13.5 and run older minor versions.

It can't generally. But this is not a problem, because the support window between two consecutive minor releases is usually tight and the official cluster always builds packages on the lowest minor version supported.

[michaelo]

In D53433#1224457, @arrowd wrote:

In D53433#1224456, @michaelo wrote:

I don't know whether it can be compiled on 13.5 and run older minor versions.

It can't generally. But this is not a problem, because the support window between two consecutive minor releases is usually tight and the official cluster always builds packages on the lowest minor version supported.

This basically means that one must strictly observe to provide wheels with an older minor version or incorporate the minor version into the wheel's platform tag.

[michaelo]

In D53433#1224443, @vishwin wrote:

In D53433#1224442, @michaelo wrote:

In D53433#1224438, @vishwin wrote:

USES=distutils is going away because upstream is removing support for this entirely. It had been deprecated for quite some time but they are finally pulling the switch.

I know, this is why I said that with PEP 517 wheels already there, they just need to be in a distinct, explicit place for consumption.

Can you explain why this change needs revision? ATM, I don't see a way to make it even simple than this. Again, I don't expect you do upload anthing to PyPI. It is just a possible usecaes. I am happy if people can host their own indexes with this.

From a meta-/framework perspective, building wheels from USE_PYTHON=distutils (ie direct setup.py invocation) is not reliable unless you don't pass any flags or variables at all. This stems from setuptools themselves continuing to deal with a littany of legacy code and design which is frustrating to rid for modernity and maintenance sakes. Trying to build a binary wheel from built artefacts in this case may result in those artefacts getting wrongly rebuilt. (I had some local modifications where USE_PYTHON=distutils only built binary wheels and eliminated its do-install in favour of the PEP-517 do-install.)

For those ports that still have a setup.py without a corresponding or replacement pyproject.toml that doesn't need to have flags or variables passed into configure or build, PEP-517 already provides that setuptools is implicitly the build backend. Those ports can switch to USE_PYTHON=pep517. For ports that need to pass flags and variables, setuptools still doesn't support config settings passed from a build frontend like devel/py-build.

I've put a couple of hours into understanding your point and the problem. Yes, I acknowledge it and understand that we cannot solve this problem unless the port fully moves to PEP 517 with a reasonable backend which passes flags or env vars. bdist_wheel can be unpredictable and will ignore everything passed to setup.py build. Ca. 50% of the ports still use distutils which is a shame.
Can we achieve a compromise with such a patch?

diff --git a/Mk/Uses/python.mk b/Mk/Uses/python.mk
index 6b618d4b2562..c4f698b6602c 100644
--- a/Mk/Uses/python.mk
+++ b/Mk/Uses/python.mk
@@ -188,6 +188,10 @@
 #          - Pass this command to distutils on build stage.
 #            default: build
 #
+# PYDISTUTILS_BUILD_WHEEL
+#          - Define to build wheel with distutils on build wheel stage.
+#            default: <not defined>
+#
 # PYDISTUTILS_BUILD_WHEEL_TARGET
 #          - Pass this command to distutils on build wheel stage.
 #            default: bdist_wheel
@@ -206,7 +210,7 @@
 #
 # PYDISTUTILS_BUILDWHEELARGS
 #          - Arguments to build wheel with distutils.
-#            default: --dist-dir ${WRKDIR}/whl
+#            default: --skip-build --dist-dir ${WRKDIR}/whl
 #
 # PYDISTUTILS_INSTALLARGS
 #          - Arguments to install with distutils.
@@ -751,7 +755,7 @@ PYDISTUTILS_SETUP?= -c \
    exec(compile(open(__file__, 'rb').read().replace(b'\\r\\n', b'\\n'), __file__, 'exec'))"
 PYDISTUTILS_CONFIGUREARGS?=    # empty
 PYDISTUTILS_BUILDARGS?=        # empty
-PYDISTUTILS_BUILDWHEELARGS?=   --dist-dir ${WRKDIR}/whl
+PYDISTUTILS_BUILDWHEELARGS?=   --skip-build --dist-dir ${WRKDIR}/whl
 PYDISTUTILS_INSTALLARGS?=  -c -O1 --prefix=${PREFIX}
 .  if defined(_PYTHON_FEATURE_DISTUTILS)
 .    if !defined(PYDISTUTILS_INSTALLNOSINGLE)
@@ -972,7 +976,9 @@ do-configure:
 .    if !target(do-build)
 do-build:
    @(cd ${BUILD_WRKSRC}; ${SETENVI} ${WRK_ENV} ${MAKE_ENV} ${PYTHON_CMD} ${PYDISTUTILS_SETUP} ${PYDISTUTILS_BUILD_TARGET} ${PYDISTUTILS_BUILDARGS})
+.      if defined(PYDISTUTILS_BUILD_WHEEL)
    @(cd ${BUILD_WRKSRC}; ${SETENVI} ${WRK_ENV} ${MAKE_ENV} ${PYTHON_CMD} ${PYDISTUTILS_SETUP} ${PYDISTUTILS_BUILD_WHEEL_TARGET} ${PYDISTUTILS_BUILDWHEELARGS})
+.      endif
 .    endif

 .    if !target(do-install)

PYDISTUTILS_BUILD_WHEEL can be accompanied with a warning about the possible erratic build behavior.

[michaelo]

Updating the patch making bdist_wheel opt-in.

[michaelo]

Just experienced the inconsistency with distutils myself:

with --skip-build:

# tar -tzf PyYAML-6.0.3-cp310-cp310-freebsd_13_5_release_p4_amd64.freebsd_13_5_release_p5_amd64.freebsd_13_5_release_p6_amd64.whl
yaml/_yaml.cpython-310.so
PyYAML-6.0.3.dist-info/LICENSE
PyYAML-6.0.3.dist-info/METADATA
PyYAML-6.0.3.dist-info/WHEEL
PyYAML-6.0.3.dist-info/top_level.txt
PyYAML-6.0.3.dist-info/RECORD

without the flag:

# tar -tzf PyYAML-6.0.3-cp310-cp310-freebsd_13_5_release_p4_amd64.freebsd_13_5_release_p5_amd64.freebsd_13_5_release_p6_amd64.whl
_yaml/__init__.py
yaml/__init__.py
yaml/_yaml.cpython-310.so
yaml/composer.py
yaml/constructor.py
yaml/cyaml.py
yaml/dumper.py
yaml/emitter.py
yaml/error.py
yaml/events.py
yaml/loader.py
yaml/nodes.py
yaml/parser.py
yaml/reader.py
yaml/representer.py
yaml/resolver.py
yaml/scanner.py
yaml/serializer.py
yaml/tokens.py
PyYAML-6.0.3.dist-info/LICENSE
PyYAML-6.0.3.dist-info/METADATA
PyYAML-6.0.3.dist-info/WHEEL
PyYAML-6.0.3.dist-info/top_level.txt
PyYAML-6.0.3.dist-info/RECORD

Removed the flag in the patch.

[michaelo]

@vishwin Have you had time to check the updated patch? The distutils approach is now behind a switch, being an opt-in.

[vishwin]

In D53433#1232130, @michaelo wrote:

@vishwin Have you had time to check the updated patch? The distutils approach is now behind a switch, being an opt-in.

I'm actually not in favour of supporting any wheel building for USE_PYTHON=distutils. For an interface that upstream have long deprecated, stopped supporting and barely maintain (almost solely to keep up with other module changes and deprecations), we should not be encouraging and exposing continued use of it. An opt-in switch does not change this position.

I grepped for PYDISTUTILS_CONFIGUREARGS and PYDISTUTILS_BUILDARGS to get a count, 10 and 28 respectively, but also remember why they are defined. Some ports are passing flags, others are passing additional commands to setup.py alongside configure or install. Sometimes extension modules are built separately from the main build via build_ext; those artefacts may not end up in the wheel. I did some further testing on passing environment variables on USE_PYTHON=pep517 ports with setuptools build backend and at least that seems to work now, at least with the latest upstream setuptools not yet in tree (older devel/py-build had some issues I believe).

As for opt-in switches, I would like to see one for the USE_PYTHON=pep517 case, at least to see how it would look. Supporting this mode shall not modify the default PEP517_{BUILD,INSTALL}_CMD since they can be overridden.

Additionally, note that unlike USE_PYTHON=distutils, wheel installation does not touch anything outside of PYTHON{,PREFIX}_SITELIBDIR, so for those with files that would belong in etc/, share/ et al, they are manually install(1) into the appropriate places during post-install.

[michaelo]

@vishwin Took me way too long to respond. you are right on the distutils,
I have dropped the entire block and will be moved to a separate discussion,
if at all.

I have now fully focused on PEP 517 and the change is minimal, I have also
reworked the title and well as the summary.

My basic idea does not change any behavior or introduce anything new being
built, but all wheels now land in a well-defined/explicit directory during build.
There is logically no opt-in or opt-out like in distutils since wheels are
ALWAYS part of the build.

That's it.

[michaelo]

Subsequent review for ports which require extra treatment: https://reviews.freebsd.org/D58861, all builds pass for me. Some of them will go away as soon as setuptools is being updated, e.g., wheel name normalization.

[michaelo]

Addon:
One thing worth spelling out for the record, since it came up earlier: a toggle (whether per-port or tree-wide) wouldn't actually change what's required here. Any port overriding PEP517_BUILD_CMD/PEP517_INSTALL_CMD directly, or invoking a build tool standalone (e.g. maturin), still needs its own explicit path update on both the build and install side regardless of what a switch defaults to — otherwise the two sides silently disagree on where the wheel lives and the port breaks. That's the entire content of the follow-up sweep commit; a toggle doesn't remove or shrink it, since those ports never go through the framework default in the first place.