Mk/Uses/python.mk: Build Python wheels next to packages

[michaelo]

Summary

Packages are being built in an explicit directory: ${WRKDIR}/pkg, wheels
however are nested somewhere in ${WRKSRC} making it for external builders like
poudriere harder to locate. Now they follow the same explicit pattern by
creating them in ${WRKDIR}/whl.

PR: 290653

[arrowd]

I don't quite get the rationale. Can you provide an example how is it going to be used?

[michaelo]

In D53433#1223067, @arrowd wrote:

I don't quite get the rationale. Can you provide an example how is it going to be used?

Yes, sure. Let's assume this lands in main and I enable in make.conf to install the wheel files to /var/cache/python-wheels. Then in pouodriere I can collect them after the build and serve a private index: https://github.com/michael-o/poudriere-python-wheels/pull/1

There are no FreeBSD wheels on PyPI and we have hefty dependencies for numpy, scipy, etc. They take ages to compile even with 24 threads. With poudriere, I have a working build system which already builds the wheels, I will simply retain them and re-use as an index. I cannot install any of the wheels as packages because uv intentionally ignores system-installed wheels:

$ git diff -U0 pyproject.toml
diff --git a/pyproject.toml b/pyproject.toml
index 673c4c1..8a95236 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -43,0 +44,3 @@ dev = [
+[[tool.uv.index]]
+name = "FREEBSD_WHEELS"
+url  = "https://dw-eng-rsc.innomotics.net/FreeBSD/python-wheels/FreeBSD:13:amd64/kona-latest/simple/"

At the end, it is about reuseability, avoid repetitive builds across CI/CD and final deployment. As soon as the index has all deps the application needs only 15 s to asseble, without 15 min.

[michaelo]

My ideal longterm goal: The Project hosts canonical poudriere builds and these could produce FreeBSD-specific wheels which could be uploaded to PyPI and would dramatically improve the situation for the Python ecosystem on FreeBSD.

[arrowd]

Then this change alone isn't really enough and I'm not even sure it is a right step forward.

The overall idea makes sense to me, though. At the moment the only artifact type the FreeBSD Ports tree can produce is a FreeBSD package. Python wheels or AppImage binaries are another examples. I'm as well planning to implement the later at some point.

But the full implementation requires work not only in Uses/python.mk, but also in Mk/bsd.port.mk and on the Poudriere side of things. I'm ready to review a complete implementation.

[michaelo]

In D53433#1223077, @arrowd wrote:

Then this change alone isn't really enough and I'm not even sure it is a right step forward.

Of course it is not enough, but that applies to the entire ports system. Without poudriere and a distribution server you can't serve packages/wheels. It is a component.

The overall idea makes sense to me, though. At the moment the only artifact type the FreeBSD Ports tree can produce is a FreeBSD package. Python wheels or AppImage binaries are another examples. I'm as well planning to implement the later at some point.

Longterm, it should be able to do.

But the full implementation requires work not only in Uses/python.mk, but also in Mk/bsd.port.mk and on the Poudriere side of things. I'm ready to review a complete implementation.

What is a full implementation for you? My poudriere change is in the works and will be a plugin in the hooks system which works protoypically. It will be open source of course. What you have in mind is a 100% solution which is not feasable right, but this is a good step to evaluate anything else. I can mark this as EXPERIMENTAL and mention this as EXPERIMENTAL in UPDATING. I'd like to get feedback from people as well how we can evolve this.

Again, this works already: Poudriere builds wheels, collects them via nullfs mount, dir2pi generates a static index and Apache HTTPd hosts this index. Exactly as for packages.

[arrowd]

Link your Poudriere changes in there, so it is possible to get an idea how this is going to work overall.

[michaelo]

In D53433#1223122, @arrowd wrote:

Link your Poudriere changes in there, so it is possible to get an idea how this is going to work overall.

I did, work in progress for poudriere: https://github.com/michael-o/poudriere-python-wheels/pull/1. This will turned into a project on GitHub soon. uv AND are already consuming it properly in a test setup.
Apache as usual:

$ cat freebsd-python-wheels.conf
Alias /FreeBSD/python-wheels/FreeBSD:13:amd64/cafe-custom-uis-latest "/var/poudriere/data/python-wheels/135-release-amd64-default-head-cafe-custom-uis"
Alias /FreeBSD/python-wheels/FreeBSD:13:amd64/kona-latest "/var/poudriere/data/python-wheels/135-release-amd64-default-head-kona"
Alias /FreeBSD/python-wheels/FreeBSD:13:amd64/cafe-custom-uis-quarterly "/var/poudriere/data/python-wheels/135-release-amd64-default-cafe-custom-uis"
Alias /FreeBSD/python-wheels/FreeBSD:13:amd64/kona-quarterly "/var/poudriere/data/python-wheels/135-release-amd64-default-kona"

<Directory "/var/poudriere/data/python-wheels">
    Options Indexes FollowSymLinks
    IndexOptions FancyIndexing FoldersFirst SuppressDescription SuppressColumnSorting NameWidth=50 VersionSort
    AllowOverride None
    Require all granted
</Directory>

[arrowd]

It should be turned into a proper pull request to the Poudriere upstream repo. One thing that already catched my eye is that wheel generation happens from the host, not within the jail. Bryan might disagree with this.

[michaelo]

In D53433#1223124, @arrowd wrote:

It should be turned into a proper pull request to the Poudriere upstream repo.

As a plugin, yes? But as long as It is not stable I don't want to have it upstream and constantly break people's stuff unless we mark it as experimental. My goal was to decouple from poduriere because people might just use portmaster.

One thing that already catched my eye is that wheel generation happens from the host, not within the jail. Bryan might disagree with this.

No*, the wheels are generated in the jail during build. What I do is to make the multiplatform because there is current no equivalent of manylinux for FreeBSD. Unfortunately, sysconfig.get_platform() is used which will contain the patch level as well. This leads to issues that a wheel isn't found if it does not matches. Unless there is a PEP for "manyfreebsd" to have a base line on major.minor one needs to rewrite the platform tags.

\* You are right, I could move that logic into the ports system, not do it in poudriere. I have never thought of that actually. That is likely even better than doing in poudriere. Shall I give it a try?

[michaelo]

@arrowd There is one issue with retagging the wheel inside of the jail. Let's say the jail has been updated from 13.5-RELEASE-p5 to 13.5-RELEASE-p7. There is a wheel for package foo. The multiplatform wheel goes only to 13.5-RELEASE-p5. Unless the port is upgraded to a newer version the multiplatform tags will never be updated and for jails running 13.5-RELEASE-p6+ unavailable. This means that I still need to perform the retagging for older wheels, but not for newer ones with a poudriere hook. So at the end it will be duplicate work. I don't know wether this is really smart to do...

[arrowd]

In D53433#1223130, @michaelo wrote:

As a plugin, yes?

If it can be completely done via a hook - yes, but not necessary. It might be as well a core feature.

But as long as It is not stable I don't want to have it upstream and constantly break people's stuff unless we mark it as experimental.

This is not a problem - we have poudriere-devel exactly for that.

My goal was to decouple from poduriere because people might just use portmaster.
You are right, I could move that logic into the ports system, not do it in poudriere. I have never thought of that actually. That is likely even better than doing in poudriere. Shall I give it a try?

I think yes. This is what I have in mind for AppImage:

• At the Ports level we introduce the make appimage target in addition to make package. This target would create an AppImage binary in ${WRKDIR} out of ${WRKSRC}. This can be used by plain make users.
• We ifdef the AppImage stuff under the APPIMAGE_BUILDING. This disables AppImage stuff by default, but allows portmaster/synth users to invoke AppImage machinery by adding APPIMAGE_BUILDING=yes into /etc/make.conf.
• At the Poudriere level we introduce poudriere appimage-bulk subcommand that works like usual bulk, but defines APPIMAGE_BUILDING and collects AppImage artifacts instead of .pkg files.

Note that my case is more complex, because a port should be built in a different way to make it possible to be converted into an AppImage. From what I gather, there are no special measures required to create .whl from a python ${WRKDIR}, so it'd be even easier for you. We just build a Python port the usual way and then invoke make wheel instead of make package.

So, I'd start with introducing the make wheel target, that is only enabled if WHEEL_BUILDING is defined. Then we can play with it on a plain make level and then we'll figure out how to properly integrate it into Poudriere.

There is one issue with retagging the wheel inside of the jail. Let's say the jail has been updated from 13.5-RELEASE-p5 to 13.5-RELEASE-p7. There is a wheel for package foo. The multiplatform wheel goes only to 13.5-RELEASE-p5. Unless the port is upgraded to a newer version the multiplatform tags will never be updated and for jails running 13.5-RELEASE-p6+ unavailable. This means that I still need to perform the retagging for older wheels, but not for newer ones with a poudriere hook.

I don't know much about Python ways of doing thing, so I'm a bit confused. Are you saying that a wheel encodes the OS version it was built on? This does not sound like a problem to me - even a boring C port may do this. Why do you want to rebuild a such a port after an OS patch update?

[michaelo]

In D53433#1223564, @arrowd wrote:

In D53433#1223130, @michaelo wrote:

As a plugin, yes?

If it can be completely done via a hook - yes, but not necessary. It might be as well a core feature.

But as long as It is not stable I don't want to have it upstream and constantly break people's stuff unless we mark it as experimental.

This is not a problem - we have poudriere-devel exactly for that.

It is a separate project how: https://github.com/michael-o/poudriere-python-wheels/pull/1

My goal was to decouple from poduriere because people might just use portmaster.
You are right, I could move that logic into the ports system, not do it in poudriere. I have never thought of that actually. That is likely even better than doing in poudriere. Shall I give it a try?

I think yes. This is what I have in mind for AppImage:

• At the Ports level we introduce the make appimage target in addition to make package. This target would create an AppImage binary in ${WRKDIR} out of ${WRKSRC}. This can be used by plain make users.
• We ifdef the AppImage stuff under the APPIMAGE_BUILDING. This disables AppImage stuff by default, but allows portmaster/synth users to invoke AppImage machinery by adding APPIMAGE_BUILDING=yes into /etc/make.conf.
• At the Poudriere level we introduce poudriere appimage-bulk subcommand that works like usual bulk, but defines APPIMAGE_BUILDING and collects AppImage artifacts instead of .pkg files.

Note that my case is more complex, because a port should be built in a different way to make it possible to be converted into an AppImage. From what I gather, there are no special measures required to create .whl from a python ${WRKDIR}, so it'd be even easier for you. We just build a Python port the usual way and then invoke make wheel instead of make package. See PEP517_INSTALL_WHEEL_CMD

So, I'd start with introducing the make wheel target, that is only enabled if WHEEL_BUILDING is defined. Then we can play with it on a plain make level and then we'll figure out how to properly integrate it into Poudriere.

There is a subtile difference where. The wheels are already there with PEP 517, distutils just requires one more step, but I expect distutils to disappear in the future. So basically either "make wheel" would be a noop or it would move/copy the *.whl to ${WRKDIR}/whl/

There is one issue with retagging the wheel inside of the jail. Let's say the jail has been updated from 13.5-RELEASE-p5 to 13.5-RELEASE-p7. There is a wheel for package foo. The multiplatform wheel goes only to 13.5-RELEASE-p5. Unless the port is upgraded to a newer version the multiplatform tags will never be updated and for jails running 13.5-RELEASE-p6+ unavailable. This means that I still need to perform the retagging for older wheels, but not for newer ones with a poudriere hook.

I don't know much about Python ways of doing thing, so I'm a bit confused. Are you saying that a wheel encodes the OS version it was built on? This does not sound like a problem to me - even a boring C port may do this. Why do you want to rebuild a such a port after an OS patch update?

Here is the definition: https://packaging.python.org/en/latest/specifications/binary-distribution-format/

There is a notion of a "platform tag" which denotes the platform for which the wheel has been built if it contains native code, by default it does "$(os}_${release}_${arch}". Run " pip debug --verbose" and see for compatible tags. If a wheel has been compiled with 13.5-RELEASE-p5 it won't be consumed by 13.5-RELEASE-p6. "wheel tags --remove --platform-tag=..." renames the file for multiplatform AND modfies the metadata (Tag: ) in WHEEL file. So I am not rebuilding, I am retagging. I first assumed a bug in poudriere: https://github.com/freebsd/poudriere/issues/1277, but then realized otherwise.

[arrowd]

In D53433#1223585, @michaelo wrote:

There is a subtile difference where. The wheels are already there with PEP 517, distutils just requires one more step, but I expect distutils to disappear in the future. So basically either "make wheel" would be a noop or it would move/copy the *.whl to ${WRKDIR}/whl/

All right, if wheels are already generated unconditionally, then this point is resolved.

Here is the definition: https://packaging.python.org/en/latest/specifications/binary-distribution-format/

There is a notion of a "platform tag" which denotes the platform for which the wheel has been built if it contains native code, by default it does "$(os}_${release}_${arch}". Run " pip debug --verbose" and see for compatible tags. If a wheel has been compiled with 13.5-RELEASE-p5 it won't be consumed by 13.5-RELEASE-p6. "wheel tags --remove --platform-tag=..." renames the file for multiplatform AND modfies the metadata (Tag: ) in WHEEL file. So I am not rebuilding, I am retagging. I first assumed a bug in poudriere: https://github.com/freebsd/poudriere/issues/1277, but then realized otherwise.

Well, to me it is the python part that should be fixed. FreeBSD guarantees ABI compatibility between minor releases, so the tag should look like py310-none-freebsd_13_amd64. This is the same how pkg handles our native packages ABI.

[michaelo]

In D53433#1223609, @arrowd wrote:

In D53433#1223585, @michaelo wrote:

There is a subtile difference where. The wheels are already there with PEP 517, distutils just requires one more step, but I expect distutils to disappear in the future. So basically either "make wheel" would be a noop or it would move/copy the *.whl to ${WRKDIR}/whl/

All right, if wheels are already generated unconditionally, then this point is resolved.

I am working on a ligher approach now which involves minimal changes to the ports system and poudriere would do the hard work.

Here is the definition: https://packaging.python.org/en/latest/specifications/binary-distribution-format/

There is a notion of a "platform tag" which denotes the platform for which the wheel has been built if it contains native code, by default it does "$(os}_${release}_${arch}". Run " pip debug --verbose" and see for compatible tags. If a wheel has been compiled with 13.5-RELEASE-p5 it won't be consumed by 13.5-RELEASE-p6. "wheel tags --remove --platform-tag=..." renames the file for multiplatform AND modfies the metadata (Tag: ) in WHEEL file. So I am not rebuilding, I am retagging. I first assumed a bug in poudriere: https://github.com/freebsd/poudriere/issues/1277, but then realized otherwise.

Well, to me it is the python part that should be fixed. FreeBSD guarantees ABI compatibility between minor releases, so the tag should look like py310-none-freebsd_13_amd64. This is the same how pkg handles our native packages ABI.

I agree with you, but that you I think require a PEP describing this (e.g., https://peps.python.org/pep-0513/). After that one needs to modify pip, setuptools, maturin, uv, and likely other code. It isn't straight forward. While I can raise the PRs for those tools, in fact I have done fixes for maturin on FreeBSD, the PEP stuff and formalization needs a formal process by someone from ?? and then I can approach to downstream. This is separate discussion I'd like to raise, but don't know with who.

[arrowd]

In D53433#1223612, @michaelo wrote:

I agree with you, but that you I think require a PEP describing this (e.g., https://peps.python.org/pep-0513/).

While this sounds complex, I think this would be just a several lines of code in a correct place. If you don't want to open a PEP for this (which also shouldn't as complex as manylinux one) we could at least patch this in our Python ports.

After that one needs to modify pip, setuptools, maturin, uv, and likely other code. It isn't straight forward.

Hmm, why so much places should be touched? Everything should start working out of the box when we get Python to use proper tags, no?

[michaelo]

In D53433#1223613, @arrowd wrote:

In D53433#1223612, @michaelo wrote:

I agree with you, but that you I think require a PEP describing this (e.g., https://peps.python.org/pep-0513/).

While this sounds complex, I think this would be just a several lines of code in a correct place. If you don't want to open a PEP for this (which also shouldn't as complex as manylinux one) we could at least patch this in our Python ports.

After that one needs to modify pip, setuptools, maturin, uv, and likely other code. It isn't straight forward.

Hmm, why so much places should be touched? Everything should start working out of the box when we get Python to use proper tags, no?

Because there are several build tools for Python wheels. Not all are written in Python, maturin is written in Rust.
https://github.com/PyO3/maturin/commit/43527f2aef4e84ab1c7fab104681e577aff4b94d this now compiles what setuptools does. Even if there would be a single place, I still would need to validate them.

[michaelo]

Simplify approach by making wheel destination explicit

[michaelo]

@arrowd I have now simplified the patch to work out of the box. It requires a subsequent patch (already prepared) for ports (seven of them) which overwrite those commands. The collection logic can fully live in poudriere: https://github.com/michael-o/poudriere-python-wheels/pull/1/files#diff-0afccccb8f3a6069f1dad4ec5c98125d10435243f7848807d2bcf65e00e88282R20-R34

This needs an exp-run of course.

[arrowd]

This looks OK to me, but I'm not a python hat wearer.

Mk/Uses/python.mk

975

I still don't think that whl building should be enabled by default for distutils.

[michaelo]

In D53433#1224026, @arrowd wrote:

This looks OK to me, but I'm a python hat wearer.

You mean you are not a Python hat wearer, don't you?

I will request an exp-run.

Mk/Uses/python.mk

975

The problem is that PEP 517 solutions always build a wheel, you can't disable it. It is the new approach. From a user's PoV it does not matter how the wheel was built: distutils or PEP 517. Adding a switch which enables/disables only a part of the ports set would, IMHO, cause more confusion than solve a problem.

I thought about a slightly different approach: Add to USES_package+=550:do-wheel which would copy the wheel from ${WRKSRC}/dist/ to ${WRKDIR}/whl, but this would result in unnecessary disk consumption.