Differential D53433
Mk/Uses/python.mk: Build Python wheels next to packages Authored by michaelo on Wed, Oct 29, 11:37 AM. Tags None Referenced Files
Subscribers
Details Packages are being built in an explicit directory: ${WRKDIR}/pkg, wheels PR: 290653
Diff Detail
Event TimelineComment Actions I don't quite get the rationale. Can you provide an example how is it going to be used? Comment Actions Yes, sure. Let's assume this lands in main and I enable in make.conf to install the wheel files to /var/cache/python-wheels. Then in pouodriere I can collect them after the build and serve a private index: https://github.com/michael-o/poudriere-python-wheels/pull/1 There are no FreeBSD wheels on PyPI and we have hefty dependencies for numpy, scipy, etc. They take ages to compile even with 24 threads. With poudriere, I have a working build system which already builds the wheels, I will simply retain them and re-use as an index. I cannot install any of the wheels as packages because uv intentionally ignores system-installed wheels: $ git diff -U0 pyproject.toml diff --git a/pyproject.toml b/pyproject.toml index 673c4c1..8a95236 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -43,0 +44,3 @@ dev = [ +[[tool.uv.index]] +name = "FREEBSD_WHEELS" +url = "https://dw-eng-rsc.innomotics.net/FreeBSD/python-wheels/FreeBSD:13:amd64/kona-latest/simple/" At the end, it is about reuseability, avoid repetitive builds across CI/CD and final deployment. As soon as the index has all deps the application needs only 15 s to asseble, without 15 min. Comment Actions My ideal longterm goal: The Project hosts canonical poudriere builds and these could produce FreeBSD-specific wheels which could be uploaded to PyPI and would dramatically improve the situation for the Python ecosystem on FreeBSD. Comment Actions Then this change alone isn't really enough and I'm not even sure it is a right step forward. The overall idea makes sense to me, though. At the moment the only artifact type the FreeBSD Ports tree can produce is a FreeBSD package. Python wheels or AppImage binaries are another examples. I'm as well planning to implement the later at some point. But the full implementation requires work not only in Uses/python.mk, but also in Mk/bsd.port.mk and on the Poudriere side of things. I'm ready to review a complete implementation. Comment Actions Of course it is not enough, but that applies to the entire ports system. Without poudriere and a distribution server you can't serve packages/wheels. It is a component.
Longterm, it should be able to do.
What is a full implementation for you? My poudriere change is in the works and will be a plugin in the hooks system which works protoypically. It will be open source of course. What you have in mind is a 100% solution which is not feasable right, but this is a good step to evaluate anything else. I can mark this as EXPERIMENTAL and mention this as EXPERIMENTAL in UPDATING. I'd like to get feedback from people as well how we can evolve this. Again, this works already: Poudriere builds wheels, collects them via nullfs mount, dir2pi generates a static index and Apache HTTPd hosts this index. Exactly as for packages. Comment Actions Link your Poudriere changes in there, so it is possible to get an idea how this is going to work overall. Comment Actions I did, work in progress for poudriere: https://github.com/michael-o/poudriere-python-wheels/pull/1. This will turned into a project on GitHub soon. uv AND are already consuming it properly in a test setup. $ cat freebsd-python-wheels.conf Alias /FreeBSD/python-wheels/FreeBSD:13:amd64/cafe-custom-uis-latest "/var/poudriere/data/python-wheels/135-release-amd64-default-head-cafe-custom-uis" Alias /FreeBSD/python-wheels/FreeBSD:13:amd64/kona-latest "/var/poudriere/data/python-wheels/135-release-amd64-default-head-kona" Alias /FreeBSD/python-wheels/FreeBSD:13:amd64/cafe-custom-uis-quarterly "/var/poudriere/data/python-wheels/135-release-amd64-default-cafe-custom-uis" Alias /FreeBSD/python-wheels/FreeBSD:13:amd64/kona-quarterly "/var/poudriere/data/python-wheels/135-release-amd64-default-kona" <Directory "/var/poudriere/data/python-wheels"> Options Indexes FollowSymLinks IndexOptions FancyIndexing FoldersFirst SuppressDescription SuppressColumnSorting NameWidth=50 VersionSort AllowOverride None Require all granted </Directory> Comment Actions It should be turned into a proper pull request to the Poudriere upstream repo. One thing that already catched my eye is that wheel generation happens from the host, not within the jail. Bryan might disagree with this. Comment Actions As a plugin, yes? But as long as It is not stable I don't want to have it upstream and constantly break people's stuff unless we mark it as experimental. My goal was to decouple from poduriere because people might just use portmaster.
No*, the wheels are generated in the jail during build. What I do is to make the multiplatform because there is current no equivalent of manylinux for FreeBSD. Unfortunately, sysconfig.get_platform() is used which will contain the patch level as well. This leads to issues that a wheel isn't found if it does not matches. Unless there is a PEP for "manyfreebsd" to have a base line on major.minor one needs to rewrite the platform tags. \* You are right, I could move that logic into the ports system, not do it in poudriere. I have never thought of that actually. That is likely even better than doing in poudriere. Shall I give it a try? Comment Actions @arrowd There is one issue with retagging the wheel inside of the jail. Let's say the jail has been updated from 13.5-RELEASE-p5 to 13.5-RELEASE-p7. There is a wheel for package foo. The multiplatform wheel goes only to 13.5-RELEASE-p5. Unless the port is upgraded to a newer version the multiplatform tags will never be updated and for jails running 13.5-RELEASE-p6+ unavailable. This means that I still need to perform the retagging for older wheels, but not for newer ones with a poudriere hook. So at the end it will be duplicate work. I don't know wether this is really smart to do... Comment Actions If it can be completely done via a hook - yes, but not necessary. It might be as well a core feature.
This is not a problem - we have poudriere-devel exactly for that.
I think yes. This is what I have in mind for AppImage:
Note that my case is more complex, because a port should be built in a different way to make it possible to be converted into an AppImage. From what I gather, there are no special measures required to create .whl from a python ${WRKDIR}, so it'd be even easier for you. We just build a Python port the usual way and then invoke make wheel instead of make package. So, I'd start with introducing the make wheel target, that is only enabled if WHEEL_BUILDING is defined. Then we can play with it on a plain make level and then we'll figure out how to properly integrate it into Poudriere.
I don't know much about Python ways of doing thing, so I'm a bit confused. Are you saying that a wheel encodes the OS version it was built on? This does not sound like a problem to me - even a boring C port may do this. Why do you want to rebuild a such a port after an OS patch update? Comment Actions It is a separate project how: https://github.com/michael-o/poudriere-python-wheels/pull/1
There is a subtile difference where. The wheels are already there with PEP 517, distutils just requires one more step, but I expect distutils to disappear in the future. So basically either "make wheel" would be a noop or it would move/copy the *.whl to ${WRKDIR}/whl/
Here is the definition: https://packaging.python.org/en/latest/specifications/binary-distribution-format/ There is a notion of a "platform tag" which denotes the platform for which the wheel has been built if it contains native code, by default it does "$(os}_${release}_${arch}". Run " pip debug --verbose" and see for compatible tags. If a wheel has been compiled with 13.5-RELEASE-p5 it won't be consumed by 13.5-RELEASE-p6. "wheel tags --remove --platform-tag=..." renames the file for multiplatform AND modfies the metadata (Tag: ) in WHEEL file. So I am not rebuilding, I am retagging. I first assumed a bug in poudriere: https://github.com/freebsd/poudriere/issues/1277, but then realized otherwise. Comment Actions All right, if wheels are already generated unconditionally, then this point is resolved.
Well, to me it is the python part that should be fixed. FreeBSD guarantees ABI compatibility between minor releases, so the tag should look like py310-none-freebsd_13_amd64. This is the same how pkg handles our native packages ABI. Comment Actions I am working on a ligher approach now which involves minimal changes to the ports system and poudriere would do the hard work.
I agree with you, but that you I think require a PEP describing this (e.g., https://peps.python.org/pep-0513/). After that one needs to modify pip, setuptools, maturin, uv, and likely other code. It isn't straight forward. While I can raise the PRs for those tools, in fact I have done fixes for maturin on FreeBSD, the PEP stuff and formalization needs a formal process by someone from ?? and then I can approach to downstream. This is separate discussion I'd like to raise, but don't know with who. Comment Actions While this sounds complex, I think this would be just a several lines of code in a correct place. If you don't want to open a PEP for this (which also shouldn't as complex as manylinux one) we could at least patch this in our Python ports.
Hmm, why so much places should be touched? Everything should start working out of the box when we get Python to use proper tags, no? Comment Actions Because there are several build tools for Python wheels. Not all are written in Python, maturin is written in Rust. Comment Actions @arrowd I have now simplified the patch to work out of the box. It requires a subsequent patch (already prepared) for ports (seven of them) which overwrite those commands. The collection logic can fully live in poudriere: https://github.com/michael-o/poudriere-python-wheels/pull/1/files#diff-0afccccb8f3a6069f1dad4ec5c98125d10435243f7848807d2bcf65e00e88282R20-R34 This needs an exp-run of course. Comment Actions This looks OK to me, but I'm not a python hat wearer.
Comment Actions You mean you are not a Python hat wearer, don't you? I will request an exp-run.
Comment Actions Not happening. We already build artefacts for ports with non-standard flags and such, which are inappropriate to share to PyPI. USES=distutils is going away because upstream is removing support for this entirely. It had been deprecated for quite some time but they are finally pulling the switch. Comment Actions I see, alternatively, the FreeBSD project could host them as a separate index in parallel to packages. That should be feasable. But as said, this is a separate discussion outside of this review.
I know, this is why I said that with PEP 517 wheels already there, they just need to be in a distinct, explicit place for consumption. Can you explain why this change needs revision? ATM, I don't see a way to make it even simple than this. Again, I don't expect you do upload anthing to PyPI. It is just a possible usecaes. I am happy if people can host their own indexes with this. Comment Actions From a meta-/framework perspective, building wheels from USE_PYTHON=distutils (ie direct setup.py invocation) is not reliable unless you don't pass any flags or variables at all. This stems from setuptools themselves continuing to deal with a littany of legacy code and design which is frustrating to rid for modernity and maintenance sakes. Trying to build a binary wheel from built artefacts in this case may result in those artefacts getting wrongly rebuilt. (I had some local modifications where USE_PYTHON=distutils only built binary wheels and eliminated its do-install in favour of the PEP-517 do-install.) For those ports that still have a setup.py without a corresponding or replacement pyproject.toml that doesn't need to have flags or variables passed into configure or build, PEP-517 already provides that setuptools is implicitly the build backend. Those ports can switch to USE_PYTHON=pep517. For ports that need to pass flags and variables, setuptools still doesn't support config settings passed from a build frontend like devel/py-build. Comment Actions Re-tagging is not necessary as they are already correct. Not every wheel will have specific platform tags, in fact most if not all ports specifying NO_ARCH will have py3-none-any which allows consumption on anything with any implementation/distribution. Patch levels are not part of specific platform tags, only minor releases, like cp312-cp312-freebsd_14_3_release_amd64 (denotes compatibility with CPython >= 3.12 on 14.3-RELEASE amd64), cp37-abi3-freebsd_16_0_current_amd64 (CPython >= 3.7 ABI 3 on -CURRENT amd64). (If we enable building ports under other implementations like lang/rustpython, that's another can of worms entirely…) Note that 3.13 and later have differing ABIs wrt free-threaded mode or not, like cp313-cp313t-freebsd_16_0_current_amd64. Changes to this system will almost certainly require a PEP or a discussion on their forums. Comment Actions I am only retagging the wheels with native code, not those with any platform tag. The patch release is, unfortunately, part of the tag: sysconfig.get_platform()/pip debug --verbose and you'll see. Example: scipy-1.15.3-cp310-cp310-freebsd_13_5_release_p5_amd64.whl to be usable by all 13.5-RELEASE-pX I need to retag to a multiplatform: https://github.com/michael-o/poudriere-python-wheels/pull/1/files#diff-e6accb6936f8964cc7cb3f78b81bfdf7c9883fdb20a1e297e93a0cbf11a11caeR43-R72
Yes, correct. I take @arrowd's statement for face value that everything build on a major version runs on any minor and patch version of that. So the scipy wheel should only be scipy-1.15.3-cp310-cp310-freebsd_13_amd64.whl although I don't know whether it can be compiled on 13.5 and run older minor versions. Built by uv: $ find uv-cache/sdists* -name *.whl uv-cache/sdists-v9/index/50675900bad406b9/scipy/1.15.3/Rhv5O1fFW5kafLR8t2Qjz/scipy-1.15.3-cp310-cp310-freebsd_13_5_release_p5_amd64.whl uv-cache/sdists-v9/index/50675900bad406b9/backports-datetime-fromisoformat/2.0.3/VYGvOgUTu1iuMOAt8OjQy/backports_datetime_fromisoformat-2.0.3-cp310-cp310-freebsd_13_5_release_p5_amd64.whl uv-cache/sdists-v9/index/50675900bad406b9/numpy/2.2.6/fZmiN98zV_54l0Ra4mdJ9/numpy-2.2.6-cp310-cp310-freebsd_13_5_release_p5_amd64.whl cannot be used on 13.5-RELEASE-p6. My wheels from poudriere after retagging: https://gist.github.com/michael-o/b79c824940017632d8301de0c28aa364 I will raise this discussion on our mailing lists after we have settled this one. Rebuilding wheels for patch versions would be insane. Comment Actions It can't generally. But this is not a problem, because the support window between two consecutive minor releases is usually tight and the official cluster always builds packages on the lowest minor version supported. Comment Actions This basically means that one must strictly observe to provide wheels with an older minor version or incorporate the minor version into the wheel's platform tag. Comment Actions I've put a couple of hours into understanding your point and the problem. Yes, I acknowledge it and understand that we cannot solve this problem unless the port fully moves to PEP 517 with a reasonable backend which passes flags or env vars. bdist_wheel can be unpredictable and will ignore everything passed to setup.py build. Ca. 50% of the ports still use distutils which is a shame. diff --git a/Mk/Uses/python.mk b/Mk/Uses/python.mk index 6b618d4b2562..c4f698b6602c 100644 --- a/Mk/Uses/python.mk +++ b/Mk/Uses/python.mk @@ -188,6 +188,10 @@ # - Pass this command to distutils on build stage. # default: build # +# PYDISTUTILS_BUILD_WHEEL +# - Define to build wheel with distutils on build wheel stage. +# default: <not defined> +# # PYDISTUTILS_BUILD_WHEEL_TARGET # - Pass this command to distutils on build wheel stage. # default: bdist_wheel @@ -206,7 +210,7 @@ # # PYDISTUTILS_BUILDWHEELARGS # - Arguments to build wheel with distutils. -# default: --dist-dir ${WRKDIR}/whl +# default: --skip-build --dist-dir ${WRKDIR}/whl # # PYDISTUTILS_INSTALLARGS # - Arguments to install with distutils. @@ -751,7 +755,7 @@ PYDISTUTILS_SETUP?= -c \ exec(compile(open(__file__, 'rb').read().replace(b'\\r\\n', b'\\n'), __file__, 'exec'))" PYDISTUTILS_CONFIGUREARGS?= # empty PYDISTUTILS_BUILDARGS?= # empty -PYDISTUTILS_BUILDWHEELARGS?= --dist-dir ${WRKDIR}/whl +PYDISTUTILS_BUILDWHEELARGS?= --skip-build --dist-dir ${WRKDIR}/whl PYDISTUTILS_INSTALLARGS?= -c -O1 --prefix=${PREFIX} . if defined(_PYTHON_FEATURE_DISTUTILS) . if !defined(PYDISTUTILS_INSTALLNOSINGLE) @@ -972,7 +976,9 @@ do-configure: . if !target(do-build) do-build: @(cd ${BUILD_WRKSRC}; ${SETENVI} ${WRK_ENV} ${MAKE_ENV} ${PYTHON_CMD} ${PYDISTUTILS_SETUP} ${PYDISTUTILS_BUILD_TARGET} ${PYDISTUTILS_BUILDARGS}) +. if defined(PYDISTUTILS_BUILD_WHEEL) @(cd ${BUILD_WRKSRC}; ${SETENVI} ${WRK_ENV} ${MAKE_ENV} ${PYTHON_CMD} ${PYDISTUTILS_SETUP} ${PYDISTUTILS_BUILD_WHEEL_TARGET} ${PYDISTUTILS_BUILDWHEELARGS}) +. endif . endif . if !target(do-install) PYDISTUTILS_BUILD_WHEEL can be accompanied with a warning about the possible erratic build behavior. Comment Actions Just experienced the inconsistency with distutils myself: with --skip-build: # tar -tzf PyYAML-6.0.3-cp310-cp310-freebsd_13_5_release_p4_amd64.freebsd_13_5_release_p5_amd64.freebsd_13_5_release_p6_amd64.whl yaml/_yaml.cpython-310.so PyYAML-6.0.3.dist-info/LICENSE PyYAML-6.0.3.dist-info/METADATA PyYAML-6.0.3.dist-info/WHEEL PyYAML-6.0.3.dist-info/top_level.txt PyYAML-6.0.3.dist-info/RECORD without the flag: # tar -tzf PyYAML-6.0.3-cp310-cp310-freebsd_13_5_release_p4_amd64.freebsd_13_5_release_p5_amd64.freebsd_13_5_release_p6_amd64.whl _yaml/__init__.py yaml/__init__.py yaml/_yaml.cpython-310.so yaml/composer.py yaml/constructor.py yaml/cyaml.py yaml/dumper.py yaml/emitter.py yaml/error.py yaml/events.py yaml/loader.py yaml/nodes.py yaml/parser.py yaml/reader.py yaml/representer.py yaml/resolver.py yaml/scanner.py yaml/serializer.py yaml/tokens.py PyYAML-6.0.3.dist-info/LICENSE PyYAML-6.0.3.dist-info/METADATA PyYAML-6.0.3.dist-info/WHEEL PyYAML-6.0.3.dist-info/top_level.txt PyYAML-6.0.3.dist-info/RECORD Removed the flag in the patch. Comment Actions @vishwin Have you had time to check the updated patch? The distutils approach is now behind a switch, being an opt-in. | ||||||||||||||||||||||||||||||||||||||||||||