Page MenuHomeFreeBSD

Python FLAVORS
ClosedPublic

Authored by mat on Sep 22 2017, 2:56 PM.

Details

Summary

What this does is, for all ports that use USE_PYTHON=distutils, autogenerate FLAVORS with python2 and/or python3 depending on python support.
There is a USE_PYTHON=flavors added to force generating flavors for python ports that do not use distutils but still are python.

Diff Detail

Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 12068
Build 12375: arc lint + arc unit

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
mat added a comment.Oct 12 2017, 5:09 PM

Rebase, and

  • Add USE_PYTHON=optsuffix to add a -pyXX suffix if the python version is
  • Mark ports that only build with python 2.7.
  • Add flavors, concurrent, optsuffix, and ${FLAVOR} where applicable.
mat updated this revision to Diff 33959.Oct 13 2017, 4:07 PM

more fixes.

mat updated this revision to Diff 33960.Oct 13 2017, 4:21 PM
  • now, poudriere does not bomb out
bdrewery requested changes to this revision.Oct 13 2017, 4:40 PM
bdrewery added a subscriber: bdrewery.
bdrewery added inline comments.
Mk/Uses/python.mk
407

So I think we need FLAVORS to be py27 py34 py35 py36, etc. Right now I only see 'python3' when there's really 3 flavors of python3. This brings the most benefit of flavors by providing all possible python packages.

This revision now requires changes to proceed.Oct 13 2017, 4:40 PM
mat added inline comments.Oct 13 2017, 4:50 PM
Mk/Uses/python.mk
407

I disagree. If someone wants to have a non default python 3, then one can change DEFAULT_VERSIONS+=python3=3.4.

I really do not see what this adds. Except about 10k packages to (re)build.

Changing like you say makes it impossible for a port to say "I am using python3, I don't care which flavor, and I do not want to be poluted with flavors."

antoine added inline comments.
Mk/Uses/python.mk
407

I thune i don't want 4 versions of python ports, there is already default_versions += python3=

mat updated this revision to Diff 34026.Oct 16 2017, 12:11 PM

rebase & update.

mat updated this revision to Diff 34060.Oct 17 2017, 2:25 PM
  • More fixes.
  • Rename devel/py27-setuptools to devel/py-setuptools.
  • some more fixes.
  • Add USE_PYTHON=cython.
mat updated this revision to Diff 34066.Oct 17 2017, 4:34 PM
  • more
  • even more
mat updated this revision to Diff 34068.Oct 17 2017, 4:59 PM
  • 'nother fix
mat updated this revision to Diff 34072.Oct 17 2017, 6:55 PM

Rebase.

I haven't tested this and so far have only read this brief overview of how this will work, but I would go with bdrewery and have flavours as py27, py34, py35, py36...

In the ports tree the default flavour list would be generated to match DEFAULT_VERSIONS of python and python3. For a local pourdriere build some people may want multiple pkg versions to be available. This should be possible by setting a custom flavours list in make.conf

In my poudriere builds I want flavours to be py27, py35, py36. I might set DEFAULT_VERSION=python3=3.5 but I also want to install python modules for 3.6 so I can test them before making 3.6 my default version.

Currently I often manually build different flavours by setting PYTHON_VERSION during a ports tree build, I expect this will continue to work.

While a little off topic, it's related to some changes you are making, lang/cython should be renamed to lang/py-cython as it is a python module that can be installed in multiple flavours.

Thanks for this work, it will be nice to finally get rid of the conflicting py3- ports with multiple dependencies.

devel/pyrex is similar to cython, it should be renamed, it has a misspelled PKGNAMEPREFIX= ${PYTHONPKGNAMEPREFIX} setting that prevents concurrent installs.

bapt added a subscriber: bapt.Oct 19 2017, 5:15 AM

I 100% agree with @mat on the number of flavors, we should only have 2, otherwise we will end up adding flavors for every single version of perl, ruby, python, etc. The repository would become gigantic etc.

Having one flavor for python2 and one for python3 is way good enough, and a huge step for users. We can still add others later if needed.

Some further musings ....

While it may be a few more years, python 2.7 will be deprecated and dropped, at which time flavours would want to use py36, py37.... and maybe later py38, py40

  • Can/should pypy and pypy3 be included in flavours? Not sure if ports can currently install/build packages for pypy.
  • devel/pyrex is similar to lang/cython - it should be renamed to py-pyrex, it currently has a typo that stops concurrent installs - PKGNAMEPREFIX= ${PYTHONPKGNAMEPREFIX}
  • Should variables be renamed to reflect being python flavours - eg. PYFLAVOUR ? ssl flavours anyone? QT flavours?
  • Can USE_PYTHON=concurrent be enabled as the default rather than manually enabled in each port? This may mean noconcurrent being available.
  • Probably related to the multiple flavours list -

Can a port that is built against a specific python version, have a dependancy that does not match the listed flavour versions?

An example is graphics/blender, it only supports being built against a single version of python. Many addons use numpy, so it would be nice to add numpy as a dependancy, while currently this needs to be done manually. If the user sets their default as python3=3.6 and blender is built against 3.5 can py35-numpy be built as a dependency?

  • Probably an idea for a later improvement - Can PYTHON_PKGNAMEPREFIX as a pkg dep prefix be assumed?

RUN_DEPENDS= ${PYTHON_PKGNAMEPREFIX}dateutil>=2.1:devel/py-dateutil
becomes -
RUN_DEPENDS= dateutil>=2.1:devel/py-dateutil@${FLAVOR}

Mk/Uses/python.mk
74

cython_run should also be here.

accessibility/py-atspi/Makefile
24

Could the USE_GNOME setting of pygobject3/py3gobject3 be combined to USE_GNOME=pygobject3 and have the framework do the choosing of the py2/py3 version? 17 ports use py3gobject3 and 31 use pygobject3

devel/py-dbus/Makefile
15

Can this be automated in the framework?

if PYTHON_VERSION>3 and exists(pkg-plist.py3) then PLIST=pkg-plist.py3

Although - you have python3_PLIST - that should also be used here?

devel/py-qt5-core/Makefile
9–12

Fairly sure this breakage is sip related, one change I have is setting SIPDIR_REL in Mk/Uses/pyqt.mk to

SIPDIR_REL= share/${PYTHON_PKGNAMEPREFIX}sip/PyQt${_PYQT_VERSION}

which also involves patching the install of py-sip.

12

A common setting that should be in Mk/Uses/pyqt.mk

devel/py-vcversioner/Makefile
5–6

This is not flavours related.

devel/pygobject3-common/Makefile
65

My idea here for supporting concurrent was to have py3-cairo install it's .pc file as py${PYTHON_SUFFIX}cairo.pc - if that is done in py-cairo for every version then the flavour test here can be replaced with one build and run entry.

net/py-netifaces/Makefile
5

Not flavours

textproc/py-MarkupSafe/Makefile
14

Not flavours change

In D12464#264067, @bapt wrote:

I 100% agree with @mat on the number of flavors, we should only have 2, otherwise we will end up adding flavors for every single version of perl, ruby, python, etc. The repository would become gigantic etc.
Having one flavor for python2 and one for python3 is way good enough, and a huge step for users. We can still add others later if needed.

I disagree. I think we've been doing python (and the others you list) packages wrong forever. The only argument against providing all of the packages is build time which is purely our problem and not the User's. Python ports generally build fast anyway. There's a lot of room for improvement in build times still.

Consider that we announce the default python3 version will change from 3.6 to 3.7. If we are only providing py36 packages then users have no real way of testing their scripts with py37 without building their own package set. By providing the packages they can actually test. There's no real technical reason a user cannot have both python36 and python37 full sets installed and use both for development/testing/whatever.
For python perhaps we only really need to provide leaf packages and dependencies for those packages *as needed* and not even have flavors. Users can use virtualenv/pip to get whatever versions they need. And yet there is a strong demand for FLAVORS, especially for python. So if we only provide the bare minimum I don't think we'll meet the demand.

I would generally agree we can add more later but the naming becomes a problem. If we add py34 py35 py36 later now we have python3 and py3* names, so perhaps we should name these py2 and py3 for now?

Or we should ask the users what they expect on the mailing lists.

mat updated this revision to Diff 34187.Oct 20 2017, 4:11 PM
  • Fix py-libpeas@python3.
  • Fix problems post-exp-run.
mat marked 5 inline comments as done.Oct 20 2017, 9:18 PM
mat marked an inline comment as done.Oct 20 2017, 9:27 PM

devel/pyrex is similar to cython, it should be renamed, it has a misspelled PKGNAMEPREFIX= ${PYTHONPKGNAMEPREFIX} setting that prevents concurrent installs.

Yes, and there are about 140 ports that have a PYTHON_PKGNAMEPREFIX that do not have a py- prefix in their directory name. This is unrelated and is left as an exercise to the reader.

devel/py-qt5-core/Makefile
9–12

Like the comment says, it is broken during the package phase. the plist is python 2 only, I'll let kde people fix it when this is done.

12

Tried, ports (one of which is py-qt4-core) using sip explode if you forcefully add --sip <path to sip>

devel/py-vcversioner/Makefile
5–6

mergin py- and py3- ports.

devel/pygobject3-common/Makefile
65

This is unrelated to flavoring python. gnome@ can go and change this after the python flavors get there.

net/py-netifaces/Makefile
5

Merging py- and py3- ports.

textproc/py-MarkupSafe/Makefile
14

merging py- and py3- ports.

mat updated this revision to Diff 34198.Oct 20 2017, 9:30 PM
mat marked an inline comment as done.
  • Really fix www/py-autobahn
  • Add cython_run.
mat updated this revision to Diff 34328.Oct 25 2017, 2:29 PM
  • Fix post-rebase.
mat updated this revision to Diff 34402.Oct 27 2017, 4:20 PM
  • post exp-run fixes.
mat updated this revision to Diff 34456.Oct 30 2017, 11:40 AM

rebase & squash.

mat updated this revision to Diff 34861.Nov 6 2017, 4:33 PM

rebase

mat updated this revision to Diff 35667.Nov 23 2017, 4:49 PM
  • Update CHANGES
  • Add a knob to build all python flavors, default to only the default
  • Removing USES=python:3 fixes.
  • rebase
lwhsu added a subscriber: lwhsu.Nov 24 2017, 3:56 PM
lwhsu added a comment.Nov 25 2017, 5:51 PM

Just want to make sure, we can set FLAVORS in a individual port, for generating packages against the required python versions, right? For example, we may need py35-numpy for graphics/blender.
Also, can we set BUILD_ALL_PYTHON_FLAVORS in a individual port, if we need it have pre-built packages for all supported python versions in the ports tree?
For example, I see this patch removes:

databases/py34-gdbm
databases/py34-sqlite3
databases/py35-gdbm
databases/py35-sqlite3
databases/py36-gdbm
databases/py36-sqlite3
x11-toolkits/py34-tkinter
x11-toolkits/py35-tkinter
x11-toolkits/py36-tkinter

However, I feel that they are still necessary (to have pre-built packages), they are the official python modules but separated to other ports by us.

mat added a comment.Nov 27 2017, 3:48 PM

Just want to make sure, we can set FLAVORS in a individual port, for generating packages against the required python versions, right? For example, we may need py35-numpy for graphics/blender.

We can, yes. At the moment though, blender does not depend on numpy, so I could not have guessed that it would have been nice for it to have numpy.

Also, can we set BUILD_ALL_PYTHON_FLAVORS in a individual port, if we need it have pre-built packages for all supported python versions in the ports tree?

BUILD_ALL_PYTHON_FLAVORS is a GLOBAL knob, one that you put in /etc/make.conf. Also, it is a tentative knob, it may end up being reversed and being called BUILD_ONLY_DEFAULT_PYTHON_FLAVORS.

For example, I see this patch removes:

databases/py34-gdbm
databases/py34-sqlite3
databases/py35-gdbm
databases/py35-sqlite3
databases/py36-gdbm
databases/py36-sqlite3
x11-toolkits/py34-tkinter
x11-toolkits/py35-tkinter
x11-toolkits/py36-tkinter

However, I feel that they are still necessary (to have pre-built packages), they are the official python modules but separated to other ports by us.

They are not, databases/py-gdbm, for example, can build all of these versions directly.

mat updated this revision to Diff 35877.Nov 27 2017, 5:53 PM
  • fixup.
  • Catch up.
In D12464#276665, @mat wrote:

Also, can we set BUILD_ALL_PYTHON_FLAVORS in a individual port, if we need it have pre-built packages for all supported python versions in the ports tree?

BUILD_ALL_PYTHON_FLAVORS is a GLOBAL knob, one that you put in /etc/make.conf. Also, it is a tentative knob, it may end up being reversed and being called BUILD_ONLY_DEFAULT_PYTHON_FLAVORS.

For example, I see this patch removes:

databases/py34-gdbm
databases/py34-sqlite3
databases/py35-gdbm
databases/py35-sqlite3
databases/py36-gdbm
databases/py36-sqlite3
x11-toolkits/py34-tkinter
x11-toolkits/py35-tkinter
x11-toolkits/py36-tkinter

However, I feel that they are still necessary (to have pre-built packages), they are the official python modules but separated to other ports by us.

They are not, databases/py-gdbm, for example, can build all of these versions directly.

That's my question, I want databases/py-gdbm automatically generate packages for all supported python versions since it is originally part of lang/python*.
Is there any other way better than manually maintaining FLAVORS=py27 py34 py35 py36 in its Makefile?
I am not sure using FLAVORS= ${_ALL_PYTHON_FLAVORS} is good, _ALL_PYTHON_FLAVORS looks an internal variable in python.mk.
I also don't want to keep one port directory databases/py*-gdbm for each supported python version.
Setting BUILD_ONLY_DEFAULT_PYTHON_FLAVORS might be too overkill, for example, we may need py27 (pythno2 default), py35 (for blender), py36 (python3 default) of py-numpy but not py34.

In D12464#276665, @mat wrote:

Just want to make sure, we can set FLAVORS in a individual port, for generating packages against the required python versions, right? For example, we may need py35-numpy for graphics/blender.

We can, yes. At the moment though, blender does not depend on numpy, so I could not have guessed that it would have been nice for it to have numpy.

So far there hasn't been a way to add numpy as a dependancy for blender, only py27-numpy builds, so it has been left as a manual step for the user to do. It should be a dependancy as the official builds include it and some addons rely on it.

mat updated this revision to Diff 35917.Nov 28 2017, 1:03 PM

rebase.

mat added a comment.Nov 28 2017, 1:19 PM

That's my question, I want databases/py-gdbm automatically generate packages for all supported python versions since it is originally part of lang/python*.
Is there any other way better than manually maintaining FLAVORS=py27 py34 py35 py36 in its Makefile?
I am not sure using FLAVORS= ${_ALL_PYTHON_FLAVORS} is good, _ALL_PYTHON_FLAVORS looks an internal variable in python.mk.
I also don't want to keep one port directory databases/py*-gdbm for each supported python version.
Setting BUILD_ONLY_DEFAULT_PYTHON_FLAVORS might be too overkill, for example, we may need py27 (pythno2 default), py35 (for blender), py36 (python3 default) of py-numpy but not py34.

The thing is, nobody will care if there is a py34-numpy, but people will complain if there is no py35-numpy. So I reversed the knob, and now it is BUILD_DEFAULT_PYTHON_FLAVORS, so it build everything, unless you only want the default ones.
Also, if you only want the default ones, you will not care that there is no py34-gdbm because, well, it is not the default, and you don't want it anyway.

lwhsu added a comment.Nov 28 2017, 2:01 PM
In D12464#277008, @mat wrote:

Setting BUILD_ONLY_DEFAULT_PYTHON_FLAVORS might be too overkill, for example, we may need py27 (pythno2 default), py35 (for blender), py36 (python3 default) of py-numpy but not py34.

Sorry that I was meaning BUILD_ALL_PYTHON_FLAVORS here, although you seem to have better solution. :-)

The thing is, nobody will care if there is a py34-numpy, but people will complain if there is no py35-numpy. So I reversed the knob, and now it is BUILD_DEFAULT_PYTHON_FLAVORS, so it build everything, unless you only want the default ones.
Also, if you only want the default ones, you will not care that there is no py34-gdbm because, well, it is not the default, and you don't want it anyway.

I just read the code so just want to check if I understand it correctly:

  • Now ports tree will generate packages for all supported python flavors (py27, py34, py35, py36) for all py-* ports by default
  • If you set BUILD_DEFAULT_PYTHON_FLAVORS, you will only have py27 and py36 packages.

Right?

I would say I like this, but you have mentioned "there will be 10k packages to (re)build" and I think it's a good point.
Just building packages for default python version is acceptable to me.
And that's why I wanted to check if we can specify FLAVORS and BUILD_ALL_PYTHON_FLAVORS (or something like that) for some special ports' Makefile

Maybe we can have something like this:

  • By default, for all ports, we build packages for default python versions (py27 and py36)
  • For some ports need packages for non-default python versions (e.g. blender), We set FLAVORS=py27 py35 py36 (or FLAVORS+=py35 or FLOAVORS_EXTRA=py35 ?)
  • For some ports need packages for all supported python versions (e.g. py-gdbm), We set FLAVORS= ${ALL_PYTHON_FLAVORS}

Just my thoughts, I like your current patch and thanks for doing this! That does make porters' life much easier.

mat added a comment.Nov 28 2017, 2:18 PM
In D12464#277008, @mat wrote:

Setting BUILD_ONLY_DEFAULT_PYTHON_FLAVORS might be too overkill, for example, we may need py27 (pythno2 default), py35 (for blender), py36 (python3 default) of py-numpy but not py34.

Sorry that I was meaning BUILD_ALL_PYTHON_FLAVORS here, although you seem to have better solution. :-)

The thing is, nobody will care if there is a py34-numpy, but people will complain if there is no py35-numpy. So I reversed the knob, and now it is BUILD_DEFAULT_PYTHON_FLAVORS, so it build everything, unless you only want the default ones.
Also, if you only want the default ones, you will not care that there is no py34-gdbm because, well, it is not the default, and you don't want it anyway.

I just read the code so just want to check if I understand it correctly:

  • Now ports tree will generate packages for all supported python flavors (py27, py34, py35, py36) for all py-* ports by default
  • If you set BUILD_DEFAULT_PYTHON_FLAVORS, you will only have py27 and py36 packages.

Right?
I would say I like this, but you have mentioned "there will be 10k packages to (re)build" and I think it's a good point.

I am not sure it is a good point, it is a technical detail, but there is no real reason aside from that to not build them all.

Just building packages for default python version is acceptable to me.
And that's why I wanted to check if we can specify FLAVORS and BUILD_ALL_PYTHON_FLAVORS (or something like that) for some special ports' Makefile

You simply set FLAVORS. The "automagic" way it is done for Python ports is special, most ports using flavors will define FLAVORS in their Makefile directly. (Say, editors/vim would have FLAVORS= normal lite)

Maybe we can have something like this:

  • By default, for all ports, we build packages for default python versions (py27 and py36)
  • For some ports need packages for non-default python versions (e.g. blender), We set FLAVORS=py27 py35 py36 (or FLAVORS+=py35 or FLOAVORS_EXTRA=py35 ?)

I would like to say to simply set FLAVORS to the versions we want. But it could be a pain to maintain when the default version change.

  • For some ports need packages for all supported python versions (e.g. py-gdbm), We set FLAVORS= ${ALL_PYTHON_FLAVORS}

For these, the list only changes when an old Python version is removed, or when a new Python version is added, a bit like it was before, so I think that having a static list is perfectly fine.

Just my thoughts, I like your current patch and thanks for doing this! That does make porters' life much easier.

mat updated this revision to Diff 35926.Nov 28 2017, 3:51 PM
  • Only build the default versions by default.
  • Build all the flavors for the modules bundled with Python.
  • Provide an extra flavor variable.
bdrewery added inline comments.Nov 29 2017, 12:03 AM
Mk/Uses/python.mk
421

EXTRA_PYTHON_FLAVORS feels funky. Can't we just append to FLAVORS if it already has a value?
At the same time I really don't care so nevermind.

lang/cython/Makefile
20

This feels like a maintenance problem. I think just using BUILD_ALL_PYTHON_FLAVORS is better here.

math/py-numpy/Makefile
26

This feels like a maintenance problem. I think just using BUILD_ALL_PYTHON_FLAVORS is better here.

mat added inline comments.Nov 29 2017, 11:46 AM
Mk/Uses/python.mk
421

Well, no, you can't really append to flavors, the "automatic" FLAVORS generation is only done if FLAVORS is empty, as to not clobber something more interesting done by the porter.

Maybe it could be part of USE_PYTHON, say:

USE_PYTHON=   extraflavor=py34,py35
lang/cython/Makefile
20

But no, BUILD_ALL_PYTHON_FLAVORS is supposed to be used globally, not per port.

mat added inline comments.Nov 29 2017, 1:07 PM
Mk/Uses/python.mk
421

Thinking about it, I could simply add a:

USE_PYTHON=  allflavors

to get all the possible versions, it would make things a bit easier.

mat updated this revision to Diff 35962.Nov 29 2017, 4:34 PM
  • Remove the extra flavors mess and add a USE_PYTHON=allflavors.
bdrewery accepted this revision.Nov 29 2017, 10:32 PM
In D12464#277359, @mat wrote:
  • Remove the extra flavors mess and add a USE_PYTHON=allflavors.

Yes perfect. I give up trying to review this thing, it's too massive. So if exp-run works, and you literally only do 'USE_PYTHON=allflavors' rather than 'EXTRA_PYTHON_FLAVORS+=py35', then ship it.

This revision is now accepted and ready to land.Nov 29 2017, 10:32 PM
mat added a comment.Nov 30 2017, 9:52 AM
In D12464#277359, @mat wrote:
  • Remove the extra flavors mess and add a USE_PYTHON=allflavors.

Yes perfect. I give up trying to review this thing, it's too massive. So if exp-run works, and you literally only do 'USE_PYTHON=allflavors' rather than 'EXTRA_PYTHON_FLAVORS+=py35', then ship it.

It really is 99% adding @${FLAVOR} to depends lines :-)

This revision was automatically updated to reflect the committed changes.