Rebase, and

• Add USE_PYTHON=optsuffix to add a -pyXX suffix if the python version is
• Mark ports that only build with python 2.7.
• Add flavors, concurrent, optsuffix, and ${FLAVOR} where applicable.

I haven't tested this and so far have only read this brief overview of how this will work, but I would go with bdrewery and have flavours as py27, py34, py35, py36...

In the ports tree the default flavour list would be generated to match DEFAULT_VERSIONS of python and python3. For a local pourdriere build some people may want multiple pkg versions to be available. This should be possible by setting a custom flavours list in make.conf

In my poudriere builds I want flavours to be py27, py35, py36. I might set DEFAULT_VERSION=python3=3.5 but I also want to install python modules for 3.6 so I can test them before making 3.6 my default version.

Currently I often manually build different flavours by setting PYTHON_VERSION during a ports tree build, I expect this will continue to work.

While a little off topic, it's related to some changes you are making, lang/cython should be renamed to lang/py-cython as it is a python module that can be installed in multiple flavours.

Thanks for this work, it will be nice to finally get rid of the conflicting py3- ports with multiple dependencies.

devel/pyrex is similar to cython, it should be renamed, it has a misspelled PKGNAMEPREFIX= ${PYTHONPKGNAMEPREFIX} setting that prevents concurrent installs.

I 100% agree with @mat on the number of flavors, we should only have 2, otherwise we will end up adding flavors for every single version of perl, ruby, python, etc. The repository would become gigantic etc.

Having one flavor for python2 and one for python3 is way good enough, and a huge step for users. We can still add others later if needed.

Some further musings ....

While it may be a few more years, python 2.7 will be deprecated and dropped, at which time flavours would want to use py36, py37.... and maybe later py38, py40

• Can/should pypy and pypy3 be included in flavours? Not sure if ports can currently install/build packages for pypy.

• devel/pyrex is similar to lang/cython - it should be renamed to py-pyrex, it currently has a typo that stops concurrent installs - PKGNAMEPREFIX= ${PYTHONPKGNAMEPREFIX}

• Should variables be renamed to reflect being python flavours - eg. PYFLAVOUR ? ssl flavours anyone? QT flavours?

• Can USE_PYTHON=concurrent be enabled as the default rather than manually enabled in each port? This may mean noconcurrent being available.

• Probably related to the multiple flavours list -

Can a port that is built against a specific python version, have a dependancy that does not match the listed flavour versions?

An example is graphics/blender, it only supports being built against a single version of python. Many addons use numpy, so it would be nice to add numpy as a dependancy, while currently this needs to be done manually. If the user sets their default as python3=3.6 and blender is built against 3.5 can py35-numpy be built as a dependency?

• Probably an idea for a later improvement - Can PYTHON_PKGNAMEPREFIX as a pkg dep prefix be assumed?

RUN_DEPENDS= ${PYTHON_PKGNAMEPREFIX}dateutil>=2.1:devel/py-dateutil
becomes -
RUN_DEPENDS= dateutil>=2.1:devel/py-dateutil@${FLAVOR}

In D12464#264067, @bapt wrote:

I 100% agree with @mat on the number of flavors, we should only have 2, otherwise we will end up adding flavors for every single version of perl, ruby, python, etc. The repository would become gigantic etc.

Having one flavor for python2 and one for python3 is way good enough, and a huge step for users. We can still add others later if needed.

I disagree. I think we've been doing python (and the others you list) packages wrong forever. The only argument against providing all of the packages is build time which is purely our problem and not the User's. Python ports generally build fast anyway. There's a lot of room for improvement in build times still.

Consider that we announce the default python3 version will change from 3.6 to 3.7. If we are only providing py36 packages then users have no real way of testing their scripts with py37 without building their own package set. By providing the packages they can actually test. There's no real technical reason a user cannot have both python36 and python37 full sets installed and use both for development/testing/whatever.
For python perhaps we only really need to provide leaf packages and dependencies for those packages *as needed* and not even have flavors. Users can use virtualenv/pip to get whatever versions they need. And yet there is a strong demand for FLAVORS, especially for python. So if we only provide the bare minimum I don't think we'll meet the demand.

I would generally agree we can add more later but the naming becomes a problem. If we add py34 py35 py36 later now we have python3 and py3* names, so perhaps we should name these py2 and py3 for now?

Or we should ask the users what they expect on the mailing lists.

In D12464#264053, @FreeBSD_ShaneWare.Biz wrote:

devel/pyrex is similar to cython, it should be renamed, it has a misspelled PKGNAMEPREFIX= ${PYTHONPKGNAMEPREFIX} setting that prevents concurrent installs.

Yes, and there are about 140 ports that have a PYTHON_PKGNAMEPREFIX that do not have a py- prefix in their directory name. This is unrelated and is left as an exercise to the reader.

Just want to make sure, we can set FLAVORS in a individual port, for generating packages against the required python versions, right? For example, we may need py35-numpy for graphics/blender.
Also, can we set BUILD_ALL_PYTHON_FLAVORS in a individual port, if we need it have pre-built packages for all supported python versions in the ports tree?
For example, I see this patch removes:

databases/py34-gdbm
databases/py34-sqlite3
databases/py35-gdbm
databases/py35-sqlite3
databases/py36-gdbm
databases/py36-sqlite3
x11-toolkits/py34-tkinter
x11-toolkits/py35-tkinter
x11-toolkits/py36-tkinter

However, I feel that they are still necessary (to have pre-built packages), they are the official python modules but separated to other ports by us.

In D12464#276045, @lwhsu wrote:

Just want to make sure, we can set FLAVORS in a individual port, for generating packages against the required python versions, right? For example, we may need py35-numpy for graphics/blender.

We can, yes. At the moment though, blender does not depend on numpy, so I could not have guessed that it would have been nice for it to have numpy.

Also, can we set BUILD_ALL_PYTHON_FLAVORS in a individual port, if we need it have pre-built packages for all supported python versions in the ports tree?

BUILD_ALL_PYTHON_FLAVORS is a GLOBAL knob, one that you put in /etc/make.conf. Also, it is a tentative knob, it may end up being reversed and being called BUILD_ONLY_DEFAULT_PYTHON_FLAVORS.

For example, I see this patch removes:

databases/py34-gdbm
databases/py34-sqlite3
databases/py35-gdbm
databases/py35-sqlite3
databases/py36-gdbm
databases/py36-sqlite3
x11-toolkits/py34-tkinter
x11-toolkits/py35-tkinter
x11-toolkits/py36-tkinter

However, I feel that they are still necessary (to have pre-built packages), they are the official python modules but separated to other ports by us.

They are not, databases/py-gdbm, for example, can build all of these versions directly.

In D12464#276665, @mat wrote:

In D12464#276045, @lwhsu wrote:

Also, can we set BUILD_ALL_PYTHON_FLAVORS in a individual port, if we need it have pre-built packages for all supported python versions in the ports tree?

BUILD_ALL_PYTHON_FLAVORS is a GLOBAL knob, one that you put in /etc/make.conf. Also, it is a tentative knob, it may end up being reversed and being called BUILD_ONLY_DEFAULT_PYTHON_FLAVORS.

For example, I see this patch removes:

databases/py34-gdbm
databases/py34-sqlite3
databases/py35-gdbm
databases/py35-sqlite3
databases/py36-gdbm
databases/py36-sqlite3
x11-toolkits/py34-tkinter
x11-toolkits/py35-tkinter
x11-toolkits/py36-tkinter

However, I feel that they are still necessary (to have pre-built packages), they are the official python modules but separated to other ports by us.

They are not, databases/py-gdbm, for example, can build all of these versions directly.

That's my question, I want databases/py-gdbm automatically generate packages for all supported python versions since it is originally part of lang/python*.
Is there any other way better than manually maintaining FLAVORS=py27 py34 py35 py36 in its Makefile?
I am not sure using FLAVORS= ${_ALL_PYTHON_FLAVORS} is good, _ALL_PYTHON_FLAVORS looks an internal variable in python.mk.
I also don't want to keep one port directory databases/py*-gdbm for each supported python version.
Setting BUILD_ONLY_DEFAULT_PYTHON_FLAVORS might be too overkill, for example, we may need py27 (pythno2 default), py35 (for blender), py36 (python3 default) of py-numpy but not py34.

In D12464#276665, @mat wrote:

In D12464#276045, @lwhsu wrote:

Just want to make sure, we can set FLAVORS in a individual port, for generating packages against the required python versions, right? For example, we may need py35-numpy for graphics/blender.

We can, yes. At the moment though, blender does not depend on numpy, so I could not have guessed that it would have been nice for it to have numpy.

So far there hasn't been a way to add numpy as a dependancy for blender, only py27-numpy builds, so it has been left as a manual step for the user to do. It should be a dependancy as the official builds include it and some addons rely on it.

In D12464#276882, @lwhsu wrote:

That's my question, I want databases/py-gdbm automatically generate packages for all supported python versions since it is originally part of lang/python*.
Is there any other way better than manually maintaining FLAVORS=py27 py34 py35 py36 in its Makefile?
I am not sure using FLAVORS= ${_ALL_PYTHON_FLAVORS} is good, _ALL_PYTHON_FLAVORS looks an internal variable in python.mk.
I also don't want to keep one port directory databases/py*-gdbm for each supported python version.
Setting BUILD_ONLY_DEFAULT_PYTHON_FLAVORS might be too overkill, for example, we may need py27 (pythno2 default), py35 (for blender), py36 (python3 default) of py-numpy but not py34.

The thing is, nobody will care if there is a py34-numpy, but people will complain if there is no py35-numpy. So I reversed the knob, and now it is BUILD_DEFAULT_PYTHON_FLAVORS, so it build everything, unless you only want the default ones.
Also, if you only want the default ones, you will not care that there is no py34-gdbm because, well, it is not the default, and you don't want it anyway.

In D12464#277008, @mat wrote:

In D12464#276882, @lwhsu wrote:

Setting BUILD_ONLY_DEFAULT_PYTHON_FLAVORS might be too overkill, for example, we may need py27 (pythno2 default), py35 (for blender), py36 (python3 default) of py-numpy but not py34.

Sorry that I was meaning BUILD_ALL_PYTHON_FLAVORS here, although you seem to have better solution. :-)

The thing is, nobody will care if there is a py34-numpy, but people will complain if there is no py35-numpy. So I reversed the knob, and now it is BUILD_DEFAULT_PYTHON_FLAVORS, so it build everything, unless you only want the default ones.
Also, if you only want the default ones, you will not care that there is no py34-gdbm because, well, it is not the default, and you don't want it anyway.

I just read the code so just want to check if I understand it correctly:

• Now ports tree will generate packages for all supported python flavors (py27, py34, py35, py36) for all py-* ports by default
• If you set BUILD_DEFAULT_PYTHON_FLAVORS, you will only have py27 and py36 packages.

Right?

I would say I like this, but you have mentioned "there will be 10k packages to (re)build" and I think it's a good point.
Just building packages for default python version is acceptable to me.
And that's why I wanted to check if we can specify FLAVORS and BUILD_ALL_PYTHON_FLAVORS (or something like that) for some special ports' Makefile

Maybe we can have something like this:

• By default, for all ports, we build packages for default python versions (py27 and py36)
• For some ports need packages for non-default python versions (e.g. blender), We set FLAVORS=py27 py35 py36 (or FLAVORS+=py35 or FLOAVORS_EXTRA=py35 ?)
• For some ports need packages for all supported python versions (e.g. py-gdbm), We set FLAVORS= ${ALL_PYTHON_FLAVORS}

Just my thoughts, I like your current patch and thanks for doing this! That does make porters' life much easier.

In D12464#277009, @lwhsu wrote:

In D12464#277008, @mat wrote:

In D12464#276882, @lwhsu wrote:

Setting BUILD_ONLY_DEFAULT_PYTHON_FLAVORS might be too overkill, for example, we may need py27 (pythno2 default), py35 (for blender), py36 (python3 default) of py-numpy but not py34.

Sorry that I was meaning BUILD_ALL_PYTHON_FLAVORS here, although you seem to have better solution. :-)

The thing is, nobody will care if there is a py34-numpy, but people will complain if there is no py35-numpy. So I reversed the knob, and now it is BUILD_DEFAULT_PYTHON_FLAVORS, so it build everything, unless you only want the default ones.
Also, if you only want the default ones, you will not care that there is no py34-gdbm because, well, it is not the default, and you don't want it anyway.

I just read the code so just want to check if I understand it correctly:

• Now ports tree will generate packages for all supported python flavors (py27, py34, py35, py36) for all py-* ports by default
• If you set BUILD_DEFAULT_PYTHON_FLAVORS, you will only have py27 and py36 packages.

Right?

I would say I like this, but you have mentioned "there will be 10k packages to (re)build" and I think it's a good point.

I am not sure it is a good point, it is a technical detail, but there is no real reason aside from that to not build them all.

Just building packages for default python version is acceptable to me.
And that's why I wanted to check if we can specify FLAVORS and BUILD_ALL_PYTHON_FLAVORS (or something like that) for some special ports' Makefile

You simply set FLAVORS. The "automagic" way it is done for Python ports is special, most ports using flavors will define FLAVORS in their Makefile directly. (Say, editors/vim would have FLAVORS= normal lite)

Maybe we can have something like this:

• By default, for all ports, we build packages for default python versions (py27 and py36)
• For some ports need packages for non-default python versions (e.g. blender), We set FLAVORS=py27 py35 py36 (or FLAVORS+=py35 or FLOAVORS_EXTRA=py35 ?)

I would like to say to simply set FLAVORS to the versions we want. But it could be a pain to maintain when the default version change.

• For some ports need packages for all supported python versions (e.g. py-gdbm), We set FLAVORS= ${ALL_PYTHON_FLAVORS}

For these, the list only changes when an old Python version is removed, or when a new Python version is added, a bit like it was before, so I think that having a static list is perfectly fine.

Just my thoughts, I like your current patch and thanks for doing this! That does make porters' life much easier.

In D12464#277503, @bdrewery wrote:

In D12464#277359, @mat wrote:

• Remove the extra flavors mess and add a USE_PYTHON=allflavors.

Yes perfect. I give up trying to review this thing, it's too massive. So if exp-run works, and you literally only do 'USE_PYTHON=allflavors' rather than 'EXTRA_PYTHON_FLAVORS+=py35', then ship it.

It really is 99% adding @${FLAVOR} to depends lines :-)