Simplify py3- slave port handling.

[AMDmi3]

Summary

Here's a common pattern with py3- slave ports:

py-foo/Makefile:

RUN_DEPENDS?=	${PYTHON_PKGNAMEPREFIX}bar>0:devel/py-bar

USES?=		python
USES+=		zip

py3-foo/Makefile:

MASTERDIR=	${.CURDIR}/../py-foo

RUN_DEPENDS=	${PYTHON_PKGNAMEPREFIX}bar>0:devel/py3-bar
USES=		python:3

Since USES and *DEPENDS must be duplicated, this is clumsy and error
prone.

What I propose is to introduce PYTHON_PORTNAMEPREFIX which may be set
by the framework to either py- or py3- and is intended to be used
in *_DEPENDS, and PY3_SLAVEPORT knob which is indended to be set
by py3- slave ports, which does two things:

• set PYTHON_PORTNAMEPREFIX to py3-
• force using of python version 3 like USES=python:3 would do

This simple change remove the need for redefining DEPENDS and USES
in a slave port, effectively reducing (most?) py3- slave ports to
three lines, removes the code duplication and also reduces chandes
needed to master ports to replacing py- with ${PYTHON_PORTNAMEPREFIX}
in depends.

The above example is now simplified to:

py-foo/Makefile:

RUN_DEPENDS=	${PYTHON_PKGNAMEPREFIX}bar>0:devel/${PYTHON_PORTNAMEPREFIX}bar

USES=		python zip

py3-foo/Makefile:

MASTERDIR=	${.CURDIR}/../py-foo

PY3_SLAVEPORT=	yes

The patch also includes a demonstration of this feature for devel/{py,py3}-flake8.

[bdrewery]

Please get more reviews before committing. I'd rather not speak for Python changes overall.

I like removing the duplication - so long as they are actually 100% duplicates.

Also FYI the root cause for needing these py3 slave ports is due to Poudriere not properly handling DEPENDS_ARGS. I am aggressively trying to get that fixed in the next few weeks and into the next Poudriere release. The work to support that feature is complimentary and required for supporting FLAVORS. Once Poudrere is fixed, we can remove slave ports (that were added purely for dependencies) - and once FLAVORS is in we can remove slave ports that were added to provide py3 versions of packages. Really FLAVORS is the more reasonable requirement to bring in before removing all of the slave ports.

[AMDmi3]

I don't think this is related to poudriere and flavors, or that we'll remove py3- slaves anytime soon.

We need two parallel independent sets of modules for python2 and python3 in any case.

[bdrewery]

In D11017#228191, @AMDmi3 wrote:

I don't think this is related to poudriere and flavors, or that we'll remove py3- slaves anytime soon.

We need two parallel independent sets of modules for python2 and python3 in any case.

It is related. FLAVORS will remove the need for py3 slave ports entirely. We would have only devel/py-flake and it would have FLAVORS=py27 py33 py34 ..., or similar. Poudriere would build multiple packages from that 1 port. I was just letting you know the plans and upcoming changes. In the meantime this change is fine.

[AMDmi3]

It is related. FLAVORS will remove the need for py3 slave ports entirely. We would have only devel/py-flake and it would have FLAVORS=py27 py33 py34 ..., or similar. Poudriere would build multiple packages from that 1 port. I was just letting you know the plans and upcoming changes. In the meantime this change is fine.

Again, poudriere is irrelevant as it's just a ports client.

How would a user build py27 variant of py-foo and py36 variant of py-bar?

[bdrewery]

In D11017#228202, @AMDmi3 wrote:

It is related. FLAVORS will remove the need for py3 slave ports entirely. We would have only devel/py-flake and it would have FLAVORS=py27 py33 py34 ..., or similar. Poudriere would build multiple packages from that 1 port. I was just letting you know the plans and upcoming changes. In the meantime this change is fine.

Again, poudriere is irrelevant as it's just a ports client.

It's not irrelevant as many of these slave ports were added specifically because Poudriere did not handle DEPENDS_ARGS properly, and it's only a problem for *package building*. The native framework, portmaster, and portupgrade all handle DEPENDS_ARGS properly and can build both py27- and py33- dependencies already.

How would a user build py27 variant of py-foo and py36 variant of py-bar?

make FLAVOR=py27

The other tools need to learn support for this. In principle there's no need for slave ports in a FLAVORS world. The tools just need to have their own syntax for specifying which flavor of a port they want to build and properly handle upgrading them using the FLAVOR they were built with. The ultimate goal of FLAVORS removing slave ports won't happen overnight, there's a lot of work left still. But the demand has largely been in PACKAGES. Already people can just change the default python they are building with to get the devel/py-flake as py3 if they want and are building locally. But they cannot easily get both py27-flake and py33-flake from packages. That's where the problems are really at.

[AMDmi3]

It's not irrelevant as many of these slave ports were added specifically because Poudriere

It is, because it's the same way if you just forget about poudriere. As long as you want two sets of ports, you need entry point for each of these. Two ports provide two entry points. Or two flavors provide two entry points. So flavors is what eliminates the need for slave ports.

make FLAVOR=py27

A bit of offtopic, but I'm curious. What would just make produce? All possible flavors? Will it be possible to set flavors the way options are set? The latter is required for the ports to remain usable. How will NxM be handled (py27/py35/py36 * pg84/pg93/pg94/pg95/pg96)?

[bdrewery]

In D11017#228204, @AMDmi3 wrote:

It's not irrelevant as many of these slave ports were added specifically because Poudriere

It is, because it's the same way if you just forget about poudriere. As long as you want two sets of ports, you need entry point for each of these. Two ports provide two entry points. Or two flavors provide two entry points. So flavors is what eliminates the need for slave ports.

Nothing is stopping the user from doing something like 'cd /usr/ports/devel/py-flake && make PYTHON_VERSION=33 install'. They can *already* get multiple versions of these without slave ports or FLAVORS. Both are a mere convenience for local builds, but a *necessity* for package building which is Poudriere.

make FLAVOR=py27

A bit of offtopic, but I'm curious. What would just make produce? All possible flavors? Will it be possible to set flavors the way options are set? The latter is required for the ports to remain usable. How will NxM be handled (py27/py35/py36 * pg84/pg93/pg94/pg95/pg96)?

See https://reviews.freebsd.org/D10327

Currently it defaults to the first FLAVOR defined, FLAVOR= ${FLAVORS:[1]}, and dependencies can specify a wanted FLAVOR.

[mat]

Yeah, I don't quite like this like Bryan says, FLAVORS will make all those useless.

[AMDmi3]

Nothing is stopping the user from doing something like 'cd /usr/ports/devel/py-flake && make PYTHON_VERSION=33 install'. They can *already* get multiple versions of these without slave ports or FLAVORS. Both are a mere convenience for local builds, but a *necessity* for package building which is Poudriere.

I don't count this as a proper support because it breaks with the next update.

[koobs]

There's two major things here:

1. Python version provided/forced by framework (replacing USES=python:<something>, via WANTS_META_PORT=3)
2. Dynamic prefix in order to reduce *_DEPENDS duplication

I'm -1 on Python version being set/forced by the framework, particularly for the proposed USES=python:3 value:

• :3 only means depend on the lang/python3 port, which provides the python3 symlink. It has not, does not, will not and cannot not ever mean "supports any 3.x version of Python". This applies to :2 and argument-less USES=python usage too.

Yes there are ports that currently and continue to use these declarations:

a) This makes them no less incorrect in their semantics
b) Some/most can be fixed and should be (there is a plan afoot to do just this)
c) Some can only be fixed to a certain extent (for now) given limitations on the granularity of version declarations in USES=python:<version-spec>.
d) This does not make setting/forcing any single value (via an incorrect method or not) in the framework for any arbitrary set of ports the right thing to do.
e) Improving the granularity of the version-spec would still not change the fact that setting USES via WANTS_META_PORT=3 (USES=python:3) is incorrect.

• The benefits (remove one line) do not outweigh the impacts, all else being equal. Further, one line is removed, but another (PY3_SLAVEPORT) is added.
• The :2, :3 and bare (argument-less) use of USES=python are very likely going to be deprecated in favour of explicit dependencies on lang/python[2,3] and explicit version declarations, given limitations and issues with how they're used (incorrectly or correctly). Adding another consumer of these bits makes it much more difficult to evolve and improve the framework.
• I have the current position also that py3-* ports are temporary workarounds given the lack of multiple-packages-per-origin (Solved in VARIANTS ("flavours/flavors") + poudriere). It concerns me that adding workarounds and single-use features to the Python framework, particularly for consumers that are themselves workarounds, wont be removed and adds more entropy to an environment already difficult to simplify, though progress has been made.
• USES is not really a duplication. At most it could be argued the py3- port supports a subset of that of the base port.

I'm -0 on adding PY3_SLAVEPORT and Dynamic prefixes:

On the downside:

• I don't like that the master/base port needs changes to accommodate this, but changes are already needed for py3- ports at all (USES= -> USES?=)
• Though dependencies are duplicated, the duplication is in a py3- port which itself is a temporary workaround. When the py3- port is removed (come VARIANTS), so is the duplication.

On the "but" side:

• I agree that practical needs (having python 3 ports and their dependencies for our users) outweigh and shouldn't be blocked by incomplete coming-soon things (to a point). In fact, I hope this and other 'workarounds' might add more impetus to prioritize VARIANTS and other improvements.
• The bulk of the 'overhead' in creating py3- ports is copy/paste *_DEPENDS lines and search/replace, though not complex
• Making life easier/quicker for committers is a good thing

If duplication of depends is actually that big an annoyance and time sink, I'm not that against it landing, as long as its not used or relied upon as a mechanism for more workarounds, marked as such, and removed as soon as possible.

I would also like to see PY3_SLAVEPORT and other framework additions commented/marked as TEMPORARY (for py3- ports)

[AMDmi3]

I gotta think about it some more, just a little idea which came to my mind.

You want to specify precisely which python versions some package supports, right? Can't we implement a more complex version syntax, e.g. USES=python:2.7,3.3,3.4-3.5? This way we'll have correct version requirements listed and PY3_SLAVEPORT implementation would still be trivial as it will only remove 2.* from the list.

If that's too complex for now, what do you think about semantics like

PY3_SLAVEPORT=3.3+

Which will also allow to specify python version requirements on per-port basis.

[koobs]

In D11017#228494, @AMDmi3 wrote:

Can't we implement a more complex version syntax, e.g. USES=python:2.7,3.3,3.4-3.5?

This refers to the same thing as (improve) granularity of version declarations in USES=python:<version-spec> in the previous comment.

However, more granular version specification (implemented in framework, set by ports) is not a pre-requisite or necessary for the changes proposed here, and is actually only one part of two things needed (see below).

The issue at hand here is not the values per se, but where and how they are set/scoped.

Version specs do not need to be made more granular (per port) in this change, just not made less correct and/or forced globally (by framework).

This way we'll have correct version requirements listed and PY3_SLAVEPORT implementation would still be trivial as it will only remove 2.* from the list.

Whether specified in USES or PY_SLAVEPORT, the only requirement is that the supported version specification continues to be done only per-port, not by the framework.

The only thing the framework can and should do (which it already does, but which can be improved as the second part/stage of the feature outlined above), is calculate or choose a specific value that is the intersection of available_python_versions and versions_supported_by_port, if their is more than 1. There are also more considerations for the feature(s) outlined here and above that are outside the scope of this change.

Note: there is a distinction between 'i support these versions' (declarative) and 'use this version' (imperative). Unfortunately USES is an imperative sounding term, so the distinction is often lost, and its used that way. There's already been talk about improving this in the python framework to make it easier and more obvious to developers, but that is also outside the scope of this change.

If that's too complex for now, what do you think about semantics like

PY3_SLAVEPORT=3.3+

It's not that it's complex, its that its not a requirement, and has no impact on the primary intended goal of this change (make making py3- ports simpler/faster)

On using <version-spec> as a PY_SLAVEPORT value, again -0.

Moving the known syntax, semantics and use of the same value/mechanism from USES to another variable means another thing for people to learn beyond standard base/sub (master/slave) port creation. I'm reminded of PEP-20's "Special cases aren't special enough to break the rules." and "Although practicality beats purity."

Further, given the net benefit is still only one line, (s/USES/PY3_SLAVEPORT), I still believe the payoff is insufficient.

I did note that doing this would appear to remove the need to s/USES/USES?= in the base (master) port, which is one less change needed.

However, this isn't necessarily a benefit given:

• Allready known and obvious semantics of master/slave port variable overriding, AND
• If correct, would also remove a minor argument for dynamic prefixes in the previous comment, namely, that downside: dynamic prefixes requires changing the master port, mitigation: but master changes are already needed to create py3- ports (s/USES=/USES?=)

Which will also allow to specify python version requirements on per-port basis.

This statement is confusing, could you clarify?

'also allow' appears to imply that they could be set per-port, in addition to something/somewhere else, not that they must be set-per-port, which is the major issue discussed.

Did you not intend this meaning?

[AMDmi3]

Obsoleted by flavors