Ports framework "variants" proof-of-concept

[bapt]

Summary

@koobs suggested that I put this up to review the implementation of variants within the ports framework.

Included in this proof of concept is ports-mgmt/poudriere-variants which will download poudriere from my feature branch.

Very few patches were needed to the ports framework to support variants, as you will see. At this stage, I have only patched the python.mk to support the pyXY- prefixes, but it demonstrates the general principle and power of this implementation.

A diff of the poudriere code can be found here. Major points of interest:

• Packages, and not their origin, become the focus of the output
• Varying levels of output depending on verbosity
• It seems to work quite well!
• Some bits aren't done yet, but aren't needed for this proof-of-concept

A real world example of this in use is provided. It includes an example of devel/py-setuptools with a VARIANTS directive, which solves the issue of having many slave ports to work around issues with the framework.

And last but not least, here is a screencast of poudriere running with various levels of verbosity generating the py-setuptools packages.

Test Plan

The purpose of this review is for you to point out as many pitfalls with this approach as you can, so we can build the most robust and logical solution to the origin variations problem.

[feld]

Is the long term approach to require port Makefiles to be modified to support VARIANTS or do you think it is possible that can be completely handled in the ports framework? Eg, could all Python ports be made VARIANTS-aware simply through Uses/python.mk instead?

[koobs]

In D5563#118764, @feld wrote:

Is the long term approach to require port Makefiles to be modified to support VARIANTS or do you think it is possible that can be completely handled in the ports framework? Eg, could all Python ports be made VARIANTS-aware simply through Uses/python.mk instead?

It can be done at any level, for example:

Per port: As in gmake and gmake-lite (the "lite" variant). Master/Slave ports can be implemented as variants

Python Framework: python.mk creates N pyXY variants based on the versions the port supports, as declared by USES:python:<blah>

Full Framework: Globally create a "debug" variant per port, that sets -g and doesn't strip symbols.

Those are just examples, but the answer is: at whatever level is needed. It was designed with this flexibility in mind

[antoine]

Did you try "poudriere bulk -a" or building openjdk?

[brendan_freebsd_bbqsrc.net]

In D5563#118766, @antoine wrote:

Did you try "poudriere bulk -a" or building openjdk?

If something doesn't have a VARIANTS definition, then the code path remains exactly the same, so there should be no change in behaviour here whatsoever.

[antoine]

This doesn't answer my questions.
And last time I tried to build openjdk with VARIANT set the build exploded.

[brendan_freebsd_bbqsrc.net]

In D5563#118771, @antoine wrote:

This doesn't answer my questions.
And last time I tried to build openjdk with VARIANT set the build exploded.

I recall you talking about this on IRC once, but I hardly remember the rest of the context. If I recall, it was if a global VARIANT was set in the environment, OpenJDK (iirc version 8) would fail to build.

I am testing it with poudriere right now to see if that is the case, though I think it will be unlikely, simply because VARIANTwon't be set at any point for that build.

VARIANT gets cleared in do-depends.sh to stop it from trickling to places it's not meant to go, so you'll need to give me an example of how you think VARIANT would ever apply to openjdk where someone didn't specifically feed it that variable. If you do think of one, I'd love to test it! :)

[antoine]

This needs to be planned beforehand, one day someone will set VARIANT for openjdk.
Otherwise you end up with bad things that have to stay forever (the breaking PYTHON_VERSION and broken PYTHON_ABIVER come to my mind...)

[brendan_freebsd_bbqsrc.net]

In D5563#118774, @antoine wrote:

This needs to be planned beforehand, one day someone will set VARIANT for openjdk.
Otherwise you end up with bad things that have to stay forever (the breaking PYTHON_VERSION and broken PYTHON_ABIVER come to my mind...)

I interpret what you've said so far as follows:

The OpenJDK origin uses a VARIANT environment variable internally and you are curious whether or not this case is handled so the VARIANT in the Makefile doesn't squash the VARIANT in the underlying build.

I have tested this specifically for you. It has had no impact. You can see the screencast here, which shows the minor changes to the file to create a stub variant, and running the build manually from the directory (you can skip to any point in the 'video' and there will be no lag, it's just a textual replay).

For completeness, here is openjdk8 build using poudriere.

[antoine]

In D5563#118777, @brendan_freebsd_bbqsrc.net wrote:

In D5563#118774, @antoine wrote:

This needs to be planned beforehand, one day someone will set VARIANT for openjdk.
Otherwise you end up with bad things that have to stay forever (the breaking PYTHON_VERSION and broken PYTHON_ABIVER come to my mind...)

I interpret what you've said so far as follows:

The OpenJDK origin uses a VARIANT environment variable internally and you are curious whether or not this case is handled so the VARIANT in the Makefile doesn't squash the VARIANT in the underlying build.

I have tested this specifically for you. It has had no impact. You can see the screencast here, which shows the minor changes to the file to create a stub variant, and running the build manually from the directory (you can skip to any point in the 'video' and there will be no lag, it's just a textual replay).

For completeness, here is openjdk8 build using poudriere.

Strange, I get those errors when I try it:

http://fpaste.org/334810/raw/

[brendan_freebsd_bbqsrc.net]

In D5563#118797, @antoine wrote:

Strange, I get those errors when I try it:

http://fpaste.org/334810/raw/

And here's what I get:

$ env VARIANT=FOO make -C /usr/ports/java/openjdk7
===>  openjdk-7.95.00,1 Unknown variant 'FOO', possible variants: .
*** Error code 1

Stop.
make: stopped in /usr/ports/java/openjdk7

Are you certain you've applied the provided patches correctly?

[antoine]

Yes, + added VARIANTS=FOO to openjdk7

[brendan_freebsd_bbqsrc.net]

In D5563#118802, @antoine wrote:

Yes, + added VARIANTS=FOO to openjdk7

Okay. The simple answer is that when you declare VARIANT as part of the env, instead of as part of the make command, it declares it as an environment variable instead of as a make variable.

From man make:

variable=value
        Set the value of the variable variable to value.  Normally, all
        values passed on the command line are also exported to sub-makes
        in the environment.  The -X flag disables this behavior.
        Variable assignments should follow options for POSIX
        compatibility but no ordering is enforced.

Variable classes
  The four different classes of variables (in order of increasing
  precedence) are:

  Environment variables
          Variables defined as part of make's environment.

  Global variables
          Variables defined in the makefile or in included makefiles.

  Command line variables
          Variables defined as part of the command line.

  Local variables
          Variables that are defined specific to a certain target.

Try setting it as a make variable as described above and you should be able to reproduce my behaviour.

[brendan_freebsd_bbqsrc.net]

In D5563#118802, @antoine wrote:

Yes, + added VARIANTS=FOO to openjdk7

Just noticed you were using openjdk7. In all of my examples, I was using openjdk8.

Okay, I can reproduce this now. Seems it doesn't care in this specific instance whether it's global or not. :)

Will look into it!

[brendan_freebsd_bbqsrc.net]

I am seeing very bizarre behaviour from make in this simple test case:

.undef name
.unexport name
.MAKEOVERRIDES:=
.MAKEFLAGS:=
name=

all:
        @echo "hello" $(name)

I get:

$ make name=foo
hello foo
$ name=foo make
hello

The output should at the _very least_ be the same, surely? Preferably both being just "hello" with no supplied name, as the Makefile clears them all in every possible way?

[feld]

I do hope this hasn't completely stalled out. Would love to see the ports tree gain this feature!

[bdrewery]

In D5563#118851, @brendan_freebsd_bbqsrc.net wrote:

I am seeing very bizarre behaviour from make in this simple test case:

.undef name
.unexport name
.MAKEOVERRIDES:=
.MAKEFLAGS:=
name=

all:
        @echo "hello" $(name)

I get:

$ make name=foo
hello foo
$ name=foo make
hello

The output should at the _very least_ be the same, surely? Preferably both being just "hello" with no supplied name, as the Makefile clears them all in every possible way?

Passing it as a make argument makes it assign the value at the point of expansion, so the .undef and name= won't matter. It is useful when passing to sub-makes to force them to use a value that they otherwise might have a default for. So for VARIANT I would say it should be passed as a make argument if the caller already knows what variant it is wanting to build for.

Thanks for working on this. I really like the idea.

For Poudriere (without VARIANTS), it currently does not pass down DEPENDS_ARGS properly. Doing so would fix many of the python ports to depend on the python3 version of packages and still have the python2 version of packages built as well for ports that needed those. It's not quite VARIANTS, but when passing down the DEPENDS_ARGS properly then a new PKGNAME may be discovered and it can be added into the queue. I've been working towards fixing that, which you've mostly solved in your patch as well since it is now fully package based. This is tracked at https://github.com/freebsd/poudriere/issues/259

Currently python.mk sets DEPENDS_ARGS+=PYTHON_VERSION=${chosen_python_version}, which is determined by a port either using the default or saying USES+= python:3, forcing it to use 3.x. This causes dependencies to be built (not in poudriere yet) with a forced PYTHON_VERSION=3 which is similar to your VARIANT=py3. For VARIANTS to work right it really needs to be integrated with DEPENDS_ARGS. DEPENDS_ARGS is used for ruby ports as well. So if you're building a python port with VARIANT=py3 it needs to also pass the right VARIANT down to its dependencies, without truncating away their own default VARIANT.

I am going to continue my work to get Poudriere fully dynamic with its queue to allow multiple packages and fix the DEPENDS_ARGS bug. I am going to try to make it a bit extensible so we can track more than just DEPENDS_ARGS or VARIANTS per package since we also want this sub-package thing that may require more information. My major concern about fixing DEPENDS_ARGS and VARIANTS so far has been performance of calculating dependencies for 25000 ports. The cache_get_* stuff is just awful too so I've been reworking a lot of it too. I would much rather have Poudriere know how to do most of the work here in the normal version and have the VARIANTS support be a very small patch until we all agree on it (just compute_deps part probably).

One problem I have here design wise is what VARIANT should build by default. For your poudriere example it is just building all VARIANTS. That's fine for me if there is some option like VARIANT=all set. Most users will still expect that adding devel/py-setuptools into their build list will only build the default one, and not all versions of it. I've been considering perhaps support in the origin list (bulk -f list) something like devel/py-setuptools:OPTIONS:OPTIONS... to allow passing something like VARIANT into the port before building. I haven't quite thought it through yet.

Something lacking here is direct ports support and portmaster/portupgrade/synth support. If we change how dependencies are handled and force using VARIANT then we must consider all of the tools. Using DEPENDS_ARGS to pass the VARIANT may be enough here as that is supposed to be respected.

Finally, we need to get more people in the discussion around the design. @bapt has had some patches and ideas on this for a while. There's little details that need to be decided, such as will PKGNAME be unique? Should changing the VARIANT always change the PKGNAME? I would much prefer it did. It's a policy question. This is an important thing for pkg and poudriere to keep things simpler.

devel/py-setuptools/Makefile

18

I think for this you ultimately could just put the VARIANTS in python.mk for all known versions. And then override it somehow in py3 ports that only support 3+.

[bapt]

Thank you for proposing this patch, since then we have the FLAVOR implementation which has hitted the tree and covers pretty much the same features