Subpackages. (deal with it.)

[mat]

Summary

Introduce subpackages to the ports framework - this allows ports to be built once, but have multiple packages created. For example, have a subpackage for docs and examples.

To add subpackages to a port, define:

SUBPACKAGES=  foo bar

and in the plist, prefix files to put in a subpackage with @@foo@@.

For each subpackage, some variables can be set, if needed.

• DESCR.<subpkg> defaults to the pkg-descr if pkg-descr.<subpkg> does not exist.
• COMMENT.<subpkg> defaults to ${COMMENT} (subpkg: <subpkg>)
• All the PKGMESSAGE/PKGINSTALL/... variable have .<subpkg> version, pointing to pkg-message.<subpkg>/pkg-install.<subpkg> files.
• A few options helpers <opt>_SUBPACKAGES and <opt>_*_DEPENDS.<subpkg> can be used. See lang/perl5-devel example.
• PLIST_FILES.<subpkg> and PLIST_DIRS.subpkg were added too.

Subpackages can depend on each other, a new variable has been introduced, SELF_DEPENDS, and a special subpackage name has been introduced to refer to the main package. For example:

If the foo subpackage depends on the main subpackage:

SUBPACKAGES=  foo
SELF_DEPENDS.foo= main

If the main package depends on the tools subpackage:

SUBPACKAGE=   tools
SELF_DEPENDS=  tools

Test Plan

EVERYTHING IS FINE.

You will probably need a patched poudriere, patches from either the branch or the pull request

[rene]

lang/perl5-devel/pkg-plist

789

Just some thoughts, feel free to ignore:

• when defining OPT_SUBPACKAGE (e.g. OPT_gdbm) it would be a nice idea to automatically move the %%GDBM%% files into the gdbm subpackage, so e.g. %%gdbm%%%%ARCH_LIB%%/GDBM_File.pm (so using lowercase option names to combine the subpackage and the regular option names). This way we can reduce some red tape.
• When GDBM_File.pm should go into multiple/different subpackages: %%GDBM%%@@doc@@%%ARCH_LIB%%/GDBM_File1.pm %%GDBM%%@@gdbm@@%%ARCH_LIB%%/GDBM_File_Explicit.pm %%gdbm%%%%ARCH_LIB%%/GDBM_File_Implicit.pm

This combining option and subpackage names is not foolproof, there are a few ports with lowercase option names (one qt4/kde4-using port IIRC).

[mat]

lang/perl5-devel/pkg-plist

789

I feel the semantics would get overly complicated and confuse people.

Options MUST be uppercase, I think there is a QA check for this. I do not remember finding ports with lowercase options, but in any case, they should be fixed.

[rene]

lang/perl5-devel/pkg-plist

789

Yes, probably better to leave the option and subpackage name split.

It was the QA check that notified me of that port ;)

[rene]

I suggest to wait for 2019Q1, then:

• update proudriere(-devel) with @mat 's patch
• update user/developer documentation
• rebase the patch
• commit the patch
• send out some emails to announce subpackages

[mat]

In D16457#396393, @rene wrote:

I suggest to wait for 2019Q1, then:

• update proudriere(-devel) with @mat 's patch
• update user/developer documentation
• rebase the patch
• commit the patch
• send out some emails to announce subpackages

Mmmm, you assume the patch has not changed in the past few months because it is ready, it is most probably not. It has not changed because I do not have had the time to work on it any more.

[rene]

Ah, I was too optimistic :(

[tcberner]

Mk/bsd.port.mk

2632

^ flavors sounds wrong here.

2653

^ I would keep it afterwards and add a DEV_WARNING

2657

How about adding a bsd.subpackages.commentsuffix.mk ala bsd.options.desc.mk:

COMMENTSUFFIX.debug?=           Debug Parts
COMMENTSUFFIX.doc?=             Documentation
COMMENTSUFFIX.l10n?=            Localization Files

And then do something like:

.    if defined(COMMENTSUFFIX.${p})
_COMMENTSUFFIX?=       ${COMMENTSUFFIX.${p}}
.    else
_COMMENTSUFFIX?=       ${p} subpackage
.    endif
COMMENT.${p}?= ${COMMENT} (${_COMMENTSUFFIX})
. endfor
.endif

[mat]

Mk/bsd.port.mk

2632

Ah, yes, too much copy&paste ^^

2653

Mmmm. You mean a DEV_WARNING when the subpackage does not have a specific description file?

2657

First, I think bsd.options.desc.mk should die, preferably in a large bonfire. Because it's all filled in with:

<foo>_DESC= <foo> support

Which is mostly useless to end users, because it does not tell them anything. Of course option foo as something to do with foo being added. But it does not tell the users what foo is, or what adding foo support actually does.

For example, if a port has the ICU option, the user will be shown "Unicode support via ICU", but what does it mean for the port? Does it mean that without it, the port will only accept 7 bit ASCII text input?

So, while your three examples are ok, I feel having this would end up as the options, and not saying much of what's in it. Or it would need to require some sort of approval to add to, unlike options.desc right now.

Also, I feel adding too many helpers defeats the purpose of having everything customizable.

(Side note, your example can be written as two lines:

COMMENTSUFFIX.${p}?=       ${p} subpackage
COMMENT.${p}?= ${COMMENT} (${COMMENTSUFFIX.${p}})

)

[mat]

rebase & add git subpackages

[mat]

rebase

[mat]

rebase

[mat]

rebase.

[mat]

Small fix in:

• Strip subpackage markers from the plist. (May not be correct.)

Rebase

[mat]

• Use sed instead of grep here, grep exists if it does not output

[mat]

Fixup in

• Start getting php72 to use subpackages.

[tcberner]

@mat is this stable enough to start play around with? Because for example Qt has many split ports that could be used as testing ground too (basically all the ports that have the same qt-dist value).

[mat]

In D16457#489658, @tcberner wrote:

@mat is this stable enough to start play around with? Because for example Qt has many split ports that could be used as testing ground too (basically all the ports that have the same qt-dist value).

Yes, this is definitively a good idea to play around with it. There are still a few things that need fixing/doing, but I doubt the way it works will change.

Note that you need a patch for poudriere to play with subpackages :-)

[mat]

rebase

[mat]

• Add automatic metapkg support for Python flavors.

[mat]

• fixup! Add automatic metapkg support for Python flavors.
• fixup! Add automatic metapkg support for Python flavors.

[mat]

• Sanitize environment here, sometime, stuff leaks from qa.sh and breaks
• fixup! Add automatic metapkg support for Python flavors.

[mat]

Rebase & remove php.

[mat]

• Ajouter une variable PKGNAME.<subpkg>.

[mat]

• Ajouter une variable PKGNAME.<subpkg>.
• Add PKGNAMES that has the main package name, and subpackage names.

[darkfiberiru_gmail.com]

@mat
Can we add this (from https://wiki.freebsd.org/Ports/FlavorsAndSubPackages)

Sub Packages
Build the port once and create multiple packages. For example, have a sub package for docs and examples.

or some similar description to summary for people who aren't familiar with subpackages.
Thanks,
Nick

[mat]

rebase.

[0mp]

Are there any blockers preventing us from committing this feature?

[mandree]

I might find breaking out one build into multiple packages very useful,
but for a successful launch I propose this would require a proper specification added to the documentation. Documentation by example is no good for developers who are new to this feature.
I wonder if there should be an option for split-out pkg-plist files, which would be much easier to understand.

Please add:

• an section for the porter's handbook, consisting of (1) introduction (the summary is mostly okay), (2) specification, (3) examples,
• proposals for portlint, portclippy/portfmt and, if needed, stage-qa and check-plist
• a proposed ports/CHANGES entry
• for new volunteers, some thoughts on the intended relation between options and subpackages, and possibly subpackages and dependencies
• an option to have pkg-plist.<subpkg> - it does not look very useful if I need to search @@subpkg@@ lines that are few and far between in a longish file of a nontrivial port.

[0mp]

Just a note to other developers interested in this feature: the current blocker is Poudriere support, additional help with the development is appreciated: https://github.com/freebsd/poudriere/pull/741

[se]

I understand that subpackages can be used to split off development specific files (e.g. header files), examples, documentation, etc. from a package.

But I think there are significant issues for non-trivial ports that are converted to use subpackages to control the installed functionality.

I do not see that the case of build options depending on subpackages been considered? (i.e. the opposite case of subpackages being selected by port options)

AFAIU the patch, the selection of subpackages to be built can be controlled by either of two mechanisms:

1. The initial method is that OPTIONS add values to ${SUBPACKAGES}. In that case, other build parameters controlled by the options may be defined.
2. The subpackage in a *_DEPENDS variable is identified with a ~subpkg modifier added to its origin. But port options that lead to this subpackage being build by the first method are not known in that case (unless a mechanism is introduced that sets those options depending on the subpackage built as a dependency).

This leads to more complications in non-trivial cases. E.g. if a subpackage exists for NLS locale dependent data files, then the main program will need to be linked against optional libraries. For thos example, either the NLS option is always defined when building the main package (adding complexity for the non-NLS use case) or a ~nls subpackage cannot be combined with the non-NLS main program.
To work around this issue, ports will always need to be build with a complete set of options (e.g. if those control the integration of hooks), but if a port is built with plain make (or a port management tool) this will lead to all those subpackages and their dependencies being installed.

If a dependency for a non-default subpackage leads to main being rebuild with a non-default option, this may lead to creation of a main package with non-default options that replace the main subpackage initially built without that option. A later upgrade could then cause the main package to again be built without that additional option, breaking the subpackage that needs main with non-default options.

This is an issue not only for port building with make or port management tools, but also for poudriere if not all subpackages are always built (and thus all options defined that are required to support any of those subpackages). It is possible that some subpackages conflict with each other and thus that not all subpackages can be built at the same time. This includes cases that need different and incompatible hooks in the main package, too.

Other issues are (IMHO):

• No automatic creation of pkg-plist files for subpackages ("make makeplist" could be used to create a list of all files, but the assignment to subpackages cannot be automated)
• Why are there no CATEGORIES.${p} and *CONFLICTS.${p}?
• WIll the subpackage selector be included in the MANIFEST and in the package db?
• What will "make deinstall" do if non-default subpackages have been installed that depend on the main package?

Mk/Scripts/do-depends.sh

145

Only "@" or "~" may occur in an origin since they are alternatives in a single case statement.

If this was changed: what order of flavor and subpackage would be expected?