Details

Reviewers

bapt
mandree
se

Group Reviewers

portmgr

Summary

Introduce subpackages to the ports framework - this allows ports to be built once, but have multiple packages created. For example, have a subpackage for docs and examples.

To add subpackages to a port, define:

SUBPACKAGES=  foo bar

and in the plist, prefix files to put in a subpackage with @@foo@@.

For each subpackage, some variables can be set, if needed.

DESCR.<subpkg> defaults to the pkg-descr if pkg-descr.<subpkg> does not exist.
COMMENT.<subpkg> defaults to ${COMMENT} (subpkg: <subpkg>)
All the PKGMESSAGE/PKGINSTALL/... variable have .<subpkg> version, pointing to pkg-message.<subpkg>/pkg-install.<subpkg> files.
A few options helpers <opt>_SUBPACKAGES and <opt>_*_DEPENDS.<subpkg> can be used. See lang/perl5-devel example.
PLIST_FILES.<subpkg> and PLIST_DIRS.subpkg were added too.

Subpackages can depend on each other, a new variable has been introduced, SELF_DEPENDS, and a special subpackage name has been introduced to refer to the main package. For example:

If the foo subpackage depends on the main subpackage:

SUBPACKAGES=  foo
SELF_DEPENDS.foo= main

If the main package depends on the tools subpackage:

SUBPACKAGE=   tools
SELF_DEPENDS=  tools

Test Plan

EVERYTHING IS FINE.

You will probably need a patched poudriere, patches from either the branch or the pull request

Diff Detail

Lint

No Lint Coverage

Unit

No Test Coverage

Build Status

Buildable 31581
Build 29177: arc lint + arc unit

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

In D16457#396393, @rene wrote:

I suggest to wait for 2019Q1, then:

update proudriere(-devel) with @mat 's patch

update user/developer documentation

rebase the patch

commit the patch

send out some emails to announce subpackages

Mmmm, you assume the patch has not changed in the past few months because it is ready, it is most probably not. It has not changed because I do not have had the time to work on it any more.

Ah, I was too optimistic :(

tcberner added a subscriber: tcberner.Feb 1 2019, 3:32 PM

tcberner added inline comments.Feb 3 2019, 9:04 AM

Mk/bsd.port.mk

2632

^ flavors sounds wrong here.

2653

^ I would keep it afterwards and add a DEV_WARNING

2657

How about adding a bsd.subpackages.commentsuffix.mk ala bsd.options.desc.mk:

COMMENTSUFFIX.debug?=           Debug Parts
COMMENTSUFFIX.doc?=             Documentation
COMMENTSUFFIX.l10n?=            Localization Files

And then do something like:

.    if defined(COMMENTSUFFIX.${p})
_COMMENTSUFFIX?=       ${COMMENTSUFFIX.${p}}
.    else
_COMMENTSUFFIX?=       ${p} subpackage
.    endif
COMMENT.${p}?= ${COMMENT} (${_COMMENTSUFFIX})
. endfor
.endif

mat marked an inline comment as done.Feb 4 2019, 2:08 PM

mat added inline comments.

Mk/bsd.port.mk
2632	Ah, yes, too much copy&paste ^^
2653	Mmmm. You mean a DEV_WARNING when the subpackage does not have a specific description file?
2657	First, I think bsd.options.desc.mk should die, preferably in a large bonfire. Because it's all filled in with: <foo>_DESC= <foo> support Which is mostly useless to end users, because it does not tell them anything. Of course option foo as something to do with foo being added. But it does not tell the users what foo is, or what adding foo support actually does. For example, if a port has the ICU option, the user will be shown "Unicode support via ICU", but what does it mean for the port? Does it mean that without it, the port will only accept 7 bit ASCII text input? So, while your three examples are ok, I feel having this would end up as the options, and not saying much of what's in it. Or it would need to require some sort of approval to add to, unlike options.desc right now. Also, I feel adding too many helpers defeats the purpose of having everything customizable. (Side note, your example can be written as two lines: COMMENTSUFFIX.${p}?= ${p} subpackage COMMENT.${p}?= ${COMMENT} (${COMMENTSUFFIX.${p}}) )

rebase & add git subpackages

Harbormaster completed remote builds in B22501: Diff 53920.Feb 14 2019, 3:02 PM

matthew added a subscriber: matthew.May 16 2019, 6:54 PM

debdrup added a subscriber: debdrup.May 28 2019, 7:54 PM

rebase

Harbormaster completed remote builds in B24980: Diff 58893.Jun 22 2019, 9:31 AM

rebase

Harbormaster completed remote builds in B24981: Diff 58894.Jun 22 2019, 11:13 AM

kai mentioned this in D21084: security/spass: Update to 3.2 and create a Qt5 slave port.Aug 14 2019, 9:15 PM

rebase.

Harbormaster completed remote builds in B27468: Diff 64226.Nov 12 2019, 11:58 AM

Small fix in:

Strip subpackage markers from the plist. (May not be correct.)

Rebase

Harbormaster completed remote builds in B27511: Diff 64321.Nov 14 2019, 2:40 PM

Use sed instead of grep here, grep exists if it does not output

Harbormaster completed remote builds in B27512: Diff 64322.Nov 14 2019, 2:47 PM

Fixup in

Start getting php72 to use subpackages.

Harbormaster completed remote builds in B27513: Diff 64324.Nov 14 2019, 3:01 PM

@mat is this stable enough to start play around with? Because for example Qt has many split ports that could be used as testing ground too (basically all the ports that have the same qt-dist value).

In D16457#489658, @tcberner wrote:

@mat is this stable enough to start play around with? Because for example Qt has many split ports that could be used as testing ground too (basically all the ports that have the same qt-dist value).

Yes, this is definitively a good idea to play around with it. There are still a few things that need fixing/doing, but I doubt the way it works will change.

Note that you need a patch for poudriere to play with subpackages :-)

iwtcex_gmail.com mentioned this in D16830: USES=lib32: add support for lib32- companion ports.Dec 11 2019, 5:21 PM

theron.tarigo_gmail.com added a subscriber: theron.tarigo_gmail.com.Dec 11 2019, 6:45 PM

zeising added a subscriber: zeising.Dec 17 2019, 11:41 AM

rebase

Harbormaster completed remote builds in B28969: Diff 67393.Jan 28 2020, 11:17 AM

Add automatic metapkg support for Python flavors.

Harbormaster completed remote builds in B28982: Diff 67410.Jan 28 2020, 1:38 PM

mat marked 7 inline comments as done.Jan 28 2020, 1:42 PM

fixup! Add automatic metapkg support for Python flavors.
fixup! Add automatic metapkg support for Python flavors.

Harbormaster completed remote builds in B28989: Diff 67421.Jan 28 2020, 2:36 PM

Sanitize environment here, sometime, stuff leaks from qa.sh and breaks
fixup! Add automatic metapkg support for Python flavors.

Harbormaster completed remote builds in B29008: Diff 67446.Jan 29 2020, 1:33 PM

Rebase & remove php.

Harbormaster completed remote builds in B29010: Diff 67448.Jan 29 2020, 2:05 PM

mat mentioned this in D23413: Start getting php72 to use subpackages..Jan 29 2020, 2:08 PM

Ajouter une variable PKGNAME.<subpkg>.

Harbormaster completed remote builds in B29022: Diff 67466.Jan 29 2020, 6:05 PM

lwhsu added a subscriber: lwhsu.Jan 30 2020, 3:09 AM

Ajouter une variable PKGNAME.<subpkg>.
Add PKGNAMES that has the main package name, and subpackage names.

Harbormaster completed remote builds in B29039: Diff 67504.Jan 30 2020, 10:52 AM

@mat
Can we add this (from https://wiki.freebsd.org/Ports/FlavorsAndSubPackages)

Sub Packages
Build the port once and create multiple packages. For example, have a sub package for docs and examples.

or some similar description to summary for people who aren't familiar with subpackages.
Thanks,
Nick

woodsb02 added a subscriber: woodsb02.Feb 9 2020, 9:23 AM

0mp added a subscriber: 0mp.Feb 12 2020, 11:50 AM

mat edited the test plan for this revision. (Show Details)May 13 2020, 7:41 AM

rebase.

Harbormaster completed remote builds in B31581: Diff 72856.Jun 9 2020, 12:03 PM

0mp awarded a token.Jun 9 2020, 12:11 PM

woodsb02 edited the summary of this revision. (Show Details)Jun 9 2020, 3:35 PM

Are there any blockers preventing us from committing this feature?

pi added a subscriber: pi.Oct 23 2020, 6:50 PM

I might find breaking out one build into multiple packages very useful,
but for a successful launch I propose this would require a proper specification added to the documentation. Documentation by example is no good for developers who are new to this feature.
I wonder if there should be an option for split-out pkg-plist files, which would be much easier to understand.

Please add:

an section for the porter's handbook, consisting of (1) introduction (the summary is mostly okay), (2) specification, (3) examples,
proposals for portlint, portclippy/portfmt and, if needed, stage-qa and check-plist
a proposed ports/CHANGES entry
for new volunteers, some thoughts on the intended relation between options and subpackages, and possibly subpackages and dependencies
an option to have pkg-plist.<subpkg> - it does not look very useful if I need to search @@subpkg@@ lines that are few and far between in a longish file of a nontrivial port.

This revision now requires changes to proceed.Oct 23 2020, 7:31 PM

Just a note to other developers interested in this feature: the current blocker is Poudriere support, additional help with the development is appreciated: https://github.com/freebsd/poudriere/pull/741

freebsd_igalic.co added a subscriber: freebsd_igalic.co.Nov 10 2020, 7:23 PM

val_packett.cool added a subscriber: val_packett.cool.Nov 23 2020, 1:50 PM

rhurlin added a subscriber: rhurlin.Nov 24 2020, 11:06 AM

arrowd added a subscriber: arrowd.Apr 29 2021, 12:00 PM

dereks_lifeofadishwasher.com added a subscriber: dereks_lifeofadishwasher.com.May 17 2021, 12:19 AM

se added a subscriber: se.Aug 19 2021, 9:53 AM

I understand that subpackages can be used to split off development specific files (e.g. header files), examples, documentation, etc. from a package.

But I think there are significant issues for non-trivial ports that are converted to use subpackages to control the installed functionality.

I do not see that the case of build options depending on subpackages been considered? (i.e. the opposite case of subpackages being selected by port options)

AFAIU the patch, the selection of subpackages to be built can be controlled by either of two mechanisms:

The initial method is that OPTIONS add values to ${SUBPACKAGES}. In that case, other build parameters controlled by the options may be defined.
The subpackage in a *_DEPENDS variable is identified with a ~subpkg modifier added to its origin. But port options that lead to this subpackage being build by the first method are not known in that case (unless a mechanism is introduced that sets those options depending on the subpackage built as a dependency).

This leads to more complications in non-trivial cases. E.g. if a subpackage exists for NLS locale dependent data files, then the main program will need to be linked against optional libraries. For thos example, either the NLS option is always defined when building the main package (adding complexity for the non-NLS use case) or a ~nls subpackage cannot be combined with the non-NLS main program.
To work around this issue, ports will always need to be build with a complete set of options (e.g. if those control the integration of hooks), but if a port is built with plain make (or a port management tool) this will lead to all those subpackages and their dependencies being installed.

If a dependency for a non-default subpackage leads to main being rebuild with a non-default option, this may lead to creation of a main package with non-default options that replace the main subpackage initially built without that option. A later upgrade could then cause the main package to again be built without that additional option, breaking the subpackage that needs main with non-default options.

This is an issue not only for port building with make or port management tools, but also for poudriere if not all subpackages are always built (and thus all options defined that are required to support any of those subpackages). It is possible that some subpackages conflict with each other and thus that not all subpackages can be built at the same time. This includes cases that need different and incompatible hooks in the main package, too.

Other issues are (IMHO):

No automatic creation of pkg-plist files for subpackages ("make makeplist" could be used to create a list of all files, but the assignment to subpackages cannot be automated)
Why are there no CATEGORIES.${p} and *CONFLICTS.${p}?
WIll the subpackage selector be included in the MANIFEST and in the package db?
What will "make deinstall" do if non-default subpackages have been installed that depend on the main package?

Mk/Scripts/do-depends.sh
145	Only "@" or "~" may occur in an origin since they are alternatives in a single case statement. If this was changed: what order of flavor and subpackage would be expected?