packages: Add meta-package sets

[ivy]

Summary

Add several metapackages which contain no files themselves, but depend
on other packages. This allows the user to install a system by using
a set like "FreeBSD-set-minimal" instead of having to know which
specific packages are required, and it means if we add more package
in the future, existing installations will get them automatically as
long as they were installed using a set.

The defined sets are significantly less granular than the actual
packages; the assumption is that users who want a very specific set of
packages installed can do that manually, while sets are intended more
for typical users who just want to install FreeBSD.

Three sets are defined:

• minimal: the set of packages required to boot the system and bring up a multi-user UNIX system. This includes hardware support, networking (wireless, DHCP), basic functionality like syslogd, cron and periodic. It does not include a kernel, because the kernels are quite large and the user might want to use a custom kernel.
  
  sendmail and DMA are not included in minimal. Since we provide two MTAs, the user should select which one they want, or install one from ports. sshd is not included because it's not required for the base system, and the user might want to install it from ports.
  
  Some other significant subsystems (e.g. NFS, Kerberos) are also not included.
  
  The minimal set does not include any libraries itself, since we rely on dependencies to pull these in. This implies it doesn't include library manpages, which is fine, since users won't want manpages for libraries unless they've also installed the development packages.

• devel: the compiler toolchain for building software, along with all "-dev" packages which include header files and static libraries.

• lib32: 32-bit compatibility packages. This includes both runtime and development packages.

• kernels: the kernel packages. For freebsd.org builds, this will be GENERIC, GENERIC-DEBUG, etc., for custom builds it will be whatever was set in $KERNCONF.

• base: everything else; this includes the entire base system.

For each set, also generate a <setname>-dbg set containing the debug
packages for the set.

The sets are built along with the rest of the packages using a new
script called create-sets.sh, which examines the "set" annotation in
each package and puts it in the appropriate set. This is in
anticipation of the later "groups" feature appearing in pkg itself, at
which point we can simply replace the set annotation with the group.

[ivy]

as discussed with @bapt, we won't have package groups in time for 15.0, so this is a temporary solution until that happens (perhaps for 16.0).

i've proposed what seems to be like a reasonable set of sets to begin with, but this is very much up for discussion, so let's build some bikesheds.

[ivy]

@ifreund_freebsdfoundation.org this may be interesting for you (assuming you're still working on the installer) because whatever sets we decide on here should probably be offered as options in the installer, rather than generating package lists by hand.

[ivy]

also put library manpage packages into the devel set

because if users install set-devel, they probably want the manpages too.

[ivy]

for reference, here's a list of every package and the set it's included in: https://people.freebsd.org/~ivy/tmp/pkgsets.txt -- but note that sets might pull in more packages than listed here due to explicit or library dependencies.

[ivy]

move some packages around

• lld, lldb and mtree should be in devel
• nuageinit should be in minimal for bootstrapping virtual machines

[markmi_dsl-only.net]

In D52412#1196934, @ivy wrote:

for reference, here's a list of every package and the set it's included in: https://people.freebsd.org/~ivy/tmp/pkgsets.txt -- but note that sets might pull in more packages than listed here due to explicit or library dependencies.

The naming below seems to be specific to main. stable/* and the later stages after that branching have nodebug content as kernel-generic and a possible kernel-generic-debug for debug content (witness and such).

FreeBSD-kernel-generic              kernels
FreeBSD-kernel-generic-dbg          kernels-dbg
FreeBSD-kernel-generic-nodebug      kernels
FreeBSD-kernel-generic-nodebug-dbg  kernels-dbg
FreeBSD-kernel-man                  kernels

There is also the issue that aarch64 [tier 1] (and armv7) have FreeBSD-dtb in main. (This avoided kernel installation conflicts, by factoring out files common across kernels.)

As stands stable/14 does not allow aarch64 or armv7 to have all the kernels installed at the same time because some have their own set of *.dtb files that conflict with others. There is no FreeBSD-dtb for stable/14 . It is not clear what style stable/15 will use for *.dtb file handling.

Other notes:

I assume that kernel-generic-mmccam (amd64 and aarch64, for example) and kernel-minimal (just amd64) not being listed are deliberately not listed.

[ivy]

In D52412#1196973, @markmi_dsl-only.net wrote:

The naming below seems to be specific to main.

i generated the list on main/amd64, so it shows the set of packages for that version and architecture. the sets are generated dynamically, so when building on stable or releng, you'll get -generic and -generic-debug automatically, or whatever you set in $KERNCONF when building the package set.

There is also the issue that aarch64 [tier 1] (and armv7) have FreeBSD-dtb in main.

the -dtb package should be in minimal when building on aarch64, see the change to release/packages/ucl/dtb-all.ucl in this diff. you can test this by building a package set with this diff applied, then looking at the "set" annotation in pkg info FreeBSD-dtb.

this change will not be MFC'd to stable/14.

[markmi_dsl-only.net]

this change will not be MFC'd to stable/14.

Note: I did not intend my comparisons to stable/14 to imply any MFC intent or request. It is just the only existing example stable/* to guess stable/* intent from.

[jrtc27]

release/packages/generate-ucl.lua

183

Can we please not hard-code lib32 as the only libcompat in this script? Please use _ALL_libcompats from share/mk/bsd.compat.pre.mk somehow. Then downstreams just need to add the UCL files for the sets they introduce, they don't need to patch this script.

[ivy]

release/packages/generate-ucl.lua

183

although i don't disagree, the string "lib32" is hardcoded all over the pkgbase scripts, and excising that is a much larger job. i'd rather leave this for now rather than introducing additional complexity that we might change later.

[markmi_dsl-only.net]

Just for reference, from looking at my live aarch64 and amd64 environments, "+" lines for unique to amd64 and "-" lines for unique to aarch64, but not based on more recent adjustments:

+FreeBSD-apm-dbg APM Utilities (debugging symbols)
+FreeBSD-apm-man APM Utilities (manual pages)
+FreeBSD-bootloader-dev Bootloader (development files)
-FreeBSD-dtb FreeBSD Devicetree Blobs
-FreeBSD-efi-tools-dbg-lib32 UEFI Utilities (32-bit debugging symbols)
-FreeBSD-efi-tools-dev-lib32 UEFI Utilities (32-bit development files)
-FreeBSD-efi-tools-lib32 UEFI Utilities (32-bit libraries)
+FreeBSD-kernel-minimal FreeBSD minimal Kernel 
+FreeBSD-kernel-minimal-dbg FreeBSD minimal Kernel -dbg (debugging symbols)
+FreeBSD-libipt Intel(R) Processor Trace decoder library
+FreeBSD-libipt-dbg Intel(R) Processor Trace decoder library (debugging symbols)
+FreeBSD-libipt-dev Intel(R) Processor Trace decoder library (development files)
+FreeBSD-libvgl Video Graphics Library
+FreeBSD-libvgl-dbg Video Graphics Library (debugging symbols)
+FreeBSD-libvgl-dbg-lib32 Video Graphics Library (32-bit debugging symbols)
+FreeBSD-libvgl-dev Video Graphics Library (development files)
+FreeBSD-libvgl-dev-lib32 Video Graphics Library (32-bit development files)
+FreeBSD-libvgl-lib32 Video Graphics Library (32-bit libraries)
+FreeBSD-libvgl-man Video Graphics Library (manual pages)

[jrtc27]

release/packages/generate-ucl.lua

183

I've tried to intercept all new ones to avoid making it worse. I would prefer that to be the approach taken.

[markmi_dsl-only.net]

the -dtb package should be in minimal when building on aarch64, see the change to release/packages/ucl/dtb-all.ucl in this diff. you can test this by building a package set with this diff applied, then looking at the "set" annotation in pkg info FreeBSD-dtb.

*.dtb files are essential for kernel operation, not world operation. Using set "minimal" looks very wrong for updating and booting the kernel(s), say before world is updated: the usual historical sequencing with reboots between. In other words: the build-time sequencing is not what is important here.

[ivy]

In D52412#1196997, @markmi_dsl-only.net wrote:

*.dtb files are essential for kernel operation, not world operation. Using set "minimal" looks very wrong for updating and booting the kernel(s), say before world is updated: the usual historical sequencing with reboots between.

this doesn't change the order that things are upgraded in: if arm64 needs to update the -dtb package along with the -kernel package, you can keep doing that. if that should be handled differently, it's probably out of scope for this change, which is only about defining package sets for installing new systems (or adding components to existing systems, which doesn't apply to the minimal set).

[markmi_dsl-only.net]

This comment was removed by markmi_dsl-only.net.

[ivy]

add devmatch to minimal

[imp]

Devmatch and devd are basically required

[markmi_dsl-only.net]

In D52412#1197023, @ivy wrote:

add devmatch to minimal

So I should not have deleted my note. (I got to thinking that i might have confused devmatch and devd needs, since devd was in minimal. So I deleted my original note about devmatch.) Thanks for adding devmatch anyway.

[ivy]

In D52412#1197067, @markmi_dsl-only.net wrote:

So I should not have deleted my note.

no, you were right: in fact i meant to add devmatch originally and then forgot until you mentioned it.

[grahamperrin]

Documentation, awareness, workarounds

If I intentionally delete an unwanted non-essential automatic package that is not a meta package, then will pkg check -d prompt to reinstall?

I assume so.

pkg-check(8)

Possible workaround: a fake package with a version height that will never be reached by the genuine package.

Preventing a given package from being installed – Dan Langille's Other Diary (2021)

[bapt]

LGTM, note I think we should flag those packages as "vital", meaning they cannot be removed beside explicitly asked by the user.

[markmi_dsl-only.net]

Note: This lead to a sizable increase in what pkgbase'd poudriere-devel jail creations include, including a full copy of /usr/src/ ( not just sys/ ) and more.