Ports Tree Base Packages

[kmoore]

Summary

This revision includes all the necessary changes and new ports required to start building and using base packages for FreeBSD.

It includes some of the following features which were discussed at BSDCan 2019's working group:
https://wiki.freebsd.org/DevSummit/201905/PackageBase

• Roughly a dozen "base packages"
• FreeBSD /usr/src available as a package
• Flavors Support
• Broken out "bootstrap" package which includes critical libs such as ld-elf, libc, libthr, libm, etc.
• A kernel "sentinel" file which will make sure ports-installed kmods are updated in lockstep with os/kernel. (Super important for sysutils/zol and others that load at bootup)

Usage:

Step one, ensure you have sources checked out to /usr/src, or else have ${BASEPKG_SRCDIR} set to the location of sources you wish to build.

Step two, build the the ports / packages

Step three, profit

If using poudriere, apply to this patch to your ports tree. Run "bulk" as usual and the os/* packages will be automatically built using the jails /usr/src sources.

Test Plan

Beyond the usual review, will be requesting an exp-run from portmgr.

Additional patches are incoming to 'pkg' which handle more of a "freebsd-update" style updating, I.E. kernel-first, and then ensuring the the bootstrap package is updated first. Those are beyond the scope of this review and will be handled separately.

[kmoore]

Exp-run requested:

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=238090

[antoine]

A few questions:

• there are 0 integrity check for os ports, while all non-os ports do check the SHA-256 of their distfiles, is this intended?
• some OPTIONS are defined for some os ports, are those OPTIONS correct for all branches of base?
• why put base and ports in the same ports tree (and consequently in the same package repository)? why not produce a base package repository from another source tree instead?
• are you aware that building 33k packages can take 3 or 4 days (or even more for fancy archs)? are you ok with this delay for base releases or base security updates?
• are you ok with portmgr now responsible for building base releases and security updates? (portmgr runs bulk -a on the ports tree, no matter what is in it)

[kmoore]

In D20394#440452, @antoine wrote:

A few questions:

• there are 0 integrity check for os ports, while all non-os ports do check the SHA-256 of their distfiles, is this intended?

Correct. We've tried it both ways, and while we can do either this seemed to be more workable. When using a poudriere jail / bulk build, you want your OS packages to be in sync with what version was used to create the local build jail. Doing it this way allows the same set of ports to be used for FreeBSD 10/11/12/13/etc without needing to constantly be bumping distfiles and versions. Especially since we build on 12-stable and 13-current weekly.

We could go the other route and hard-define specific revisions, but it'll just require a lot more active maintenance to keep those up to date. Not impossible, but just wasn't sure it was worth the effort.

• some OPTIONS are defined for some os ports, are those OPTIONS correct for all branches of base?

Those are correct for HEAD. Once we get some consensus on which versions of FreeBSD folks find this useful for, we'd want to probably add version-specific options. However having these options doesn't seem to break anything today, they just may not be used if you build FreeBSD 9.x.

• why put base and ports in the same ports tree (and consequently in the same package repository)? why not produce a base package repository from another source tree instead?

We've done both. Started off using base packages in their own repo. While it sounds ideal, in practice its not so good. Using pkg with multi-repo updating caused us all kinds of grief on the runtime side, where repos weren't in sync, one got updated without the other, etc. Additionally we wanted to 100% ensure that ports-installed kmods are kept updated in lockstep with the os/kernel package. This wasn't possible at this time using the two repo method. The submitted patch handles that with the kernel-kmod-sentinel package.

• are you aware that building 33k packages can take 3 or 4 days (or even more for fancy archs)? are you ok with this delay for base releases or base security updates?

Initially sure, however it doesn't have to be that way. If there are security updates to base OS, it is entirely possible to update the jail, and rebuild only the os/* ports without doing the entire ports tree. At most you'll only be rebuilding world/kernel/kmods and that may only take a few hours. Thats more of a process decision on behalf of whomever is doing the building of packages.

• are you ok with portmgr now responsible for building base releases and security updates? (portmgr runs bulk -a on the ports tree, no matter what is in it)

Yup! I think some process would need to be checked by portmgr to ensure they are updating in best possible manner. I.E. perhaps some logic that if SA's are announce, the next run is "bulk os/userland@all os/kernel@all" instead of "bulk -a", but those decisions can be made as part of a process refinement at a later time if necessary.

[seanc]

Integrity of the base distfiles is left to the administrator. Maintaining the chain of trust is outside of the scope of building base packages, whereas pkg for ports packages and distfiles does require pkg to maintain a chain of trust. Source code acquisition via ssh, https, or fetching tarballs and comparing the SHAs all seem like prerequisites for building a package base system, therefore out of scope for the proposed set of patches here. Said differently, if a release engineer can't ensure integrity of the source used to build a jail, the release engineer has bigger problems.

[adamw]

In D20394#441438, @seanc wrote:

Integrity of the base distfiles is left to the administrator. [...] Said differently, if a release engineer can't ensure integrity of the source used to build a jail, the release engineer has bigger problems.

I know admittedly very little about what's being proposed and the obstacles here, but I have a conceptual problem with that statement. You're asking us to teach our end-users that they shouldn't trust the ports tree. I can't recall us ever taking that position with our end-users. Checksums are sacrosanct in the ports tree, and they are a basic mechanism of the trust our end-users have in us. If you're asking us to roll src into the ports tree, then it really ought to come with the same basic security expectations we have for everything else in the tree.

My major problem here is that base packaging simply isn't the purview of portmgr. We do the ports tree, and packing for ports. If core's expectation really is that we are going to be solely responsible for packaging base, then we have to fork pkg-building to its own team, and it needs to involve both ports and src people.

When the INDEX breaks because of a src commit---and INDEX breaks all the time---now ports is being asked to amend src commits; when it breaks things entirely, we'd have an unclear chain of command if we back out a src commit. One of our core policies is called "Just Fix It," where anybody can play with anybody else's toys when something is wrong; I can't see src committers being amused when that happens.

TL;DR---If src is coming to ports, then it needs to abide by all the expectations of the ports tree. If src packages are being built inside of the ports tree, then pkg-building can no longer be managed purely by the ports team, and pkgmgr will need to be created with both ports and src people.

[bapt]

I don't think it belongs to the ports tree at all, everything in the ports should should be verifiable via the ports tree. If one want to bring in the ability to build this kind of base package, the right way to go, it so work on the framework to enable the use some kind of overlays. There has been some work in progress in that area, but nothing has been committed yet.

As far as I am concerned (ports tree side) I am strongly against integrating such things in the ports tree as such.

[kmoore]

I'm going to have some cycles to work on this in the next few days. What I'm seeing and hearing is that the #1 objection is to the lack of distfiles which is outside the norm of ports. There are some process concerns as well about portmgr "shipping" base bits, which is understandable, however for the project to evolve and grow we're going to have to tackle that in the future anyway.

Working on the first issue, what I'm considering is the following:

• Change os/src port to properly pull sources from tagged releases, with a port option that can be set to pick which release to build. I.E. (12.0-RELEASE, 12-STABLE, HEAD, etc)
• This will allow building base packages from a known set of system sources, more inline with how ports behave today.

The downside to this is that when portmgr or anybody rolls packages, they will have to ensure that the correct "base" set is selected that matches the poudriere build jail. Or, alternatively consider boot-strapping the build jail directly from these ports, ala how we do in this patch to poudriere. That is the only way to ensure 1:1 consistency with the version of FreeBSD that the packages are built on will match the base package bits building from the tree.

To tackle the build questions, does it makes sense to consider a formal package base working-group to address those process issues and any other potential problems that may need to be addressed on that side?

[kmoore]

We've had a chance internally to have some further discussion around this topic. Before proceeding on any changes to the port distfiles mechanisms we feel it may be necessary to have some conversations about potential drawbacks and chains of trust.

The current patch was implemented to essentially build and package the current running FreeBSD world/kernel sources that are checked out $somewhere. From the portmgr perspective this would be the poudriere build jail and sources (in /usr/src) within that jail. These are assumed to be "trusted" since they are built from either FreeBSD official sources (git/svn) or pulled directly from the published mirrors in the form of dist-files. If portmgr is concerned that we cannot trust these sources then we may have a bigger issue to worry about since the entire published package sets could be compromised by building on or compiling against suspect OS sources.

Now, if we switch the os/ ports to pulling sources in directly, say from a USE_GITHUB= or other dist-files mechanism this will introduce a new element, which is that the packages produced may *not* be in-sync with the OS that the other ports are built on top of. This could lead to some disastrous runtime errors for user, unless port-mgr or the poudriere builder switches to boot-strapping the build jails from the os/* ports directly. Plus a maintenance cost of dealing with bumping os/* port versions constantly in the case of -STABLE or -HEAD builds. This would require changes to work-flow for an exp-run for example.

On the flip-side, the proposed changes as exists now, would be able to be imported directly into an exp-run and packages would get built properly and work in today's work-flow, enough to produce usable base-packages and allow testing to begin.

Nobody has asked that this become the officially supported set of os packages (yet), that would require much larger discussion around process changes as it relates to portmgr/releng duties. Which I think is a conversation worth having eventually anyway for the project to grow and evolve in the long run. Many of us downstream consumers of FreeBSD view both ports and base as inseparable bits of the same product, and think that some unification of the way we ship those bits will need to eventually happen. Package base seems the ideal jumping off point to start having those discussions in earnest, but that is very much a separate conversation that may need some wider involvement.

Anyway, open to suggestions on where to next take this conversation to reach some consensus around these topics.

[mat]

In D20394#441905, @kmoore wrote:

Anyway, open to suggestions on where to next take this conversation to reach some consensus around these topics.

The consensus is that this does not belong in the ports tree.

[adamw]

In D20394#444436, @mat wrote:

In D20394#441905, @kmoore wrote:

Anyway, open to suggestions on where to next take this conversation to reach some consensus around these topics.

The consensus is that this does not belong in the ports tree.

I want to piggyback on this. The ports tree needs to be about ports, and it just doesn't make sense to mix src and ports in the same tree.

If I were you(pl.), I'd make a new tree for building src packages. It can piggyback ports/Mk, and with the addition of some src-tree Mk/Uses files you should be in business. The process and resources are identical, but it should live in its own tree.

To build pkgs out of src, I would have suggested calling it /usr/pkgsrc but that name means something else :-)

[imp]

In D20394#444447, @adamw wrote:

In D20394#444436, @mat wrote:

In D20394#441905, @kmoore wrote:

Anyway, open to suggestions on where to next take this conversation to reach some consensus around these topics.

The consensus is that this does not belong in the ports tree.

I want to piggyback on this. The ports tree needs to be about ports, and it just doesn't make sense to mix src and ports in the same tree.

If I were you(pl.), I'd make a new tree for building src packages. It can piggyback ports/Mk, and with the addition of some src-tree Mk/Uses files you should be in business. The process and resources are identical, but it should live in its own tree.

To build pkgs out of src, I would have suggested calling it /usr/pkgsrc but that name means something else :-)

I'm having trouble understanding how this is materially different than all the kmod ports relying on /usr/src for their builds, and that's not verified at all either. Also, in the future, we may have a src git repo, and we're looking at having signed commits so we'll know that a tree has integrity.

[adamw]

I'm having trouble understanding how this is materially different than all the kmod ports relying on /usr/src for their builds, and that's not verified at all either. Also, in the future, we may have a src git repo, and we're looking at having signed commits so we'll know that a tree has integrity.

It's different in that kmod ports are 3rd-party software, with precomputed checksums, interdependent with other ports, and committed to by ports committers.

A pristine src repo ensures that ports-only and src-only committers don't step on each others' toes or inadvertently affect areas they don't intend you, facilitates checksum-less builds (if that's really the route you want to go), and makes it possible to check out a src tree without checking out ~30,000 ports directories.

[mat]

In D20394#447314, @imp wrote:

In D20394#444447, @adamw wrote:

In D20394#444436, @mat wrote:

In D20394#441905, @kmoore wrote:

Anyway, open to suggestions on where to next take this conversation to reach some consensus around these topics.

The consensus is that this does not belong in the ports tree.

I want to piggyback on this. The ports tree needs to be about ports, and it just doesn't make sense to mix src and ports in the same tree.

If I were you(pl.), I'd make a new tree for building src packages. It can piggyback ports/Mk, and with the addition of some src-tree Mk/Uses files you should be in business. The process and resources are identical, but it should live in its own tree.

To build pkgs out of src, I would have suggested calling it /usr/pkgsrc but that name means something else :-)

I'm having trouble understanding how this is materially different than all the kmod ports relying on /usr/src for their builds, and that's not verified at all either. Also, in the future, we may have a src git repo, and we're looking at having signed commits so we'll know that a tree has integrity.

Building packages out of the source tree should be self-contained, we already have make packages in there that almost works effort should probably go in that direction. It really does not belong in the ports tree.

[kmoore]

So, what I'm hearing from this most recent comment, is that because there's an alternative implementation in src, we shouldn't be allowed to explore and/or use other implementations? Is this the new rule for FreeBSD ports? If so, should we start removing all the alternative implementations of other things from ports as well? We could greatly simplify if we reduced down to a single supported implementation of other things, such as a compiler, sound-system, desktop, browser, SSL, chat program, etc.

Obviously that is completely ridiculous. We have many ways of getting base "bits" today already. This is just another one. It may work for lots of users in some use-cases, just as other implementations may work in other use-cases.

At the end of the day, our proposed patch:

1. Does not impact any other ports / packages
2. Does not require any process change in how we build packages
3. Does not force anybody to use our implementation over any other (As you are suggesting we do ironically)

So please forgive me for being insistent on this, but I'm failing to see any real issue here.

If I may cut straight to the chase, it seems this is mostly an objection over the political concern that this is another implementation which may or may not hinder efforts on the other base package work. Nowhere have I suggested that interested parties cannot continue to work on that implementation. I'm not even making an argument for which one should be officially "supported", only asking for a chance to give back to the community and let users run whatever is most useful to them.

[bapt]

What I think is the matter of how the base system is turned into packages should be a project decision, and yes I would like to see some broader discussion about before adding all of this in the ports tree. Further more it adds lots of maintenance to portmgr to the tree. if you were to compare it with something it would be comparable to the base/ fake category. And I as the one who added it in the ports would gladly remove it if that is what is proposed. With a side note on this case, that it was added to avoid duplication with other binutils and gcc ports in the first place and as always been thought to only concern those 2 things.

regarding this patch, it can perfectly leave in a different tree that the ports tree, given all it needs beside the "vital" flags part is bsd.port.mk which any of those ports should be able to find easily.

[kmoore]

In D20394#447608, @bapt wrote:

What I think is the matter of how the base system is turned into packages should be a project decision, and yes I would like to see some broader discussion about before adding all of this in the ports tree. Further more it adds lots of maintenance to portmgr to the tree. if you were to compare it with something it would be comparable to the base/ fake category. And I as the one who added it in the ports would gladly remove it if that is what is proposed. With a side note on this case, that it was added to avoid duplication with other binutils and gcc ports in the first place and as always been thought to only concern those 2 things.

I'm fine with having those conversations. I'm curious about the possible maintenance burden to portmgr though. Since it doesn't really touch anything else, what is portmgr on the hook for? That the packages themselves fail to build? Would portmgr handle that directly, or won't those bugs get sent to the maintainer (me in this case)? Or is there some implicit quality guarantee by virtue of having packages available in the "FreeBSD" repo? If there is, that would be news to many of us, since its not at all uncommon to pkg install things that are broken. Would a strong pkg-message warning that these are not "Quality-Tested base packages" suffice in this case?

regarding this patch, it can perfectly leave in a different tree that the ports tree, given all it needs beside the "vital" flags part is bsd.port.mk which any of those ports should be able to find easily.

Its already in a bunch of different "trees" today, we have it in TrueOS/FreeNAS/Trident/GhostBSD/PenBSD. The whole idea here is to get it into upstream so that packages get built, so a wider audience can also test and use them, help find issues, etc. Putting it into yet another tree does not accomplish that, even though it is technically very easy.

[kmoore]

This diff brings in some of the latest changes as it relates to base packages. Specifically:

• Cleanup of some whitespace and other house-keeping
• Adds a new 'base-update' tool/script which is included with the userland-base port/package.

This tool is an analog to 'freebsd-update' in the sense that it allows you to:

• Check for updates to base
• Perform updates to base (In a safe-manner, kernel first, followed by rtld, libc, and other "critical" libraries, then the rest of base)
• Merge conflicts with files in /etc that pkg couldn't auto-merge.

[adamw]

os/buildkernel-debug/Makefile

73

$(STAGEDIR} ends with a slash. This should be ${STAGEDIR}kernel-debug etc.

80

${TAR}

81

${CHFLAGS} ... ${STAGEDIR}kernel-debug

os/buildworld/Makefile

77

Files should always be installed into STAGEDIR with ${INSTALL_DATA/SCRIPT/LIB/KLD/MAN}

os/kernel-debug-symbols/Makefile

39

Can the commands be unmuted? As a rule, we show every step except mkdir. It’s nice for users monitoring the build to know what’s happening.

os/userland-base/Makefile

16

LOCALBASE/bin is automatically searched. This can just be:

jq:textproc/jq

25

Why not put NO_OPTIONS_SORT in Makefile.common or .options.mk?

[kmoore]

Adam - Updated the patch to reflect your comments. Appreciate the review.

FYI - CHFLAGS didn't exist in bsd.commands.mk, added it.

[adamw]

So, I'm just going to throw this out here. This is just a neat feature of the ports tree that the src framework might choose to leverage.

Many src OPTIONS will necessarily depend upon each other. For instance, enabling ZFS but not CDDL will (afaik) not produce working zfs. Similarly, TELNET without INET will not produce a working telnet. I know that there are a great number of options here, and that this would add some complexity, but it might save some users from shooting themselves in the foot.

We have a syntax that specifies that one option requires another, or that enabling one option should prevent another option from being enabled.

ZFS_IMPLIES=         CDDL
SENDMAIL_IMPLIES=    MAIL MAILWRAPPER INET

Here, if SENDMAIL is enabled, once the options form is submitted, MAIL, MAILWRAPPER, and INET will be automatically enabled.

SENDMAIL_PREVENTS=    RATIONAL_THOUGHT

Here, if SENDMAIL and RATIONAL_THOUGHT are both enabled when the options form is submitted, the build will die with a (configurable) error explaining that those options are incompatible.

[kmoore]

Address a few more comments from Adamw

[kmoore]

In D20394#449200, @adamw wrote:

So, I'm just going to throw this out here. This is just a neat feature of the ports tree that the src framework might choose to leverage.

Many src OPTIONS will necessarily depend upon each other. For instance, enabling ZFS but not CDDL will (afaik) not produce working zfs. Similarly, TELNET without INET will not produce a working telnet. I know that there are a great number of options here, and that this would add some complexity, but it might save some users from shooting themselves in the foot.

We have a syntax that specifies that one option requires another, or that enabling one option should prevent another option from being enabled.

ZFS_IMPLIES=         CDDL
SENDMAIL_IMPLIES=    MAIL MAILWRAPPER INET

Here, if SENDMAIL is enabled, once the options form is submitted, MAIL, MAILWRAPPER, and INET will be automatically enabled.

SENDMAIL_PREVENTS=    RATIONAL_THOUGHT

Here, if SENDMAIL and RATIONAL_THOUGHT are both enabled when the options form is submitted, the build will die with a (configurable) error explaining that those options are incompatible.

@adamw

Yes, this is something really nice about the ports framework. We started with a more basic list of options, so you can run 'make config' and select your knobs at will. But we've anticipated going back and figuring out the groupings and conflicts between the options and setting those as well. We can see there being a lot of users who appreciate being able to see what options are available via the dialog menus, but also have that guidance on which ones are safe to set/unset as they build a custom kernel/userland or whatever.

[imp]

In D20394#449248, @kmoore wrote:

In D20394#449200, @adamw wrote:

So, I'm just going to throw this out here. This is just a neat feature of the ports tree that the src framework might choose to leverage.

Many src OPTIONS will necessarily depend upon each other. For instance, enabling ZFS but not CDDL will (afaik) not produce working zfs. Similarly, TELNET without INET will not produce a working telnet. I know that there are a great number of options here, and that this would add some complexity, but it might save some users from shooting themselves in the foot.

We have a syntax that specifies that one option requires another, or that enabling one option should prevent another option from being enabled.

ZFS_IMPLIES=         CDDL
SENDMAIL_IMPLIES=    MAIL MAILWRAPPER INET

Here, if SENDMAIL is enabled, once the options form is submitted, MAIL, MAILWRAPPER, and INET will be automatically enabled.

SENDMAIL_PREVENTS=    RATIONAL_THOUGHT

Here, if SENDMAIL and RATIONAL_THOUGHT are both enabled when the options form is submitted, the build will die with a (configurable) error explaining that those options are incompatible.

@adamw

Yes, this is something really nice about the ports framework. We started with a more basic list of options, so you can run 'make config' and select your knobs at will. But we've anticipated going back and figuring out the groupings and conflicts between the options and setting those as well. We can see there being a lot of users who appreciate being able to see what options are available via the dialog menus, but also have that guidance on which ones are safe to set/unset as they build a custom kernel/userland or whatever.

If we're going full on src options, then we have most, if not all, of this information available in the base system. If we don't, then we should add it there and generate this stuff, if possible. This may help keep down the hard coding and could also enable other src config tools that might not be tied to packages or the ports system. The usual worry here is overwhelming the user with options because we have so many...

[adamw]

In D20394#449271, @imp wrote:

The usual worry here is overwhelming the user with options because we have so many...

There are indeed a ton of options. Two things you can do to help:

• Use OPTIONS groups. Put the filesystem stuff in a group, put the NIC stuff in a group, the network services in a group, etc. It's easy to skip over groups you don't care about.

• We support a file named pkg-help in the main port dir. When it exists, users get a message to press ^E to view that file. You could put advice in there. Even listing which manpages are the most useful would be quite helpful for end-users.

[kmoore]

This update breaks the userland package down into some smaller sub-packages. This addresses concerns that 1-2 monolithic userland packages are "too big" to download on slow connections, especially in the case of just a single file update. The new compressed package sizes (on HEAD/amd64) are roughly as follows:

40M	os-generic-kernel
1M	os-generic-userland
9M	os-generic-userland-base
1M	os-generic-userland-base-bootstrap
22M	os-generic-userland-bin
1M	os-generic-userland-boot
1M	os-generic-userland-conf
153M	os-generic-userland-debug
99M	os-generic-userland-devel
12M	os-generic-userland-docs
8M	os-generic-userland-lib
8M	os-generic-userland-lib32
51M	os-generic-userland-lib32-devel
5M	os-generic-userland-rescue
7M	os-generic-userland-sbin
8M	os-generic-userland-tests

[adamw]

Thanks, those new categories make more sense to me.

Can we please clean up user-facing strings?

• COMMENT strings like Buildkernel portition of os packages tell me nothing about whether it contains stuff I want to install (plus "portition" is a typo).

• Some of the COMMENT strings reference it being for trueos.

• The pkg-descr text here is one line, and usually a direct recitation of the COMMENT string. What does it install? What is its purpose?
  
  This FreeBSD OS port installs programs, libraries, and header files that are used to build other software. Some major programs like clang, make, and lldb are included. If you want to build and/or compile anything, you will need to install this port. Restricted jails may want to skip this port, to minimize the arsenal available to intruders.

• The WWW for every port here is trueos. I know that it's being developed by our trueos brothers and sisters, but casual users may look at it and decide that these ports don't apply to their FreeBSD box.

[adamw]

I should add here:

Because the pkg-plist is being dynamically generated here, the pkg-descr is our only opportunity to make it clear what components are installed by each port. This is an essential service that I'd love to see us provide for our users.

Perhaps we could do one or both of:

1. Define in the descr specifically what kind of software is in each port

2. List all the important components in the descr

[kmoore]

Expand the COMMENT sections and pkg-descr files thanks to feedback from AdamW.

[kmoore]

Update diff, remove a src-head/ port which was added by mistake from a previous git branch.

[kmoore]

Updated patch to include 'os/ports' port as well. This is something I've personally wanted in FreeBSD for a LONG time. The ability to 'pkg install ports' and get a ports tree installed which exactly matches the packages I'm running. Over the years its been very frustrating to need to rebuild something from ports, only to find out that the tree I've grabbed from portsnap, git or subversion doesn't match, and my build is now trying to upgrade things I didn't want to upgrade.

[adamw]

I appreciate the work there, Kris, but a ports tree package is simply counterproductive. I am 100% opposed to a ports tree being in the release images in the first place; giving users another way to install a stale version of the tree is setting users up for failure. We do not support stale ports trees in any way; we shouldn't make it possible for users to get them in the first place.

Ports trees should only be installed through means that allow users to keep it up-to-date. Having a ports tree package in os/ is simply overstepping its mandate, and is precisely emblematic of the reservations that portmgr has about os/ living in the ports tree.

We already have several simple ways to get a ports tree in one command, including portsnap fetch extract, that provide a system that can stay up-to-date. It could even be theoretically possible to make /usr/ports a completely empty shallow checkout, so that people could cd /usr/ports && svnlite co and get the entire tree right away.

[kmoore]

Adam,

I'll be happy to back out that bit if its a huge concern, but let me first lay out the scenario we're trying to solve, and I'm curious to know if there's some better way to solve this that we've been unaware of for years.

First of all, this isn't for developers, this is looking at things with my "end-user" hat on. We've for years had a mantra that you don't mix ports and packages. Packages tend to lag too far behind ports (for understandable reasons, takes time to build them after all). However this has lead to a LOT of frustration for our IT staff. For example, we like most FreeBSD users try to run services inside jails. I can't tell you how many times we've had to setup and maintain some *AMP stack, and naturally we gravitate towards using the project-supplied packages for this task. However more times than I can count, we run into the situation where we need just 1, or maybe 2 packages built with some different combination of flags (PHP I'm looking at you here). This often has led to much frustration where trying to go into ports and re-compile that single port breaks down, since in between the week where packages were built, and we performed the upgrade something in upstream ports has changed, and now instead of rebuilding just the one package, we're having to re-compile large parts of all the packages we just installed / upgraded. After years of doing this, we've had IT teams then go spin up their own poudriere infrastructure to build their own packages, however that's certainly not very "user friendly" and has led to more than one awkward conversation where some disgruntled IT-type asks why is this so difficult on FreeBSD, we could run something on Linux far easier :(

This potential "solution" at least helps solve one huge challenge for that scenario, since getting a ports tree thats consistent with the packages we're running would have made this process far less painful, so much so we could have scripted solutions to run "pkg upgrade" along with rebuilding a single port and kept jails updated much easier. Granted, if you are security paranoid and want the absolute latest ports all the time, this won't work for you, but then again why the heck are you running project-provided packages in the first place?

Anyway, I'm curious if you or others have had some solution to this that we've missed for years now, beyond just "spin up your own poudriere/synth build". I'll be happy to remove this from the review, but we'd probably keep it in TrueOS since I want some way to see the exact ports tree that my packages came from, without needing to go run 'svn/git' and figure out what the right revision was.

[swills]

In D20394#451916, @kmoore wrote:

Adam,

I'll be happy to back out that bit if its a huge concern, but let me first lay out the scenario we're trying to solve, and I'm curious to know if there's some better way to solve this that we've been unaware of for years.

First of all, this isn't for developers, this is looking at things with my "end-user" hat on. We've for years had a mantra that you don't mix ports and packages. Packages tend to lag too far behind ports (for understandable reasons, takes time to build them after all).

We have separate branches so we have quarterly "stable" packages, not because of build time. You can switch to latest packages if you want or the quarterly branch of ports if you want and build the packages you need from there.

However this has lead to a LOT of frustration for our IT staff. For example, we like most FreeBSD users try to run services inside jails. I can't tell you how many times we've had to setup and maintain some *AMP stack, and naturally we gravitate towards using the project-supplied packages for this task. However more times than I can count, we run into the situation where we need just 1, or maybe 2 packages built with some different combination of flags (PHP I'm looking at you here). This often has led to much frustration where trying to go into ports and re-compile that single port breaks down, since in between the week where packages were built, and we performed the upgrade something in upstream ports has changed, and now instead of rebuilding just the one package, we're having to re-compile large parts of all the packages we just installed / upgraded.

You could also consider submitting patches which change the defaults and see if the maintainer will accept them, and if not, create an OPTIONS_SLAVE port that has what you need turned on.

After years of doing this, we've had IT teams then go spin up their own poudriere infrastructure to build their own packages, however that's certainly not very "user friendly" and has led to more than one awkward conversation where some disgruntled IT-type asks why is this so difficult on FreeBSD, we could run something on Linux far easier :(

This potential "solution" at least helps solve one huge challenge for that scenario, since getting a ports tree thats consistent with the packages we're running would have made this process far less painful, so much so we could have scripted solutions to run "pkg upgrade" along with rebuilding a single port and kept jails updated much easier. Granted, if you are security paranoid and want the absolute latest ports all the time, this won't work for you, but then again why the heck are you running project-provided packages in the first place?

We would welcome help identifying security updates that need to be merged to the quarterly branch, and also ones that need to be committed to the head branch and documented in vuxml.

Anyway, I'm curious if you or others have had some solution to this that we've missed for years now, beyond just "spin up your own poudriere/synth build". I'll be happy to remove this from the review, but we'd probably keep it in TrueOS since I want some way to see the exact ports tree that my packages came from, without needing to go run 'svn/git' and figure out what the right revision was.

I do run a poudriere build in order to build customized packages on top of a customized OS, I don't see it as a major roadblock personally. I find poudriere very easy to setup and use, and with the right config it builds things pretty quickly, really as quickly as possible given the requirements, I think.

Back to the topic at hand, this review just seems too large to me. I would like us to at least consider breaking it up into smaller pieces.

By the way, SUBDIR in ports/Makefile and VALID_CATEGORIES in Mk/bsd.port.mk are both defined with += from the start, so you can define additions in /etc/make.conf and they won't be overridden. I'm not saying this is a solution to the whole base package problem of course, but it might be interesting to consider it in conjunction with a git sub-module.

[swills]

Another issue to consider, -devel in package/port name already means something else in ports (development version), so perhaps a different name for those is needed. Personally, I dislike not having all of a thing when I install it and I don't like getting errors when I go to build something because I have to install headers, so having headers in a separate package isn't something I'm terribly fond of, but if it's really required maybe it is OK.

[kmoore]

In D20394#451999, @swills wrote:

In D20394#451916, @kmoore wrote:

Adam,

I'll be happy to back out that bit if its a huge concern, but let me first lay out the scenario we're trying to solve, and I'm curious to know if there's some better way to solve this that we've been unaware of for years.

First of all, this isn't for developers, this is looking at things with my "end-user" hat on. We've for years had a mantra that you don't mix ports and packages. Packages tend to lag too far behind ports (for understandable reasons, takes time to build them after all).

We have separate branches so we have quarterly "stable" packages, not because of build time. You can switch to latest packages if you want or the quarterly branch of ports if you want and build the packages you need from there.

Very true, stable does move slower. We've gone back and forth on them, but at the moment have settled on running more of the HEAD ports / packages.

However this has lead to a LOT of frustration for our IT staff. For example, we like most FreeBSD users try to run services inside jails. I can't tell you how many times we've had to setup and maintain some *AMP stack, and naturally we gravitate towards using the project-supplied packages for this task. However more times than I can count, we run into the situation where we need just 1, or maybe 2 packages built with some different combination of flags (PHP I'm looking at you here). This often has led to much frustration where trying to go into ports and re-compile that single port breaks down, since in between the week where packages were built, and we performed the upgrade something in upstream ports has changed, and now instead of rebuilding just the one package, we're having to re-compile large parts of all the packages we just installed / upgraded.

You could also consider submitting patches which change the defaults and see if the maintainer will accept them, and if not, create an OPTIONS_SLAVE port that has what you need turned on.

That's fine for you or me (I usually do slave-ports), but we're talking about end-users, IT staff or other folks who don't necessarily have the time or inclination to do that :)

After years of doing this, we've had IT teams then go spin up their own poudriere infrastructure to build their own packages, however that's certainly not very "user friendly" and has led to more than one awkward conversation where some disgruntled IT-type asks why is this so difficult on FreeBSD, we could run something on Linux far easier :(

This potential "solution" at least helps solve one huge challenge for that scenario, since getting a ports tree thats consistent with the packages we're running would have made this process far less painful, so much so we could have scripted solutions to run "pkg upgrade" along with rebuilding a single port and kept jails updated much easier. Granted, if you are security paranoid and want the absolute latest ports all the time, this won't work for you, but then again why the heck are you running project-provided packages in the first place?

We would welcome help identifying security updates that need to be merged to the quarterly branch, and also ones that need to be committed to the head branch and documented in vuxml.

We may end up taking you up on that going forward. @miwi is now on SO and will be helping us audit and keep our ports tree updated with security fixes for FreeNAS.

Anyway, I'm curious if you or others have had some solution to this that we've missed for years now, beyond just "spin up your own poudriere/synth build". I'll be happy to remove this from the review, but we'd probably keep it in TrueOS since I want some way to see the exact ports tree that my packages came from, without needing to go run 'svn/git' and figure out what the right revision was.

I do run a poudriere build in order to build customized packages on top of a customized OS, I don't see it as a major roadblock personally. I find poudriere very easy to setup and use, and with the right config it builds things pretty quickly, really as quickly as possible given the requirements, I think.

Huge fan of poudriere personally and thats how I've solved these problems here. But I'm talking about my IT folks or those who maybe don't have time / patience or a system around to go run poudriere on. They have a jail, want to update it and often just need to rebuild a single port with a different flag or option. But maybe going forward we'll try to introduce them more to that workflow to see how it goes.

Back to the topic at hand, this review just seems too large to me. I would like us to at least consider breaking it up into smaller pieces.

By the way, SUBDIR in ports/Makefile and VALID_CATEGORIES in Mk/bsd.port.mk are both defined with += from the start, so you can define additions in /etc/make.conf and they won't be overridden. I'm not saying this is a solution to the whole base package problem of course, but it might be interesting to consider it in conjunction with a git sub-module.

I'd prefer to not break it apart. I started with that approach submitting a single patch for review to enable the VITAL flag for base packages. That got shot down until we submitted something in full context, which we've done here :)

But yes, agreed, the SUBDIR / VALID_CATEGORIES could be used fairly easily. Its the changed bits in Mk/Scripts and Mk/Uses and bsd.port.mk that are somewhat more tricky to deal with.

[kmoore]

In D20394#452000, @swills wrote:

Another issue to consider, -devel in package/port name already means something else in ports (development version), so perhaps a different name for those is needed. Personally, I dislike not having all of a thing when I install it and I don't like getting errors when I go to build something because I have to install headers, so having headers in a separate package isn't something I'm terribly fond of, but if it's really required maybe it is OK.

100% agree, I find it somewhat confusing as well. Here's the possible names I can think of:

"-compiler"
"-buildtools"
"-devtools"
"-development"

I don't really have any strong preference for one vs another, any thoughts on your end which makes the most sense?

And yes, this single package includes the compiler/linker bits, along with headers and other files required to do building on FreeBSD. Single package to enable the build system in its entirety.

[swills]

In D20394#452005, @kmoore wrote:

In D20394#452000, @swills wrote:

Another issue to consider, -devel in package/port name already means something else in ports (development version), so perhaps a different name for those is needed. Personally, I dislike not having all of a thing when I install it and I don't like getting errors when I go to build something because I have to install headers, so having headers in a separate package isn't something I'm terribly fond of, but if it's really required maybe it is OK.

100% agree, I find it somewhat confusing as well. Here's the possible names I can think of:

"-compiler"
"-buildtools"
"-devtools"
"-development"

I don't really have any strong preference for one vs another, any thoughts on your end which makes the most sense?

And yes, this single package includes the compiler/linker bits, along with headers and other files required to do building on FreeBSD. Single package to enable the build system in its entirety.

-devtools might be OK.

[adamw]

In D20394#451916, @kmoore wrote:

I'll be happy to back out that bit if its a huge concern, but let me first lay out the scenario we're trying to solve, and I'm curious to know if there's some better way to solve this that we've been unaware of for years.

Hi Kris,

I don't personally understand your use-case yet, but that's okay. We can spend some more time brainstorming and implementing an approach together.

For now though, I suggest that you back that part out. Let's keep this review just about building base pkgs, and move that to a separate review. Base pkgs and portstree pkgs shouldn't be dependent on each other. Is that ok?

[adamw]

In D20394#451999, @swills wrote:

By the way, SUBDIR in ports/Makefile and VALID_CATEGORIES in Mk/bsd.port.mk are both defined with += from the start, so you can define additions in /etc/make.conf and they won't be overridden.

Piggybacking on this statement, Kris I think you should keep VALID_CATEGORIES+=os, but remove SUBDIR+=os. Here's my reasoning.

VALID_CATEGORIES makes it possible to build ports with CATEGORIES=os. But SUBDIR+=os includes it in distfile, INDEX, and pointyhat pkg runs. If we drop the SUBDIR+=os, then ports pkg runs won't build the os/ ports, and when base pkg runs occur, poudriere will still happily build things even if they're not listed in SUBDIR.

Tl;dr--Please remove SUBDIR+=os from /usr/ports/Makefile. That will be the key to building ports and base pkgs separately without doing pointyhat cluster contortions.

[kmoore]

Remove 'os' from SUBDIR+=
Remove 'ports/' port
Rename userland-devel -> userland-devtools

[kmoore]

In D20394#452020, @swills wrote:

In D20394#452005, @kmoore wrote:

In D20394#452000, @swills wrote:

Another issue to consider, -devel in package/port name already means something else in ports (development version), so perhaps a different name for those is needed. Personally, I dislike not having all of a thing when I install it and I don't like getting errors when I go to build something because I have to install headers, so having headers in a separate package isn't something I'm terribly fond of, but if it's really required maybe it is OK.

100% agree, I find it somewhat confusing as well. Here's the possible names I can think of:

"-compiler"
"-buildtools"
"-devtools"
"-development"

I don't really have any strong preference for one vs another, any thoughts on your end which makes the most sense?

And yes, this single package includes the compiler/linker bits, along with headers and other files required to do building on FreeBSD. Single package to enable the build system in its entirety.

-devtools might be OK.

Works for me. Done!

[kmoore]

In D20394#452073, @adamw wrote:

In D20394#451916, @kmoore wrote:

I'll be happy to back out that bit if its a huge concern, but let me first lay out the scenario we're trying to solve, and I'm curious to know if there's some better way to solve this that we've been unaware of for years.

Hi Kris,

I don't personally understand your use-case yet, but that's okay. We can spend some more time brainstorming and implementing an approach together.

For now though, I suggest that you back that part out. Let's keep this review just about building base pkgs, and move that to a separate review. Base pkgs and portstree pkgs shouldn't be dependent on each other. Is that ok?

Yep, no worries. Removed!

[kmoore]

Update to the 'base-update' utility:

• Fixes issue on userland update not getting all the newly split out packages
• Adds ability to list available flavors and change between them.

[kmoore]

After discussing with the team, we're going to go ahead and abandon this revision. While we wanted to see something usable happen in this space, it's just not cost or time effective to spend far more effort fighting politics than the actual implementation took. We will continue using a non-package solution for our products for the future.

Thanks!