release: Build the release media from packages

[ivy]

Summary

Instead of using installworld to build the release media, use the
base package repository we already created.  This is much faster
than running installworld again, and in future will allow us to
build install media from an external package repository.

The sets used for each medium are:

* disc1: kernels, optional.  This provides a basic live system.

* bootonly: kernels, minimal.  This saves about 100MB compared to
installing optional.

* dvd: kernels, optional, devel.  We might want to drop devel in
the future to include more packages, but for now, this matches
the current contents of the medium.

Since the new install-system target always installs pkg, we can also
simplify the target-specific logic a bit to only handle the extra
packages.

Keep the existing non-package method for the NOPKGBASE case so that
downstream users can continue building distset install media without
having to depend on pkg.  This should be removed once we vendor pkg.

Add a warning about distset support being deprecated.  This is only
printed if NOPKGBASE is set and can be disabled entirely by setting
RELEASE_SKIP_DISTSET_WARNING.

[ivy]

fix the metalog mode for /var/db/services.db

(although this doesn't really matter as /var is tmpfs in the installer environment.)

[emaste]

release/Makefile

48

Is this because we can use the dvd as a somewhat usable live system as well?

50

IMO we should uncomment this - the bootonly is intended to be as small as possible including only what is necessary for remote installation.

329

When we switch to building release images from pkgbase this code path will start to bitrot. We should probably emit a warning indicating that this is no longer regularly tested.

[ivy]

release/Makefile

48

i did wonder about that - does anyone actually use the compiler on the DVD? would it be useful to remove it (all the devel libraries are quite big) and put more packages on there?

i'm happy to remove devel, but i thought it was best to start by replicating what we already have.

[cperciva]

release/Makefile

48

I agree with replicating what we already have as a starting point; but yes, we should have a conversation about release images, possibly in the direction of separating "install media" from "live system".

[emaste]

Yes, agreed on starting with an approximation that is as close as possible w/ pkgbase to what we have today.

[ivy]

use minimal for bootonly by default

print a warning about distsets if NOPKGBASE is set

instead of hardcoding bsdinstall and bsdconfig in install-system, add
a new variable, RELEASE_BASE_PACKAGES

[ivy]

use a "." form for the release sets, so RELEASE_MEDIA_SETS_BOOTONLY
becomes RELEASE_MEDIA_SETS.bootonly. this makes it more clear that
it can be set for any media type.

[ivy]

are there any objections to landing this? i'd like to get started on multiple kernel support for bsdinstall, and it would be easier to get the other release changes in first.

[cperciva]

In D54542#1250350, @ivy wrote:

are there any objections to landing this? i'd like to get started on multiple kernel support for bsdinstall, and it would be easier to get the other release changes in first.

I haven't reviewed the code but in principle I'm ok with the change. Please check with the CHERI people in case this will cause problems for them though -- I think they were holding off on pkgbase for a bit, and they're an important downstream for the project.

Not sure if you were planning on MFCing this, but if you are we should discuss further before that happens. There has been some release build breakage in HEAD lately so I want to make sure all the issues are fixed there before anything gets MFCed.

[emaste]

I want to make sure all the issues are fixed there before anything gets MFCed.

My preference is for all of the release building changes to be merged to stable/15, but after a significant amount of time & testing in main. Perhaps after the 14.4 process is complete?

[cperciva]

In D54542#1250514, @emaste wrote:

I want to make sure all the issues are fixed there before anything gets MFCed.

My preference is for all of the release building changes to be merged to stable/15, but after a significant amount of time & testing in main. Perhaps after the 14.4 process is complete?

Probably not necessary to wait that long. I just want to have a couple weeks of 16.x snapshots building successfully first.

[ivy]

In D54542#1250509, @cperciva wrote:

Not sure if you were planning on MFCing this, but if you are we should discuss further before that happens. There has been some release build breakage in HEAD lately so I want to make sure all the issues are fixed there before anything gets MFCed.

i wasn't planning to MFC this because it changes the release process in a surprising way. is that the sort of change we want to make in a release branch?

[ivy]

In D54542#1250509, @cperciva wrote:

Please check with the CHERI people in case this will cause problems for them though -- I think they were holding off on pkgbase for a bit, and they're an important downstream for the project.

i tried to be careful not to break the distset release build and i did test it while developing this. i also previously confirmed on IRC that this approach is acceptable to them in principle. still, it's possible i missed something since i don't use / care about dist sets myself.

@jrtc27 any objections here?

[ivy]

would anyone like to review this? are there any objections?

[imp]

Not on the RE team, but the build it from pkg lines look fine.

Honestly, though, we should see how the Nanobsd retooling stuff goes this summer with GSoC to see if we can move a lot of the repeated code that's added here can be moved to one place instead of a dozen... But that's more a 'Check back in September and we might need a small redo' than anything actionable today.

[ivy]

In D54542#1282756, @imp wrote:

Not on the RE team, but the build it from pkg lines look fine.

Honestly, though, we should see how the Nanobsd retooling stuff goes this summer with GSoC to see if we can move a lot of the repeated code that's added here can be moved to one place instead of a dozen... But that's more a 'Check back in September and we might need a small redo' than anything actionable today.

i've been working on this since last year (this review was opened on Jan 5) and frankly "let's wait until summer" makes me want to just stop working on pkgbase because why bother if no one cares or someone else is going to do it in a completely different way?

that's not an attack on you, i understand your position. and i don't know what's going on with GSoC, maybe someone implemented this in a better way already. but it's frustrating to spend hours and hours working on improving the system and to get no feedback or the feedback is "well, you submitted this change six months ago and now it's overcome by events". you know?

i think a lot of the existing pkgbase code is bad, and yes, redundant, and i have branches on my local system to fix that and make it better, but i can't commit it because it depends on existing reviews like this one, so everything is stalled due to lack of review. it's frustrating. the same thing happened with bridge(4), and i ended up committing a bunch of changes without review because no one would review it, but i feel less comfortable doing that with pkgbase/releng related stuff.

[imp]

In D54542#1282766, @ivy wrote:

In D54542#1282756, @imp wrote:

Not on the RE team, but the build it from pkg lines look fine.

Honestly, though, we should see how the Nanobsd retooling stuff goes this summer with GSoC to see if we can move a lot of the repeated code that's added here can be moved to one place instead of a dozen... But that's more a 'Check back in September and we might need a small redo' than anything actionable today.

i've been working on this since last year (this review was opened on Jan 5) and frankly "let's wait until summer" makes me want to just stop working on pkgbase because why bother if no one cares or someone else is going to do it in a completely different way?

that's not an attack on you, i understand your position. and i don't know what's going on with GSoC, maybe someone implemented this in a better way already. but it's frustrating to spend hours and hours working on improving the system and to get no feedback or the feedback is "well, you submitted this change six months ago and now it's overcome by events". you know?

i think a lot of the existing pkgbase code is bad, and yes, redundant, and i have branches on my local system to fix that and make it better, but i can't commit it because it depends on existing reviews like this one, so everything is stalled due to lack of review. it's frustrating. the same thing happened with bridge(4), and i ended up committing a bunch of changes without review because no one would review it, but i feel less comfortable doing that with pkgbase/releng related stuff.

This isn't "let's wait til summer" but "let's move ahead with this now and make a note to refactor at the end of summer if there's something there we can use".

[ivy]

okay, i understood "we should see how the Nanobsd retooling stuff goes this summer" to mean we should wait until that code appears and how it affects this diff. if you're saying we should land this as-is then i don't understand why you mentioned that. but in any case i appreciate the review, thanks.

but i'd really like it if Colin or someone else from releng could review it too... but otherwise i intend to land this in a week or two.

[imp]

In D54542#1282768, @ivy wrote:

okay, i understood "we should see how the Nanobsd retooling stuff goes this summer" to mean we should wait until that code appears and how it affects this diff. if you're saying we should land this as-is then i don't understand why you mentioned that. but in any case i appreciate the review, thanks.

Mostly as a heads up so you know of other work in related areas. Sorry for doing it in a confusing way.

but i'd really like it if Colin or someone else from releng could review it too... but otherwise i intend to land this in a week or two.

That sounds like a good plan.

[emaste]

release/Makefile

43

maybe "used to create the release media" to make it very explicit?

286

We should have a tracking ticket (somewhere?) for this