release: install wireless firmware onto disc1 and dvd

[bz]

Summary

Wireless driver firmware is no longer added to the src tree.
In order to have wireless support in the installer for the new drivers
we install the firmware packages onto disc1 (and memstick) and dvd.

Sponsored by: The FreeBSD Foundation
Submitted by: cperciva (the commands and where to place them)
MFC after: 3 days

Test Plan

Done a build as root on amd64 with
make -C /usr/src/bz_wifi_release/release WITH_DVD=yes clean release

Need to boot the various artefacts still.

[bz]

Given rtw88 and iwlwifi still have firmware in src.git I ran a test with rtw89 which came up booting disc1 as well.
So this part works.

Trying to load binary firmware from /boot/firmware/rtw89/rtw8852a_fw.bin
firmware: '/boot/firmware/rtw89/rtw8852a_fw.bin' version 0: 1423232 bytes loaded at 0xfffffe0051e00000
rtw89/rtw8852a_fw.bin: Loaded binary firmware using /boot/firmware/rtw89/rtw8852a_fw.bin

[bz]

I also tried memstick and dvd1; I manually (so far) afterwards ran fwget from the shell (and in case of rtw8x added the tunable to loader.conf) and the system came up fine after a reboot.

@cperciva releng do we need to test the NO_ROOT bits as well (I saw other bits are not implemented for it yet either).

[jrtc27]

Please don't insert unconditional use of pkg for disc1, it will break the build for non-FreeBSD systems as they normally don't have a pkg binary. Whilst eventually we'll need to figure out a story there for pkgbase, I suggest just guarding it by NO_ROOT for now. It really exists to support non-FreeBSD where NO_ROOT is required, though eventually would ideally be used for the real release builds on FreeBSD rather than requiring root.

[cperciva]

@jrtc27 We should make this conditional on "is x86 or arm64" since there's no guarantee that the FreeBSD project will have package sets for any other platforms. Would that address your concern about the availability of a pkg binary?

[jrtc27]

In D47407#1081558, @cperciva wrote:

@jrtc27 We should make this conditional on "is x86 or arm64" since there's no guarantee that the FreeBSD project will have package sets for any other platforms. Would that address your concern about the availability of a pkg binary?

Well, x86 and arm64 macOS/Linux will still fall under that. But yes, making sure it doesn't fail for architectures without package sets is also a good point.

[emaste]

FYI in the new year I plan to start work on making the whole artifact build process NO_ROOT. If we need to use NO_ROOT as a flag that changes the built artifacts please make the missing functionality explicit (e.g. by leaving the NO_ROOT case with a commented body, or so).

Or even better, have it controlled by whether the build host is FreeBSD or a new flag that controls these cases.

[bz]

How do we envision a build/install media creation for pkgbase in the future? I would highly assume you'll need to sort that issue somehow anyway.

That said we use pkg for VM images already to my understanding; the simple fact that they use shell scripts but are not integrated into a unified build process just makes it a mood point.

From all I can tell by now and after a brief discussion the artifacts we build today are mostly of historic nature and need overhaul anyway (disc1 being way to big for a CD, memstick not having the pkg-set on it but only the optional dvd1, ...). VM images looking like a 2nd class citizen.

I would love this all to be NO_ROOT @emaste and I don't agree with the view that NO_ROOT is for non-FreeBSD.

This is all discussions for another day and place though I assume.

What .if should we really put around it now?

.if ${.MAKE.OS} == "FreeBSD"

[cperciva]

In D47407#1081562, @jrtc27 wrote:

In D47407#1081558, @cperciva wrote:

@jrtc27 We should make this conditional on "is x86 or arm64" since there's no guarantee that the FreeBSD project will have package sets for any other platforms. Would that address your concern about the availability of a pkg binary?

Well, x86 and arm64 macOS/Linux will still fall under that. But yes, making sure it doesn't fail for architectures without package sets is also a good point.

Oh, people are building FreeBSD release images on non-FreeBSD? I didn't realize that. This sounds like even more reason to have pkg in the base system (and as a cross-tool).

[bz]

Guard pkg invokations by only running them on FreeBSD.

[jrtc27]

In D47407#1081787, @cperciva wrote:

In D47407#1081562, @jrtc27 wrote:

In D47407#1081558, @cperciva wrote:

@jrtc27 We should make this conditional on "is x86 or arm64" since there's no guarantee that the FreeBSD project will have package sets for any other platforms. Would that address your concern about the availability of a pkg binary?

Well, x86 and arm64 macOS/Linux will still fall under that. But yes, making sure it doesn't fail for architectures without package sets is also a good point.

Oh, people are building FreeBSD release images on non-FreeBSD? I didn't realize that. This sounds like even more reason to have pkg in the base system (and as a cross-tool).

Yeah, we do all our CheriBSD CI builds on Linux hosts and also support building release media on macOS for ease of local testing. That’s only (mini-)memstick/bootonly/disc1/ftp though, no dvd1 (due to pkg), and no VM images.

[bz]

Yeah, we do all our CheriBSD CI builds on Linux hosts and also support building release media on macOS for ease of local testing. That’s only (mini-)memstick/bootonly/disc1/ftp though, no dvd1 (due to pkg), and no VM images.

Are you happy with the added guards?

release/Makefile

77

Remove -c I have no idea where that crept in. I am doing a NO_ROOT build test currently.

[jrtc27]

In D47407#1081914, @bz wrote:

Yeah, we do all our CheriBSD CI builds on Linux hosts and also support building release media on macOS for ease of local testing. That’s only (mini-)memstick/bootonly/disc1/ftp though, no dvd1 (due to pkg), and no VM images.

Are you happy with the added guards?

It deals with the non-FreeBSD problem, but it still assumes that there's a package repo available with an installable wifi-firmware-kmod-release in it. The config file is also release-dvd.conf, which you're making no longer true.

Also, what about bootonly? Surely that's the most important case for having working networking (and dvd the least given it has more distribution sets and some packages)?

What about NOPKG? Should it not be extended to include *installing* third-party packages to the release media, not just *distributing*?

release/Makefile

75–81

Something like this to avoid duplication, and mirroring the style the top-level Makefile.inc1 often uses?

I'd also suggest using ${PKG_CMD} to find the pkg binary like Makefile.inc1 (default set by bsd.own.mk) and choosing a name for PKGCMD that doesn't sound so close to it. Maybe just PKG. Or put an R in front of all these things (for Release) like how we do [BWK]MAKE(|ENV|FLAGS) etc in Makefile.inc1

[bz]

• also guard wifi-firmware with NOPKG
• correct the commands for NO_ROOT installations
• for pkg-stage: check for NO_ROOT and email a loud warning that it is not yet supported though it seems easy to make it work for FreeBSD at this point; move the XXX into the check; this allows me to test build a dvd with NO_ROOT

I am NOT:

• going to remane the PKGCMD for another arbitrary name currently
• change pkg for a variable as it is caled "pkg" throughout the release builds of various artefacts
• do further polishing of the addiiton for NO_ROOT builds to be special if @emaste wants to make it all NO_ROOT anyway it'll be easier to just drop one of the cases like this

[bz]

I would love feedback from releng on this at this point as BETA2 is approaching soon...

[bz]

Do we still want this (and the accompanying bsdinstall change) in 14.2-R or not? As otherwise I'll have to add rtw89 firmware at least there.

[bz]

Unless releng has any objections I'll commit this around Tue 19.11. afternoon UTC.
I would still hope releng to comment on it before if not at least by accepting or requesting changes.

I understand that mixing this in while the other dust settles was not a good idea so I kept waiting a bit longer...

[cperciva]

@jrtc27 Any objections to this patch as it is now? I'd like to get it into the tree ASAP so we can have wifi on 14.2-RC1 ISOs.

[jrtc27]

I'd prefer some of the other cleanups I suggested to have been done as I see no downside to doing things more properly (beyond taking the time to test, but there's been plenty of that in the past 2 weeks), but I'm not going to block the review on those. There is one other point I do have to make though that I really do object to being here.

release/Makefile

328

This doesn't belong, and were we to do anything it should be error rather than echo, but still not in *this* commit. You only added this to the diff so you could test dvd builds, but it's not required for the rest of the diff otherwise.

[bz]

Remove the bits about the broken NO_ROOT bit.

[bz]

"cleanup" the PKGCMD bits as requested by @jrtc27 .
I have not tested NO_ROOT again.

PKG_CMD is indirectly defined thorugh bsd.obj.mk -> bsd.own.mk but I am
not going to adjust that given this is needed in less than 48 hours for
14.2-RC1.

I opted for the "ports" way to call the command PKG_INSTALL and added the
install target which makes it all more clear.

[jrtc27]

Thanks, no more objections from me