release: add -DPKGBASE option

[emaste]

Summary

If this option is set, an offline repo of pkgbase packages corresponding
to base.txz and kernel.txz will be included in the disc1 release media
rather than the base.txz and kernel.txz tarballs.

Sponsored by: The FreeBSD Foundation

[ifreund_freebsdfoundation.org]

Git branch: https://github.com/ifreund/freebsd-src/tree/bsdinstall-offline-pkgbase

[ifreund_freebsdfoundation.org]

release/scripts/pkgbase-stage.lua

29

I don't like the duplicated logic from usr.sbin/bsdinstall/scripts/pkgbase.in. I don't see a better solution for this though aside from implementing pkg groups in pkg.

[emaste]

release/Makefile

25

also memstick (which is equivalent to disc1)

184

We should later add a knob to skip the packagesystem dependency and inclusion of MANIFEST (and have bsdinstall not ask the dist set/pkgbase question if MANIFEST does not exist)

208–211

I'd like to dedup the MANIFEST addition

[emaste]

Trying this out I get:

mkdir -p disc1/usr/freebsd-packages/repos/
/home/emaste/src/freebsd-git/main/release/scripts/pkgbase-stage.lua disc  /usr/obj/home/emaste/src/freebsd-git/main/amd64.amd64/release/pkgbase-repo/$(pkg -o ABI_FILE=disc1/usr/bin/uname config ABI)/latest  disc1/usr/freebsd-packages/offline  "32"
Updating FreeBSD-base repository catalogue...
pkg: file:///usr/obj/home/emaste/src/freebsd-git/main/amd64.amd64/release/pkgbase-repo/FreeBSD:15:amd64/latest/meta.txz: No such file or directory
repository FreeBSD-base has no meta file, using default settings
pkg: file:///usr/obj/home/emaste/src/freebsd-git/main/amd64.amd64/release/pkgbase-repo/FreeBSD:15:amd64/latest/data.pkg: No such file or directory
pkg: file:///usr/obj/home/emaste/src/freebsd-git/main/amd64.amd64/release/pkgbase-repo/FreeBSD:15:amd64/latest/data.tzst: No such file or directory 
pkg: file:///usr/obj/home/emaste/src/freebsd-git/main/amd64.amd64/release/pkgbase-repo/FreeBSD:15:amd64/latest/packagesite.pkg: No such file or directory  
pkg: file:///usr/obj/home/emaste/src/freebsd-git/main/amd64.amd64/release/pkgbase-repo/FreeBSD:15:amd64/latest/packagesite.tzst: No such file or directory
Unable to update repository FreeBSD-base
Error updating repositories!
/usr/libexec/flua: ...e/src/freebsd-git/main/release/scripts/pkgbase-stage.lua:130: exit
stack traceback:
        [C]: in function 'assert'
        ...e/src/freebsd-git/main/release/scripts/pkgbase-stage.lua:130: in local 'main'
        ...e/src/freebsd-git/main/release/scripts/pkgbase-stage.lua:138: in main chunk
        [C]: in ?
*** Error code 1

Stop.

[emaste]

Trying this out I get:

I tried in isolation (cd release && make -DPKGBASE pkgbase-repo) which succeeded, and then a subsequent make -DPKGABSE disc1.iso was successful. Could be yet more fallout from release/Makefile generally not handling re-runs particularly well, especially after a failure. I am able to reproduce the problem I encountered above if the pkgbase-rep directory exists, but is empty.

[ziaee]

Can we please add this to release(7) in this commit?

edit: unless it goes somewhere else? It seems strange to me the all caps like a variable and the flag like a flag.

edit2: so make has a pkgbase-repo target? What is the difference between make packages make update-packages and make pkgbase-repo?

[emaste]

edit2: so make has a pkgbase-repo

That is not a user-facing target and won't be documented. I would use this via e.g. (cd release && make -DPKGBASE disc1.iso). Presumably this could be set in release.conf but I haven't checked that. Perhaps it should be WITH_PKGBASE for consistency with other settings like WITH_DVD, although that might not be quite right because the change in this patch excludes the legacy dist sets when pkgbase is enabled.

[ivy]

i'm not familiar enough with the release scripts to feel comfortable reviewing this, but the basic idea seems fine.

release/scripts/FreeBSD-base-offline.conf

6

is this right? it seems odd to configure FreeBSD (which is the ports repository) in a file called FreeBSd-base-offline.conf.

[emaste]

release/Makefile

282

This one will also need to depend on ${PKGBASE_REPO}

release/scripts/FreeBSD-base-offline.conf

6

@ifreund_freebsdfoundation.org probably won't be able to comment in the near term so perhaps @bapt can confirm my understanding. This is only for use by pkg invoked from the installer, and the ports repo would be enabled by default otherwise.

release/scripts/pkgbase-stage.lua

29

I don't think this is too much of a concern; it's a little awkward but will be reworked significantly in the future (pkg groups, etc.) and shouldn't be a long-term maintenance burden.

81

I think this is a fine way to start, mirroring the dist sets currently on the installer, but I think we will want to rework this soon. In particular, with pkgbase we can do a minimal set of packages to get a booting system, and I think we could replace bootonly (with no packages) with that set. In the Cirrus CI test we do a boot smoke test via:

ASSUME_ALWAYS_YES=true INSTALL_AS_USER=true pkg \
    -o ABI_FILE=$OBJTOP/bin/sh/sh \
    -C ${ROOTDIR}/pkg.conf -r ${ROOTDIR} install \
    FreeBSD-kernel-generic FreeBSD-bootloader \
    FreeBSD-clibs FreeBSD-runtime

(It is unfortunate that knowledge about which sets of packages are needed for various purposes is proliferating in various spots.)

The dvd set should probably be "everything."

Anyhow, all of that can be iterated on in the tree.

[bapt]

release/scripts/FreeBSD-base-offline.conf

6

I would not disable it here, I would prefer to call it directly via pkg install -r FreeBSD-base which will only activate the FreeBSD-base repo during that execution of pkg.

release/scripts/pkgbase-stage.lua

135

pkg fetch -r therepositorytoenable -o

[emaste]

Eliminate disabling of FreeBSD repo - installer change will need an update as @bapt showed

[emaste]

@bapt the installation support came in via D49822 and -r is already passed:

while not os.execute(pkg .. "update") do
        if not prompt_yn("Updating repositories failed, try again?") then                               
                os.exit(1)
        end
end             

local packages = table.concat(select_packages(pkg, options), " ")
                
while not os.execute(pkg .. "install -U -F -y -r FreeBSD-base " .. packages) do
        if not prompt_yn("Fetching packages failed, try again?") then
                os.exit(1)
        end                     
end                    

if not os.execute(pkg .. "install -U -y -r FreeBSD-base " .. packages) then

maybe we just need to add -r to more pkg invocations?

[emaste]

release/Makefile

208–211

In D50383