sysutils/firstboot-pkg-upgrade: New port

[ziaee]

Summary

Introduce an rc.d service to upgrade all packages on first boot,
ensuring cloud images are deployed with no known vulnerabilities. By
default, it will patch everything from all enabled repos, and record
this in syslog. It accepts an optional additional line that specifies
a list of space-separated specific repos to limit the upgrade to.

Note specifying bogus repos results in the upgrade aborting,
it will log This and will not try to not run again.

Sponsored by: Amazon
Sponsored by: Google Cloud
Sponsored by: OVHcloud
Discussed with: bapt, cperciva, delphij, lwhsu

[ziaee]

sysutils/firstboot-pkg-upgrade/files/firstboot_freebsd_update.in

13 ↗

(On Diff #175451)

I don't know if this is true

27 ↗

(On Diff #175451)

i don't know if it's reasonable to prefix the logs with pkg: because it seems reasonable to me but could cause confusion since this isn't actually pkg, or maybe it will print as pkg: pkg:

44 ↗

(On Diff #175451)

this HTTP_TIMEOUT is cargo culting, I don't know if pkg even reads this

[ziaee]

remove HTTP_TIMEOUT since I didn't see that in the pkg code. Add -y so
that pkg upg can run non-interactive.

[ziaee]

ok, i figured out how to test this, and fixed the errors. it's working
on my machine by touching /firstboot. this may be ready for review now.

[cperciva]

sysutils/firstboot-pkg-upgrade/files/firstboot_pkg_upgrade.in

45

Can you add functionality here to limit this to a particular repo?

In particular I'd like to be able to tell EC2 (and probably other cloud) images to update FreeBSD-base packages (since those updates will be security and errata updates) but not FreeBSD-ports packages (since those will include functional changes).

[ziaee]

Limited to FreeBSD-base repo and tested on my laptop. Thanks Colin!

[cperciva]

Sorry, I should have been clearer. Can you make limiting to a specific repository *optional*? There may be people who want to update everything, especially if they build their own images which access internal repositories.

[ziaee]

adjust logic to accept a firstboot_pkg_upgrade_repos variable that
limits the repos to specific repos if provided. i adapted the logic
from sysutils/firstboot-pkgs, but this is the most complex rc.d script
ive tried to write, so i don't know what i dont know. thanks Colin!

i also rewrapped the comment for my workflow.

[bapt]

sysutils/firstboot-pkg-upgrade/files/firstboot_pkg_upgrade.in

52

you should run pkg upgrade -r ${repo1} -r ${repo2} ... -y
it will help the solver doing a bette job.

[ziaee]

apply bapt feedback. note that a bogus repo being specified will abort all the others. im not sure if i need to initialize the empty variable first, but that's what i did. thanks @bapt!

[delphij]

sysutils/firstboot-pkg-upgrade/files/firstboot_pkg_upgrade.in

33

I think 'pkg' don't have to be quoted here because it's one single word constant.

56

OPTIONAL: Purely personal opinion: It might be reasonable to pass AUTOCLEAN=on here. If we downloaded some packages for a fresh installed system, the packages are used for installation but probably never read from again.

On sidenote, I wish we could have a way to tell if pkg upgrade really did some actions (and possibly also what changes it has made) instead of requesting reboot regardless, but for the initial implementation let's keep it simple for now.

[ziaee]

remove log prefix overquoting and add autoclean to pkg. thanks @delphij!
overquoting is ugly and confusing, and autoclean is definitely ideal.
the space savings will be a lot across the entire cloud.

[lwhsu]

Hi, I am also working on this and didn't know this is earlier than me. I think this one is more complete while mine is only focusing on FreeBSD-base repo. However I guess there are still something useful (or not, as I'm still testing):

• bootstrap/update pkg
• only reboot when there is pkg updated

It's short so I guess I can just paste the core part of mine here. Hope this helps.

firstboot_base_upgrade_run()
{

        # Calculate the checksum of the orignal state
        state_orig=`pkg info -g FreeBSD-\* | sha256`

        # Bootstrap and update pkg to ensure synchronization with the repository
        env ASSUME_ALWAYS_YES=YES pkg bootstrap -f | cat
        env ASSUME_ALWAYS_YES=YES pkg update -f | cat

        # Upgrade the FreeBSD-* packages
        env ASSUME_ALWAYS_YES=YES pkg upgrade \
                --repository FreeBSD-base -g FreeBSD-\* </dev/null | cat

        # Calculate the checksum of the state again
        state_new=`pkg info -g FreeBSD-\* | sha256`

        if [ $state_orig != $state_new ]; then
                echo "Requesting reboot after installing updates."
                touch ${firstboot_sentinel}-reboot
        fi
}

[lwhsu]

btw, you can try to run it with rclint, but I don't usually follow "Do not quote values unless necessary" error.

sysutils/firstboot-pkg-upgrade/files/firstboot_pkg_upgrade.in

63

trailing white line?

[lwhsu]

sysutils/firstboot-pkg-upgrade/files/firstboot_pkg_upgrade.in

45

In particular I'd like to be able to tell EC2 (and probably other cloud) images to update FreeBSD-base packages (since those updates will be security and errata updates) but not FreeBSD-ports packages (since those will include functional changes).

That's exactly the same as what I'm thinking and that's why my work is named sysutils/firstboot-base-upgrade. I am thinking updating the sysutils/firstboot-pkgs to let it can do both install and/or upgrade.

I am thinking in the future those two ports might be merged so there will not be duplicated pkg operations.

[ziaee]

implement bootstrap pkg if it is not done so already, based on lwhsu feedback, thanks! note, i like a blank line at the end of scripts for cat reasons. if we don't want this, i can remove it, but it doesn't hurt anything and we do it all over the place.

[ziaee]

use logic inspired from @lwhsu to prevent reboot if nothing happened. compare the file size in bytes of the pkg database instead of using pkg inf and checksum to save water/coal. tested on my laptop

[ivy]

@lwhsu out of interest, how do you end up with a system that has FreeBSD-base packages installed but doesn't have pkg bootstrapped?

[lwhsu]

In D56381#1295532, @ivy wrote:

@lwhsu out of interest, how do you end up with a system that has FreeBSD-base packages installed but doesn't have pkg bootstrapped?

hmm, in the past there is a step in the vm image build to remove the pkg repo database:

mount -t devfs devfs ${DESTDIR}/dev

# The firstboot_pkgs rc.d script will download the repository
# catalogue and install or update pkg when the instance first
# launches, so these files would just be replaced anyway; removing
# them from the image allows it to boot faster.
chroot ${DESTDIR} ${EMULATOR} env ASSUME_ALWAYS_YES=yes \
        /usr/sbin/pkg delete -f -y pkg
umount ${DESTDIR}/dev
rm -r ${DESTDIR}/var/db/pkg/repos/FreeBSD-ports
rm -r ${DESTDIR}/var/db/pkg/repos/FreeBSD-ports-kmods

I didn't know this got removed in 54e006369c9aab4f3a22f026eb6924c0f9cafda8 (and it would be good if we can find a way to do this under NO_ROOT)

But still, for the VM, it's still safer to get the pkg re-bootstrapped on the first boot, in the past there was an issue that the pkg in the VM image is too old and cannot work with the repo with newer format on pkg.freebsd.org. That caused many CI failures in other projects, and made many projects need to write those env ASSUME_ALWAYS_YES=YES pkg bootstrap -f and env ASSUME_ALWAYS_YES=YES pkg update -f two lines in the beginning of their CI script before installing the required packages.

[lwhsu]

In D56381#1295529, @ziaee wrote:

use logic inspired from @lwhsu to prevent reboot if nothing happened. compare the file size in bytes of the pkg database instead of using pkg inf and checksum to save water/coal. tested on my laptop

I know that the chance could be low, but I feel in the process, some packages increase the pkg database the size and some decrease it, there is chance to be the same size before/after the operation.

Yes, there is still the possibility of checksum collision, and that's why the distinfo has both checksum and size.

[lwhsu]

sysutils/firstboot-pkg-upgrade/files/firstboot_pkg_upgrade.in

52

I haven't tested, but will this only find /usr/sbin/pkg and think pkg is bootstrapped?

[lwhsu]

In D56381#1295504, @ziaee wrote:

note, i like a blank line at the end of scripts for cat reasons. if we don't want this, i can remove it, but it doesn't hurt anything and we do it all over the place.

Do you mean you want a newline character or a blank line? It doesn't seem to me it's our common practice to have a whole blank line in the end of the file (that's actually 2 newline characters), but yes we do need a newline character in the end of the file.

[ziaee]

incorperate suggestions by @bapt. Do not reboot unless kernel is updated. In all other situations, including situations where only the pkg database is updated, just restart all services.

Can we get this in? We really need this in 15.1.

[cperciva]

I'm inclined to skip the "figure out if we need to reboot or just restart services" complexity at this point because

1. Trying to restart services when we're in running from inside /etc/rc sounds like a recipe for getting things started in the wrong order.
2. Odds are that there's going to be a kernel update and we'll need to reboot anyway.

sysutils/firstboot-pkg-upgrade/files/firstboot_pkg_upgrade.in

24

It's true that re@ only provides *base system* updates for those, but people running -STABLE can still get updates from FreeBSD-ports just fine. Maybe worth excluding base specifically for the nonstandard case?

65

Rather than checking if /var/db/pkg/local.sqlite changed *and* whether something got logged -- which could fail if we're fast enough to check before syslogd writes to the log file -- can we check if the mtime on sysctl kern.bootfile has changed?

For base system updates that should suffice since pkgbase ships new kernels even if the only substantive changes are in kernel modules... it won't catch kernel module updates, though, but neither would the current code.

69

@bapt Is it safe to run service -R *in the middle of /etc/rc*?

[ziaee]

greatly simplify based on cperciva feedback.

basically, we're always rebooting unless there's no internet, which will
never happen for this use case (cloud), so remove the logic to compare
state. remove the logic for nonstandard, since we can always upgrade
ports, and if the operator has a custom repo they're specifying, it
won't matter. Tested on my laptop running 15.0 and GCE running 15-stable

if the script is not running on base, it will skip the reboot.

maybe in the future we can add all kinds of conditional logic but I'm
honestly not that interested in it right now. this patch is going to be
very reliable and predictable. the one we've been using does not have
any of that. reboot on cloud is very very fast, and this will only
happen once.

[ziaee]

sysutils/firstboot-pkg-upgrade/files/firstboot_pkg_upgrade.in

52

pkg -N is from /usr/sbin/pkg and exists to determine if pkg is bootstrapped. I tested this on GCE by deleting pkg and touching /firstboot, and it works as intended. On prerelease, it does a very good behavior by putting only a single line in /var/log/messages

[ziaee]

ok, what if we separate the pkg update, and take the state after updating, but before upgrading, then we will actually know if the upgrade did anything. maybe because of this we can simplify the logic at the end a bit more.

[cperciva]

In D56381#1302386, @ziaee wrote:

ok, what if we separate the pkg update, and take the state after updating, but before upgrading, then we will actually know if the upgrade did anything. maybe because of this we can simplify the logic at the end a bit more.

I think pkg info | sha256 would work? If any packages are updated (or installed or removed) then that hash will change.

[ziaee]

In D56381#1302390, @cperciva wrote:

In D56381#1302386, @ziaee wrote:

ok, what if we separate the pkg update, and take the state after updating, but before upgrading, then we will actually know if the upgrade did anything. maybe because of this we can simplify the logic at the end a bit more.

I think pkg info | sha256 would work? If any packages are updated (or installed or removed) then that hash will change.

I think it would work too, but I don't see why we wouldn't want to stat the mtime in between updating and upgrading instead. By my rough estimate, time(1) shows 0.08 real for the pkg info | sha256, but for stat -f %c /var/db/pkg/local.sqlite it shows 0.00 real, so, that's less electricity, right?

[cperciva]

In D56381#1302566, @ziaee wrote:

In D56381#1302390, @cperciva wrote:

In D56381#1302386, @ziaee wrote:

ok, what if we separate the pkg update, and take the state after updating, but before upgrading, then we will actually know if the upgrade did anything. maybe because of this we can simplify the logic at the end a bit more.

I think pkg info | sha256 would work? If any packages are updated (or installed or removed) then that hash will change.

I think it would work too, but I don't see why we wouldn't want to stat the mtime in between updating and upgrading instead. By my rough estimate, time(1) shows 0.08 real for the pkg info | sha256, but for stat -f %c /var/db/pkg/local.sqlite it shows 0.00 real, so, that's less electricity, right?

Fair point, but I'm a little bit concerned about this being brittle -- /var/db/pkg/local.sqlite is not a public interface for pkg, so there's no guarantee that a future pkg won't (a) change that path, or (b) touch that file even if nothing has changed. I don't want to suddenly discover in FreeBSD 16.3 that VMs aren't rebooting after applying security updates because something changed in pkg.

[bapt]

In D56381#1302615, @cperciva wrote:

In D56381#1302566, @ziaee wrote:

In D56381#1302390, @cperciva wrote:

In D56381#1302386, @ziaee wrote:

ok, what if we separate the pkg update, and take the state after updating, but before upgrading, then we will actually know if the upgrade did anything. maybe because of this we can simplify the logic at the end a bit more.

I think pkg info | sha256 would work? If any packages are updated (or installed or removed) then that hash will change.

I think it would work too, but I don't see why we wouldn't want to stat the mtime in between updating and upgrading instead. By my rough estimate, time(1) shows 0.08 real for the pkg info | sha256, but for stat -f %c /var/db/pkg/local.sqlite it shows 0.00 real, so, that's less electricity, right?

Fair point, but I'm a little bit concerned about this being brittle -- /var/db/pkg/local.sqlite is not a public interface for pkg, so there's no guarantee that a future pkg won't (a) change that path, or (b) touch that file even if nothing has changed. I don't want to suddenly discover in FreeBSD 16.3 that VMs aren't rebooting after applying security updates because something changed in pkg.

that is a valid concern from @cperciva I don't think I will change anything in that area, but yes the content of /var/db/pkg is supposed to be opaque. (I know I am the one who recommended the mtime check in the first place).

[cperciva]

In D56381#1302616, @bapt wrote:

In D56381#1302615, @cperciva wrote:

Fair point, but I'm a little bit concerned about this being brittle -- /var/db/pkg/local.sqlite is not a public interface for pkg, so there's no guarantee that a future pkg won't (a) change that path, or (b) touch that file even if nothing has changed. I don't want to suddenly discover in FreeBSD 16.3 that VMs aren't rebooting after applying security updates because something changed in pkg.

that is a valid concern from @cperciva I don't think I will change anything in that area, but yes the content of /var/db/pkg is supposed to be opaque. (I know I am the one who recommended the mtime check in the first place).

Yeah I'm not just concerned about deliberate changes in pkg itself either -- it's possible that sqlite's behaviour might change.

[ziaee]

implement cperciva/lwhsu sha256 suggestion, thanks guys!

[ziaee]

fix comment typos, thanks @cperciva!

[cperciva]

LGTM!

[bapt]

sysutils/firstboot-pkg-upgrade/files/firstboot_pkg_upgrade.in

69

I think it is, but I am not sure.