Page MenuHomeFreeBSD

sysutils/firstboot-pkg-upgrade: New port
Needs ReviewPublic

Authored by ziaee on Mon, Apr 13, 7:32 PM.
Referenced Files
Unknown Object (File)
Sun, Apr 19, 12:55 AM
Unknown Object (File)
Sat, Apr 18, 6:32 PM
Unknown Object (File)
Sat, Apr 18, 9:41 AM
Unknown Object (File)
Sat, Apr 18, 9:41 AM
Unknown Object (File)
Sat, Apr 18, 9:41 AM
Unknown Object (File)
Sat, Apr 18, 9:41 AM
Unknown Object (File)
Sat, Apr 18, 9:41 AM
Unknown Object (File)
Sat, Apr 18, 9:41 AM
Subscribers

Details

Summary

Introduce an rc.d service to upgrade packages on first boot, ensuring
cloud images are deployed with no known vulnerabilities. By default,
it will patch everything from all enabled repos, and record this in
syslog. It accepts an optional additional line that specifies a list
of space-separated specific repos to upgrade.

Note specifying bogus repos result in the upgrade aborting, and it will
not run again.

Sponsored by: Google
Discussed with: bapt, cperciva, delphij

Diff Detail

Repository
R11 FreeBSD ports repository
Lint
Lint Skipped
Unit
Tests Skipped
Build Status
Buildable 72242
Build 69125: arc lint + arc unit

Event Timeline

ziaee requested review of this revision.Mon, Apr 13, 7:32 PM
ziaee created this revision.
sysutils/firstboot-pkg-upgrade/files/firstboot_freebsd_update.in
13 ↗(On Diff #175451)

I don't know if this is true

27 ↗(On Diff #175451)

i don't know if it's reasonable to prefix the logs with pkg: because it seems reasonable to me but could cause confusion since this isn't actually pkg, or maybe it will print as pkg: pkg:

44 ↗(On Diff #175451)

this HTTP_TIMEOUT is cargo culting, I don't know if pkg even reads this

remove HTTP_TIMEOUT since I didn't see that in the pkg code. Add -y so
that pkg upg can run non-interactive.

ok, i figured out how to test this, and fixed the errors. it's working
on my machine by touching /firstboot. this may be ready for review now.

sysutils/firstboot-pkg-upgrade/files/firstboot_pkg_upgrade.in
45

Can you add functionality here to limit this to a particular repo?

In particular I'd like to be able to tell EC2 (and probably other cloud) images to update FreeBSD-base packages (since those updates will be security and errata updates) but not FreeBSD-ports packages (since those will include functional changes).

Limited to FreeBSD-base repo and tested on my laptop. Thanks Colin!

Sorry, I should have been clearer. Can you make limiting to a specific repository *optional*? There may be people who want to update everything, especially if they build their own images which access internal repositories.

adjust logic to accept a firstboot_pkg_upgrade_repos variable that
limits the repos to specific repos if provided. i adapted the logic
from sysutils/firstboot-pkgs, but this is the most complex rc.d script
ive tried to write, so i don't know what i dont know. thanks Colin!

i also rewrapped the comment for my workflow.

sysutils/firstboot-pkg-upgrade/files/firstboot_pkg_upgrade.in
51

you should run pkg upgrade -r ${repo1} -r ${repo2} ... -y
it will help the solver doing a bette job.

apply bapt feedback. note that a bogus repo being specified will abort all the others. im not sure if i need to initialize the empty variable first, but that's what i did. thanks @bapt!

delphij added inline comments.
sysutils/firstboot-pkg-upgrade/files/firstboot_pkg_upgrade.in
33

I think 'pkg' don't have to be quoted here because it's one single word constant.

56

OPTIONAL: Purely personal opinion: It might be reasonable to pass AUTOCLEAN=on here. If we downloaded some packages for a fresh installed system, the packages are used for installation but probably never read from again.

On sidenote, I wish we could have a way to tell if pkg upgrade really did some actions (and possibly also what changes it has made) instead of requesting reboot regardless, but for the initial implementation let's keep it simple for now.

This revision is now accepted and ready to land.Thu, Apr 16, 5:55 PM

remove log prefix overquoting and add autoclean to pkg. thanks @delphij!
overquoting is ugly and confusing, and autoclean is definitely ideal.
the space savings will be a lot across the entire cloud.

This revision now requires review to proceed.Thu, Apr 16, 6:28 PM
ziaee added projects: releng, pkgbase.

Hi, I am also working on this and didn't know this is earlier than me. I think this one is more complete while mine is only focusing on FreeBSD-base repo. However I guess there are still something useful (or not, as I'm still testing):

  • bootstrap/update pkg
  • only reboot when there is pkg updated

It's short so I guess I can just paste the core part of mine here. Hope this helps.

firstboot_base_upgrade_run()
{

        # Calculate the checksum of the orignal state
        state_orig=`pkg info -g FreeBSD-\* | sha256`

        # Bootstrap and update pkg to ensure synchronization with the repository
        env ASSUME_ALWAYS_YES=YES pkg bootstrap -f | cat
        env ASSUME_ALWAYS_YES=YES pkg update -f | cat

        # Upgrade the FreeBSD-* packages
        env ASSUME_ALWAYS_YES=YES pkg upgrade \
                --repository FreeBSD-base -g FreeBSD-\* </dev/null | cat

        # Calculate the checksum of the state again
        state_new=`pkg info -g FreeBSD-\* | sha256`

        if [ $state_orig != $state_new ]; then
                echo "Requesting reboot after installing updates."
                touch ${firstboot_sentinel}-reboot
        fi
}

btw, you can try to run it with rclint, but I don't usually follow "Do not quote values unless necessary" error.

sysutils/firstboot-pkg-upgrade/files/firstboot_pkg_upgrade.in
63

trailing white line?

sysutils/firstboot-pkg-upgrade/files/firstboot_pkg_upgrade.in
45

In particular I'd like to be able to tell EC2 (and probably other cloud) images to update FreeBSD-base packages (since those updates will be security and errata updates) but not FreeBSD-ports packages (since those will include functional changes).

That's exactly the same as what I'm thinking and that's why my work is named sysutils/firstboot-base-upgrade. I am thinking updating the sysutils/firstboot-pkgs to let it can do both install and/or upgrade.

I am thinking in the future those two ports might be merged so there will not be duplicated pkg operations.

implement bootstrap pkg if it is not done so already, based on lwhsu feedback, thanks! note, i like a blank line at the end of scripts for cat reasons. if we don't want this, i can remove it, but it doesn't hurt anything and we do it all over the place.

use logic inspired from @lwhsu to prevent reboot if nothing happened. compare the file size in bytes of the pkg database instead of using pkg inf and checksum to save water/coal. tested on my laptop

@lwhsu out of interest, how do you end up with a system that has FreeBSD-base packages installed but doesn't have pkg bootstrapped?

In D56381#1295532, @ivy wrote:

@lwhsu out of interest, how do you end up with a system that has FreeBSD-base packages installed but doesn't have pkg bootstrapped?

hmm, in the past there is a step in the vm image build to remove the pkg repo database:

mount -t devfs devfs ${DESTDIR}/dev

# The firstboot_pkgs rc.d script will download the repository
# catalogue and install or update pkg when the instance first
# launches, so these files would just be replaced anyway; removing
# them from the image allows it to boot faster.
chroot ${DESTDIR} ${EMULATOR} env ASSUME_ALWAYS_YES=yes \
        /usr/sbin/pkg delete -f -y pkg
umount ${DESTDIR}/dev
rm -r ${DESTDIR}/var/db/pkg/repos/FreeBSD-ports
rm -r ${DESTDIR}/var/db/pkg/repos/FreeBSD-ports-kmods

I didn't know this got removed in 54e006369c9aab4f3a22f026eb6924c0f9cafda8 (and it would be good if we can find a way to do this under NO_ROOT)

But still, for the VM, it's still safer to get the pkg re-bootstrapped on the first boot, in the past there was an issue that the pkg in the VM image is too old and cannot work with the repo with newer format on pkg.freebsd.org. That caused many CI failures in other projects, and made many projects need to write those env ASSUME_ALWAYS_YES=YES pkg bootstrap -f and env ASSUME_ALWAYS_YES=YES pkg update -f two lines in the beginning of their CI script before installing the required packages.

use logic inspired from @lwhsu to prevent reboot if nothing happened. compare the file size in bytes of the pkg database instead of using pkg inf and checksum to save water/coal. tested on my laptop

I know that the chance could be low, but I feel in the process, some packages increase the pkg database the size and some decrease it, there is chance to be the same size before/after the operation.

Yes, there is still the possibility of checksum collision, and that's why the distinfo has both checksum and size.

sysutils/firstboot-pkg-upgrade/files/firstboot_pkg_upgrade.in
52

I haven't tested, but will this only find /usr/sbin/pkg and think pkg is bootstrapped?

note, i like a blank line at the end of scripts for cat reasons. if we don't want this, i can remove it, but it doesn't hurt anything and we do it all over the place.

Do you mean you want a newline character or a blank line? It doesn't seem to me it's our common practice to have a whole blank line in the end of the file (that's actually 2 newline characters), but yes we do need a newline character in the end of the file.