freebsd-update(8): add new command to 'check' for available updates only

[egypcio]

Summary

by applying this patch we:

• would introduce the check command;
• have an alternative to avoid having the fetch command downloading all binary updates;
  • folks can now use check to only fetch (and verify sanity) of a remote tag to report if there's an update available, or not;
• save FreeBSD's update servers from getting unnecessary HTTP requests - by not serving every single fetch request from freebsd-update(8);
• help improving users to just verify if updates are available, or not;
  • freebsd-update(8) will just report the status of an available update be ready or not, and exit.

it's also important to note that at least the tag must be fetched (and sanitized). we are doing that - still preventing replay attacks and unsupported downgrades.
we are mainly focusing on supporting RELEASE here, as other versions (ALPHA/BETA/RC) can only be upgraded instead of patched using the install command.

the motivation to produce this changes come from:

• https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=265594

Test Plan

root@freebsd:~ # uname -abiUK
FreeBSD freebsd.kvm.localdomain 13.1-RELEASE FreeBSD 13.1-RELEASE releng/13.1-n250148-fc952ac2212 GENERIC amd64 GENERIC 1301000 1301000 5b12d97f9629e35915bb91fb9145e851f0b437fe

root@freebsd:~ # freebsd-update check
src component not installed, skipped
Looking up update.FreeBSD.org mirrors... 2 mirrors found.
Fetching public key from update2.freebsd.org... done.
Fetching metadata signature for 13.1-RELEASE from update2.freebsd.org... done.
There is an update available: 13.1-RELEASE-p1

root@freebsd:~ # find /var/db/freebsd-update/
/var/db/freebsd-update/
/var/db/freebsd-update/files
/var/db/freebsd-update/tINDEX.present
/var/db/freebsd-update/serverlist_tried
/var/db/freebsd-update/serverlist_full
/var/db/freebsd-update/serverlist
/var/db/freebsd-update/pub.ssl
/var/db/freebsd-update/tag.new

root@freebsd:~ # du -sh /var/db/freebsd-update/
 28K    /var/db/freebsd-update/

root@freebsd:~ # freebsd-update fetch
src component not installed, skipped
Looking up update.FreeBSD.org mirrors... 2 mirrors found.
Fetching public key from update1.freebsd.org... done.
Fetching metadata signature for 13.1-RELEASE from update1.freebsd.org... done.
Fetching metadata index... done.
Fetching 2 metadata files... done.
Inspecting system... done.
Preparing to download files... done.
Fetching 8 patches..... done.
Applying patches... done.

root@freebsd:~ # find /var/db/freebsd-update/
/var/db/freebsd-update/
/var/db/freebsd-update/files
/var/db/freebsd-update/files/0bd6d0e9205702e46ec1ab85e03c75b82b3d349655f4a8f12a45c84f5cc57be5.gz
/var/db/freebsd-update/files/1b5d50195d5c5d6275d05df7a353da2a6f23769ed6d6c88ea6760618c8d5b766.gz
/var/db/freebsd-update/files/202884e06b5cd7a4a3cbc605d31f1b86cd5f3459a81e4cc2945cd8246eebcbc1.gz
/var/db/freebsd-update/files/f9da837ee70dc8b8f48c9b3b0599e3a667d511b2e3fe005ff4dd00a3f3db47c1.gz
/var/db/freebsd-update/files/d91fe3dc61ec4faf35fd7ff70e79da823fdbacbd0f06489a79480d523c10e813.gz
/var/db/freebsd-update/files/20c6c61e3a5a17257a3649026dcc26d873914bce75a6e9560c1e5343720c097f.gz
/var/db/freebsd-update/files/fc47fc445d95785d69a22f5b8a57c430bf44add537786219a0883e4239a149c5.gz
/var/db/freebsd-update/files/7e03e998979665edd655a6cc6289e8a23aa0f26477f562376e4e017c72bfd757.gz
/var/db/freebsd-update/files/9f7830221f50a2689e65842241550bb73589a1e40041168b9d538ad610440199.gz
/var/db/freebsd-update/files/a374f3997f7a94c8aeae10c28cca1a86d5e25361103c5b5d50b90c3330de36bb.gz
/var/db/freebsd-update/files/bd2bb9ebe7e59750f43a886eee2cb79da9d0ef8a54f57736fa1f5894c0bb6e10.gz
/var/db/freebsd-update/files/3fb7e32f9cae1cd88a530b744bda9418291bab494f42ed505dc63b2449c2d97f.gz
/var/db/freebsd-update/files/982bfe44a873f9ba2f2a2adf79db3daa1dfdcd2b452fa92a9b3b73902c6a92c8.gz
/var/db/freebsd-update/files/cfe3b430100563fbb4ea07035a97753e0fcee589624176bd699c6cd9587b0395.gz
/var/db/freebsd-update/files/02d7d8a57a692cfc1c6596fc9de5d081d3e6919565284f70c2158c8233ef16df.gz
/var/db/freebsd-update/files/a40b21d2cb1bf04d8f96c76fa47928128f6ccd5fda44d0d954918c6c3c79c83c.gz
/var/db/freebsd-update/files/be6bcc0bd7ba797dd6d89d9c225f1e6518ca40e747ec043354ce3ad743d1a4b4.gz
/var/db/freebsd-update/files/f63e85cc386e3fbc2563481bc7013ad304acafee944cce14f63f5af59c269969.gz
/var/db/freebsd-update/tINDEX.present
/var/db/freebsd-update/serverlist_tried
/var/db/freebsd-update/serverlist_full
/var/db/freebsd-update/serverlist
/var/db/freebsd-update/pub.ssl
/var/db/freebsd-update/tag
/var/db/freebsd-update/install.jVMf0I
/var/db/freebsd-update/install.jVMf0I/INDEX-OLD
/var/db/freebsd-update/install.jVMf0I/INDEX-NEW
/var/db/freebsd-update/f465c3739385890c221dff1a05e578c6cae0d0430e46996d319db7439f884336-install

root@freebsd:~ # du -sh /var/db/freebsd-update/
 26M    /var/db/freebsd-update/

taking 12.3-RELEASE as example now, and checking how fast check can be (and again how less storage it would consume):

root@freebsd:~ # env PAGER=cat time freebsd-update fetch >/dev/null
       38.63 real        20.64 user         1.81 sys

root@freebsd:~ # cat /var/db/freebsd-update/tag
freebsd-update|amd64|12.3-RELEASE|6|c63d0d5b3974423f39310a515f177873bea6b649510643bd0781788c062fa41d|1675123200

root@freebsd:~ # du -sh /var/db/freebsd-update/
150M    /var/db/freebsd-update/

running the very same command again would present us the following:

root@freebsd:~ # env PAGER=cat time freebsd-update fetch >/dev/null
       12.51 real        10.94 user         0.88 sys

if there's the check command available as an alternative option to just check for updates, well:

root@freebsd:~ # time freebsd-update check >/dev/null
        0.41 real         0.09 user         0.06 sys

[egypcio]

update the COMMANDS section on the manpage, to show commands sorted just like they are printed by freebsd-update -h

[asomers]

I guess the main difference between check and updatesready is that the latter requires you to fetch the updates first, right? Is the workflow you envision a cron or periodic job that only runs freebsd-update check and informs the user if there are any updates, but makes them actually fetch those updates manually? For a user who eventually applies every update, will this new workflow result in less total load on the update servers, or about the same?

[egypcio]

In D36241#823751, @asomers wrote:

I guess the main difference between check and updatesready is that the latter requires you to fetch the updates first, right? Is the workflow you envision a cron or periodic job that only runs freebsd-update check and informs the user if there are any updates, but makes them actually fetch those updates manually? For a user who eventually applies every update, will this new workflow result in less total load on the update servers, or about the same?

indeed the main difference is that check only fetches a bare minimum (and sanitized) tag, which tells us which version is available on FreeBSD's update servers.

check the TEST PLAN above

updatesready do not work in that way; it downloads like all binary files and at the end report back that one is able to run e.g.: freebsd-update install.

the so-called footprint and impact coming from the "client machines" against the update servers using this proposed check command is way smaller compared to running others like install - and, IMHO, that new check command can also be used in a way to verify for available upgrades as well (again, without really fetching all required files to perform such action).

how would that be done, you may ask? by setting a temporary UNAME_r env. variable and running freebsd-update check I'd say.

the further available flexibility delivered by this new check command would also allow users (sysadmins?) to:

• save their download traffic (paying less money?, in case of regular unwanted binary files download);
• avoid unnecessary disk I/O, when fetch writes small or mid size files to the cache directory;
• enhance a way to monitor for available updates (as mentioned before, by not fetching all files at once);
  • it includes monitoring available updates using either a simple shell script, a Nagios plugin, or more complexes scenarios using Grafana or whatever;
• reduce the time of automated deployments on workflows and pipelines that rely on freebsd-update to create virtual machines, physical servers or just jails;
• let users (sysadmins?) define a e.g.: "policy to update FreeBSD machines" only when -p2/-p4/-p5 versions are available -- or whatever the new flexibility provides them.

[asomers]

indeed the main difference is that check only fetches a bare minimum (and sanitized) tag, which tells us which version is available on FreeBSD's update servers.

check the TEST PLAN above

updatesready do not work in that way; it downloads like all binary files and at the end report back that one is able to run e.g.: freebsd-update install.

the so-called footprint and impact coming from the "client machines" against the update servers using this proposed check command is way smaller compared to running others like install - and, IMHO, that new check command can also be used in a way to verify for available upgrades as well (again, without really fetching all required files to perform such action).

But every user who runs "freebsd-update check" will eventually also run "freebsd-update fetch". So is the overall load any less this way? Does "freebsd-update fetch" download all files on every invocation, even if there are already downloaded files that haven't been installed?

[egypcio]

In D36241#824095, @asomers wrote:

indeed the main difference is that check only fetches a bare minimum (and sanitized) tag, which tells us which version is available on FreeBSD's update servers.

check the TEST PLAN above

updatesready do not work in that way; it downloads like all binary files and at the end report back that one is able to run e.g.: freebsd-update install.

the so-called footprint and impact coming from the "client machines" against the update servers using this proposed check command is way smaller compared to running others like install - and, IMHO, that new check command can also be used in a way to verify for available upgrades as well (again, without really fetching all required files to perform such action).

But every user who runs "freebsd-update check" will eventually also run "freebsd-update fetch". So is the overall load any less this way? Does "freebsd-update fetch" download all files on every invocation, even if there are already downloaded files that haven't been installed?

that's correct, indeed: eventually we do need to run freebsd-update fetch in order to get our updates.

should one runs that routine in a crontab(1) to check for updates every day - which I believe a few people do - then these folks are repeatedly running "fetch_inspect_system" and having subtle little spikes on their machines' load (I know: not the perfect terminology here. just wish to try illustrating a point).

the proposed check command interrupts the verification for available updates before reaching that point and still serves the purpose (the purpose of only checking for updates and existing).

by having the option to run check, we do not need to:

1. fire up unnecessary HTTP requests fetching a) metadata indexes, b) sort them, and c) download all the binaries;
2. stop by the "Inspecting system..." stage;
3. copy unmodified files and prepare a rollback (which is only a thing after we have installed something).

there's also something else to bring into consideration

maybe for some folks downloading 28MB or 200MB is not a big deal, but in some places around the globe we do have users/admins that would pretty much need an alternative option like check to avoid downloading that amount of data;
running a simple check in one place and take the opportunity to download the data in a different one can be handy, or just download in a different time of the day.

here we have maybe a different point of view that could be sweet to share:

• 12.3-RELEASE

taking the 12.3-RELEASE as example, the very first run of freebsd-update fetch running on a fresh installed system would produce the following:

root@freebsd:~ # env PAGER=cat time freebsd-update fetch >/dev/null
       38.63 real        20.64 user         1.81 sys

root@freebsd:~ # cat /var/db/freebsd-update/tag
freebsd-update|amd64|12.3-RELEASE|6|c63d0d5b3974423f39310a515f177873bea6b649510643bd0781788c062fa41d|1675123200

root@freebsd:~ # du -sh /var/db/freebsd-update/
150M    /var/db/freebsd-update/

running the very same command again would present us the following:

root@freebsd:~ # env PAGER=cat time freebsd-update fetch >/dev/null
       12.51 real        10.94 user         0.88 sys

if there's the check command available as an alternative option to just check for updates, well:

root@freebsd:~ # time freebsd-update check >/dev/null
        0.41 real         0.09 user         0.06 sys

[egypcio]

ping?

[allanjude]

usr.sbin/freebsd-update/freebsd-update.sh

3381

Should this be using freebsd-version -r instead of uname -r so that non-kernel -pX updates don't falsely trigger issues?

When there is a patch that doesn't change the kernel (like tzdata updates), the kernel version is not updated, just the userland, and we wouldn't want to keep telling people they need to update when they do not.

3433

using a case here would avoid needing to spawn new processes

case "$COMMANDS" in
  "*check*")
    ;;
  *)
    fetch_run || exit 1
    ;;
esac

or something to that effect

3523

same advice about using case here

[egypcio]

usr.sbin/freebsd-update/freebsd-update.sh

3381

I really do not think so; RELPX comes from the fetch_tagsanity() function. inside the case we do take care of the proper checks. same base from all @cperciva's code, tbh, and kept as is.

pretty much all PoC to test this diff worked as expected using this implementation (documented on this revision).

[egypcio]

usr.sbin/freebsd-update/freebsd-update.sh

3433

interesting suggestion. applying.

[egypcio]

• update diff, to sync latest changes from https://reviews.freebsd.org/D36516;
• apply suggestions from @allanjude (to avoid spawn new processes, using case).

[egypcio]

any further suggestions or inputs about this change request? anyone?

I've being running that for quite a while and the fleet is green ever since.

[gbe]

LGTM for the man page parts.

[egypcio]

thanks @gbe !
it would be awesome to get approval from src, so we can land this one -- which I am using solo for quite a time

[egypcio]

any new input from @src would be really appreciated here; this is running smooth for me since past updates (for all supported releases).

ping @asomers @cperciva, and others maybe?

[asomers]

The only problem I have with this new feature is that it is 75% redundant with updatesready. I think that having two so-similar features will confuse users. Is there any way to get the benefits of both, while not adding any new commands?

[egypcio]

In D36241#861449, @asomers wrote:

The only problem I have with this new feature is that it is 75% redundant with updatesready. I think that having two so-similar features will confuse users. Is there any way to get the benefits of both, while not adding any new commands?

they are used for different purposes, once we look and analyze the code;

• check verifies the remote update servers if there's something to be fetched, and returns a status for that;
• updatesready checks if the local machine has binaries (which were already downloaded) on a particular BASEDIR ready to be installed.

that said, would you mind to raise us attention to those "75% of code redundancy" you say check is if compared against updatesready? I am not sure if I was able to follow that, or you missed the point while analyzed them in the way described above.

on top of the past descriptions flagged on this very same comment, I shall ask for a quick look on the fact of how this new proposed command to the freebsd-update has a positive impact on performance and disk I/O -- in which way? here you must read the description, with its detailed information.

[egypcio]

ping

[egypcio]

ping @src