libfetch: Send If-Range with ranged requests

[rcm]

Summary

A resumed transfer asks for the tail of whatever the server has now.
If the file changed in between, the caller cannot tell and ends up
with a corrupt copy.

Record the strong entity tag of each successful HTTP response in a new
validator field appended to struct url, and send it as If-Range
whenever the same struct url is reused with a non-zero offset. A
server that no longer has that version then answers 200 with the full
document, which is reported through url->offset as today. A 206 with
a different entity tag, or a 416 to a conditional request, fails with
the new FETCH_RESTART error code; the former catches servers that
honor Range but ignore If-Range.

Only a caller that reuses the struct url benefits: fetch -a retries
do, a later fetch -r run starts without a validator and does not.

struct url is part of the ABI, so bump SHLIB_MAJOR to 7 and take the
occasion to enable symbol versioning: only the interfaces declared in
fetch.h are exported now, tagged FBSD_1.8.

fetch(1): resume each -a retry from the bytes written so far, into the
file being written. A retry used to take its offset from stat(2) of
the destination, which misses buffered output and names the wrong file
when a temporary file is in use, so resumed data could land at the
wrong offset. Keep that file open across retries, discard whatever
the server does not resume, and start over on FETCH_RESTART. A -r
run whose local mtime did not match the server's went on reading the
stream it had just closed when the server ignored the range or the
local file was empty; refetch in that case too.

Relnotes: yes
Sponsored by: Rubicon Communications, LLC ("Netgate")

Test Plan

Built lib/libfetch and usr.bin/fetch on 16.0-CURRENT (aarch64) with the
default WARNS (-Werror); no diagnostics. mandoc -Tlint on fetch.3 and
fetch.1 reports nothing new. nm -D on libfetch.so.7 shows only the
versioned fetch* symbols; fetch(1), pkg(7) and bsdinstall's distfetch
pick up libfetch.so.7 after installworld and nothing in the installed
tree references libfetch.so.6.

Scripted HTTP server driving fetch(1), checking the resulting file byte
for byte and the headers fetch(1) sent, 18 scenarios, all pass: 206 with
matching ETag resumes; 200 ignoring Range is written from offset 0;
changed document -> 200 replaces the file; 416 to a conditional range ->
FETCH_RESTART -> unranged restart; 206 carrying a different ETag ->
FETCH_RESTART; weak ETag, a Last-Modified within a minute of Date and an
old Last-Modified are not sent as If-Range; trailing whitespace in the
ETag is dropped; a redirect target receives Range and If-Range; complete
local file -> single 416, nothing to do, file untouched; complete file
against a range-ignoring server; mtime mismatch with the range ignored
refetches cleanly; -r on an empty local file with a different mtime;
chunked 200 without a size to a resumed request; -a without -r resumes
into the file; -r -R rewrites in place (same inode); -a -r -R through
the FETCH_RESTART path.

A second server cuts every response after 600000 bytes of a 2 MiB
document with a strong ETag; fetch(1) from main and from this change:

case	main	patched
-a, new file	exit 0, corrupt (30 KB too long)	correct
-a, replacing an existing file	loops until killed	correct, no temp file left
-r -a, partial local file	correct	correct
-r, complete file, server ignores Range	no-op	no-op
-a, document changes after the first response	exit 0, corrupt	new version, correct
-a, changes, server ignores If-Range	exit 0, corrupt	FETCH_RESTART, new version, correct
-r -a, local mtime mismatch	exit 0 with a 600 KB file	correct

The server log shows If-Range on every retry, a 200 of the new version
when the document changed, and a 206 with the new ETag rejected when
If-Range is ignored.

Real servers, nginx 1.30 from packages, 8 cases, all pass: HTTP and
HTTPS cut-and-resume through a TCP relay that closes the first connection
1.2 MB into a 3 MB file (If-Range carries nginx's ETag, 206, file
identical); file replaced between attempts over HTTP and HTTPS (nginx
ignores the range, 200, file rewritten and identical to the new
content); fetch -r with a complete local file (one 416, size taken from
Content-Range, file and mtime untouched); nginx as a forward proxy in
front of the scripted server (origin sees If-Range; 206, 200 and 416
paths correct); a small C program calling fetchStat() then fetchXGet()
with an offset on the same struct url (HEAD records the validator, GET
sends If-Range and resumes at the requested offset; after replacing the
file it gets a 200 from offset 0 with the new size and validator).

The chunked-coding cases from D61267 give the same results with this
change applied, and all of the above passes with the minimum-rate,
connection-cache and keep-alive changes stacked on top.

[des]

This changes the public API and ABI, so we have to bump SHLIB_MAJOR. This might be a good occasion to enable symbol versioning.

[rcm]

Enable symbol versioning

[rcm]

Reworked to keep the change minimal, and to answer the ABI point properly:

• SHLIB_MAJOR 6 -> 7, libfetch.so.6 in ObsoleteFiles.inc. Symbol versioning is enabled: lib/libfetch/Symbol.map exports the 35 interfaces declared in fetch.h under FBSD_1.8, following libusb (1c1bce5bbd4d), which tagged its initial set with the latest stable version. The fetch_*, http_request* and ftp_request* helpers from common.h are no longer exported; nothing in the tree used them.
• The sys/modules/zfs/zfs_config.h hunk is gone, and so is the summary's claim that libzfs dlopen()s libfetch: the LIBFETCH_* block there sits inside #if 0 /* TBD: fetch(3) support */, so the base libzfs does not use libfetch and the soname is dead text. The OpenZFS configure check (config/user-libfetch.m4) still names libfetch.so.6 for builds outside the base system and can follow at the next vendor import.
• ETag only: the Last-Modified/Date fallback is gone (about 40 lines). Every server fetch(1) and pkg(8) meet in practice sends a strong ETag for static files; a response without one simply resumes unconditionally, as today. http_parse_mtime() is unchanged, one small ETag parser replaces three helpers, FETCH_RESTART is set inline, a repeated ETag no longer clears the validator (the last one wins, as for other headers), and a 416 to an unconditional range is handled exactly as in main again.

• fetch -a resumed from stat(2) of the destination, which misses buffered output and names the wrong file when a temporary file is in use: against a server that drops the connection every 600 KB, main produces a file 30 KB too long, and loops forever when the destination already existed. Retries now resume from the bytes actually written, into the file being written, which stays open across retries; whatever the server does not resume is truncated away.
• fetch -r with a local mtime that does not match the server's closed the stream and then, when the server ignored the range or the local file was empty, went on reading it ("Bad file descriptor", pre-existing in main). It refetches now.
• The previous diff's fetch -r of a complete file from a server that ignores Range truncated it and downloaded it again; it is a no-op again, as in main, and the -r path is otherwise as in main (no truncation on the first attempt, the "local file is longer" check is back). tries is gone, since retries no longer go through the stat at the top.

[des]

lib/libfetch/Symbol.map

1

lib/libfetch/fetch.h

53

I think the validator should be a pointer to a separately allocated string.

lib/libfetch/http.c

1619

ifrange should be bool

1623

cf. style(9), local variables should be sorted by descending order of size first, and name second (yes, I'm aware that the existing order is incorrect)