Set hw.usb.no_boot_wait=1 by default

[cperciva]

Summary

A large number of people run ZFS on laptops and suffer a 9 second delay
in the boot process for USB probing, even though they are not in fact
booting from a USB device. (This is not an issue with UFS, since USB
probing is skipped if the root filesystem is UFS and on a device which
is present.)

Set the default behaviour to skip this wait; but set the loader tunable
to retain the wait on install media, since that's the one circumstance
where many users will boot from USB.

MFC after: never
Relnotes: By default, FreeBSD will not wait for USB device probing

		to complete before booting; if you have filesystems on
		USB devices, set hw.usb.no_boot_wait=0 to wait for them
		to be available before the system boots.

[imp]

In the past... we'd talked about having a just in time waiting for all the pools.. i hate the negative option... but we're stuck with ot...

For laptops this is good.. but many people run servers that have pools on usb for various reasons. More than I'd expect

[cperciva]

In D45954#1047551, @imp wrote:

In the past... we'd talked about having a just in time waiting for all the pools.. i hate the negative option... but we're stuck with ot...

Yeah, no progress was being made on that front so I was looking for other options.

For laptops this is good.. but many people run servers that have pools on usb for various reasons. More than I'd expect

Huh, that really surprises me. From my cloudy perspective I didn't think servers would be using USB at all.

[imp]

In D45954#1047558, @cperciva wrote:

In D45954#1047551, @imp wrote:

In the past... we'd talked about having a just in time waiting for all the pools.. i hate the negative option... but we're stuck with ot...

Yeah, no progress was being made on that front so I was looking for other options.

For laptops this is good.. but many people run servers that have pools on usb for various reasons. More than I'd expect

Huh, that really surprises me. From my cloudy perspective I didn't think servers would be using USB at all.

I see threads from time to time. Home deployment and the pools are for local, fast bachups.

What i don't know is relative number. More threads than I would have expected...

[0mp]

In D45954#1047551, @imp wrote:

In the past... we'd talked about having a just in time waiting for all the pools.. i hate the negative option... but we're stuck with ot...

Do we have a way of deprecating a sysctl, maybe even hiding it, and introducing a new, properly named replacement?

[mjg]

can this only wait for usb if no root device is found? then whack the tunable

[cperciva]

In D45954#1047826, @mjg wrote:

can this only wait for usb if no root device is found? then whack the tunable

We do that on UFS. The problem is if root is on a zpool which is a mirror of two USB devices (or a USB and a non-USB device). We might mount root when we only have half of the mirror, and then have to resync the entire USB disk when it shows up a few seconds later.

If we had a way of saying "hey ZFS, do you have any pools which are missing devices", we could skip the wait. I don't think there's any way to do that right now, but maybe it could be added?

[mjg]

one could add a flag to mountroot indicating the system knows some stuff may be missing and to fail the mount if that's the case, should be very easy to handle. then a wait round and another mount call but without the flag

[imp]

edited, stupid phone

In D45954#1047836, @mjg wrote:

one could add a flag to mountroot indicating the system knows some stuff may be missing and to fail the mount if that's the case, should be very easy to handle. then a wait round and another mount call but without the flag

We do special things for zfs that we shouldn't. We always wait for all the root holds. That's why we wait so long...

We do this because zfs is cute and doesn't mount a device, but instead a named thing that doesn't map to a dev we can look up.

The real answer isn't to hack usb here. The real answer is to not be so stupid in vfs_mountroot_wait_if_necessary. That's the root cause of all this nonsense. The comments say we don't have to wait, but yet we wait the very next line.

Something like bumping the default timeout for root to 30 seconds and just not stupidly waiting should do the trick:

diff --git a/sys/kern/vfs_mountroot.c b/sys/kern/vfs_mountroot.c
index e0d1cec5bd71..88f4c3194698 100644
--- a/sys/kern/vfs_mountroot.c
+++ b/sys/kern/vfs_mountroot.c
@@ -128,8 +128,8 @@ static enum action root_mount_onfail = A_CONTINUE;
 static int root_mount_mddev;
 static int root_mount_complete;
 
-/* By default wait up to 3 seconds for devices to appear. */
-static int root_mount_timeout = 3;
+/* By default wait up to 30 seconds for devices to appear. */
+static int root_mount_timeout = 30;
 TUNABLE_INT("vfs.mountroot.timeout", &root_mount_timeout);
 
 static int root_mount_always_wait = 0;
@@ -1013,15 +1013,21 @@ vfs_mountroot_wait_if_neccessary(const char *fs, const char *dev)
 {
        int delay, timeout;
 
+ /*
+  * Wait if the user forced that behaviour by setting
+  * vfs.root_mount_always_wait=1.
+  */
+ if (root_mount_always_wait) {
+         vfs_mountroot_wait();
+         return (0);
+ }
+
        /*
         * In case of ZFS and NFS we don't have a way to wait for
-    * specific device.  Also do the wait if the user forced that
-    * behaviour by setting vfs.root_mount_always_wait=1.
+  * specific device.
         */
        if (strcmp(fs, "zfs") == 0 || strstr(fs, "nfs") != NULL ||
-       strcmp(fs, "p9fs") == 0 ||
-       dev[0] == '\0' || root_mount_always_wait != 0) {
-           vfs_mountroot_wait();
+     strcmp(fs, "p9fs") == 0) {
                return (0);
        }
 

ZFS may need to still wait for weird cases. Mostly it won't. If we want to bogusly hyperoptimize booting a mirror with an nvme drive (ready instantly) and a USB mirror to not wait for that, ZFS will have to not wait. But this way, we try to mount, fail, and then try again. the ui might be a bit better (!bootverbose we print a dot every second, say)...

Not sure about the dev[0] = "" case. It came in with marcel's refactor, but there's no comments on it in the commit message, nor comments in the code. I junked it since I don't know when we'd try to mountroot with dev = "", but it looks like the behavior in that case, likely related to his scripting, is to drop to a prompt (default) or "do something" after all the devices arrived. I think we can just drop it, but maybe asking him first wouldn't be amiss...

Note: we still must wait for USB to be done probing later in the boot (when we import all the pools). That will also need to be fixed to make sure we can import all the pools we know about (form the zfs mapping cache), otherwise wait there...

[imp]

btw, my suggestion already works (should work?) because we have error retry in the main loop.
If it helps people (I've only lightly tested it since my laptop still has UFS root), we should look at refining it by having the timeout be part of what's configurable rather than being a weird wart on things as a tuneable holdover from pre-scripting days if I'm reading the commit history correctly.

[zlei]

I have not read though sys/kern/vfs_mountroot.c. Some thoughts,

1. If we know the rootfs is on / requiring USB disks, then certainly we have to wait for USB.
2. If we know the rootfs is not on / requiring USB disks, then certainly we should not wait for USB.

then, can the kernel have an explicit signal, for example echo "hw.usb.rootfs=1" >> /boot/loader.conf, so that the logic, if wait for USB, will be much clear ?

[imp]

In D45954#1047984, @zlei wrote:

I have not read though sys/kern/vfs_mountroot.c. Some thoughts,

1. If we know the rootfs is on / requiring USB disks, then certainly we have to wait for USB.
2. If we know the rootfs is not on / requiring USB disks, then certainly we should not wait for USB.

then, can the kernel have an explicit signal, for example echo "hw.usb.rootfs=1" >> /boot/loader.conf, so that the logic, if wait for USB, will be much clear ?

That solves the wrong problem. We should only ever wait long enough to mount root read only. USB is a red herring and has no business even having this option. Optimized this f'd up option is not the path to happiness. At best it is a short term bandaid that may help a few people (so the installer should set it imho: it knows or can know) and at worst make things less crappy enough that nobody tries to fix the actual problem like I proposed.