nvmf: Auto-reconnect periodically after a disconnect

[jhb]

Summary

Use a timer in the nvmf(4) driver to periodically trigger a
devctl "reconnect" notification. A trigger in the /etc/devd/nvmf.conf
file invokes "nvmecontrol reconnect nvmeX" upon each notification.

Similar to nvme-cli, the interval between reconnection attempts is
specified in seconds by the --reconnect-delay argument to the connect
and reconnect commands. Note that nvme-cli uses -c for short letter
of this command, but that was already taken so nvmecontrol uses -r.
The default is 10 seconds to match Linux.

In addition, a second timeout can be used to force a full detach of a
disconnected the nvmeX device after the controller loss timeout
expires. The timeout for this is specified in seconds by the
--ctrl-loss-tmo/-l options (identical to nvme-cli). The default is
600 seconds.

Either of these timers can be disabled by setting the timer to 0. In
that case, the associated action (devctl notifications or full detach)
will not occur after a disconnect.

Note that this adds a dedicated taskqueue for nvmf tasks as the
controller loss task could deadlock waiting for tasks it's task
handler enqueued on taskqueue_thread.

Sponsored by: Chelsio Communications

[jhb]

The events from devd look like:

# cat /var/run/devd.pipe
!system=nvmf subsystem=nvme0 type=reconnect
!system=nvmf subsystem=nvme0 type=reconnect
!system=nvmf subsystem=nvme0 type=reconnect

Perhaps though the subsystem should be CONTROLLER and the device name should be in a separate name property to match the nvme events?

[jhb]

Also, devctl events seem to be rather inconsistent in naming (upper vs lower case for system, etc.)

[jhb]

Also, not sure if it's worth saying, but I chose to use a devctl event that triggers nvmecontrol reconnect rather than adding a new nvmfd daemon to the system similar to iscsid. One reason is that nvmfd is already taken (though I plan to add nvmf support to ctld and retire nvmfd eventually), but adding an nvmfd would also mean either duplicating a bunch of code from nvmecontrol or having to factor it out into a library, etc. I also find the design of iscsid a bit odd as iscsictl basically sends message to iscsid via ioctls using the kernel as a kind of proxy. With this approach, the additional state of a timer for reconnects is kept in the kernel instead, but I think overall that this is simpler.