I took the liberty to give this another try, since I'm picking up on my ceph work.
Already got Ceph v21/22 running on FBSD 14 and 15.
THis tests with a small driver I made for a memory based backend.
The RBD plugin needs some more work, but is getting there.
But getting the block_if mux in place will be a first requirement.
Now trying a less invasive approach.
Let blockif_open() hand a disk over to a backend that lives outside of
bhyve. A path of the form "<name>:<spec>", that does not exist as a
file, loads BLOCKIF_PLUGIN_DIR/block_<name>.so and uses the
struct block_backend found in it. The interface is described in the new
block_backend.h and uses no bhyve symbols, so backends can be packaged
separately, e.g. one for Ceph RBD.
The backend methods are run by the existing blockif worker threads, so
the request queue, ordering, flush, delete, snapshot pause and resume
and close handling stay in block_if.c and are shared by all disks.
A request that is in flight in a backend cannot be interrupted, so
blockif_cancel() returns EBUSY for it, and disk resize notifications are
not supported for backends.
The ro, nodelete, sectorsize and bootindex options apply to a backend
as they do to a file. The nocache, sync and direct options are ignored.
Backends are loaded while the disks are opened, before bhyve enters
capability mode. A backend that needs resources which are not
available in capability mode, such as network sockets, cannot be used
with a bhyve that has Capsicum enabled.
The new path syntax is documented in bhyve.8 and bhyve_config.5.
This is a much smaller take on the earlier proposal in D23010
(https://reviews.freebsd.org/D23010), which renamed the blockif API and
moved the local disk code out of block_if.c.
As discussed there, the backend is selected by the name before the first
colon and is loaded dynamically, and the frontends are not touched.
Earlier out-of-tree work on Ceph RBD support for bhyve:
https://github.com/freebsd/freebsd-src/pull/426
https://reviews.freebsd.org/D23010