Page MenuHomeFreeBSD

nvmf: Add infrastructure kernel module for NVMe over Fabrics

Authored by jhb on Apr 9 2024, 11:03 PM.
Referenced Files
Unknown Object (File)
Sun, Jun 16, 7:29 PM
Unknown Object (File)
Mon, May 27, 12:14 PM
Unknown Object (File)
Mon, May 27, 2:11 AM
Unknown Object (File)
May 19 2024, 5:59 AM
Unknown Object (File)
May 3 2024, 7:10 AM
Unknown Object (File)
Apr 26 2024, 4:58 AM
Unknown Object (File)
Apr 19 2024, 3:11 AM
Unknown Object (File)
Apr 13 2024, 3:55 AM



nvmf_transport.ko provides routines for managing NVMeoF queue pairs
and capsules. It provides a glue layer between transports (such as
TCP or RDMA) and an NVMeoF host (initiator) and controller (target).

Unlike the synchronous API exposed to the host and controller by
libnvmf, the kernel's transport layer uses an asynchronous API built
on callbacks. Upper layers provide callbacks on queue pairs that are
invoked for transport errors (error_cb) or anytime a capsule is
received (receive_cb).

Data transfers for a command are usually associated with a callback
that is invoked once a transfer has finished either due to an error
or successful completion.

For an upper layer that is a host, command capsules are allocated and
populated with an NVMe SQE by calling nvmf_allocate_command. A data
buffer (described by a struct memdesc) can be associated with a
command capsule before it is transmitted via nvmf_capsule_append_data.
This function accepts a direction (send vs receive) as well as the
data transfer callback. The host then transmits the command via
nvmf_transmit_capsule. The host must ensure that the data buffer
described by the 'struct memdesc' remains valid until the data
transfer callback is called. The queue pair's receive_cb callback
should match received response capsules up with previously transmitted

For the controller, incoming commands are received via the queue
pair's receive_cb callback. nvmf_receive_controller_data is used to
retrieve any data from a command (e.g. the data for a WRITE command).
It can be called multiple times to split the data transfer into
smaller sizes. This function accepts an I/O completion callback that
is invoked once the data transfer has completed.
nvmf_send_controller_data is used to send data to a remote host in
response to a command. In this case a callback function is not used
but the status is returned synchronously. Finally, the controller can
allocate a response capsule via nvmf_allocate_response populated with
a supplied CQE and send the response via nvmf_transmit_capsule.

Sponsored by: Chelsio Communications

Diff Detail

rG FreeBSD src repository
Lint Not Applicable
Tests Not Applicable

Event Timeline

jhb requested review of this revision.Apr 9 2024, 11:03 PM
jhb created this revision.
This revision is now accepted and ready to land.Apr 13 2024, 11:50 PM

Switch to SPDX-only license blocks for C files

This revision now requires review to proceed.May 1 2024, 10:51 PM
This revision was not accepted when it landed; it landed in state Needs Review.May 3 2024, 12:16 AM
This revision was automatically updated to reflect the committed changes.