Support for TLS offload of TOE connections on T6 adapters.

[jhb]

Summary

The TOE engine in Chelsio T6 adapters supports offloading of TLS
encryption and TCP segmentation for offloaded connections. Sockets
using TLS are required to use a set of custom socket options to upload
RX and TX keys to the NIC and to enable RX processing. Currently
these socket options are implemented as TCP options in the vendor
specific range. A patched OpenSSL library will be made available in a
port / package for use with the TLS TOE support.

TOE sockets can either offload both transmit and reception of TLS
records or just transmit. TLS offload (both RX and TX) is enabled by
setting the dev.t6nex.<x>.tls sysctl to 1 and requires TOE to be
enabled on the relevant interface. Transmit offload can be used on
any "normal" or TLS TOE socket by using the custom socket option to
program a transmit key. This permits most TOE sockets to
transparently offload TLS when applications use a patched SSL library
(e.g. using LD_LIBRARY_PATH to request use of a patched OpenSSL
library). Receive offload can only be used with TOE sockets using the
TLS mode. The dev.t6nex.0.toe.tls_rx_ports sysctl can be set to a
list of TCP port numbers. Any connection with either a local or
remote port number in that list will be created as a TLS socket rather
than a plain TOE socket. Note that although this sysctl accepts an
arbitrary list of port numbers, the sysctl(8) tool is only able to set
sysctl nodes to a single value. A TLS socket will hang without
receiving data if used by an application that is not using a patched
SSL library. Thus, the tls_rx_ports node should be used with care.
For a server mostly concerned with offloading TLS transmit, this node
is not needed as plain TOE sockets will fall back to software crypto
when using an unpatched SSL library.

New per-interface statistics nodes are added giving counts of TLS
packets and payload bytes (payload bytes do not include TLS headers or
authentication tags/MACs) offloaded via the TOE engine, e.g.:

dev.cc.0.stats.rx_tls_octets: 149
dev.cc.0.stats.rx_tls_records: 13
dev.cc.0.stats.tx_tls_octets: 26501823
dev.cc.0.stats.tx_tls_records: 1620

TLS transmit work requests are constructed by a new variant of
t4_push_frames() called t4_push_tls_records() in tom/t4_tls.c.

TLS transmit work requests require a buffer containing IVs. If the
IVs are too large to fit into the work request, a separate buffer is
allocated when constructing a work request. This buffer is associated
with the transmit descriptor and freed when the descriptor is ACKed by
the adapter.

Received TLS frames use two new CPL messages. The first message is a
CPL_TLS_DATA containing the decryped payload of a single TLS record.
The handler places the mbuf containing the received payload on an
mbufq in the TOE pcb. The second message is a CPL_RX_TLS_CMP message
which includes a copy of the TLS header and indicates if there were
any errors. The handler for this message places the TLS header into
the socket buffer followed by the saved mbuf with the payload data.
Both of these handlers are contained in tom/t4_tls.c.

A few routines were exposed from t4_cpl_io.c for use by t4_tls.c
including send_rx_credits(), a new send_rx_modulate(), and
t4_close_conn().

TLS keys for both transmit and receive are stored in onboard memory
in the NIC in the "TLS keys" memory region.

In some cases a TLS socket can hang with pending data available in the
NIC that is not delivered to the host. As a workaround, TLS sockets
are more aggressive about sending CPL_RX_DATA_ACK messages anytime that
any data is read from a TLS socket. In addition, a fallback timer will
periodically send CPL_RX_DATA_ACK messages to the NIC for connections
that are still in the handshake phase. Once the connection has
finished the handshake and programmed RX keys via the socket option,
the timer is stopped.

A new function select_ulp_mode() is used to determine what sub-mode a
given TOE socket should use (plain TOE, DDP, or TLS). The existing
set_tcpddp_ulp_mode() function has been renamed to set_ulp_mode() and
handles initialization of TLS-specific state when necessary in
addition to DDP-specific state.

Since TLS sockets do not receive individual TCP segments but always
receive full TLS records, they can receive more data than is available
in the current window (e.g. if a 16k TLS record is received but the
socket buffer is itself 16k). To cope with this, just drop the window
to 0 when this happens, but track the overage and "eat" the overage as
it is read from the socket buffer not opening the window (or adding
rx_credits) for the overage bytes.

[jhb]

None of the OpenSSL changes would be committed upstream. They are just in my branch and it is easier to generate a review from the full branch. Some of the VERBOSE_TRACES in t4_tls.c might be a bit overboard (multiple traces per received TLS record, etc.) and we might want to trim some of those eventually (they were useful during development but probably less useful now).

sys/dev/cxgbe/firmware/t6fw_cfg.txt

170

These values were just taken from the uwire config.

sys/dev/cxgbe/tom/t4_cpl_io.c

140

I adopted this helper macro from the Linux driver, in particular since the two TLS-related options can be added to the end using variable indices. I could revert back to the expanded version and set paramidx in the two branches of the if and only use paramidx for the two TLS options if you'd prefer that.

493

Technically we could permit this for other socket types as well using the same logic if I just moved rcv_over out into the generic toep instead being in the TLS-specific structure. Other CPL handlers for RX data would have to be patched to set rcv_over if they wanted to make use of it. I think only RDMA connections currently permit rcv_wnd to go negative and it's not clear to me if this would be a help for those or not.

513

TPF_FORCE_CREDITS may need a better name since in this else it is sending a CPL_RX_DATA_ACK with zero credits, but with the RX_MODULATE "force" flag.

734

TLS sockets use the plain TOE path during the handshake phase before the transmit key is negotiated and sent to the card via the socket option.

1880

When using TLS transmit offload, TOE sockets use sb_off and sb_sndptr more like "normal" TCP sockets since we can't assume that work requests operate on mbuf boundaries (the TLS header in the next record might be in the middle of an mbuf in the socket buffer).

2360

We don't support any of the AIO zero-copy stuff for TLS offload. It would perhaps not be too bad to do for TX, but SSL libraries don't use aio_write() so no existing software is going to be using aio_write() for sockets using SSL anyway.

sys/dev/cxgbe/tom/t4_ddp.c

1942

This change and related cleanup in t4_mod_load() I will probably want to commit before the TLS commit itself.

sys/dev/cxgbe/tom/t4_tls.c

570

I've left this debugging bit in my branch but will not included it in the commit.

1082

More debugging that will stay in the branch only.

1408

Last bit of debugging.

sys/dev/cxgbe/tom/t4_tls.h

324

Some of the things in this header that are defining structures used by the firmware it would be nice if they ended up moving to a shared header in common. These are currently all in the t4_tls.h header in the Linux driver as well.

sys/dev/cxgbe/tom/t4_tom.c

1207

The module framework in the kernel invokes MOD_UNLOAD on any module whose MOD_LOAD event fails (I do find this a bit odd, but it is how it works). I would want to commit this separately before the TLS commit itself.