NIC KTLS for Chelsio T6 adapters.

[jhb]

Summary

This adds support for ifnet (NIC) KTLS using Chelsio T6 adapters.
Unlike the TOE-based KTLS in r353328, NIC TLS works with non-TOE
connections.

NIC KTLS on T6 is not able to use the normal TSO (LSO) path to segment
the encrypted TLS frames output by the crypto engine. Instead, the
TOE is placed into a special setup to permit "dummy" connections to be
associated with regular sockets using KTLS. This permits using the
TOE to segment the encrypted TLS records. However, this approach does
have some limitations:

1. Regular TOE sockets cannot be used when the TOE is in this special mode. One can use either TOE and TOE-based KTLS or NIC KTLS, but not both at the same time.

2. In NIC KTLS mode, the TOE is only able to accept a per-connection timestamp offset that varies in the upper 4 bits. Put another way, only connections whose timestamp offset has the 28 lower bits cleared can use NIC KTLS and generate correct timestamps. The driver will refuse to enable NIC KTLS on connections with a timestamp offset with any of the lower 28 bits set. To use NIC KTLS, users can either disable TCP timestamps by setting the net.inet.tcp.rfc1323 sysctl to 0, or by applying a local patch to the tcp_new_ts_offset() function to clear the lower 28 bits of the generated offset.

3. Because the TCP segmentation relies on fields mirrored in a TCB in the TOE, not all fields in a TCP packet can be sent in the TCP segments generated from a TLS record. Specifically, for packets containing TCP options other than timestamps, the driver will inject an "empty" TCP packet holding the requested options (e.g. a SACK scoreboard) along with the segments from the TLS record. These empty TCP packets are counted by the dev.cc.N.txq.M.kern_tls_options sysctls.

Unlike TOE TLS which is able to buffer encrypted TLS records in
on-card memory to handle retransmits, NIC KTLS must re-encrypt TLS
records for retransmit requests as well as non-retransmit requests
that do not include the start of a TLS record but do include the
trailer. The T6 NIC KTLS code tries to optimize some of the cases for
requests to transmit partial TLS records. In particular it attempts
to minimize sending "waste" bytes that have to be given as input to
the crypto engine but are not needed on the wire to satisfy mbufs sent
from the TCP stack down to the driver.

TCP packets for TLS requests are broken down into the following
classes (with associated counters):

• Mbufs that send an entire TLS record in full do not have any waste bytes (dev.cc.N.txq.M.kern_tls_full).

• Mbufs that send a short TLS record that ends before the end of the trailer (dev.cc.N.txq.M.kern_tls_short). For sockets using AES-CBC, the encryption must always start at the beginning, so if the mbuf starts at an offset into the TLS record, the offset bytes will be "waste" bytes. For sockets using AES-GCM, the encryption can start at the 16 byte block before the starting offset capping the waste at 15 bytes.

• Mbufs that send a partial TLS record that has a non-zero starting offset but ends at the end of the trailer (dev.cc.N.txq.M.kern_tls_partial). In order to compute the authentication hash stored in the trailer, the entire TLS record must be sent as input to the crypto engine, so the bytes before the offset are always "waste" bytes.

In addition, other per-txq sysctls are provided:

• dev.cc.N.txq.M.kern_tls_cbc: Count of sockets sent via this txq using AES-CBC.

• dev.cc.N.txq.M.kern_tls_gcm: Count of sockets sent via this txq using AES-GCM.

• dev.cc.N.txq.M.kern_tls_fin: Count of empty FIN-only packets sent to compensate for the TOE engine not being able to set FIN on the last segment of a TLS record if the TLS record mbuf had FIN set.

• dev.cc.N.txq.M.kern_tls_records: Count of TLS records sent via this txq including full, short, and partial records.

• dev.cc.N.txq.M.kern_tls_octets: Count of non-waste bytes (TLS header and payload) sent for TLS record requests.

• dev.cc.N.txq.M.kern_tls_waste: Count of waste bytes sent for TLS record requests.

To enable NIC KTLS with T6, set the following tunables prior to
loading the cxgbe(4) driver:

hw.cxgbe.config_file=kern_tls
hw.cxgbe.kern_tls=1

Test Plan

• tested with KTLS in stock FreeBSD and also tested rather extensively at Netflix

[jhb]

sys/dev/cxgbe/t4_main.c

1278

The atid table change I could break out into a separate commit first if that's better.

2286

I could perhaps break out the snd tag changes to split cxgbe_snd_tag from cxgbe_rate_tag out as well, basically keeping these wrapper routines but removing the KERN_TLS bits from them and handle the structure renames in other places for cxgbe_rate_tag.

sys/dev/cxgbe/t4_sge.c

2938

This change is because certain TLS mbufs might span multiple work requests that in total require more than SGE_MAX_WR_NDESC descriptors even though each individual WR is smaller than that limit. This ensures we have enough descriptors for the next mbuf.

sys/dev/cxgbe/tom/t4_tls.c

435

This change is mostly unrelated (just removing an unused argument) and I will do it separately.

[jhb]

• Rebase.
• Fix build after if_ratelimit_query.
• Use kmod.opts.mk.
• Small cleanups.
• Diff reduce.
• Trim some spurious bits in the diff.

[jhb]

This should now be a commit candidate once the kmod.opts.mk change lands.

[jhb]

• Rebase and update after t4_keyctx.c commit.