I've included the Chelsio changes for context in the review, but would split that out into a separate commit.

NIC TLS RX will require additional changes to deal with marking not-yet-decrypted data in the socket buffer as NOTREADY and teaching soreceive() to not return NOTREADY mbufs. I have a sketch in my head for how to extend this further to permit "software" offload, though the intended use case would not be pure software (that works fine in userland) but things like ccr(4) as it is almost certainly more efficient to let hardware crypto engines do the decrypt in the kernel before handing it off to userland than to have userland try to use /dev/crypto to copy the data back into the kernel and out again with extra syscalls, etc.

SO_WANT_KTLS is the part of the design I'm less sure of. It is convenient for TOE TLS, but a software-based KTLS RX would not need it. I'm not sure if it would be beneficial for NIC TLS since the T6 doesn't support NIC TLS for RX. Hans, would it be helpful for you to know "early" that a connection might want to use KTLS RX, or would you allocate a queue, etc. when the RX key is programmed via TCP_TLSRX_ENABLE?

The OpenSSL changes for this are available at https://github.com/openssl/openssl/compare/master...bsdjhb:ktls_rx Note that this also includes two changes to support BIO_conn at the start of the branch so that openssl s_time supports KTLS. I've kept SO_WANT_KTLS separate in that branch so its easy to revert if we don't keep the option.

It would still be useful to know if SO_WANT_KTLS would be useful for Mellanox NIC TLS, but I think I want to at least do an initial software TLS before defining the interface. One thing I noticed while reading the OpenSSL code again is that for the RX case, the Linux KTLS support checks for TLS records already read in the userland openssl buffer and only enables KTLS if only full records are stored, and passes along the number of stored records via an initial sequence number. Software KTLS will need the same thing, so I'm going to need to add a uint64_t sequence number field to the tls_enable structure for that. I can add a temporary combat shim for head for the TXENABLE sockopt to deal with old vs new fields, but I'd like the RX case to work correctly from the get-go. For TOE KTLS, that sequence number should always be zero.

In D22917#505555, @jhb wrote:

It would still be useful to know if SO_WANT_KTLS would be useful for Mellanox NIC TLS, but I think I want to at least do an initial software TLS before defining the interface. One thing I noticed while reading the OpenSSL code again is that for the RX case, the Linux KTLS support checks for TLS records already read in the userland openssl buffer and only enables KTLS if only full records are stored, and passes along the number of stored records via an initial sequence number. Software KTLS will need the same thing, so I'm going to need to add a uint64_t sequence number field to the tls_enable structure for that. I can add a temporary combat shim for head for the TXENABLE sockopt to deal with old vs new fields, but I'd like the RX case to work correctly from the get-go. For TOE KTLS, that sequence number should always be zero.

Eventually the Mellanox NIC TLS will support TLS RX offloading, but I'm a little bit uncertain about the best implementation approach, especially when TLS records are receive out of sequence.

Can't you add some reserved fields to the TLS enable structure perhaps?

So this now supports software KTLS RX as well as TOE. For software KTLS RX, the socket buffer now effectively holds two chains of mbufs. New mbufs from TCP are appending to a new "TLS" chain. When a full TLS record's worth of data is queued, the socket is scheduled to the KTLS worker thread pool. The worker thread walks the TLS chain decrypting individual TLS records. Decrypted TLS records are then appended to the "normal" mbuf chain that is available for reading as records with a control message holding the TLS header. When KTLS RX is enabled, any pending data in the socket buffer is moved to the TLS chain and scheduled for decryption. In addition, the TLS enable structure for the socket option now takes an initial sequence number to handle the case that userland might have done a read-ahead and have pending encrypted TLS records in userland already. OpenSSL already handles this for Linux and doesn't enable KTLS if it has any "partial" records in userland.

This doesn't include support for NIC / ifnet TLS, but it shouldn't be a lot of work to add. For NIC TLS, we could add a new type of send tag for TLSRX (except it would never be stored as a "send tag" in an mbuf). When a NIC decrypts a TLS mbuf, it would pass a M_NOMAP mbuf with the populated TLS header and a length corresponding to the header + trailer up to TCP. In sbappend_ktls we would treat M_NOMAP mbufs with a valid header as an already encrypted mbuf. When they are at the head of the queue we would just allocate the control mbuf for the TLS header and strip the header and trailer from the M_NOMAP mbuf and move them to the regular mbuf chain. If the NIC needs to pass up "plain" data due to a retransmit, it can make the sw_decrypt function pointer point to a routine that decrypts the reassembled TLS frame using the NIC or software, etc. and then resumes decrypting TLS frames in the NIC. Note that this routine would also be invoked for any data already in the socket buffer at the time of the setsockopt call allowing the NIC to start its decryption with the sequence number from the socket option. I think this will work for what Hans has described to me previously for NIC TLS on mlx.

I'm still not sure about whether the SO_WANT_KTLS will be useful for anyone but TOE TLS. We may also want some additional controls (sysctls or otherwise) to control software KTLS RX. In my testing it wasn't really a win compared to AESNI in userland, though it may be useful for lower powered CPUs where you can use a co-processor engine, and NFS over TLS in the kernel will want to use it.

Anyone have any further thoughts? I will probably try to split this up a bit into an initial commit to add the API changes (adding 'rec_seq' and socket options, etc.) along with TOE infrastructure (but perhaps without SO_WANT_KTLS), and then a commit for T6 TOE TLS, then maybe the changes for SW RX separately as they are more invasive (and probably deserve being reviewed on their own). Mostly I wanted to get the SW fleshed out to validate the API changes like adding the rec_seq and additional errors.

I've added the first batch of commit candidates adding the user interface and TOE support. Once those are settled I will rebase this branch and put up a review for the software path. My plan is to use this review as a kind of meta review to track the smaller commits going forward.