Add support for KTLS via the TOE on Chelsio's T6 adapters.

[jhb]

Summary

Add a TOE KTLS mode and a TOE hook for allocationg TLS sessions.

This adds the glue to allocate TLS sessions and invokes it from
the TLS enable socket option handler. This also adds some counters
for active TOE sessions.

The TOE KTLS mode is returned by getsockopt(TLSTX_TLS_MODE) when
TOE KTLS is in use on a socket, but cannot be set via setsockopt().

Add support for KTLS in the Chelsio TOE module.

This adds a TOE hook to allocate a KTLS session. It also recognizes
TLS mbufs in the socket buffer and sends those to the NIC using a TLS
work request to encrypt the record before segmenting it.

Test Plan

• tested with nginx and KTLS, able to send at about the same rate (70 Gbps) as NIC TLS on T6, though all the tests performed with TOE KTLS have been client-limited.

[gallatin]

sys/dev/cxgbe/tom/t4_tls.c

1686 ↗

(On Diff #62886)

Is all this trickery to reduce sgl lengths really worth it for what's going to be 4K chunks? That's the largest PCIe max read request size anyway, so the device is going to have to split the segments that you've combined regardless. All we're doing here is maybe compressing the SGL,s but it just doesn't seem worth the effort to me.

I think the tradeoff is different here than in what might be a series of 128b mbufs written on a socket, for example.

sys/kern/uipc_ktls.c

706 ↗

(On Diff #62886)

Shouldn't this be ktls_toe_{cbc,gcm} ?

[jhb]

sys/dev/cxgbe/tom/t4_tls.c

1686 ↗

(On Diff #62886)

Hmm. Ironically pretty much all the bugs I had were in this function, though they weren't due to dealing with coalescing (and at this point they are all resolved). The other interesting thing is that the main driver also tries to coalesce ext_pgs mbufs as well as the NIC TLS code, and arguably it would simplify all of those to not bother with coalescing. I'll talk to np@ to see what the tradeoff is for coalescing SGL entries and what effect that might have (e.g. it might also use less resources on the NIC itself for all I know to store the more compact SGL which might matter when you have a lot of active connections)

sys/kern/uipc_ktls.c

706 ↗

(On Diff #62886)

Yes, I fixed this yesterday when i noticed the counters were wrong and need to update the diff.

[np]

sys/dev/cxgbe/tom/t4_tls.c

1686 ↗

(On Diff #62886)

Compressing SGLs so that they use fewer tx descriptors is definitely worth it for any TOE txq (ulp mode crypto or not). The the per-tid toep->tx_credits are in high demand during tx and it makes sense to pack as much as possible into as few len16 tx credits for bulk transmit.

[jhb]

• Increment correct counters.

[jhb]

sys/kern/uipc_ktls.c

1032 ↗

(On Diff #63005)

Should be 'tls_new->mode'. Will fix.