Change Details

When we implemented NIC kTLS state, we set a flag in the tx socket buffer (SB_TLS_IFNET) to indicate NIC kTLS. This flag meant that now, or in the past, NIC kTLS was active on a socket. Later, I added code to switch TLS sessions to software in the case of lossy TCP connections that have a high retransmit rate. Because TCP was using SB_TLS_IFNET, this meant that even long after a TLS session was switched to software, we'd be doing math to calculate the rxmt rate in tcp_account_for_send(), and making potentially spurious calls into ktls_disable_ifnet(). This patch carefully tracks whether or not ifnet ktls is still enabled on a TCP connection. Because the inp is now embedded in the tcbcb, and because tcp is the most frequent accessor of this state, it made sense to move this from the socket buffer to the tcpcb. Because we now need reliable access to the tcbcb, we take a ref on the inp when creating a tx ktls session.. this actually seems to simplify a few things. While here, I noticed that rack/bbr were incorrectly implementing tfb_hwtls_change(), and applying the change to all pending sends, when it should apply only to future sends. This patch reduces spurious calls to ktls_disable_ifnet() by 95% or so in our environment.

When we implemented NIC kTLS state, we set a flag in the tx socket buffer (SB_TLS_IFNET) to indicate NIC kTLS. This flag meant that now, or in the past, NIC kTLS was active on a socket. Later, I added code to switch TLS sessions to software in the case of lossy TCP connections that have a high retransmit rate. Because TCP was using SB_TLS_IFNET, this meant that even long after a TLS session was switched to software, we'd be doing math to calculate the rxmt rate in tcp_account_for_send(), and making potentially spurious calls into ktls_disable_ifnet(). This patch carefully tracks whether or not ifnet ktls is still enabled on a TCP connection. Because the inp is now embedded in the tcbcb, and because tcp is the most frequent accessor of this state, it made sense to move this from the socket buffer to the tcpcb. Because we now need reliable access to the tcbcb, we take a ref on the inp when creating a tx ktls session.. this actually seems to simplify a few things. While here, I noticed that rack/bbr were incorrectly implementing tfb_hwtls_change(), and applying the change to all pending sends, when it should apply only to future sends. This patch reduces spurious calls to ktls_disable_ifnet() by 95% or so in our environment. Note that the new flags in the tcpcb fit into an alignment hole, and do not increase the size of the struct.

When we implemented NIC kTLS state, we set a flag in the tx socket buffer (SB_TLS_IFNET) to indicate NIC kTLS. This flag meant that now, or in the past, NIC kTLS was active on a socket. Later, I added code to switch TLS sessions to software in the case of lossy TCP connections that have a high retransmit rate. Because TCP was using SB_TLS_IFNET, this meant that even long after a TLS session was switched to software, we'd be doing math to calculate the rxmt rate in tcp_account_for_send(), and making potentially spurious calls into ktls_disable_ifnet(). This patch carefully tracks whether or not ifnet ktls is still enabled on a TCP connection. Because the inp is now embedded in the tcbcb, and because tcp is the most frequent accessor of this state, it made sense to move this from the socket buffer to the tcpcb. Because we now need reliable access to the tcbcb, we take a ref on the inp when creating a tx ktls session.. this actually seems to simplify a few things. While here, I noticed that rack/bbr were incorrectly implementing tfb_hwtls_change(), and applying the change to all pending sends, when it should apply only to future sends. This patch reduces spurious calls to ktls_disable_ifnet() by 95% or so in our environment. Note that the new flags in the tcpcb fit into an alignment hole, and do not increase the size of the struct.