net/tso: use the most restrictive segment size to perform the calculations

[royger]

Summary

The current TSO segment size accounting code was not taking into account
that the link segment size can be smaller than the maximum segment size
supported by the network card. Without this change, the if_hw_tsomaxsegcount
limit could be ignored, and the mbuf chain created by the TSO code could
indeed have more segments than supported.

Sponsored by: Citrix Systems R&D

Test Plan

FreeBSD DomU PVHVM guests cannot 'route' traffic for other Xen PV guests on same Dom0 Host.
I am linking to the PR that this is suppose to be part of the fix for.

[gallatin]

I'm confused. t_maxsegsize is basically the connection's MSS, eg, what the NIC is supposed to break the TSO into. However if_hw_tsomaxsegsize is the max size of the TSO to be broken up. Eg, on a NIC without oddball limitations, if_hw_tsomaxsegcount might be 64, if_hw_tsomaxsegsize might be 65535. That means that TSO is supposed to send a max of 64K down to the NIC, in a max of 64 packets. If t_maxseg is less than 1000, then we'd have to limit the TSO to 64 packets.

It seems like by restricting if_hw_tsomaxsegsize to t_maxseg, you're essentially turning off TSO, since you're only allowing one MSS to be sent at a time.

[gallatin]

In D6656#578326, @gallatin wrote:

I'm confused. t_maxsegsize is basically the connection's MSS, eg, what the NIC is supposed to break the TSO into. However if_hw_tsomaxsegsize is the max size of the TSO to be broken up. Eg, on a NIC without oddball limitations, if_hw_tsomaxsegcount might be 64, if_hw_tsomaxsegsize might be 65535. That means that TSO is supposed to send a max of 64K down to the NIC, in a max of 64 packets. If t_maxseg is less than 1000, then we'd have to limit the TSO to 64 packets.

It seems like by restricting if_hw_tsomaxsegsize to t_maxseg, you're essentially turning off TSO, since you're only allowing one MSS to be sent at a time.

Ugh, I'm wrong. I was confusing t_tsomax with t_tsomaxsegsize. I forgot that some NICs are apparently so broken that they limit TSO seg sizes.

[royger]

In D6656#578863, @gallatin wrote:

In D6656#578326, @gallatin wrote:

I'm confused. t_maxsegsize is basically the connection's MSS, eg, what the NIC is supposed to break the TSO into. However if_hw_tsomaxsegsize is the max size of the TSO to be broken up. Eg, on a NIC without oddball limitations, if_hw_tsomaxsegcount might be 64, if_hw_tsomaxsegsize might be 65535. That means that TSO is supposed to send a max of 64K down to the NIC, in a max of 64 packets. If t_maxseg is less than 1000, then we'd have to limit the TSO to 64 packets.

It seems like by restricting if_hw_tsomaxsegsize to t_maxseg, you're essentially turning off TSO, since you're only allowing one MSS to be sent at a time.

Ugh, I'm wrong. I was confusing t_tsomax with t_tsomaxsegsize. I forgot that some NICs are apparently so broken that they limit TSO seg sizes.

So this is correct? Or am I missing something else?

[gallatin]

In D6656#579330, @royger wrote:

In D6656#578863, @gallatin wrote:

In D6656#578326, @gallatin wrote:

I'm confused. t_maxsegsize is basically the connection's MSS, eg, what the NIC is supposed to break the TSO into. However if_hw_tsomaxsegsize is the max size of the TSO to be broken up. Eg, on a NIC without oddball limitations, if_hw_tsomaxsegcount might be 64, if_hw_tsomaxsegsize might be 65535. That means that TSO is supposed to send a max of 64K down to the NIC, in a max of 64 packets. If t_maxseg is less than 1000, then we'd have to limit the TSO to 64 packets.

It seems like by restricting if_hw_tsomaxsegsize to t_maxseg, you're essentially turning off TSO, since you're only allowing one MSS to be sent at a time.

Ugh, I'm wrong. I was confusing t_tsomax with t_tsomaxsegsize. I forgot that some NICs are apparently so broken that they limit TSO seg sizes.

So this is correct? Or am I missing something else?

I think it is correct, but I would prefer somebody from the transport team to review it.

[• hselasky]

Hi Drew,

t_tsomaxsegsize usually only affects the number of DMA segments and the mbuf chain. At least the Mellanox NICS do it this way.

How does t_maxseg affect the generated DMA chain?

--HPS

[• hselasky]

I mean DMA -> mbuf chain.

[gallatin]

In D6656#579632, @hselasky wrote:

I mean DMA -> mbuf chain.

if_hw_tsomaxsegsize is used to calculate the size of the packets the NIC is going to emit, and tcp_m_copym uses it to solve other constraints (eg, if_hw_tsomaxsegcount)

I do think that this change is probably correct. If so, it needs to also be applied to rack and bbr.

(/rant) I honestly hate this weird impedance matching attempt between TSO and DMA. We should just do what linux does, and punt to software TSO for broken hardware that can't send something. (/end rant)