tcp/lro: Add network driver configurable LRO entry depth

[sepherosa_gmail.com]

Summary

When there is only tiny amount of TCP connections and the host is slow, e.g. in VM, holding too much TCP segments in an LRO entry will cause RX performance degradation. We now allow network drivers to configure how deep one LRO entry should be.

https://reviews.freebsd.org/D4824 has a disabled network driver usage example.

Reviewed by: Hongjiang Zhang <honzhan microsoft com>, Dexuan Cui <decui microsoft com>, Jun Su <junsu microsoft com>
Tested by: me (local), Hongjiang Zhang <honzhan microsoft com> (directly connected 40Ge)
Sponsored by: Microsoft OSTC

BTW, I think some drivers already put a limit on the # of drivers holding TCP segments, e.g. oce(4), though oce(4) does not use per-LRO entry depth.

[np]

Have you considered a time-based approach similar to cxgbe(4)? A couple of years ago I experimented with both payload thresholds (like the one you're suggesting) and time-based flush of LRO entries. I found the time-based flush worked better overall and implemented tcp_lro_flush_inactive for general use. The problem with any payload based scheme is that pure ACKs still get stuck with the driver for unhealthy amounts of time.

[np]

This comment was removed by np.

[sepherosa_gmail.com]

In D4825#102524, @np wrote:

Have you considered a time-based approach similar to cxgbe(4)? A couple of years ago I experimented with both payload thresholds (like the one you're suggesting) and time-based flush of LRO entries. I found the time-based flush worked better overall and implemented tcp_lro_flush_inactive for general use. The problem with any payload based scheme is that pure ACKs still get stuck with the driver for unhealthy amounts of time.

I just checked the cxgbe(4). I will give it a try. However, find a proper 'limit' still could be pretty tricky...

[• hselasky]

FYI

D1761 might be related to this one.

Should you check that "lc->lro_hiwat" is greater or equal to "lc->ifp->if_mtu" ?

--HPS

[sepherosa_gmail.com]

In D4825#110653, @hselasky wrote:

FYI

D1761 might be related to this one.

Should you check that "lc->lro_hiwat" is greater or equal to "lc->ifp->if_mtu" ?

--HPS

I have discarded this one, please take a look at this:
https://reviews.freebsd.org/D5185

Thanks,
sephe

[sepherosa_gmail.com]

Updated version at:
https://reviews.freebsd.org/D5185