tcp/lro: Add network driver configurable LRO entry depth
AbandonedPublic
Actions

Authored by sepherosa_gmail.com on Jan 8 2016, 2:29 AM.

Details

Reviewers

howard0su_gmail.com
honzhan_microsoft.com
decui_microsoft.com
delphij
glebius
adrian

Group Reviewers

network
transport

Summary

When there is only tiny amount of TCP connections and the host is slow, e.g. in VM, holding too much TCP segments in an LRO entry will cause RX performance degradation. We now allow network drivers to configure how deep one LRO entry should be.

https://reviews.freebsd.org/D4824 has a disabled network driver usage example.

Reviewed by: Hongjiang Zhang <honzhan microsoft com>, Dexuan Cui <decui microsoft com>, Jun Su <junsu microsoft com>
Tested by: me (local), Hongjiang Zhang <honzhan microsoft com> (directly connected 40Ge)
Sponsored by: Microsoft OSTC

BTW, I think some drivers already put a limit on the # of drivers holding TCP segments, e.g. oce(4), though oce(4) does not use per-LRO entry depth.

Diff Detail

Lint

Lint Skipped

Unit

Tests Skipped

Event Timeline

sepherosa_gmail.com updated this revision to Diff 12026.Jan 8 2016, 2:29 AM

sepherosa_gmail.com retitled this revision from to tcp/lro: Add network driver configurable LRO entry depth.

sepherosa_gmail.com updated this object.

sepherosa_gmail.com edited the test plan for this revision. (Show Details)

sepherosa_gmail.com added reviewers: network, adrian, delphij, decui_microsoft.com, honzhan_microsoft.com, howard0su_gmail.com, glebius.

sepherosa_gmail.com added a subscriber: freebsd-net-list.

Herald added a reviewer: transport. · View Herald TranscriptJan 8 2016, 2:29 AM

Have you considered a time-based approach similar to cxgbe(4)? A couple of years ago I experimented with both payload thresholds (like the one you're suggesting) and time-based flush of LRO entries. I found the time-based flush worked better overall and implemented tcp_lro_flush_inactive for general use. The problem with any payload based scheme is that pure ACKs still get stuck with the driver for unhealthy amounts of time.

np added a comment.Jan 8 2016, 6:25 AM

This comment was removed by np.

In D4825#102524, @np wrote:

Have you considered a time-based approach similar to cxgbe(4)? A couple of years ago I experimented with both payload thresholds (like the one you're suggesting) and time-based flush of LRO entries. I found the time-based flush worked better overall and implemented tcp_lro_flush_inactive for general use. The problem with any payload based scheme is that pure ACKs still get stuck with the driver for unhealthy amounts of time.

I just checked the cxgbe(4). I will give it a try. However, find a proper 'limit' still could be pretty tricky...

FYI

D1761 might be related to this one.

Should you check that "lc->lro_hiwat" is greater or equal to "lc->ifp->if_mtu" ?

--HPS

In D4825#110653, @hselasky wrote:

FYI

D1761 might be related to this one.

Should you check that "lc->lro_hiwat" is greater or equal to "lc->ifp->if_mtu" ?

--HPS

I have discarded this one, please take a look at this:
https://reviews.freebsd.org/D5185

Thanks,
sephe

Updated version at:
https://reviews.freebsd.org/D5185