Page MenuHomeFreeBSD

Track TCP connection's NUMA domain in the inpcb
ClosedPublic

Authored by gallatin on Apr 23 2019, 5:10 PM.
Tags
None
Referenced Files
Unknown Object (File)
Feb 22 2024, 11:30 PM
Unknown Object (File)
Feb 12 2024, 11:54 PM
Unknown Object (File)
Jan 23 2024, 7:27 AM
Unknown Object (File)
Dec 20 2023, 1:06 AM
Unknown Object (File)
Dec 10 2023, 1:09 PM
Unknown Object (File)
Oct 30 2023, 9:01 PM
Unknown Object (File)
Oct 30 2023, 2:37 AM
Unknown Object (File)
Oct 10 2023, 6:00 PM
Subscribers

Details

Summary

This review teaches two example drivers to pass NUMA domain information up via the mbuf numa domain field. This information is then used by TCP syncache_socket() to associate that information with inpcb. The domain information is then fed back into transmitted mbufs in ip{6}_output(). This mechanism is nearly identical to what is done to track RSS hash values in the inp_flowid.

In this patch, since all of these things are in the critical path, I have been careful to put everything inside #ifdef NUMA.

Once the inpcb has NUMA information, we can do several more interesting things, such as teach lacp which egress port to use based on NUMA domain, bind the TCP pacers to NUMA domains and pace connections on the local domain, teach sendfile(9) where to allocate backing pages, and filter inpcblb_group (SO_REUSEPORT_LB) listen sockets by NUMA domain. I have all this working in the Netflix tree, and will feed these patches in after this. In combination, they reduce cross domain QPI traffic by roughly 50% for a web workload running on two-socket Xeon and increase throughput from 140Gb/s to almost 200Gb/s.

Diff Detail

Repository
rS FreeBSD src repository - subversion
Lint
Lint Skipped
Unit
Tests Skipped
Build Status
Buildable 23850

Event Timeline

LGTM... waiting for more beef :-)

jtl added a subscriber: jtl.

Accepting the inp change as transport role to unblock the review.

This revision is now accepted and ready to land.Apr 25 2019, 2:43 PM

Looks good. mlx driver +1

This revision was automatically updated to reflect the committed changes.