Page MenuHomeFreeBSD
Differential D26523

Enable multipath for outbound connections
ClosedPublic
Actions

Authored by melifaro on Sep 22 2020, 7:47 PM.
Tags
None
Referenced Files
F106197393: D26523.id78337.diff
Fri, Dec 27, 12:19 AM
F106164058: D26523.diff
Thu, Dec 26, 10:56 AM
Unknown Object (File)
Thu, Dec 5, 6:48 PM
Unknown Object (File)
Wed, Dec 4, 11:49 AM
Unknown Object (File)
Mon, Dec 2, 10:32 AM
Unknown Object (File)
Mon, Dec 2, 10:31 AM
Unknown Object (File)
Mon, Dec 2, 10:31 AM
Unknown Object (File)
Mon, Dec 2, 10:31 AM
View All Files
Subscribers
ae
asomers
bz
emaste
imp

Details

Reviewers
glebius
Group Reviewers
network
transport
Commits
rS366813: Implement flowid calculation for outbound connections to balance
Summary

Implement flowid calculation for outbound connections to enable splitting connections over multiple paths.

Multipath routing relies on mbuf flowid data for both transit and outbound traffic.
Current code fills mbuf flowid from inp_flowid for connection-oriented sockets.
However, inp_flowid is currently not calculated for outbound connections.

This diff creates simple hashing functions and starts calculating hashes for TCP,UDP/UDP-Lite and raw IP if multipath routes are present in the system.

RSS and ROUTE_MPATH

Multipath routing currently does not require any special properties from the hashing function, except good flow distribution.
RSS for outbound connections (which is actually XPS) require hashing function to be compatible with card hashing to achieve optimal TX ring assignment.

RSS requires PCBGROUP, which makes coupling RSS and ROUTE_MPATH by requiring RSS inconvenient.

With that in mind,

  • use RSS-derived hashing functions for outbound connections when RSS in enabled and
  • use light-weight ROUTE_MPATH hashing functions otherwise
  • implement a simple wrapper layer (see net/route.h) to switch between the implementation without touching callers.
NOTE: I intend to commit this change on Sunday, October 16

Diff Detail

Repository
rS FreeBSD src repository - subversion
Lint
Lint Not Applicable
Unit
Tests Not Applicable

Event Timeline

melifaro created this revision.Sep 22 2020, 7:47 PM
Herald added a subscriber: ae. · View Herald TranscriptSep 22 2020, 7:47 PM
Harbormaster completed remote builds in B33739: Diff 77373.Sep 22 2020, 7:47 PM
melifaro requested review of this revision.Sep 22 2020, 7:47 PM
melifaro edited the summary of this revision. (Show Details)Sep 22 2020, 7:49 PM
melifaro added reviewers: glebius, network.
melifaro edited the summary of this revision. (Show Details)Sep 22 2020, 7:52 PM
melifaro mentioned this in D26449: Stage 2: Introduce scalable route multipath.Sep 22 2020, 8:00 PM
melifaro edited the summary of this revision. (Show Details)Sep 22 2020, 8:03 PM
melifaro added a parent revision: D26449: Stage 2: Introduce scalable route multipath.
ae added a comment.Sep 23 2020, 1:38 PM

It seems you missed hash calculation for udp[6]_output(), when socket isn't connected and destination address is specified by caller.

melifaro mentioned this in D24141: Implementation of scalable multipath & nexthop objects introduction..Sep 23 2020, 8:48 PM
glebius accepted this revision.Sep 23 2020, 9:09 PM
This revision is now accepted and ready to land.Sep 23 2020, 9:09 PM
melifaro mentioned this in rS366390: Introduce scalable route multipath..Oct 3 2020, 10:48 AM
emaste added a subscriber: emaste.Oct 4 2020, 5:12 PM
emaste added inline comments.
sys/net/route/mpath_ctl.c
72 ↗(On Diff #77373)

Where does this come from?

melifaro added inline comments.Oct 4 2020, 6:30 PM
sys/net/route/mpath_ctl.c
72 ↗(On Diff #77373)

That's the default key from Microsoft RSS spec, copied from rss_config.c

melifaro updated this revision to Diff 77988.Oct 7 2020, 7:45 PM

Fix raw IPv6, add connectionless UDP.
Fix tests.

This revision now requires review to proceed.Oct 7 2020, 7:45 PM
Herald added a reviewer: transport. · View Herald TranscriptOct 7 2020, 7:45 PM
Herald added subscribers: bz, asomers, imp. · View Herald Transcript
Harbormaster completed remote builds in B34045: Diff 77988.Oct 7 2020, 7:45 PM
ae added inline comments.Oct 9 2020, 12:00 PM
sys/netinet/udp_usrreq.c
1480 ↗(On Diff #77988)

It seems you forgot #include "opt_route.h" and this code was not built. :)
Note, a few lines below we have code that sets flowid and flowtype.

sys/netinet6/udp6_usrreq.c
79 ↗(On Diff #77988)

Also opt_route.h is missing.

968 ↗(On Diff #77988)

Probably we don't need to calculate flowid twice when both ROUTE_MPATH and RSS are defined.

melifaro updated this revision to Diff 78336.Oct 16 2020, 10:03 PM

Improve ROUTE_MPATH & RSS configurations.

Harbormaster completed remote builds in B34222: Diff 78336.Oct 16 2020, 10:03 PM
melifaro edited the summary of this revision. (Show Details)Oct 16 2020, 10:20 PM
melifaro updated this revision to Diff 78337.Oct 16 2020, 10:34 PM
melifaro marked an inline comment as done.

Address ae@ comment on udp6_usrreq.

Harbormaster completed remote builds in B34223: Diff 78337.Oct 16 2020, 10:34 PM
melifaro edited the summary of this revision. (Show Details)Oct 16 2020, 10:35 PM
ae added a comment.Oct 17 2020, 3:17 PM

LGTM. However it would be nice, if you consider my comments :)

sys/net/route.h
136 ↗(On Diff #78337)

I think it would be better to name these all with upper case, to have understanding from the code that uses them that we use macro. Or maybe move all hashing related code to the same place, and do conditional build there.

sys/net/route/mpath_ctl.c
68 ↗(On Diff #78337)

Maybe we should allow set it from loader.conf? I.e. use CTLFLAG_RDTUN flag?

This revision was not accepted when it landed; it landed in state Needs Review.Oct 18 2020, 5:16 PM
Closed by commit rS366813: Implement flowid calculation for outbound connections to balance (authored by melifaro). · Explain Why
This revision was automatically updated to reflect the committed changes.
melifaro added a commit: rS366813: Implement flowid calculation for outbound connections to balance.

Revision Contents
Changeset List

PathSize
head/
sys/
conf/
2 lines
net/
17 lines
34 lines
route/
25 lines
4 lines
netinet/
3 lines
35 lines
11 lines
2 lines
42 lines
23 lines
32 lines
netinet6/
3 lines
34 lines
12 lines
3 lines
44 lines
6 lines
13 lines
42 lines

Diff 78382

View Options

head/sys/conf/files

Loading...
View Options

head/sys/net/radix.c

Loading...
View Options

head/sys/net/route.h

Loading...
View Options

head/sys/net/route/mpath_ctl.c

Loading...
View Options

head/sys/net/route/route_var.h

Loading...
View Options

head/sys/netinet/in_fib.h

Loading...
View Options

head/sys/netinet/in_fib.c

Loading...
View Options

head/sys/netinet/in_pcb.c

Loading...
View Options

head/sys/netinet/in_rss.h

Loading...
View Options

head/sys/netinet/in_rss.c

Loading...
View Options

head/sys/netinet/raw_ip.c

Loading...
View Options

head/sys/netinet/udp_usrreq.c

Loading...
View Options

head/sys/netinet6/in6_fib.h

Loading...
View Options

head/sys/netinet6/in6_fib.c

Loading...
View Options

head/sys/netinet6/in6_pcb.c

Loading...
View Options

head/sys/netinet6/in6_rss.h

Loading...
View Options

head/sys/netinet6/in6_rss.c

Loading...
View Options

head/sys/netinet6/ip6_output.c

Loading...
View Options

head/sys/netinet6/raw_ip6.c

Loading...
View Options

head/sys/netinet6/udp6_usrreq.c

Loading...