Enable ROUTE_MPATH support in GENERIC kernels.

[melifaro]

Summary

Enable ROUTE_MPATH support in GENERIC kernels.

Ability to load-balance traffic over multiple path is a must-have thing for routers.
It may be used by the servers to balance outgoing traffic over multiple default gateways.

The previous implementation, RADIX_MPATH stayed in the shadow for too long. It was not well maintained, which lead us to a vicious circle - people were using non-contiguous mask or firewalls to achieve similar goals. As a result, some routing daemons implementation still don't have multipath support enabled for FreeBSD.

Turning on ROUTE_MPATH by default would fix it. It will allow to reduce networking feature gap to other operating systems. Linux and OpenBSD has the support for at least 5 years.

ROUTE_MPATH does not consume memory unless actually used. It enables around ~1k LOC.

It does not bring any behaviour changes for userland.
Additionally, feature is (temporarily) turned off by the net.route.multipath sysctl defaulting to 0.

Timeline

1. Enable ROUTE_MPATH code compile by default (this review, targetting first week of December)
2. Turn on net.route.multipath sysctl to 1 by default. (followup review, targetting middle of December)

[olivier]

Excellent idea to enable it!

Here here my regression test bench results (only 2 static routes) by adding this kernel option (sysctl still using the default, so not enabled):

On a small device (AMD GX-412TC 4Cores):

x r368200: inet packets-per-second forwarded
+ r368200 with D27428: inet packets-per-second forwarded
+--------------------------------------------------------------------------+
|+                               xxx    +                +        + +      |
|                                |A                                        |
|                  |__________________________A__________M________________||
+--------------------------------------------------------------------------+
    N           Min           Max        Median           Avg        Stddev
x   5      744726.5        749883        747013      747129.5     2330.0344
+   5      659212.5        839278        809395      781445.1     74499.772
No difference proven at 95.0% confidence

On a Xeon E5-2650 8Cores with Chelsio_T540-CR (10Gb/s): small impact.

x r368200: inet4 packets-per-second forwarded
+ r368200 with D27428: inet4 packets-per-second forwarded
+--------------------------------------------------------------------------+
|                                           +                              |
|+                                       ++ +                      xx x x x|
|                                                                  |__A__| |
|               |_________________A_______M__________|                     |
+--------------------------------------------------------------------------+
    N           Min           Max        Median           Avg        Stddev
x   5     9803502.5     9843579.5       9822232     9822433.6     15632.275
+   5       9449022       9682297       9670171     9629134.8     100921.96
Difference at 95.0% confidence
        -193299 +/- 105319
        -1.96793% +/- 1.07174%
        (Student's t, pooled s = 72213.6)

[melifaro]

In D27428#612943, @olivier wrote:

Excellent idea to enable it!

Here here my regression test bench results (only 2 static routes) by adding this kernel option (sysctl still using the default, so not enabled):

Thank you for testing it!

On a Xeon E5-2650 8Cores with Chelsio_T540-CR (10Gb/s): small impact.

x r368200: inet4 packets-per-second forwarded
+ r368200 with D27428: inet4 packets-per-second forwarded
+--------------------------------------------------------------------------+
|                                           +                              |
|+                                       ++ +                      xx x x x|
|                                                                  |__A__| |
|               |_________________A_______M__________|                     |
+--------------------------------------------------------------------------+
    N           Min           Max        Median           Avg        Stddev
x   5     9803502.5     9843579.5       9822232     9822433.6     15632.275
+   5       9449022       9682297       9670171     9629134.8     100921.96
Difference at 95.0% confidence
        -193299 +/- 105319
        -1.96793% +/- 1.07174%
        (Student's t, pooled s = 72213.6)

That's interesting. Effective datapath changes contains only 1 additional branch (condition inside nhop_select()) and still that corresponds to 2% difference.
Do you by any chance have remember, what is the mean deviation between the tests on the same revision?
Do you by any chance have a historical graph with other revision's performance?

[olivier]

The latest historical data I have on this hardware are those:

x r364730: inet packets-per-second forwarded
+ r367714: inet packets-per-second forwarded
* r368200: inet packets-per-second forwarded
+--------------------------------------------------------------------------+
|x xx   x x      *** **        +                               + +++       |
||__MA___|                                                                 |
|                                          |______________A______M________||
|                |_A__|                                                    |
+--------------------------------------------------------------------------+
    N           Min           Max        Median           Avg        Stddev
x   5       9697454       9755700       9716136     9724960.1     24760.638
+   5       9899883      10149413      10132167      10088669     105990.12
Difference at 95.0% confidence
        363709 +/- 112248
        3.73995% +/- 1.1565%
        (Student's t, pooled s = 76964.3)
*   5     9803502.5     9843579.5       9822232     9822433.6     15632.275
Difference at 95.0% confidence
        97473.5 +/- 30198.2
        1.0023% +/- 0.312751%
        (Student's t, pooled s = 20705.8)

[melifaro]

I'm going to commit this diff on Saturday, December 5, unless I get any objections.