Change smarter epoch support for NIC drivers to opt-in.

[glebius]

Summary

Although most of the drivers are epoch ready, due to peer pressure
switch over to opt-in instead of opt-out for epoch.

Instead of IFF_NEEDSEPOCH, provide IFF_KNOWSEPOCH. If driver marks
itself with IFF_KNOWSEPOCH, when ether_input() would not enter epoch
when processing its packets.

Now this will create recursive entrance in epoch in >90% network
drivers, but will guarantee safeness of the transition.

Mark tested drivers as IFF_KNOWSEPOCH.

[• hselasky]

Looks good to me.

[jeff]

What about the interrupt changes?

[olivier]

So, if I've correctly understood:

• This will create recursive entrance in epoch with drivers that didn't support IFF_KNOWSEPOCH
• iflib supports IFF_KNOWSEPOCH (because this patch add it to sys/net/iflib.c) but Chelsio didn't support it (because I don't see any modification to the Chelsio drivers here)

So theoretically, I shouldn't find performance difference on my forwarding lab using iflib drivers, but "perhaps" a small degradation with Chelsio because the recursive entrance:

• My results show no performance impact with iflib neither with Chelsio.

Number of packets-per-second forwarded with a 10G Chelsio:

x FreeBSD r357884
+ FreeBSD r357884 and D23674
+--------------------------------------------------------------------------+
|x                                      +   +        x    + *+        x    |
|                   |__________________________A_____M____________________||
|                                          |_________A____M___|            |
+--------------------------------------------------------------------------+
    N           Min           Max        Median           Avg        Stddev
x   5     7301523.5       8641651       8316751     8202397.8     521321.49
+   5     8069901.5       8464468       8419314       8307376     187557.28
No difference proven at 95.0% confidence

Same but with Intel 1G:

x FreeBSD r357884
+ FreeBsD r357884 with D23674
+--------------------------------------------------------------------------+
|x                                                     +*  * x*   +        |
|                     |_________________________A__________M______________||
|                                                      |___A____|          |
+--------------------------------------------------------------------------+
    N           Min           Max        Median           Avg        Stddev
x   5        916923       1123971     1115717.5     1076802.5     89713.215
+   5       1100496       1138467       1114737     1116772.6     15947.773
No difference proven at 95.0% confidence