Enter network epoch for network interrupts

[glebius]

Summary

This change pushes the epoch further up in the stack. Basically any
context that is associated with networking and isn't allowed to sleep
runs in epoch.

There are few interfaces that have special context where they
process incoming packets and those are marked with IFF_NEEDSEPOCH.
For such interfaces ether_input() would enter epoch. The plan is
to eliminate this crutch before 13.0-RELEASE.

To prevent endless epoch under heavy interrupt storm there is a
tunable that stops batching packet processing at certain count.

Rough commit split: https://github.com/glebius/FreeBSD/commits/intr-epoch

[jhb]

I'm pretty sure all the NET_EPOCH's in the cxgbe(4) TOM code can go away as part of this.

sys/net/if.h

147 ↗

(On Diff #66933)

Does this mean "driver calls if_input without epoch", or "network stack should call if_input without epoch"? I think the former, but the comment is a bit ambiguous.

[glebius]

Clarify unclear comment.

[olivier]

First forwarding performance impact benches.
On an old PC Engines APU2 (AMD GX 4 core 1Ghz and Intel Gigabit) there is a +6% improvement of inet4 packets-per-second forwarded:

x r356840
+ r356840 with D23242
+--------------------------------------------------------------------------+
|x x   xx   x                                               +   +    ++   +|
| |___AM___|                                                               |
|                                                             |____A_M___| |
+--------------------------------------------------------------------------+
    N           Min           Max        Median           Avg        Stddev
x   5      763950.5      772812.5      768790.5      768164.7     3504.7937
+   5      812003.5        823150        818868      817803.5     4336.1657
Difference at 95.0% confidence
        49638.8 +/- 5749.84
        6.462% +/- 0.767994%
        (Student's t, pooled s = 3942.46)

On a Atom device (8core and 10G Chelsio) there is no improvement on the packets-per-second forwarded rate:

x 356840
+ 356840 with D23242
+--------------------------------------------------------------------------+
|x                              xx        x  ++            +        +   + x|
|         |______________________M__A__________________________|           |
|                                             |___________AM__________|    |
+--------------------------------------------------------------------------+
    N           Min           Max        Median           Avg        Stddev
x   5       4309884       4606689       4439183     4454028.1     106157.11
+   5       4487479       4598784       4547141     4542038.6     50110.922
No difference proven at 95.0% confidence

Result on bigger hardware later (16 cores Xeon and 40G).

[olivier]

And on 16core with 40Gb NIC, + 4% of forwarding speed improvement (pps unit):

x 356840
+ 356840 with D23242
+--------------------------------------------------------------------------+
|x     x    xx      x                                    +     +   + +    +|
|  |_______AM_____|                                                        |
|                                                           |_____AM____|  |
+--------------------------------------------------------------------------+
    N           Min           Max        Median           Avg        Stddev
x   5      30651580      31116776      30914356      30882113     172145.91
+   5      32002188      32402450      32244079      32214895     150605.94
Difference at 95.0% confidence
        1.33278e+06 +/- 235881
        4.31571% +/- 0.782654%
        (Student's t, pooled s = 161735)

[glebius]

I'd really appreciate some reviewing. I'm going to commit this week.

[melifaro]

sys/kern/kern_intr.c

98 ↗

(On Diff #66936)

Could you please elaborate a bit on the reasoning of picking this relatively high default?
I was under impression that batch size of 16 or 32 could amortise the costs pretty close to "ideal", while maintaining worst-case epoch duration lower.

[bz]

Can you please update the diff? I doen't apply cleanly to head (anymore):

bz% patch -C -s -p0 < D23242\?download=true
Reversed (or previously applied) patch detected! Assume -R? [y] n
Apply anyway? [n] n
2 out of 2 hunks ignored while patching sys/kern/uipc_domain.c
Reversed (or previously applied) patch detected! Assume -R? [y] n
Apply anyway? [n] n
2 out of 2 hunks ignored while patching sys/net/if.c
Reversed (or previously applied) patch detected! Assume -R? [y] n
Apply anyway? [n] n
2 out of 2 hunks ignored while patching sys/net/if_var.h
2 out of 6 hunks failed while patching sys/net/pfil.c
bz% svn status sys/kern/uipc_domain.c sys/net/if.c sys/net/if_var.h sys/net/pfil.c
bz%

[glebius]

Rebased to fresh master.

[glebius]

sys/kern/kern_intr.c

98 ↗

(On Diff #66936)

Number is pretty arbitrary. I didn't do much experimenting with it, yet.

[bz]

One small nit on a comment.

If you do the split commits can you please try to tell people as to why the bits are needed the way they are as well. The split github changes are massively helpful to see and understand individual changes, the commit messages there not so much ;-)

sys/sys/interrupt.h

122 ↗

(On Diff #67133)

Start with upper case and ends in '.'