User Details
- User Since
- Feb 5 2016, 2:19 PM (197 w, 3 d)
Oct 16 2019
Sep 27 2019
I prefer this change over my previous improvements.
Sep 24 2019
Forgot to close this old revision.
Sep 20 2019
After removing those 2 lines:
3875 if (txq->ift_in_use) 3876 txq->ift_br->drain(txq->ift_br, NULL, 0, 0);
The result is now different (7% improvement):
x bypass_mpring=0 (default) and tx_abdicate=1: inet4 packets-per-second forwarded
+ bypass_mpring=1 and tx_abdicate=1: inet4 packets-per-second forwarded
+--------------------------------------------------------------------------+
|x x x x x + ++ + +|
||_______M___A___________| |
| |____M_A_____| |
+--------------------------------------------------------------------------+
N Min Max Median Avg Stddev
x 5 1047806.5 1091725 1059590 1066031.6 17660.615
+ 5 1132410 1156922 1141373 1144724.6 9694.1874
Difference at 95.0% confidence
78693 +/- 20776.4
7.38186% +/- 2.0604%
(Student's t, pooled s = 14245.6)Sep 19 2019
So new result with new patch's version:
x bypass_mpring=0 (default) and tx_abdicate=1: inet4 packets-per-second forwarded
+ bypass_mpring=1 and tx_abdicate=1: inet4 packets-per-second forwarded
+--------------------------------------------------------------------------+
|+ ++ + + x x x x |
| |_______A_M_____||
| |______M_A________| |
+--------------------------------------------------------------------------+
N Min Max Median Avg Stddev
x 5 1043042 1084017 1074558 1070992.4 16759.158
+ 5 938950.5 984677 959011 963327.5 17971.342
Difference at 95.0% confidence
-107665 +/- 25341.6
-10.0528% +/- 2.25866%
(Student's t, pooled s = 17375.8)Sep 17 2019
tx_abdicate was not set (default value of 0) for these benches.
If I'm setting tx_abdicate, here are the new results:
x bypass_mpring=0 (default) and tx_abdicate=1: inet4 packets-per-second forwarded
+ bypass_mpring=1 and tx_abdicate=1: inet4 packets-per-second forwarded
+--------------------------------------------------------------------------+
| + |
| + |
|+ ++ x x xx x|
| |____MA_____| |
| |___A_M__| |
+--------------------------------------------------------------------------+
N Min Max Median Avg Stddev
x 5 1060792 1099642 1069370 1073776.1 15108.116
+ 5 906915.5 934633 930094 926433 11078.667
Difference at 95.0% confidence
-147343 +/- 19320.7
-13.722% +/- 1.64298%
(Student's t, pooled s = 13247.5)Wit tx_abdicate is enabled there is about -13% of performance reduction once bypass_mpring is enabled.
Sep 12 2019
Following the code with TUNABLE_INT_FETCH("dev.%s.%d.iflib.bypass_mpring"), I've added these line in the /boot/loader.conf.local:
dev.igb.0.iflib.bypass_mpring=1 dev.igb.1.iflib.bypass_mpring=1 dev.igb.2.iflib.bypass_mpring=1 dev.igb.3.iflib.bypass_mpring=1 dev.igb.4.iflib.bypass_mpring=1 dev.igb.5.iflib.bypass_mpring=1
New results with this new version on atom C2558 (4 cores).
Aug 28 2019
Aug 27 2019
Yes you've right! The correct label should be "bypass_mpring" disabled/enabled... which revert the logic.
Aug 26 2019
Flamegraph (fixed labels):
Aug 24 2019
Aug 15 2019
On another hardware (Atom 4cores):
x inet4 forwarding of small packet size, bypass_mpring=0 in packets-per-second
+ inet4 forwarding of small packet size, bypass_mpring=1 in packets-per-second
+--------------------------------------------------------------------------+
|+ + ++ + x x x x |
| |______A__M____||
| |_______________A_____M________| |
+--------------------------------------------------------------------------+
N Min Max Median Avg Stddev
x 4 927858.5 983285 971934 963752.88 24606.11
+ 5 749042 872241 859294 839484.8 51106.313
Difference at 95.0% confidence
-124268 +/- 66405
-12.8942% +/- 6.76603%
(Student's t, pooled s = 41856.6)Bigger impact her: -12%
Aug 14 2019
Results on PC Engines APU2 (notice that I had to set back iflib.tx_abdicate=0, because if=1, all packets are dropped in another dimension):
x inet4 forwarding of small packet size, bypass_mpring=0 in packets-per-second
+ inet4 forwarding of small packet size, bypass_mpring=1 in packets-per-second
+--------------------------------------------------------------------------+
|+ + + + + x x x xx|
| |____A___||
| |_________A____M_____| |
+--------------------------------------------------------------------------+
N Min Max Median Avg Stddev
x 5 462506 464435 463851 463697.4 759.63235
+ 5 451908 456286 455316 454538 1785.9727
Difference at 95.0% confidence
-9159.4 +/- 2001.51
-1.9753% +/- 0.430346%
(Student's t, pooled s = 1372.36)Aug 10 2019
Jul 15 2019
Update following Enji's advice.
Jul 14 2019
Jul 12 2019
Jun 27 2019
May 28 2019
May 13 2019
Let's try the benefit of this patch on a 8-core Atom with a 10G Chelsio, during a forwarding bench with tcdpump running:
x r347519: inet4 smallest size packets-per-second forwarded with tcpdump running
+ r347519 and D20224: inet4 smallest size packets-per-second forwarded with tcpdump running
+--------------------------------------------------------------------------+
| xx x + + + ++|
||___M__A______| |
| |___AM___||
+--------------------------------------------------------------------------+
N Min Max Median Avg Stddev
x 5 3682162 3816554 3684703 3711296.2 58889.489
+ 5 4179733.5 4274492 4240164 4234216.7 39505.425
Difference at 95.0% confidence
522920 +/- 73130.8
14.09% +/- 2.16578%
(Student's t, pooled s = 50143)Apr 30 2019
Apr 23 2019
Applying ngie's comments
Apr 19 2019
Apr 18 2019
Apr 16 2019
Apr 15 2019
Apr 5 2019
Apr 4 2019
Apr 1 2019
Mar 29 2019
Mar 28 2019
Mar 26 2019
Mar 21 2019
Mar 14 2019
Feb 14 2019
Feb 3 2019
Jan 14 2019
I'm generating about 2000 flows and I'm seeing a big improvement by enabling tx_abduction too with iflib.
But my understanding is this review is not about tx_abduction but about TX_BATCH_SIZE, and that I had to check if this TX_BATCH_SIZE patch will bring better performances in forwarding mode.
This is why I've made 2 DoS forwarding benches:
- One comparing TX_BATCH_SIZE patch against 2 head configured identically
- One comparing performance impact brings by this TX_BATCH_SIZE patch against the performance brings by tx_abduction
It's the standard "DoS" method: I'm unidirectional sending line-rate of smallest size packet.
Jan 11 2019
Ok, so let's try again with this latest version (I'm calling this one D18532v3):
Jan 8 2019
Jan 5 2019
Here are my classical DoS benches with ipfw (stateless with one rule "allow ip from any to any") results on 3 different setup:
Jan 4 2019
Hi,
I've got igb only NIC, not em, so I can't use my lab to bench this new review version.
Dec 17 2018
Other platform (supermicro 5018A-FTN4, Atom C2758 8-cores at 2.4Ghz with intel 82599ES 10-Gigabit):
x r342020 with D18532(old: diff51924) and tx_abdicate disabled (default): inet4 pps
+ r342020 with D18532(new: diff51986) and tx_abdicate disabled (default): inet4 pps
+--------------------------------------------------------------------------+
| x xx + + * + +|
||______________M____A____________________| |
| |_____________MA_______________| |
+--------------------------------------------------------------------------+
N Min Max Median Avg Stddev
x 5 3179260 3260787 3200772 3208284.5 30758.531
+ 5 3235848 3287655 3260728 3262531.6 23384.011
Difference at 95.0% confidence
54247.1 +/- 39846.4
1.69084% +/- 1.25533%
(Student's t, pooled s = 27321.2)Comparing DoS benches results with the previous Diff version applied and this one didn't show difference.
Dec 13 2018
Here is my DoS benches results on my smallest 2 hardwares.
First serie comparing the benefit of D18532, so with tx_abdicate disabled (default behaviour)
Dec 3 2018
Benches results using Denial-of-Service mode (line-rate at maximum pps) on small hardware with pf+pfysnc (using loopback interface):
PC Engines APU 2 (4 cores AMD GX 1Ghz):
Nov 29 2018
Nov 25 2018
Nov 13 2018
Nov 12 2018
Nov 11 2018
Yes, I confirm a little improvement on small devices (same iflib.tx_abdicate=1 and ip_redirect disabled).
Nov 10 2018
Nov 9 2018
I've forgot to disable IP redirect in my previous bench.
Here are the other results, on 3 different small platforms (-5% on all).
All of these are using common tuning features for routing/firewalling which are:
- net.inet.ip.redirect=0 and net.inet6.ip6.redirect=0, this allow to re-enable fastforwarding path
- dev.igb|ix.X.iflib.tx_abdicate=1, because they are receiving maximum link packet rate
Nov 8 2018
Here is the first result on forwarding performance impact on low-end hardware (AMD GX-412TC, 4Cores and Intel i210AT NIC) :