User Details
- User Since
- Feb 5 2016, 2:19 PM (270 w, 22 h)
Wed, Mar 24
Mon, Mar 22
Fri, Mar 19
Wed, Mar 17
You've right, those are not the source of the problem.
For history, here is the problem sometimes happen in parallel mode:
You've right, after more tests on my side, the failing in parallel mode is not due to that.
For history, here is the error message that happen sometime:
Tue, Mar 16
Mon, Mar 15
Sun, Mar 14
Impact of D24094 "Fix allocation of queues to CPUs in iflib" on FreeBSD 14 current n257351 on forwarding performance:
- SuperMicro SuperServer 5018A-FTN4 (8 cores Atom C2758 at 2.4GHz)
- Dual port Intel Intel 82599
- 5000 flows of smallest UDP packets
- 2 static routes
- Traffic load at 14.88 Mpps
- ICMP redirect disabled
Mar 8 2021
Mar 5 2021
Mar 4 2021
Feb 26 2021
Feb 17 2021
Feb 10 2021
Feb 8 2021
Feb 3 2021
Feb 2 2021
Feb 1 2021
Jan 26 2021
Jan 25 2021
Jan 4 2021
Dec 28 2020
Intel Xeon E5-2697Av4 (16Cores, 32 threads) with Mellanox ConnectX-4 MCX416A-CCAT (100GBase-SR4):
Dec 27 2020
Small device:
x 7f4e724829e (2020/12/27): inet packets-per-second forwarded + 7f4e724829e (2020/12/27) with D27755: inet packets-per-second forwarded +--------------------------------------------------------------------------+ |+ + + + + x xx x x| | |__________M___A______________| | | |_________A__________| | +--------------------------------------------------------------------------+ N Min Max Median Avg Stddev x 5 897815 910126.5 903309 904485.3 4890.7439 + 5 886306 894945 889997 889957.2 3447.4139 Difference at 95.0% confidence -14528.1 +/- 6170.78 -1.60623% +/- 0.674941% (Student's t, pooled s = 4231.08)
Dec 19 2020
Benches on multiples hardwares (click their links for full data and flame-graphs):
So, here are benches result against the diff 80909 (not the latest one):
Dec 18 2020
Dec 17 2020
So due to a bug in netmap pkt-gen, the previous benches were generating only 200 UDP flows (in place of 2000 for the smallest device and 5000 for the others).
New benches were done, and the new results with lots more flows seems concluding the same as previous.
Dec 15 2020
I need to redo all my benches!!!
Dec 14 2020
New benches on Diff 80677 on multiples hardwares (click their links for full data and flame-graphs):
Dec 13 2020
The previous unexpected improvement should became from Chelsio drivers, because on small hardware with Intel NIC, there is no such difference:Full report here.
Just find unexpected improvement (not related to this review) on latest -head: On my 10Gb/s Chelsio server (8-cores) is now reaching the line-rate of 14.8Mpps.
So I will start another bench on biggest hardware (40 and 100Gb/s).
Meanwhile results and flamegraphs here (but seems not useful now):
https://github.com/ocochard/netbenches/blob/master/Xeon_E5-2650_8Cores-Chelsio_T540-CR/forwarding-pf-ipfw/results/fbsd13-r368606.D27401v3/README.md
Dec 5 2020
Result on a small and medium hardware (huge difference between those hardware).
pmc data no more available on the small AMD cpu (regression because it was).
Dec 3 2020
Yes, it builds and boot, but seems quiet verbose, the dmesg is full of messages like those:
[rt_algo] inet6.0 (radix6_lockless) handle_rtable_change_cb: Scheduling rebuilt [rt_algo] inet6.0 fib_check_best_algo: candidate_algos: 2, curr: radix6_lockless(255) result: NULL(255) [rt_algo] inet6.0 (radix6_lockless) handle_rtable_change_cb: Scheduling rebuilt [rt_algo] inet6.0 (radix6_lockless) fib_get_nhop_idx: REF nhop 1 0xfffff8000bb01e00 [rt_algo] inet6.0 (radix6_lockless) fib_get_nhop_idx: REF nhop 3 0xfffff8000bb01c00 [rt_algo] inet6.0 (radix6_lockless) fib_get_nhop_idx: REF nhop 2 0xfffff8000bb01d00 [rt_algo] inet6.0 (radix6_lockless) sync_algo: initial dump completed. [rt_algo] inet6.0 (radix6_lockless) try_setup_instance: DUMP completed successfully. [rt_algo] inet.0 (radix4_lockless) handle_rtable_change_cb: Scheduling rebuilt [rt_algo] inet6.0 (radix6_lockless) replace_rtables_family: [vnet 0xfffff80003088f00] replace with f:0xffffffff80de82b0 arg:0xfffff800039739 00 [rt_algo] inet6.0 (radix6_lockless) replace_rtables_family: OLD FFI: 0xfffff8000b3cde00 NEW FFI: 0xfffff8000b3cd380 [rt_algo] inet6.0 (radix6_lockless) replace_rtables_family: update 0xfffff8000b3cde00 -> 0xfffff8000b3cd380 [rt_algo] inet6.0 setup_instance: try 0: fib algo result: 0 [rt_algo] inet6.0 (radix6_lockless) rebuild_callout: switched to new instance [rt_algo] inet6.0 (radix6_lockless) schedule_destroy_instance: DETACH [rt_algo] inet6.0 (radix6_lockless) schedule_destroy_instance: destroying old instance [rt_algo] inet.0 fib_check_best_algo: candidate_algos: 2, curr: radix4_lockless(255) result: NULL(255) [rt_algo] inet6.0 (radix6_lockless) destroy_instance: destroy fd 0xfffff80003985d00 [rt_algo] inet.0 (radix4_lockless) fib_get_nhop_idx: REF nhop 1 0xfffff8000bb13e00 [rt_algo] inet.0 (radix4_lockless) sync_algo: initial dump completed. [rt_algo] inet.0 (radix4_lockless) try_setup_instance: DUMP completed successfully. [rt_algo] inet.0 (radix4_lockless) replace_rtables_family: [vnet 0xfffff80003088f00] replace with f:0xffffffff80d98f10 arg:0xfffff80003973c0 0 [rt_algo] inet.0 (radix4_lockless) replace_rtables_family: OLD FFI: 0xfffff8000b3cde40 NEW FFI: 0xfffff80003778bc0 [rt_algo] inet.0 (radix4_lockless) replace_rtables_family: update 0xfffff8000b3cde40 -> 0xfffff80003778bc0 [rt_algo] inet.0 setup_instance: try 0: fib algo result: 0 [rt_algo] inet.0 (radix4_lockless) rebuild_callout: switched to new instance [rt_algo] inet.0 (radix4_lockless) schedule_destroy_instance: DETACH [rt_algo] inet.0 (radix4_lockless) schedule_destroy_instance: destroying old instance [rt_algo] inet.0 (radix4_lockless) destroy_instance: destroy fd 0xfffff80003985b00 [rt_algo] inet.0 (radix4_lockless) handle_rtable_change_cb: Scheduling rebuilt [rt_algo] inet6.0 (radix6_lockless) handle_rtable_change_cb: Scheduling rebuilt [rt_algo] inet.0 fib_check_best_algo: candidate_algos: 2, curr: radix4_lockless(255) result: NULL(255) [rt_algo] inet.0 (radix4_lockless) fib_get_nhop_idx: REF nhop 1 0xfffff8000bb13e00 [rt_algo] inet.0 (radix4_lockless) fib_get_nhop_idx: REF nhop 2 0xfffff8000bc48e00 [rt_algo] inet.0 (radix4_lockless) fib_get_nhop_idx: REF nhop 3 0xfffff8000bc48d00 [rt_algo] inet.0 (radix4_lockless) sync_algo: initial dump completed. [rt_algo] inet.0 (radix4_lockless) try_setup_instance: DUMP completed successfully. [rt_algo] inet.0 (radix4_lockless) replace_rtables_family: [vnet 0xfffff80003088f00] replace with f:0xffffffff80d98f10 arg:0xfffff8000bc7cd8 0 [rt_algo] inet.0 (radix4_lockless) replace_rtables_family: OLD FFI: 0xfffff80003778bc0 NEW FFI: 0xfffff8000b360a80 [rt_algo] inet.0 (radix4_lockless) replace_rtables_family: update 0xfffff80003778bc0 -> 0xfffff8000b360a80 [rt_algo] inet.0 setup_instance: try 0: fib algo result: 0 [rt_algo] inet.0 (radix4_lockless) rebuild_callout: switched to new instance [rt_algo] inet.0 (radix4_lockless) schedule_destroy_instance: DETACH [rt_algo] inet.0 (radix4_lockless) schedule_destroy_instance: destroying old instance [rt_algo] inet6.0 fib_check_best_algo: candidate_algos: 2, curr: radix6_lockless(255) result: NULL(255) [rt_algo] inet.0 (radix4_lockless) destroy_instance: destroy fd 0xfffff80003985d00 [rt_algo] inet6.0 (radix6_lockless) fib_get_nhop_idx: REF nhop 1 0xfffff8000bb01e00 [rt_algo] inet.0 (radix4_lockless) destroy_instance: FREE nhop 1 0xfffff8000bb13e00 [rt_algo] inet6.0 (radix6_lockless) fib_get_nhop_idx: REF nhop 5 0xfffff8000bc48b00 [rt_algo] inet6.0 (radix6_lockless) fib_get_nhop_idx: REF nhop 4 0xfffff8000bc48c00 [rt_algo] inet6.0 (radix6_lockless) fib_get_nhop_idx: REF nhop 3 0xfffff8000bb01c00 [rt_algo] inet6.0 (radix6_lockless) fib_get_nhop_idx: REF nhop 2 0xfffff8000bb01d00 [rt_algo] inet6.0 (radix6_lockless) sync_algo: initial dump completed. [rt_algo] inet6.0 (radix6_lockless) try_setup_instance: DUMP completed successfully. [rt_algo] inet6.0 (radix6_lockless) replace_rtables_family: [vnet 0xfffff80003088f00] replace with f:0xffffffff80de82b0 arg:0xfffff80003973a 80 [rt_algo] inet6.0 (radix6_lockless) replace_rtables_family: OLD FFI: 0xfffff8000b3cd380 NEW FFI: 0xfffff8000b3609c0 [rt_algo] inet6.0 (radix6_lockless) replace_rtables_family: update 0xfffff8000b3cd380 -> 0xfffff8000b3609c0 [rt_algo] inet6.0 setup_instance: try 0: fib algo result: 0 [rt_algo] inet6.0 (radix6_lockless) rebuild_callout: switched to new instance [rt_algo] inet6.0 (radix6_lockless) schedule_destroy_instance: DETACH [rt_algo] inet6.0 (radix6_lockless) schedule_destroy_instance: destroying old instance
The error is different:
ld: error: undefined symbol: nhgrp_get_count >>> referenced by route_algo.c:962 (/usr/local/BSDRP/TESTING/FreeBSD/src/sys/net/route/route_algo.c:962) >>> route_algo.o:(fib_get_rtable_info) >>> referenced by route_algo.c:962 (/usr/local/BSDRP/TESTING/FreeBSD/src/sys/net/route/route_algo.c:962) >>> route_algo.o:(fib_check_best_algo) *** [kernel.full] Error code 1
Dec 1 2020
Kernel build failed with this error (on r368264):
--- kernel.full --- ld: error: undefined symbol: fib_destroy_rib >>> referenced by route.c:158 (/usr/local/BSDRP/TESTING/FreeBSD/src/sys/net/route.c:158) >>> route.o:(rt_table_destroy)
The latest historical data I have on this hardware are those:
x r364730: inet packets-per-second forwarded + r367714: inet packets-per-second forwarded * r368200: inet packets-per-second forwarded +--------------------------------------------------------------------------+ |x xx x x *** ** + + +++ | ||__MA___| | | |______________A______M________|| | |_A__| | +--------------------------------------------------------------------------+ N Min Max Median Avg Stddev x 5 9697454 9755700 9716136 9724960.1 24760.638 + 5 9899883 10149413 10132167 10088669 105990.12 Difference at 95.0% confidence 363709 +/- 112248 3.73995% +/- 1.1565% (Student's t, pooled s = 76964.3) * 5 9803502.5 9843579.5 9822232 9822433.6 15632.275 Difference at 95.0% confidence 97473.5 +/- 30198.2 1.0023% +/- 0.312751% (Student's t, pooled s = 20705.8)
Excellent idea to enable it!
Nov 30 2020
Nov 25 2020
Nov 24 2020
Nov 14 2020
Nov 11 2020
Nov 10 2020
Nov 9 2020
Nov 8 2020
Nov 6 2020
Oct 30 2020
Oct 25 2020
Oct 5 2020
What the current status of this review ? Wasn't it supposed to be committed at the end of August ?
Oct 2 2020
Oct 1 2020
Sep 28 2020
Sep 25 2020
switching a variable to const.
Sep 24 2020
Sep 23 2020
Tested with net/frr7 compiled with MULTIPATH option, and it is working great:
Sep 22 2020
Sep 21 2020
Tested on my ECMP lab (https://bsdrp.net/documentation/examples/ecmp#testing_load_balancing) and the flow-id load-balancing seems working great.
Would you like test using bird and FRR too ?
Sep 18 2020
Look like the review contains a patch to file usr.bin/netstat/nhgrp.c that doesn't exist on -head
Sep 10 2020
Sep 7 2020
My jail/vnet burn test is still generating a panic:
Sep 5 2020
Sep 4 2020
@olivier Thanks for the analysis. Could you share some pmc profiles from unhalted core cycles?
Sep 1 2020
Comparing wireguard userland vs kernel module on a small device: +300% (from 117Mb/s to 483Mb/s)
- PC Engines APU2C4 (quad core AMD GX-412T Processor 1 GHz)
- 3 Intel i210AT Gigabit Ethernet ports
- FreeBSD 13-head r365033
- Wireguard kernel: D26137
- Wireguard userland: 1.0.20200827
- 2000 flows of UDP packets
- 500Bytes UDP load => packet size: 528B => Ethernet frame size:542B
Aug 28 2020
Aug 25 2020
Aug 24 2020
Aug 20 2020
Aug 18 2020
Updating the patch to latest -head.