Before commit 81dc00331d1 "Make msgbuf_peekbytes() not return leading zeroes.", the dmesg just after the reboot is this output:

In D33854#765599, @tuexen wrote:

I could add support for TOS and TTL (and its cousins for IPv6) to packetdrill. Then you could use a simple packetdrill scripts. I think this was added to the upstream version after I forked it. So it is a matter of picking the right lines of code...

In D33854#765560, @melifaro wrote:

It's great we'll get DSCP tests!
Nit: maybe it's worth trying to have it encoded with python-scapy? It has both sending and receiving support for all kind of packets.
Happy to chat more on that.

Shouldn't the root cause be fixed (geom be revert back to non-verbose mode by default) in place of fixing the regression test ?

Does it mean that from an updated FreeBSD main-n249987-b158d4d7a12 (commit at "rescue: fix after with to a ldscript of libncursesw.a"), I should be able to build those few weeks old nanobsd ?
Or should I patch those old weeks sources with this patch?

Thanks, bsdlabel/disklabel/fdisk are correctly installed now

Thanks!
No more core dump once your patch applied.

Tested to display trace with gdb:

% gdb --args /usr/bin/sockstat -P tcp -p 8080
(...)
Reading symbols from /usr/bin/sockstat...
Reading symbols from /usr/lib/debug//usr/bin/sockstat.debug...
(gdb) run
Starting program: /usr/bin/sockstat -P tcp -p 8080
Assertion failed: (chan != NULL), function cap_xfer_nvlist, file /usr/src/lib/libcasper/libcasper/libcasper.c, line 304.

sockstat is broken since this commit, preventing to run the tcptestsuite regression that is using it:

Thanks! This patch fixes the panic I have met with BBR each time my system was initiating a TCP session:

Once this patch applied, how to check it is dxr_lpm4 that is used ?

Thanks!
it fixes the regression:

In case of system builded without epair, previously those tests were skipped because the line 'ATF_REQUIRE_KERNEL_MODULE("if_epair");'.
But now it seems they are failing in place of being skipped if epair not builded with message like:

/tests/sys/net/routing/rtsock_config.h:134: Expected errno 17, got 2, in true

This fixed my following panic:

You've right, those are not the source of the problem.
For history, here is the problem sometimes happen in parallel mode:

You've right, after more tests on my side, the failing in parallel mode is not due to that.
For history, here is the error message that happen sometime:

Impact of D24094 "Fix allocation of queues to CPUs in iflib" on FreeBSD 14 current n257351 on forwarding performance:

• SuperMicro SuperServer 5018A-FTN4 (8 cores Atom C2758 at 2.4GHz)
• Dual port Intel Intel 82599
• 5000 flows of smallest UDP packets
• 2 static routes
• Traffic load at 14.88 Mpps
• ICMP redirect disabled

In D28581#641508, @rene wrote:

In D28581#641503, @lwhsu wrote:

portmgr : this looks good and trivial, Can we have this approved soon?

I don't expect anyone to object, feel free to remind me later this week.

Intel Xeon E5-2697Av4 (16Cores, 32 threads) with Mellanox ConnectX-4 MCX416A-CCAT (100GBase-SR4):

Small device:

x 7f4e724829e (2020/12/27): inet packets-per-second forwarded
+ 7f4e724829e (2020/12/27) with D27755: inet packets-per-second forwarded
+--------------------------------------------------------------------------+
|+  +       +   +          +        x               xx               x    x|
|                                         |__________M___A______________|  |
| |_________A__________|                                                   |
+--------------------------------------------------------------------------+
    N           Min           Max        Median           Avg        Stddev
x   5        897815      910126.5        903309      904485.3     4890.7439
+   5        886306        894945        889997      889957.2     3447.4139
Difference at 95.0% confidence
        -14528.1 +/- 6170.78
        -1.60623% +/- 0.674941%
        (Student's t, pooled s = 4231.08)

Benches on multiples hardwares (click their links for full data and flame-graphs):

So, here are benches result against the diff 80909 (not the latest one):

So due to a bug in netmap pkt-gen, the previous benches were generating only 200 UDP flows (in place of 2000 for the smallest device and 5000 for the others).
New benches were done, and the new results with lots more flows seems concluding the same as previous.

• tiny AMD 1Ghz 4 cores with Intel Gb
• medium Xeon 8-cores with Chelsio 10Gb

I need to redo all my benches!!!

New benches on Diff 80677 on multiples hardwares (click their links for full data and flame-graphs):

The previous unexpected improvement should became from Chelsio drivers, because on small hardware with Intel NIC, there is no such difference:Full report here.

Just find unexpected improvement (not related to this review) on latest -head: On my 10Gb/s Chelsio server (8-cores) is now reaching the line-rate of 14.8Mpps.
So I will start another bench on biggest hardware (40 and 100Gb/s).
Meanwhile results and flamegraphs here (but seems not useful now):
https://github.com/ocochard/netbenches/blob/master/Xeon_E5-2650_8Cores-Chelsio_T540-CR/forwarding-pf-ipfw/results/fbsd13-r368606.D27401v3/README.md