I just looked for a short time.. more comments later..

Yes, I meant adding a KASSERT or at least a comment.

I like this a lot. Note that I only looked at mxge and a few iflib drivers.

Moved PORTREVISION down a line to fix portlint complaint as pointed out by jhb

In my tests of UDP transmit using N copies of netperf on a 14c/28t Xeon E5-2697 v3 (haswell) using a 40Gbe ixl, I see nearly a 100% performance improvement (eg, double the packet rate) as compared to using mp_ring. In fact, for a single-threaded test the packet rate is 4x what it is for mp_ring with tx-abdicate enabled.

Applied suggested changes prior to committing.

In D19111#476197, @hselasky wrote:

@gallatin: Network drivers can also use worker threads instead of IRQ's to process packets. Especially USB network devices.

In D21801#476448, @jhb wrote:

Actually, we need to update the logic that populates the TLS header to stop writing the 8 byte nonce for GCM for 1.3 in ktls_seq? It probably doesn't hurt for it to stay, but it would be cleaner if we removed it. I've actually thought about reworking what we do anyway. The 8 byte nonce doesn't have to be the sequence number, it just has to be unique per record. OpenSSL picks a random 8 byte value and increments it. We could actually do the something similar and generate the TLS header always in ktls_frame(). We would generate an 8 byte random value that is saved in the TLS session when it is created and just increment it for each record that is framed. There's no requirement that the nonce's be monotonically increasing, etc. just unique per record. This would simplify ktls_seq() and would make the 1.3 vs 1.2 handling a bit clearer.

Re-submit diff with -U999999 for full context.

Updated the port to handle TLS 1.3

Applied suggested changes in what I'm about to commit.

• I've taken linimon's changes from bugzilla
• I've hooked the port to the Makefile in the parent

Updated to address feedback from Hans:

• add comment that record_type must be first in the union
• added a constant for the 1.3 GCM iv len and checked it where we check it for 1.2

I tested this, and it performed well and worked fine.

Would it make sense then to assert that the link is down?

I'm not sure, but I think the only way this can happen is when the link is down. Is that the caee here?

Uncrappy is nice. But if we've got lock contention, it would be nice to know why.

In D21122#472928, @olivier wrote:

In D21636#471988, @kbowling wrote:

This isn't worth much consternation but I do wonder if we can try to limit the proliferation of these sockopts. For instance, if you generalize the existing _LB to take a generic integer arg, would that and cpuset achieve the same effect (and generalize to however people want to use those numbered groups?)

• Fixed line wrap issues pointed out by igor

Address bz's concerns:

• Shorten lines in man page
• Make numa_domain consistently uint8_t
• simplify logic in places
• fix some style errors

In D21122#466175, @olivier wrote:

Flamegraph:

• mpring enabled
• mpring disabled

In D21122#462385, @olivier wrote:

On another hardware (Atom 4cores):

x inet4 forwarding of small packet size, bypass_mpring=0 in packets-per-second
+ inet4 forwarding of small packet size, bypass_mpring=1 in packets-per-second
+--------------------------------------------------------------------------+
|+                              +  ++  +                x           x x x  |
|                                                          |______A__M____||
|            |_______________A_____M________|                              |
+--------------------------------------------------------------------------+
    N           Min           Max        Median           Avg        Stddev
x   4      927858.5        983285        971934     963752.88      24606.11
+   5        749042        872241        859294      839484.8     51106.313
Difference at 95.0% confidence
        -124268 +/- 66405
        -12.8942% +/- 6.76603%
        (Student's t, pooled s = 41856.6)

Bigger impact her: -12%

Oh, and in the documentation, it would be handy to note that BIOSes will sometimes disable this functionality and you may need to enable or disable certain common settings in the BIOS.

I just cherry picked this into my tree to try on some servers. As a *USER*, I have the following comments:

Just wanted to chime in and say that this, in combination with https://reviews.freebsd.org/D20523, allows me to pass an add-in USB XHCI controller to a guest. Before this patch, bhyve would abort when passing through the problematic controller.

Just wanted to chime in that this, in combination with https://reviews.freebsd.org/D20525, allows me to pass through an add-in USB XHCI controller to a guest OS. Before this patch, bhyve would segv.

I tend to think that neither of these is actually a problem, since in both cases, if the "if" is taken, we return or jump. But the fix seems harmless.

In D20163#434567, @jhb wrote:

FWIW, my limited testing of IPsec doesn't use if_ipsec, but instead I used setkey. I think having the rc.d scripts for 'ipsec_enable' autoloading ipsec.ko is reasonable. Maybe if there are ports scripts for IKE daemons (raccoon, strongswan?) those rc.d scripts should also try to load modules they need.

Thank you!

In D20163#434345, @cem wrote:

No objection to moving IPSEC out of GENERIC (but continuing to build it as a module). Check with gnn@, as he originally added it.
I agree that teaching ifconfig about ipsec (option 1) seems the most straightforward way to provide compatibility through that interface.

To make this change less painful for users, it seems like we should enable ifconfig to auto-load ipsec when ifconfig is invoked with ipsecX. ifconfig will currently try to load if_ipsec, and this does not work because the kernel module is ipsec.ko. So it seems like there are 3 obvious workarounds:

• Restored the fixes to the manpage formatting and xref's requested by markj (as noticed by bz)
• Updated the date on the manpage as requested by bz (speculative date for now)
• Changed ifdef / { matching as requested by bz
• Changed net.link.lagg.default_use_numa init as requested by bz (i think)
• Fixed printing of USE_NUMA flag in ifconfig, as noticed by me.