In D21122#462385, @olivier wrote:

On another hardware (Atom 4cores):

x inet4 forwarding of small packet size, bypass_mpring=0 in packets-per-second
+ inet4 forwarding of small packet size, bypass_mpring=1 in packets-per-second
+--------------------------------------------------------------------------+
|+                              +  ++  +                x           x x x  |
|                                                          |______A__M____||
|            |_______________A_____M________|                              |
+--------------------------------------------------------------------------+
    N           Min           Max        Median           Avg        Stddev
x   4      927858.5        983285        971934     963752.88      24606.11
+   5        749042        872241        859294      839484.8     51106.313
Difference at 95.0% confidence
        -124268 +/- 66405
        -12.8942% +/- 6.76603%
        (Student's t, pooled s = 41856.6)

Bigger impact her: -12%

Oh, and in the documentation, it would be handy to note that BIOSes will sometimes disable this functionality and you may need to enable or disable certain common settings in the BIOS.

I just cherry picked this into my tree to try on some servers. As a *USER*, I have the following comments:

Just wanted to chime in and say that this, in combination with https://reviews.freebsd.org/D20523, allows me to pass an add-in USB XHCI controller to a guest. Before this patch, bhyve would abort when passing through the problematic controller.

Just wanted to chime in that this, in combination with https://reviews.freebsd.org/D20525, allows me to pass through an add-in USB XHCI controller to a guest OS. Before this patch, bhyve would segv.

I tend to think that neither of these is actually a problem, since in both cases, if the "if" is taken, we return or jump. But the fix seems harmless.

In D20163#434567, @jhb wrote:

FWIW, my limited testing of IPsec doesn't use if_ipsec, but instead I used setkey. I think having the rc.d scripts for 'ipsec_enable' autoloading ipsec.ko is reasonable. Maybe if there are ports scripts for IKE daemons (raccoon, strongswan?) those rc.d scripts should also try to load modules they need.

Thank you!

In D20163#434345, @cem wrote:

No objection to moving IPSEC out of GENERIC (but continuing to build it as a module). Check with gnn@, as he originally added it.
I agree that teaching ifconfig about ipsec (option 1) seems the most straightforward way to provide compatibility through that interface.

To make this change less painful for users, it seems like we should enable ifconfig to auto-load ipsec when ifconfig is invoked with ipsecX. ifconfig will currently try to load if_ipsec, and this does not work because the kernel module is ipsec.ko. So it seems like there are 3 obvious workarounds:

• Restored the fixes to the manpage formatting and xref's requested by markj (as noticed by bz)
• Updated the date on the manpage as requested by bz (speculative date for now)
• Changed ifdef / { matching as requested by bz
• Changed net.link.lagg.default_use_numa init as requested by bz (i think)
• Fixed printing of USE_NUMA flag in ifconfig, as noticed by me.

In D20117#433106, @adrian wrote:

I see you've done a bit of net80211 checking there; I think I'm going to use this change as motivation to speed up my desire to make rcvif manipulation a bit less insane and error prone.

In D20117#433037, @ae wrote:

I'm sorry, I completely missed this change in the past. But it looks like it can break ipfw firewall rules, since rcvif is now union with snd_tag. And this means, rcvif can be initialized for packets that were not actually received on specified interface. ipfw uses rcvif in rules to check that a packet was received on specified interface, and this check was correct even for outgoing packets. Now it looks like such checks can be incorrect.

In D20062#432787, @alc wrote:

I think that there is a better approach than what you are attempting here. Once you have the patch that you describe as "My patch to alter the behavior of SO_REUSEPORT_LB listen sockets ...", the thread executing sendfile(2) will be running on the same domain as the socket. Recall Kostik's first comment, "Vm objects can have domain policies assigned, and object policy trumps a thread policy, if present." In other words, if you don't define an object policy, the page allocations will be governed by the thread's policy. A thread policy that says allocate from the local domain should achieve your desired outcome.

In D20062#432560, @kib wrote:

I still do not quite understand the benefits of this change.

As alc pointed out (and as I've confirmed), the vm_object will linger with the domain policy set, causing an increase in cross domain traffic for our workload. I've restored the feature from the older patch where we override the current domain policy. This reduces cross domain traffic to previous levels.

I'm going to have to re-test with this

Address comments by markj and kib

• add a comment explaining what is happening
  • only replace domain policy when it is NULL
  • re-check policy after acquiring object lock

Fix off-by-one error pointed out by hselasky

In D20062#431501, @kib wrote:

IMO it is too special-purpose, and potentially affecting other loads.
Vm objects can have domain policies assigned, and object policy trumps a thread policy, if present. I would prefer that we added infrastructure to assign policy to the file' vm_object, e.g. using some new ioctl on file descriptor, and userspace did the job of explicitly stating the desired policy.

I've updated the patch to remove my hand-plumbed path down to vm_page_alloc_domain_after(), and changed to setting the domain policy on the object.

In D20062#431501, @kib wrote:

IMO it is too special-purpose, and potentially affecting other loads.
Vm objects can have domain policies assigned, and object policy trumps a thread policy, if present. I would prefer that we added infrastructure to assign policy to the file' vm_object, e.g. using some new ioctl on file descriptor, and userspace did the job of explicitly stating the desired policy.

Fix style issues pointed out by markj in the ifconfig man page change, and add cross-references to the numa(4) man page, as he suggests.

shurd: I have been waiting for you to test with bnxt & jumbo frames, but I think this is approaching a reasonable timeout (almost 2 weeks since the last feedback). I plan to commit in the next day or so unless I hear from you.

Removed redundant parens around bus_get_domain() call as pointed out by markj

Address kib's feedback

• remove unneeded _domainset.h include
• re-write if_alloc_dev() to be more concise

• Added if_alloc_dev() and changed example mlx5en and cxgbe code to use them, as suggested by kib.
• Added links to if_alloc_domain and if_alloc_dev for man pages
• Fixed style issues pointed out by kib

In D19930#428329, @kib wrote:

I wonder if a KPI of the format

struct ifnet *if_alloc_dev(u_char type, dev_t device);

would be more useful. It would hide all NUMA/non-NUMA/bus_get_domain() failed details that driver authors will copy/paste ad infinitum.
I do not propose to not add if_alloc_domain(), just think that if_alloc_dev() should be the primari KPI. Also it might allow to get more device_t context in if_alloc().

Address review feedback from marius & shurd

YES! Death to M_DEVBUF!

Fixed comments, as pointed out by ejg

I think this is a mostly theoretical issue, as you'll have had a chain which was impossible to send (eg, far too large, etc) even after defrag.

• Use new pfil_mem2mbuf() to avoid unaligned access of mbuf pointer
• Remove unneeded breaks

Much cleaner, as usual, than the hack I gave you to start with.

Guard against null pfil at device detach by deregistering pfil hook later. While here, check for a null pfil hook and cache it in a local to make the code easier to read.