Looks good for content to me.

Under a heavy UDP packet flood this helps quite a bit.

Looks awesome. My only issue is that I'd strongly prefer that the STAILQ_HEAD / STAILQ_INIT / STAILQ_ENTRY macros be prefixed with CK_ so that readers realize that these lists are used with epochs, and cannot be safely used with the normal STAILQ macros. I tried to point out most of them, but I may have missed a few.

It would be nice to have this patch together with at least one example of a caller and one example of a NIC driver implementing the API.

I'd just update this review.

As mentioned on the other review, maybe remove the print in the caller, since it is now redundant..

Change to using sched_prio() rather than sched_lend_prio, as suggested by @kib

In D15246#321533, @jhb wrote:

Adding kib@.

It's not clear to me why to prefer PI_NET - 1 over, say, PRI_MIN in this case. There's nothing about powerd that is network specific.

I've replaced the arbitrary PI_NET - 1 to PRI_MIN, as suggested by John

Well thats embarrassing. Thanks for the fix.

I think you'll get a lot less pushback if you serialize the multicast stuff in the stack, rather than the driver framework. This will allow you to put warnings / asserts into all the ioctl entry points above the drivers, so as to lock in the "you can't hold a lock while calling into a driver" rule.

So let me try to sum up in my own words what's going on here:

In D12638#263636, @kib wrote:

I patched this into our netflix tree. I confirmed that, even with rx timestamps enabled, the new functionality does not cause a significant performance loss in our case. There seems to be roughly one or two cache misses per mlx5e_poll_rx_cq() to check the p->priv->clbr_done > 2 condition, and roughly 8x as many for the timestamp handling itself. This pushes the cost of mlx5e_build_rx_mbuf() higher, but not horribly so.

I have not tested it yet, but this looks fantastic. Thank you for this!

In D12615#262156, @glebius wrote:

The change doesn't affect sfbufs at all.

What is LGTM?

I think this is intended to piggyback on the recent iflib change which claims a speedup from chaining the packets. However, I'm afraid that I don't understand where this speedup is coming from. The stated reason to allow chaining in ether_input() is to allow drivers to amortize the release/acquire of the rx lock. However, no decent driver even uses an rx lock anymore, certainly not iflib or mlx5. So is there a benefit? If yes, then can you explain where it is coming from?

We currently disable entropy collection (or hoarding, as you aptly describe it) for NET_ETHER. It looks like if the IFF_ NO_ENTROPY flag was present for our 100G nics, this might be slightly cheaper for us, as it will avoid the function call, and the load of the entropy mask, since if_flags will already be hot in cache from the IFF_UP check. As it is, it will be no worse .

In D11525#251667, @wblock wrote:

Added my probably-weak mdoc suggestions, and added brueffer as a reviewer.

In D11525#251340, @wblock wrote:

This is a bit confusing. The new -I and -X options take the same irq argument as the old -x?

It has been years since I've had to think about this, but I remember that the problem that all drivers fight is that you wind up being called in your ioctl routine with potentially some lock held by something that is calling you, so you cannot sleep. Is that still the issue? Can you remind me what lock is held?

Thanks for the feedback. I'll reach out to wblock about the doc changes. I'm quite weak in man page fu, and this is a slightly odd one at that..

• Feedback from jbh regarding wording

I finally tried this, and sadly it does not seem to speed things up at all.

Address kib's feedback

This seems reasonable to me

I really hate the idea of spreading the linux kpi, but I totally understand why you used it. We should probably move a lot of that stuff to a native interface.

I added Scott, as he's been quite involved in the busdma code. I want to make sure this looks OK to him too.

• Add sanity checking to reject bad configs.
• Make original physical domain count available

via sysctl

Thanks again for all the style fixes (we really need a checkpatch, a-la linux, or a c-style like opensolaris -- having hopped between OSes so much, i'm terrible at style(9). And thanks for catching the uninitialized variable.

• style fixes, as well as making sure to initialize the domain.

Thanks for the feedback

• Address Kib's feedback

• we could add another 4 bytes to mbuf pkthdr and it still wouldn't fill a cacheline;

Hi Navdeep,

This seems like a great idea. I don't see any downside.

Yep.. i missed it. So nevermind then. Darn, I was hoping this would be easy.

I'm not sure where ipi_flags is initialized in the main iflib.c routine; it appears that it is used just for v4/v6 determination and those bits are OR'ed in. So having stack garbage in the flags could lead to IPI_TX_IPV4 being set, and the ixgbe driver taking the wrong offload path. Perhaps the first thing to try would be to change those OR's to simple assignments:

TSO implies checksum offload. If you could reverse the order of your test, that would be helpful. Eg, does +txcsum6 -tso6 work?