Also, this needs to be MFC'ed to 15 for the release..

To be clear, igc worked fine when configured with multiple queues. I had configured it with a single rx queue in order to debug something else, and it took the somewhat reasonable approach of returning garbage as the hash since it didn't need to hash packets to multiple queues. Thank you for coming up with this... it is much cleaner than my solution, and should ensure that any other non-iflib drivers avoid this behavior as well.

Update patch to address @markj 's feedback, and to properly fix the bug I encountered where doorbells were withheld when using the mp_ring transmit method.

@markj I've marked things as done as I've done them in my local tree. I'll update the patch once I do some more research into the doorbell issue. thank you for the reveiw!

• remove extra space in iflib_if_get_counter as pointed out by @kbowling

In D52369#1197238, @markj wrote:

Instead of making each driver report 0 if it doesn't collect a drop counter from the MAC, and treating these counters as a special case in iflib_if_get_counter(), wouldn't it be simpler and cleaner to make them all drivers return the value from if_get_counter_default()?

reversed logic and made the driver responsible to query the generic counters for odrop/oerr rather than iflib.

Updated drivers that were getting generic counters using if_get_counter_default() as part of their ifdi_get_counter method to avoid getting generic counters for output drops and output errors, as suggested by @markj .

• Call iflib_remove_mbuf() to unmap and remove the mbuf from list of mbufs to be freed when we free an mbuf due to ENOBUFS

In D51772#1185535, @markj wrote:

removing forwarding checks ... LRO already does this

But LRO only checks the forwarding configuration when it actually comes time to process packets. With this patch, if forwarding is enabled, we'll do a bunch of needless sorting.

Address Eric's style(9) nits and remove space in front of parans in sizeof

Update diff to address Gleb's feedback

Thank you for moving to this solution, away from a global atomic.

In D50546#1154747, @markj wrote:

This generally seems fine to me, but I'm not familiar with the original change and wonder why the compat shims are so coupled to the one application.

In D50546#1154136, @ziaee wrote:

Should we put the sysctl in the manual?

Update to use hlen, which already holds the size of the ip6 hdr, as suggested by glebius

Thank you!

I'd very much like for this to be backed out. I feel like users will be chasing the breakage caused by this for years. It broke some tests, iocage, and nstat that I know of in the 5 minutes I've been aware of the commit.

Super helpful review, John. I just opened a new review (https://reviews.freebsd.org/D47583) for the simplest suggested change. Will work on your other suggestions.