Mea culpa, it is due to -t

Mea culpa.

In D18339#426772, @ae wrote:

Ok. It is more flexible, but produces additional options. I think ipfw(8) is already very complex.
What if we will make "missing"+"flush" behavior as default.
It seems if user wants to create table, it is expected that later this table will be filled. So, if we are creating some table, and it is already exist, we will check that the table has the same configuration and then flush it.
If configuration is different, then we return error. What you think?

In D18339#426430, @ae wrote:

I think you can add to the beginning of your ipfw rules script something like this:

ipfw -q flush
ipfw -q table all destroy

And then create needed tables and fill them.

In D19094#410062, @mav wrote:

While I see the problem you are fixing, the fix looks ugly to me, that is why I would look for something nicer. I agree that according to logic of remove_reference() dropping last reference for header in ghost state is a failure, but how can remove_reference() be called before the arc_access() just on following line? I would guess from description telling about the case of prefetch read it should happen no sooner then we actually initiate the I/O, which is done much later then those two lines. So while I agree it is somewhat odd to have buffer for header in ghost state, is that a criminal.

And without this patch

In D18532#402275, @olivier wrote:

I'm generating about 2000 flows and I'm seeing a big improvement by enabling tx_abduction too with iflib.

I'm testing TWO scenarios:

• First is «LAN to WAN» and flows are 10.1.0.2:2000-10.1.0.5:2004 → 10.10.10.2:2000-10.10.10.128:2006 — it should be 4×5×127×7 = 17780 flows.
• Second is «WAN to DMZ» and flows are 10.10.10.2:2000-10.10.10.254 → 10.1.0.2:2000 — it should be only 253 flows.

Other trick is, I tests not only «raw» routing, but throw in IPsec (and gre and gif and ipfw with and without NAT, so my configuration space contains 87 configurations, but here I'm speaking only about simplest cases), which always works from 10.1.0.1/24 to 10.10.10.0/24 between DUT and traffic mirror (which is much more powerful). So, first and second cases becomes even more asymmetrical.
First case becomes «receive, encrypt, send through tunnel» and second cases becomes «receive from tunnel, decrypt, send in clear», which should affect RSS and flow distribution, as far as I understand.

In D18532#402157, @olivier wrote:

It's the standard "DoS" method: I'm unidirectional sending line-rate of smallest size packet.

Question is — how many source/destination IPs and ports are used? It is what determine usability of tx_abduction for me, is it some-to-many («from LAN to WAN») or many-to-one («WAN to DMZ box in LAN».

In D18532#401754, @olivier wrote:

Ok, so let's try again with this latest version (I'm calling this one D18532v3):

What is your benchmark? I'm using your equilibrium script and see very different effect of tx_abdicate depending on «direction» of test: when I emulate «small network sends to big Internet» result is different to «Big Internet sends to small network». Unfortunately, there is no easy way to emulate real traffic, as equilibrium is strictly unidirectional.

These drops of performance with tx_abdicate which is almost 2 times looks like RSS failure?..

With this patch and *with* tx_abdicate results are mixed.

I could say, that with this patch and *without* tx_abdicate results are:

1. Without IPsec is the same both in bandwidth (kb/s) and throughput (pps) is not worse than without it. It is hard to say, that it is better as it is near ability of my test rig to generate traffic anyway.
2. With IPsec it is slightly better both for bandwidth and throughput in both directions.

It helps for ix, can not test for ixl, as I don't have one/

Address comments by @mizhka_gmail.com

Who could/should commit this? I don't have src commit bit.

Yep, it helps, at least at em

It changes nothing visible and doesn't help.

This helps. It doen't show BSD partitions, but shows all four MBR slices without crash.

It fix ix0 for me.

Second version works, both without VLAN_HWTAGGING and with it, physical network and VLANs.

This patch (first version) breaks my I210 completely. With it physical interface without any VLANs could not mount NFS share, for example. Something simple, like "ping" works, but all complex protocols are broken.
I didn't check VLANs, because host without NFS shares and other such protocols renders itself useless.

This patch helps with all my convoluted rules :-)

Fix all issues pointed out by @yuripv_yuripv.net

Looks good.
Thank you!

This change helps me on real hardware I've had this problem.
Thank you!

Address review notes by a @ae : better new-style printing and more context to diff.
Also, update to r334094

Update to be applied to r334006 or later.

It is only solution to live-lock problem I encounter on my server when there are massive-parallel fast download.

Update diff to latest CURRENT version