Looks good to me.

I'm not glebius@, but the patch looks correct.

Looks good to me. Also I want to note, that all

do 
 read();
} while()

loops are affected to the problem described in rS303374. It would be nice to fix this problem too. :)

In D10154#211730, @olivier wrote:

In D10154#211425, @ae wrote:

I think some performance increasing is possible with static rules.

What do you mean by "static rules" ?

In D10154#211415, @olivier wrote:

I didn't see any difference (I'm building world&kernel using WITH_META_MODE, I hope it didn't impact build):

Maybe the name without _all suffix will be better choice?
Also, since there are no protection from concurrent access to queues, I think this should be noted in the comment too. The caller must have exclusive access to both queues, otherwise it is possible to get the wrong result.

In D10154#209953, @eri wrote:

Just curious, do you have any comparison/profiling data if this improves anything?

In D10154#209915, @adrian wrote:

So this just changes the locking model to use a per-vnet lock rather than a per-chain lock for stuff?

Modify some comments and error messages. Fix checksum modification
for forwarded traffic.

This is something that I mean when said about setting RT_LLE_CACHE flags in the TCP/UDP code. This even looks better.

Committed in r315305.

Julian, if you have no objection I'll commit it tomorrow.

I think it would be better move all local variables to the top of function according to style(9) and reuse some already existing variable for this.

In D9929#205426, @julian wrote:

we are going to run out of available tricks for this in ipfw at some stage.. Can we re-use () like used in table? or make the : part of the keep-state.. so that "keep_state: state1" or keep_state(state1) vs "keep_state :state1"

It seems that the 'R' format character of sockaddr_snprintf() function has become useless. In the rest looks good for me.

In D9721#204562, @adrian wrote:

Well, it's allocating an mbuf, adding mtags, then freeing it if it fails to queue. A lot of our network stack does this because of well, history reasons.

Ok, so on the output path once the netisr path is called, how much of that work is done without a global lock (eg crypto?) I wonder how much of this change in throughput is because the netisr thread != the NIC input CPU path, and RX is saturating (more) cores. Versus, say, some parallel processing.

Because my guess is that the RX -> IP stuff -> IP TX stuff -> ipsec encrypt -> NIC if_transmit() path is happening in one complete thread, to completion each time, so there's backpressure (the NIC for example will drop frames early if the CPU is saturated for example, and/or send pause frames, etc.)

I think we have three options:

• do not change anything and require kern.kstack_pages=4 for working IPsec without panics;
• apply this patch to slowly work without panics when we have kern.kstack_pages below 4;
• rework the patch to use different solution.

In D9721#204352, @adrian wrote:

Interesting. Ok. Well the existing ipsec code for transmit just runs to completion to ip output right?

In D9721#204350, @eri wrote:

Actually that is not true it will run in parallell in the netisr thread.

In D9721#204330, @adrian wrote:

oh hm, one ipsec flow? so ok. So, can you also dump out the NIC statistics so we can see if the NIC is seeing all the traffic on one RX ring, or whether it's being load balanced between multiple RX rings.

In D9721#204264, @olivier wrote:

• 5 416795 440933.5 423271 426451.8 9207.3367

Difference at 95.0% confidence

-122302 +/- 9631.37
-22.2872% +/- 1.74536%
(Student's t, pooled s = 6603.87)

In D9721#204240, @olivier wrote:

Difference at 95.0% confidence

2906.3 +/- 1989.79
0.529619% +/- 0.363866%
(Student's t, pooled s = 1364.33)

Do you want flamegraph too?

In D9721#204240, @olivier wrote:

Difference at 95.0% confidence

2906.3 +/- 1989.79
0.529619% +/- 0.363866%
(Student's t, pooled s = 1364.33)

Do you want flamegraph too?

So, if there is no objection, I'll commit this patch.