Thanks for the review, kib@.

Serial all VXLAN start/stops (even across different VXLAN interfaces).

Add a global sx and hold it around the VXLAN start/stop events.

Remove debug leftovers.

And the refcount underflow check is for the case where a VXLAN is created/started before the NIC driver is kldloaded. It will miss the create/start event but will see the destroy/stop and get confused. The real fix would be to have a count of VXLAN interfaces somewhere and have the driver seed its initial refcount with this value. But if_vxlan itself is a loadable module so this counter would have to be in the kernel proper, or else there would have to be a query function in if_vxlan that is called indirectly (like the vlan functions). I thought all this was too complicated for an unusual case. If users make sure all NIC drivers are loaded before creating VXLANs then there is no problem with missed events.

In D25873#579567, @kib wrote:

This patch naturally splits into many chunks, for instance RFC7348 port, inner checksums infra, events, driver bits. Do you plan to do more fine-grained commits that one big ci ?

incorporate some feedback from kib@

Tx counters that track hw assistance should consider all inner traffic
irrespective of the IP version of the outer encapsulation.

Handle rxcsum and rxcsum6 correctly.

Disable TSO when tx checksumming is disabled.
Add counters for checksums and TSO.

In D25873#573657, @kib wrote:

In D25873#573638, @np wrote:

The goal is to improve vxlan's performance. mbuf tags would add at least an alloc/free and extra access(es) to the tags (which are on a linked list) on the hot path. Why even consider any other location for the flags if there _are_ bits available in the proper place right now. We'll be out of them after this change, we aren't out already.

Where VNI is currently stored when packet is pushed up or down to driver, for stateless offload ?

In D25873#573628, @lutz_donnerhacke.de wrote:

In D25873#573627, @np wrote:

In D25873#573456, @lutz_donnerhacke.de wrote:

In D25873#573447, @hselasky wrote:

Aren't we soon running out of flag bits in certain integer values?

Given that the inner flags mirror the outer ones, why not use a different variable inner_flags with the same set of flags?

That would work, but it would increase the size of struct mbuf.

So let's move the vxnet specific flags into a mbuf_tag.

In D25873#573456, @lutz_donnerhacke.de wrote:

In D25873#573447, @hselasky wrote:

Aren't we soon running out of flag bits in certain integer values?

Given that the inner flags mirror the outer ones, why not use a different variable inner_flags with the same set of flags?

In D25873#573447, @hselasky wrote:

Aren't we soon running out of flag bits in certain integer values?

Remove comment.

In D25873#573049, @lutz_donnerhacke.de wrote:

May you please consider splitting the patch into functional groups?

• VXLAN HW capabilities
• Spelling errors
• Logic cleanup "xxx == 0" vs "!xxx"
• style cleanup

Incorporate feedback. atomic_load_64 is now used everywhere to read state.

Remove unrelated hunk from sge.c

This change works once the compilation issues are fixed.

Seems like t4_listen is missing an include.

I will try this and let you know how it went by June 10.

Looks good (other than the yoda style in one conditional).

This was committed in r360232

Incorporate feedback.