Freeing completed transmit mbufs can be time consuming (due to them
being cold in cache, and due to ext free routines taking locks),
especially when we batch tx completions. If we do this when holding
the tx ring mutex, this can cause lock contention on the tx ring mutex
when using iflib_simple_transmit.
To resolve this, this patch opportunistically copies completed mbuf
pointers into a new array (ifsd_m_defer) so they can be freed after
dropping the transmit mutex. The ifsd_m_defer array is
opportunistically used, and may be NULL. If its NULL, then we free
mbufs in the old way. The ifsd_m_defer array is atomically nulled
when a thread is using it, and atomically restored when the freeing
thread is done with it. The use of atomics here avoids
acquire/release of the tx lock to restore the array after freeing
mbufs.
Since we're no longer always freeing mbufs inline, peeking into them to see if a
transmit used TSO or not will cause a useless cache miss, as nothing
else in the mbuf is likely to be accessed soon. To avoid that cache
miss, we encode a TSO or not TSO flag in the lower bits of the mbuf
pointer stored in the ifsd_m array. Note that the IFLIB_NO_TSO flag
exists primarily for sanity/debugging.
iflib_completed_tx_reclaim() was refactored to break out
iflib_txq_can_reclaim() and _iflib_completed_tx_reclaim()
so the that the tx routine can call iflib_tx_credits_update()
just once, rather than twice.
Note that deferred mbuf freeing is not enabled by default, and can be
enabled using the dev.$DEV.$UNIT.iflib.tx_defer_mfree sysctl.