With direct dispatch, the receive task transmits each forwarded packet
from within if_input(), and on a lightly used ring every packet rings
the doorbell. 4a0ec469934a stopped iflib from ringing when nothing was
pending, which took three register writes per packet down to one; this
rings the remaining one once per receive batch. On an Intel I226 (igc)
a write costs about 400 cycles, a fifth of the cost of forwarding a
packet with pf disabled.

For the whole of a receive task's pass over its ring, including the LRO
flush, the transmit queues it uses, up to eight, defer their doorbells,
and the task rings each of them once at the end of the pass. The whole
pass is covered, not just the if_input() call after it, because with
LRO enabled and IP forwarding on, tcp_lro_queue_mbuf() hands each
packet to if_input() from inside the receive loop; igc, em, igb, ix and
ixl enable LRO by default. A queue defers only while it is at most an
eighth full, where iflib_txd_db_check() would ring for every packet;
above that iflib already batches on its own. Both the mp_ring and the
simple transmit paths take part. Other transmitters, such as the
transmit task when tx_abdicate is set, ring as before, and nothing
changes when net.iflib.min_tx_latency is set. Batched doorbells are
off by default until they have been exercised on more drivers;
net.iflib.tx_db_batched=1 turns them on.

The batch lives on the receive task's stack, the task stays pinned, and
the transmit path finds it on a per-CPU list by thread, so a thread
that preempts the task does not take part. A queue counts the batches
holding its doorbell, and on the mp_ring path the transmit task leaves
such descriptors to them: taking over the ring for them would move the
draining to a task that rings per packet, and a backlog builds whenever
it falls behind. mlx5en has held its send queues' doorbells during
receive processing since 2d5e5a0d75b0; DragonFly's ifq staging and
OpenBSD's transmit mitigation batch packets at the ifq layer instead.

Sponsored by: Rubicon Communications, LLC ("Netgate")

On a Netgate 4200 (4 Atom cores, Intel I226) forwarding 64-byte UDP,
one core rings once per 16 packets instead of once per packet and
forwards 1.35 instead of 1.10 Mpps with pf disabled and 0.44 instead of
0.38 Mpps with pf and 198k states; with LRO enabled it forwards 1.31
instead of 1.08 Mpps. With four queues per port it forwards up to
3.5-3.6 instead of 2.6 Mpps one way and 5.4-5.6 instead of 3.9-4.2 Mpps
both ways, on either transmit path. As a TCP endpoint with LRO, it
receives at line rate with one doorbell per 3.5 ACKs instead of one per
ACK, and sends with one per 9 segments instead of nearly one per
segment.