Only chain non-LRO mbufs when LRO is not possible

[shurd]

Summary

Preserve packet order between tcp_lro_rx() and if_input() to avoid
creating extra corner cases. If no packets can be LROed, combine them
into one chain for submission via if_input(). If any packet can
potentially be LROed however, retain old behaviour and call if_input()
for each packet.

This should keep the 12% improvement for small packet forwarding intact,
but mostly avoids impacting the LRO case.

Test Plan

Ensure small packet forwarding case is still improved, check
that LRO performance isn't degraded.

[sbruno]

Found while testing the iflib'd version of ixgbe(4).

[cem]

If no packets can be LROed, combine them into one change

Probably "chain"

sys/net/iflib.c

2461

It seems like a more accurate name might be check_ipforwarding_disabled. No?

2600–2601

Could combine these to drop a level of indentation

[shurd]

Incorporate feedback.

[shurd]

sys/net/iflib.c

2461

That's what it currently does, but the intent is to indicate if it's possible the packet could be LROed. I'm considering copying the TCP header checks into the function as well, but can't make a strong case for them at this point.

Added a comment to clarify.

[cem]

sys/net/iflib.c

2598

nit: parens aren't needed for this line

2599–2600

I was suggesting to combine the two lines above, not move the assignment into an if(). Like:

if (lro_possible && mf != NULL) {
    ifp->if_input(ifp, mf);
    ...
}

[shurd]

Much better way of dropping a level of indentation. :-)

[shurd]

Fix typo

[shurd]

committed as r325487