if_epair: Hash the IP header for fanout not the Ethernet header

[rcm]

Summary

With RSS, epair_select_queue() hashes a packet that carries no RSS hash
with rss_soft_m2cpuid_v4() or rss_soft_m2cpuid_v6(), which read the IP
header at the start of the mbuf. On transmit the mbuf starts with the
Ethernet header, so the hash covered the MAC addresses, the EtherType
and the first bytes of the IP header, the IPv4 total length and
identification among them. The packets of one flow were spread over the
queues by their IP ID and delivered out of order, and the value was
stored in the mbuf as an RSS hash for the receiving side to use.

Step over the Ethernet header, and over inline 802.1Q or 802.1ad tags,
while the software hash runs. A tag is inline when vlan(4) or a bridge
member runs without VLAN_HWTAGGING, always for 802.1ad, and
for the inner tag of stacked vlans; those frames previously went to the
first queue unhashed. Frames with more than two tags still do.

Fixes: 24f0bfbad57b ("if_epair: implement fanout")
MFC after: 2 weeks
Sponsored by: Rubicon Communications, LLC ("Netgate")

Test Plan

Measured with an 8-CPU RSS kernel sending UDP datagrams from a raw IP
socket, whose packets carry no RSS hash, through an epair to a netmap
sink that checks a sequence number in the payload: at 200k packets per
second 52 % of the datagrams of one flow arrived behind a later one,
displaced by up to 3800; at 50k per second 3 %. With this change none
do, at any rate, and the flow stays on one queue. Datagrams from a UDP
socket were in order both ways: udp_output() hashes them itself.

[kp]

Should we try to account for a VLAN header here too?

[rcm]

In D61037#1399489, @kp wrote:

Should we try to account for a VLAN header here too?

Ah that's a good point, yes we probably should.