Implement L2 lookup caching for TCP and UDP

[• karels]

Summary

This change re-adds L2 caching for TCP and UDP, as originally added in D4306
but removed due to other changes in the system. Restore the llentry pointer
to the "struct route", and use it to cache the L2 lookup (ARP or ND6) as
appropriate.

Test Plan

This change is based on production code in Sidewinder, aka McAfee Firewall
Enterprise. Testing has included changing of MAC addresses on neighboring
systems and verifying that connections continue to work.

[• karels]

Hello? Is anyone looking at this?

[• karels]

Update to current HEAD; fix to work with flowtable

[gnn]

I've tested this in a two host test using iperf. https://github.com/gvnn3/netperf/tree/master/VANILLA/Tests/iperf-twohost-nooffload

and not seen any issues on 10G Chelsio hardware.

[bz]

Just came across this commit tracing other problems ...

head/sys/netinet6/in6_pcb.c

836

Question (post commit review :) Why is this inp_route and not inp_route6? I know the heads of them are identical and it's a union, yet ... for clarity if nothing else?

853

Same here, inp_route6?

[• karels]

You are right, inp_route6 would be better. I'm sure I copied and pasted, then didn't make as many changes as I should have. I will happily approve a review to make that change if you test that it compiles :-).