ip_forward() looked the route up with NHR_REF and released the nexthop on
every exit. The function runs in the network epoch, and nexthops are
destroyed through NET_EPOCH_CALL(), so the nexthop stays valid until
ip_forward() returns without a reference, as ip_tryforward() already
relies on.
The reference costs two atomic operations per packet on a
counter that every CPU forwarding to that nexthop shares.
ip_output() may drop the nexthop of the route it is given, when the
nexthop is no longer valid or a pfil hook asks for a new lookup, and store
a referenced one in its place. Add a route flag, RT_NH_NOREF, for a route
whose nexthop was looked up without a reference. RO_NHFREE() and
RO_INVALIDATE_CACHE() drop such a nexthop without releasing it and clear
the flag, so a nexthop stored in the route later is released as before.
ip_forward() sets the flag on its on-stack route.
The flag must not be set on a route that outlives the epoch section of the
lookup, such as the route of a PCB. ip_output() asserts that a route
passed along with a PCB does not have it.
This is RT_NORTREF, which dcfa556b02dd removed as unused in 2017, with
the same bit and the same clear-on-release behaviour.
No functional change intended.
MFC after: 2 weeks
Sponsored by: Rubicon Communications, LLC ("Netgate")