mbuf: do not restore dying interfaces

[kp]

Summary

When we remove an interface it is first removed from the interface list
V_ifnet (by if_unlink_ifnet()) and marked as IFF_DYING. We then wait for
any possible references to stop being used (i.e.
epoch_wait/epoch_drain_callbacks) before we tear it fully down.

However, the index in ifindex_table is not removed, so m_rcvif_restore()
can still find the (now dying) interface.

This results in panics, for example when dummynet restores the rcvif
pointer and passes a packet to ip6_input() we can panic because the
AF_INET6 domain has already been removed (so we end up dereferencing a
NULL pointer there).

Check that the interface is not dying before we restore it, which is
equivalent to checking its presence in V_ifnet, and thus ensures that
future accesses (while in NET_EPOCH) are safe.

Sponsored by: Rubicon Communications, LLC ("Netgate")

[kp]

Remove mostly unrelated dummynet fix

[glebius]

Why can't we remove it from the ifindex, too? That would be a normal delayed free practice: remove all references to a structure, mark it as being deleted and wait for threads that hold a reference to finish.

[glebius]

Note: we set this flag twice: if_unlink_vnet() and if_free(). This doesn't look consistent.

[glebius]

Ok, this second IFF_DYING was added in a716ad4a35627bfe68d6681e1a4bb0c1e922325d

[kp]

In D34076#770623, @glebius wrote:

Why can't we remove it from the ifindex, too? That would be a normal delayed free practice: remove all references to a structure, mark it as being deleted and wait for threads that hold a reference to finish.

That might be better. I'll take a closer look at that tomorrow, when I'm sure I'm not going to break if_vmove().

[glebius]

I'm also digging that. I think your patch is fine as temporary measure.