It seems with the parts tested a top-down network stack cleanup works.

The exception to the rules are cloned interfaces which unfortunately
need to go first, which is a constrained we inherit as the clone destroy
functions cleanup themselves and we can't break that for the normal
ifconfig destroy case. We introduce a VNET state field for this to
check. It is unclear to me if we want that state just be the SI_SUB_*
level we are currently on instead of manually defining state. That way
we would be more flexible?

In general this changes is very careful (and introduces flags in various
places for that) to distinguish between the cases. On VNET shutdown we
do not want to do the normal interface detach and cleanup as we cleanup
upper layers, protocol layers, etc.

Extra fun had with:

• bpf, on vmove we lose the if_bpf due to the event handler on detach but we do not re-attach. It did not seem to be 100% reliable which made me wonder but also how this may have worked in the past and not been noticed. I saw panics in epair on the BPF_MTAP right after attach from nd6 sending a packet.
• de-tangling multicast and it's assumptions on interfaces and a proper cleanup needed to be handled carefully.
• plugging the rtsock zone leak turned out to be interface routes never being removed properly in the top-down teardown. It's unclear to me if they are in the current head version.
• all the inter-dependencies between layers and lists and hashes and locks and the various bits called on on teardown or from various places causing problems when suddenly certain layers had already been "destroyed" yet their locks were tried to be acquired.

Sponsored by: The FreeBSD Foundation