vxlan: support moving to a different vnet

[kp]

Summary

Allow the underlay network (i.e. the UDP socket) to be in a different vnet than
the overlay (i.e. whatever we transport throug the vxlan tunnel).
This allows jails using a vxlan interface to be unaware of the required routing.

vxlan interfaces are backed by a socket, which we can't open or close from a
different vnet, so automatically set the interface up when it's moved out of
its home vnet. Also don't change DRV_RUNNING, i.e. keep the socket open even if
the interface is set down as long as it is not in its home vnet.

Sponsored by: Rubicon Communications, LLC ("Netgate")

[glebius]

if_vmove bites again? I'm fine with adding more kludges around this problem as long as we all agree that eventually this thing needs to be removed and interfaces shall be fully destroyed and fully instantiated in a different jail.

[kp]

In D52852#1207545, @glebius wrote:

if_vmove bites again? I'm fine with adding more kludges around this problem as long as we all agree that eventually this thing needs to be removed and interfaces shall be fully destroyed and fully instantiated in a different jail.

That wouldn't quite work for this specific scenario. Without this patch jails could already create their own vxlan interfaces (with the previous patch in this series), but that requires them to know where to route the tunnel traffic to.
With this change it's possible to separate the world the jail sees from the host. You could even have conflicting IP ranges between the two, which isn't going to work if the jail itself creates the vxlan interace. I don't have an immediate use for this myself, but I expect it to be useful to other vxlan users.

[zlei]

I've ever considered this approach, but this adds too many headaches. Well I'd propose to use vlxan(4) + bridge(4) + epair(4) if the underlay network is in different VNET.

For example, how can the admin change the tunnel parameters ( vni / vxlanlocal / vxlanremote / ports ) when the vxlan(4) interface is vmoved to another VNET ?

[kp]

In D52852#1208672, @zlei wrote:

I've ever considered this approach, but this adds too many headaches. Well I'd propose to use vlxan(4) + bridge(4) + epair(4) if the underlay network is in different VNET.

For example, how can the admin change the tunnel parameters ( vni / vxlanlocal / vxlanremote / ports ) when the vxlan(4) interface is vmoved to another VNET ?

In this scenario it can't. Arguably it's a feature for the jail to be unable to change the parameters, but it's not ideal that the host admin can't either (or not without moving the interface out and back into the jail).