if_ovpn: avoid LOR between ovpn and UDP locks
When we install the tunneling function we had the ovpn lock, and then took the
UDP lock. During normal data flow we are called with the UDP lock held and then
take the ovpn lock.
This naturally produces a lock order reversal warning.
Avoid this by releasing the ovpn lock before installing the tunnel function.
This is safe, in that installing the tunnel function does not fail (other than
with EBUSY, which would mean another thread has already installed the function).
On cleanup the problem is more difficult, in that we cannot reasonably release
the ovpn lock before we can remove the tunneling function callback.
Solve this by delaying the removal of the tunnel callback until the ovpn_softc
is cleaned up. It's still safe for ovpn_udp_input() to be caled when all peers
are removed. That will only increment counters (which are still allocated),
discover there are no peers and then pass the message on to userspace, if
any userspace users of the socket remain.
We ensure that the socket object remains valid by holding a reference, which
we release when we remove the ovpn_softc. This removes the need for per-peer
reference counting on the socket, so remove that.
Reviewed by: zlei
Sponsored by: Rubicon Communications, LLC ("Netgate")
Differential Revision:: https://reviews.freebsd.org/D46616