Tie up a few lose ends on pf vnet cleanup and in theory if we could
unload the module try to also end the kthread again.

Sponsored by: The FreeBSD Foundation