pf: Fix panic on unload

[kp]

Summary

vnet_pf_uninit() is called through vnet_deregister_sysuninit() and
linker_file_unload() when the pf module is unloaded. This is executed
after pf_unload() so we end up trying to take locks which have been
destroyed already.

Ensure we don't call pf_unload_vnet() from vnet_pf_uninit() for the root
vnet. We still need to do that cleanup, so call it directly from
pf_unload().

Test Plan

kldload pf
pfctl -g -f pf.conf
pfctl -e
kldunload -f pf

[eri]

It feels a lot like a hack.
Shouldn't the proper VNET accessor be called on creation and teardown?

[kp]

In D10025#207198, @eri wrote:

It feels a lot like a hack.
Shouldn't the proper VNET accessor be called on creation and teardown?

Yes, but that's essentially what's happening here. The root/default vnet is getting torn down.
The problem is the order of operations. pf_modevent() gets called before vnet_pf_uninit().

[eri]

I would rather remove the pf_unload and pf_load to empty stubs and do their operation in pf_vnet_[un]init wiht DEFAULT_VNET wrapping than this.

Check ipfw(4) code and it has the same solution and makes consistency.

The other way is to use a semaphore pattern but the above is much simpler.

[kp]

I think you have a point, but I don't want to move pf_load(), because that can fail.

[kp]

Use SYSUNINIT() to ensure pf_load() isn't called until after all vnet_pf_uninit()s have been called