vfs: use vn_lock_pair to avoid establishing an ordering on mount

[mjg]

Summary

This fixes some of the LORs of this form:

lock order reversal: 
 1st 0xfffff80002c50930 ufs (ufs, lockmgr) @ /usr/src/sys/kern/vfs_mount.c:2052
 2nd 0xfffff80002994cb0 devfs (devfs, lockmgr) @ /usr/src/sys/kern/vfs_subr.c:3009
lock order devfs -> ufs established at:
#0 0xffffffff8053bb7d at witness_checkorder+0x46d
#1 0xffffffff804a38e8 at lockmgr_lock_flags+0x188
#2 0xffffffff806c608d at ffs_lock+0x8d
#3 0xffffffff807b6390 at VOP_LOCK1_APV+0x40
#4 0xffffffff805cb2a4 at _vn_lock+0x54
#5 0xffffffff805a8e6d at vfs_domount+0xf1d
#6 0xffffffff805a6de2 at vfs_donmount+0x872
#7 0xffffffff805abcf7 at kernel_mount+0x57
#8 0xffffffff805ae831 at parse_mount+0x4a1
#9 0xffffffff805acca7 at vfs_mountroot+0x587
#10 0xffffffff8046187f at start_init+0x1f
#11 0xffffffff8048e990 at fork_exit+0x80
#12 0xffffffff8075c45e at fork_trampoline+0xe
lock order ufs -> devfs attempted at:
#0 0xffffffff8053c4dc at witness_checkorder+0xdcc
#1 0xffffffff804a5325 at lockmgr_xlock+0x55
#2 0xffffffff807b6390 at VOP_LOCK1_APV+0x40
#3 0xffffffff805cb2a4 at _vn_lock+0x54
#4 0xffffffff805b2f5a at vget_finish+0x7a
#5 0xffffffff8042b94f at devfs_allocv+0xbf
#6 0xffffffff8042b103 at devfs_root+0x43
#7 0xffffffff805b88d0 at vfs_cache_root_fallback+0x120
#8 0xffffffff805b42cc at vflush+0x4c
#9 0xffffffff8042b00e at devfs_unmount+0x3e                        
#10 0xffffffff805aa2ae at dounmount+0x81e  
#11 0xffffffff805b5ab1 at vfs_unmountall+0xc1    
#12 0xffffffff80586efe at bufshutdown+0x2ce
#13 0xffffffff804ccd83 at kern_reboot+0x213    
#14 0xffffffff804ccb0f at sys_reboot+0x39f  
#15 0xffffffff80786644 at amd64_syscall+0x124        
#16 0xffffffff8075bcfe at fast_syscall_common+0xf8

There is still ordering established at unmount which is going to be taken care of with a different patch. The unmount one results in LORs if filesystems get interleaved, for example tmpfs -> ufs -> tmpfs

Test Plan

tested by pho

[kib]

Chuck has a patch to handle unmount, by some lock reordering.

sys/kern/vfs_mount.c

1196

I would move the setting of VIRF_MOUNTPOINT before potential unlock of vp inside vn_lock_pair(), just in case.

[mckusick]

I concur with kib's suggestion. Otherwise looks like a good use of vn_lock_pair().

[mjg]

• handle some of the stuff earlier