Since non-doomed unionfs vnodes always share their primary lock with
either the lower or upper vnode, any forwarded call to the base FS
which transiently drops that upper or lower vnode lock may result in
the unionfs vnode becoming completely unlocked during that transient
window. The unionfs vnode may then become doomed by a concurrent
forced unmount, which can lead to either or both of the following:
--Complete loss of the unionfs lock: in the process of being
doomed, the unionfs vnode switches back to the default vnode lock, so even if the base FS VOP reacquires the upper/lower vnode lock, that no longer translates into the unionfs vnode being relocked. This will then violate that caller's locking assumptions as well as various assertions that are enabled with DEBUG_VFS_LOCKS.
--Complete less of reference on the upper/lower vnode: the caller
normally holds a reference on the unionfs vnode, while the unionfs vnode in turn holds references on the upper/lower vnodes. But in the course of being doomed, the unionfs vnode will drop the latter set of references, which can effectively lead to the base FS VOP executing with no references at all on its vnode, violating the assumption that vnodes can't be recycled during these calls and (if lucky) violating various assertions in the base FS.
Fix this by adding two new functions, unionfs_forward_vop_start()
and unionfs_forward_vop_finish(), which are intended to bookend
any forwarded VOP which may transiently unlock the relevant vnode(s).
These functions are currently only applied to VOPs that modify file
state (and require vnode reference and lock state to be identical at
call entry and exit), as the common reason for transiently dropping
locks is to update filesystem metadata.
MFC after: 2 weeks