Style

Oops, I didn't realize that the mount path doesn't hold the covered vnode lock while updating the relevant fields.
It looks as though the unmount path does hold the lock across the relevant operations, so we should be safe from these fields being cleared out from under us, correct?
Would it make sense to add a comment here to that effect?

Apologies for letting this review stall, $WORK got in the way in a big way for a while.
I've rebased, incorporated some of @mjg's optimization suggestions, and retested.

Fix wording in comment, incoporate some suggested performance optimizations

Style

Remove remaining WITNESS directives and no-longer used variables

Is the idea here to proactively update the DRIVER_MODULE because the devclass compat shims will be removed in a future version, or was there some build error on 14?

• Fix typo, restore support for LK_INTERLOCK on vp_crossmp

• Replace runtime check with KASSERT, add comment on CROSSLOCK behavior

I don't really like this change, it feels like a hack to me. At the same time, I don't have another idea that seems clearly better, so I thought I would post this to at least start a discussion.

prune whitespace

Remove unnecessary MNT_RDONLY check

Return EACCES if we can't find a suitable upper vnode; add comments and assertion

KASSERT->VNASSERT

Add comment explaining VV_ROOT check

Restore VV_ROOT management to unionfs_nodeget()

In D33914#766976, @jah wrote:

In D33914#766971, @kib wrote:

You can see yourself what would be the breakage for dotdots: in vfs_lookup.c, the block starting at line 1038 if (cnp->cn_flags & ISDOTDOT) { checks for VV_ROOT to detect the case when it should walk over the mount point instead of calling VOP_LOOKUP(). Most likely, it would result in either upper or lower filesystem call to VOP_LOOKUP(), which again would either break VFS invariant that dotdot lookup is never done on VV_ROOT, or (if unionfs allows sub-directory covering, nullfs does allow it) causes exposition of the vnodes not supposed to be accessed through this mount.

Right now I think that indeed, there are only forced unmount and debugging sysctls that result in the referenced vnode reclamation. But VFS does not guarantee that, and we might grow additional cases for reclaim without breaking the promises.

The other reason I think this *might* not be an issue in practice is that the unionfs node keeps a reference to the parent directory vnode, which is returned for ISDOTDOT lookups instead of calling unionfs_nodeget().
But it also seems as though there are enough corner cases and things that might change in the future here, that it's probably better to restore the VV_ROOT logic in unionfs_nodeget().

In D33914#766971, @kib wrote:

You can see yourself what would be the breakage for dotdots: in vfs_lookup.c, the block starting at line 1038 if (cnp->cn_flags & ISDOTDOT) { checks for VV_ROOT to detect the case when it should walk over the mount point instead of calling VOP_LOOKUP(). Most likely, it would result in either upper or lower filesystem call to VOP_LOOKUP(), which again would either break VFS invariant that dotdot lookup is never done on VV_ROOT, or (if unionfs allows sub-directory covering, nullfs does allow it) causes exposition of the vnodes not supposed to be accessed through this mount.

Right now I think that indeed, there are only forced unmount and debugging sysctls that result in the referenced vnode reclamation. But VFS does not guarantee that, and we might grow additional cases for reclaim without breaking the promises.

In D33914#766955, @kib wrote:

In D33914#766951, @jah wrote:

In D33914#766950, @kib wrote:

What if um_rootvp is reclaimed?

Would that matter as far as the check for VV_ROOT is concerned? It looks as though v_vflag isn't cleared until the vnode is freed, and we always keep a reference to um_rootvp.
Also, unionfs_root() directly returns um_rootvp instead of calling unionfs_nodeget().

Suppose that um_rootvp is reclaimed, and another unionfs vnode is instantiated over corresponding upper or lower vnode. In this case that new vnode does not get VV_ROOT set. This breaks dotdot lookups at least.

In D33914#766950, @kib wrote:

What if um_rootvp is reclaimed?

In D33729#763777, @markj wrote:

I'm sure you've observed this already, but this behaviour of potentially dropping the vnode lock seems dubious: the vnode interface doesn't say anything about whether a particular VOP implementation is allowed to do that, and callers might assume that it doesn't happen. Moreover, lock upgrades will often succeed, so problems that arise from dropping the lock will be rare and hard to debug. Maybe unionfs can use an internal lock to provide mutual exclusion without having to upgrade, but that is probably a simplistic idea, or maybe this is actually not much of a problem in practice.

Seems good to me, but I didn't really look at the tests.

Factor out lock upgrade/downgrade

Check for reclaimed vnodes in various places, chmod +x unionfs1[0-2].sh

Improve assert message

Also simplify writecount management in unionfs_noderem()

Only allow write refs on upper vnode