Ok. For the other VFS operations besides mount, would it be preferable to drop MBF_NOWAIT and allow vfs_busy() to block?
I avoided that out of concern for possible deadlock, and it didn't seem necessary given that failure would only happen with a pending concurrent unmount. But I hadn't considered something like the automount case.

Specifically the concern I had regarding deadlock would be if there could ever be a case in which the lower mount would be busied on the call here, then an umount would happen on another thread, followed by our attempt to busy the mount. So basically the same thing that led to unbusying requirements around quota file I/O in VFS_QUOTACTL().

The same concern would apply both here and in the corresponding unionfs change. But for both nullfs and unionfs, would it instead be better to simply prevent even forced unmount of any lower FS while the upper is mounted? mnt_uppers wouldn't work as-is for unionfs, and it seems to imply some extra functionality anyway, but perhaps something like a new mnt_kern_flag ?

(Or rather, a mnt_upper_count or mnt_hold_count field in struct mount, since a mount can have multiple upper mounts)

I agree, busying shouldn't be needed for nullfs_mount(). I'm more concerned with the other vfs_* entrypoints, and also the corresponding entrypoints in unionfs.

I'm hesitant to use mnt_uppers to keep the lower mount from going away for two reasons:

1. The mnt_uppers list seems to be intended as a notification mechanism for managing the nullfs vnode cache, rather than a way to keep the lower mount pinned. I'd rather avoid confusing those 2 goals if possible.

2. The same approach won't work for unionfs, as it assumes no more than 1 lower mount with the list linkage stored in upper->mnt_upper_link.

Why not use my earlier suggestion to add a counter to struct mount?

The changes here and in D30152 would be replaced with a much simpler set of changes that would:

1. Add a mnt_pinned_count field to struct mount
2. Stacked filesystems would increment this field for their lower filesystem(s) in their .vfs_mount callback and decrement it in .vfs_unmount, while holding the lower ilock. unionfs would do this for both "lower" filesystems.
3. The TAILQ_EMPTY(&mp->mnt_uppers) check in dounmount would be replaced with a check of "mnt_pinned_count != 0"

This would work for both nullfs and unionfs, and wouldn't cause any unnecessary VFS_NOTIFY_UPPER_* callbacks in the nocache case for nullfs.

What it wouldn't do is allow unionfs to participate in the VFS_NOTIFY_UPPER_ * scheme, but I think unionfs has bigger issues to be fixed before worrying about that.

I would not want the increment to block. I would want it to fail if unmount is already started, that is if lowermp->mnt_kern_flag & MNTK_UNMOUNT != 0, also checked within the same ilock section as the increment of mnt_pinned_count. Since the increment only happens on mount of the upper filesystem, the mount attempt shouldn't be allowed if there is already a pending unmount request, even if that request doesn't ultimately succeed.

In dounmount(), mnt_uppers is checked while holding the ilock, just before setting MNTK_UNMOUNT. dounmount_cleanup() drops the ilock if we decide to return EBUSY.

Registration in mnt_uppers won't happen in the nocache case.