Currently several paths in the NFS client upgrade the shared vnode lock to exclusive, which might cause temporal dropping of the lock. This action appears to be fatal for nullfs mounts over NFS. If the operation is performed over nullfs vnode, then bypassed down to NFS VOP, and the lock is dropped, other thread might reclaim the upper nullfs vnode. Since on reclaim the nullfs vnode lock and NFS vnode lock are split, the original lock state of the nullfs vnode is not restored. As result, VFS operations receive not locked vnode after a VOP call. See https://people.freebsd.org/~pho/stress/log/kostik1019.txt as an example.
The solution I propose is to stop upgrading the vnode lock when we check the consistency or flush buffers as result of detected inconsistency. Instead, each NFS vnode gets a new lockmgr lock which is locked exclusive instead of the vnode lock upgrade. In other words, the other parallel modification of the vnode are excluded by either vnode lock conflict or exclusivity of the new lock when the vnode lock is shared.
This mostly worked fine, the only VFS change I have to add is to allow vinvalbuf() to operate with the shared vnode lock. This mode allows other clean buffers to arrive while we flush the buf lists for the vnode, which is fine. In fact, only the assert have to be relaxed.
I considered that adding a new lockmgr lock to each NFS vnode is fine even with the associated node structure grow. If considered to be excessive, a per-mount lock can be used instead, but the upgrade lock is used often and contended highly enough.