ffs: lock buffers after snaplk with LK_NOWITNESS

[kib]

Summary

Also:

Add GB_NOWITNESS flag

ffs_alloc.c: order includes alphabetically

[markj]

sys/kern/vfs_bio.c

4010 ↗

(On Diff #102029)

IMO it would look nicer written this way:

lockflags = LK_EXCLUSIVE | LK_INTERLOCK |
    ((flags & GB_LOCK_NOWAIT) != 0 ? LK_NOWAIT : LK_SLEEPFAIL);
#ifdef WITNESS
lockflags |= (flags & GB_NOWITNESS) != 0 ? LK_NOWITNESS : 0;
#endif

sys/ufs/ffs/ffs_alloc.c

89

Why did these get reordered? It's not alphabetical.

sys/ufs/ffs/ffs_balloc.c

72 ↗

(On Diff #102029)

Why not after mount.h?

633 ↗

(On Diff #102029)

I think some comment is deserved. I also do not quite understand this given Kirk's comment in D33921:

Turning to buffer deadlocks, the only way that the snapshot can be updated is when it internally decides to make a copy. Here we need another buffer in which to grab the old contents. As you noted, this will always be a new buffer so cannot conflict with any existing buffer, so that is not a problem.

If we're always locking a new buffer, then isn't it sufficient to use LK_NOWAIT in buf_alloc() as is done in D34072? I think witness will not complain when LK_NOWAIT is used. If not, then why is this safe?

[kib]

sys/ufs/ffs/ffs_alloc.c

89

There were even more. I think I fixed everything now.

sys/ufs/ffs/ffs_balloc.c

633 ↗

(On Diff #102029)

I suspect LK_NOWAIT would not work. It is perfectly fine to have ufs_balloc_ufs2() to wait for some buffer lock, which is after the snap lock. For instance, the indirect block buffer might be LK_KERNPROC still after other CoW action allocated some indirect block and wrote the indirect buffer asynchronously.

[kib]

Update patch with the Mark' notes changes. This is the diff on top of D34072.

[mckusick]

Once passing Peter Holm's testing, this looks good to go.

[kib]

Peter confirms that this patch hides snaplk->bufwait order from witness.