devfs: introduce a per-dev lock to protect ->si_devsw

[mjg]

Summary

This allows bumping threadcount without taking the global devmtx lock.

markj reports it eliminates contention on said lock on a sample system with 32 instances of bhyve. Also it went went through stress testing while hosting syzkaller. No issues found.

[kib]

This absolutely requires a bump for D_VERSION.
May be add explicit order between devmtx and si_threadlock to the witness static table.

[mjg]

• bump D_VERSION
• annotate the lock in witness

[kib]

sys/sys/conf.h

174

After closer look, can we avoid this bump ? si_threadlock is used (should be used) only be devfs internals, so if you move the lock into cdev_priv it should still work.

[mjg]

• move the lock to cdev_privdata

[kib]

Except for the headers pollution, I am fine with the change.

sys/fs/devfs/devfs_int.h

43 ↗

(On Diff #64923)

Is that needed ? Even if it is, devfs_int.h is included in 5 or 6 files at all, so it is easy to arrange sys/mutex.h before it in .c files.

[mjg]

• drop _lock.h and _mtx.h

I included them towards self-contianment of the header file. universe MAKE_JUST_KERNELS builds just fine (except for powerpc* which were note tested)