lockmgr locks:

Cause contention on some lockmgr lock (ufs directory vnode lock):

# dtrace -w -n 'fbt::vfs_hash_insert:entry {chill(100000000)}' &
# jot 0 | xargs -P10 -I %% sh -c "touch %%; rm %%" &

# sysctl debug.kdb.enter=1
debug.kdb.enter:KDB: enter: sysctl debug.kdb.enter
[ thread pid 6279 tid 100204 ]
Stopped at      kdb_sysctl_enter+0x95:  movq    $0,0x121d0c0(%rip)
db> show allchains
chain 1:
 thread 100342 (pid 6283, sh) is blocked on lock 0xfffff8003ca02228 (lockmgr) "ufs" EXCL
 thread 100249 (pid 6273, touch) is running on CPU 0
chain 2:
 thread 100351 (pid 6281, touch) is blocked on lock 0xfffff8003ca02228 (lockmgr) "ufs" EXCL
 thread 100249 (pid 6273, touch) is running on CPU 0
chain 3:
 thread 100361 (pid 6280, touch) is blocked on lock 0xfffff8003ca02228 (lockmgr) "ufs" EXCL
 thread 100249 (pid 6273, touch) is running on CPU 0
chain 4:
 thread 100353 (pid 6278, sh) is sleeping on 0xfffffe00d4d43008 "wait"
chain 5:
 thread 100240 (pid 6277, sh) is sleeping on 0xfffffe00d4d28ab0 "wait"
chain 6:
 thread 100373 (pid 6276, touch) is blocked on lock 0xfffff8003ca02228 (lockmgr) "ufs" EXCL
 thread 100249 (pid 6273, touch) is running on CPU 0
chain 7:
 thread 100380 (pid 6275, touch) is blocked on lock 0xfffff8003ca02228 (lockmgr) "ufs" EXCL
 thread 100249 (pid 6273, touch) is running on CPU 0
chain 8:
db> show lock 0xfffff8003ca02228
 class: lockmgr
 name: ufs
 state: XLOCK: 0xfffff80008699000 (tid 100249, pid 6273, "touch")
 waiters: shared and exclusive
 spinners: none
db> c
 0 -> 0

sx locks:

First, compile kernel with NO_ADAPTIVE_SX so that we actually get waiters
(should we somehow make it so show lockchain can show spin waits?)

Cause contention on some sx lock:

dtrace -w -n 'fbt::fail_point_garbage_collect:entry {chill(10000000)}' &
while true; do sysctl debug.fail_point.test_fail_point=off{,,,,,,,,,,,,,,,} > /dev/null; done &
while true; do sysctl debug.fail_point.test_fail_point=off{,,,,,,,,,,,,,,,} > /dev/null; done &
while true; do sysctl debug.fail_point.test_fail_point=off{,,,,,,,,,,,,,,,} > /dev/null; done &
while true; do sysctl debug.fail_point.test_fail_point=off{,,,,,,,,,,,,,,,} > /dev/null; done &

# sysctl debug.kdb.enter=1
debug.kdb.enter:KDB: enter: sysctl debug.kdb.enter
[ thread pid 3495 tid 100207 ]
Stopped at      kdb_sysctl_enter+0x95:  movq    $0,0x121d0c0(%rip)
db> show alltrace
No such command; use "help" to list available commands
db> show allchains
chain 1:
 thread 100280 (pid 3494, sysctl) is blocked on lock 0xffffffff81da07b0 (sx) "fail point set sx" XLOCK
 thread 100206 (pid 3491, sysctl) is running on CPU 2
chain 2:
 thread 100257 (pid 3492, sysctl) is blocked on lock 0xffffffff81da07b0 (sx) "fail point set sx" XLOCK
 thread 100206 (pid 3491, sysctl) is running on CPU 2
chain 3:
 thread 100295 (pid 3490, sysctl) is blocked on lock 0xffffffff81da07b0 (sx) "fail point set sx" XLOCK
 thread 100206 (pid 3491, sysctl) is running on CPU 2
chain 4:
 thread 100313 (pid 3058, ftcleanup) is sleeping on 0xffffffff82b015cc "ftcl"
chain 5:
 thread 100299 (pid 1795, zsh) is sleeping on 0xfffffe009460b620 "sigsusp"
chain 6:
 thread 100298 (pid 1793, zsh) is sleeping on 0xfffffe00d4d1c0b0 "sigsusp"
chain 7:
 thread 100297 (pid 1791, zsh) is sleeping on 0xfffffe00d4d1c608 "sigsusp"
chain 8:
 thread 100296 (pid 1789, zsh) is sleeping on 0xfffffe00d4d1cb60 "sigsusp"
chain 9:
db> show lock 0xffffffff81da07b0
 class: sx
 name: fail point set sx
 state: XLOCK: 0xfffff80008722000 (tid 100206, pid 3491, "sysctl")
 waiters: exclusive
db> c
 0 -> 0