sleepqueue: Address a lock order reversal

[markj]

Summary

After commit 74cf7cae4d22 ("softclock: Use dedicated ithreads for
running callouts."), there is a lock order reversal between the per-CPU
callout lock and the scheduler lock. softclock_thread() locks callout
lock then the scheduler lock, when preparing to switch off-CPU, and
sleepq_remove_thread() stops the timed sleep callout while potentially
holding a scheduler lock. In the latter case, it's the thread itself
that's locked, and if the thread is sleeping then its lock will be a
sleepqueue lock, but if it's still in the process of going to sleep
it'll be a scheduler lock.

We could perhaps change softclock_thread() to try to acquire locks in
the opposite order, but that'd require dropping and re-acquiring the
callout lock, which seems expensive for an operation that will happen
quite frequently. We can instead perhaps avoid stopping the
td_slpcallout callout if the thread is still going to sleep, which is
what this patch does. This will result in a spurious call to
sleepq_timeout(), but some counters suggest that this is very rare.

PR: 261198
Fixes: 74cf7cae4d22 ("softclock: Use dedicated ithreads for running callouts.")

[markj]

For reference, I'm running 16 busy bhyve VMs and a -j32 buildworld on a 32-cpu system. sleepq_timeout() gets called 1000-2000 times per second and after 90 minutes debug.sleepq.spurious_timeouts is 368.

[kib]

The sleepq+timeout() was specifically changed to ignore spurious wakeups, esp. from the non-stopped callout.

[kib]

sys/kern/subr_sleepqueue.c

154

I suppose you will remove this counter for final commit, or at least move it under SLEEPQUEUE_PROFILING.

[markj]

sys/kern/subr_sleepqueue.c

154

Indeed, I won't commit this, I just tried to verify my intuition about the frequency of spurious callouts.

[thj]

This seems to fix the panic I was hitting. Before I could reliably panic by doing buildworld with -j 16 in a bhyve guest with 16 cores.