callout: Wait for the softclock thread to switch before rescheduling

[markj]

Summary

When a softclock thread prepares to go off-CPU, the following happens in
the context of the thread:

1. callout state is locked
2. thread state is set to IWAIT
3. thread lock is switched from the tdq lock to the callout lock
4. tdq lock is released
5. sched_switch() sets td_lock to &blocked_lock
6. sched_switch() releases old td_lock (callout lock)
7. sched_switch() removes td from its runqueue
8. cpu_switch() sets td_lock back to the callout lock

Suppose a timer interrupt fires while the softclock thread is switching
off, and callout_process() schedules the softclock thread. Then there
is a window where callout_process() can call sched_add() while td_lock
is &blocked_lock, but this is not permitted since the thread is not
logically locked.

callout_process() needs to spin waiting for the softclock thread to
finish switching off (i.e., after step 8 completes) before rescheduling
it.

Reported by: [e-mail]
Fixes: 74cf7cae4d22 ("softclock: Use dedicated ithreads for running callouts.")

[jhb]

Thanks, was just debugging this. I guess the fact that ithreads still don't use a real thread lock is what threw me off.