softclock: Use dedicated ithreads for running callouts.

[jhb]

Summary

Rather than using the swi infrastructure, rewrite softclock() as a
thread loop (softclock_thread()) and use it as the main routine of the
softclock threads. The threads use the CC_LOCK as the thread lock
when idle.

Sponsored by: Netflix

[kib]

The use of cc_lock as thread lock is to avoid lost wakeups from callout_process()?

[mav]

It is a bit like particle physics, but I don't see anything wrong here.

[mav]

One possible downside I can guess in sched_add() calling sched_pickcpu() under the CC_LOCK, if I understand it right, but may be it will be some compensated by reduced complexity.

[imp]

This seems fine... Have a question about the locks that likely could be clarified with a few words...

sys/kern/kern_timeout.c

839

A comment somewhere would be useful about the weird locking that's going on here...

[jhb]

In D33683#761169, @kib wrote:

The use of cc_lock as thread lock is to avoid lost wakeups from callout_process()?

Yes, but in general a thread needs some sort of thread_lock when it is idle. When you are asleep on a sleep queue it is the associated SC_LOCK, or when blocked on a turnstile the TC_LOCK. ithreads are currently a bit of an odd ball as they end up staying with the "running lock" even when idle. Reusing the cc_lock as the thread lock means you don't have to grab the "running lock" just to schedule the thread but can instead reuse the spin lock that is already held to interlock against going to sleep in softclock_thread().

[jhb]

sys/kern/kern_timeout.c

839

It's actually a bit typical in other places like sleepq_switch() and turnstile_wait(). However, it is certainly not immediately obvious (it wasn't to me) that mi_switch() drops the current thread_lock and returns without it held.

[jhb]

• Add some comments on thread_lock locking.
• Reset profile counters in each loop iteration.