fix callout lock contention during thread creation/destruction

[mjg]

Summary

The lock is top of the profile when applied to kernel with other patches (will post later):

294846572 (spin mutex:callout)   
1832520 (sleep mutex:TID lock)

disappears with the patch. bumps thread creation/destruction rate 2 -> 3 mln ops/s with 104 workers doing it.

commit 885ad80eaa83261a32815a4f5ec094d35cd5b620
Author: Mateusz Guzik <[e-mail]>
Date:   Thu Aug 25 22:42:42 2022 +0000

    thread: use callout_init_cpu(..., curcpu)
    
    Sorts out callout lock contention during heavy thread
    creation/destruction test.
    
    Reviewed by:
    Differential Revision:

commit fdd578e33ad1be7410ad5384b0e22ab351fb0803
Author: Mateusz Guzik <[e-mail]>
Date:   Thu Aug 25 22:42:08 2022 +0000

    callout: add callout_init_cpu
    
    Reviewed by:
    Differential Revision:

[markj]

Why are we reinitializing td_slpcallout each time a thread struct is reused? The callout is drained before the thread structure is reused. Can't the callout be initialized just once in thread_init()?

[markj]

In D36352#825589, @markj wrote:

Why are we reinitializing td_slpcallout each time a thread struct is reused? The callout is drained before the thread structure is reused. Can't the callout be initialized just once in thread_init()?

I guess that wouldn't fix the actual problem, since in your microbenchmark the callout is never getting scheduled?

[mjg]

right. threads just show up and die.

[markj]

Missing a man page update, and other callout init functions (callout_init_mtx() etc.) still don't have this variant. I'd expect it to be useful for p_limco as well?

IMO the following approach is more general and future-friendly:

• add callout_init_flags() instead of callout_init_cpu()
• add a new callout flag which means "bind it to the current CPU instead of the default CPU"
• then there's no need for callout_init_mtx_cpu() etc., and callout_init_flags() can be further extended in the future without needing yet another function