Boost thread priority while changing CPU frequency

[gallatin]

Summary

When powerd tries to adjust the frequency of a core which is CPU bound running a kernel thread (typically a NIC driver ithread, or iflib gtaskq), it can get "stuck" since its lower priority than the kernel thread. Worse, it is stuck holding Giant, which causes many things to seem to "freeze". This leads to the CPU-bound kernel thread continuing to run on a down-clocked core, and terrible performance.

If we temporarily boost powerd's priority for the duration of the frequency adjustment, then the core will get up-clocked immediately, and we'll have much better performance.

PI_NET - 1 was chosen for the boosted prio, as I've seen this most frequently with NIC ithreads and iflib gtaskqs.

[imp]

This seems sane to me, but my suspend/resume skills aren't what they were in 2005....

[jhb]

Adding kib@.

It's not clear to me why to prefer PI_NET - 1 over, say, PRI_MIN in this case. There's nothing about powerd that is network specific.

[gallatin]

I've replaced the arbitrary PI_NET - 1 to PRI_MIN, as suggested by John

[gallatin]

In D15246#321533, @jhb wrote:

Adding kib@.

It's not clear to me why to prefer PI_NET - 1 over, say, PRI_MIN in this case. There's nothing about powerd that is network specific.

Sorry about that.. That was just the first thing I found that worked in the only scenario where I could easily reproduce it. I've updated the patch to use PRI_MIN.

[kib]

I am not sure why do you use sched_lend_prio(). This method is supposed to be used when the thread lends priority from another thread, which is not the case there. I would just use sched_prio().

If you do not use sched_lend_prio(), then you can rely on the subr_trap.c:userret() to return the user thread priority reset.

[gallatin]

Change to using sched_prio() rather than sched_lend_prio, as suggested by @kib