ithreads: Support priority adjustment by schedulers.

[jhb]

Summary

Use sched_wakeup instead of sched_add when marking an ithread
runnable. This allows schedulers to reset their internal time slice
tracking state and restore the base ithread priority when an ithread
resumes from idle.

Sponsored by: Netflix

[kib]

So this change would be not needed if you do the prio adjustments in sched_add(). I think more explanations are needed if there is a reason to handle ithreads prio in sched_wakeup(). Is it just a micro-optimization to avoid one more if() in sched_add()?

[jhb]

One more if that would end up duplicating logic. However, it's also true that you don't want to do it on every sched_add(). For example, if an ithread is preempted the old thread gets put on the runqueues via sched_add() and you don't want that step to reset the scheduler's internal notion of CPU usage (how much of a time slice has been used). You only want to reset the usage back to zero when the ithread goes idle. In practice though it is simpler to mimic what happens now for other threads which is to reset it not when going idle, but when resumed from being idle. This is how it works today for other threads that only reset their time slice usage when resumed from sleeping, not every time they are put on a run queue.