Although it looks improbable, it might be the cause of the observed latency problem at Netflix.

We may have hit this one but that's not the one I'm testing a patch for right now. If it looks good in our setup I'll share it. It's a bit more of a fundamental scheduler change that we may or may not want to have enabled by default.

Do you have your tree for the stuff from your talk because, I'd like to compare notes. I had a few other odd behaviors I found. Some of them I didn't fix yet but couldn't tell if you did from your talk.

Also can someone with access on review make a sched group?

For now I don't have enough time to visit this, so I'll take a look at this later. If I don't give any comment by Saturday evening, you can commit this.

Wouldn't this clamp threads with different priorities to the same index (tdq->tdq_ts_deq_off - 1)?

Is there any way that we can ensure tdq_ts_deq_off is always behind tdq_ts_off + MODULO and replace the whole thing as MPASS() instead?

In D57829#1330688, @minsoochoo0122_proton.me wrote:

Wouldn't this clamp threads with different priorities to the same index (tdq->tdq_ts_deq_off - 1)?

Is there any way that we can ensure tdq_ts_deq_off is always behind tdq_ts_off + MODULO and replace the whole thing as MPASS() instead?

Forget what I said above, we cannot do that under current design. I think clamping is fine if 1) it only happens to a few lowest priorities and 2) this situation doesn't happen often.

I think the current runq design is good for 4BSD, but it sometimes fails to address cases like this for ULE. A future improvement can be making the circular queue long enough so this doesn't happen, but it requires refactoring runq code as well both on 4BSD and ULE side.

Hence, LGTM:)

In D57829#1330701, @minsoochoo0122_proton.me wrote:

Forget what I said above, we cannot do that under current design. I think clamping is fine if 1) it only happens to a few lowest priorities and 2) this situation doesn't happen often.

That's a legitimate remark. Yes, clamping happens and leads to threads with the highest priority values to be treated similarly. The effect, however is small for a number of reasons: Prior to this change, the last two priority values were treated the same, and now the three last are, to be compared with the number of queues allocated to the timesharing policy (109); also, the threads concerned by this are necessarily niced ones with high nice values and a high %CPU on the same CPU, which is the same case as described in the commit message and occurs fairly rarely; finally, nice handling is already largely broken in ULE (i.e., very weak), so needs to be revisited anyway.

I think the current runq design is good for 4BSD, but it sometimes fails to address cases like this for ULE. A future improvement can be making the circular queue long enough so this doesn't happen, but it requires refactoring runq code as well both on 4BSD and ULE side.

Well, actually, the part of the runqueue treated as circular is as long as it can get given the current levels reserved for other policies and the total number of runqueues. But there's no need to change anything here, just the runqueue index computations could be changed to account for possible aliasing. I don't see anything that would be needed on 4BSD's side?