sched_pickcpu() returns early only for pinned threads (td_pinned != 0).
A thread whose cpuset holds a single CPU, as set by
cpuset_setaffinity(2) or pthread_setaffinity_np(3), is not pinned in
that sense. Unless its last CPU is idle, it falls through to
cpu_search_lowest() over the whole topology, even though the only
possible answer is that one CPU.
The search is costly in a common layout: one worker thread bound to each
CPU, with a NIC interrupt thread on the same CPU waking its worker for
every batch of received packets. The worker's CPU is busy running the
interrupt thread at wakeup, so the idle-CPU shortcut never applies, and
every wakeup walks the cpu_group tree.
Track a single-CPU mask with a new cpuset flag, CPU_SET_ONECPU. It is
updated wherever cs_mask changes (cpuset_init(), cpuset_update(), which
recurses into child sets, and the root set), so it always matches the
mask. sched_pickcpu() checks it and returns that CPU without the search.
Add a SCHED_STATS counter, pickcpu_onecpu.
On a 64-thread AMD EPYC 7502P (8-CPU LLC groups) under a sustained UDP
receive load with one bound worker per CPU, kernel CPU time in the
receive path fell by 60%, from 1.72 to 0.69 profile samples per second.
sched_pickcpu() went from 73% of the receive path to 4.8%, and
cpu_search_lowest() dropped out of the profile.
Sponsored by: Netflix