Implement per-thread counters for PMC sampling --- redux

[mmacy]

Summary

This is D4227 with the changes requested by @jhb plus some additional bug fixes exposed on a cursory review. Not entirely sure how to test this until I bring in D7350.

Depends on D15155

[kib]

sys/dev/hwpmc/hwpmc_mod.c

1626

Why is reload subtraction only done for process counter ?

1680

Blank line is excessive.

1694

There too.

2297

Why freelist_mtx needs to be spinlock ?

2304

return (pt);

2341

So if some other consumer drain the pool in parallel, you can end up allocating the entries only to free them. Why not chomp the head of the list of needed length under the freelist_mtx, and then free the entries after unlock ? This also removes lock/unlock for each free.

sys/kern/kern_thr.c

270

Why not simply check PMC_PROC_IS_USING_PMCS() after the unlock, without else ? You call pmc hook after the unlock anyway, so it gives the same race.

[mmacy]

Apply review suggestions

sys/dev/hwpmc/hwpmc_mod.c

1626

We automatically reset the counter on read. For processes we need to keep track of updates across all threads and then track the delta until the next reload. For threads the value won't exceed reloadcount.

2297

It's called from sched_switch with the thread_lock held.

[kib]

sys/dev/hwpmc/hwpmc_mod.c

2346

Don't you need to remove pt from pmc_threadfreelist first ?

[mmacy]

sys/dev/hwpmc/hwpmc_mod.c

2346

DOH

[mmacy]

add missed LIST_REMOVE