On non-trivial SMP systems the contention on the pmc_owner mutex leads to a substantial number of samples captured being from the pmc process itself. This change a) makes buffers larger to avoid contention on the global list b) makes the working sample buffer per cpu.
will-it-scale/page_fault1_processes -t 96 -s 30 >& /dev/null &
pmcstat -S UNHALTED_CORE_CYCLES -n 21000000 -O pf1.pmcstat sleep 10
pmcstat -R pf1.pmcstat -z100 -G pf1.stacks
before:
https://github.com/mattmacy/profiling/blob/master/2018.04.22/pf1_orig.svg
after:
https://github.com/mattmacy/profiling/blob/master/2018.04.22/pf1.svg
without pmcstat running:
mmacy@anarchy [~/devel/freebsd|0:09|32] time make -j96 buildkernel -s >& /dev/null
make -j96 buildkernel -s >&/dev/null 2508.41s user 1047.14s system 6000% cpu 59.259 total
before:
https://github.com/mattmacy/profiling/blob/master/2018.04.22/pf1_orig.svg
after:
https://github.com/mattmacy/profiling/blob/master/2018.04.22/pf1.svg
with pmcstat running before:
pmcstat -S UNHALTED_CORE_CYCLES -n 21000000 -O /dev/null sleep 600 &
[1] 8918
mmacy@anarchy [~/devel/freebsd|0:29|68] time make -j96 buildkernel -s >& /dev/null
make -j96 buildkernel -s >&/dev/null 3336.68s user 24684.10s system 7442% cpu 6:16.50 total
with pmcstat running after:
pmcstat -S UNHALTED_CORE_CYCLES -n 21000000 -O /dev/null sleep 180 &
mmacy@anarchy [~/devel/freebsd|0:14|48] time make -j96 buildkernel -s >& /dev/null
make -j96 buildkernel -s >&/dev/null 2688.13s user 1547.66s system 6053% cpu 1:09.97 total