tests/sys/amd/pmc: add AMD PMC ATF test suite

[janeri_gmail.com]

Summary

Add an ATF test suite for AMD Performance Monitoring Counter (PMC)
features exposed by hwpmc(4) on amd64.

The suite is organized under tests/sys/amd/pmc/ and covers:

hwpmc_exterr_test       -- extended error propagation for PMCALLOCATE,
                           PMCATTACH and PMCRW ioctls (AMD K8 and IBS)
hwpmc_pidtrack_test     -- per-process PMC lifecycle: context-switch
                           save/restore, no-leakage across siblings,
                           live self-read semantics
hwpmc_callchain_ur_test -- PMC_UR callchain regression (rGaf3929c5152b,
                           rG66118c3f1011)
pmu_detect_test         -- AMD vendor check, hwpmc init, PMC_CLASS_K8,
                           Zen-generation pool sizing
pmu_event_test          -- pmu-events table lookup, evsel/umask encoding
                           including EventSelect[11:8] for Zen4+,
                           end-to-end counter read
pmu_event_neg_test      -- rejection of reserved bits, guest/host bits,
                           PRECISERETIRE without capability
pmu_mux_test            -- slot count from CPUID Fn80000022 at runtime,
                           fill/over-allocate/reuse
pmu_cross_class_alloc_test -- simultaneous K8+RAPL allocation
rapl_test               -- PMC_CLASS_RAPL: detect, PKG/CORES delta
                           under load, 30-sample monotonicity, DRAM
                           and process-mode rejection on AMD
pmcstat_tsc_test        -- pmcstat TSC column format and flamegraph
pmcstat_grouping_test   -- system-event grouping; AMD core runtime
                           tests opt-in via amd.pmc.grouping.runtime=true
pmcstat_ibs_errata_test -- IBS errata #1238 suppression (Zen 3 B0)
pmcstat_group_alloc_atomicity_test -- concurrent process allocations
pmc_cli_smoke_test      -- pmc(8) frontend: record, info, IBS trace

Shared helpers included in this diff:

amd_pmc_test_common.h      -- CPUID vendor check, skip guards, pmc_release
                              wrapper; used by all ATF-C programs
tests/sys/amd/umcdf/amd_umcdf_decode.h, amd_umcdf_common.h --
                              CPU generation detection and skip helpers
                              (also used by the follow-on umcdf suite)

tests/sys/Makefile is modified to wire tests/sys/amd/ into the build
via _amd, following the _pmc/_audit pattern, restricted to amd64.

All test cases pass or skip with documented reasons on AMD EPYC 9654
(Zen 4, 192 logical CPUs) running FreeBSD 16.0-CURRENT with hwpmc(4).
Result: 70/79 passed, 0 failed, 9 skipped.

Expected skips:

• PMC_CLASS_PERF (APERF/MPERF): requires a follow-on hwpmc diff
• pmcstat grouping process tests: opt-in via amd.pmc.grouping.runtime
• IBS Zen3 B0 errata test: skip on non-affected CPUIDs

The following programs are present in the source tree but excluded from
this diff and submitted separately:

hwpmc_aperf_mperf_test     -- requires PMC_CLASS_PERF
hwpmc_grouping_test        -- requires PerfMonV2/grouping kernel structs
hwpmc_group_api_test       -- same
pmu_perfmonv2_test         -- same
pmcstat_batch_grouping_test -- requires pmcstat -b

Tested on: amd64 (EPYC 9654, Zen 4, 192 logical CPUs)
Sponsored by: Advanced Micro Devices, Inc.
Signed-off-by: Osvaldo Janeri Filho <[e-mail]>

[netchild]

• End of last month I added some tests/sys/pmc tests. While the items in this review seem to be AMD specific, they are testing parts of the PMC subsystem. I would expect the tests in the pmc directory (if you want in an amd subdirectory, but I have already some amd specific tests in the main OMC directory), than as it is vice versa.
• Is it feasible to use speaking names for model in amd_umcdf_map_zen()?
• hwpmc_exterr_test.c is a near-copy of the in-tree tests/sys/pmc/pmc_exterr_test.c. 13 of 14 case names identical; adds amd_missing_pmu_flag, drops amd_invalid_df_config_bits. Copyright/author line changed from the original.
• The directory never builds. tests/sys/Makefile gains TESTS_SUBDIRS+= ${_amd} but _amd is defined nowhere in the diff (the _pmc/_audit pattern needs the .if … _amd= amd .endif block, and _pmc is also gated on MK_PMC, which this isn't).
• tests/sys/amd/pmc/Makefile starts with a UTF-8 BOM and has no trailing newline. make may choke on the BOM.
• Content for other diffs shipped here. msr_snapshot.[ch] is not compiled ("see follow-on diff").
• The two umcdf/ headers seem to belong to to something that isn't in this diff, yet five pmc tests #include "../umcdf/amd_umcdf_common.h" for Zen-generation decoding. Not sure if the decde should belong together with amd_pmc_test_common.h or with this something else not included here.
• hwpmc_rdpmc_test is in the Makefile and the diff but not in the summary.
• Roughly half the suite is not AMD-specific. The callchain-UR regression (af3929c5152b, 66118c3f1011 — both vendor-neutral hwpmc_mod.c fixes), pmcstat TSC column, group-alloc atomicity, pidtrack, pmc(8) CLI smoke: they only use AMD event names. Under tests/sys/pmc/ with a per-vendor event table they'd cover Intel too. Having this in a vendor-specific subdirectory would need a reorg to add intel support. Better to not use a vendor specific directory (I do not ask someone from AMD to provide the Intel parts, but not making it harder to add them would be polite).
• Vendor detection via /dev/cpuctl0 (needs cpuctl.ko + root) in the C tests vs. kern.hwpmc.cpuid in the sh tests. <machine/cpufunc.h> do_cpuid() needs no device.
• AMDID2_* redefined instead of <machine/specialreg.h>.
• "../amd_pmc_test_common.h" relative includes and -I${SRCTOP}/tests/sys/amd — one is redundant.

I also suggest per-file arch gating and runtime vendor skips.

[janeri_gmail.com]

Thank you for the thorough review.

Addressed in the updated diff:

• Moved vendor-neutral tests to tests/sys/pmc/: pmc_pidtrack_test.c, pmcstat_tsc_test.sh, pmc_callchain_ur_test.sh. The pidtrack test uses find_counting_event() for portability; the callchain-UR test retains its amd64 uname guard internally.

• Removed hwpmc_exterr_test.c -- rGe555692d1bbb already covers those cases. If any AMD/IBS cases from our version are not in the landed file we will submit them as an incremental patch to pmc_exterr_test.c separately.

• Added hwpmc_rdpmc_test.c to tests/sys/amd/pmc/ (RDPMC userspace validation: pmc_get_msr() ECX encoding, CR4.PCE grant, rdpmc vs pmc_read() agreement).

• amd_test_do_cpuid() rewritten to use do_cpuid() from <machine/cpufunc.h>; no /dev/cpuctl0, no cpuctl.ko dependency. Removed <sys/cpuctl.h>, <sys/ioctl.h>, <fcntl.h> from the header.

• Removed AMDID2_PNXC and AMDID2_PTSCEL2I redefinitions from amd_pmc_test_common.h -- both are in <machine/specialreg.h>.

• Replaced relative #include "../amd_pmc_test_common.h" with #include "amd_pmc_test_common.h" throughout; -I${SRCTOP}/tests/sys/amd in the Makefile resolves it. Same for the umcdf header.

• Removed msr_snapshot.[ch] from this diff. Those files are used by hwpmc_grouping_test, which depends on the PMC grouping kernel API currently pending upstream. We will address msr_snapshot together with the grouping tests once that API merges.

• Added product names as inline comments to amd_umcdf_map_zen() (Pinnacle Ridge, Rome, Matisse, Genoa, Turin, etc.).

Not yet addressed (pending decisions or separate diffs):

• Regarding the suggestion to colocate under tests/sys/pmc/amd/ instead of tests/sys/amd/pmc/: the AMD-specific tests (RAPL, pmu-events, IBS errata, RDPMC, grouping) are tightly coupled to AMD hardware interfaces and would be difficult to generalize. We kept the vendor-specific directory for those and moved the portable tests to tests/sys/pmc/ as suggested.

• The two umcdf/ headers are shared with the follow-on umcdf test suite. We can extract the Zen-generation decode helpers into a smaller header under amd/ if that is preferred over shipping the full umcdf header here.

[netchild]

Please update to a recent version of the main branch. You created a new pmc/Makefile, but the tip of main has one already. As-is the patch does not apply, and is not self-sufficient. It also contains a build-host hack (include dir and lib dir pointing hardcoded to /usr/local, whereas ATF should be in the base and doesn't need that. I have seen several internal Jira IDs. Not sure if you want to have that in public. And tests are skipped unless amd.pmc.grouping.runtime is set (which nothing does in this patch). umcdf/amd_umcdf_common.h still uses /dev/cpuctl0 (do_cpuid() went into amd_pmc_test_common.h only). The family/model messages and the AuthenticAMD-25-00-1 fixture names still print hex where the map now has the names.

I can understand from a vendor point of view, to have all related code together. I'm grateful that AMD is sponsoring this work. Please keep in mind that we do not have a vendor directory in sys/ and that our organisation in sys/ is functional. Having amd-only stuff in subdirectories or in the same directory as the subsystem is in my opinion a personal-taste question (at least here). So I don't care if it is pmc/amd/test1.x or pmc/amd_test1.x, but sys/amd/ looks wrong.

Reviews target a commit, and the patch here in this review provides parts which don't get used, references parts which are not included, and does not seem to build/test everything you want it to build/test. I suggest to update to a recent main branch, apply the subset of the bigger picture you want to have reviewed there, cut the diff, apply to to a fresh copy of main, and check if it does all what you want it to do (= check if it is self-contained, and complete).

[janeri_gmail.com]

Updated diff against current FreeBSD main (43347e8).

Structural change: moved everything from tests/sys/amd/ to
tests/sys/pmc/amd/ per your feedback. Patch applies cleanly
against main and is self-contained.

Also fixed in this revision:

• do_cpuid() in amd_umcdf_common.h (missed in v2)
• removed remaining internal ticket references
• tests/sys/Makefile adds _amd= amd gated on amd64, following the _pmc/_audit pattern
• tests/sys/pmc/Makefile adds SUBDIR+= amd

[netchild]

Makefiles:

• tests/sys/Makefile descends into tests/sys/amd, which this diff no longer creates.
• tests/sys/pmc/Makefile says SUBDIR+= amd. bsd.test.mk reserves SUBDIR for helper/data dirs; only TESTS_SUBDIRS gets include("amd/Kyuafile") in the parent Kyuafile, so as posted kyua test from /usr/tests are not run. And it's ungated: MK_PMC is default-on everywhere, the C tests need do_cpuid() from <machine/cpufunc.h> (amd64/i386 only) → aarch64/riscv/powerpc break. The architecture guard needs to be here.
• pmc_pidtrack_test is a .c file listed under ATF_TESTS_SH. atf.test.mk makes it depend on pmc_pidtrack_test.sh, which doesn't exist. ATF_TESTS_C is the correct one.

The three vendor-neutral tests (pidtrack, tsc, callchain-UR) that the previous patch had rightly placed in tests/sys/pmc/ came back into the vendor directory.

You have two "QA branch not installed?" texts in pmcstat_tsc_test.sh. This does not make sense here in FreeBSD.

[janeri_gmail.com]

v9: address all reviewer feedback

• pmc_pidtrack_test.c: move uint64_t out[3] declaration before any executable statement in pidtrack_ctxswitch body (KNF C89, mhorne)
• pmc_pidtrack_test.c: fork(idle)<0 error path now closes all six descriptors including busyfd[1] and startfd[0] (mhorne)
• pmc_callchain_ur_test.sh: replace private commit hashes with descriptive text (netchild)
• pmcstat_tsc_test.sh: remove empty () placeholders from comments and trim duplicate phrase (netchild)
• amd/Makefile: strip CI-only block with hardcoded developer path; this block must not appear in the upstream tree (netchild)
• pmu_mux_test.c: fix brace alignment in cleanup for-loop, tighten Zen4-only guard to zen != ZEN_4 (ali)
• rapl_test.c: reorder ATF_CHECK_MSG guard before fprintf; guard unsigned subtraction (mhorne)
• Author lines in shell scripts: full name + email format
• amd_pmc_test_common.h: add Copyright line

Tested on AMD EPYC 9654 (Zen 4, 192 logical CPUs),
FreeBSD 16.0-CURRENT HWPMC_DBG kernel:

44/51 passed, 0 failed, 0 broken, 7 skipped (all expected)

[janeri_gmail.com]

please ignore older comments.

addressed all items raised:

• tests/sys/Makefile: removed from this diff; wiring goes through tests/sys/pmc/Makefile only
• tests/sys/pmc/Makefile: SUBDIR changed to TESTS_SUBDIRS; added .if ${MACHINE_ARCH} == "amd64" guard so aarch64/riscv/powerpc are not affected
• pmc_pidtrack_test: moved from ATF_TESTS_SH to ATF_TESTS_C
• Vendor-neutral tests (pmc_pidtrack_test, pmcstat_tsc_test, pmc_callchain_ur_test) are in tests/sys/pmc/ directly
• "QA branch not installed?" text removed
• amd/Makefile: CI-only block with hardcoded developer path removed
• pmc_pidtrack_test.c: uint64_t out[3] declaration moved before any executable statement (KNF C89)
• pmc_pidtrack_test.c: fork(idle)<0 error path closes all six fds
• pmu_mux_test.c: brace alignment fixed; Zen4-only guard tightened
• rapl_test.c: ATF_CHECK_MSG guard reordered before fprintf
• Author lines in shell scripts: full name + email format

Tested on AMD EPYC 9654 (Zen 4, 192 logical CPUs),
FreeBSD 16.0-CURRENT with hwpmc(4):

44/51 passed, 0 failed, 0 broken, 7 skipped (all expected).

[paulo_nlink.com.br]

Tested Diff 187278 on an AMD Ryzen 5 5600X (Zen 3, Family 19h,
Model 21h, Stepping 0, Id 0xa20f10, 12 threads), FreeBSD
16.0-CURRENT at 10a3562a019b, kernel and world in sync.

Applying: "arc patch D59549" fails (the new files come out as
modifications of nonexistent files), but the raw diff applies
cleanly with "git apply".

Build/install issues:

1. tests/sys/pmc/pmc_pidtrack_test.c:282: kill() is undeclared; <signal.h> is not included. Fatal with -Werror.

2. tests/sys/pmc/pmc_pidtrack_test.c:386: "uint64_t out[3]" in the child branch shadows the declaration at line 357 (-Wshadow, fatal with -Werror). Since the child works on its own copy after fork(), dropping the inner declaration should be enough.

3. etc/mtree/BSD.tests.dist is not updated, so /usr/tests/sys/pmc/amd is not created and "make install" fails. installworld would fail the same way.

4. pmcinfo_thread_count is installed to /pmcinfo_thread_count (debug file to /.debug/), apparently because it has no install directory in tests/sys/pmc/amd/Makefile.

Because of 1 and 2, the vendor-neutral tests in tests/sys/pmc
were not run. For the AMD suite, after creating the directory by
hand: 51 total, 27 passed, 24 skipped, 0 failed, 0 broken.

Skips:

• 6 grouping tests: opt-in, as documented.
• 4 pmu_mux_test: Zen 4 only; expected, no CPUID Fn8000_0022 here.
• 7 pmu_event_test and 6 pmu_event_neg_test skip with "CPU is Zen 3; requires Zen4+". Zen 3 has pmu-events tables (pmccontrol -L lists ex_ret_instr, l3_lookup_state.*, etc. on this machine), so lookup by name, os/user and umask encoding, and the e2e count look applicable to Zen 3 as well. Would it be possible to gate only the Zen 4 specific cases (EventSelect[11:8])?
• pmcstat_ibs_errata_test skips with "erratum #1238 suppression not present". It uses synthetic pmclog fixtures, so the result does not depend on the host CPU: it skips because pmcstat in main lacks the suppression. The summary describes it as skipping on non-affected CPUIDs. Maybe it belongs with the tests that depend on pending changes?

Happy to rerun on Zen 3 after an update.