Page MenuHomeFreeBSD

hwpmc: add MPERF/APERF MSR support for AMD/Intel CPUs
AcceptedPublic

Authored by anascime_amd.com on Tue, Aug 4, 5:53 PM.
Tags
None
Referenced Files
Unknown Object (File)
Sat, Aug 22, 11:20 PM
Unknown Object (File)
Sat, Aug 22, 6:55 AM
Unknown Object (File)
Sat, Aug 22, 5:42 AM
Unknown Object (File)
Sat, Aug 22, 2:35 AM
Unknown Object (File)
Fri, Aug 21, 9:06 PM
Unknown Object (File)
Thu, Aug 20, 10:20 PM
Unknown Object (File)
Wed, Aug 19, 8:45 AM
Unknown Object (File)
Tue, Aug 18, 3:17 PM
Subscribers

Details

Summary

Add support for reading the MPERF (MSR 0xE7) and APERF (MSR 0xE8)
model-specific registers on AMD/Intel CPUs through hwpmc(4). These
counters track maximum and actual performance frequency respectively,
and are used to compute effective CPU frequency scaling independent
of the nominal TSC rate. The name of the class was chosen as PERF
because later support for other PERF MSRs can be added to the same
class.

Extend libpmc(3) to expose the AMD/Intel MPERF/APERF counters added
to hwpmc(4) in the companion kernel change, so userland consumers
(pmcstat(8), etc.) can allocate and read these events by name.

Document the new PERF class and its MPERF/APERF counters in a new
pmc.perf.3 manual page, describing their semantics and how to read
them via pmc(3) and pmcstat(8).

Signed-off-by: Anderson Nascimento <anascime@amd.com>
Sponsored by: AMD

Diff Detail

Repository
rG FreeBSD src repository
Lint
Lint Skipped
Unit
Tests Skipped
Build Status
Buildable 75609
Build 72492: arc lint + arc unit

Event Timeline

This manual looks gorgeous. The structure and narrative flow is really just delightful.

lib/libpmc/pmc.perf.3
2–4

Please consider using this format for conformance with our policy and consistency with C style comments. You can see the policy here: https://docs.freebsd.org/en/articles/license-guide/

Please do not infer this as legal advice, but it must have a license declaration or we can not use it because all rights are reserved, with or without that line -- https://en.wikipedia.org/wiki/All_rights_reserved -- therefore our style guides no longer recommend it.

7

The .Os macro shouldn't have any argument

17

Please consider using Dq, according to the reference manual on this language, mdoc(7). Qq quotes are a pain in some of the myriad output formats manpages scale to.

https://man.freebsd.org/cgi/man.cgi?query=mdoc&apropos=0&sektion=7&manpath=FreeBSD+15.1-RELEASE+and+Ports.quarterly&format=html

20

ditto

23

ditto

80–83

It might make sense to spell these out here to assist with skimming, but I'm not sure.

85–87

ditto

105

ditto

sys/dev/hwpmc/hwpmc_perf.c
2–5

We removed the hyphen on the opening line from our style guides, and the copyright line to come first when using SPDX only license declarations -- https://docs.freebsd.org/en/articles/license-guide/

Resubmit as patch 1/3: hwpmc driver changes only

Resubmit as patch 1/3: hwpmc driver changes only

Who told you to do this? It's very nice for the one logical change to be one complete atomic change. It's harder to reference everything on different pages, harder to bisect on split commits, and harder to follow review conversation.

Update: consolidate all changes into a single revision

Resubmit as patch 1/3: hwpmc driver changes only

Who told you to do this? It's very nice for the one logical change to be one complete atomic change. It's harder to reference everything on different pages, harder to bisect on split commits, and harder to follow review conversation.

That was the way another co-worker was submitting his work. I've consolidated everything here and abandoned the other revisions. I hope it's right now. I've also addressed your comments, thank you!

Thanks for bringing this back here! Just found two little typos, and then the manual looks good to me!

lib/libpmc/pmc.perf.3
2–4

Forgot the blank comment line at the top.

8

These newlines seems to have run together.

Request (if they're available) some colleagues who I regard highly w/rt MPERF/APERF MSR expertise.

I thought that APERF/MPERF are architectural, and not specific to AMD or Intel. Should this go with the arch description string and enablement based on CPU feature only, regardless of the vendor?

ali_mashtizadeh.com added inline comments.
sys/amd64/include/pmc_mdep.h
70

AMD/Intel MPERF/APERF

sys/dev/hwpmc/hwpmc_amd.c
1181

APERF/MPERF are architectural MSRs that work on both Intel and AMD. Can you add the registration to Intel as well.

sys/dev/hwpmc/hwpmc_perf.c
7

x86 or AMD/Intel

73

I'm confused the module uses priv_check already shouldn't we just concern ourselves with the USER/SYSTEM CAPS here? All system mode counters have this enforced already.

https://github.com/freebsd/freebsd-src/blob/200de1b70e2b4f809d1d3a4c430db80b24124468/sys/dev/hwpmc/hwpmc_mod.c#L3451

170

I know the naming isn't great but the purpose of this is to return the index that can be used by RDPMC for a particular performance counter. IIRC neither MPERF nor APERF are accessible using RDPMC on Intel and AMD. You should just return an error.

sys/sys/pmc.h
144

AMD/Intel or x86

This revision now requires changes to proceed.Wed, Aug 5, 4:20 PM
sys/dev/hwpmc/hwpmc_amd.c
1181

Yes, I'll do that.

sys/dev/hwpmc/hwpmc_perf.c
73

I'm confused the module uses priv_check already shouldn't we just concern ourselves with the USER/SYSTEM CAPS here? All system mode counters have this enforced already.

https://github.com/freebsd/freebsd-src/blob/200de1b70e2b4f809d1d3a4c430db80b24124468/sys/dev/hwpmc/hwpmc_mod.c#L3451

The idea is to treat those MSRs as privileged, like the RAPL case. If the sysctl knob security.bsd.unprivileged_syspmcs is set to 1, unprivileged users will have access to those MSRs, and there are known attacks that could use them to break cryptographic code, such as Hertzbleed. The operating frequency is an indirect measurement of power consumption.

Implementing it this way makes the security model stronger, I think. It allows access to the MSRs only to the root user while the sysctl security.bsd.unprivileged_syspmcs is 1. An administrator can, for example, allow access to the TSC without exposing those counters. What do you think?

Address reviewer comments: add Intel support, fix manpage typos, update
manpage and code comments to cover AMD/Intel, and make perf_get_msr()
return an error when accessed via RDPMC, since these MSRs are not
readable through that path.

anascime_amd.com retitled this revision from hwpmc: add APERF/MPERF MSR support for AMD CPUs to hwpmc: add MPERF/APERF MSR support for AMD/Intel CPUs.Mon, Aug 10, 6:08 PM
anascime_amd.com edited the summary of this revision. (Show Details)

Add missing hwpmc_perf.c and hwpmc_perf.h

Generally looks fine.

I note that this is coming shortly after the addition of the RAPL class. I do not think it is sustainable to add too many new classes each covering a couple MSRs/events. It is too much boilerplate.

It would be good to consider how these could be combined into one class, maybe even the TSC class. It is possible to provide naming aliases. I am not blocking this review based on this, but the interested parties should add it to their discussion.

sys/dev/hwpmc/hwpmc_amd.c
1104

This nclasses thing has really become an anti-pattern. IMO it could be easily reworked to be friendlier to this type of dynamic class.

Not for you to fix in this change, however :)

1153–1160

I think it is better to mimic the pattern of the RAPL class below, rather than IBS above.

That is, call pmc_perf_initialize() unconditionally, and fold the cpu_power_ecx check inside that function. If unsupported, return an error, but make this not fatal.

Basically, we don't want to abort the basic AMD counter initialization if the PERF class somehow fails.

sys/dev/hwpmc/hwpmc_perf.c
26

Perhaps?

sys/modules/hwpmc/Makefile
42

Not really needed; i386 is EOL on 16-CURRENT and 15-STABLE.

sys/sys/pmc.h
144

I'm not sure if we can reuse 0x1 yet. Better to be safe and place this at the end of the list.

Thank you for the review, @mhorne. I've addressed all the comments:

1 - Moved the cpu_power_ecx/tsc_perf_stat check inside pmc_perf_initialize() to mimic the RAPL pattern.
2 - Moved the PERF event block to 0x14800, after RAPL.
3 - Removed the i386 pmc_mdep.h include.

Additionally, I detected and fixed some minor issues during a second review:

4 - Stylistic issues (spaces, tabs).
5 - Added a comment clarifying the reasoning behind requiring PRIV_PMC_SYSTEM for the PERF class and placed it before the caps validation.
6 - Compiler annotations (diagused, unused) being misused.
7 - Typo in perf_allocate_pmc() in libpmc.c.

anascime_amd.com added inline comments.
sys/dev/hwpmc/hwpmc_amd.c
1153–1160

The only thing the pmc_perf_initialize() function does is allocating memory and initializing it. Even though there's the check, it shouldn't return error in any circumstance. I updated the code to include the validation in the perf functions.

sys/dev/hwpmc/hwpmc_perf.c
26

Done!

sys/modules/hwpmc/Makefile
42

Done! I've also removed the other i386 changes.

sys/sys/pmc.h
144

It makes sense, I moved it to be after RAPL.

Overall looks good, I mostly read the man page that I didn’t look at before and some minor nits.

lib/libpmc/pmc.perf.3
34

Sometimes trying to be too specific makes it harder to read simplify if you can.

The ratio of APERF to MPERF scaled by the processor’s nominal frequency, yields the average effective frequency.

46

Both counters expose the unmodified contents of the underlying MSRs.

50

Probably you can simplify the above text since it’s repetitive.

53

I would just turn line 53 into its own sentence and remove 47-52 to eliminate repetition.

56

3rd time

110

Last bit (after the comma) about it being read only should probably be removed it’s confusing because you only expose a read-only counter.

113
sys/amd64/include/pmc_mdep.h
45

Doesn’t p go between 44/45

sys/dev/hwpmc/hwpmc_perf.c
294

Nit move next to read

336

Nit, move write just below read

sys/modules/hwpmc/Makefile
28

I think we meant to keep these alphabetical

anascime_amd.com marked 5 inline comments as done.

I've addressed all comments

anascime_amd.com added inline comments.
lib/libpmc/pmc.perf.3
110

Reading it again, I agree! Thanks! The read-only part was really inadequate in this case.

113

The read-only bit stated in the manual was inadequate. The two conditions don't have anything to do with it. For the PERF class to be available, two conditions need to be met: the CPUID bit is advertised, and two consecutive reads return different values. As stated in the comment, the CPUID bit might be set but the counters don't increase in some scenarios, making them unfit, so the implementation depends on the tsc_perf_stat variable being 1, which tests that second condition.

With regards to priv_check If you care about side channels you can't expose any of the counters. I don't see this one as any different. In linux PERF is exposed a general facility, while you are only exposing this under hwpmc.

This revision is now accepted and ready to land.Tue, Aug 18, 10:12 PM