Paths

Table of Contentst

amdrapl: Introduce amdrapl driver
AbandonedPublic
Actions

Authored by afscoelho_gmail.com on Mar 1 2026, 11:21 AM.

Details

Reviewers

aokblast
ali_mashtizadeh.com
imp
mhorne

Summary

amdrapl exposes per-package and per-core energy from AMD RAPL MSRs via sysctl. Sampled on demand at read time. Self-disarming overflow guard bounds 32-bit counter wrap during read silence. Suspend/resume re-baselines counters so reset folds no spurious delta into totals.

Power in milliwatts over measured interval. Fine-grained energy values privileged-only: power side channel (PLATYPUS, CVE-2020-8694).

Binds to physical sockets/cores via cpu_get_pkg_id() and cpu_get_core_id() -> socket-scoped or SMT-shared MSR sampled once per real domain. Requires SMP. Wired into NOTES and module build.

Sponsored by: Advanced Micro Devices (AMD)
MFC after: 2 weeks
Co-authored-by: ShengYi Hung <aokblast@FreeBSD.org>
Signed-off-by: Andre Silva <andasilv@amd.com>

Diff Detail

Repository

rG FreeBSD src repository

Lint

Lint Passed

Unit

No Test Coverage

Build Status

Buildable 71111
Build 67994: arc lint + arc unit

Event Timeline

aokblast created this revision.Mar 1 2026, 11:21 AM

Herald added a subscriber: imp. · View Herald TranscriptMar 1 2026, 11:21 AM

aokblast requested review of this revision.Mar 1 2026, 11:21 AM

Harbormaster completed remote builds in B71110: Diff 172960.Mar 1 2026, 11:21 AM

Support numa node report

Harbormaster completed remote builds in B71111: Diff 172962.Mar 1 2026, 12:01 PM

Quick response:

also need to add to sys/modules/Makefile
Do you have a plan to write a short man page?

Fix Makefile and use recursive lock to make witness happy

Harbormaster completed remote builds in B71128: Diff 172997.Mar 2 2026, 8:34 AM

In D55603#1272290, @lwhsu wrote:

Quick response:

also need to add to sys/modules/Makefile

Do you have a plan to write a short man page?

Sure, but I am thinking about creating a uniform monitor interface for kernel sensors that can also be used by amdtemp and coretemp driver. So the sysctl may chages.

Hi @aokblast, I've found your rapl driver for amd and I'm interested in continuing the work. It's been sitting since March and I'd like to pick it up. Are you still planning to work on it, or would you be OK with me taking over? Thanks

In D55603#1311570, @afscoelho_gmail.com wrote:

Hi @aokblast, I've found your rapl driver for amd and I'm interested in continuing the work. It's been sitting since March and I'd like to pick it up. Are you still planning to work on it, or would you be OK with me taking over? Thanks

Hi, according to the PPR. This is all we can do for the Zen5 pre CPUs. The mailbox based features require AMD AI series CPU or Zen5 EPYC, which I am unable to test and debug. So don't hesitate to take this task.

Notice: I don't know whether we will merge this feature as RAPL has some known security flaws.

Hi @aokblast, thanks for starting amdrapl!. I've been working on top of this driver and would like to carry the review forward to land. Would you be willing to reassign this revision to me, or add me so I can arc update it? Happy to keep you as co-author on the commit (I've added a Co-authored-by: trailer crediting you). I did work that covers on-demand energy sampling and socket-aware package mapping for example

afscoelho_gmail.com commandeered this revision from aokblast.Jun 9 2026, 1:10 PM

afscoelho_gmail.com added a reviewer: aokblast.

Hi @aokblast, just a heads-up that I've taken ownership of this revision to carry it forward. Thanks for laying the groundwork with the initial driver; I'll credit you with a Co-authored-by: trailer on the commit. I'll be pushing an updated diff shortly. Feel free to stay subscribed, your feedback is welcome!

afscoelho_gmail.com added a subscriber: ali_mashtizadeh.com.Jun 9 2026, 1:24 PM

Correct stack onto D55603 and add Co-authored-by trailer for ShengYi Hung.

Harbormaster completed remote builds in B74130: Diff 180345.Tue, Jun 23, 2:50 AM

Correcting the stack: this driver update continues D55603 and depends on the prerequisite AMD RAPL MSR definitions in D57759.

afscoelho_gmail.com mentioned this in D57760: amdrapl: add AMD RAPL energy/power telemetry driver.Tue, Jun 23, 2:50 AM

afscoelho_gmail.com edited the summary of this revision. (Show Details)

afscoelho_gmail.com added subscribers: mhorne, alfredo, raghavendra.kt_amd.com.Tue, Jun 23, 2:57 AM

afscoelho_gmail.com added a subscriber: pmc.

janeri_gmail.com added a subscriber: janeri_gmail.com.Tue, Jun 23, 9:33 AM

Just a summary of this driver interface.

 Telemetry sysctls:

 - dev.amd_rapl.0.package_mwatt
   Comma-separated package/socket power, in milliwatts, averaged over the most recent sampling interval.

 - dev.amd_rapl.0.cores_mwatt
   Comma-separated per-physical-core power, in milliwatts, averaged over the most recent sampling interval.

 - dev.amd_rapl.0.package_energy_uj
   Comma-separated cumulative package/socket energy, in microjoules.

- dev.amd_rapl.0.cores_energy_uj
   Comma-separated cumulative per-physical-core energy, in microjoules.

 Status/configuration sysctls:

 - dev.amd_rapl.0.package_energy_lapsed
   Comma-separated 0/1 values. A 1 means the package counter had a sampling gap longer than the wrap-safe interval, so package_energy_uj may have missed one or more wraps.

 - dev.amd_rapl.0.cores_energy_lapsed
   Same as above, for per-core counters.

 - dev.amd_rapl.0.energy_unit
   Hardware RAPL energy unit shift. One raw counter unit equals 1 / 2^energy_unit joules.

 - dev.amd_rapl.0.max_energy_uj
   Microjoules represented by the maximum 32-bit hardware energy counter value.

 - dev.amd_rapl.0.force_guard
   Runtime/tunable boolean. 1 keeps the overflow guard sampler always armed; 0 lets it self-disarm after readers go idle.

 - dev.amd_rapl.0.idle_guard_mult
   Runtime/tunable multiplier controlling how long the overflow guard stays armed after the last read.

 Example usage:

 Load the module:

     kldload amdrapl

 List all amdrapl sysctls:

     sysctl dev.amd_rapl.0

 Read package power:

     sysctl dev.amd_rapl.0.package_mwatt

 Read only the raw comma-separated package power values:

     sysctl -n dev.amd_rapl.0.package_mwatt

 Read per-core power:

     sysctl -n dev.amd_rapl.0.cores_mwatt

 Read cumulative package energy:

     sysctl -n dev.amd_rapl.0.package_energy_uj

 Poll package power once per second:

     while :; do sysctl -n dev.amd_rapl.0.package_mwatt; sleep 1; done

 Keep the overflow guard always enabled:

     sysctl dev.amd_rapl.0.force_guard=1

 Return to demand-gated sampling:

     sysctl dev.amd_rapl.0.force_guard=0

 Check whether a counter may have lapsed while idle:

     sysctl dev.amd_rapl.0.package_energy_lapsed
     sysctl dev.amd_rapl.0.cores_energy_lapsed

 The power and energy telemetry sysctls require privilege;

Note that package power readings are strongly affected by CPU idle state.

On my test machine, idle package power was around 138 W. Splitting into package and summed per-core power showed:

package = 138 W
cores   =  31 W   (sum of all 80 physical cores)
uncore  = 107 W   (package - cores)

The system was staying in a shallow ACPI idle state. With hw.acpi.cpu.cx_lowest at C1, package power stayed high. Letting the CPUs enter C2 dropped it to about 27 W:

sysctl hw.acpi.cpu.cx_lowest=C2 hw.acpi.cpu.cx_lowest: C1 -> C2
while :; do sysctl -n dev.amd_rapl.0.package_mwatt | \ awk -F, '{printf "\rpkg: %.1f W ", $1 / 1000}'; sleep 1; done pkg: 27.1 W

alfredo added inline comments.Wed, Jun 24, 7:22 PM

sys/dev/amdrapl/amdrapl.c
162	Is it missing an 'else' to set 'val->lapsed = false' ?
247	Is `mtx_unlock` here intentional?

Just to follow up with our call can you quickly layout the plan for integrating with:

hwpmc
You don't need to implement it but are we going to use one driver for Intel/AMD? I'd like the hwpmc counters exposed in a unified way like Linux does.

ali_mashtizadeh.com requested changes to this revision.Thu, Jun 25, 4:55 PM

ali_mashtizadeh.com added inline comments.

sys/dev/amdrapl/amdrapl.c
577	Shouldn't we expose each core counter separately? Particularly with big-little we need to see how much power we're using on a per core basis.

This revision now requires changes to proceed.Thu, Jun 25, 4:55 PM

ali_mashtizadeh.com added inline comments.Thu, Jun 25, 4:57 PM

sys/dev/amdrapl/amdrapl.c
509	Wouldn't it be simpler to structure the code to use x86_msr_op rather than rolling your own logic to read MSRs on specific cores? I think you could eliminate a bunch of the code and dance around the mutex/dying/etc.

In D55603#1326261, @ali_mashtizadeh.com wrote:

Just to follow up with our call can you quickly layout the plan for integrating with:

hwpmc

You don't need to implement it but are we going to use one driver for Intel/AMD? I'd like the hwpmc counters exposed in a unified way like Linux does.

one thing is that these RAPL MSR's don't fit clen in a allocate/start/stop programmable model, they are always "on", I know that in linux "power" is just another PMU uniformly accessed by perf, it just seems that we are forcing something here. what you think? The closest thing in hwpmc for a free-running counter is the TSC class, so a RAPL class would look TSC-like, not K8 like.

About th Intel support, it is definitely possible, I think the natural shape is a shared MI core + vendor specific MSR`s.

sys/dev/amdrapl/amdrapl.c
162	It is intended this way. 'lapse' is a flag: once it records that data was lost (because of the UINT32_MAX limit of the energy counter), it keeps reporting "data was lost" forever, on purpose, because the cumulative energy total really did lose information and stays wrong for the rest of the session. It is initialized as false during driver attach.
247	yes. The callout enters here with sc->mtx already held by the callout subsystem. We have to unlock it before smp_rendezvous_cpus (no rendezvous under the lock), then re-acquire later to check if dying/reschedule and return unlocked
509	The problem is that x86_msr_op reads just one value (*res in the signature) and if the op runs on more than one CPU, "it is unspecified on which CPU the value was read.". When sampling the energy values, we need different values at once, one energy reading per core and per package, otherwise x86_msr_op would have to be executed N times, and each time would execute a rendezvous. The current code does the opposite: just one smp_rendezvous_cpus where each core runs the handler and writes into its own slot, sc->core_value[sc->cpu_core_slot[curcpu]]. About the mutex/dying, they are not in the read path, they exist just to control the 32-bit overflow guard
577	The data is already per-physical-core, cores_mwatt/cores_energy_uj are comma-separated, one field per physical core (SMT siblings fuse into their core's slot). For example: $ sudo sysctl dev.amd_rapl.0.cores_mwatt dev.amd_rapl.0.cores_mwatt: 228,220,225,30,30,30,32,213,396,394,398,397,30,387,30,386,30,30,246,258,247,30,249,247,30,31,347,346,339,340,30,30,30,304,30,30,46,300,310,28,30,260,269,30,3022,257,30,257,30,247,370,381,238,331,30,246,217,31,30,220,214,30,30,212,30,202,201,199,30,193,27,194,342,341,336,336,144,219,333,30 It is possible to use big.little, do you think it is better?

Also take a look at the Intel Fixed function counters IAF that are also always present. Start/stop are nops.

Superseded by an hwpmc-integrated implementation, per review feedback to integrate RAPL into hwpmc(4) rather than ship a standalone driver. The new work is a three-part stack:

D58027: x86: add AMD RAPL MSR definitions to specialreg.h
D58028: hwpmc: add RAPL energy-counter class (AMD + Intel)
D58029: libpmc: userland support and pmc.rapl.3 for the RAPL class

Abandoning this revision in favor of that stack.