Differential D56546
hmp(4): introduce Heterogeneous MultiProcessing support Authored by mchoo on Apr 21 2026, 10:21 AM. Tags None Referenced Files
Details Add initial support for HMP. The hmp(4) framework sits between scheduler and providers. It aims to Support for controlling hardware from scheduler (e.g. Arm SCMI) will be For now, disable this option by default and enable when it becomes Sponsored by: FreeBSD Foundation Signed-off-by: Minsoo Choo <minsoochoo0122@proton.me>
Diff Detail
Event TimelineComment Actions Changes from D56547:
Comment Actions Thank you very much for improving the HMP code and creating the manual. Understood. I will proceed with supporting this version. Comment Actions
Comment Actions Thank you very much for the improvements. If the intention behind removing it is that it might not be immediately used in the ULE scheduler, I understand the reasoning. I am currently learning about the ULE scheduler while testing intelhfi. Since ULE focuses on distributing loads to the lowest-load CPUs while remaining cache-aware, I agree that the Throttled flag isn't strictly necessary for ULE itself. However, I view HMP(4) as a forward-looking mechanism designed to share scores across heterogeneous multicore architectures, rather than something optimized solely for the ULE scheduler. Also, considering that the Intel Hardware Feedback Interface defines a "Throttled" state, I assume this flag was initially introduced with Arm architecture development in mind. On the other hand, since "Capacity" represents a static capability, dynamic updates would be handled by a Provider that sets the "Score." Thus, I agree that the "Dynamic" flag can be safely removed. Additionally, the Intel Hardware Feedback Interface defines a state for notifying power-efficient cores (such as LP-E cores) to aggregate all tasks. Specifically, when the HFI table's Efficiency score reaches its maximum value of 255, it requests aggregating all tasks to that specific core to maximize battery efficiency. This is conceptually the opposite of the "Throttled" state. Comment Actions Currently the sole goal for hmp(4) is integrating with the ULE scheduler (or any other future scheduler). When I added the throttled flag, I thought it would be useful for thread placement so the scheduler can avoid placing thread to throttled core. But this comes with two drawbacks:
hmp(4) subsystem works like a greatest common divisor. In other words, it only abstracts what seems to be common across many providers. But this seems to be intelhfi specific. Moving a bulk of tasks at a time is generally avoided because thread replacement without reason is bad (e.g. cache miss, TLB miss, etc). And even without making a bulk replacement, if a core's efficiency score is high enough, the tasks should be balanced gradually over time according to the algorithm, so I don't see any reason why the aggregation needs to happen at the same time. But as you said, if it's good for battery efficiency, it's worth discussing with people interested in schedulers and laptop projects. Comment Actions Thank you very much for the explanation. I understand it much better now. I understand that "throttled" should indeed be handled by the thermal management mechanism and that it also affects performance. Based on your feedback, I have uploaded an intelhfi patch with the HMP flags removed. I would appreciate it if you could check it. Comment Actions
Comment Actions (This may not be the appropriate place to post this, but please forgive me as I don't know of any other suitable location.) As part of testing intelhfi(4), I attempted to modify sched_ule(4) to utilize hmp(4), and I was able to achieve a performance improvement of up to +10-15% in some cases. I plan to post the diff later.
I apologize for suggesting that this would involve removing part of the wonderful patch you have so kindly created. Would you be able to forgive me? All of the above modifications are synchronized with sched_ule(4), so I have also created a patch for hmp(4) addressing the above points. Therefore, I would like to post it together with this update. Finally, I am currently dealing with the above post and the advice from intelhfi(4) in response to it, so please understand that it will take me a few weeks to complete everything. Comment Actions You are always welcome to share hmp-related stuff here.
Thank you very much! I'm currently in school so I can't spend too much time working on this..... But if you are willing to help, I appreciate a lot:) Please feel free to take some time for posting the diff if you need to do so.
The concept of capacity is actually from hardware and ACPI people, not from me. At least on device tree, it accurately represents the maximum wordload a processor can do (work done per Hz * maximum frequency). However, if the diff tells that there is some compelling reasons to do so, I'm okay with it.
amdhfi uses u32 for perf/eff scores. I can check the rationale behind this in your ULE patch.
This was something I was worrying about in the design phase. I think the diff will clarify it.
If that's what we should do, I'm happy to do so.
Of course! Thank you for dedicating your time on this:)
Please don't overwrite this revision, but instead create a new one with hmp modification and add that as its child revision.
No worries. Comment Actions You should probably merge this revision with D56547. I have reviewed the merged result of this one, that one and D57939 (Koine Yuusuke's). Inline comments have been put in the relevant reviews as much as possible, but general comments are all listed below. The current name, Heterogeneous MultiProcessing, looks misleading for multiple reasons. This is a framework, without drivers nor scheduler support, and by itself does not make any real user-visible difference, although the role it fulfills is certainly necessary. The current schedulers do work on heterogeneous cores, it's just that they will make poor decision by lack of knowledge (this is different, e.g., from the SMP option, that one actually enabling running on multiple cores; "SMP" itself could be seen as a misnomer, as it supports different cores provided they report the same functional capabilities (but not necessarily the same performance); the choice of "SMP" at the time it was introduced is understandable, probably there was nothing else on the horizon back then). As you said yourself at the EuroBSDCon presentation, its usefulness goes beyond heterogeneous multiprocessing: Even when all cores are the same, it's useful to have hardware feedback on which core to schedule preferentially because some of them are throttled for thermal (and perhaps other) reasons. Finally, the fate of hybrid general purpose CPU looks uncertain, and Intel has officially announced its intention to stop producing this kind of packages. So I would think about another, more generic name, even if less "catchy". Haven't given much thought to that, off the top of my head: Hardware Data for Scheduling (HDS), which feels much more to the point with respect to what this framework really is. Proposals welcome. A priori, I would not enable the framework by default in GENERIC at the moment, that will depend a lot on how the reviews and testing go, but ultimately that's certainly the goal. By contrast, I'd put it in LINT right now. I think we should aim for a linear range of possible capacity values, with finer-grain granularity. Using the inverse of a 0-1024 range with numerator 1024 for the performance factor gives ~0.1% precision per increment near 1024, but already ~0.5% around 512 (factor 2), ~1.5% around 256 (factor 4), and already ~6.5% around 128 (factor 8). Let's make things more uniform and future proof by just using a 64-bit integer instead (still a separate type, hmp_capacity_t) whose value's meaning completely depends on the capacity provider, which is free to treat this number as fixed point instead if need be (in a similar fashion as we do with fixpt_t, but the precise scale to be used would be provider-dependent, instead of the fixed FSCALE (2^11)); I expect frequencies could be stored in there by some providers (see below). There is generally a lot of redundancy in information exported via sysctl knobs, and whether that brings much value is unclear. Also, I'd organize the tree differently, and clarify some of the exported data. All this is detailed in inline comments. The functions selecting highest/lowest capacity or the best CPU are premature. ULE's policy to select CPU is to follow the highest load on the highest loaded path, or conversely the lowest load on the lowest loaded path, and those functions require more fine-grained information than the functions here can provide. In passing, as long as capacities are unchanging, having O(n) algorithms in the scheduler would be hard to accept when there are better options. Some Koine's later revision removes them, that should be merged here. I'd like that we keep in mind that, even if it doesn't need to be the case right now (although it would likely help debug, see below), this framework should support adding/removing providers at runtime, or changing their parameters. Having atomic accessors is not enough for this to happen. The easiest way to do it, while retaining some abstraction, seems to be to have a transactional-like API, where basically you'd call some kind of prepare() function, which allocates an opaque structure on which you then call all necessary accessors, and then commit the new structure with a commit() function. The difficulty here is that the values are spread out into per-CPU structures, so a more clever strategy has to be used to have atomicity across all of them. The first coming to mind would be to have two instances of struct hmp_pcpu per-CPU, and a global index in a global variable that indicates which structure to use. prepare() would possibly initialize all the not-currently-used struct hmp_pcpu structures, setters would modify them and commit() would just update the global index. For maximal robustness, the actual scheme probably should also prevent ABA problems, if for some reason hardware updates are too fast, so what is described above is only a sketch (not going to detail it for now; you can think about what that entails and we can discuss that later). Last (for now) but not least, I'd like to point out that you're considering capacity as fixed (as the maximum capacity), but it may not be the most appropriate design. (If we were to keep it, I'd rename capacity into the more explicit maximum_capacity.) The actual capacity itself is not fixed and varies over time, depending among others on power (through P-states/frequency control or CPPC) and thermal settings/conditions, and it is this number which ultimately matters for schedulers to make decisions. We should keep bridging the frequency control/CPPC information in mind when computing the current capacity. Having the schedulers factor in these things into the equation themselves looks quite unwise. Consequently, we need a way for performance-controlling drivers (such as hwpstate_amd(4) and hwpstate_intel(4)) to change the capacity. Whether applying a multiplier to preexisting fixed maximum capacities is the most appealing mechanism remains to be seen. If, e.g., we intend to use CPU CPPC, as Koine is already trying to do in intelhfi(4), then it seems we are going to be able to immediately get the current capacity without having to compute it from a maximum capacity (and actually, I don't see where maximum capacities would come from on amd64 platforms, other than trying to compute them with some dummy workload?). I only gave a small peek at Arm PSCA/SCMI, it looks like it reports CPPC information (it may report other ones, I haven't dug enough to know). Same if we are only considering processor frequencies (when all cores are the same), we would get immediately the current capacity, and computing a multiplier to the maximum capacity appears as an unnecessary detour. Stopping here, as there is already a lot to process, and I'll be mostly AFK for a long while (~4 weeks). Feel free to ask for more details, to respond to exposed points, or perhaps raise other concerns or questions.
Comment Actions Thank you very much for the review comment above.
How about "Hardware Feedback for Scheduling (HFS)," "Processor Feedback for Scheduling (PFS)," or simply "Hardware Feedback Interface (HFI)"? Comment Actions
Everyone, why don't we consider the value ranges and their meanings in conjunction with how they are used in sched_ule(4)? I have submitted a proposal for the sched_ule(4) implementation in D57941. Could you please review it? I also agree that, from the perspective of precision, the value range could be wider than the current 0–1024. However, regarding the implementation proposed in D57941, it seems that unless we use uint64_t within sched_ule(4) - effectively making it exclusive to 64-bit architectures - I wouldn't be able to fully utilize the increased precision even if the range were expanded. I also believe that, ideally, this value should represent a capacity rather than a performance ratio. That said, unless we undertake a major overhaul of sched_ule(4), we would still need to convert it to a performance ratio for calculations, resulting in increased overhead. In my proposal (D57939), I suggested reversing the meaning of the value; that suggestion arose from the necessity I identified while experimenting with how best to handle this within sched_ule(4). One more point and this is strictly my own opinion: | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||