hmp(4): introduce Heterogeneous MultiProcessing support

[minsoochoo0122_proton.me]

Summary

Add initial support for HMP.

The hmp(4) framework sits between scheduler and providers. It aims to
forward information from provider to scheduler to enable hybrid
scheduling.

Support for controlling hardware from scheduler (e.g. Arm SCMI) will be
added later.

For now, disable this option by default and enable when it becomes
stable enough.

Sponsored by: FreeBSD Foundation

Signed-off-by: Minsoo Choo <[e-mail]>

[minsoochoo0122_proton.me]

Changes from D56547:

• Moved from struct pcpu to DPCPU so we can avoid options HMP on pcpu.
• Reworked scores system. Since scheduler only cares about perf and eff scores, we don't need other capabilities.
• Capacities don't need atomic operation since it is written once on boot.
• Added sysctls.

[minsoochoo0122_proton.me]

Move sysctl

[minsoochoo0122_proton.me]

@koinec_yahoo.co.jp Could you please work on intelhfi based on this patch series?

[koinec_yahoo.co.jp]

Thank you very much for improving the HMP code and creating the manual.

Understood. I will proceed with supporting this version.

[minsoochoo0122_proton.me]

Update sysctl

[minsoochoo0122_proton.me]

• Remove dynamic and throttle flags as they are not needed anymore
• Add hmp_lowest_capacity_cpu()

[koinec_yahoo.co.jp]

In D56546#1306730, @minsoochoo0122_proton.me wrote:

• Remove dynamic and throttle flags as they are not needed anymore

Thank you very much for the improvements.
Is it possible to retain the flag field and the definition of the "Throttled" flag?

If the intention behind removing it is that it might not be immediately used in the ULE scheduler, I understand the reasoning. I am currently learning about the ULE scheduler while testing intelhfi. Since ULE focuses on distributing loads to the lowest-load CPUs while remaining cache-aware, I agree that the Throttled flag isn't strictly necessary for ULE itself.

However, I view HMP(4) as a forward-looking mechanism designed to share scores across heterogeneous multicore architectures, rather than something optimized solely for the ULE scheduler. Also, considering that the Intel Hardware Feedback Interface defines a "Throttled" state, I assume this flag was initially introduced with Arm architecture development in mind.
Therefore, notification of the "Throttled" state could become essential in the future for managing heterogeneous multicore environments. For this reason, I would like to request keeping the Throttled state configurable rather than removing it.

On the other hand, since "Capacity" represents a static capability, dynamic updates would be handled by a Provider that sets the "Score." Thus, I agree that the "Dynamic" flag can be safely removed.

Additionally, the Intel Hardware Feedback Interface defines a state for notifying power-efficient cores (such as LP-E cores) to aggregate all tasks. Specifically, when the HFI table's Efficiency score reaches its maximum value of 255, it requests aggregating all tasks to that specific core to maximize battery efficiency. This is conceptually the opposite of the "Throttled" state.
To support this behavior, would it be possible to add a "Consolidated" flag to the flag field?

[minsoochoo0122_proton.me]

In D56546#1306763, @koinec_yahoo.co.jp wrote:

In D56546#1306730, @minsoochoo0122_proton.me wrote:

• Remove dynamic and throttle flags as they are not needed anymore

Thank you very much for the improvements.
Is it possible to retain the flag field and the definition of the "Throttled" flag?

If the intention behind removing it is that it might not be immediately used in the ULE scheduler, I understand the reasoning. I am currently learning about the ULE scheduler while testing intelhfi. Since ULE focuses on distributing loads to the lowest-load CPUs while remaining cache-aware, I agree that the Throttled flag isn't strictly necessary for ULE itself.

However, I view HMP(4) as a forward-looking mechanism designed to share scores across heterogeneous multicore architectures, rather than something optimized solely for the ULE scheduler. Also, considering that the Intel Hardware Feedback Interface defines a "Throttled" state, I assume this flag was initially introduced with Arm architecture development in mind.
Therefore, notification of the "Throttled" state could become essential in the future for managing heterogeneous multicore environments. For this reason, I would like to request keeping the Throttled state configurable rather than removing it.

Currently the sole goal for hmp(4) is integrating with the ULE scheduler (or any other future scheduler). When I added the throttled flag, I thought it would be useful for thread placement so the scheduler can avoid placing thread to throttled core. But this comes with two drawbacks:

1. Assume we have two core arm64 board and buildworld (or do any heavy work) on it. Then both cores are likely to be flagged as throttled. The scheduler won't assign thread to ether core, but that's impossible. Otherwise, the scheduler will conclude that all cores are throttled and place the thread at a random core, but then we wasted O(n) time to make that conclusion. (O(n) for 2c board won't be a huge issue, but imagine this is happening to a 256c server)
2. Throttled should be a part of thermal subsystem like cpufreq. For Arm specifically, I believe writing cpufreq_scmi driver would be better for handling throttled flag. You might ask what if we control the cores through hmp(4), but then we have two subsystems (cpufreq, hmp) for same work which introduces another layer of uncertainty.

Additionally, the Intel Hardware Feedback Interface defines a state for notifying power-efficient cores (such as LP-E cores) to aggregate all tasks. Specifically, when the HFI table's Efficiency score reaches its maximum value of 255, it requests aggregating all tasks to that specific core to maximize battery efficiency. This is conceptually the opposite of the "Throttled" state.
To support this behavior, would it be possible to add a "Consolidated" flag to the flag field?

hmp(4) subsystem works like a greatest common divisor. In other words, it only abstracts what seems to be common across many providers. But this seems to be intelhfi specific.

Moving a bulk of tasks at a time is generally avoided because thread replacement without reason is bad (e.g. cache miss, TLB miss, etc). And even without making a bulk replacement, if a core's efficiency score is high enough, the tasks should be balanced gradually over time according to the algorithm, so I don't see any reason why the aggregation needs to happen at the same time. But as you said, if it's good for battery efficiency, it's worth discussing with people interested in schedulers and laptop projects.

[koinec_yahoo.co.jp]

Thank you very much for the explanation. I understand it much better now.

I understand that "throttled" should indeed be handled by the thermal management mechanism and that it also affects performance.
Also, regarding "Consolidated" flag as you pointed out, I also recognize it as a feature specific to Intel HFI.

Based on your feedback, I have uploaded an intelhfi patch with the HMP flags removed. I would appreciate it if you could check it.

[minsoochoo0122_proton.me]

• total_capacity should have type hmp_capacity_t
• hmp_capacity_t is now uint32_t since total_capacity will overflow when there are 64 or more cores. hmp_score_t has also changed to uint32_t to reflect the change.