intelhfi - Intel TD/HFI driver - Part3: Add CPU core performance/efficiency score variable to SMP's cpu_group struct.

[koinec_yahoo.co.jp]

Summary

I developed the Intel Thread Director (ITD) / Hardware Feedback Interfce (HFI) device driver to obtain performance/efficiency information for each CPU core, which was implemented to improve the performance of Intel hybrid architecture CPUs. (e.g. Raptor Lake (refresh), Alder Lake, LakeField processors)

This driver simply obtains performance/efficiency information from the CPU and stores it in the "cpu_group" struct data referenced by the ULE scheduler.
However, since the ULE scheduler side is not yet supported, performance/efficiency cannot be improved by installing this driver at this time.

I will try to modify the ULE scheduler side in the future, but I posted this driver first because it can be implemented independently of this driver and it is difficult to modify the ULE scheduler.

There are seven patches, and this is the Part 3.
The ULE scheduler seems to refer to the "cpu_group" structure generated by the SMP code to determine which cores to allocate tasks to.
However, currently it seems that the performance of all cores is treated equally, so in order to be able to change this, I will add the score variable array for each CPU core group to the "cpu_group" structure.

Test Plan

With the remaining patches to be submitted later, I ran the tests listed below.

• FreeBSD 15-current: Be able to apply patches and build to the latest source tree as of 2024/03/20. After building, restart the OS and be able to start the kernel correctly.
• FreeBSD 14.0-Release: The source tree can be patched and built. After building, it is possible to load the driver and correctly obtain performance values from the CPU.

Due to my development environment, I developed it on FreeBSD 14.0-RELEASE and ported it to FreeBSD 15-current on a virtual environment.
However, I have also confirmed that the parts related to the code modified this time have hardly changed between 14.0-RELEASE and 15-current.
For this reason, I believe that it will probably work with 15-current, but if you are able to test it, please help confirm that it works.

[koinec_yahoo.co.jp]

Attach the full context patch file.

part3_add_score_variable_to_cpu_group_struct_fbsd15c.patch117 KBDownload

[koinec_yahoo.co.jp]

Attach the patch file created with the git format-patch command.

0003-Add-CPU-core-performance-efficiency-score-variable-t.patch118 KBDownload

[minsoochoo0122_proton.me]

sys/sys/smp.h

82

I assume you mean performance/efficiency cores here. How does it work when there are three types of cores including LP-E?

[minsoochoo0122_proton.me]

sys/sys/smp.h

82

LP-E will share cpuid 1Ah with E core.

[minsoochoo0122_proton.me]

sys/conf/options

641

I suggest renaming this to HMP so sched_ule code looks more consistent in future.

[koinec_yahoo.co.jp]

• Fix macro define name (CPUGRP_SCORE -> HMP)
• Rebase for freebsd current source tree.

[minsoochoo0122_proton.me]

sys/conf/options

48

This file removes bunch of options and I guess this is caused by rebasing issue. Could restore removed ones?

[minsoochoo0122_proton.me]

Could you rebase main?

sys/conf/options

48

Ignore what I said here. It's another confusion caused by Phab.

[koinec_yahoo.co.jp]

Rebase current src-tree.

Although I have rebased it, I understand that this patch will need to be integrated with https://reviews.freebsd.org/D54674 (sys:add hmp) in the future. I understand that depending on the progress of the integration work, this patch may become unnecessary.

[minsoochoo0122_proton.me]

LGTM, although we need approval from committer.

[minsoochoo0122_proton.me]

After talk with other FreeBSD dev at AsiaBSDCon, we concluded that score should be stored in pcpu structure, not in cpu_group for the following reasons:

1. cpu_group is topology structure and it is a layer for every level of cpu hierarchy (e.g. L2 cache, cores, packages, etc). This patch wouldn't work where multiple packages exist such as 2 cpu servers. Also, adding score entry for every layer is waste of memory.
2. You made a very good observation that the scheduler is oriented around cpu_group, but schedulers can also access to pcpu as well. Actually, in case like this where we need 1:1 mapping for each core, pcpu is more suitable as scheduler can make decisions based on each core not topology layer.

I know you've spent tremendous amount of time on this, but I think cpu core information should be stored in pcpu. Transitioning to pcpu isn't hard though—you can refer to my HMP patch (D54674) and add fields accordingly.

This comment also applies to D44456, of course.

[koinec_yahoo.co.jp]

Thank you for letting me know the results of discussion on the implementation policy. I understand the policy.
I will start working on the revisions now, but given my current situation, I think it will take several months. Please give me some time.