Page MenuHomeFreeBSD

powernv: Include NUMA locality information in the CPU topology
ClosedPublic

Authored by markj on Feb 10 2021, 9:11 PM.

Details

Summary

This helps ULE make locality-preserving scheduling decisions. To make
use of the generic subr_smp.c topology-building routes, I have to assume
that all NUMA domains are the same size and that the set of CPUs
belonging to any domain is contiguous.

Test Plan

I don't have a NUMA POWER system to test on. Would anyone be willing to try it and show output from the kern.sched.topology_spec sysctl?

Diff Detail

Repository
R10 FreeBSD src repository
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

markj requested review of this revision.Feb 10 2021, 9:11 PM
markj added reviewers: jhibbits, bdragon, luporl.

On a dual socket POWER9:

<groups>
 <group level="1" cache-level="0">
  <cpu count="144" mask="ffffffffffffffff,ffffffffffffffff,ffff,0">0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143</cpu>
  <children>
   <group level="2" cache-level="3">
    <cpu count="72" mask="ffffffffffffffff,ff,0,0">0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71</cpu>
    <children>
     <group level="3" cache-level="1">
      <cpu count="4" mask="f,0,0,0">0, 1, 2, 3</cpu>
      <flags><flag name="THREAD">THREAD group</flag><flag name="SMT">SMT group</flag></flags>
     </group>
     <group level="3" cache-level="1">
      <cpu count="4" mask="f0,0,0,0">4, 5, 6, 7</cpu>
      <flags><flag name="THREAD">THREAD group</flag><flag name="SMT">SMT group</flag></flags>
     </group>
     <group level="3" cache-level="1">
      <cpu count="4" mask="f00,0,0,0">8, 9, 10, 11</cpu>
      <flags><flag name="THREAD">THREAD group</flag><flag name="SMT">SMT group</flag></flags>
     </group>
     <group level="3" cache-level="1">
      <cpu count="4" mask="f000,0,0,0">12, 13, 14, 15</cpu>
      <flags><flag name="THREAD">THREAD group</flag><flag name="SMT">SMT group</flag></flags>
     </group>
     <group level="3" cache-level="1">
      <cpu count="4" mask="f0000,0,0,0">16, 17, 18, 19</cpu>
      <flags><flag name="THREAD">THREAD group</flag><flag name="SMT">SMT group</flag></flags>
     </group>
     <group level="3" cache-level="1">
      <cpu count="4" mask="f00000,0,0,0">20, 21, 22, 23</cpu>
      <flags><flag name="THREAD">THREAD group</flag><flag name="SMT">SMT group</flag></flags>
     </group>
     <group level="3" cache-level="1">
      <cpu count="4" mask="f000000,0,0,0">24, 25, 26, 27</cpu>
      <flags><flag name="THREAD">THREAD group</flag><flag name="SMT">SMT group</flag></flags>
     </group>
     <group level="3" cache-level="1">
      <cpu count="4" mask="f0000000,0,0,0">28, 29, 30, 31</cpu>
      <flags><flag name="THREAD">THREAD group</flag><flag name="SMT">SMT group</flag></flags>
     </group>
     <group level="3" cache-level="1">
      <cpu count="4" mask="f00000000,0,0,0">32, 33, 34, 35</cpu>
      <flags><flag name="THREAD">THREAD group</flag><flag name="SMT">SMT group</flag></flags>
     </group>
     <group level="3" cache-level="1">
      <cpu count="4" mask="f000000000,0,0,0">36, 37, 38, 39</cpu>
      <flags><flag name="THREAD">THREAD group</flag><flag name="SMT">SMT group</flag></flags>
     </group>
     <group level="3" cache-level="1">
      <cpu count="4" mask="f0000000000,0,0,0">40, 41, 42, 43</cpu>
      <flags><flag name="THREAD">THREAD group</flag><flag name="SMT">SMT group</flag></flags>
     </group>
     <group level="3" cache-level="1">
      <cpu count="4" mask="f00000000000,0,0,0">44, 45, 46, 47</cpu>
      <flags><flag name="THREAD">THREAD group</flag><flag name="SMT">SMT group</flag></flags>
     </group>
     <group level="3" cache-level="1">
      <cpu count="4" mask="f000000000000,0,0,0">48, 49, 50, 51</cpu>
      <flags><flag name="THREAD">THREAD group</flag><flag name="SMT">SMT group</flag></flags>
     </group>
     <group level="3" cache-level="1">
      <cpu count="4" mask="f0000000000000,0,0,0">52, 53, 54, 55</cpu>
      <flags><flag name="THREAD">THREAD group</flag><flag name="SMT">SMT group</flag></flags>
     </group>
     <group level="3" cache-level="1">
      <cpu count="4" mask="f00000000000000,0,0,0">56, 57, 58, 59</cpu>
      <flags><flag name="THREAD">THREAD group</flag><flag name="SMT">SMT group</flag></flags>
     </group>
     <group level="3" cache-level="1">
      <cpu count="4" mask="f000000000000000,0,0,0">60, 61, 62, 63</cpu>
      <flags><flag name="THREAD">THREAD group</flag><flag name="SMT">SMT group</flag></flags>
     </group>
     <group level="3" cache-level="1">
      <cpu count="4" mask="0,f,0,0">64, 65, 66, 67</cpu>
      <flags><flag name="THREAD">THREAD group</flag><flag name="SMT">SMT group</flag></flags>
     </group>
     <group level="3" cache-level="1">
      <cpu count="4" mask="0,f0,0,0">68, 69, 70, 71</cpu>
      <flags><flag name="THREAD">THREAD group</flag><flag name="SMT">SMT group</flag></flags>
     </group>
    </children>
   </group>
   <group level="2" cache-level="3">
    <cpu count="72" mask="0,ffffffffffffff00,ffff,0">72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143</cpu>
    <children>
     <group level="3" cache-level="1">
      <cpu count="4" mask="0,f00,0,0">72, 73, 74, 75</cpu>
      <flags><flag name="THREAD">THREAD group</flag><flag name="SMT">SMT group</flag></flags>
     </group>
     <group level="3" cache-level="1">
      <cpu count="4" mask="0,f000,0,0">76, 77, 78, 79</cpu>
      <flags><flag name="THREAD">THREAD group</flag><flag name="SMT">SMT group</flag></flags>
     </group>
     <group level="3" cache-level="1">
      <cpu count="4" mask="0,f0000,0,0">80, 81, 82, 83</cpu>
      <flags><flag name="THREAD">THREAD group</flag><flag name="SMT">SMT group</flag></flags>
     </group>
     <group level="3" cache-level="1">
      <cpu count="4" mask="0,f00000,0,0">84, 85, 86, 87</cpu>
      <flags><flag name="THREAD">THREAD group</flag><flag name="SMT">SMT group</flag></flags>
     </group>
     <group level="3" cache-level="1">
      <cpu count="4" mask="0,f000000,0,0">88, 89, 90, 91</cpu>
      <flags><flag name="THREAD">THREAD group</flag><flag name="SMT">SMT group</flag></flags>
     </group>
     <group level="3" cache-level="1">
      <cpu count="4" mask="0,f0000000,0,0">92, 93, 94, 95</cpu>
      <flags><flag name="THREAD">THREAD group</flag><flag name="SMT">SMT group</flag></flags>
     </group>
     <group level="3" cache-level="1">
      <cpu count="4" mask="0,f00000000,0,0">96, 97, 98, 99</cpu>
      <flags><flag name="THREAD">THREAD group</flag><flag name="SMT">SMT group</flag></flags>
     </group>
     <group level="3" cache-level="1">
      <cpu count="4" mask="0,f000000000,0,0">100, 101, 102, 103</cpu>
      <flags><flag name="THREAD">THREAD group</flag><flag name="SMT">SMT group</flag></flags>
     </group>
     <group level="3" cache-level="1">
      <cpu count="4" mask="0,f0000000000,0,0">104, 105, 106, 107</cpu>
      <flags><flag name="THREAD">THREAD group</flag><flag name="SMT">SMT group</flag></flags>
     </group>
     <group level="3" cache-level="1">
      <cpu count="4" mask="0,f00000000000,0,0">108, 109, 110, 111</cpu>
      <flags><flag name="THREAD">THREAD group</flag><flag name="SMT">SMT group</flag></flags>
     </group>
     <group level="3" cache-level="1">
      <cpu count="4" mask="0,f000000000000,0,0">112, 113, 114, 115</cpu>
      <flags><flag name="THREAD">THREAD group</flag><flag name="SMT">SMT group</flag></flags>
     </group>
     <group level="3" cache-level="1">
      <cpu count="4" mask="0,f0000000000000,0,0">116, 117, 118, 119</cpu>
      <flags><flag name="THREAD">THREAD group</flag><flag name="SMT">SMT group</flag></flags>
     </group>
     <group level="3" cache-level="1">
      <cpu count="4" mask="0,f00000000000000,0,0">120, 121, 122, 123</cpu>
      <flags><flag name="THREAD">THREAD group</flag><flag name="SMT">SMT group</flag></flags>
     </group>
     <group level="3" cache-level="1">
      <cpu count="4" mask="0,f000000000000000,0,0">124, 125, 126, 127</cpu>
      <flags><flag name="THREAD">THREAD group</flag><flag name="SMT">SMT group</flag></flags>
     </group>
     <group level="3" cache-level="1">
      <cpu count="4" mask="0,0,f,0">128, 129, 130, 131</cpu>
      <flags><flag name="THREAD">THREAD group</flag><flag name="SMT">SMT group</flag></flags>
     </group>
     <group level="3" cache-level="1">
      <cpu count="4" mask="0,0,f0,0">132, 133, 134, 135</cpu>
      <flags><flag name="THREAD">THREAD group</flag><flag name="SMT">SMT group</flag></flags>
     </group>
     <group level="3" cache-level="1">
      <cpu count="4" mask="0,0,f00,0">136, 137, 138, 139</cpu>
      <flags><flag name="THREAD">THREAD group</flag><flag name="SMT">SMT group</flag></flags>
     </group>
     <group level="3" cache-level="1">
      <cpu count="4" mask="0,0,f000,0">140, 141, 142, 143</cpu>
      <flags><flag name="THREAD">THREAD group</flag><flag name="SMT">SMT group</flag></flags>
     </group>
    </children>
   </group>
  </children>
 </group>
</groups>
This revision is now accepted and ready to land.Feb 10 2021, 11:14 PM

Since jhibbits tried for Talos II already, I will try on TYAN POWER8 instead.

On a deca-core POWER8 (SMT-8):

<groups>
 <group level="1" cache-level="0">
  <cpu count="80" mask="ffffffffffffffff,ffff,0,0">0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79</cpu>
  <children>
   <group level="2" cache-level="1">
    <cpu count="8" mask="ff,0,0,0">0, 1, 2, 3, 4, 5, 6, 7</cpu>
    <flags><flag name="THREAD">THREAD group</flag><flag name="SMT">SMT group</flag></flags>
   </group>
   <group level="2" cache-level="1">
    <cpu count="8" mask="ff00,0,0,0">8, 9, 10, 11, 12, 13, 14, 15</cpu>
    <flags><flag name="THREAD">THREAD group</flag><flag name="SMT">SMT group</flag></flags>
   </group>
   <group level="2" cache-level="1">
    <cpu count="8" mask="ff0000,0,0,0">16, 17, 18, 19, 20, 21, 22, 23</cpu>
    <flags><flag name="THREAD">THREAD group</flag><flag name="SMT">SMT group</flag></flags>
   </group>
   <group level="2" cache-level="1">
    <cpu count="8" mask="ff000000,0,0,0">24, 25, 26, 27, 28, 29, 30, 31</cpu>
    <flags><flag name="THREAD">THREAD group</flag><flag name="SMT">SMT group</flag></flags>
   </group>
   <group level="2" cache-level="1">
    <cpu count="8" mask="ff00000000,0,0,0">32, 33, 34, 35, 36, 37, 38, 39</cpu>
    <flags><flag name="THREAD">THREAD group</flag><flag name="SMT">SMT group</flag></flags>
   </group>
   <group level="2" cache-level="1">
    <cpu count="8" mask="ff0000000000,0,0,0">40, 41, 42, 43, 44, 45, 46, 47</cpu>
    <flags><flag name="THREAD">THREAD group</flag><flag name="SMT">SMT group</flag></flags>
   </group>
   <group level="2" cache-level="1">
    <cpu count="8" mask="ff000000000000,0,0,0">48, 49, 50, 51, 52, 53, 54, 55</cpu>
    <flags><flag name="THREAD">THREAD group</flag><flag name="SMT">SMT group</flag></flags>
   </group>
   <group level="2" cache-level="1">
    <cpu count="8" mask="ff00000000000000,0,0,0">56, 57, 58, 59, 60, 61, 62, 63</cpu>
    <flags><flag name="THREAD">THREAD group</flag><flag name="SMT">SMT group</flag></flags>
   </group>
   <group level="2" cache-level="1">
    <cpu count="8" mask="0,ff,0,0">64, 65, 66, 67, 68, 69, 70, 71</cpu>
    <flags><flag name="THREAD">THREAD group</flag><flag name="SMT">SMT group</flag></flags>
   </group>
   <group level="2" cache-level="1">
    <cpu count="8" mask="0,ff00,0,0">72, 73, 74, 75, 76, 77, 78, 79</cpu>
    <flags><flag name="THREAD">THREAD group</flag><flag name="SMT">SMT group</flag></flags>
   </group>
  </children>
 </group>
</groups>

One possible problem with your assumptions is that arbitray cores may be GUARDed out for a given boot. So some cores may exist but be offline due to being deconfig'd during IPL. I'm not sure whether that would have an effect on this or not. The IPL logic will always deconfig the entire core, and therefore the entire thread/smt group when there is a fault entry. So the domains might have differing numbers of online cores.

One possible problem with your assumptions is that arbitray cores may be GUARDed out for a given boot. So some cores may exist but be offline due to being deconfig'd during IPL. I'm not sure whether that would have an effect on this or not. The IPL logic will always deconfig the entire core, and therefore the entire thread/smt group when there is a fault entry. So the domains might have differing numbers of online cores.

When is this done in practice?

I'm not a huge fan of assuming that all domains have an identical number of cores, this assumption is just needed to use the somewhat simplistic smp_topo_*() functions. I'll revisit this and see if I can handle asymmetric domains without having to complicate the code too much.

On a deca-core POWER8 (SMT-8):

Presumably this is not a NUMA system? Or should we be seeing an additional layer in the topology?

One possible problem with your assumptions is that arbitray cores may be GUARDed out for a given boot. So some cores may exist but be offline due to being deconfig'd during IPL. I'm not sure whether that would have an effect on this or not. The IPL logic will always deconfig the entire core, and therefore the entire thread/smt group when there is a fault entry. So the domains might have differing numbers of online cores.

When is this done in practice?

I'm not a huge fan of assuming that all domains have an identical number of cores, this assumption is just needed to use the somewhat simplistic smp_topo_*() functions. I'll revisit this and see if I can handle asymmetric domains without having to complicate the code too much.

The most common way for this to happen is on POWER9 DD2.2, where driver bugs can cause an erroneous checksum fault in the load/store unit when trying to mix a CI mapping with a non-CI mapping for the same physical page. The RAS recovery code assumes that it was a problem with the CPU hardware and takes the core that the driver had been running on out of service until the fault is manually cleared.

We currently warn on boot when the system is in that state but allow the boot to continue.

On a deca-core POWER8 (SMT-8):

Presumably this is not a NUMA system? Or should we be seeing an additional layer in the topology?

It's a single-ram-domain system, yeah. (although the ram is in a fanout hierarchy behind the Centaur buffer chips, I think it still counts as uniform access.) I mainly posted it because I wanted to verify that the 8-way multithreading was being represented correctly.

Stop assuming that domain masks are contiguous and symmetric.

We still assume that adjacent CPU IDs correspond to sibling threads.

This revision now requires review to proceed.Feb 25 2021, 4:12 PM

I did a full test run with the updated version of this patch in (among others) and it worked fine.

Here's my output again.

If you need me to test with a noncontig set, I can probably crash my machine on purpose and guard a core out. (I'd rather not if it's not needed though)

Another thing about the guard behavior -- it works a core at a time, so all 4 threads (for POWER9) will get removed at the same time.

Also, this has a dependency on bd94c8ab29 -- without that commit, powernv was not properly propagating the associativity information for the second domain.

kern.sched.topology_spec: <groups>
 <group level="1" cache-level="0">
  <cpu count="144" mask="ffffffffffffffff,ffffffffffffffff,ffff,0">0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143</cpu>
  <children>
   <group level="2" cache-level="3">
    <cpu count="72" mask="ffffffffffffffff,ff,0,0">0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71</cpu>
    <children>
     <group level="3" cache-level="1">
      <cpu count="4" mask="f,0,0,0">0, 1, 2, 3</cpu>
      <flags><flag name="THREAD">THREAD group</flag><flag name="SMT">SMT group</flag></flags>
     </group>
     <group level="3" cache-level="1">
      <cpu count="4" mask="f0,0,0,0">4, 5, 6, 7</cpu>
      <flags><flag name="THREAD">THREAD group</flag><flag name="SMT">SMT group</flag></flags>
     </group>
     <group level="3" cache-level="1">
      <cpu count="4" mask="f00,0,0,0">8, 9, 10, 11</cpu>
      <flags><flag name="THREAD">THREAD group</flag><flag name="SMT">SMT group</flag></flags>
     </group>
     <group level="3" cache-level="1">
      <cpu count="4" mask="f000,0,0,0">12, 13, 14, 15</cpu>
      <flags><flag name="THREAD">THREAD group</flag><flag name="SMT">SMT group</flag></flags>
     </group>
     <group level="3" cache-level="1">
      <cpu count="4" mask="f0000,0,0,0">16, 17, 18, 19</cpu>
      <flags><flag name="THREAD">THREAD group</flag><flag name="SMT">SMT group</flag></flags>
     </group>
     <group level="3" cache-level="1">
      <cpu count="4" mask="f00000,0,0,0">20, 21, 22, 23</cpu>
      <flags><flag name="THREAD">THREAD group</flag><flag name="SMT">SMT group</flag></flags>
     </group>
     <group level="3" cache-level="1">
      <cpu count="4" mask="f000000,0,0,0">24, 25, 26, 27</cpu>
      <flags><flag name="THREAD">THREAD group</flag><flag name="SMT">SMT group</flag></flags>
     </group>
     <group level="3" cache-level="1">
      <cpu count="4" mask="f0000000,0,0,0">28, 29, 30, 31</cpu>
      <flags><flag name="THREAD">THREAD group</flag><flag name="SMT">SMT group</flag></flags>
     </group>
     <group level="3" cache-level="1">
      <cpu count="4" mask="f00000000,0,0,0">32, 33, 34, 35</cpu>
      <flags><flag name="THREAD">THREAD group</flag><flag name="SMT">SMT group</flag></flags>
     </group>
     <group level="3" cache-level="1">
      <cpu count="4" mask="f000000000,0,0,0">36, 37, 38, 39</cpu>
      <flags><flag name="THREAD">THREAD group</flag><flag name="SMT">SMT group</flag></flags>
     </group>
     <group level="3" cache-level="1">
      <cpu count="4" mask="f0000000000,0,0,0">40, 41, 42, 43</cpu>
      <flags><flag name="THREAD">THREAD group</flag><flag name="SMT">SMT group</flag></flags>
     </group>
     <group level="3" cache-level="1">
      <cpu count="4" mask="f00000000000,0,0,0">44, 45, 46, 47</cpu>
      <flags><flag name="THREAD">THREAD group</flag><flag name="SMT">SMT group</flag></flags>
     </group>
     <group level="3" cache-level="1">
      <cpu count="4" mask="f000000000000,0,0,0">48, 49, 50, 51</cpu>
      <flags><flag name="THREAD">THREAD group</flag><flag name="SMT">SMT group</flag></flags>
     </group>
     <group level="3" cache-level="1">
      <cpu count="4" mask="f0000000000000,0,0,0">52, 53, 54, 55</cpu>
      <flags><flag name="THREAD">THREAD group</flag><flag name="SMT">SMT group</flag></flags>
     </group>
     <group level="3" cache-level="1">
      <cpu count="4" mask="f00000000000000,0,0,0">56, 57, 58, 59</cpu>
      <flags><flag name="THREAD">THREAD group</flag><flag name="SMT">SMT group</flag></flags>
     </group>
     <group level="3" cache-level="1">
      <cpu count="4" mask="f000000000000000,0,0,0">60, 61, 62, 63</cpu>
      <flags><flag name="THREAD">THREAD group</flag><flag name="SMT">SMT group</flag></flags>
     </group>
     <group level="3" cache-level="1">
      <cpu count="4" mask="0,f,0,0">64, 65, 66, 67</cpu>
      <flags><flag name="THREAD">THREAD group</flag><flag name="SMT">SMT group</flag></flags>
     </group>
     <group level="3" cache-level="1">
      <cpu count="4" mask="0,f0,0,0">68, 69, 70, 71</cpu>
      <flags><flag name="THREAD">THREAD group</flag><flag name="SMT">SMT group</flag></flags>
     </group>
    </children>
   </group>
   <group level="2" cache-level="3">
    <cpu count="72" mask="0,ffffffffffffff00,ffff,0">72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143</cpu>
    <children>
     <group level="3" cache-level="1">
      <cpu count="4" mask="0,f00,0,0">72, 73, 74, 75</cpu>
      <flags><flag name="THREAD">THREAD group</flag><flag name="SMT">SMT group</flag></flags>
     </group>
     <group level="3" cache-level="1">
      <cpu count="4" mask="0,f000,0,0">76, 77, 78, 79</cpu>
      <flags><flag name="THREAD">THREAD group</flag><flag name="SMT">SMT group</flag></flags>
     </group>
     <group level="3" cache-level="1">
      <cpu count="4" mask="0,f0000,0,0">80, 81, 82, 83</cpu>
      <flags><flag name="THREAD">THREAD group</flag><flag name="SMT">SMT group</flag></flags>
     </group>
     <group level="3" cache-level="1">
      <cpu count="4" mask="0,f00000,0,0">84, 85, 86, 87</cpu>
      <flags><flag name="THREAD">THREAD group</flag><flag name="SMT">SMT group</flag></flags>
     </group>
     <group level="3" cache-level="1">
      <cpu count="4" mask="0,f000000,0,0">88, 89, 90, 91</cpu>
      <flags><flag name="THREAD">THREAD group</flag><flag name="SMT">SMT group</flag></flags>
     </group>
     <group level="3" cache-level="1">
      <cpu count="4" mask="0,f0000000,0,0">92, 93, 94, 95</cpu>
      <flags><flag name="THREAD">THREAD group</flag><flag name="SMT">SMT group</flag></flags>
     </group>
     <group level="3" cache-level="1">
      <cpu count="4" mask="0,f00000000,0,0">96, 97, 98, 99</cpu>
      <flags><flag name="THREAD">THREAD group</flag><flag name="SMT">SMT group</flag></flags>
     </group>
     <group level="3" cache-level="1">
      <cpu count="4" mask="0,f000000000,0,0">100, 101, 102, 103</cpu>
      <flags><flag name="THREAD">THREAD group</flag><flag name="SMT">SMT group</flag></flags>
     </group>
     <group level="3" cache-level="1">
      <cpu count="4" mask="0,f0000000000,0,0">104, 105, 106, 107</cpu>
      <flags><flag name="THREAD">THREAD group</flag><flag name="SMT">SMT group</flag></flags>
     </group>
     <group level="3" cache-level="1">
      <cpu count="4" mask="0,f00000000000,0,0">108, 109, 110, 111</cpu>
      <flags><flag name="THREAD">THREAD group</flag><flag name="SMT">SMT group</flag></flags>
     </group>
     <group level="3" cache-level="1">
      <cpu count="4" mask="0,f000000000000,0,0">112, 113, 114, 115</cpu>
      <flags><flag name="THREAD">THREAD group</flag><flag name="SMT">SMT group</flag></flags>
     </group>
     <group level="3" cache-level="1">
      <cpu count="4" mask="0,f0000000000000,0,0">116, 117, 118, 119</cpu>
      <flags><flag name="THREAD">THREAD group</flag><flag name="SMT">SMT group</flag></flags>
     </group>
     <group level="3" cache-level="1">
      <cpu count="4" mask="0,f00000000000000,0,0">120, 121, 122, 123</cpu>
      <flags><flag name="THREAD">THREAD group</flag><flag name="SMT">SMT group</flag></flags>
     </group>
     <group level="3" cache-level="1">
      <cpu count="4" mask="0,f000000000000000,0,0">124, 125, 126, 127</cpu>
      <flags><flag name="THREAD">THREAD group</flag><flag name="SMT">SMT group</flag></flags>
     </group>
     <group level="3" cache-level="1">
      <cpu count="4" mask="0,0,f,0">128, 129, 130, 131</cpu>
      <flags><flag name="THREAD">THREAD group</flag><flag name="SMT">SMT group</flag></flags>
     </group>
     <group level="3" cache-level="1">
      <cpu count="4" mask="0,0,f0,0">132, 133, 134, 135</cpu>
      <flags><flag name="THREAD">THREAD group</flag><flag name="SMT">SMT group</flag></flags>
     </group>
     <group level="3" cache-level="1">
      <cpu count="4" mask="0,0,f00,0">136, 137, 138, 139</cpu>
      <flags><flag name="THREAD">THREAD group</flag><flag name="SMT">SMT group</flag></flags>
     </group>
     <group level="3" cache-level="1">
      <cpu count="4" mask="0,0,f000,0">140, 141, 142, 143</cpu>
      <flags><flag name="THREAD">THREAD group</flag><flag name="SMT">SMT group</flag></flags>
     </group>
    </children>
   </group>
  </children>
 </group>
</groups>
This revision is now accepted and ready to land.Mar 31 2021, 4:22 AM

Thanks for re-testing. I had tested it on my blackbird but it has only one socket populated.