This addresses iflib CPU mapping oddities that were discussed on
the 20190731 iflib call.
Providing sensible defaults for, and giving the user intelligible
control over, the mapping of iflib device queues to CPUs is
complicated by the possible presence/absence of logical cores (in
addition to the physical cores), the possibility that the device
might not be able to know which cores are physical/logical, the
desire to be able to control whether logical cores are used, the
desire to be able to map TX/RX queues to CPUs in (TX,RX) pairs or
not, the desire to support unequal numbers of TX and RX queues,
and the desire to continue to support the ability to experiment
with placing TX queues on logical cores that share L2 with
corresponding RX queues.
This code provides sensible defaults for and gives the user
intelligible control over the mapping of iflib device queues to
CPUs in light of the above, and preserves all existing non-broken
behaviors (such as advancing through available CPUs across
devices when no core_offset is provided). The device sysctls that
govern CPU assignment are now:
iflib.core_offset (existing, CPU ID to begin assigning cores at)
iflib.separate_txrx (existing, whether to pair TX and RX queues on the same core, defaults to 0)
iflib.use_logical_cores (new, whether to try to assign {TX,RX} queue pairs to logical cores that share an L2 cache, defaults to 0)
See the comment on get_cpuid_for_queue() in this patch for all of
the possible behaviors.
This code also ensures that when a reference IRQ is not provided,
TX softirqs are bound to CPUs using the same allocation strategy
that would be used to allocate CPUs to TX IRQs, instead of
leaving it up to the grouptaskqueue code, as is now the case.
(The Intel drivers' practice of providing the corresponding RX IRQ as a reference when creating TX soft IRQs should be looked at, as it's not clear why this is being done, and doing so, whether with the existing iflib code or with this patch, has the effect of asking the grouptaskqueue code to rebind the RX IRQ to a different CPU than it has already been bound to at that point).