x86: Round robin disabled interrupt sources as well

[cem]

Summary

Once all processor cores are started and active interrupts have been spread
amongst them, spread out any interrupts which have been allocated but not
yet activated as well. This ensures that drivers which allocate interrupt
vectors at boot time but don't assign handlers until they're actually opened
will have their interrupts distributed too.

This was encountered with a 40Gbps NIC, delivering all interrupts to core 0.

Submitted by: Anton Rang <anton.rang AT isilon.com>
Sponsored by: Dell EMC Isilon

[kib]

Why it requires the separate loop with exactly the same loop body under the condition ?

[jhb]

This is unnecessary on stock FreeBSD. Interrupt vectors are assigned a round-robin CPU on first use. See x86/x86/ioapic.c:ioapic_enable_intr(). Also, your stated use case can't possibly be affected by this as multiple interrupts for a multi-queue device are MSI interrupts which are round-robin assigned to CPUs when they are first allocated via msi_alloc() or msix_alloc(). The only interrupts this loop would affect are the non-MSI IRQs that are all created in response to enumerating PICs (8259As and I/O APICs).

[cem]

In D12149#252290, @kib wrote:

Why it requires the separate loop with exactly the same loop body under the condition ?

To round-robin enabled sources first. The allocation would be different if the is_handlers condition was just removed from the first loop instead.

In D12149#252338, @jhb wrote:

This is unnecessary on stock FreeBSD. Interrupt vectors are assigned a round-robin CPU on first use. See x86/x86/ioapic.c:ioapic_enable_intr(). Also, your stated use case can't possibly be affected by this as multiple interrupts for a multi-queue device are MSI interrupts which are round-robin assigned to CPUs when they are first allocated via msi_alloc() or msix_alloc(). The only interrupts this loop would affect are the non-MSI IRQs that are all created in response to enumerating PICs (8259As and I/O APICs).

Anton, maybe you can address this?

[cem]

Attempted to upstream code I didn't author; I don't know why this was deemed necessary for OneFS and didn't hear back from Isilonians. Dropping.