Page MenuHomeFreeBSD

intrng: Shuffle unhandled interrupts too
Needs ReviewPublic

Authored by cperciva on Sat, Feb 14, 12:46 AM.
Tags
None
Referenced Files
F144995874: D55284.id.diff
Sat, Feb 14, 9:02 PM
Unknown Object (File)
Sat, Feb 14, 9:07 AM
Unknown Object (File)
Sat, Feb 14, 8:03 AM
Unknown Object (File)
Sat, Feb 14, 5:59 AM
Subscribers

Details

Reviewers
andrew
mhorne
Summary

When interrupt vectors are first allocated, they get assigned to
CPU #0; at SI_SUB_SMP / SI_ORDER_SECOND (aka once we have multiple
CPUs), the intr_irq_shuffle SYSINIT clears their CPU sets with the
effect of forcing them to be assigned to new CPUs later.

In case where interrupt vectors were allocated *but not yet bound*
this code did not run, with the effect that those interrupts would
remain pinned to CPU #0 forever. This affected the ena(4) driver,
which allocates interrupts for I/O when the device is attached but
doesn't set them up until the interface is brought up much later in
the boot process (and, crucially, long after intr_irq_shuffle runs).

Adjust intr_irq_shuffle to clear the CPU set for an interrupt source
even if it currently has no handlers, so that it will be properly
assigned to a CPU when it is used later.

MFC after: 1 month
Sponsored by: Amazon

Diff Detail

Repository
rG FreeBSD src repository
Lint
Lint Skipped
Unit
Tests Skipped
Build Status
Buildable 70672
Build 67555: arc lint + arc unit

Event Timeline

I have tested in EC2 that this patch results in ena_handle_msix being called on 8 different CPUs (on this hardware we have 8 I/O queues) instead of only being called on CPU 0.

I put down a one-month MFC since this seems entirely MFCable from an interface-stability perspective but interrupt routing is the sort of thing which might take a while for problems to show up. But I'm hoping to land this in 15.1 and maybe also 14.5 (I'm not sure how much this code diverged between 14 and 15 so I'd be very happy to get feedback from experts about whether it makes sense to MFC this all the way to stable/14).