Page MenuHomeFreeBSD

amd64: allow parallel shootdown IPIs
Needs ReviewPublic

Authored by kib on Mon, Jun 29, 9:51 PM.



Stop using smp_ipi_mtx to protect global shootdown state, and move the global state into pcpu. Now each CPU can initiate shootdown IPI independently from other CPUs. Initiator enters critical section, then fills its local PCPU shootdown info (pc_smp_tlb_XXX), then clears scoreboard generation at location (cpu, my_cpuid) for each target cpu. After that IPI is sent to all targets which scan for zeroed scoreboard generation words. Upon finding such word the shootdown data is read from corresponding cpu pcpu, and generation is set. Meantime initiator loops waiting for all zeroed generations in scoreboard to update.

Initiator does not disable interrupts, which should allow non-invalidation IPIs from deadlocking.

Handlers loop until they do not see zeroed scoreboard generations. This, together with one pending IPI in LAPIC IRR should prevent lost shootdowns.


  1. The generation is set before the actual invalidation is performed in handler. It is safe because target CPU cannot return to userspace before handler finishes. In principle only NMI can preempt the handler, but NMI would see the kernel handler frame and not touch not-invalidated user page table.
  1. The code does touch LAPIC ICR without exclusion. I believe this is fine because we in fact do not send IPIs from interrupt handlers. More for !x2APIC mode where ICR access for write requires two registers write, we disable interrupts around it. If considered incorrect, I can add per-cpu spinlock around ipi_send().
  1. Scoreboard lines owned by given target CPU can be padded to the cache line, to reduce ping-pong. This is not done in the prototype.

Diff Detail

rS FreeBSD src repository
Lint Skipped
Unit Tests Skipped
Build Status
Buildable 32046

Event Timeline

kib created this revision.Mon, Jun 29, 9:51 PM
kib requested review of this revision.Mon, Jun 29, 9:51 PM
kib edited the summary of this revision. (Show Details)Mon, Jun 29, 10:01 PM
kib edited the summary of this revision. (Show Details)Mon, Jun 29, 10:04 PM
kib edited the summary of this revision. (Show Details)
emaste added a subscriber: emaste.Tue, Jun 30, 2:35 PM