Page MenuHomeFreeBSD

Fix an occasional hang in smp_after_idle_runnable
AbandonedPublic

Authored by mhorne on Sep 20 2020, 10:13 PM.

Details

Reviewers
jhb
jrtc27
markj
Summary

Okay, here's a fun one.

In smp_after_idle_runnable() we execute the following loop to ensure
APs enter the scheduler before freeing their boot stack:

pc = pcpu_find(cpu);
while (atomic_load_ptr(&pc->pc_curpcb) == NULL)
    cpu_spinwait();
kmem_free((vm_offset_t)bootstacks[cpu], PAGE_SIZE);

Strangely, the resulting assembly for this loop is:

ffffffc00044dbe6:   ld      a0,32(a0)
ffffffc00044dbe8:   bnez    a0,ffffffc00044dbaa <smp_after_idle_runnable+0x26>
ffffffc00044dbea:   j       ffffffc00044dbea <smp_after_idle_runnable+0x66>

Which will spin forever if pc->pc_curpcb == NULL on the first check.

This usually doesn't happen as APs enter the scheduler quickly, relative
to the BSP entering smp_after_idle_runnable. However, I've noticed that
when booting with loader(8) and -smp 2, the secondary processor will
sometimes get interrupted to handle disk I/O before it is finished in
init_secondary(), which increases the window for this to occur.

Our definition of cpu_spinwait() on RISC-V is empty, so as a
workaround defining it as a single nop instruction seems to fix the
issue.

I have two follow-up questions to this:

Is this a compiler bug, or is this empty while loop abusing some
undefined behaviour? I'll note that gcc doesn't emit the same sequence,
but so far I've had trouble replicating this as a smaller test case.

Second, would it be desirable to ensure that APs don't receive external
interrupts until they have finished init_secondary()? I can't name
an explicit harm from allowing this, but it definitely caused me some
confusion while debugging. The reason it is currently possible is that
intr_irq_shuffle() executes at SI_ORDER_SECOND, while
smp_after_idle_runnable() executes at SI_ORDER_ANY.

Diff Detail

Lint
Lint OK
Unit
No Unit Test Coverage
Build Status
Buildable 33676
Build 30915: arc lint + arc unit