Enable interrupts on the BSP once all PICs are initialized.

[jhb]

Summary

Enable interrupts on the BSP once all PICs are initialized.

This moves the enabling of interrupts slightly earlier (the old location
was still before devices were enumerated and probed) and does it in the
interrupt code (rather than in the device configuration code). This
also avoids tripping over an assertion on the first TLB shootdown with
earlier AP startup.

Test Plan

• booted in kernels both with and without early AP startup enabled in both VMs and real hardware

[imp]

Seems sane to me.

Do we need code on other platforms though to do something similar? this is an area I'd think we'd want to keep as uniform as possible across the stable of architectures we have.

[kib]

So the goal is to avoid the

	if (!(read_eflags() & PSL_I))
		panic("%s: interrupts disabled", __func__);

with earlier TLB shootdown ? I am fine with this.

As a side note, TLB shootdowns are meaningless until APs are started in scheduler, and probably cannot even work. E.g., the interrupt handlers must send EOI, but we only configure LAPIC, including properly setting x2APIC mode, only after the APs are unfrozen. We even do not configure MTRRs before that point, which probably have 'interesting' consequences for early TLB shootdowns.

APs implicitely flush TLB on the route to full operation after unreeze.

[jhb]

In D5710#121985, @imp wrote:

Seems sane to me.

Do we need code on other platforms though to do something similar? this is an area I'd think we'd want to keep as uniform as possible across the stable of architectures we have.

Well, this part of startup is typically MD. However, I would expect other platforms might need a similar change as they switch to EARLY_AP_STARTUP. I don't think it can be done in a centralized place. OTOH, doing it in SI_SUB_INTR is probably more consistent overall.

[jhb]

In D5710#122059, @kib wrote:

So the goal is to avoid the

	if (!(read_eflags() & PSL_I))
		panic("%s: interrupts disabled", __func__);

with earlier TLB shootdown ? I am fine with this.

Correct.

As a side note, TLB shootdowns are meaningless until APs are started in scheduler, and probably cannot even work. E.g., the interrupt handlers must send EOI, but we only configure LAPIC, including properly setting x2APIC mode, only after the APs are unfrozen. We even do not configure MTRRs before that point, which probably have 'interesting' consequences for early TLB shootdowns.

APs implicitely flush TLB on the route to full operation after unreeze.

Yes, we already conditionalize TLB shootdowns on 'smp_started' which isn't set until the APs are out of the pen. However, in the pending patches to move AP unpenning earlier (before device probe), the APs are now unpenned and running before the old SYSINIT ran which triggered the panic.