Committed as a935c2a63 (don't feel like reverting and updating the commit message)

Address feedback. I hope I got everything.

RANDOM_KEYBOARD and RANDOM_MOUSE can be masked away by sysctls. What is the need to remove them this way?

Address feedback from @ziaee and @0mp. Thanks!

I'd make the summary more like: dev/ofw: Register ofw_cpu xref.

In D51622#1216461, @kib wrote:

I really become curious in which way firmware parks APs after the initial configuration. Coreboot might have something for this.

And indeed, there is very interesting https://github.com/coreboot/coreboot/blob/main/src/cpu/x86/lapic/lapic_cpu_stop.c that is used to park APs. From my reading, they send INIT IPI to itself, which, by the claim in the source file, is equivalent to asserting the INIT# pin.

In D51622#1179546, @kib wrote:

The loop is only "Just in case" an interrupt comes in that's not disabled (not being an x86 guy, I don't know what's disabled or not with disable_intr() and lapic_disable()). If nothing can come in, then there's no need for the safe memory, since once it halts it's dead.

As I said below, I believe SMI is still possible even in 'cli;hlt' loop. On AMD there is a way to disable SMI and NMI, but I do not think the method to do that is usable for kexec, and there is also Intel without something equivalent.

Address @kib's feedback. kexec_do_reboot() no longer takes an argument, and hasn't even in the first commit, so remove the argument from the prototype. Removed the wbinvd from the trampoline.

Address @kib's feedback further. I didn't reproduce the problem we solved with mfence, so removed that.

Address @kib's feedback.

In D51623#1209399, @kib wrote:

I might suggest, to not delay the commit even more, simply refuse kexec for now if we are in LA57. Then somebody would work out the missing code in the trampoline later.

In D51623#1209045, @kib wrote:

It seems that you always build 4-level intermediate page table. Wouldn't it blow up if the source kernel is running in LA57 mode? [Kernel always expect LA48 on start nonetheless]

Address feedback. I hope I got it all now.

Address feedback on ioctl list.

Address feedback. I think I got it all.

Further updates. vdso maps correctly.

Address feedback from @kib and @brooks.

Address feedback. I think I got it all.

Update after latest probe changes.

Address @kib's feedback.

If it builds, it ships!

Address feedback. Added comment noting empty strings, and changed naked 0x12 to
the right constant.

In D51771#1182947, @imp wrote:

Weird, all the other devices that attach via SPCR don't need this entry. Are you sure it's still needed? I reworked things a bit ago...
Or is 0x12 some super magical thing...

In D51771#1182758, @imp wrote:

Where / how is this used?

Address feedback.

Address feedback

Address feedback. The change to vm_radix_insert, etc, is untested still (will
test in my VM shortly).

Address feedback.

Address @jhb's feedback.

In D51623#1181603, @kib wrote:

In D51623#1181602, @jhibbits wrote:

In D51623#1181601, @kib wrote:

In D51623#1181462, @jhibbits wrote:

One global question I have, most likely you must execute the handoff and start executing the new kernel on AP, not BSPs. In other words. the reboot must migrate to AP if it not already did it.

Do you mean it needs to migrate to the BSP, and not run on the APs? kern_reboot() already binds to CPU_FIRST().

If you mean it should execute from an AP, is there a reason for that?

Yes, of course I mean 'kexec must occur on BSP'. Silly thinko.

I think the sched_bind(curthread, CPU_FIRST()) in kern_reboot() should suffice, then, but correct me if I'm wrong.

When we eventually add rescue support, to run from panic, that might need to change, but would likely be a different entry, too.

Generally CPU_FIRST() is not BSP, but it happen on amd64. On x86 we define BSP as cpuid == 0.

In D51623#1181601, @kib wrote:

In D51623#1181462, @jhibbits wrote:

One global question I have, most likely you must execute the handoff and start executing the new kernel on AP, not BSPs. In other words. the reboot must migrate to AP if it not already did it.

Do you mean it needs to migrate to the BSP, and not run on the APs? kern_reboot() already binds to CPU_FIRST().

If you mean it should execute from an AP, is there a reason for that?

Yes, of course I mean 'kexec must occur on BSP'. Silly thinko.

One global question I have, most likely you must execute the handoff and start executing the new kernel on AP, not BSPs. In other words. the reboot must migrate to AP if it not already did it.

In D51620#1181458, @imp wrote:

You don't need to get this reviewed.

Abandoned in favor of the kexec stack.

In D51622#1179455, @kib wrote:

In fact, I retract my proposal with the INIT IPI. If SMI is broadcasted, other CPUs would not enter the SMI handler, and the sending CPU most likely hang waiting for the reply.

So yes, the cli;hlt loop is the best, but it should be executed from the memory which is not overwritten during kexec.

I should add that this is sort of how Linux does it as well (synchronize, disable interrupts, go catatonic and hope for the best)

In D51622#1179431, @kib wrote:

In D51622#1179429, @kib wrote:

In D51622#1179405, @jhibbits wrote:

In D51622#1179347, @kib wrote:

Why do we need this at all? Why cannot you use e.g. stop_cpus() or stop_cpus_hard() or even smp_rendezvous() to do that? It can be done in MI, and definitely does not require new IPI vector.

The purpose of this is to make the CPU go catatonic so that it requires a core reset to continue. The handler disables all interrupts before going catatonic, so it requires the LAPIC INIT sequence to restore. We don't want the CPU to execute any other instructions until reset because the memory may have been overwritten.

Then explain this, at least as a comment in the code.

In D51622#1179347, @kib wrote:

Why do we need this at all? Why cannot you use e.g. stop_cpus() or stop_cpus_hard() or even smp_rendezvous() to do that? It can be done in MI, and definitely does not require new IPI vector.