Page MenuHomeFreeBSD

amd64: Add cpu_stop() support to go UP after SMP
Needs ReviewPublic

Authored by jhibbits on Jul 29 2025, 6:56 PM.
Tags
None
Referenced Files
Unknown Object (File)
Thu, Oct 9, 5:32 PM
Unknown Object (File)
Thu, Oct 9, 5:32 PM
Unknown Object (File)
Thu, Oct 9, 3:30 PM
Unknown Object (File)
Sep 11 2025, 7:15 AM
Unknown Object (File)
Sep 6 2025, 12:30 PM
Unknown Object (File)
Sep 2 2025, 3:01 PM
Unknown Object (File)
Sep 1 2025, 11:05 AM
Unknown Object (File)
Aug 27 2025, 1:00 PM
Subscribers

Details

Diff Detail

Repository
rG FreeBSD src repository
Lint
Lint Skipped
Unit
Tests Skipped
Build Status
Buildable 65835
Build 62718: arc lint + arc unit

Event Timeline

Why do we need this at all? Why cannot you use e.g. stop_cpus() or stop_cpus_hard() or even smp_rendezvous() to do that? It can be done in MI, and definitely does not require new IPI vector.

In D51622#1179347, @kib wrote:

Why do we need this at all? Why cannot you use e.g. stop_cpus() or stop_cpus_hard() or even smp_rendezvous() to do that? It can be done in MI, and definitely does not require new IPI vector.

The purpose of this is to make the CPU go catatonic so that it requires a core reset to continue. The handler disables all interrupts before going catatonic, so it requires the LAPIC INIT sequence to restore. We don't want the CPU to execute any other instructions until reset because the memory may have been overwritten.

In D51622#1179347, @kib wrote:

Why do we need this at all? Why cannot you use e.g. stop_cpus() or stop_cpus_hard() or even smp_rendezvous() to do that? It can be done in MI, and definitely does not require new IPI vector.

The purpose of this is to make the CPU go catatonic so that it requires a core reset to continue. The handler disables all interrupts before going catatonic, so it requires the LAPIC INIT sequence to restore. We don't want the CPU to execute any other instructions until reset because the memory may have been overwritten.

Then explain this, at least as a comment in the code.

But what happens if e.g. BIOS broadcasts SMI to all cores? Wouldn't this break the core out of the halt state if the memory address is overwritten?

In D51622#1179429, @kib wrote:
In D51622#1179347, @kib wrote:

Why do we need this at all? Why cannot you use e.g. stop_cpus() or stop_cpus_hard() or even smp_rendezvous() to do that? It can be done in MI, and definitely does not require new IPI vector.

The purpose of this is to make the CPU go catatonic so that it requires a core reset to continue. The handler disables all interrupts before going catatonic, so it requires the LAPIC INIT sequence to restore. We don't want the CPU to execute any other instructions until reset because the memory may have been overwritten.

Then explain this, at least as a comment in the code.

But what happens if e.g. BIOS broadcasts SMI to all cores? Wouldn't this break the core out of the halt state if the memory address is overwritten?

I suspect that the right answer is to deliver the INIT IPI to put all other cores into the init state.

In D51622#1179431, @kib wrote:
In D51622#1179429, @kib wrote:
In D51622#1179347, @kib wrote:

Why do we need this at all? Why cannot you use e.g. stop_cpus() or stop_cpus_hard() or even smp_rendezvous() to do that? It can be done in MI, and definitely does not require new IPI vector.

The purpose of this is to make the CPU go catatonic so that it requires a core reset to continue. The handler disables all interrupts before going catatonic, so it requires the LAPIC INIT sequence to restore. We don't want the CPU to execute any other instructions until reset because the memory may have been overwritten.

Then explain this, at least as a comment in the code.

This is (crudely) explained in mp_x86.c, but I can expand on it.

But what happens if e.g. BIOS broadcasts SMI to all cores? Wouldn't this break the core out of the halt state if the memory address is overwritten?

I suspect that the right answer is to deliver the INIT IPI to put all other cores into the init state.

Do you mean that the right thing would be instead of an IPI, to send the INIT IPI for catatonia? I thought about that, but was confused reading the manual, because it looks like the BIOS can start initialization, so running code, with the INIT IPI, it doesn't put it into a wait state.

Do you mean that the right thing would be instead of an IPI, to send the INIT IPI for catatonia? I thought about that, but was confused reading the manual, because it looks like the BIOS can start initialization, so running code, with the INIT IPI, it doesn't put it into a wait state.

Yes, send INIT IPI.

Can you expand on 'BIOS can start initialization'? I do not understand this.

I should add that this is sort of how Linux does it as well (synchronize, disable interrupts, go catatonic and hope for the best)

In fact, I retract my proposal with the INIT IPI. If SMI is broadcasted, other CPUs would not enter the SMI handler, and the sending CPU most likely hang waiting for the reply.

So yes, the cli;hlt loop is the best, but it should be executed from the memory which is not overwritten during kexec.

In D51622#1179455, @kib wrote:

In fact, I retract my proposal with the INIT IPI. If SMI is broadcasted, other CPUs would not enter the SMI handler, and the sending CPU most likely hang waiting for the reply.

So yes, the cli;hlt loop is the best, but it should be executed from the memory which is not overwritten during kexec.

Hm, would it be better to add another kexec MD method for stopping all APs? Something like kexec_md_stop_aps() or such? If amd64 needs its catatonia memory to be safe then we need a (trivial) page table and page for this, so it would need to know the available memory. All this said, the new kernel could still overwrite the memory used for this, since it would be tacked on to the end of the image. The chance of that happening is pretty low.

The loop is only "Just in case" an interrupt comes in that's not disabled (not being an x86 guy, I don't know what's disabled or not with disable_intr() and lapic_disable()). If nothing can come in, then there's no need for the safe memory, since once it halts it's dead.

In D51622#1179455, @kib wrote:

In fact, I retract my proposal with the INIT IPI. If SMI is broadcasted, other CPUs would not enter the SMI handler, and the sending CPU most likely hang waiting for the reply.

So yes, the cli;hlt loop is the best, but it should be executed from the memory which is not overwritten during kexec.

Hm, would it be better to add another kexec MD method for stopping all APs? Something like kexec_md_stop_aps() or such? If amd64 needs its catatonia memory to be safe then we need a (trivial) page table and page for this, so it would need to know the available memory. All this said, the new kernel could still overwrite the memory used for this, since it would be tacked on to the end of the image. The chance of that happening is pretty low.

Might be.

The loop is only "Just in case" an interrupt comes in that's not disabled (not being an x86 guy, I don't know what's disabled or not with disable_intr() and lapic_disable()). If nothing can come in, then there's no need for the safe memory, since once it halts it's dead.

As I said below, I believe SMI is still possible even in 'cli;hlt' loop. On AMD there is a way to disable SMI and NMI, but I do not think the method to do that is usable for kexec, and there is also Intel without something equivalent.