Why do we need this at all? Why cannot you use e.g. stop_cpus() or stop_cpus_hard() or even smp_rendezvous() to do that? It can be done in MI, and definitely does not require new IPI vector.

In D51622#1179347, @kib wrote:

Why do we need this at all? Why cannot you use e.g. stop_cpus() or stop_cpus_hard() or even smp_rendezvous() to do that? It can be done in MI, and definitely does not require new IPI vector.

The purpose of this is to make the CPU go catatonic so that it requires a core reset to continue. The handler disables all interrupts before going catatonic, so it requires the LAPIC INIT sequence to restore. We don't want the CPU to execute any other instructions until reset because the memory may have been overwritten.

In D51622#1179405, @jhibbits wrote:

In D51622#1179347, @kib wrote:

Why do we need this at all? Why cannot you use e.g. stop_cpus() or stop_cpus_hard() or even smp_rendezvous() to do that? It can be done in MI, and definitely does not require new IPI vector.

The purpose of this is to make the CPU go catatonic so that it requires a core reset to continue. The handler disables all interrupts before going catatonic, so it requires the LAPIC INIT sequence to restore. We don't want the CPU to execute any other instructions until reset because the memory may have been overwritten.

Then explain this, at least as a comment in the code.

But what happens if e.g. BIOS broadcasts SMI to all cores? Wouldn't this break the core out of the halt state if the memory address is overwritten?

In D51622#1179429, @kib wrote:

In D51622#1179405, @jhibbits wrote:

In D51622#1179347, @kib wrote:

Why do we need this at all? Why cannot you use e.g. stop_cpus() or stop_cpus_hard() or even smp_rendezvous() to do that? It can be done in MI, and definitely does not require new IPI vector.

The purpose of this is to make the CPU go catatonic so that it requires a core reset to continue. The handler disables all interrupts before going catatonic, so it requires the LAPIC INIT sequence to restore. We don't want the CPU to execute any other instructions until reset because the memory may have been overwritten.

Then explain this, at least as a comment in the code.

But what happens if e.g. BIOS broadcasts SMI to all cores? Wouldn't this break the core out of the halt state if the memory address is overwritten?

I suspect that the right answer is to deliver the INIT IPI to put all other cores into the init state.

In D51622#1179431, @kib wrote:

In D51622#1179429, @kib wrote:

In D51622#1179405, @jhibbits wrote:

In D51622#1179347, @kib wrote:

Why do we need this at all? Why cannot you use e.g. stop_cpus() or stop_cpus_hard() or even smp_rendezvous() to do that? It can be done in MI, and definitely does not require new IPI vector.

The purpose of this is to make the CPU go catatonic so that it requires a core reset to continue. The handler disables all interrupts before going catatonic, so it requires the LAPIC INIT sequence to restore. We don't want the CPU to execute any other instructions until reset because the memory may have been overwritten.

Then explain this, at least as a comment in the code.

This is (crudely) explained in mp_x86.c, but I can expand on it.

But what happens if e.g. BIOS broadcasts SMI to all cores? Wouldn't this break the core out of the halt state if the memory address is overwritten?

I suspect that the right answer is to deliver the INIT IPI to put all other cores into the init state.

Do you mean that the right thing would be instead of an IPI, to send the INIT IPI for catatonia? I thought about that, but was confused reading the manual, because it looks like the BIOS can start initialization, so running code, with the INIT IPI, it doesn't put it into a wait state.

In D51622#1179439, @jhibbits wrote:

Do you mean that the right thing would be instead of an IPI, to send the INIT IPI for catatonia? I thought about that, but was confused reading the manual, because it looks like the BIOS can start initialization, so running code, with the INIT IPI, it doesn't put it into a wait state.

Yes, send INIT IPI.

Can you expand on 'BIOS can start initialization'? I do not understand this.

I should add that this is sort of how Linux does it as well (synchronize, disable interrupts, go catatonic and hope for the best)

In fact, I retract my proposal with the INIT IPI. If SMI is broadcasted, other CPUs would not enter the SMI handler, and the sending CPU most likely hang waiting for the reply.

So yes, the cli;hlt loop is the best, but it should be executed from the memory which is not overwritten during kexec.

In D51622#1179455, @kib wrote:

In fact, I retract my proposal with the INIT IPI. If SMI is broadcasted, other CPUs would not enter the SMI handler, and the sending CPU most likely hang waiting for the reply.

So yes, the cli;hlt loop is the best, but it should be executed from the memory which is not overwritten during kexec.

Hm, would it be better to add another kexec MD method for stopping all APs? Something like kexec_md_stop_aps() or such? If amd64 needs its catatonia memory to be safe then we need a (trivial) page table and page for this, so it would need to know the available memory. All this said, the new kernel could still overwrite the memory used for this, since it would be tacked on to the end of the image. The chance of that happening is pretty low.

The loop is only "Just in case" an interrupt comes in that's not disabled (not being an x86 guy, I don't know what's disabled or not with disable_intr() and lapic_disable()). If nothing can come in, then there's no need for the safe memory, since once it halts it's dead.

In D51622#1179456, @jhibbits wrote:

In D51622#1179455, @kib wrote:

In fact, I retract my proposal with the INIT IPI. If SMI is broadcasted, other CPUs would not enter the SMI handler, and the sending CPU most likely hang waiting for the reply.

So yes, the cli;hlt loop is the best, but it should be executed from the memory which is not overwritten during kexec.

Hm, would it be better to add another kexec MD method for stopping all APs? Something like kexec_md_stop_aps() or such? If amd64 needs its catatonia memory to be safe then we need a (trivial) page table and page for this, so it would need to know the available memory. All this said, the new kernel could still overwrite the memory used for this, since it would be tacked on to the end of the image. The chance of that happening is pretty low.

Might be.

The loop is only "Just in case" an interrupt comes in that's not disabled (not being an x86 guy, I don't know what's disabled or not with disable_intr() and lapic_disable()). If nothing can come in, then there's no need for the safe memory, since once it halts it's dead.

As I said below, I believe SMI is still possible even in 'cli;hlt' loop. On AMD there is a way to disable SMI and NMI, but I do not think the method to do that is usable for kexec, and there is also Intel without something equivalent.