arm64: Make SMP `spin-table` boot method work for more than Apple Silicon

[jhibbits]

Summary

The spin-table AP boot method on arm64 uses a single shared release
address, which the BSP sets to the entry point. This is in contrast to
other platforms, such as powerpc, which utilize a separate release
address for each core, containing a multi-word descriptor. This
difference therefore requires that the AP holdoff be managed entirely by
the OS. Support this by adding a new ap_cpuid variable, holding the
64-bit cpu device tree reg value, which should map to the mpidr_el1,
masked out to the CPU affinity bits. With this, we also need a new
entry point to spin until the it's the given core's turn to let go.

Sponsored by: Juniper Networks, Inc.

[jhibbits]

sys/arm64/arm64/mp_machdep.c

119

I didn't put this into machdep.h or cpu.h because it's not a stable variable, its only purpose is AP hold-off, holding the "current" AP in bringup.

[andrew]

sys/arm64/arm64/locore.S

209

952–959

This is in the .init_pagetable section where you probably don't want it. It might be better to just move it to mp_machdep.c.

sys/arm64/arm64/mp_machdep.c

427

Are these needed? cpu_flush_dcache is a nop on arm64 so should probably be cpu_dcache_wb_range if they are.

[jhibbits]

sys/arm64/arm64/mp_machdep.c

427

You're right. Changing that.

[andrew]

sys/arm64/arm64/locore.S

209

Should be ldr x2, =CPU_AFF_MASK

[jhibbits]

Address feedback.

[andrew]

I've thought about this and think we should only read ap_cpuid after the MMU is enabled. The same physical address should be mapped with the same memory type for all virtual addresses, however this isn't possible for ap_cpuid when the MMU is disabled.

This might need a new wfe/sev section to ensure the CPU is running.