kexec: Introduce basic arm64 support
ClosedPublic
Actions

Authored by jhibbits on Jul 29 2025, 6:55 PM.

Details

Reviewers

imp
markj
jhb
kib
jmg
stevek
andrew
manu

Commits

rG1ae25866767d: kexec: Introduce basic arm64 support

Summary

This works on older arm64 platforms, but may not work with arm64 devices
using GICv3, due to a quirk in the GICv3, where some registers are
write-once.

Most of the kexec reboot work on arm64 can be done entirely in C code,
by disabling the MMU, as the kernel is carved out of the vm_phys_segs
array, so cannot be overwritten.

Diff Detail

Repository

rG FreeBSD src repository

Lint

Lint Skipped

Unit

Tests Skipped

Build Status

Buildable 65834
Build 62717: arc lint + arc unit

Event Timeline

jhibbits created this revision.Jul 29 2025, 6:55 PM

Herald added a reviewer: andrew. · View Herald TranscriptJul 29 2025, 6:55 PM

Herald added a reviewer: andrew. · View Herald Transcript

Herald added a reviewer: manu. · View Herald Transcript

Herald added a subscriber: emaste. · View Herald Transcript

jhibbits requested review of this revision.Jul 29 2025, 6:55 PM

Harbormaster completed remote builds in B65834: Diff 159418.Jul 29 2025, 6:55 PM

jhibbits added a parent revision: D51620: syscalls: Regen after adding kexec_load syscall.Jul 29 2025, 6:55 PM

jhibbits added a child revision: D51622: amd64: Add cpu_stop() support to go UP after SMP.

Do you plan to return to EL2 on systems that booted from it but don't have VHE?

sys/arm64/arm64/kexec_support.c
140	Do you need a `dsb` here to ensure any prior stores to the image have completed?
143–144	Why are you invalidating the cache? It looks like this is the image we are about to branch to, so I would expect it be a `cpu_dcache_wbinv_range`.
sys/arm64/arm64/locore.S
475	This can be a `br`
476	Do we need a BTI target here?
482	Could also be a `br` as the next instruction isn't useful
sys/arm64/arm64/mp_machdep.c
396	Does this need `__dead2`?

jhibbits added inline comments.Jul 30 2025, 1:54 PM

sys/arm64/arm64/kexec_support.c
140	I don't think so. I already did the cpu_dcache_wbinv_range() (via cpu_dcache_sync) at load time, so it's stable now (wired memory, so can't have paged out).
143–144	I copy the image with the cache disabled, so I need to invalidate the cache to make sure anyone else participating in the cache protocol invalidates their copy (empirically it looks like the CPUs participate in the protocol even when their cache is disabled).
sys/arm64/arm64/locore.S
476	Probably.
482	I ran into problems testing on my M3 MBP, where it faulted because apparently `br` would only work with x16, or such like that (it's been a few months). Pointer authentication or something.

andrew added inline comments.Jul 30 2025, 3:00 PM

sys/arm64/arm64/kexec_support.c
140	I assume you mean `cpu_flush_dcache`.
180–185	You probably don't need to set `ATTR_S1_GP`, it's for BTI.
sys/arm64/arm64/locore.S
482	It will be BTI related, if the target has the `GP` bit in the page table it will need the correct bti instruction based on the caller. `br x16` or `br x17` is compatible with both `j` and `c` instructions.
sys/arm64/arm64/machdep.c
315	Based on the above comment, and original commit message (rGdbb95048da3a) this is stronger than is needed by this function. It might be better to create a new cache function for kexec.

jhibbits added inline comments.Jul 30 2025, 6:59 PM

sys/arm64/arm64/kexec_support.c
140	Of course. ECAFFEINE (and going by uncaffeinated memory..)
sys/arm64/arm64/machdep.c
315	Can you elaborate? This function is supposed to guarantee that the data is flushed out to memory before it returns; my reading of cpu_dcache_wb_range() is that it does this very thing.

andrew added inline comments.Jul 31 2025, 9:42 AM

sys/arm64/arm64/machdep.c
315	To synchronise the D and I caches on arm64 the most we need to do is write back to the point of unification. `cpu_dcache_wb_range` will write back to the point of coherency which I expect to be further from the CPU. If `CTR_EL0.IDC` is set then no D-cache operation is needed to make the I-cache coherent so nothing would be needed here.

jhibbits added inline comments.Aug 4 2025, 6:35 PM

sys/arm64/arm64/machdep.c
315	Thinking on this, the name `cpu_flush_cache()` implies that once the function is complete the contents of the cache are flushed out to main memory, while pmap_sync_icache() implies that its purpose is to synchronize the D and I caches,. So just from the name cpu_flush_cache() provides a stronger guarantee of state at return.