amd64: implement support for disabling splitlocks

[kib]

Summary

If hw supports it, on atomic operation that requires exclusive ownership of more than one cache line, #AC is generated.
Thread can control it with sysarch(I386_SET_SPLITLOCK). The global default is set with hw.splitlock_force.

[kib]

Test program I used with simics:

/* $Id: splitlock.c,v 1.5 2026/09/18 16:39:53 kostik Exp kostik $ */

#include <sys/param.h>
#include <sys/mman.h>
#include <machine/atomic.h>
#include <machine/sysarch.h>
#include <err.h>
#include <signal.h>
#include <stdlib.h>
#include <stdio.h>
#include <string.h>

#ifndef I386_SET_SPLITLOCK
#define	I386_SET_SPLITLOCK	14
#define	I386_GET_SPLITLOCK	15
#endif

static void
sigbus_handler(int signo, siginfo_t *si, void *mc)
{
	printf("sig%d si_code %d addr %p\n", signo, si->si_code, si->si_addr);
	exit(0);
}

int
main(void)
{
	char *m;
	struct sigaction sa;
	int error, val;

	val = 1;
	error = sysarch(I386_SET_SPLITLOCK, &val);
	if (error == -1)
		err(1, "I386_SET_SPLITLOCK");
	error = sysarch(I386_GET_SPLITLOCK, &val);
	if (error == -1)
		err(1, "I386_GET_SPLITLOCK");
	printf("setlock: %d\n", val);

	memset(&sa, 0, sizeof(sa));
	sa.sa_sigaction = sigbus_handler;
	sa.sa_flags = SA_SIGINFO;
	if (sigaction(SIGBUS, &sa, NULL) == -1)
		err(1, "sigaction");

	m = mmap(NULL, PAGE_SIZE, PROT_READ | PROT_WRITE,
	    MAP_PRIVATE | MAP_ANON, -1, 0);
	if (m == MAP_FAILED)
		err(1, "mmap");

	m += 256 - 1;
	atomic_fetchadd_long((unsigned long *)m, 1);
}

[kib]

Fix reversed handling of the msr bit on context switch.

[kib]

Fix splitlock control on exec for non-native ABIs.
Cache MSR_MEMORY_CTL, avoiding msr read on the context switch.
Use non-seializing immediate form for WRMSR when available.

[markj]

sys/amd64/amd64/pmap.c

10257 ↗

(On Diff #187206)

This should load flags with atomic_load_*.

sys/amd64/amd64/sys_machdep.c

457 ↗

(On Diff #187206)

Extra break.

sys/amd64/amd64/vm_machdep.c

185 ↗

(On Diff #187206)

Shouldn't it be loading td1->td_md.md_td_flags?

[markj]

If I understand correctly, #AC isn't raised if a splitlock operation happens while in kernel mode. But, what about places that use casuword() and friends? They will behave the same as before even if splitlock detection is enabled.

[kib]

In D59815#1373664, @markj wrote:

If I understand correctly, #AC isn't raised if a splitlock operation happens while in kernel mode. But, what about places that use casuword() and friends? They will behave the same as before even if splitlock detection is enabled.

Isn't casuword() executed in kernel mode?

Hm, I also had an impression that #AC only happen when CPL is 3. But re-checking SDM 11.1.2.3 Features to Disable Bus Locks and description of the #AC exception, I started doubt it.
I think it costs nothing to add the recovery for #AC handler in kernel mode.

[kib]

Add kernel-mode #AC handler.
Copy md_td_flags from the right source.
Use atomic_load() in context switch code.

[kib]

In D59815#1373713, @kib wrote:

In D59815#1373664, @markj wrote:

If I understand correctly, #AC isn't raised if a splitlock operation happens while in kernel mode. But, what about places that use casuword() and friends? They will behave the same as before even if splitlock detection is enabled.

Isn't casuword() executed in kernel mode?

Hm, I also had an impression that #AC only happen when CPL is 3. But re-checking SDM 11.1.2.3 Features to Disable Bus Locks and description of the #AC exception, I started doubt it.
I think it costs nothing to add the recovery for #AC handler in kernel mode.

I checked, simics does generates #AC in CPL 0, the current patch correctly returns EFAULT from umtx_op UMTX_OP_MUTEX_LOCK on mis-aligned struct umutex.

[markj]

I wonder if it should be a procctl feature instead of sysarch()? If your intent is to set ia32_splitlock_force = 1 by default, then it would be nice if proccontrol could be used to override application settings without having to modify the program or change the global default. Hmm, but I guess this wouldn't work if the policy is to reset the splitlock setting upon exec.

sys/amd64/amd64/exec_machdep.c

415 ↗

(On Diff #187285)

What if !ia32_splitlock is true? disable_splitlock() asserts MPASS(ia32_splitlock).

sys/amd64/amd64/pmap.c

10256 ↗

(On Diff #187285)

I think the oldtd != td check is redundant: if oldtd == td then the oldpmap == pmap case above should be taken.

[kib]

In D59815#1373855, @markj wrote:

I wonder if it should be a procctl feature instead of sysarch()? If your intent is to set ia32_splitlock_force = 1 by default, then it would be nice if proccontrol could be used to override application settings without having to modify the program or change the global default. Hmm, but I guess this wouldn't work if the policy is to reset the splitlock setting upon exec.

My first version of the patch controlled it with MD procctl. But I felt it was too 'tweaky' for such minor and probably not too popular feature. After all, we do not provide a procctl knob to enable %rflags.AC, and nobody asked for that.

If somebody wants to enable either alignment checks or disable splitlocks, this can be done with a trivial LD_PRELOADed dso. The static binaries are off (like Go-compiled binaries) but I do not feel guilty.

[kib]

In exec_splitlock(), do not call disable_splitlock() if hw does not support it.
Remove oldtd != td check in context switch.

[markj]

sys/amd64/amd64/pmap.c

10256 ↗

(On Diff #187285)

Hmm, but this is a per-thread setting, not per-process. So in the oldpmap == pmap case, where we are switching between two threads in the same proc, the per-thread splitlock setting might not be applied correctly.

sys/amd64/include/proc.h

72 ↗

(On Diff #187315)

It should be (k) I believe.

[kib]

Fix handling of context switches.

[markj]

sys/amd64/amd64/sys_machdep.c

436 ↗

(On Diff #187356)

I guess we don't need the proc lock here.

sys/x86/include/sysarch.h

56 ↗

(On Diff #187356)

Why does it have the I386_ prefix? Is it just to avoid having to translate in freebsd32_sysarch()?

[kib]

sys/amd64/amd64/sys_machdep.c

436 ↗

(On Diff #187356)

Indeed. md_td_flags was annotated with (c).

sys/x86/include/sysarch.h

56 ↗

(On Diff #187356)

To mark it as available to the i386 userspace as well. I see no sense in adding the parallel AMD64_GET/SET_SPLITLOCK constants.

[kib]

Remove proc lock in sysarch()

This was also tested on real hardware

(uarch synth) = Intel Emerald Rapids {Raptor Cove, optim of Golden Cove}, Intel 7
(synth) = Intel Xeon Scalable (5th Gen) Bronze/Silver/Gold/Platinum (Emerald Rapids A1/R1) {Raptor Cove, optim of Golden Cove}, Intel 7

which has the splitlock control.

[markj]

Presumably sysarch.2 should also be updated.

[kib]

In D59815#1374449, @markj wrote:

Presumably sysarch.2 should also be updated.

Rather, it should be written. Or might be sysarch.x86.2. Right now the man page is a stub that proclaims the existence of the syscall.
I might do it later.

[markj]

In D59815#1374524, @kib wrote:

In D59815#1374449, @markj wrote:

Presumably sysarch.2 should also be updated.

Rather, it should be written. Or might be sysarch.x86.2. Right now the man page is a stub that proclaims the existence of the syscall.
I might do it later.

Oops, ok. I was thinking about pkru(3) and forgot that it is a separate page.

[brooks]

rG20c09e6fece29dbcf4835b4dd90e969f1b9b3e59 raised the minimum LLVM version to build FreeBSD to 20. That's probably too far given that 15.1 still has 19.

[ehem_freebsd_m5p.com]

What @brooks wrote. Building HEAD on 15.1 doesn't work now and reverting rG:20c09e6fece makes the build work on 15.1.

[brooks]

In D59815#1377193, @ehem_freebsd_m5p.com wrote:

What @brooks wrote. Building HEAD on 15.1 doesn't work now and reverting rG:20c09e6fece makes the build work on 15.1.

Fixed in 53fe018630d06eeef6791cfacaa6dc9acdf4d669