Page MenuHomeFreeBSD

umtx: add FAIR_LINUX_FUTEX_HASHING for a distribution-fair chain hash
Needs ReviewPublic

Authored by nick_spun.io on Sun, Jul 19, 5:47 AM.
Tags
None
Referenced Files
F164619723: D58337.diff
Sun, Aug 2, 1:45 PM
F164614347: D58337.id182200.diff
Sun, Aug 2, 12:02 PM
Unknown Object (File)
Sun, Aug 2, 6:03 AM
Unknown Object (File)
Fri, Jul 31, 2:05 PM
Unknown Object (File)
Thu, Jul 30, 8:43 PM
Unknown Object (File)
Wed, Jul 29, 3:49 AM
Unknown Object (File)
Tue, Jul 28, 9:59 AM
Unknown Object (File)
Tue, Jul 28, 1:26 AM
Subscribers

Details

Reviewers
adrian
Summary

umtxq_hash() computes ((n * GOLDEN_RATIO_PRIME) >> UMTX_SHIFTS) %
UMTX_CHAINS, which is exactly Linux's hash_32(n, 9): multiply by an odd
constant and keep the high bits. The multiplier, 2654404609
(0x9E370001), is Linux's old GOLDEN_RATIO_PRIME_32, adopted here in
3f76af0f4ac4 (2004) when umtxq gained per-chain locking. Linux chose
it to be bit-sparse -- 2^31 + 2^29 - 2^25 + 2^22 - 2^19 - 2^16 + 1 --
so that the multiply could be reduced to shifts and adds on machines
with no hardware multiplier. Mixing quality was the thing traded away.

That trade makes it degenerate for keys whose spacing carries trailing
zero bits, which is what aligned allocation produces. 0x9E370001 is
0x9E37 * 2^16 + 1, so multiplying by it adds a copy of the key, shifted
16 bits up, to the key itself. When the stride carries s trailing
zeros that copy is pushed s bits further out of the word; at s = 16,
(n * 0x9E370001) mod 2^32 == n and the hash is the identity. In a 512
key sample a 4 KiB stride reaches 16 of the 512 chains, and any stride
from 16 KiB to 64 KiB reaches 4. Only the trailing zeros count, so the
floor sits exactly where allocator arena granularity sits.

This has gone unremarked since 2004 because the base system does not
exercise it. libthr allocates one object per lock -- pthread_mutex_t
is a pointer to a struct libthr allocates on first use -- so a caller
never places a wait word, and the objects land one malloc size class
apart, 128 bytes. A 128 byte stride is s = 7, and 2^(16-7) is 512, so
the sparse multiplier saturates the chain table exactly. In the one
place the base system uses this hash by default it is flawless.
d52d7aa87149 (2013) records that an earlier profiling bug "brought us
questioning in the past the quality of the umtx hash table
distribution", with no benchmark then able to reproduce an issue: a
benchmark built from pthread primitives cannot, because libthr places
those wait words, not the benchmark.

Consumers of _umtx_op(2) have no such shield: they wait on addresses
they place themselves. A Linux-ABI runtime reaching futex(2) through
linux_futex.c passes its own uaddr straight to umtx_key_get(), and one
that gives each thread its own allocation arena places those words a
power-of-two stride apart. The Bun JavaScript runtime, built on
mimalloc whose arenas are carved in 64 KiB slices, parks its per-thread
futex words 32 to 64 KiB apart -- squarely on the floor.

Because the sparse multiplier is what saturates libthr's 128 byte
stride, and a fairer one costs a little there (467 of 512 chains, worst
chain two waiters rather than one), this is a build option rather than
a change of default. FAIR_LINUX_FUTEX_HASHING, off by default, selects
1640531527 (0x61C88647), the multiplier Linux moved to in ef703f49a6c5
after judging the sparse constants "actively bad for hashing". It is
2^32 * (1 - phi), the negative golden ratio, which Knuth (vol 3,
section 6.4) notes distributes as well as phi itself; it is not prime,
and primality is irrelevant to a multiplicative hash, where only the
mixing of the retained high bits matters. The option changes the one
umtx chain hash, so it affects native _umtx_op(2) as well as the Linux
futex path; enable it on kernels that host futex-heavy runtimes placing
their own wait words at allocator strides.

Measured with options UMTX_PROFILING on two kernels differing only in
this option, by parking 512 threads on UMTX_OP_WAIT_UINT_PRIVATE wait
words at a fixed stride and reading debug.umtx.chains.<n>.max_length0.
With every thread parked, a chain's length is the number of distinct
wait words that hashed to it:

	          chains used      deepest chain
	stride    default/fair     default/fair
	 4 KiB      18 /  229         32 / 4
	64 KiB       7 /  476        128 / 3

At a 64 KiB stride the default puts 128 of the 512 waiters onto one
chain mutex; the fair multiplier leaves at most 3. A single-bit change
in the key alters 26.9% of the default hash bits, against 45.1% for the
fair one and 50% for an ideal hash. /usr/tests/lib/libthr passes 70/70
under either multiplier; the change alters hash distribution, not
semantics.

Signed-off-by: Nick Price <nick@spun.io>

Diff Detail

Repository
rG FreeBSD src repository
Lint
Lint Skipped
Unit
Tests Skipped
Build Status
Buildable 74925
Build 71808: arc lint + arc unit

Event Timeline

I think it'll be worthwhile putting a comment in here explaining the behavioural differences between the two values. Eg, referencing the linux commit, explanation about sparse alloc'ed arenas and waits, etc.
Maybe like three or four lines tops?

I do not see how this could be an option. Either there is no (bad) impact on the native umtxes, and then we should just change the constant, perhaps adding a comment explaining its origin and reasoning behind. Or there is the impact, but then you could check that the curproc ABI is linux, and use the proposed constant for Linux processes.