x86 tsc: use RDTSCP in preference of fence + RDTSC

[kib]

Summary

Both kernel and userspace. It is recommended by both Intel and AMD. Also, on AMD this means that more appropriate fence is used instead of too harsh MFENCe.

My crash box uses TSC and not TSC-low, so half of the patch with more delicate asm is not yet tried.

[gallatin]

I tested the kernel part of this on a Netflix server with an EPYC 7502P using TSC-low. It saves roughly 2% CPU and tsc disappears entirely from profiling output, making profiling output match that from our Xeons much more closely.

Note that the bulk of the tsc usage is coming from microuptime called via rack tcp, so I cannot comment on accuracy, only performance and lack of any problems large enough to be noticed in quick system wide testing.

[kib]

Switch to use LFENCE;RDTSC for Zen+.
Rework ifunc resolver to avoid code duplication.

[gallatin]

After talking with our AMD rep, the AMD advise for this is:

before Zen (fam < 0x17): MFENCE;RDTSC is preferred
Zen1 & Zen2 : There isn't much of a difference between RDTSCP and LFENCE;RDTSC
Zen3: RDTSCP will be slower than LFENCE;RDTSC

So it seems like, for simplicity, Zen should use LFENCE;RDTSC; and older AMD should use MFENCE;RDTSC

[pho]

I ran stress tests on two different hosts for an hour. No problems seen.

[markj]

lib/libc/x86/sys/__vdso_gettc.c

93 ↗

(On Diff #81754)

IMO it is a bit neater to dereference th in rdtsc_low() rather than its callers, like rdtscp_low().

sys/x86/x86/tsc.c

813 ↗

(On Diff #81754)

This is rdtscp32().

832 ↗

(On Diff #81754)

Doesn't tc_priv need to be cast to 32 bits like in tsc_get_timecount_low()?

[kib]

sys/x86/x86/tsc.c

813 ↗

(On Diff #81754)

Well, except that we do not have such helper in cpufunc.h. I considered adding it, but decided that single-use does not make sense, additional argument for me was that this file contains enough asm already.

If you prefer rdtscp32(), I will add it.

832 ↗

(On Diff #81754)

Casting would make it into r-value from l-value, which does not allow to use m specifier. Inline asm is happy without casting as well, because movl %1,%%ecx explicitly informs assembler about operand size (both 'l' and target size).

I believe the cast in tsc_get_timecount_low() is not strictly needed, since there is no harm from loading whole 64bit of tc_priv into %rcx on amd64. But I do not want to touch it ATM.

[kib]

Make rdtsc_low() and rdtscp_low() similar by both taking struct timehands *.

[markj]

sys/x86/x86/tsc.c

813 ↗

(On Diff #81754)

There would be 2 uses: here and in __vdso_gettc.c, where rdtscp32() is already defined locally. Sorry for not being more clear, the duplication is why I made the suggestion.

[kib]

Add rdtscp32() to cpufunc.h.