Page MenuHomeFreeBSD
Differential D27986

x86 tsc: use RDTSCP in preference of fence + RDTSC
ClosedPublic
Actions

Authored by kib on Jan 5 2021, 11:02 PM.
Tags
None
Referenced Files
F113289024: D27986.id81998.diff
Fri, Mar 28, 4:17 PM
Unknown Object (File)
Mon, Mar 10, 9:25 PM
Unknown Object (File)
Thu, Mar 6, 10:50 AM
Unknown Object (File)
Thu, Mar 6, 4:15 AM
Unknown Object (File)
Wed, Mar 5, 11:17 PM
Unknown Object (File)
Wed, Mar 5, 10:05 PM
Unknown Object (File)
Tue, Mar 4, 8:50 PM
Unknown Object (File)
Tue, Mar 4, 10:59 AM
View All Files
Subscribers
pho

Details

Reviewers
gallatin
markj
jhb
jkim
Commits
rGcd240c9cf100: x86 vdso gettc: Add RDTSCP support
rG5bf4bafd13a4: x86 vdso gettc: eliminate duplicated code in ifunc selectors.
rGde898cb96042: x86 vdso gettc: reorganize ifunctions.
rG9e680e4005b7: tsc: add RDTSCP or faster variants of get_timecount()
rG45974de8fb23: x86: Add rdtscp32() into cpufunc.h.
rG826fc3cc3d59: tsc: use u_int for return type for prototype, same as in definitions.
rGaa9450e44b5f: x86 identcpu.c: fix formatting of the comment.
Summary

Both kernel and userspace. It is recommended by both Intel and AMD. Also, on AMD this means that more appropriate fence is used instead of too harsh MFENCe.

My crash box uses TSC and not TSC-low, so half of the patch with more delicate asm is not yet tried.

Diff Detail

Repository
rG FreeBSD src repository
Lint
Lint Skipped
Unit
Tests Skipped

Event Timeline

kib requested review of this revision.Jan 5 2021, 11:02 PM
kib created this revision.
gallatin accepted this revision.Jan 6 2021, 3:18 PM

I tested the kernel part of this on a Netflix server with an EPYC 7502P using TSC-low. It saves roughly 2% CPU and tsc disappears entirely from profiling output, making profiling output match that from our Xeons much more closely.

Note that the bulk of the tsc usage is coming from microuptime called via rack tcp, so I cannot comment on accuracy, only performance and lack of any problems large enough to be noticed in quick system wide testing.

This revision is now accepted and ready to land.Jan 6 2021, 3:18 PM
kib updated this revision to Diff 81754.Jan 6 2021, 4:57 PM

Switch to use LFENCE;RDTSC for Zen+.
Rework ifunc resolver to avoid code duplication.

This revision now requires review to proceed.Jan 6 2021, 4:57 PM
gallatin added a comment.Jan 7 2021, 8:57 PM

After talking with our AMD rep, the AMD advise for this is:

before Zen (fam < 0x17): MFENCE;RDTSC is preferred
Zen1 & Zen2 : There isn't much of a difference between RDTSCP and LFENCE;RDTSC
Zen3: RDTSCP will be slower than LFENCE;RDTSC

So it seems like, for simplicity, Zen should use LFENCE;RDTSC; and older AMD should use MFENCE;RDTSC

gallatin accepted this revision.Jan 7 2021, 9:00 PM
This revision is now accepted and ready to land.Jan 7 2021, 9:00 PM
kib added a subscriber: pho.Jan 7 2021, 9:21 PM
pho added a comment.Jan 8 2021, 11:42 AM

I ran stress tests on two different hosts for an hour. No problems seen.

markj added inline comments.Jan 8 2021, 3:30 PM
lib/libc/x86/sys/__vdso_gettc.c
79

IMO it is a bit neater to dereference th in rdtsc_low() rather than its callers, like rdtscp_low().

sys/x86/x86/tsc.c
813

This is rdtscp32().

829

Doesn't tc_priv need to be cast to 32 bits like in tsc_get_timecount_low()?

kib marked an inline comment as done.Jan 8 2021, 7:19 PM
kib added inline comments.
sys/x86/x86/tsc.c
813

Well, except that we do not have such helper in cpufunc.h. I considered adding it, but decided that single-use does not make sense, additional argument for me was that this file contains enough asm already.

If you prefer rdtscp32(), I will add it.

829

Casting would make it into r-value from l-value, which does not allow to use m specifier. Inline asm is happy without casting as well, because movl %1,%%ecx explicitly informs assembler about operand size (both 'l' and target size).

I believe the cast in tsc_get_timecount_low() is not strictly needed, since there is no harm from loading whole 64bit of tc_priv into %rcx on amd64. But I do not want to touch it ATM.

kib updated this revision to Diff 81907.Jan 8 2021, 7:21 PM

Make rdtsc_low() and rdtscp_low() similar by both taking struct timehands *.

This revision now requires review to proceed.Jan 8 2021, 7:21 PM
markj accepted this revision.Jan 9 2021, 4:49 PM
markj added inline comments.
sys/x86/x86/tsc.c
813

There would be 2 uses: here and in __vdso_gettc.c, where rdtscp32() is already defined locally. Sorry for not being more clear, the duplication is why I made the suggestion.

This revision is now accepted and ready to land.Jan 9 2021, 4:49 PM
kib updated this revision to Diff 81971.Jan 9 2021, 9:18 PM
kib marked 2 inline comments as done.

Add rdtscp32() to cpufunc.h.

This revision now requires review to proceed.Jan 9 2021, 9:18 PM
This revision was not accepted when it landed; it landed in state Needs Review.Jan 10 2021, 2:43 AM
Closed by commit rGaa9450e44b5f: x86 identcpu.c: fix formatting of the comment. (authored by kib). · Explain Why
This revision was automatically updated to reflect the committed changes.
kib added a commit: rGaa9450e44b5f: x86 identcpu.c: fix formatting of the comment..
kib added a commit: rG826fc3cc3d59: tsc: use u_int for return type for prototype, same as in definitions..
kib added a commit: rG45974de8fb23: x86: Add rdtscp32() into cpufunc.h..
kib added a commit: rG9e680e4005b7: tsc: add RDTSCP or faster variants of get_timecount().
kib added a commit: rGde898cb96042: x86 vdso gettc: reorganize ifunctions..
kib added a commit: rG5bf4bafd13a4: x86 vdso gettc: eliminate duplicated code in ifunc selectors..
kib added a commit: rGcd240c9cf100: x86 vdso gettc: Add RDTSCP support.

Revision Contents
Changeset List

PathSize
lib/
libc/
x86/
sys/
150 lines
sys/
amd64/
include/
9 lines
i386/
include/
9 lines
x86/
x86/
2 lines
51 lines

Diff 81971

View Options

lib/libc/x86/sys/__vdso_gettc.c

Show First 20 Lines • Show All 47 Lines • ▼ Show 20 Lines
#include <machine/specialreg.h> #include <machine/specialreg.h>
#include <dev/acpica/acpi_hpet.h> #include <dev/acpica/acpi_hpet.h>
#ifdef WANT_HYPERV #ifdef WANT_HYPERV
#include <dev/hyperv/hyperv.h> #include <dev/hyperv/hyperv.h>
#endif #endif
#include <x86/ifunc.h> #include <x86/ifunc.h>
#include "libc_private.h" #include "libc_private.h"
static void static inline u_int
rdtsc_mb_lfence(void) rdtsc_low(const struct vdso_timehands *th)
{ {
u_int rv;
lfence(); __asm __volatile("rdtsc; shrd %%cl, %%edx, %0"
: "=a" (rv) : "c" (th->th_x86_shift) : "edx");
return (rv);
} }
static void static inline u_int
rdtsc_mb_mfence(void) rdtscp_low(const struct vdso_timehands *th)
{ {
u_int rv;
mfence(); __asm __volatile("rdtscp; movl %%edi,%%ecx; shrd %%cl, %%edx, %0"
: "=a" (rv) : "D" (th->th_x86_shift) : "ecx", "edx");
return (rv);
} }
static void static u_int
rdtsc_mb_none(void) rdtsc_low_mb_lfence(const struct vdso_timehands *th)
{ {
lfence();
markjUnsubmitted
Done
Inline Actions

IMO it is a bit neater to dereference th in rdtsc_low() rather than its callers, like rdtscp_low().

markj: IMO it is a bit neater to dereference `th` in rdtsc_low() rather than its callers, like…
return (rdtsc_low(th));
} }
DEFINE_UIFUNC(static, void, rdtsc_mb, (void)) static u_int
rdtsc_low_mb_mfence(const struct vdso_timehands *th)
{ {
u_int p[4]; mfence();
/* Not a typo, string matches our do_cpuid() registers use. */ return (rdtsc_low(th));
static const char intel_id[] = "GenuntelineI";
if ((cpu_feature & CPUID_SSE2) == 0)
return (rdtsc_mb_none);
do_cpuid(0, p);
return (memcmp(p + 1, intel_id, sizeof(intel_id) - 1) == 0 ?
rdtsc_mb_lfence : rdtsc_mb_mfence);
} }
static u_int static u_int
__vdso_gettc_rdtsc_low(const struct vdso_timehands *th) rdtsc_low_mb_none(const struct vdso_timehands *th)
{ {
u_int rv; return (rdtsc_low(th));
}
rdtsc_mb(); static u_int
__asm __volatile("rdtsc; shrd %%cl, %%edx, %0" rdtsc32_mb_lfence(void)
: "=a" (rv) : "c" (th->th_x86_shift) : "edx"); {
return (rv); lfence();
return (rdtsc32());
} }
static u_int static u_int
__vdso_rdtsc32(void) rdtsc32_mb_mfence(void)
{ {
mfence();
return (rdtsc32());
}
rdtsc_mb(); static u_int
rdtsc32_mb_none(void)
{
return (rdtsc32()); return (rdtsc32());
} }
static u_int
rdtscp32_(void)
{
return (rdtscp32());
}
struct tsc_selector_tag {
u_int (*ts_rdtsc32)(void);
u_int (*ts_rdtsc_low)(const struct vdso_timehands *);
};
static const struct tsc_selector_tag tsc_selector[] = {
[0] = { /* Intel or AMD Zen+, LFENCE */
.ts_rdtsc32 = rdtsc32_mb_lfence,
.ts_rdtsc_low = rdtsc_low_mb_lfence,
},
[1] = { /* AMD, MFENCE */
.ts_rdtsc32 = rdtsc32_mb_mfence,
.ts_rdtsc_low = rdtsc_low_mb_mfence,
},
[2] = { /* No SSE2 */
.ts_rdtsc32 = rdtsc32_mb_none,
.ts_rdtsc_low = rdtsc_low_mb_none,
},
[3] = { /* RDTSCP */
.ts_rdtsc32 = rdtscp32_,
.ts_rdtsc_low = rdtscp_low,
},
};
static int
tsc_selector_idx(u_int cpu_feature)
{
u_int amd_feature, cpu_exthigh, cpu_id, p[4], v[3];
static const char amd_id[] = "AuthenticAMD";
static const char hygon_id[] = "HygonGenuine";
bool amd_cpu;
if (cpu_feature == 0)
return (2); /* should not happen due to RDTSC */
do_cpuid(0, p);
v[0] = p[1];
v[1] = p[3];
v[2] = p[2];
amd_cpu = memcmp(v, amd_id, sizeof(amd_id) - 1) == 0 ||
memcmp(v, hygon_id, sizeof(hygon_id) - 1) == 0;
do_cpuid(1, p);
cpu_id = p[0];
if (amd_cpu && CPUID_TO_FAMILY(cpu_id) >= 0x17)
return (0);
if (cpu_feature != 0) {
do_cpuid(0x80000000, p);
cpu_exthigh = p[0];
} else {
cpu_exthigh = 0;
}
if (cpu_exthigh >= 0x80000001) {
do_cpuid(0x80000001, p);
amd_feature = p[3];
} else {
amd_feature = 0;
}
if ((amd_feature & AMDID_RDTSCP) != 0)
return (3);
if ((cpu_feature & CPUID_SSE2) == 0)
return (2);
return (amd_cpu ? 1 : 0);
}
DEFINE_UIFUNC(static, u_int, __vdso_gettc_rdtsc_low,
(const struct vdso_timehands *th))
{
return (tsc_selector[tsc_selector_idx(cpu_feature)].ts_rdtsc_low);
}
DEFINE_UIFUNC(static, u_int, __vdso_gettc_rdtsc32, (void))
{
return (tsc_selector[tsc_selector_idx(cpu_feature)].ts_rdtsc32);
}
#define HPET_DEV_MAP_MAX 10 #define HPET_DEV_MAP_MAX 10
static volatile char *hpet_dev_map[HPET_DEV_MAP_MAX]; static volatile char *hpet_dev_map[HPET_DEV_MAP_MAX];
static void static void
__vdso_init_hpet(uint32_t u) __vdso_init_hpet(uint32_t u)
{ {
static const char devprefix[] = "/dev/hpet"; static const char devprefix[] = "/dev/hpet";
char devname[64], *c, *c1, t; char devname[64], *c, *c1, t;
▲ Show 20 Lines • Show All 79 Lines • ▼ Show 20 Lines__vdso_hyperv_tsc(struct hyperv_reftsc *tsc_ref, u_int *tc)
uint64_t disc, ret, tsc, scale; uint64_t disc, ret, tsc, scale;
uint32_t seq; uint32_t seq;
int64_t ofs; int64_t ofs;
while ((seq = atomic_load_acq_int(&tsc_ref->tsc_seq)) != 0) { while ((seq = atomic_load_acq_int(&tsc_ref->tsc_seq)) != 0) {
scale = tsc_ref->tsc_scale; scale = tsc_ref->tsc_scale;
ofs = tsc_ref->tsc_ofs; ofs = tsc_ref->tsc_ofs;
rdtsc_mb(); mfence(); /* XXXKIB */
tsc = rdtsc(); tsc = rdtsc();
/* ret = ((tsc * scale) >> 64) + ofs */ /* ret = ((tsc * scale) >> 64) + ofs */
__asm__ __volatile__ ("mulq %3" : __asm__ __volatile__ ("mulq %3" :
"=d" (ret), "=a" (disc) : "=d" (ret), "=a" (disc) :
"a" (tsc), "r" (scale)); "a" (tsc), "r" (scale));
ret += ofs; ret += ofs;
Show All 15 Lines
__vdso_gettc(const struct vdso_timehands *th, u_int *tc) __vdso_gettc(const struct vdso_timehands *th, u_int *tc)
{ {
volatile char *map; volatile char *map;
uint32_t idx; uint32_t idx;
switch (th->th_algo) { switch (th->th_algo) {
case VDSO_TH_ALGO_X86_TSC: case VDSO_TH_ALGO_X86_TSC:
*tc = th->th_x86_shift > 0 ? __vdso_gettc_rdtsc_low(th) : *tc = th->th_x86_shift > 0 ? __vdso_gettc_rdtsc_low(th) :
__vdso_rdtsc32(); __vdso_gettc_rdtsc32();
return (0); return (0);
case VDSO_TH_ALGO_X86_HPET: case VDSO_TH_ALGO_X86_HPET:
idx = th->th_x86_hpet_idx; idx = th->th_x86_hpet_idx;
if (idx >= HPET_DEV_MAP_MAX) if (idx >= HPET_DEV_MAP_MAX)
return (ENOSYS); return (ENOSYS);
map = (volatile char *)atomic_load_acq_ptr( map = (volatile char *)atomic_load_acq_ptr(
(volatile uintptr_t *)&hpet_dev_map[idx]); (volatile uintptr_t *)&hpet_dev_map[idx]);
if (map == NULL) { if (map == NULL) {
Show All 28 Lines
View Options

sys/amd64/include/cpufunc.h

Show First 20 Lines • Show All 406 Lines • ▼ Show 20 Lines
rdtsc32(void) rdtsc32(void)
{ {
uint32_t rv; uint32_t rv;
__asm __volatile("rdtsc" : "=a" (rv) : : "edx"); __asm __volatile("rdtsc" : "=a" (rv) : : "edx");
return (rv); return (rv);
} }
static __inline uint32_t
rdtscp32(void)
{
uint32_t rv;
__asm __volatile("rdtscp" : "=a" (rv) : : "ecx", "edx");
return (rv);
}
static __inline void static __inline void
wbinvd(void) wbinvd(void)
{ {
__asm __volatile("wbinvd"); __asm __volatile("wbinvd");
} }
static __inline void static __inline void
write_rflags(u_long rf) write_rflags(u_long rf)
▲ Show 20 Lines • Show All 626 Lines • Show Last 20 Lines
View Options

sys/i386/include/cpufunc.h

Show First 20 Lines • Show All 406 Lines • ▼ Show 20 Lines
rdtsc32(void) rdtsc32(void)
{ {
uint32_t rv; uint32_t rv;
__asm __volatile("rdtsc" : "=a" (rv) : : "edx"); __asm __volatile("rdtsc" : "=a" (rv) : : "edx");
return (rv); return (rv);
} }
static __inline uint32_t
rdtscp32(void)
{
uint32_t rv;
__asm __volatile("rdtscp" : "=a" (rv) : : "ecx", "edx");
return (rv);
}
static __inline void static __inline void
wbinvd(void) wbinvd(void)
{ {
__asm __volatile("wbinvd"); __asm __volatile("wbinvd");
} }
static __inline void static __inline void
write_eflags(u_int ef) write_eflags(u_int ef)
▲ Show 20 Lines • Show All 402 Lines • Show Last 20 Lines
View Options

sys/x86/x86/identcpu.c

Loading...
View Options

sys/x86/x86/tsc.c

Loading...