arm64: Add Arm SPE thread mode support

[sarah.walker2_arm.com]

Summary

Add support for the using the Arm Statistical Profiling Extension (SPE) in
the hardware tracing framework, hwt(4), thread mode.

SPE is an optional extension added in Armv8.1 to provide profiling of
software using randomised instruction sampling. Thread mode allows tracing
of a single thread, regardless of the CPU it is scheduled on.

Sponsored by: Arm Ltd

[andrew]

How does the buffering work in threading mode on SMP? It looks like there is still a per-CPU buffer for SPE to write to where I would expect it to be per-thread. e.g. If userspace maps the CPU0 buffer, then the thread migrates to CPU1 would userspace still get SPE data?

[sarah.walker2_arm.com]

In D53738#1228675, @andrew wrote:

How does the buffering work in threading mode on SMP? It looks like there is still a per-CPU buffer for SPE to write to where I would expect it to be per-thread. e.g. If userspace maps the CPU0 buffer, then the thread migrates to CPU1 would userspace still get SPE data?

The buffers are per thread - they get mapped on/off the appropriate CPU via the thread switch hook. The /dev/hwt_*_* files are also per thread.

[andrew]

I see, hwt_switch_in calls hwt_backend_configure so the thread is configured on the new CPU

sys/arm64/spe/arm_spe_backend.c

654–670

Is s->info a CPU ID or a thread ID in threaded mode? It looks like it's something userspace sets & it shouldn't need to know what CPU a thread is on.

739

It looks like this is only used in the snprintf & could be moved to the format string.

742

Is there a reason to use M_NOWAIT here? If we can use M_WAITOK then the allocation is guaranteed to succeed.

[sarah.walker2_arm.com]

Address review comments

[sarah.walker2_arm.com]

sys/arm64/spe/arm_spe_backend.c

654–670

Do you mean s->ident? In threaded mode that's set in spe_backend_configure(), from an ID provided by the HWT scheduler hooks.

[andrew]

sys/arm64/spe/arm_spe_backend.c

654–670

Yes it should have been s->ident. If it comes from info->ident then that is a thread index, while spe_info is indexed by CPU ID. You probably need the same TAILQ_FOREACH loop as in spe_backend_configure() to find the info pointer.

[sarah.walker2_arm.com]

sys/arm64/spe/arm_spe_backend.c

654–670

You are correct. Will fix.

[sarah.walker2_arm.com]

Fix spe_info indexing in threading mode