The cpuset_t could cause ABI issues when we bump MAXCPU. vmm.ko on amd64 was recently changed to not embed it in the struct in rGe17eca327633efc511450310afb5e4a662724e3d

Thanks I have updated; cpusetsize is now supplied by a user; cpuset_t is now a pointer

In what situations do we have more than one hwt_context? This allocates a unique ident from 0 to 1<<8 but as far as I can tell in both hwt_ioctl_alloc_mode_{thread,cpu} we only allocate a single context aka ident 0.

I guess this may happen if two users simultaneously using hwt. This is not possible in coresight, but possible in Intel PT and probably SPE ?

Ah yep - okay makes sense. It should be possible to run Coresight and SPE simultaneously at least.

On SPE everything is per core, so you *should* be able to trace on all cores separately. Two users tracing the same core would not be possible. Not sure how that would work with THREAD_MODE, as would depend on where the task is scheduled - I guess no concurrent SPE users will be possible there. I've only implemented CPU_MODE so far.

We definitely don't need all of these includes.

We should only do this work if the hook is actually enabled.

These tags shouldn't appear in new code.

How is this used? You are not passing the size of the mapping anywhere, which seems wrong.

Move the ent declaration under the if().

That said, why do you hook elf image activator at all? Such hook should be generic and set somewhere in kern_exec.c.

I doubt that two vars deserve a dedicated kern.c file. Find a place to put them elsewere.

What about the same hooks for SCHED_4BSD?

Move vars declarations under if () block.

Move ent under if().

Also, it seems you only hook vnodes mappings. What about anon memory, or swap-backed non-anon?

Also, it seems you only hook vnodes mappings. What about anon memory, or swap-backed non-anon?

The userspace decoder needs to have access to the binaries that generated the execution trace (and their location within the traced process's address space) to decode the trace, which is we only hook vnode mappings.
These hooks forward the mapping information to the userspace decoder, which will load all executable sections from the mapped binary.

I assume that this approach doesn't covers things like tracing JIT-generated code, but that can probably be fixed by placing another hook, if need be.

Agree, I have it moved to generic kern_exec.c!

Moved to kern_pmc.c

Ok, added and tested!

There should be a blank line after vars definition block.

What does the hook do? In particular, I wonder why it needs to image text vnode unlocked.

Blank line is needed after vars block.

The hook makes a copy of ent (using uma_zalloc) and stores it into the tail queue of records to the corresponding tracing context. Then it puts the thread to sleep if needed (see hwt_hook_mmap()).
Userspace periodically fetches the queue of records, locates the address of executable/library in the records, reconfigures hardware tracing units and wakes up the sleeping thread.
Sleep is only needed when the user sets up the 'address range filtering' feature to trace a particular function of an executable/dynamic library.

Based on my experiments, unlocking image text vnode is not needed, so I remove it.

Sleeping while owning a vnode lock is not prohibited, but is highly undesirable. It is tabooed when the duration of the sleep is controlled by userspace, since it causes a user-controllable live-lock: any process trying to do anything with this vnode, would be put to sleep until the sleeping thread wakes up and unlocks the vnode.

So from your description, the situation is reverse: you do need to unlock the vnode around the hook call.