That's pretty terrible, given that all flavors of xo_emit* turn into the same codepath (xo_do_emit). Can you isolate this into something I can debug, or send me your current patch (phil@freebsd.org) and I'll see if I can do it?

$ sudo dtrace -n 'BEGIN { mod(0xffffffff8261a000); exit(0); }'
dtrace: description 'BEGIN ' matched 1 probe
CPU ID FUNCTION:NAME

6      1                           :BEGIN

opensolaris.ko

@phil I've tried to do a basic implementation using xo_create_to_file and then implementing dt_emit() by redirecting buffered output and sprintf output to the regular printf output (it becomes dt_vprintf since i have a va_list instead of variadic arguments). However, I've noticed that if I do regular text output using

Update the diff. This diff should address the following:

Thanks for the detailed reviews @markj and @phil! The libxo clarifications have been very helpful, much appreciated. I've added a few comments in places that can be discussed independently and will aim to update the diff addressing the comments sometime this week.

In D41745#951971, @domagoj.stolfa_gmail.com wrote:

In D41745#951858, @stephane.rochoy_stormshield.eu wrote:

I suggest to think about adding --libxo foo as an alias to -x oformat=foo to make things more inline with what is actually done for, e.g., netstat(1).

Thanks for the feedback! I don't take a strong view on whether or not xo_parse_args() should be called at the start, though it might require some finessing to make it work happily with libdtrace because xo_parse_args() sets a number of flags for libxo that I don't with -x oformat (such as colored output in some cases). Another thing worth pondering is that this is also an argument to libdtrace rather than dtrace(1), so one would inevitably need to still call dtrace_setopt(). Now, however libxo configuration would be performed twice: once in dtrace(1), and once in libdtrace. dtrace(1) itself would explicitly need to include libxo and link against it, but this isn't a big issue IMO, it just "feels" cleaner to me for it to only link against libdtrace and for libdtrace to provide all the libxo handling and configuration and for any consumer code (including dtrace(1)) to simply request it.

However I do agree with the general point that there is an inconsistency with the rest of the base system when it comes to libxo integration (though it's inconsistent in a fair bit of other ways too). One way to work around the issue without xo_parse_args() would be perhaps to add a - flag and manually parse out the -libxo ... that follows? I'm not sure what the best approach would be that doesn't get slightly messy.

In D41745#951971, @domagoj.stolfa_gmail.com wrote:

In D41745#951858, @stephane.rochoy_stormshield.eu wrote:

I suggest to think about adding --libxo foo as an alias to -x oformat=foo to make things more inline with what is actually done for, e.g., netstat(1).

Thanks for the feedback! I don't take a strong view on whether or not xo_parse_args() should be called at the start, though it might require some finessing to make it work happily with libdtrace because xo_parse_args() sets a number of flags for libxo that I don't with -x oformat (such as colored output in some cases).

Fix off->val type causing a build failure.

Fix struct indentation.

Add missing information to the man page and fix some bits in it (e.g. using .Fn for action names).

Updated the diff with documentation in the dtrace(1) man page, as well as a bug fix when it comes to naming aggregations. Namely, min, max, sum and count were all called count in the final output due to missing checks. These are now addressed.

Please address the comments from markj@ but otherwise this looks good, and more extensive than what we came up with on CADETS.

In D41745#951858, @stephane.rochoy_stormshield.eu wrote:

I suggest to think about adding --libxo foo as an alias to -x oformat=foo to make things more inline with what is actually done for, e.g., netstat(1).

I suggest to think about adding --libxo foo as an alias to -x oformat=foo to make things more inline with what is actually done for, e.g., netstat(1).

Attempt #2 at addressing @markj's comments... Forgot one gettimeofday

Address some comments by @markj. The man page comment is still true, as I will be updating that when all the documentation comes in.

I don't see a reason to avoid adding it to dtrace.1?

Perhaps its own man page is warranted in order to avoid polluting dtrace(1)?

Fix return offset handling.

Approved.

Address comments, fix stuff that was also fixed in riscv.

Update

Address Mark's comments. Depends on D40962.

Disable adr/adrp emulation.

Use per-probe trampolines. Depends on D40962

Updated description, still some unsolved bugs, but mostly done.

In D39884#930242, @jrtc27 wrote:

In D39884#930234, @christos wrote:

In D39884#930225, @mhorne wrote:

In D39884#928340, @markj wrote:

This provider can trigger what appears to be a QEMU/TCG bug. The test case:

• boot a riscv VM with more than one vCPU
• run dtrace -q -n 'kinst::riscv_cpu_intr:', i.e., trace all instructions in riscv_cpu_intr()

Seems I was a bit slow to report my findings, but I did observe some of the same kinds of panics on real hardware that I saw in QEMU. I will update to the committed version and provide a more detailed answer of what is still not working.

Can you recompile kinst with this patch applied? https://reviews.freebsd.org/P592
If we run it with a single trampoline per page, the crashes seem to go away.

Is it just a case of needing to smp_rendezvous a fence_i since it's not broadcast by the hardware like on arm64?

In D39884#930242, @jrtc27 wrote:

Is it just a case of needing to smp_rendezvous a fence_i since it's not broadcast by the hardware like on arm64?

In D39884#930234, @christos wrote:

In D39884#930225, @mhorne wrote:

In D39884#928340, @markj wrote:

This provider can trigger what appears to be a QEMU/TCG bug. The test case:

• boot a riscv VM with more than one vCPU
• run dtrace -q -n 'kinst::riscv_cpu_intr:', i.e., trace all instructions in riscv_cpu_intr()

Seems I was a bit slow to report my findings, but I did observe some of the same kinds of panics on real hardware that I saw in QEMU. I will update to the committed version and provide a more detailed answer of what is still not working.

Can you recompile kinst with this patch applied? https://reviews.freebsd.org/P592
If we run it with a single trampoline per page, the crashes seem to go away.

In D39884#930225, @mhorne wrote:

In D39884#928340, @markj wrote:

This provider can trigger what appears to be a QEMU/TCG bug. The test case:

• boot a riscv VM with more than one vCPU
• run dtrace -q -n 'kinst::riscv_cpu_intr:', i.e., trace all instructions in riscv_cpu_intr()

Seems I was a bit slow to report my findings, but I did observe some of the same kinds of panics on real hardware that I saw in QEMU. I will update to the committed version and provide a more detailed answer of what is still not working.

Updates.

In D39884#930225, @mhorne wrote:

In D39884#928340, @markj wrote:

This provider can trigger what appears to be a QEMU/TCG bug. The test case:

• boot a riscv VM with more than one vCPU
• run dtrace -q -n 'kinst::riscv_cpu_intr:', i.e., trace all instructions in riscv_cpu_intr()

Seems I was a bit slow to report my findings, but I did observe some of the same kinds of panics on real hardware that I saw in QEMU. I will update to the committed version and provide a more detailed answer of what is still not working.

In D39884#928340, @markj wrote:

This provider can trigger what appears to be a QEMU/TCG bug. The test case:

• boot a riscv VM with more than one vCPU
• run dtrace -q -n 'kinst::riscv_cpu_intr:', i.e., trace all instructions in riscv_cpu_intr()

Retain CDDL license.

Restore original trampoline size.

Approved.

WIP.

Address comments, update LICENSE headers, split some changes to other PRs.

This provider can trigger what appears to be a QEMU/TCG bug. The test case:

• boot a riscv VM with more than one vCPU
• run dtrace -q -n 'kinst::riscv_cpu_intr:', i.e., trace all instructions in riscv_cpu_intr()

Address comments, fix undefined behavior when working with potentially unaligned pointers.

Address comments.

Address comments.

In D39884#918563, @markj wrote:

I've been doing some testing in QEMU again. I'm occasionally able to trigger random "impossible" panics by running dtrace -n 'kinst::vm_fault:' -q & and running various programs in the foreground. top(1), running tests, etc.. It's rather difficult to reproduce though. :(

I've been doing some testing in QEMU again. I'm occasionally able to trigger random "impossible" panics by running dtrace -n 'kinst::vm_fault:' -q & and running various programs in the foreground. top(1), running tests, etc.. It's rather difficult to reproduce though. :(

Check for the function prologue.

Do not use kinst_memcpy outside of probe context.

Use kinst_memcpy() and also add kinst_md_excluded().

Closed by commit 9c80ad6839cd30ecfeff2fb946d86888815da600, but had to do it manually because the commit message ate up the 'D' from 'Differential Revision'...

Approved

Remove cpu_switch() from kinst_excluded().

Add kinst_md_excluded() as well.

Remove amd64 exception handlers.

Get rid of the 'push rbp' removal in this patch.

Address comments.

In D39229#916822, @markj wrote:

The "push %rbp" check isn't sufficient though. Today you can do this: dtrace -n 'kinst::dtrace_probe:0'. This doesn't work very well.

I would like to at least fix that problem before reconsidering the push %rbp check.