dtrace: add a knob to control maximum size of principal buffers

[avg]

Summary

We had a hardcoded limit of 1/128-th of physical memory that was further
subdivided between all CPUs as principal buffers are allocated on the
per-CPU basis. Actually, the buffers could use up 1/64-th of the
memmory because with the default switch policy there are two buffers per
CPU.

This commit allows to change that limit.

Note that the discussed limit is per dtrace command invocation.
The idea is to limit the size of a single malloc(9) call, not the total
memory size used by DTrace buffers.

[avg]

sys/cddl/contrib/opensolaris/uts/common/dtrace/dtrace.c

12202

I am curious about this condition which seems to be equivalent to !__i386__.
Maybe it's related to PAE.

[markj]

I have no objection to this change, but did you ever try to remove the check entirely? I wonder if some of the patches imported from illumos have mitigated the need for this check. I'll try running the test suite with that change at least.

sys/cddl/contrib/opensolaris/uts/common/dtrace/dtrace.c

12202

Because physmem * PAGE_SIZE will overflow if sizeof(long) == 4 and there is more than 4GB of physical memory? Could be, yes. Perhaps we should cast the multiplication properly and remove the ifdef?

12209–12210

I think mp_maxid + 1 should really be mp_ncpus.

[avg]

sys/cddl/contrib/opensolaris/uts/common/dtrace/dtrace.c

12209–12210

You are right. Will correct.

[avg]

In D33648#761332, @markj wrote:

I have no objection to this change, but did you ever try to remove the check entirely? I wonder if some of the patches imported from illumos have mitigated the need for this check. I'll try running the test suite with that change at least.

No, I haven't yet. Let me try.

[markj]

In D33648#761332, @markj wrote:

I have no objection to this change, but did you ever try to remove the check entirely? I wonder if some of the patches imported from illumos have mitigated the need for this check. I'll try running the test suite with that change at least.

I don't see any new failures from the test suite with the check removed, at least.

[avg]

Does it test things like setting bufsize to a significant portion of total memory?
Might be worthwhile testing that directly.

[markj]

In D33648#761370, @avg wrote:

Does it test things like setting bufsize to a significant portion of total memory?
Might be worthwhile testing that directly.

It tests some absurd values, but not things like physmem/2. If I try it, it works eventually, but the allocation takes a very long time and other processes are blocked, probably because kmem_back_domain() holds a global lock for a long period. Other than that, it looks like dtrace handles buffer allocation failures gracefully. For instance, if I try -x bufsize=64g on a system with 64GB of RAM, dtrace hits a page allocation failure and eventually reduces the buffer size to 512MB.

So ok, the seatbelt is somewhat useful for now at least. I'm ok with the change as-is, it would be nice to fix the other nits, perhaps in a separate patch.