ddb(4): Add 'sysctl' command

[cem]

Summary

Implement sysctl in ddb by overriding SYSCTL_OUT. When handling the
req, we install custom ddb in/out handlers. The out handler prints straight
to the debugger, while the in handler ignores all input. This is intended
to allow us to print just about any sysctl.

Submitted by: Travis Lane <travis.lane AT isilon.com>

Test Plan

Eric and Don already reviewed this on the other side of the fence, so I've
taggged you two again here. If you can think of someone else to rope in,
please do.

Travis is no longer at Isilon so that email address isn't current, but I don't
have a newer one for him.

Note that if used on a live system via debug.kdb.enter=1, there is a panic when
leaving ddb; I intend to at least look at that before committing.

db> sysctl kern.hostname
kern.hostname: testvm
db> sysctl kern.shutdown.dumpdevname
kern.shutdown.dumpdevname: gpt/swapfs
db> c
panic: mutex Giant not owned at /usr/home/conrad/src/freebsd/sys/kern/kern_mutex.c:281
cpuid = 3
time = 1557436278
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x35/frame 0xfffffe00157d45a0
kdb_backtrace() at kdb_backtrace+0x36/frame 0xfffffe00157d4650
vpanic() at vpanic+0x111/frame 0xfffffe00157d46a0
panic() at panic+0x3c/frame 0xfffffe00157d4700
__mtx_assert() at __mtx_assert+0xe7/frame 0xfffffe00157d4710
__mtx_unlock_flags() at __mtx_unlock_flags+0x65/frame 0xfffffe00157d4740
sysctl_root_handler_locked() at sysctl_root_handler_locked+0x10a/frame 0xfffffe00157d4790
sysctl_root.isra.12() at sysctl_root.isra.12+0x17f/frame 0xfffffe00157d4810
userland_sysctl() at userland_sysctl+0x136/frame 0xfffffe00157d48c0
sys___sysctl() at sys___sysctl+0x7b/frame 0xfffffe00157d4980

[dgmorris_earthlink.net]

Looks like a straightforward copy of what we reviewed, yeah.

Regarding the live version panic - it is a little odd... __mtx_assert() does check for SCHEDULER_STOPPED... I haven't poked around the live breakpoint in quite a while... wonder if that isn't lit? That would do it. Anyway, I trust you to dig a little deeper on that. Thanks. Otherwise, I'm fine with things.

[cem]

In D20219#435661, @dgmorris_earthlink.net wrote:

Regarding the live version panic - it is a little odd... __mtx_assert() does check for SCHEDULER_STOPPED... I haven't poked around the live breakpoint in quite a while... wonder if that isn't lit? That would do it. Anyway, I trust you to dig a little deeper on that. Thanks. Otherwise, I'm fine with things.

This assert is in unlock after we're leaving ddb, not a direct mtx_assert while in ddb. I think this is related to a problem we hit internally where "unlock" isn't conditional on scheduler_stopped in ddb. So in this case, we have Giant when we enter ddb because it's taken for sysctl operations, then we "lock it" again (noop) while running ddb sysctl, ddb sysctl unlocks it (releasing the lock — bug), and when we return from ddb we release it again, but it is already unlocked, so we trip this. Or that's the theory.