Add a way to temporarily suspend and resume virtual CPUs.

[jhb]

Summary

This is used as part of implementing run control in bhyve's debug
server. The hypervisor now maintains a set of "debugged" CPUs.
Attempting to run a debugged CPU will fail to execute any guest
instructions and will instead report a VM_EXITCODE_DEBUG exit to
the userland hypervisor. Virtual CPUs are placed into the debugged
state via vm_suspend_cpu() (implemented via a new VM_SUSPEND_CPU ioctl).
Virtual CPUs can be resumed via vm_resume_cpu() (VM_RESUME_CPU ioctl).

The debug server suspends virtual CPUs when it wishes them to stop
executing in the guest (for example, when a debugger attaches to the
server). The debug server can choose to resume only a subset of CPUs
(for example, when single stepping) or it can choose to resume all
CPUs. The debug server must explicitly mark a CPU as resumed via
vm_resume_cpu() before the virtual CPU will successfully execute any
guest instructions.

Test Plan

• used to implement halt on attach, halt for Ctrl-C, resuming after continue, resuming after detach, and resuming for single-stepping in the GDB stub / debug server for bhyve
• only tested on Intel to date

[jhb]

Ping?

[grehan]

Anything available I could use to test this on AMD ?

[jhb]

In D14466#310258, @grehan wrote:

Anything available I could use to test this on AMD ?

I just updated the bhyve_gdb branch at github.com/bsdjhb/freebsd.git. I don't have a better test case than the debug server itself unfortunately. You should be able to add '-G w1234' to the bhyve command line in which case it will pause at the first instruction until a debugger connects on port 1234. You can then use 'info registers', 'info threads' (each vCPU is a thread, but only once the vCPU shows up). You can also read memory (e.g. x/i $rip). You can use 'continue' and then wait a bit and hit Ctrl-C to break back into the debugger. Detach should also work. I have pushed some untested changes last night after rebasing that should let it work with capsicum, but you might want to build with "-DWITHOUT_CAPSICUM=yes" for now to be safe for testing.

[jhb]

Andriy, does that mean you were able to test the debug server itself on AMD?

[avg]

In D14466#313263, @jhb wrote:

Andriy, does that mean you were able to test the debug server itself on AMD?

I've only reviewed it.
I can do a test if you'd like, just give me some instructions to get started and a couple of days.

[jhb]

In D14466#313304, @avg wrote:

In D14466#313263, @jhb wrote:

Andriy, does that mean you were able to test the debug server itself on AMD?

I've only reviewed it.
I can do a test if you'd like, just give me some instructions to get started and a couple of days.

See my last comment above to Peter. You can just pull my 'bhyve_gdb' git branch from github/bsdjhb/freebsd.git. It contains the debug server on top of this change. You have to add '-G 1234' to bhyve and then you can use 'target remote localhost:1234' with gdb on the host to connect to it.

[avg]

I have tested bsdjhb/freebsd/bhyve_gdb@3db8f96fc0 (before the capsicum change) on an AMD system (Phenom II X4 955).
Below is a transcript from kgdb, looks sane to me (the image is a FreeBSD installation, the kernel is without symbols):

(kgdb) target remote localhost:1234
Remote debugging using localhost:1234
0xffffffff802e2530 in btext ()
(kgdb) bt
#0  0xffffffff802e2530 in btext ()
(kgdb) i reg
rax            0x0      0
rbx            0x0      0
rcx            0x0      0
rdx            0x0      0
rsi            0x0      0
rdi            0x0      0
rbp            0x0      0x0
rsp            0x1000   0x1000
r8             0x0      0
r9             0x0      0
r10            0x0      0
r11            0x0      0
r12            0x0      0
r13            0x0      0
r14            0x0      0
r15            0x0      0
rip            0xffffffff802e2530       0xffffffff802e2530 <btext>
eflags         0x2      [ ]
cs             0x8      8
ss             0x10     16
ds             0x10     16
es             0x10     16
fs             0x10     16
gs             0x10     16
fs_base        <unavailable>
gs_base        <unavailable>
(kgdb) x/i $rip
=> 0xffffffff802e2530 <btext>:  movw   $0x1234,0x472
(kgdb) i thr
  Id   Target Id         Frame
* 1    Thread 1 (vCPU 0) 0xffffffff802e2530 in btext ()
(kgdb) c
Continuing.
^C[New Thread 2]

Thread 1 received signal SIGTRAP, Trace/breakpoint trap.
0xffffffff80d3fd23 in cpu_idle_acpi ()
(kgdb) i thr
  Id   Target Id         Frame 
* 1    Thread 1 (vCPU 0) 0xffffffff80d3fd23 in cpu_idle_acpi ()
  2    Thread 2 (vCPU 1) 0xffffffff80d3fd23 in cpu_idle_acpi ()
(kgdb) i reg
rax            0x0      0
rbx            0xffffffff816cb500       -2123582208
rcx            0x0      0
rdx            0x6a9ea7b        111798907
rsi            0xfffff800026b3c80       -8796052439936
rdi            0xd06c7f9        218548217
rbp            0xfffffe007b73cae0       0xfffffe007b73cae0
rsp            0xfffffe007b73cad0       0xfffffe007b73cad0
r8             0x1a0d8e 1707406
r9             0xfffffe000088c000       -2199014293504
r10            0x10e0   4320
r11            0xfffffe0093551cc0       -2196551426880
r12            0xffffffff815fa088       -2124439416
r13            0xffffffff815fa09a       -2124439398
r14            0xd06c7f9        218548217
r15            0x1      1
rip            0xffffffff80d3fd23       0xffffffff80d3fd23 <cpu_idle_acpi+67>
eflags         0x246    [ PF ZF IF ]
cs             0x20     32
ss             0x28     40
ds             0x3b     59
es             0x3b     59
fs             0x13     19
gs             0x1b     27
fs_base        <unavailable>
gs_base        <unavailable>
(kgdb) x/i $rip
=> 0xffffffff80d3fd23 <cpu_idle_acpi+67>:       movl   $0x0,(%rbx)
(kgdb) bt
#0  0xffffffff80d3fd23 in cpu_idle_acpi ()
#1  0xffffffff80d3fdc0 in cpu_idle ()
#2  0xffffffff8097e4f5 in sched_idletd ()
#3  0xffffffff8091adca in fork_exit ()
#4  <signal handler called>
(kgdb) thr 2
[Switching to thread 2 (Thread 2)]
#0  0xffffffff80d3fd23 in cpu_idle_acpi ()
(kgdb) i reg
rax            0x0      0
rbx            0xffffffff816cb900       -2123581184
rcx            0x0      0
rdx            0x6a17b49        111246153
rsi            0xfffff800026b3c80       -8796052439936
rdi            0x7e983545       2123904325
rbp            0xfffffe007b741ae0       0xfffffe007b741ae0
rsp            0xfffffe007b741ad0       0xfffffe007b741ad0
r8             0xfd306a 16593002
r9             0xfffffe000088c000       -2199014293504
r10            0x10bf   4287
r11            0x0      0
r12            0xffffffff815fad88       -2124436088
r13            0xffffffff815fad9a       -2124436070
r14            0x7e983545       2123904325
r15            0xf      15
rip            0xffffffff80d3fd23       0xffffffff80d3fd23 <cpu_idle_acpi+67>
eflags         0x246    [ PF ZF IF ]
cs             0x20     32
ss             0x28     40
ds             0x3b     59
es             0x3b     59
fs             0x13     19
gs             0x1b     27
fs_base        <unavailable>
gs_base        <unavailable>

[jhb]

Andriy, looks sane to me too. Thanks! Peter, are you ok with this now?

[grehan]

Fine by me. Looking forward to using this :)