Otherwise we don't do anything to kick vcpu threads out of a sleep
state when destroying a VM. For instance, suppose a guest executes hlt
on amd64 or wfi on arm64 with interrupts disabled. Then,
bhyvectl --destroy will hang until the vcpu thread somehow comes out of
vm_handle_hlt()/vm_handle_wfi() since destroy_dev() is waiting for vCPU
threads to drain.
Note that on amd64, if hw.vmm.halt_detection is set to 1 (the default),
the guest will automatically exit in this case since it's treated as a
shutdown. But, the above should not hang if halt_detection is set to 0.
Here, vm_suspend() wakes up vcpu threads, and a subsequent attempt to
run the vCPU will result in an error which gets propagated to userspace,
allowing destroy_dev() to proceed.
Add a new suspend code for this purpose. Modify bhyve to exit with
status 4 ("exited due to an error") when it's received, since that's
what'll happen generally when the VM is destroyed asynchronously.
Reported by: def