Page MenuHomeFreeBSD
Differential D52968

vmm: My attempt at fixing the rendezvous deadlock
ClosedPublic
Actions

Authored by markj on Oct 7 2025, 8:13 PM.
Tags
None
Referenced Files
Unknown Object (File)
Mon, Nov 3, 6:54 PM
Unknown Object (File)
Mon, Nov 3, 7:58 AM
Unknown Object (File)
Mon, Nov 3, 4:56 AM
Unknown Object (File)
Sun, Nov 2, 6:30 AM
Unknown Object (File)
Wed, Oct 29, 1:51 PM
Unknown Object (File)
Mon, Oct 27, 3:25 AM
Unknown Object (File)
Sun, Oct 26, 10:41 PM
Unknown Object (File)
Thu, Oct 23, 9:07 AM
View All 33 Files
Subscribers
bcran
emaste
imp
jbo
rgrimes
riscv

Details

Reviewers
corvink
andrew
manu
Group Reviewers
bhyve
Commits
rGf39768e52e51: vmm: Fix a deadlock between vm_smp_rendezvous() and vcpu_lock_all()
Summary

This is a WIP, not intended to be committed as-is.

Implement vcpu_lock_all() in a way that avoids deadlocks with
vm_smp_rendezvous(). In particular, when traversing vCPUs, invoke the
rendezvous callback on the vCPU's behalf to help the initiator finish.
We can only safely do so when the vCPU is IDLE or we have already locked
it.

Diff Detail

Repository
rG FreeBSD src repository
Lint
Lint Not Applicable
Unit
Tests Not Applicable

Event Timeline

markj created this revision.Oct 7 2025, 8:13 PM
Herald added a reviewer: andrew. · View Herald TranscriptOct 7 2025, 8:13 PM
Herald added a reviewer: andrew. · View Herald Transcript
Herald added a reviewer: manu. · View Herald Transcript
Herald added a reviewer: bhyve. · View Herald Transcript
Herald added subscribers: riscv, bcran, rgrimes and 2 others. · View Herald Transcript
markj requested review of this revision.Oct 7 2025, 8:13 PM
Harbormaster completed remote builds in B67625: Diff 163723.Oct 7 2025, 8:13 PM
markj mentioned this in D52781: sys/amd64/vmm: fix Windows 11 deadlock.Oct 7 2025, 8:14 PM
corvink added a comment.Tue, Oct 14, 12:57 PM

It doesn't look like it fully freezes. However, it gets extremely slow. The circle of the Windows boot loader spins very slowly. Don't know if it will ever reach the desktop.

markj added a comment.Wed, Oct 15, 4:31 PM
In D52968#1212719, @corvink wrote:

It doesn't look like it fully freezes. However, it gets extremely slow. The circle of the Windows boot loader spins very slowly. Don't know if it will ever reach the desktop.

What output do you see if you run procstat -kk <bhyve PID> while this is happening?

markj updated this revision to Diff 164242.Wed, Oct 15, 4:46 PM

Fix a lock order reversal.

Harbormaster completed remote builds in B67810: Diff 164242.Wed, Oct 15, 4:46 PM
corvink added a comment.EditedThu, Oct 16, 6:34 AM
In D52968#1213405, @markj wrote:
In D52968#1212719, @corvink wrote:

It doesn't look like it fully freezes. However, it gets extremely slow. The circle of the Windows boot loader spins very slowly. Don't know if it will ever reach the desktop.

What output do you see if you run procstat -kk <bhyve PID> while this is happening?

Almost always one vCPU hangs in vm_handle_rendezvous. The output is from a single VM run. I've called procstat a few times within a few seconds:

# procstat -kk 1688
  PID    TID COMM                TDNAME              KSTACK                                                                                                                                                                                                                                                                 
 1688 100214 bhyve               mevent              mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1db kqueue_scan+0x9e1 kqueue_kevent+0x13b kern_kevent_fp+0x4b kern_kevent_generic+0xd6 sys_kevent+0x61 amd64_syscall+0x10e fast_syscall_common+0xf8                                             
 1688 100394 bhyve               blk-1:0-0           mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1db umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x10e fast_syscall_common+0xf8                                                             
 1688 100395 bhyve               blk-1:0-1           mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1db umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x10e fast_syscall_common+0xf8                                                             
 1688 100396 bhyve               blk-1:0-2           mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1db umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x10e fast_syscall_common+0xf8                                                             
 1688 100397 bhyve               blk-1:0-3           mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1db umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x10e fast_syscall_common+0xf8                                                             
 1688 100398 bhyve               blk-1:0-4           mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1db umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x10e fast_syscall_common+0xf8                                                             
 1688 100399 bhyve               blk-1:0-5           mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1db umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x10e fast_syscall_common+0xf8                                                             
 1688 100400 bhyve               blk-1:0-6           mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1db umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x10e fast_syscall_common+0xf8                                                             
 1688 100401 bhyve               blk-1:0-7           mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1db umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x10e fast_syscall_common+0xf8                                                             
 1688 100402 bhyve               nvme-aen-1:0        mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1db umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x10e fast_syscall_common+0xf8                                                             
 1688 100403 bhyve               vtnet-4:0 tx        mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1db umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x10e fast_syscall_common+0xf8                                                             
 1688 100405 bhyve               rfb                 mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1db sbwait+0x6a soreceive_generic_locked+0x225 soreceive_generic+0xd6 soreceive+0x2f dofileread+0x80 sys_read+0xb3 amd64_syscall+0x10e fast_syscall_common+0xf8                                 
 1688 100406 bhyve               tpm_intf_crb        mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1db umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x10e fast_syscall_common+0xf8                                                             
 1688 100407 bhyve               vcpu 0              kern_ioctl+0x11b sys_ioctl+0x10a amd64_syscall+0x10e fast_syscall_common+0xf8                                                                                                                                                                                          
 1688 100408 bhyve               vcpu 1              mi_switch+0xbb sleepq_timedwait+0x2f _sleep+0x1c8 vm_handle_rendezvous+0xc0 vm_run+0x3d4 vmmdev_ioctl+0xf45 devfs_ioctl+0xcb vn_ioctl+0xce devfs_ioctl_f+0x1e kern_ioctl+0x255 sys_ioctl+0x10a amd64_syscall+0x10e fast_syscall_common+0xf8                            
 1688 100423 bhyve               rfbout              mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_timedwait_sig+0x12 _cv_timedwait_sig_sbt+0x116 seltdwait+0x72 kern_select+0x94f sys_select+0x57 amd64_syscall+0x10e fast_syscall_common+0xf8                                                                          

# procstat -kk 1688                                                                                                                                                                                                                                                                       
  PID    TID COMM                TDNAME              KSTACK                                                                                                                                                                                                                                                                 
 1688 100214 bhyve               mevent              mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1db kqueue_scan+0x9e1 kqueue_kevent+0x13b kern_kevent_fp+0x4b kern_kevent_generic+0xd6 sys_kevent+0x61 amd64_syscall+0x10e fast_syscall_common+0xf8                                             
 1688 100394 bhyve               blk-1:0-0           mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1db umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x10e fast_syscall_common+0xf8                                                             
 1688 100395 bhyve               blk-1:0-1           mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1db umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x10e fast_syscall_common+0xf8                                                             
 1688 100396 bhyve               blk-1:0-2           mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1db umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x10e fast_syscall_common+0xf8                                                             
 1688 100397 bhyve               blk-1:0-3           mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1db umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x10e fast_syscall_common+0xf8                                                             
 1688 100398 bhyve               blk-1:0-4           mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1db umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x10e fast_syscall_common+0xf8                                                             
 1688 100399 bhyve               blk-1:0-5           mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1db umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x10e fast_syscall_common+0xf8                                                             
 1688 100400 bhyve               blk-1:0-6           mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1db umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x10e fast_syscall_common+0xf8                                                             
 1688 100401 bhyve               blk-1:0-7           mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1db umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x10e fast_syscall_common+0xf8                                                             
 1688 100402 bhyve               nvme-aen-1:0        mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1db umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x10e fast_syscall_common+0xf8                                                             
 1688 100403 bhyve               vtnet-4:0 tx        mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1db umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x10e fast_syscall_common+0xf8                                                             
 1688 100405 bhyve               rfb                 mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1db sbwait+0x6a soreceive_generic_locked+0x225 soreceive_generic+0xd6 soreceive+0x2f dofileread+0x80 sys_read+0xb3 amd64_syscall+0x10e fast_syscall_common+0xf8                                 
 1688 100406 bhyve               tpm_intf_crb        mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1db umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x10e fast_syscall_common+0xf8                                                             
 1688 100407 bhyve               vcpu 0              vmx_run+0x12ce vm_run+0x172 vmmdev_ioctl+0xf45 devfs_ioctl+0xcb vn_ioctl+0xce devfs_ioctl_f+0x1e kern_ioctl+0x255 sys_ioctl+0x10a amd64_syscall+0x10e fast_syscall_common+0xf8                                                                                         
 1688 100408 bhyve               vcpu 1              mi_switch+0xbb sleepq_timedwait+0x2f _sleep+0x1c8 vm_handle_rendezvous+0xc0 vm_run+0x3d4 vmmdev_ioctl+0xf45 devfs_ioctl+0xcb vn_ioctl+0xce devfs_ioctl_f+0x1e kern_ioctl+0x255 sys_ioctl+0x10a amd64_syscall+0x10e fast_syscall_common+0xf8 
 1688 100423 bhyve               rfbout              mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_timedwait_sig+0x12 _cv_timedwait_sig_sbt+0x116 seltdwait+0x72 kern_select+0x94f sys_select+0x57 amd64_syscall+0x10e fast_syscall_common+0xf8 

# procstat -kk 1688
  PID    TID COMM                TDNAME              KSTACK                                                                                                                                                                                                                                                                 
 1688 100214 bhyve               mevent              mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1db kqueue_scan+0x9e1 kqueue_kevent+0x13b kern_kevent_fp+0x4b kern_kevent_generic+0xd6 sys_kevent+0x61 amd64_syscall+0x10e fast_syscall_common+0xf8                                             
 1688 100394 bhyve               blk-1:0-0           mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1db umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x10e fast_syscall_common+0xf8                                                             
 1688 100395 bhyve               blk-1:0-1           mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1db umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x10e fast_syscall_common+0xf8                                                             
 1688 100396 bhyve               blk-1:0-2           mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1db umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x10e fast_syscall_common+0xf8                                                             
 1688 100397 bhyve               blk-1:0-3           mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1db umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x10e fast_syscall_common+0xf8                                                             
 1688 100398 bhyve               blk-1:0-4           mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1db umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x10e fast_syscall_common+0xf8                                                             
 1688 100399 bhyve               blk-1:0-5           mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1db umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x10e fast_syscall_common+0xf8                                                             
 1688 100400 bhyve               blk-1:0-6           mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1db umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x10e fast_syscall_common+0xf8                                                             
 1688 100401 bhyve               blk-1:0-7           mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1db umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x10e fast_syscall_common+0xf8                                                             
 1688 100402 bhyve               nvme-aen-1:0        mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1db umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x10e fast_syscall_common+0xf8                                                             
 1688 100403 bhyve               vtnet-4:0 tx        mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1db umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x10e fast_syscall_common+0xf8                                                             
 1688 100405 bhyve               rfb                 mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1db sbwait+0x6a soreceive_generic_locked+0x225 soreceive_generic+0xd6 soreceive+0x2f dofileread+0x80 sys_read+0xb3 amd64_syscall+0x10e fast_syscall_common+0xf8                                 
 1688 100406 bhyve               tpm_intf_crb        mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1db umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x10e fast_syscall_common+0xf8                                                             
 1688 100407 bhyve               vcpu 0              mi_switch+0xbb sleepq_timedwait+0x2f _sleep+0x1c8 vm_handle_rendezvous+0xc0 vm_run+0x3d4 vmmdev_ioctl+0xf45 devfs_ioctl+0xcb vn_ioctl+0xce devfs_ioctl_f+0x1e kern_ioctl+0x255 sys_ioctl+0x10a amd64_syscall+0x10e fast_syscall_common+0xf8                            
 1688 100408 bhyve               vcpu 1              vmx_run+0x12ce vm_run+0x172 vmmdev_ioctl+0xf45 devfs_ioctl+0xcb vn_ioctl+0xce devfs_ioctl_f+0x1e kern_ioctl+0x255 sys_ioctl+0x10a amd64_syscall+0x10e fast_syscall_common+0xf8                                                                                         
 1688 100423 bhyve               rfbout              mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_timedwait_sig+0x12 _sleep+0x1b6 kern_clock_nanosleep+0x1cf sys_nanosleep+0x3b amd64_syscall+0x10e fast_syscall_common+0xf8                                                                                            

# procstat -kk 1688
  PID    TID COMM                TDNAME              KSTACK                       
 1688 100214 bhyve               mevent              mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1db kqueue_scan+0x9e1 kqueue_kevent+0x13b kern_kevent_fp+0x4b kern_kevent_generic+0xd6 sys_kevent+0x61 amd64_syscall+0x10e fast_syscall_common+0xf8 
 1688 100394 bhyve               blk-1:0-0           mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1db umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x10e fast_syscall_common+0xf8 
 1688 100395 bhyve               blk-1:0-1           mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1db umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x10e fast_syscall_common+0xf8 
 1688 100396 bhyve               blk-1:0-2           mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1db umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x10e fast_syscall_common+0xf8 
 1688 100397 bhyve               blk-1:0-3           mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1db umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x10e fast_syscall_common+0xf8 
 1688 100398 bhyve               blk-1:0-4           mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1db umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x10e fast_syscall_common+0xf8 
 1688 100399 bhyve               blk-1:0-5           mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1db umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x10e fast_syscall_common+0xf8 
 1688 100400 bhyve               blk-1:0-6           mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1db umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x10e fast_syscall_common+0xf8 
 1688 100401 bhyve               blk-1:0-7           mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1db umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x10e fast_syscall_common+0xf8 
 1688 100402 bhyve               nvme-aen-1:0        mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1db umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x10e fast_syscall_common+0xf8 
 1688 100403 bhyve               vtnet-4:0 tx        mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1db umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x10e fast_syscall_common+0xf8 
 1688 100405 bhyve               rfb                 mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1db sbwait+0x6a soreceive_generic_locked+0x225 soreceive_generic+0xd6 soreceive+0x2f dofileread+0x80 sys_read+0xb3 amd64_syscall+0x10e fast_syscall_common+0xf8 
 1688 100406 bhyve               tpm_intf_crb        mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1db umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x10e fast_syscall_common+0xf8 
 1688 100407 bhyve               vcpu 0              mi_switch+0xbb sleepq_timedwait+0x2f _sleep+0x1c8 vm_handle_rendezvous+0xc0 vm_run+0x3d4 vmmdev_ioctl+0xf45 devfs_ioctl+0xcb vn_ioctl+0xce devfs_ioctl_f+0x1e kern_ioctl+0x255 sys_ioctl+0x10a amd64_syscall+0x10e fast_syscall_common+0xf8 
 1688 100408 bhyve               vcpu 1              sys_ioctl+0x10a amd64_syscall+0x10e fast_syscall_common+0xf8 
 1688 100423 bhyve               rfbout              mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_timedwait_sig+0x12 _cv_timedwait_sig_sbt+0x116 seltdwait+0x72 kern_select+0x94f sys_select+0x57 amd64_syscall+0x10e fast_syscall_common+0xf8
markj updated this revision to Diff 164347.Thu, Oct 16, 9:56 PM

Fix a silly mistake

Harbormaster completed remote builds in B67835: Diff 164347.Thu, Oct 16, 9:56 PM
markj added a comment.Thu, Oct 16, 9:57 PM
In D52968#1213618, @corvink wrote:
In D52968#1213405, @markj wrote:
In D52968#1212719, @corvink wrote:

It doesn't look like it fully freezes. However, it gets extremely slow. The circle of the Windows boot loader spins very slowly. Don't know if it will ever reach the desktop.

What output do you see if you run procstat -kk <bhyve PID> while this is happening?

Almost always one vCPU hangs in vm_handle_rendezvous. The output is from a single VM run. I've called procstat a few times within a few seconds:

Could you please try the updated patch?

corvink added a comment.Fri, Oct 17, 6:22 AM
In D52968#1214028, @markj wrote:
In D52968#1213618, @corvink wrote:
In D52968#1213405, @markj wrote:
In D52968#1212719, @corvink wrote:

It doesn't look like it fully freezes. However, it gets extremely slow. The circle of the Windows boot loader spins very slowly. Don't know if it will ever reach the desktop.

What output do you see if you run procstat -kk <bhyve PID> while this is happening?

Almost always one vCPU hangs in vm_handle_rendezvous. The output is from a single VM run. I've called procstat a few times within a few seconds:

Could you please try the updated patch?

It seems to work. I was able to reboot the Windows VM multiple times without any issues. Thanks!

corvink accepted this revision.Fri, Oct 17, 6:23 AM
This revision is now accepted and ready to land.Fri, Oct 17, 6:23 AM
Closed by commit rGf39768e52e51: vmm: Fix a deadlock between vm_smp_rendezvous() and vcpu_lock_all() (authored by markj). · Explain WhyFri, Oct 17, 2:31 PM
This revision was automatically updated to reflect the committed changes.
markj added a commit: rGf39768e52e51: vmm: Fix a deadlock between vm_smp_rendezvous() and vcpu_lock_all().
jbo added a subscriber: jbo.Mon, Oct 20, 2:16 PM
corvink mentioned this in D37972: bhyve: help rendezvous request when waiting for IDLE vcpu state.Tue, Oct 21, 6:42 AM

Revision Contents
Changeset List

PathSize
sys/
amd64/
include/
3 lines
vmm/
177 lines
arm64/
include/
2 lines
vmm/
4 lines
dev/
vmm/
28 lines
riscv/
include/
2 lines
vmm/
4 lines

Diff 164417

View Options

sys/amd64/include/vmm.h

Loading...
View Options

sys/amd64/vmm/vmm.c

Loading...
View Options

sys/arm64/include/vmm.h

Loading...
View Options

sys/arm64/vmm/vmm.c

Loading...
View Options

sys/dev/vmm/vmm_dev.c

Loading...
View Options

sys/riscv/include/vmm.h

Loading...
View Options

sys/riscv/vmm/vmm.c

Loading...