Page MenuHomeFreeBSD
Differential D52968

vmm: My attempt at fixing the rendezvous deadlock
ClosedPublic
Actions

Authored by markj on Oct 7 2025, 8:13 PM.
Tags
None
Referenced Files
Unknown Object (File)
Tue, Nov 25, 10:56 PM
Unknown Object (File)
Mon, Nov 24, 4:24 PM
Unknown Object (File)
Mon, Nov 24, 3:05 PM
Unknown Object (File)
Sat, Nov 22, 3:25 PM
Unknown Object (File)
Sat, Nov 22, 8:46 AM
Unknown Object (File)
Sat, Nov 22, 2:33 AM
Unknown Object (File)
Thu, Nov 20, 7:28 AM
Unknown Object (File)
Tue, Nov 18, 8:51 AM
View All 44 Files
Subscribers
bcran
emaste
imp
jbo
rgrimes
riscv

Details

Reviewers
corvink
andrew
manu
Group Reviewers
bhyve
Commits
rG300a8977bcfd: vmm: Fix a deadlock between vm_smp_rendezvous() and vcpu_lock_all()
rGf39768e52e51: vmm: Fix a deadlock between vm_smp_rendezvous() and vcpu_lock_all()
Summary

This is a WIP, not intended to be committed as-is.

Implement vcpu_lock_all() in a way that avoids deadlocks with
vm_smp_rendezvous(). In particular, when traversing vCPUs, invoke the
rendezvous callback on the vCPU's behalf to help the initiator finish.
We can only safely do so when the vCPU is IDLE or we have already locked
it.

Diff Detail

Repository
rG FreeBSD src repository
Lint
Lint Not Applicable
Unit
Tests Not Applicable

Event Timeline

markj created this revision.Oct 7 2025, 8:13 PM
Herald added a reviewer: andrew. · View Herald TranscriptOct 7 2025, 8:13 PM
Herald added a reviewer: andrew. · View Herald Transcript
Herald added a reviewer: manu. · View Herald Transcript
Herald added a reviewer: bhyve. · View Herald Transcript
Herald added subscribers: riscv, bcran, rgrimes and 2 others. · View Herald Transcript
markj requested review of this revision.Oct 7 2025, 8:13 PM
Harbormaster completed remote builds in B67625: Diff 163723.Oct 7 2025, 8:13 PM
markj mentioned this in D52781: sys/amd64/vmm: fix Windows 11 deadlock.Oct 7 2025, 8:14 PM
corvink added a comment.Oct 14 2025, 12:57 PM

It doesn't look like it fully freezes. However, it gets extremely slow. The circle of the Windows boot loader spins very slowly. Don't know if it will ever reach the desktop.

markj added a comment.Oct 15 2025, 4:31 PM
In D52968#1212719, @corvink wrote:

It doesn't look like it fully freezes. However, it gets extremely slow. The circle of the Windows boot loader spins very slowly. Don't know if it will ever reach the desktop.

What output do you see if you run procstat -kk <bhyve PID> while this is happening?

markj updated this revision to Diff 164242.Oct 15 2025, 4:46 PM

Fix a lock order reversal.

Harbormaster completed remote builds in B67810: Diff 164242.Oct 15 2025, 4:46 PM
corvink added a comment.EditedOct 16 2025, 6:34 AM
In D52968#1213405, @markj wrote:
In D52968#1212719, @corvink wrote:

It doesn't look like it fully freezes. However, it gets extremely slow. The circle of the Windows boot loader spins very slowly. Don't know if it will ever reach the desktop.

What output do you see if you run procstat -kk <bhyve PID> while this is happening?

Almost always one vCPU hangs in vm_handle_rendezvous. The output is from a single VM run. I've called procstat a few times within a few seconds:

# procstat -kk 1688
  PID    TID COMM                TDNAME              KSTACK                                                                                                                                                                                                                                                                 
 1688 100214 bhyve               mevent              mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1db kqueue_scan+0x9e1 kqueue_kevent+0x13b kern_kevent_fp+0x4b kern_kevent_generic+0xd6 sys_kevent+0x61 amd64_syscall+0x10e fast_syscall_common+0xf8                                             
 1688 100394 bhyve               blk-1:0-0           mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1db umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x10e fast_syscall_common+0xf8                                                             
 1688 100395 bhyve               blk-1:0-1           mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1db umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x10e fast_syscall_common+0xf8                                                             
 1688 100396 bhyve               blk-1:0-2           mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1db umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x10e fast_syscall_common+0xf8                                                             
 1688 100397 bhyve               blk-1:0-3           mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1db umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x10e fast_syscall_common+0xf8                                                             
 1688 100398 bhyve               blk-1:0-4           mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1db umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x10e fast_syscall_common+0xf8                                                             
 1688 100399 bhyve               blk-1:0-5           mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1db umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x10e fast_syscall_common+0xf8                                                             
 1688 100400 bhyve               blk-1:0-6           mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1db umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x10e fast_syscall_common+0xf8                                                             
 1688 100401 bhyve               blk-1:0-7           mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1db umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x10e fast_syscall_common+0xf8                                                             
 1688 100402 bhyve               nvme-aen-1:0        mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1db umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x10e fast_syscall_common+0xf8                                                             
 1688 100403 bhyve               vtnet-4:0 tx        mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1db umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x10e fast_syscall_common+0xf8                                                             
 1688 100405 bhyve               rfb                 mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1db sbwait+0x6a soreceive_generic_locked+0x225 soreceive_generic+0xd6 soreceive+0x2f dofileread+0x80 sys_read+0xb3 amd64_syscall+0x10e fast_syscall_common+0xf8                                 
 1688 100406 bhyve               tpm_intf_crb        mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1db umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x10e fast_syscall_common+0xf8                                                             
 1688 100407 bhyve               vcpu 0              kern_ioctl+0x11b sys_ioctl+0x10a amd64_syscall+0x10e fast_syscall_common+0xf8                                                                                                                                                                                          
 1688 100408 bhyve               vcpu 1              mi_switch+0xbb sleepq_timedwait+0x2f _sleep+0x1c8 vm_handle_rendezvous+0xc0 vm_run+0x3d4 vmmdev_ioctl+0xf45 devfs_ioctl+0xcb vn_ioctl+0xce devfs_ioctl_f+0x1e kern_ioctl+0x255 sys_ioctl+0x10a amd64_syscall+0x10e fast_syscall_common+0xf8                            
 1688 100423 bhyve               rfbout              mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_timedwait_sig+0x12 _cv_timedwait_sig_sbt+0x116 seltdwait+0x72 kern_select+0x94f sys_select+0x57 amd64_syscall+0x10e fast_syscall_common+0xf8                                                                          

# procstat -kk 1688                                                                                                                                                                                                                                                                       
  PID    TID COMM                TDNAME              KSTACK                                                                                                                                                                                                                                                                 
 1688 100214 bhyve               mevent              mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1db kqueue_scan+0x9e1 kqueue_kevent+0x13b kern_kevent_fp+0x4b kern_kevent_generic+0xd6 sys_kevent+0x61 amd64_syscall+0x10e fast_syscall_common+0xf8                                             
 1688 100394 bhyve               blk-1:0-0           mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1db umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x10e fast_syscall_common+0xf8                                                             
 1688 100395 bhyve               blk-1:0-1           mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1db umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x10e fast_syscall_common+0xf8                                                             
 1688 100396 bhyve               blk-1:0-2           mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1db umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x10e fast_syscall_common+0xf8                                                             
 1688 100397 bhyve               blk-1:0-3           mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1db umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x10e fast_syscall_common+0xf8                                                             
 1688 100398 bhyve               blk-1:0-4           mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1db umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x10e fast_syscall_common+0xf8                                                             
 1688 100399 bhyve               blk-1:0-5           mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1db umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x10e fast_syscall_common+0xf8                                                             
 1688 100400 bhyve               blk-1:0-6           mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1db umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x10e fast_syscall_common+0xf8                                                             
 1688 100401 bhyve               blk-1:0-7           mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1db umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x10e fast_syscall_common+0xf8                                                             
 1688 100402 bhyve               nvme-aen-1:0        mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1db umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x10e fast_syscall_common+0xf8                                                             
 1688 100403 bhyve               vtnet-4:0 tx        mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1db umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x10e fast_syscall_common+0xf8                                                             
 1688 100405 bhyve               rfb                 mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1db sbwait+0x6a soreceive_generic_locked+0x225 soreceive_generic+0xd6 soreceive+0x2f dofileread+0x80 sys_read+0xb3 amd64_syscall+0x10e fast_syscall_common+0xf8                                 
 1688 100406 bhyve               tpm_intf_crb        mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1db umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x10e fast_syscall_common+0xf8                                                             
 1688 100407 bhyve               vcpu 0              vmx_run+0x12ce vm_run+0x172 vmmdev_ioctl+0xf45 devfs_ioctl+0xcb vn_ioctl+0xce devfs_ioctl_f+0x1e kern_ioctl+0x255 sys_ioctl+0x10a amd64_syscall+0x10e fast_syscall_common+0xf8                                                                                         
 1688 100408 bhyve               vcpu 1              mi_switch+0xbb sleepq_timedwait+0x2f _sleep+0x1c8 vm_handle_rendezvous+0xc0 vm_run+0x3d4 vmmdev_ioctl+0xf45 devfs_ioctl+0xcb vn_ioctl+0xce devfs_ioctl_f+0x1e kern_ioctl+0x255 sys_ioctl+0x10a amd64_syscall+0x10e fast_syscall_common+0xf8 
 1688 100423 bhyve               rfbout              mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_timedwait_sig+0x12 _cv_timedwait_sig_sbt+0x116 seltdwait+0x72 kern_select+0x94f sys_select+0x57 amd64_syscall+0x10e fast_syscall_common+0xf8 

# procstat -kk 1688
  PID    TID COMM                TDNAME              KSTACK                                                                                                                                                                                                                                                                 
 1688 100214 bhyve               mevent              mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1db kqueue_scan+0x9e1 kqueue_kevent+0x13b kern_kevent_fp+0x4b kern_kevent_generic+0xd6 sys_kevent+0x61 amd64_syscall+0x10e fast_syscall_common+0xf8                                             
 1688 100394 bhyve               blk-1:0-0           mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1db umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x10e fast_syscall_common+0xf8                                                             
 1688 100395 bhyve               blk-1:0-1           mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1db umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x10e fast_syscall_common+0xf8                                                             
 1688 100396 bhyve               blk-1:0-2           mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1db umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x10e fast_syscall_common+0xf8                                                             
 1688 100397 bhyve               blk-1:0-3           mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1db umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x10e fast_syscall_common+0xf8                                                             
 1688 100398 bhyve               blk-1:0-4           mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1db umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x10e fast_syscall_common+0xf8                                                             
 1688 100399 bhyve               blk-1:0-5           mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1db umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x10e fast_syscall_common+0xf8                                                             
 1688 100400 bhyve               blk-1:0-6           mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1db umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x10e fast_syscall_common+0xf8                                                             
 1688 100401 bhyve               blk-1:0-7           mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1db umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x10e fast_syscall_common+0xf8                                                             
 1688 100402 bhyve               nvme-aen-1:0        mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1db umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x10e fast_syscall_common+0xf8                                                             
 1688 100403 bhyve               vtnet-4:0 tx        mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1db umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x10e fast_syscall_common+0xf8                                                             
 1688 100405 bhyve               rfb                 mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1db sbwait+0x6a soreceive_generic_locked+0x225 soreceive_generic+0xd6 soreceive+0x2f dofileread+0x80 sys_read+0xb3 amd64_syscall+0x10e fast_syscall_common+0xf8                                 
 1688 100406 bhyve               tpm_intf_crb        mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1db umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x10e fast_syscall_common+0xf8                                                             
 1688 100407 bhyve               vcpu 0              mi_switch+0xbb sleepq_timedwait+0x2f _sleep+0x1c8 vm_handle_rendezvous+0xc0 vm_run+0x3d4 vmmdev_ioctl+0xf45 devfs_ioctl+0xcb vn_ioctl+0xce devfs_ioctl_f+0x1e kern_ioctl+0x255 sys_ioctl+0x10a amd64_syscall+0x10e fast_syscall_common+0xf8                            
 1688 100408 bhyve               vcpu 1              vmx_run+0x12ce vm_run+0x172 vmmdev_ioctl+0xf45 devfs_ioctl+0xcb vn_ioctl+0xce devfs_ioctl_f+0x1e kern_ioctl+0x255 sys_ioctl+0x10a amd64_syscall+0x10e fast_syscall_common+0xf8                                                                                         
 1688 100423 bhyve               rfbout              mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_timedwait_sig+0x12 _sleep+0x1b6 kern_clock_nanosleep+0x1cf sys_nanosleep+0x3b amd64_syscall+0x10e fast_syscall_common+0xf8                                                                                            

# procstat -kk 1688
  PID    TID COMM                TDNAME              KSTACK                       
 1688 100214 bhyve               mevent              mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1db kqueue_scan+0x9e1 kqueue_kevent+0x13b kern_kevent_fp+0x4b kern_kevent_generic+0xd6 sys_kevent+0x61 amd64_syscall+0x10e fast_syscall_common+0xf8 
 1688 100394 bhyve               blk-1:0-0           mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1db umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x10e fast_syscall_common+0xf8 
 1688 100395 bhyve               blk-1:0-1           mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1db umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x10e fast_syscall_common+0xf8 
 1688 100396 bhyve               blk-1:0-2           mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1db umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x10e fast_syscall_common+0xf8 
 1688 100397 bhyve               blk-1:0-3           mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1db umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x10e fast_syscall_common+0xf8 
 1688 100398 bhyve               blk-1:0-4           mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1db umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x10e fast_syscall_common+0xf8 
 1688 100399 bhyve               blk-1:0-5           mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1db umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x10e fast_syscall_common+0xf8 
 1688 100400 bhyve               blk-1:0-6           mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1db umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x10e fast_syscall_common+0xf8 
 1688 100401 bhyve               blk-1:0-7           mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1db umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x10e fast_syscall_common+0xf8 
 1688 100402 bhyve               nvme-aen-1:0        mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1db umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x10e fast_syscall_common+0xf8 
 1688 100403 bhyve               vtnet-4:0 tx        mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1db umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x10e fast_syscall_common+0xf8 
 1688 100405 bhyve               rfb                 mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1db sbwait+0x6a soreceive_generic_locked+0x225 soreceive_generic+0xd6 soreceive+0x2f dofileread+0x80 sys_read+0xb3 amd64_syscall+0x10e fast_syscall_common+0xf8 
 1688 100406 bhyve               tpm_intf_crb        mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_wait_sig+0x9 _sleep+0x1db umtxq_sleep+0x2cd do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x10e fast_syscall_common+0xf8 
 1688 100407 bhyve               vcpu 0              mi_switch+0xbb sleepq_timedwait+0x2f _sleep+0x1c8 vm_handle_rendezvous+0xc0 vm_run+0x3d4 vmmdev_ioctl+0xf45 devfs_ioctl+0xcb vn_ioctl+0xce devfs_ioctl_f+0x1e kern_ioctl+0x255 sys_ioctl+0x10a amd64_syscall+0x10e fast_syscall_common+0xf8 
 1688 100408 bhyve               vcpu 1              sys_ioctl+0x10a amd64_syscall+0x10e fast_syscall_common+0xf8 
 1688 100423 bhyve               rfbout              mi_switch+0xbb sleepq_catch_signals+0x2c6 sleepq_timedwait_sig+0x12 _cv_timedwait_sig_sbt+0x116 seltdwait+0x72 kern_select+0x94f sys_select+0x57 amd64_syscall+0x10e fast_syscall_common+0xf8
markj updated this revision to Diff 164347.Oct 16 2025, 9:56 PM

Fix a silly mistake

Harbormaster completed remote builds in B67835: Diff 164347.Oct 16 2025, 9:56 PM
markj added a comment.Oct 16 2025, 9:57 PM
In D52968#1213618, @corvink wrote:
In D52968#1213405, @markj wrote:
In D52968#1212719, @corvink wrote:

It doesn't look like it fully freezes. However, it gets extremely slow. The circle of the Windows boot loader spins very slowly. Don't know if it will ever reach the desktop.

What output do you see if you run procstat -kk <bhyve PID> while this is happening?

Almost always one vCPU hangs in vm_handle_rendezvous. The output is from a single VM run. I've called procstat a few times within a few seconds:

Could you please try the updated patch?

corvink added a comment.Oct 17 2025, 6:22 AM
In D52968#1214028, @markj wrote:
In D52968#1213618, @corvink wrote:
In D52968#1213405, @markj wrote:
In D52968#1212719, @corvink wrote:

It doesn't look like it fully freezes. However, it gets extremely slow. The circle of the Windows boot loader spins very slowly. Don't know if it will ever reach the desktop.

What output do you see if you run procstat -kk <bhyve PID> while this is happening?

Almost always one vCPU hangs in vm_handle_rendezvous. The output is from a single VM run. I've called procstat a few times within a few seconds:

Could you please try the updated patch?

It seems to work. I was able to reboot the Windows VM multiple times without any issues. Thanks!

corvink accepted this revision.Oct 17 2025, 6:23 AM
This revision is now accepted and ready to land.Oct 17 2025, 6:23 AM
Closed by commit rGf39768e52e51: vmm: Fix a deadlock between vm_smp_rendezvous() and vcpu_lock_all() (authored by markj). · Explain WhyOct 17 2025, 2:31 PM
This revision was automatically updated to reflect the committed changes.
markj added a commit: rGf39768e52e51: vmm: Fix a deadlock between vm_smp_rendezvous() and vcpu_lock_all().
jbo added a subscriber: jbo.Oct 20 2025, 2:16 PM
corvink mentioned this in D37972: bhyve: help rendezvous request when waiting for IDLE vcpu state.Oct 21 2025, 6:42 AM
markj added a commit: rG300a8977bcfd: vmm: Fix a deadlock between vm_smp_rendezvous() and vcpu_lock_all().Fri, Nov 7, 3:00 PM
jbo added a comment.EditedTue, Nov 11, 1:49 PM

Can you guys confirm that 300a8977bcfd2f43bc6df81d9bdad6b79a740729 fixes the Windows 11 passthru boot-stuck issue?
D52781 was actually working for me. However, this patch (D52968) results in the same symptoms for me: Windows 11 gets stuck during boot when attempting to passthru a PCIe device.

markj added a comment.Tue, Nov 11, 2:40 PM
In D52968#1225802, @jbo wrote:

Can you guys confirm that 300a8977bcfd2f43bc6df81d9bdad6b79a740729 fixes the Windows 11 passthru boot-stuck issue?
D52781 was actually working for me. However, this patch (D52968) results in the same symptoms for me: Windows 11 gets stuck during boot when attempting to passthru a PCIe device.

What are you testing exactly? The commit that went into the src tree, or a hand-applied patch from this revision?

When the hang happens, can you please run "procstat -kk <pid of bhyve processes>" and share the output?

jbo added a comment.Tue, Nov 11, 3:18 PM
In D52968#1225836, @markj wrote:

What are you testing exactly? The commit that went into the src tree, or a hand-applied patch from this revision?

I'm on an unmodified stable/14, specifically, commit c64b00a1fa8ce7f53ec33949a418adfa6f816ee7

In D52968#1225836, @markj wrote:

When the hang happens, can you please run "procstat -kk <pid of bhyve processes>" and share the output?

Shows no output:

# procstat -kk 5593
  PID    TID COMM                TDNAME              KSTACK
markj added a comment.Tue, Nov 11, 3:25 PM
In D52968#1225861, @jbo wrote:
In D52968#1225836, @markj wrote:

What are you testing exactly? The commit that went into the src tree, or a hand-applied patch from this revision?

I'm on an unmodified stable/14, specifically, commit c64b00a1fa8ce7f53ec33949a418adfa6f816ee7

Then you don't have the patch at all?

In D52968#1225836, @markj wrote:

When the hang happens, can you please run "procstat -kk <pid of bhyve processes>" and share the output?

Shows no output:

# procstat -kk 5593
  PID    TID COMM                TDNAME              KSTACK

But bhyve is running? That's pretty weird. Are you sure that's the right PID? Also make sure that you're running procstat as root.

jbo added a comment.Tue, Nov 11, 3:30 PM
In D52968#1225867, @markj wrote:

Then you don't have the patch at all?

c64b00a1fa8ce7f53ec33949a418adfa6f816ee7 was pushed to stable/14 2025-11-10 19:28:30 +0000 (i.e. yesterday, when I specifically updated to the latest stable/14 branch to get this patch).
Am I missing something?

In D52968#1225836, @markj wrote:

But bhyve is running? That's pretty weird. Are you sure that's the right PID? Also make sure that you're running procstat as root.

Didn't run it as root - apologies.

5593 106744 bhyve               mevent              mi_switch+0xbd sleepq_catch_signals+0x2a2 sleepq_wait_sig+0x9 _sleep+0x1ec kqueue_scan+0xa11 kqueue_kevent+0x13b kern_kevent_fp+0x4b kern_kevent_generic+0xdf sys_kevent+0x61 amd64_syscall+0x117 fast_syscall_common+0xf8 
5593 117101 bhyve               blk-4:0-0           mi_switch+0xbd sleepq_catch_signals+0x2a2 sleepq_wait_sig+0x9 _sleep+0x1ec umtxq_sleep+0x2c4 do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x117 fast_syscall_common+0xf8 
5593 117102 bhyve               blk-4:0-1           mi_switch+0xbd sleepq_catch_signals+0x2a2 sleepq_wait_sig+0x9 _sleep+0x1ec umtxq_sleep+0x2c4 do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x117 fast_syscall_common+0xf8 
5593 117103 bhyve               blk-4:0-2           mi_switch+0xbd sleepq_catch_signals+0x2a2 sleepq_wait_sig+0x9 _sleep+0x1ec umtxq_sleep+0x2c4 do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x117 fast_syscall_common+0xf8 
5593 117104 bhyve               blk-4:0-3           mi_switch+0xbd sleepq_catch_signals+0x2a2 sleepq_wait_sig+0x9 _sleep+0x1ec umtxq_sleep+0x2c4 do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x117 fast_syscall_common+0xf8 
5593 117105 bhyve               blk-4:0-4           mi_switch+0xbd sleepq_catch_signals+0x2a2 sleepq_wait_sig+0x9 _sleep+0x1ec umtxq_sleep+0x2c4 do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x117 fast_syscall_common+0xf8 
5593 117106 bhyve               blk-4:0-5           mi_switch+0xbd sleepq_catch_signals+0x2a2 sleepq_wait_sig+0x9 _sleep+0x1ec umtxq_sleep+0x2c4 do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x117 fast_syscall_common+0xf8 
5593 117107 bhyve               blk-4:0-6           mi_switch+0xbd sleepq_catch_signals+0x2a2 sleepq_wait_sig+0x9 _sleep+0x1ec umtxq_sleep+0x2c4 do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x117 fast_syscall_common+0xf8 
5593 117108 bhyve               blk-4:0-7           mi_switch+0xbd sleepq_catch_signals+0x2a2 sleepq_wait_sig+0x9 _sleep+0x1ec umtxq_sleep+0x2c4 do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x117 fast_syscall_common+0xf8 
5593 117109 bhyve               nvme-aen-4:0        mi_switch+0xbd sleepq_catch_signals+0x2a2 sleepq_wait_sig+0x9 _sleep+0x1ec umtxq_sleep+0x2c4 do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x117 fast_syscall_common+0xf8 
5593 117110 bhyve               vtnet-5:0 tx        mi_switch+0xbd sleepq_catch_signals+0x2a2 sleepq_wait_sig+0x9 _sleep+0x1ec umtxq_sleep+0x2c4 do_wait+0x244 __umtx_op_wait_uint_private+0x54 sys__umtx_op+0x7e amd64_syscall+0x117 fast_syscall_common+0xf8 
5593 117111 bhyve               rfb                 mi_switch+0xbd sleepq_catch_signals+0x2a2 sleepq_wait_sig+0x9 _sleep+0x1ec sbwait+0x6b soreceive_generic_locked+0x228 soreceive_generic+0xd6 soreceive+0x2f dofileread+0x80 sys_read+0xb3 amd64_syscall+0x117 fast_syscall_common+0xf8 
5593 117112 bhyve               vcpu 0              mi_switch+0xbd sleepq_timedwait+0x2f _sleep+0x1dd vm_handle_rendezvous+0x31e vm_run+0x3ca vmmdev_ioctl+0xd70 devfs_ioctl+0xcb vn_ioctl+0xc8 devfs_ioctl_f+0x1e kern_ioctl+0x255 sys_ioctl+0x106 amd64_syscall+0x117 fast_syscall_common+0xf8 
5593 117113 bhyve               vcpu 1              mi_switch+0xbd sleepq_timedwait+0x2f _sleep+0x1dd vm_handle_rendezvous+0x31e vm_run+0x3ca vmmdev_ioctl+0xd70 devfs_ioctl+0xcb vn_ioctl+0xc8 devfs_ioctl_f+0x1e kern_ioctl+0x255 sys_ioctl+0x106 amd64_syscall+0x117 fast_syscall_common+0xf8 
5593 117114 bhyve               vcpu 2              mi_switch+0xbd sleepq_timedwait+0x2f _sleep+0x1dd vm_handle_rendezvous+0x31e vm_run+0x3ca vmmdev_ioctl+0xd70 devfs_ioctl+0xcb vn_ioctl+0xc8 devfs_ioctl_f+0x1e kern_ioctl+0x255 sys_ioctl+0x106 amd64_syscall+0x117 fast_syscall_common+0xf8 
5593 117115 bhyve               vcpu 3              mi_switch+0xbd sleepq_timedwait+0x2f _sleep+0x1dd vm_handle_rendezvous+0x31e vm_run+0x3ca vmmdev_ioctl+0xd70 devfs_ioctl+0xcb vn_ioctl+0xc8 devfs_ioctl_f+0x1e kern_ioctl+0x255 sys_ioctl+0x106 amd64_syscall+0x117 fast_syscall_common+0xf8 
5593 117116 bhyve               vcpu 4              mi_switch+0xbd sleepq_timedwait+0x2f _sleep+0x1dd vm_handle_rendezvous+0x31e vm_run+0x3ca vmmdev_ioctl+0xd70 devfs_ioctl+0xcb vn_ioctl+0xc8 devfs_ioctl_f+0x1e kern_ioctl+0x255 sys_ioctl+0x106 amd64_syscall+0x117 fast_syscall_common+0xf8 
5593 117117 bhyve               vcpu 5              mi_switch+0xbd sleepq_timedwait+0x2f _sleep+0x1dd vm_handle_rendezvous+0x31e vm_run+0x3ca vmmdev_ioctl+0xd70 devfs_ioctl+0xcb vn_ioctl+0xc8 devfs_ioctl_f+0x1e kern_ioctl+0x255 sys_ioctl+0x106 amd64_syscall+0x117 fast_syscall_common+0xf8 
5593 117118 bhyve               vcpu 6              mi_switch+0xbd sleepq_timedwait+0x2f _sleep+0x1dd vm_handle_rendezvous+0x31e vm_run+0x3ca vmmdev_ioctl+0xd70 devfs_ioctl+0xcb vn_ioctl+0xc8 devfs_ioctl_f+0x1e kern_ioctl+0x255 sys_ioctl+0x106 amd64_syscall+0x117 fast_syscall_common+0xf8 
5593 117119 bhyve               vcpu 7              mi_switch+0xbd sleepq_timedwait+0x2f msleep_spin_sbt+0xd5 vcpu_set_state_locked+0xa2 vcpu_set_state+0x43 vcpu_lock_all+0x6e vmmdev_ioctl+0x355 devfs_ioctl+0xcb vn_ioctl+0xc8 devfs_ioctl_f+0x1e kern_ioctl+0x255 sys_ioctl+0x106 amd64_syscall+0x117 fast_syscall_common+0xf8 
5593 117120 bhyve               vcpu 8              mi_switch+0xbd sleepq_timedwait+0x2f _sleep+0x1dd vm_handle_rendezvous+0x31e vm_run+0x3ca vmmdev_ioctl+0xd70 devfs_ioctl+0xcb vn_ioctl+0xc8 devfs_ioctl_f+0x1e kern_ioctl+0x255 sys_ioctl+0x106 amd64_syscall+0x117 fast_syscall_common+0xf8 
5593 117121 bhyve               vcpu 9              mi_switch+0xbd sleepq_timedwait+0x2f _sleep+0x1dd vm_handle_rendezvous+0x31e vm_run+0x3ca vmmdev_ioctl+0xd70 devfs_ioctl+0xcb vn_ioctl+0xc8 devfs_ioctl_f+0x1e kern_ioctl+0x255 sys_ioctl+0x106 amd64_syscall+0x117 fast_syscall_common+0xf8 
5593 117122 bhyve               vcpu 10             mi_switch+0xbd sleepq_timedwait+0x2f _sleep+0x1dd vm_handle_rendezvous+0x31e vioapic_mmio_rw+0x1cd vioapic_mmio_write+0x3b vmm_emulate_instruction+0xa95 vm_run+0x70d vmmdev_ioctl+0xd70 devfs_ioctl+0xcb vn_ioctl+0xc8 devfs_ioctl_f+0x1e kern_ioctl+0x255 sys_ioctl+0x106 amd64_syscall+0x117 fast_syscall_common+0xf8 
5593 117123 bhyve               vcpu 11             mi_switch+0xbd sleepq_timedwait+0x2f _sleep+0x1dd vm_handle_rendezvous+0x31e vm_run+0x3ca vmmdev_ioctl+0xd70 devfs_ioctl+0xcb vn_ioctl+0xc8 devfs_ioctl_f+0x1e kern_ioctl+0x255 sys_ioctl+0x106 amd64_syscall+0x117 fast_syscall_common+0xf8 
5593 117124 bhyve               vcpu 12             mi_switch+0xbd sleepq_timedwait+0x2f _sleep+0x1dd vm_handle_rendezvous+0x31e vm_run+0x3ca vmmdev_ioctl+0xd70 devfs_ioctl+0xcb vn_ioctl+0xc8 devfs_ioctl_f+0x1e kern_ioctl+0x255 sys_ioctl+0x106 amd64_syscall+0x117 fast_syscall_common+0xf8 
5593 117125 bhyve               vcpu 13             mi_switch+0xbd sleepq_timedwait+0x2f _sleep+0x1dd vm_handle_rendezvous+0x31e vm_run+0x3ca vmmdev_ioctl+0xd70 devfs_ioctl+0xcb vn_ioctl+0xc8 devfs_ioctl_f+0x1e kern_ioctl+0x255 sys_ioctl+0x106 amd64_syscall+0x117 fast_syscall_common+0xf8 
5593 117126 bhyve               vcpu 14             mi_switch+0xbd sleepq_timedwait+0x2f _sleep+0x1dd vm_handle_rendezvous+0x31e vm_run+0x3ca vmmdev_ioctl+0xd70 devfs_ioctl+0xcb vn_ioctl+0xc8 devfs_ioctl_f+0x1e kern_ioctl+0x255 sys_ioctl+0x106 amd64_syscall+0x117 fast_syscall_common+0xf8 
5593 117127 bhyve               vcpu 15             mi_switch+0xbd sleepq_timedwait+0x2f _sleep+0x1dd vm_handle_rendezvous+0x31e vm_run+0x3ca vmmdev_ioctl+0xd70 devfs_ioctl+0xcb vn_ioctl+0xc8 devfs_ioctl_f+0x1e kern_ioctl+0x255 sys_ioctl+0x106 amd64_syscall+0x117 fast_syscall_common+0xf8 
5593 117133 bhyve               rfbout              mi_switch+0xbd sleepq_catch_signals+0x2a2 sleepq_timedwait_sig+0x12 _sleep+0x1ce kern_clock_nanosleep+0x1d2 sys_nanosleep+0x3b amd64_syscall+0x117 fast_syscall_common+0xf8
markj added a comment.Tue, Nov 11, 3:35 PM
In D52968#1225871, @jbo wrote:
In D52968#1225867, @markj wrote:

Then you don't have the patch at all?

c64b00a1fa8ce7f53ec33949a418adfa6f816ee7 was pushed to stable/14 2025-11-10 19:28:30 +0000 (i.e. yesterday, when I specifically updated to the latest stable/14 branch to get this patch).
Am I missing something?

Well, that commit is a random whitespace change... and the fix for the deadlock is not in stable/14 at all. I will try to backport it if you can test the resulting patch.

jbo added a comment.Tue, Nov 11, 3:39 PM
In D52968#1225880, @markj wrote:

Well, that commit is a random whitespace change... and the fix for the deadlock is not in stable/14 at all. I will try to backport it if you can test the resulting patch.

I could have sworn I saw this patch being MFC'd into stable/14...
Guessing I'm just having a bad day - apologies for the noise then :/

I will glady test any backported patch.

markj added a comment.Wed, Nov 19, 1:14 AM
In D52968#1225882, @jbo wrote:
In D52968#1225880, @markj wrote:

Well, that commit is a random whitespace change... and the fix for the deadlock is not in stable/14 at all. I will try to backport it if you can test the resulting patch.

I could have sworn I saw this patch being MFC'd into stable/14...
Guessing I'm just having a bad day - apologies for the noise then :/

I will glady test any backported patch.

Apologies for the delay, here's a backport to stable/14: https://people.freebsd.org/~markj/patches/0001-vmm-Fix-a-deadlock-between-vm_smp_rendezvous-and-vcp.patch

Please give it a try and let me know whether it helps (or even makes things worse).

Revision Contents
Changeset List

PathSize
sys/
amd64/
include/
3 lines
vmm/
177 lines
arm64/
include/
2 lines
vmm/
4 lines
dev/
vmm/
28 lines
riscv/
include/
2 lines
vmm/
4 lines

Diff 164417

View Options

sys/amd64/include/vmm.h

Loading...
View Options

sys/amd64/vmm/vmm.c

Loading...
View Options

sys/arm64/include/vmm.h

Loading...
View Options

sys/arm64/vmm/vmm.c

Loading...
View Options

sys/dev/vmm/vmm_dev.c

Loading...
View Options

sys/riscv/include/vmm.h

Loading...
View Options

sys/riscv/vmm/vmm.c

Loading...