bhyve: virtio: do not lock the softc mutex again to raise an interrupt

[wanpengqian_gmail.com]

Summary

Without MSI-X, vi_interrupt() takes the device's vs_mtx to set the ISR
bit and assert INTx. Most callers already hold that mutex: vi_pci_write()
calls the queue notification handler with it held, and backends such as
pci_vtblk_done() take it before vq_endchains().

With FreeBSD's default (error-checking) mutexes the second lock fails
with EDEADLK, which is ignored, and the matching unlock then releases
the caller's lock early. With a normal mutex the thread deadlocks: a
Linux guest that has no MSI domain hangs at its first block read.

The lock dates from 3cbf3585cbab (January 2014), which added INTx
support to the virtio backends and took VS_LOCK() around the ISR update
and pci_lintr_assert(); c06676bee3d2 (June 2021) moved that code into
vi_interrupt() unchanged. Because the relock only returns EDEADLK on
FreeBSD, it went unnoticed until bhyve was built against a pthread
implementation with non-error-checking mutexes.

Give the ISR state and the INTx level a small mutex of their own, taken
last, and use it in the three places that touch them.

Fixes: 3cbf3585cbab ("Enhance the support for PCI legacy INTx interrupts and enable them in the virtio backends.")
Signed-off-by: Wanpeng Qian <[e-mail]>
Sponsored by: keelos.dev

Test Plan

Built usr.sbin/bhyve on main 7cb4d4dfd with and without the change, plain
and WITH_BHYVE_SNAPSHOT, -Werror: 0 warnings.

Reproducer: in both trees the three pthread_mutex_lock(vs->vs_mtx) calls
(VS_LOCK() in virtio.h, two in virtio.c) were temporarily changed to trap
when the lock does not return 0, i.e. on the EDEADLK that FreeBSD's
error-checking mutexes return for a relock. FreeBSD main guest with
virtio-blk, virtio-net, virtio-rnd and virtio-console, MSI-X disabled in
the guest (hw.virtio.pci.disable_msix=1, so the devices use MSI and
vi_interrupt() takes the non-MSI-X path):

without the change: bhyve traps (SIGILL) at the first virtio interrupt
with the change:    boots; 256 MB written to vtbd0 and read back with
                    the same MD5, 50/50 pings over vtnet0, vtrnd0 and
                    vtcon0 attach

With MSI-X enabled both versions pass (that path does not touch vs_isr).
Without the trap the relock is silent on FreeBSD: the second lock fails,
and the inner unlock drops the caller's lock early.