bhyve: xhci: deassert INTx when no enabled interrupt is pending

[wanpengqian_gmail.com]

Summary

pci_xhci_deassert_interrupt() asserted the INTx line when MSI was off,
and nothing else ever deasserted it. The UEFI firmware writes IMAN with
IE clear when it starts the controller, so from then on the line stayed
up until the guest driver enabled MSI. A guest that uses INTx, or a
device sharing the pin with the xHCI, sees an interrupt after every EOI.

Follow the specification instead: INTx stays asserted only while
USBCMD.INTE is set and the interrupter has IMAN.IP and IMAN.IE set.
Re-evaluate that after every write to IMAN and USBCMD, and after a
write to ERDP that clears EHB, which here also clears IMAN.IP. Without
the latter the line stays up when an event arrives while the guest's
handler runs: the handler consumes it and acknowledges it through ERDP,
and then sees thousands of interrupts with nothing pending until the
next event.

Signed-off-by: Wanpeng Qian <[e-mail]>
Sponsored by: keelos.dev

Test Plan

main (f958aa7e7), FreeBSD 16.0-CURRENT guest booted with the UEFI firmware (BHYVE_UEFI.fd from edk2-bhyve), on FreeBSD main running nested under KVM. The guest has hw.pci.enable_msi=0 and hw.pci.enable_msix=0 in loader.conf, so its xhci(4) uses INTx. Devices: ahci-hd in slot 2, e1000 in slot 3, xhci,tablet in slot 5 or 10, fbuf, lpc.

1. The xHCI shares its pin with another device (slot 10 and ahci0 in slot 2 both get IOAPIC pin 23). This is how it was found: on a Windows guest the xHCI shared its pin with an e1000, and a resume from hibernation got stuck in an interrupt storm before the xHCI's driver had re-enabled MSI.

	before	after
boot	hangs after em0 attaches, before xhci0 is probed: ahci0's handler is unmasked on pin 23 and the line never goes down. Nothing more on the console for 60 s; 48,000 VM exits per second, mostly interrupt-window exits	boots; irq23: ahci0 xhci0, 0 interrupts per second when idle

2. The xHCI has a pin of its own (slot 5, irq 18). The guest boots in both cases.

idle, 10 s	before	after
xhci_interrupt() calls (dtrace fbt::xhci_interrupt:entry)	18,099	0
vmstat -i irq18	2,065/s	0/s
VM exits of the guest	25,234/s	1,099/s

3. The tablet still works with INTx (after). Moving the VNC pointer delivers events on /dev/input/event4 (hms0): 3,672 bytes for 50 moves, 106 interrupts on irq 23. devctl detach xhci0; devctl attach pci0:0:5:0 re-enumerates the tablet; the interrupt rate is 0 again afterwards.

4. Every INTx interrupt corresponds to an event (diff 2). The first diff re-evaluated the line only after writes to IMAN and USBCMD. With it the guest still took about 22,000 interrupts while booting and 250 per devctl detach/attach of xhci0, against 43 with MSI. A test build of bhyve that logs the line, the events and the IMAN/ERDP writes showed why: when an event arrives while xhci_interrupt() runs (after it wrote IMAN, before it reads the ring), the handler consumes it and writes ERDP with EHB set, which in bhyve also clears IMAN.IP, but the line stays up. The next interrupts find IMAN.IP clear, so the handler never writes IMAN and the line stays up until the next event, e.g. for 16 ms during boot. Diff 2 re-evaluates the line after that ERDP write too (one line).

guest with xhci in slot 5, xhci_interrupt() calls	diff 1	diff 2	MSI
boot to login (vmstat -i)	21,820	27	42
devctl detach xhci0; devctl attach pci0:0:5:0 (dtrace)	249	28	44
50 VNC pointer moves while reading /dev/input/event4 (dtrace, 20 s)	100	100	100
line raised by bhyve during boot / ERDP write found the line up	29 / 16	43 / 16	(99 MSIs)

With diff 2 there are never more handler calls than times bhyve raised the line; several events while the line is up give one interrupt, which is why INTx needs fewer calls than MSI (bhyve sends an MSI for every event batch). Cases 1 to 3 were rerun with diff 2: slot 10 sharing pin 23 with ahci0 boots, 0 interrupts per second idle, 105 handler calls for 50 pointer moves; slot 5 idle 0 per second.

Also checked: with MSI enabled (the default) the guest behaves as before (42 interrupts during boot, the tablet attaches and delivers pointer events). bhyve builds with and without WITH_BHYVE_SNAPSHOT.

Scripts: run8.sh KIND SLOT starts the guest (KIND picks the bhyve binary, SLOT the xHCI's slot); xhcirate.sh prints the interrupts per second of xhci0/ahci0 from vmstat -i in the guest and the guest's VM exits per second from bhyvectl --get-stats.

run8.sh

#!/bin/sh
# run8.sh KIND [XSLOT]: guest g1 booted with the UEFI firmware, an xHCI with a USB tablet in slot
# XSLOT (default 10: same IOAPIC pin as ahci0 in slot 2) and a frame buffer for VNC on :5900.
K=${1:-xhci}; X=${2:-10}
bhyvectl --destroy --vm=g1 2>/dev/null
daemon -o /root/bhyve.log -p /root/bhyve.pid /root/kit/$K/bhyve -c 2 -m 2G -H -A -s 0,hostbridge -s 2,ahci-hd,/root/guest.raw \
    -s 3,e1000,tap0 -s $X,xhci,tablet -s 29,fbuf,tcp=127.0.0.1:5900,w=800,h=600 -s 31,lpc -l com1,tcp=127.0.0.1:4567 \
    -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd g1

[wanpengqian_gmail.com]

Also re-evaluate the line after a write to ERDP that clears EHB: bhyve clears IMAN.IP there, and without it the line stayed up after the guest acknowledged an event that arrived while its handler ran (thousands of interrupts with nothing pending; see the Test Plan).