vmm: save the event waiting for injection in a vCPU's snapshot

[wanpengqian_gmail.com]

Summary

vmx_run() injects a pending interrupt before it checks whether the vCPU
has to leave the loop: vmx_inject_interrupts() lets the vlapic accept
the interrupt (the ISR bit is set) and writes it into the VM-entry
interruption information field. bhyve stops the vCPUs for a snapshot
with a debug suspend, which is one of those checks, so a vCPU can be
stopped with an accepted interrupt that has not reached the guest yet.
Without a snapshot it is injected at the next VM entry.

vmx_vcpu_snapshot() did not save the VM-entry event injection fields.
After a restore the guest never got that interrupt, never wrote the EOI,
the ISR bit stayed set, and the vCPU took no interrupt of the same or a
lower priority any more: its timer, then every device.

Save VM-entry interruption information, exception error code and
instruction length. This changes the format of the snapshot.

Signed-off-by: Wanpeng Qian <[e-mail]>
Sponsored by: keelos.dev

Test Plan

main (f958aa7e7) with WITH_BHYVE_SNAPSHOT. FreeBSD 16.0-CURRENT guest with two vCPUs, started with bhyveload: AHCI root disk, -s 3,e1000,tap0 (the guest's em0 uses INTx), an NVMe data disk (D60234).

The test: the host floods the guest with pings (ping -f), bhyvectl --suspend two seconds into the flood, then bhyve -r; 15 seconds later the host tries an ssh command in the guest and looks at the CPU time of the two vCPU threads. Repeated in a loop (lintrtest.sh below; run5.sh starts or restores the guest).

(With D60239 and D60241, without which this test fails for other reasons.)

Before: 2 of 20 restored guests did not answer. Their vCPUs were idle and the AHCI root disk was gone: the completion interrupt of a command had been accepted by the vlapic but not injected when the vCPU was stopped, and was not in the snapshot

ahcich0: Timeout on slot 14 port 0
ahcich0: is 00000008 cs 00000000 ss 00000000 rs 00004000 tfd 50 serr 00000000 cmd 0001ce17
(ada0:ahcich0:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 40 33 d4 4c 40 00 00 00 00 00 00
(ada0:ahcich0:0:0:0): CAM status: Command timeout
(aprobe0:ahcich0:0:0:0): Error 5, Retries exhausted
ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
ada0: <BHYVE SATA DISK 001> s/n BHYVE-421E-8AAE-7E36 detached

The device has completed the command (cs and ss are 0) and raised the interrupt (is), the driver still waits for it (rs), and the recovery fails too because no interrupt of that priority gets through any more.

After: 30 of 30 restored guests answer.

How it was found (FreeBSD 14.5): about one in five restores of an idle guest came back half dead: a vCPU sat in cpu_idle_acpi() with interrupts enabled and the timer's vector (239) in service in its local APIC, in the snapshot already. A pause request wakes a vCPU that is in HLT, and if its timer is due at that moment the interrupt is accepted on the way out of vmx_run().

Only the Intel side is changed. I have no AMD machine to look at svm_vcpu_snapshot() with.

lintrtest.sh

#!/bin/sh
# lintrtest.sh N: on the host, N times: suspend the guest g1 while it is flooded with pings (its
# e1000 uses INTx, so the line is asserted most of the time), restore it, and see whether it answers.
# The guest is the one run5.sh starts (bhyveload, AHCI root disk, NVMe data disk). KIND=... picks
# the bhyve of the restore and of the restarts.
export LD_LIBRARY_PATH=/root/kit/fix
K=${KIND:-fix}
ok=0
for i in $(seq 1 ${1:-5}); do
	rm -f /root/ckp/*
	(ping -f -t 6 10.9.0.2 > /dev/null 2>&1 &)
	sleep 2
	/root/kit/fix/bhyvectl --suspend=/root/ckp/s.ckp --vm=g1 2>/dev/null
	for n in $(seq 1 60); do pgrep -q bhyve || break; sleep 1; done
	pkill nc; pkill ping; sleep 2
	(/root/run5.sh $K /root/ckp/s.ckp > /dev/null 2>&1 < /dev/null &)
	sleep 15
	cpu=$(top -b -H -p $(cat /root/bhyve.pid) 2>/dev/null | grep vcpu | sed 's/.* \([0-9.]*%\) bhyve.*/\1/' | tr '\n' ' ')
	if timeout 15 /root/gs true 2>/dev/null; then
		echo "round $i: the guest answers (vCPU threads at $cpu)"; ok=$((ok + 1))
	else
		echo "round $i: the guest does NOT answer (vCPU threads at $cpu)"
		kill $(cat /root/bhyve.pid); sleep 2; pkill nc
		(/root/run5.sh $K > /dev/null 2>&1 < /dev/null &); sleep 45
		for n in $(seq 1 40); do timeout 8 /root/gs true 2>/dev/null && break; sleep 3; done
	fi
done
echo "$ok of ${1:-5} restores ok"