bhyve: e82545: pause the device for a snapshot

[wanpengqian_gmail.com]

Summary

The device had no pause method, so it went on receiving and sending
while a snapshot was taken. The memory of the guest is saved before the
state of the device: a receive descriptor filled in between is not in
the saved memory, but the saved RDH counts it. After the restore the
guest finds no completed descriptor and never moves RDT, while the
device waits for free descriptors: the restored guest receives nothing
any more. The snapshot also recorded what the transmit thread was doing
(esc_tx_active); restored as 1, it kept e82545_tx_start() from waking
the thread, and the guest could not send either.

Add pause and resume methods. A pause stops the receive events of the
backend and waits until the receive callback and the transmit thread are
idle; the resume starts both again and wakes the transmit thread for the
packets the guest has queued.

While here, check the receive descriptor ring against its own length in
e82545_snapshot(), not that of the transmit ring.

Signed-off-by: Wanpeng Qian <[e-mail]>
Sponsored by: keelos.dev

Test Plan

main (f958aa7e7) with WITH_BHYVE_SNAPSHOT. FreeBSD 16.0-CURRENT guest with two vCPUs, started with bhyveload: AHCI root disk, -s 3,e1000,tap0 (the guest's em0 uses INTx), an NVMe data disk (D60234).

The test: the host floods the guest with pings (ping -f), bhyvectl --suspend two seconds into the flood, then bhyve -r; 15 seconds later the host tries an ssh command in the guest and looks at the CPU time of the two vCPU threads. Repeated in a loop (lintrtest.sh below; run5.sh starts or restores the guest).

Before (with D60239, which a restore of this guest needs as well): 0 of 5 restored guests answer. Their vCPUs are idle and the guest is alive on its serial console, but em0 receives nothing: the host's tap0 counts the packets it sends, em0's input count stays (788 before and after a ping from the guest; the guest's own pings arrive on tap0).

After (with D60239): 16 restores, no guest without network. One of the 16 did not answer for another reason, its AHCI disk timed out; that is D60240. With all three, 30 of 30 restored guests answer.

Also built without WITH_BHYVE_SNAPSHOT.

lintrtest.sh

#!/bin/sh
# lintrtest.sh N: on the host, N times: suspend the guest g1 while it is flooded with pings (its
# e1000 uses INTx, so the line is asserted most of the time), restore it, and see whether it answers.
# The guest is the one run5.sh starts (bhyveload, AHCI root disk, NVMe data disk). KIND=... picks
# the bhyve of the restore and of the restarts.
export LD_LIBRARY_PATH=/root/kit/fix
K=${KIND:-fix}
ok=0
for i in $(seq 1 ${1:-5}); do
	rm -f /root/ckp/*
	(ping -f -t 6 10.9.0.2 > /dev/null 2>&1 &)
	sleep 2
	/root/kit/fix/bhyvectl --suspend=/root/ckp/s.ckp --vm=g1 2>/dev/null
	for n in $(seq 1 60); do pgrep -q bhyve || break; sleep 1; done
	pkill nc; pkill ping; sleep 2
	(/root/run5.sh $K /root/ckp/s.ckp > /dev/null 2>&1 < /dev/null &)
	sleep 15
	cpu=$(top -b -H -p $(cat /root/bhyve.pid) 2>/dev/null | grep vcpu | sed 's/.* \([0-9.]*%\) bhyve.*/\1/' | tr '\n' ' ')
	if timeout 15 /root/gs true 2>/dev/null; then
		echo "round $i: the guest answers (vCPU threads at $cpu)"; ok=$((ok + 1))
	else
		echo "round $i: the guest does NOT answer (vCPU threads at $cpu)"
		kill $(cat /root/bhyve.pid); sleep 2; pkill nc
		(/root/run5.sh $K > /dev/null 2>&1 < /dev/null &); sleep 45
		for n in $(seq 1 40); do timeout 8 /root/gs true 2>/dev/null && break; sleep 3; done
	fi
done
echo "$ok of ${1:-5} restores ok"