bhyve: e82545: deassert the interrupt when its last cause is masked

[wanpengqian_gmail.com]

Summary

A write to IMC only cleared the bits in the mask. If the device was
asserting INTx for a cause that the write masked, the line stayed up
until the guest cleared the cause in ICR. The hardware deasserts INTx
as soon as no unmasked cause is left, and a guest may rely on that:
Windows masks all causes and unloads its driver when it shuts down, and
then spends its time in the handlers of whatever shares the interrupt.

Do what the comment in the code asked for.

Signed-off-by: Wanpeng Qian <[e-mail]>
Sponsored by: keelos.dev

Test Plan

main (f958aa7e7), FreeBSD 16.0-CURRENT guest started with bhyveload. The guest has 16 e1000 devices, so that the last one (slot 18) is the 17th device asking for INTx and shares IOAPIC pin 16 with ahci0, and it boots with hw.pci.enable_msi=0 and hw.pci.enable_msix=0, so that ahci0 has a handler on that pin.

In the guest the driver is detached from that e1000 and e1k (below) writes its registers through /dev/mem: IMS and ICS to assert INTx, then IMC = 0xffffffff, then a read of ICR. The numbers are the interrupts per second on irq16 over 3 seconds (vmstat -i); ahci0's handler runs each time and finds nothing to do.

	before	after
idle	0	0
INTx asserted (IMS = ICS = RXT0)	1698	1798
all causes masked (IMC = 0xffffffff)	1758	0
ICR read	0	0

How it was found (FreeBSD 14.5): a Windows guest with an e1000 did not get past "Restarting". Windows masks every cause and unloads its driver; the line stayed asserted, no handler claimed it, and the vCPU took the level triggered interrupt again after every EOI. With this change it restarts.

e1k.c

/*
 * e1k: poke the interrupt registers of an e1000 that has no driver attached.
 *
 *	e1k BAR0-ADDRESS assert|mask|ack
 *
 * assert: unmask RXT0 (IMS) and raise it (ICS): the device asserts INTx
 * mask:   mask every cause (IMC)
 * ack:    read ICR, which clears the causes
 */
#include <sys/types.h>
#include <sys/mman.h>
#include <err.h>
#include <fcntl.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>

#define	ICR	0xc0
#define	ICS	0xc8
#define	IMS	0xd0
#define	IMC	0xd8
#define	RXT0	0x80

int
main(int argc, char **argv)
{
	volatile uint32_t *reg;
	int fd;

	if (argc != 3)
		errx(1, "usage: e1k BAR0-ADDRESS assert|mask|ack");
	if ((fd = open("/dev/mem", O_RDWR)) < 0)
		err(1, "/dev/mem");
	reg = mmap(NULL, 0x20000, PROT_READ | PROT_WRITE, MAP_SHARED, fd,
	    strtoull(argv[1], NULL, 0));
	if (reg == MAP_FAILED)
		err(1, "mmap");
	if (strcmp(argv[2], "assert") == 0) {
		reg[IMS / 4] = RXT0;
		reg[ICS / 4] = RXT0;
	} else if (strcmp(argv[2], "mask") == 0)
		reg[IMC / 4] = 0xffffffff;
	else if (strcmp(argv[2], "ack") == 0)
		printf("ICR %#x\n", reg[ICR / 4]);
	else
		errx(1, "unknown command");
	return (0);
}

intrtest.sh

#!/bin/sh
# intrtest.sh: the e1000 at pci0:0:18:0 shares its interrupt line with ahci0. Without a driver
# on the e1000: the rate of that interrupt when the e1000 asserts INTx, after all causes are
# masked with IMC, and after ICR is read.
sel=pci0:0:18:0
devctl detach $sel 2>/dev/null
pciconf -w -h $sel 0x04 0x0002
bar=$(pciconf -lb $sel | awk '/bar.*\[10\]/ {print $6}' | sed 's/,//')
irq=$(vmstat -i | awk '/ahci0/ {print $1}')
rate() { a=$(vmstat -i | awk '/ahci0/ {print $3}'); sleep 3; b=$(vmstat -i | awk '/ahci0/ {print $3}'); echo "$1: $(( (b - a) / 3 )) interrupts per second on $irq"; }
echo "e1000 at $sel, BAR0 $bar"
rate "idle                  "
/root/e1k $bar assert
rate "INTx asserted         "
/root/e1k $bar mask
rate "after IMC = 0xffffffff"
/root/e1k $bar ack
rate "after reading ICR     "