vmm ppt: Enable busmastering and BAR decoding while a device is assigned

[jhb]

Summary

Fixes: f44ff2aba2d6 bhyve: Treat the COMMAND register for PCI passthru devices as emulated
Sponsored by: Chelsio Communications

[jhb]

So this is actually a bit worse than I had imagined. *sigh* The ppt(4) driver doesn't actually map its BARs (so does not ensure that MEMEN and/or PORTEN are set), and the ioctls used via /dev/pci to do I/O explicitly depend on that and would break if we did start allocating resources. I would rather have ppt(4) be more intentional and be responsible for mapping its BARs (right now memory BARs are only mapped by luck if the host firmware feels like it, the driver doesn't ensure they are allocated). One thing I considered doing earlier but did not do is establish IOMMU mappings for BARs not by passing in the HPA, but by passing in a BAR, offset, and length, and then ppt(4) could ensure the BAR was allocated. This might break the MSI-X shared BAR case though with the current /dev/pci approach. I really would much rather we had a /dev/ppt0 device and that all operations were done via ioctls on that. Some of them get a bit tricky because for assign you do need a way to get to the VM.

Anyway, my other problem is I'm currently dealing with a temperamental device that takes a while to FLR, and the end result is that the register state gets lost when the FLR takes too long. This happened to work before when bhyve was trying to set the command register from userspace, but it's really not ideal. I might be able to do this slightly hackier though if I enable MEMEN when an IOMMU mapping is created.

[corvink]

sys/amd64/vmm/io/ppt.c

432

Should we save the initial busmaster state and restore it here? Anyways, I'm unsure whether we need this here at all. We perform an FLR/power reset on ppt_pci_reset two lines below. Don't know if it resets busmastering too.

[jhb]

I have found a simpler approach for my MEMEN requirement I mentioned earlier that I will upload later today. Basically, I walk the BARs of the device during ppt_assign_device, and if it has any memory BARs I enable MEMEN in the command register at the same place we enable bus mastering. IO BARs are implicitly enabled by the /dev/pci ioctl to access the BAR.

sys/amd64/vmm/io/ppt.c

432

But the pci_save_state/pci_restore_state around the FLR means we turn bus mastering back on since it would be in the saved command register value.

[jhb]

Handle MEMEN

[jhb]

sys/amd64/vmm/io/ppt.c

432

Possibly this should also disable I/O port decoding during unassign?

[jhb]

sys/amd64/vmm/io/ppt.c

182–183

Note that we disable busmastering during detach. I wonder if we want to disable it during attach instead so that it stays disabled after attach until it is assigned to a VM?

[jhb]

More thoroughly update and coalesce writes

[jhb]

I've tested this with a temperamental device whose FLR tends to take a bit long and it works fine across guest reboots.

[corvink]

sys/amd64/vmm/io/ppt.c

418

It's a bit confusing that we don't do that for IO BARs too. Would be a good idea to add a command explaining why we don't need to enable IO encoding here.

[jhb]

sys/amd64/vmm/io/ppt.c

418

Hmm, I could just enable I/O decoding so long as there is an I/O BAR. It won't hurt and might be more intuitive.

[jhb]

Enable I/O decoding as well