When we run behyve without passthru host devices can DMA to all VMs
When we add a passthru device that is no longer the case as we are
removing the weird VM memory from the host domain.
Now if we are using physical host devices from bhyve user-mode,
e.g., physical disks /dev/ada<n> /dev/da<n> with AHCI or over USB (umass)
then the guest physical address (GPA) is passed to the bhyve user mode
process which in turn will translate it to its virtual mappin in the
host space and pass it to the preadv syscall which will then go to,
e.g. ahci and use the then mapped address for the physical device.
That address is a guest mapped physical address which is no longer valid
in the host domain (as we removed the weird VM mappings from that) and
as a result the DMA will fail.
This will not result in an error in AHCI, or the syscall returned to
bhyve but the passed buffer will stay untouched essentially resulting
in non-working IO in the guest.
It is unclear why initially the mapped guest address space was removed
from the host domain, but it does not seem to give any extra security
for the host or guest compared to a non-passthru VM.
In conclusion rather than adding an extra bounce layer (as initially
drafted for a proof of concent in D34535), keep the GPA mappings valid
in the host domain and allow IO to work. That solves a long-standing
problem when using passthru devices and physical disks in the same VM.
With lots of help and patience from: grehan