virtio: remove byte conversion function calls in modern pci

[timo.voelker_fh-muenster.de]

Summary

Remove function calls in virtio_pci_modern.c that convert the byte order. This fixes a bug reported here:

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=294706

Reading a value from the PCI bus with bus_read_x() (with x > 1) requires a byte conversion on a big-endian system, because the PCI bus uses little endian. However, this conversion is done by the bus_space_read_x() functions, which are used (bus_read_x() are macros defined in sys/sys/bus.h that map to bus_space_read_x()). Thus, another byte conversion in virtio_pci_modern.c is incorrect.

The byte order the PCI device uses is irrelevant. For instance, in a big-endian guest, the VirtIO PCI device uses big endian in legacy mode and little endian in modern mode. However, since the data are transferred over the PCI bus, single values arrive in little endian no matter how the PCI device stores them.

QEMU tends to provide a transitional VirtIO PCI device, which means it can run in both modes, legacy and modern. In a FreeBSD guest, the loader tunable hw.virtio.pci.transitional decides which mode is used. If it is 0, legacy mode is used. If it is 1, modern mode is used. The default value had been 0 until D55894, which changed the default to 1. Since then, modern mode is probably used more often, and that discovered this long-existing bug.

Test Plan

I tested this with a FreeBSD 16-CURRENT guest in three environments and got the same result: modern mode did not work until applying this patch.

1
Host: MacBook with Apple Silicon CPU and macOS (Little Endian).
Hypervisor: QEMU (UTM)
Guest config:
-machine pseries-10.0
(CPU default)
-smp cpus=1,sockets=1,cores=1,threads=1
-device virtio-net-pci,...

2
Host: Raptor Blackbird with Power9 CPU and Linux Debian (Little Endian).
Hypervisor: QEMU (Virt-Manager)
Guest config:
-machine pseries-10.0,usb=off,cap-hpt-max-page-size=4k,dump-guest-core=off,memory-backend=ppc_spapr.ram
-accel kvm
-cpu POWER9
-smp 16,sockets=16,cores=1,threads=1
-device {"driver":"virtio-net-pci",...}

3
Host: Raptor Blackbird with Power9 CPU and FreeBSD 16-CURRENT (Big Endian).
Hypervisor: QEMU (qemu-system-ppc64)
Guest config:
qemu-system-ppc64 -name bsdppc-vm -cpu POWER9 -smp 16 -M pseries-10.0 -m 8G -drive file=/home/msvoelker/bsdppc-vm/disk.qcow2,if=virtio,format=qcow2 -netdev tap,id=net0,ifname=tap0,script=no,downscript=no -device virtio-net-pci,netdev=net0 -nographic

[adrian]

Neat! So hm what devices are on the virtio pci bus? storage, net, anything else? I'm just trying to imagine the device testing map for this change.

Thanks for chasing this down!

[timo.voelker_fh-muenster.de]

In D57392#1315045, @adrian wrote:

Neat! So hm what devices are on the virtio pci bus? storage, net, anything else? I'm just trying to imagine the device testing map for this change.

Thanks for chasing this down!

I'm not a VirtIO expert. As far as I know, every VirtIO device can use the PCI bus to attach to the system.

In sys/dev/virtio, each subfolder stands for one implemented VirtIO driver implementation (except pci). QEMU can emulate the corresponding device. On my macOS, I get this list

qemu-system-aarch64 -device help | grep virt.\*pci\"
name "virtio-9p-pci", bus PCI, alias "virtio-9p"
name "virtio-blk-pci", bus PCI, alias "virtio-blk"
name "virtio-scsi-pci", bus PCI, alias "virtio-scsi"
name "virtio-net-pci", bus PCI, alias "virtio-net"
name "virtio-keyboard-pci", bus PCI, alias "virtio-keyboard"
name "virtio-mouse-pci", bus PCI, alias "virtio-mouse"
name "virtio-multitouch-pci", bus PCI
name "virtio-serial-pci", bus PCI, alias "virtio-serial"
name "virtio-tablet-pci", bus PCI, alias "virtio-tablet"
name "virtio-gpu-pci", bus PCI, alias "virtio-gpu"
name "virtio-sound-pci", bus PCI, alias "virtio-sound", desc "Virtio Sound"
name "virtio-balloon-pci", bus PCI, alias "virtio-balloon"
name "virtio-crypto-pci", bus PCI
name "virtio-iommu-pci", bus PCI, alias "virtio-iommu"
name "virtio-rng-pci", bus PCI, alias "virtio-rng"

I focused my test on virtio-net-pci but used some other VirtIO devices (mostly virtio-blk-pci) as well. If you have the chance to test other VirtIO devices, that would be great.

[tuexen]

In D57392#1315323, @timo.voelker_fh-muenster.de wrote:

In D57392#1315045, @adrian wrote:

Neat! So hm what devices are on the virtio pci bus? storage, net, anything else? I'm just trying to imagine the device testing map for this change.

Thanks for chasing this down!

I'm not a VirtIO expert. As far as I know, every VirtIO device can use the PCI bus to attach to the system.

In sys/dev/virtio, each subfolder stands for one implemented VirtIO driver implementation (except pci). QEMU can emulate the corresponding device. On my macOS, I get this list

qemu-system-aarch64 -device help | grep virt.\*pci\"
name "virtio-9p-pci", bus PCI, alias "virtio-9p"
name "virtio-blk-pci", bus PCI, alias "virtio-blk"
name "virtio-scsi-pci", bus PCI, alias "virtio-scsi"
name "virtio-net-pci", bus PCI, alias "virtio-net"
name "virtio-keyboard-pci", bus PCI, alias "virtio-keyboard"
name "virtio-mouse-pci", bus PCI, alias "virtio-mouse"
name "virtio-multitouch-pci", bus PCI
name "virtio-serial-pci", bus PCI, alias "virtio-serial"
name "virtio-tablet-pci", bus PCI, alias "virtio-tablet"
name "virtio-gpu-pci", bus PCI, alias "virtio-gpu"
name "virtio-sound-pci", bus PCI, alias "virtio-sound", desc "Virtio Sound"
name "virtio-balloon-pci", bus PCI, alias "virtio-balloon"
name "virtio-crypto-pci", bus PCI
name "virtio-iommu-pci", bus PCI, alias "virtio-iommu"
name "virtio-rng-pci", bus PCI, alias "virtio-rng"

I focused my test on virtio-net-pci but used some other VirtIO devices (mostly virtio-blk-pci) as well. If you have the chance to test other VirtIO devices, that would be great.

When using an arm64 VM with UTO on macOS, I get

tuexen@head:~ % dmesg | grep "VirtIO PCI"
virtio_pci0: <VirtIO PCI (modern) Network adapter> port 0x140-0x15f mem 0x1006e000-0x1006efff,0x10058000-0x1005bfff at device 1.0 on pci0
virtio_pci1: <VirtIO PCI (modern) Network adapter> port 0x120-0x13f mem 0x1006d000-0x1006dfff,0x10054000-0x10057fff at device 2.0 on pci0
virtio_pci2: <VirtIO PCI (modern) GPU adapter> mem 0x1006c000-0x1006cfff,0x10050000-0x10053fff at device 3.0 on pci0
virtio_pci3: <VirtIO PCI (modern) Block adapter> mem 0x1006b000-0x1006bfff,0x1004c000-0x1004ffff at device 7.0 on pci0
virtio_pci4: <VirtIO PCI (modern) Console adapter> port 0xc0-0xff mem 0x1006a000-0x1006afff,0x10048000-0x1004bfff at device 8.0 on pci0
virtio_pci5: <VirtIO PCI (modern) Entropy adapter> port 0x100-0x11f mem 0x10069000-0x10069fff,0x10044000-0x10047fff at device 9.0 on pci0
virtio_pci6: <VirtIO PCI (modern) Balloon adapter> port 0x80-0xbf mem 0x10068000-0x10068fff,0x10040000-0x10043fff at device 10.0 on pci0

These work fine with that patch. Which is expected since guest and host are little endian, but it is a data point.