bhyve: Add virtio modern support handling

[khng]

Summary

This adds virtio modern support handling to bhyve.
pci_emul_add_capability is exposed to support PCIY_VENDOR-type
virtio modern capabilities.

Current virtio devices are slightly changed to adopt
vi_pci_cfgread/vi_pci_cfg_write.

Virtio modern is not exposed to device emulation code currently.

Sponsored by: The FreeBSD Foundation

[khng]

Rebased after virtio-input support was added

[khng]

VS_LOCK/VS_UNLOCK in vi_interrupt causes deadlock in case MSI-X is disabled.

[corvink]

What's the state of this patch? Does it still works?

[novel]

I was looking into getting virtio-input working for Linux guests which apparently do not support legacy virtio input devices.

I was able to rebase this patch onto a fresh -CURRENT, it wasn't particularly straightforward, so I'm not sure I did it properly.

Anyway, I was able to boot a Linux guest, and lspci shows these devices like:

00:04.0 Ethernet controller: Red Hat, Inc. Virtio network device
        Subsystem: Red Hat, Inc. Device 0001
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 64
        Interrupt: pin A routed to IRQ 17
        Region 0: I/O ports at 2080 [size=64]
        Region 1: Memory at c2004000 (32-bit, non-prefetchable) [size=8K]
        Region 2: Memory at 800000000 (64-bit, non-prefetchable) [size=32K]
        Capabilities: [40] MSI-X: Enable+ Count=3 Masked-
                Vector table: BAR=1 offset=00000000
                PBA: BAR=1 offset=00001000
        Capabilities: [4c] MSI: Enable- Count=1/1 Maskable- 64bit+
                Address: 0000000000000000  Data: 0000
        Capabilities: [5c] Vendor Specific Information: VirtIO: CommonCfg
                BAR=2 offset=00000000 size=00001000
        Capabilities: [6c] Vendor Specific Information: VirtIO: Notify
                BAR=2 offset=00001000 size=00002000 multiplier=00001000
        Capabilities: [80] Vendor Specific Information: VirtIO: DeviceCfg
                BAR=2 offset=00003000 size=00001000
        Capabilities: [90] Vendor Specific Information: VirtIO: ISR
                BAR=2 offset=00004000 size=00001000
        Capabilities: [a0] Vendor Specific Information: VirtIO: <unknown>
                BAR=0 offset=00000000 size=00000000
        Kernel driver in use: virtio-pci

...

00:06.0 Input device controller: Red Hat, Inc. Virtio 1.0 input (rev 01)
        Subsystem: Oracle/SUN Device 1100
        Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV+ VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Interrupt: pin A routed to IRQ 19
        Region 1: Memory at c2002000 (32-bit, non-prefetchable) [size=8K]
        Capabilities: [40] MSI-X: Enable- Count=3 Masked-
                Vector table: BAR=1 offset=00000000
                PBA: BAR=1 offset=00001000
        Capabilities: [4c] MSI: Enable- Count=1/1 Maskable- 64bit+
                Address: 0000000000000000  Data: 0000

00:07.0 SCSI storage controller: Red Hat, Inc. Virtio block device
        Subsystem: Red Hat, Inc. Device 0002
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 64
        Interrupt: pin A routed to IRQ 20
        Region 0: I/O ports at 2000 [size=128]
        Region 1: Memory at c2000000 (32-bit, non-prefetchable) [size=8K]
        Region 2: Memory at 800008000 (64-bit, non-prefetchable) [size=16K]
        Capabilities: [40] MSI-X: Enable+ Count=2 Masked-
                Vector table: BAR=1 offset=00000000
                PBA: BAR=1 offset=00001000
        Capabilities: [4c] MSI: Enable- Count=1/1 Maskable- 64bit+
                Address: 0000000000000000  Data: 0000
        Capabilities: [5c] Vendor Specific Information: VirtIO: CommonCfg
                BAR=2 offset=00000000 size=00001000
        Capabilities: [6c] Vendor Specific Information: VirtIO: Notify
                BAR=2 offset=00001000 size=00001000 multiplier=00001000
        Capabilities: [80] Vendor Specific Information: VirtIO: DeviceCfg
                BAR=2 offset=00002000 size=00001000
        Capabilities: [90] Vendor Specific Information: VirtIO: ISR
                BAR=2 offset=00003000 size=00001000
        Capabilities: [a0] Vendor Specific Information: VirtIO: <unknown>
                BAR=0 offset=00000000 size=00000000
        Kernel driver in use: virtio-pci

Virtio-input doesn't work still though:

kernel: virtio-pci 0000:00:06.0: virtio_pci: leaving for legacy driver

That's probably expected as in the original patch I don't see vc_en_modern and vc_hv_caps_modern set for the virtio-input.

When I update the code to set it, bhyve complains with:

vtinput: writing bad value to device_feature_select

Any hints how to update the virtio-input related bits?

FWIW, I pushed a rebased branch on github if somebody wants to experiment https://github.com/novel/freebsd-src/tree/D29708-rebase