bhyve: add suport for other mappings in the MSI-X table page

[bz]

Summary

Intel AX200 seems to map other usable address space into the
same page holding the the MSI-X table (which may be against the spec).
Map the host view of MSI-X Table page of the bar and perform
reads/writes on that for anything not covered by the MSI-X PBA or
the MSI-X Table.

Many thanks to grehan and jhb for helping to figure this out.

Sponsored by: The FreeBSD Foundation

[bz]

There is still an issue with this I believe in case the Table grows larger than 1 page.
now I would hope that we would never see these kinds of devices but I'll leave this and other things to the review.

For others showing the MSI-X Table not in a different bar:

bar   [10] = type Memory, range 64, base 0x9e200000, size 16384, enabled
    cap 11[80] = MSI-X supports 16 messages, enabled
                 Table in map 0x10[0x2000], PBA in map 0x10[0x3000]

and a random read/write from within the same page as the MSI-X Table as seen:

passthru_read: baridx 0 msixtabbar 0 offset 0x2804 size 4 val 0x2affe0 error 0
passthru_write:1071: baridx 0 msixtabbar 0 offset 0x2804 size 4 val 0x28ffe0 error 0

[markj]

In D31241#703457, @bz wrote:

There is still an issue with this I believe in case the Table grows larger than 1 page.
now I would hope that we would never see these kinds of devices but I'll leave this and other things to the review.

I posted a related review to avoid using /dev/mem for this: D31534. Right now it only maps the single page containing the PBA, but I suppose we could just map the entire BAR and use that...?

For others showing the MSI-X Table not in a different bar:

bar   [10] = type Memory, range 64, base 0x9e200000, size 16384, enabled
    cap 11[80] = MSI-X supports 16 messages, enabled
                 Table in map 0x10[0x2000], PBA in map 0x10[0x3000]

and a random read/write from within the same page as the MSI-X Table as seen:

passthru_read: baridx 0 msixtabbar 0 offset 0x2804 size 4 val 0x2affe0 error 0
passthru_write:1071: baridx 0 msixtabbar 0 offset 0x2804 size 4 val 0x28ffe0 error 0

Any idea what this region is used for?

[bz]

Abandon in favour of D32359