MFC SR-IOV to 10-STABLE

[kbowling]

Summary

This is an attempt to bring back the SR-IOV infrastructure to 10-STABLE. The following commits were inspected:
r279442
r279446
r279447
r279448
r279449

• changes uint8_t pm_reg; to uint16_t pm_reg; in struct pci_map, not sure why seems like an ABI break though, leaving as is until clarified

r279450
r279452
r279453
r279466
r279868
r283670

Test Plan

1. buildworld/buildkernel.
2. Bring back ixgbe PCI_IOV changes in another review and test them on top of this

[kbowling]

I haven't build tested this yet, will in a bit. Off the bat, there may be nvlist changes that need to come back. But I'd appreciate basic review of the commits selected and ABI confirmation.

[rstone]

nvlist not being backmerged yet was why I haven't yet MFC'ed SR-IOV. That's long overdue, though. It was originally promised to me that would happen for 10.1-RELEASE.

[kbowling]

Note to self, this is missing ioctl rc.d https://people.freebsd.org/~rstone/patches/iov/0019-Add-an-rc.d-script-to-invoke-iovctl-8-during-boot.patch

[kbowling]

Adding iovctl rc stuff from HEAD

[kbowling]

This applies cleanly on top of D4249 which brings back nvlist and requires it.

I brought back some missed headers, r279443, build changes for 10-STABLE and have shaken out all the build errors.

@rstone can you take a look?

Any help on what to do with r279449:

• changes uint8_t pm_reg; to uint16_t pm_reg; in struct pci_map, not sure why seems like an ABI break though, leaving as is until clarified

[kbowling]

I've tested this with my ixgbe MFC insofar as I can create both non-passthrough ixv interfaces, and can also create and pass through to bhyve interfaces that come up as ixv.

Passthrough requires setting hw.dmar.enable="1" in /boot/loader.conf

I created /etc/iovctl.conf lile:

PF {
 device : "ix1";
 num_vfs : 2;
}

DEFAULT {
 passthrough :  true;
}

VF-0 {
 passthrough : false;
}

And loaded it with iovctl. However, in both passthrough and non, the interface comes up as 'no carrier' and I haven't figured out how to get past that. I don't know if the same would happen on CURRENT or not.

[kbowling]

@rstone it looks like my libnv and ixgbe patches will land shortly after some massaging. It would be cool to land this so 10.3 ships with it. Any thoughts?

[ngie]

r279441 seems to be missing from the list. I'm trying to figure out how to backport it (haven't gotten there yet..)

[kbowling]

@ngie from what I could tell everything was ABI safe (static functions, etc) aside from my question about the struct pci_map change.

[rstone]

Changing pm_reg to uint16_t was done because the SR-IOV BARs are located outside of the 256-byte PCI configuration space (they are in the extended config space).

I can't remember if anything other than pciconf -b cares about that offset, though. I also can't comment on whether this is part of the stable ABI or not.

[ngie]

I've pulled in everything but r283670 .. there are some conflicts that need to be resolved here with manpages and ObsoleteFiles.inc

[jhb]

In D4232#100926, @rstone wrote:

Changing pm_reg to uint16_t was done because the SR-IOV BARs are located outside of the 256-byte PCI configuration space (they are in the extended config space).

I can't remember if anything other than pciconf -b cares about that offset, though. I also can't comment on whether this is part of the stable ABI or not.

pm_reg is only used for the in-kernel API. The user-facing API for pciconf -lb is struct pci_bar_io which uses an 'int' for pbi_reg. I don't think anything outside of the PCI code uses struct pci_map.