powerpc64/powernv: Enable Partitionable Endpoint (PE) support

[tpearson_raptorengineering.com]

Summary

powerpc64/powernv: Enable Partitionable Endpoint (PE) support

This is a fairly major rewrite of the IODA2/IODA3 codebase to enable
PE allocation on a per-bus basis, and to lay the groundwork required
to wire the IODA3 controller into the FreeBSD IOMMU framework.

While the 1:1 DMA mapping is still present, it is now enabled via
a proper TVT (TCE table) with associated setup functions. Likewise,
we no longer place all devices in PE#1, which (among other benefits)
prevents spurious MSI interrupts from interfering with unrelated
devices on the same PHB.

Because the segmented memory model is now in use for 64-bit BARs,
the PE allocation and DMA setup needs to run in a secondary pass
after PCI bus and device resource allocation. This mirrors the
Linux reference code for IODA2/IODA3, and allows further isolation
between PCI devices.

Tested to operate without regressions on an RCS Blackbird system.

Test Plan

Tested on RCS POWER9 hardware (IODA3) and allocations look reasonable / no device regressions noted.

Ideally this should be tested on POWER8; there were some offers on the mailing list for old POWER8 hardware access that could be used here. However, as this patch is foundational for IOMMU support, and most relevant hardware is POWER9, I prefer this merge not be held up waiting for POWER8 testing.

[jhibbits]

Adding the wider powerpc umbrella, so others can take a look as well.

My initial review is just style. Also, make sure your style is consistent within a file. You have some return (foo) and some return foo in the same file, stick with one.

Finally, if you upload the diff from the Phabricator UI, generate the diff via git diff -U99999 (or git show -U999999), so that we can see the full context, instead of just the immediate context.

Aside from the style nits, which are all minor, it looks good, and will be a big improvement over what we have currently. I want a second set of eyes to either review or test this, even if testing is just a power8 qemu powernv VM.

sys/powerpc/powernv/opal_pci.c

621–622

else shares the line with the closing '}' above. There are more of these below.

998

Minor style nit: Wrap these bitwise boolean tests in another set of parentheses, so it'd read more as:
if ((sc->off_sc.sc_range_mask & ((uint64_t)1 << i)) != 0)

Double-check the other conditionals as well.

sys/powerpc/powernv/opal_pci.h

96

Keep style consistent here, '{' at the end of the opening line.

[jhibbits]

sys/powerpc/powernv/opal_pci.c

207

Just remove this whole block, no need to #if 0 it.

[adrian]

I'm in the process of documenting / getting powernv8 and powernv9 qemu guests up and running.
(And I now have a power8 booting freebsd powernv so I can test it on real hardware as well.)

Stay tuned, thanks!