Add support for unspecified ranges on ARM64 ThunderX system

[wma_semihalf.com]

Summary

When one is trying to allocate a resource with unspecified range,
read already configured BAR values (by UEFI or whatever).
This is necessary to make VFs working and to allow them to be
properly allocated.

[andrew]

sys/arm64/cavium/thunder_pcie.c

102

Why 4?

470

What is happening here? I don't understand how this works, a comment would be useful to help explain it.

477

You talk about 3 64-bit bars, then use 5 here. Why?

[imp]

I think this is OK, but I don't know much about VF

sys/arm64/cavium/thunder_pcie.c

470

You're supposed to write all f's to the BAR to size it. The bits for the decode lines will be 0's, the address range it occupies are 1's. So this flips the upper bits that are set and adds 1 to get the size. The masking above masks out the low-order bits that contain attributes for the mapping.

477

0, 2 and 4 are valid valid values for 3 64-bit BARs. 5 is the first invalid one.

[emaste]

sys/arm64/cavium/thunder_pcie.c

477

Alternatively, 6 is the 4th 64-bit BAR?

[wma_semihalf.com]

sys/arm64/cavium/thunder_pcie.c

477

PCI supports up to 6 BAR registers. They can be used either as 6x 32-bit regs, or 3x 64-bit. Anyway, PCIR_BAR(5) is the last valid entry inside configuration header, so anything above means we're accessing VFs.