pci: wait for a function to answer again after a reset

[nick_spun.io]

Summary

A function that has not finished resetting answers configuration
requests with all ones. pcie_flr() read PCIER_DEVICE_STA once after a
fixed sleep and so reported a transaction that does not exist:

pci0:9:0:0: Transactions pending after FLR!

pci_power_reset() waited only the 10 ms that pci_set_powerstate()
applies leaving D3.

The stale report is the smaller problem: callers restore configuration
state as soon as the reset returns, and a restore into a function that
is not answering is lost, leaving the BARs and command register clear.

Poll for the function to answer before returning.

PR: 296662

[kbowling]

sys/dev/pci/pci.c

7030

I wonder if this can be more link https://reviews.freebsd.org/D59477 PCIe aware timeout

[kbowling]

sys/dev/pci/pci.c

7026

Healthy SR-IOV VFs can return 0xffff there, use Command instead.

7030

On deeper inspection these should be logically similar (same units, etc) but the timeout formula will be unique for each.

7247

There needs to be some differentiation here, and I don't think simply flipping this to false solves it.. if the device isn't ready the caller needs the timeout propagated.

[jhb]

sys/dev/pci/pci.c

7026

Hummm, that is a bit sketchy as normally the PCI spec is very clear when doing bus scans, etc. you have to query vendor ID first and ignore all 1's. In this case we are hoping for the device to "come back" after a reset vs scanning an unknown DBSF so it is slightly different. Maybe check the dinfo to see if it is a VF and only do sketchy things for a VF explicitly? For that you could just have a
u_int reg that you pass here to pci_read_config that is PCIR_VENDOR for PFs and PCIR_COMMAND for VFs.

7031

This might be easier to read (less duplication at least):

todo = MIN(max_delay, 10);
pause_sbt("pcirdy", todo * SBT_1MS, 0, C_HARDCLOCK);
max_delay -= todo;

7243

Note that the reason we waited for 100ms and did a single read here is that is what the spec said (devices were required to come back in 100ms or less). That said, I'm sure there are problem children in the field. In my experience, when this fails, the device is out to lunch entirely and never comes back until a full reset.

7247

We could flip this to an int return and use ETIMEDOUT vs EINVAL (or maybe EOPNOTSUPP)