ixgbe: improve MDIO performance by reducing semaphore/IPC delays

[adrian]

Summary

Each MDIO transfer takes 8ms, and all of that is spent in a set
of DELAY() calls, which is consuming a LOT of CPU.

• Change the timeout in ixgbe_hic_unlocked() - doing IPC to the on-chip firmware - to a 1uS delay and bump timeout appropriately. This seems to finish in a couple of microseconds on my local test devices.

• Change the 2ms sleep in ixgbe_release_swfw_sync_X540() to 2ms for EEPROM/flash, and 10uS otherwise. 10uS seems to work fine on my local testing, but the config EEPROM doesn't read right without this extra delay.

The first change shouldn't change the driver behaviour, but the
latter change is more intrusive and needs some wider testing.
(My guess here is that there SHOULD have been some completion
check somewhere in the EEPROM/flash IO path, and these sleeps
are masking them.)

Locally tested:

• C3558 (Denverton) w/ X553 backplane

[kbowling]

The first change seems sensible.

For the second change, It would be nice to have some citation on the timing or behavior (for instance a data sheet, another driver) or corroboration from an intel insider that this is thought to be ok. My own curiosity.. what is the delay even doing after the swfw release, are we missing or working around some mmio sync, or is the fw side not obeying memory ordering? Maybe we can just get rid of it.. inquiring minds want to know :)

[kgalazka]

sys/dev/ixgbe/ixgbe_x540.c

882

I've found a vouge note that this delay was added to prevent some issue with link establishment after a HW reset. It doesn't explain though why it was needed here, instead of e.g. somewhere in ixgbe_reset_hw_X540. Let me dig a bit more.

[adrian]

sys/dev/ixgbe/ixgbe_x540.c

882

thanks! have you found anything yet? Can you say which chipset(s) it was trying to fix?

[adrian]

rebase

[adrian]

swizzle when it's done to explicitly do it for mdio/management;
add comments in the code.