iflib: Add rx_wait_irq configuration option

[kgalazka]

Summary

This is an observation when receiving bulk traffic with LRO enabled. Checking for available RX descriptors at the end of iflib_rxeof() leads to aggregating only 2-3 frames, causes higher CPU utilization, and regresses the latency of large packet processing (e.g. 64K). So we add a sysctl allowing us the flexibility of switching it back to IRQ-driven RX. When enabled, this has shown to improve CPU utilization by about 20% on our X550 line of adaptors.

[stallamr_netapp.com]

This change with re-arm IRQ i.e, rx_wait_irq = 1, also helped NetApp to improve latency in 64K writes.

[markj]

So is this intended to supersede D27344?