mlx5en: add cqe zipping support

[markb_mellanox.com]

Summary

CQE zipping reduces PCI overhead by coalescing and zipping multiple CQEs into a
single merged CQE. We introduce a new format to be used when the HW compress
CQEs. because of the way this feature works, we need to separate mlx5_cqwq_pop
from mlx5e_get_cqe.

The feature is enabled by default, a sysctl is added if one wants to disable it.

Sponsored by: Mellanox Technologies

[gallatin]

o The device.h change is just whitespace & should probably be removed
o Have you measured the cost in terms of CPU overhead of doing this unzipping on a transmit-mostly workload (eg, busy web server, or even just net/iperf sender)? I'm assuming that it is in the noise, but it would be nice to get confirmation.

[markb_mellanox.com]

In D4598#97286, @gallatin wrote:

o The device.h change is just whitespace & should probably be removed
o Have you measured the cost in terms of CPU overhead of doing this unzipping on a transmit-mostly workload (eg, busy web server, or even just net/iperf sender)? I'm assuming that it is in the noise, but it would be nice to get confirmation.

• The new line in device.h is just something that needs to be fixed so I did it now. I'm also adding struct, enum and a static function to device.h so it's not like I'm touching the file just for the newline change.

• I'll add that the NIC will be zipping CQEs only if it has no idle periods (i.e the PCI bus is the bottleneck). If your're already CPU-bound, the NIC won't be doing any zipping.

• On our testing we almost never saw zipping being done on the transmit side, but we couldn't test with the amount of connections your guys are handling, that's why I added a sysctl to turn it off. I knew you would like to test it.

[• hselasky]

Committed as r292838.