Incorporate feedback from jhb.

I went over all of this and related revisions recently. The high level comments are here and the rest will go next to the code they apply to.

{F86253090} I uploaded an attachment with all the changes I had to make to get this working. Please incorporate it into this revision.

This PF change looks correct but there needs to be matching change in the VF driver that reads the VLAN settings and acts on it.

In D43166#986878, @rscheff wrote:

In D43166#986875, @np wrote:

b) If FIN/PSH are set in tcp flags then they will be set in the flags for the last segment only.
c) If CWR is set in the TCP hdr it is set in the hdr of the first segment only. The ECE bit is copied to all segments.
d) If there are IP options they are copied into each segment unaltered. This means if TCP timestamp option is in use the chip will use the same timestamp in all the segments for the TSO.

Thanks Navdeep!

Is there a r/w mask register similar to what Intel NICs offer available in the cxgbe hardware, to modify specifically the CWR behavior?

In D43166#984624, @gallatin wrote:

OK, I'm sorry, I was not aware of AccECN and its desired behavior of setting CWR on all segments.

In any case, a system wide tunable is probably not the correct approach. We would want to have the driver set a bit to advertise which ECN modes it supports. If there is hardware that supports both, and the driver cannot determine easily from the packet itself if normal ECN or AccECN should be used, we probably also need to hint to the driver which ECN mode should be used.

I'd like to see some input from NIC vendors here, so I'm glad @np is on the review.