tcp: DSCP Codepoint not being propagated in the rack fast path.

[rrs]

Summary

So it turns out that when you change DSCP codepoint's using the appropriate V6 or V4 socket option, if
the connection you do that on has exchanged data prior to the option set, then you will not get an updated
codepoint in the rack fast path. The solution for this is to have an API twiddle that the socket option code
sets calling into tcp to basically set a flag. This then can be picked up by the appropriate stack and cause
it to "update" its fast send block. Note that we use this method to make sure we only do the changes
when the codepoint changes and only touching fields (t_flags2) in the tcb that the fast path already touches.
This makes sure we don't incur any added overhead.

Test Plan

Get a connection up, exchange some data, and then change the codepoint. Then use
dtrace to validate that you see outbound the proper codepoint.

[rrs]

Fix my DHCP/DSCP typos :)

[rrs]

Let's not mess in ip or ip6 but intercept this in TCP. This also makes it so
we don't try this on UDP.

[glebius]

I'm in the process of writing more generic patch to cover this. I would ask not to commit the intermediate version.

Note that already existing IPV6_USE_MIN_MTU handling in tcp_ctloutput is pretty much the same problem.

For IPv4 options and rack, there is also IP_TTL socket option that (I believe) needs same handling as IP_TOS.

[rrs]

This remix's things so that we create a t_fb function that can be called
by the codepoint changes i.e. the socket option if the transport wants
it and this updates the rack fsb.

[rrs]

In D32600#736198, @glebius wrote:

I'm in the process of writing more generic patch to cover this. I would ask not to commit the intermediate version.

Note that already existing IPV6_USE_MIN_MTU handling in tcp_ctloutput is pretty much the same problem.

For IPv4 options and rack, there is also IP_TTL socket option that (I believe) needs same handling as IP_TOS.

Consider this patch as input to yours.. no worries I am fine with whatever you have in mind.. make the tfb_dscp_change() into
a generic socket option change with option argument etc if thats what you want ;)

[rrs]

I believe this is the more generic approach or at least closer to what Gleb wants. The names
might be wrong but I think it gets at what he has been saying..

[rrs]

Opps one typo to fix

[rrs]

Update the diff with offline comments from Jonathan

[rrs]

Michael has shown me how to add the right magic to git to get full context yeah!

[glebius]

My version https://reviews.freebsd.org/D32655