Mitigate a case where TCP rack can send an extra ack.

[rrs]

Summary

Rack will in theory send an extra rate limited ack when we get to a closing state (sending a FIN) so that
if we have only 1 packet outstanding we might encourage the connection to close out. However it does this
always which is not always wise. Change it so that it only does that if its been more than an srtt since
we have had some activity i.e. a send or a receive of a packet.

Test Plan

This little pkt-drill script will fail without the patch but will pass once we
get the prevention in place. It makes it so two extra acks are not sent
at close() (one for the close call and one for the disconnect() that happens under
the covers)

Without this patch the following packetdrill scripts passes:

--ip_version=ipv4
--mtu=16384

 0.000 `kldload -n tcp_rack cc_htcp`
+0.000 `sysctl -w net.inet.tcp.sack.enable=1`
+0.000 `sysctl -w net.inet.tcp.ecn.enable=3`
+0.000 `sysctl -w net.inet.tcp.hostcache.purgenow=1`

+0.000 socket(..., SOCK_STREAM, IPPROTO_TCP) = 3
+0.000 setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
+0.000 setsockopt(3, IPPROTO_TCP, TCP_FUNCTION_BLK, {function_set_name="rack",
                                                     pcbcnt=0}, 36) = 0
+0.000 setsockopt(3, IPPROTO_TCP, TCP_CONGESTION, "htcp", 4) = 0
+0.000 bind(3, ..., ...) = 0
+0.000 listen(3, 1) = 0
+0.000 < SEWA     0:0(0)                win 65535 <mss 16344,sackOK,eol,eol>
+0.000 > SW.      0:0(0)       ack    1 win 65535 <mss 16344,sackOK,eol,eol>
+0.050 <   .2     1:1(0)       ack    1 win 65535
+0.000 accept(3, ..., ...) = 4
+0.000 setsockopt(4, IPPROTO_TCP, TCP_LOG, [4], 4) = 0
+0.000 close(3) = 0
+0.100 send(4, ..., 8192, 0) = 8192 
+0.000 >  P.5     1:8193(8192) ack    1 win 65535
+0.050 <   .      1:1(0)       ack 8193 win 32000
+0.100 send(4, ..., 1038, 0) = 1038 
+0.000 close(4) = 0
+0.000 >  P.5  8193:9231(1038) ack    1 win 65535
+0.002 >   .5  8192:8192(0)    ack    1 win 65535 // XXX
+0.050 <  F.5     1:1(0)       ack 9231 win 32000 
+0.000 >  F.5  9231:9231(0)    ack    2 win 65535
+0.050 <   .5     2:2(0)       ack 9232 win 32000

With this patch, the behavior is:

--ip_version=ipv4
--mtu=16384

 0.000 `kldload -n tcp_rack cc_htcp`
+0.000 `sysctl -w net.inet.tcp.rfc1323=1`
+0.000 `sysctl -w net.inet.tcp.blackhole=0`
+0.000 `sysctl -w net.inet.tcp.sack.enable=1`
+0.000 `sysctl -w net.inet.tcp.ecn.enable=3`
+0.000 `sysctl -w net.inet.tcp.rexmit_slop=200`
+0.000 `sysctl -w net.inet.tcp.rexmit_drop_options=0`
+0.000 `sysctl -w net.inet.tcp.hostcache.purgenow=1`

+0.000 socket(..., SOCK_STREAM, IPPROTO_TCP) = 3
+0.000 setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
+0.000 setsockopt(3, IPPROTO_TCP, TCP_FUNCTION_BLK, {function_set_name="rack",
                                                     pcbcnt=0}, 36) = 0
+0.000 setsockopt(3, IPPROTO_TCP, TCP_CONGESTION, "htcp", 4) = 0
+0.000 bind(3, ..., ...) = 0
+0.000 listen(3, 1) = 0
+0.000 < SEWA     0:0(0)                win 65535 <mss 16344,sackOK,eol,eol>
+0.000 > SW.      0:0(0)       ack    1 win 65535 <mss 16344,sackOK,eol,eol>
+0.000 <   .2     1:1(0)       ack    1 win 65535
+0.000 accept(3, ..., ...) = 4
+0.000 setsockopt(4, IPPROTO_TCP, TCP_LOG, [4], 4) = 0
+0.000 close(3) = 0
+0.100 send(4, ..., 8192, 0) = 8192 
+0.000 >  P.5     1:8193(8192) ack    1 win 65535
+0.000 <   .      1:1(0)       ack 8193 win 32000
+0.100 send(4, ..., 1038, 0) = 1038 
+0.000 close(4) = 0
+0.000 >  P.5  8193:9231(1038) ack    1 win 65535
+0.000 <  F.5     1:1(0)       ack 9231 win 32000 
+0.000 >  F.5  9231:9231(0)    ack    2 win 65535
+0.000 <   .5     2:2(0)       ack 9232 win 32000

[tuexen]

I added a script which show the behavior before the patch. I also cleaned up the formatting of the script showing the behavior after the script.

[rscheff]

Reducing the number of challenge ACKs when closing out the connection makes sense.

[rrs]

And I have found the case where we are getting the initial ack that causes the ack-war beginning.. not the
src of the continued ack-war (thats in tcp_timewait) but the initial extra ack :)

[rrs]

Fix the compile bug and get the tested code in.

[rrs]

fix comment

[rrs]

commited