Change blackhole detection and mtu clamping trigger.

[hiren]

Summary

Blackhole detection and mtu clamping begins with just first time an RTO fires.
This seems too aggressive and probably not correct. This change makes it so
that it only triggers when the retransmit also fails. i.e. it gets triggered after 2 chances.

The changes also make sure that each mtu probe stage (usually 1448 -> 1188 -> 524) gets 2 chances before further clamping down.

Test Plan

This is how it behaves now with simulated loss:

1. A packet with mtu 1448 gets dropped
2. We receive 3 dup acks to let us know about the loss
3. 1st retransmit attempt with mtu 1448 which also gets dropped, t_rxtshift = 1
4. 2nd retransmit attempt with mtu 1448 which also gets dropped, t_rxtshift = 2
5. Now, we decide to trigger blackhole mtu reduction and go from 1448 to 1188.
6. 3rd retransmit attempt now with newer mtu 1188 which gets dropped rxtshift = 3
7. 4th retransmit attempt now with mtu 1188 which also gets dropped rxtshift = 4
8. At this point, we further reduce MTU to 524, 5th retransmit attempt which gets dropped rxtshift = 5
9. 6th retransmit attempt with mtu 524 which gets dropped rxtshift = 6. Now, because 524 is net.inet.tcp.mssdflt, we don't clamp down more.
10. 7th retransmit attempt with mtu 524. If this gets dropped, we assume that this is not actually a blackhole and we undo the clamping down we've done and increment net.inet.tcp.pmtud_blackhole_failed

If a valid ack comes at any time during this, t_rxtshift gets reset to 0.

[bz]

I wished our TCP comments would all cite the RFC/section for the appropriate thing to make it a lot easier to see if implementation and theory line up. At least for commit messages it might be a good idea to cite them?

[hiren]

In D3434#69986, @bz wrote:

I wished our TCP comments would all cite the RFC/section for the appropriate thing to make it a lot easier to see if implementation and theory line up. At least for commit messages it might be a good idea to cite them?

Apologies for not being clear about this.
rfc 4821 doesn't specify the implementation details. Section "2. Overview" says:

This document does not contain a complete description of an
implementation.  It only sketches details that do not affect
interoperability with other implementations....

I can mention this in the commitlog.

[hiren]

I've discovered some anomalies with the current approach. I'd come back with more results/conclusions.

[hiren]

Intention of this feature is to be able to crank mtu down till we can
successfully send data or if mtu gets reduced to decided min mss. This
was not happening with the earlier proposed patch. We could only reduce
mtu once.

Current patch makes it possible where we transition from, for example,
1448 -> 1188 -> 524 till we successfully can send a packet. (542 is the
predetermined min mss here.)

[hiren]

Any comments/concerns regarding this patch?

[hiren]

Improving the patch by making sure each probe stage gets 2 chances of recovery
before further clamping down of mtu.

[sbruno]

This is an adequate solution to "searching" for a useable MTU when ICMP is blocked. Linux does a binary search starting at MIN and working its way up. This is a nice way to do a similar technique.

[hiren]

@bz / others: any objection/comments? If not, I'll commit this in a few days.