For reference fastpath version of this hasn't been done, which need also need the same changes to:

tcp_stacks/fastpath.c

Apologies for the delay in getting to this.

So, to go back to first principles for a second, the basic goal of RX sockbuf autoscaling is to avoid the window advertised to the sender becoming the limiting factor in the sender's ability to fully utilise the pipe. The sender's theoretically ideal window at any given moment will be == the path's BDP, which the receiver makes no attempt to infer/measure, so the current algorithm takes the simplistic approach of only ensuring that the largest window that can be advertised (== RX sb_hiwat) is at least 1/8 or ~12% greater than the connection's largest ever measured send window (subject to the various autoscaling and sockbuf limits). The sender's window is measured approximately once per RTT by counting bytes received over the current RTT. The algorithm makes no attempt to deal with other causes of lost transmission opportunity (e.g. if the receiving app is not adequately servicing the RX sockbuf every RTT), which arguably should be improved given the goal of this feature.

Regarding the use of SRTT to measure send window: given the imprecise requirements for the current algorithm, I suspect it's entirely reasonable to ditch the dual RTT measurement code paths and use SRTT exclusively even though it's less correct to do so. If SRTT is larger than raw RTT, we'll have a longer measurement period and likely count more bytes, which may cause a premature increase. Conversely, we may delay an increase if SRTT < raw. Neither is a show stopper.

Regarding bulk receive mode reset: it seems that perhaps the whole autoscaling code block is in the wrong place stuffed into the header prediction "fast path". Bottom line is that to trigger an increase, the connection has to be in "bulk receive mode" on account of the send window being measured as > 7/8 RX sockbuf. I also can't think of a reason why bi-directional connections shouldn't also scale their RX sockbuf.

As an aside, the "to.to_tsecr - tp->rfbuf_ts < hz" check in the TOF_TS path is completely bogus for connections with legitimate RTTs >1s (which would stand to benefit a lot from RX sockbuf scaling).

Updated dtrace:

dtrace
#!/usr/sbin/dtrace -s

#pragma D option quiet
#pragma D option switchrate=25hz

int last[int];
int last_check[int];

dtrace:::BEGIN
{
       printf(" %12s %7s %9s %9s %9s %12s %12s %12s\n",
           "dt(us)",  "srtt", "newsize", "rfbuf_ts", "rtttime", "ack", "seq", "rtseq");
}

tcp:::receive-autoresize
/(args[4]->tcp_sport == 80 || args[4]->tcp_sport == 443)/
{
       elapsed = (timestamp - last[args[1]->cs_cid]) / 1000;
       printf(" %12d %7d %9d %9d %9d %12d %12d %12d\n", elapsed,
           args[3]->tcps_srtt, args[5], args[3]->tcps_rfbuf_ts, args[3]->tcps_rtttime, args[4]->tcp_ack, args[4]->tcp_seq, args[3]->tcps_rtseq);
       last[args[1]->cs_cid] = timestamp;
}

Thanks for the feedback Lawrence, based on that I've updated to use just the SRTT check, added the fastpath version and generally cleaned up.

Hopefully this captures everything, putting it in a good place to merge?

Lawrence is now happy with this in is current form, so just a prod to see if I get any comments for or against from transport members / gnn?