Abandoning this due to insufficient proof of a problem.

Based on evaluating this more, feedback in https://reviews.freebsd.org/D12501, and some production analysis, I think the only potential exposure here is tcp_usr_send adding bytes to the socket with sbappendstream followed by tcp_connect returning EAGAIN.

We're running a modified patch of this in production where a counter is incremented, and we aren't seeing a counter increment.

I think a reasonable question here is, why the gate on rem <= 0 for goto noerr;

• detect when EINTR and EAGAIN can be squelched in sendfile

In D12501#259391, @jtl wrote:

OK, thanks! I agree this concept appears consistent with the current user-space send equivalent.

I'm not positive that pru_send is guaranteed to not return EINTR or ERESTART, so it might be worth considering them.

Also, in this particular context (TCP behavior), note that the EAGAIN about which you are concerned is probably not even possible, since it only occurs in the implicit connect case, which sendfile doesn't support.

Ok, let's dig in. pru_send, aka tcp_usr_send calls both tcp_connect in some cases, and tcp_output in most cases. Both calls are after sbappendstream() is called. tcp_connect has a pretty clear if rare path to EAGAIN via in_pcbbind and might be the source of the issues we are seeing.

I've created another diff for the simpler approach. If that looks better, we can abandon the more extensive changes.

The primary goal is to ensure that userspace and the kernel are in lock step as far as how many bytes were actually added to the socket, as long as sendfile returns no error or an error that is not fatal to the connection.

I'm not seeing where that was discussed but, sounds good to me.

In D12492#258950, @jtl wrote:

Can you explain how these are different? Both functionally set error to so->so_error, and then set so->so_error to 0. If (so) is properly locked, it looks like these should be equivalent. If (so) is not properly locked, then the new formulation is not guaranteed to overcome this deficiency.

But, I could be missing something obvious... (It is one of "those days" today. :-) )

revert so_error change

fix so_error handling

Ah, yes you are correct with regard to the PPPoE case.

We can't revert this patch, for one extremely simple reason, and another more subtle reason.

In D10272#213260, @sepherosa_gmail.com wrote:

Everyone fall back to syncookie under heavy load, if you reap syncache using callout.

Here's an idea from gnn that I agree with:

I think the problems with the current behavior are a bit larger than a RST. As far as we can tell, we send a RST, then delete the syncache entry, which allows the request packet (for http anyway) to create a session... because syncookies were in use.

First, an explanation of the formula. This is the key equation, the missing assumption in my original email:

This formula: