sendfile: don't hack sb_lowat for sockets that manage the watermark

[glebius]

Summary

In the sendfile(2) we carry an old hack (originating from d99b0dd2c5297)
to help dumb benchmarks and applications to achieve higher performance. We
would modify low watermark on the socket send buffer to avoid socket being
reported as writable too early, which would result in lots of small
writes.

Skip that hack for applications that do setsockopt(SO_SNDLOWAT) or that
register the socket in kevent(2) with NOTE_LOWAT feature. First, we don't
want the hack to rewrite the watermark value explicitly specified by the
user. Second, in certain cases that can lead to real performance
regressions. A kevent(2) with NOTE_LOWAT would report socket as writable,
but then sendfile(2) would write 0 bytes and return EAGAIN.

The change also disables the hack for unix(4) sockets, leaving only TCP.

[jtl]

sys/kern/uipc_socket.c

4520

I'll note that it is possible that there are applications which will only set NOTE_LOWAT some of the time. Perhaps, they don't even do it themselves, but some library they are calling to do some work on the socket does it for them. In those cases, we will permanently clear this flag for the applications, which seems like a potential POLA violation. (Of course, sendfile() modifying the low-water mark is also a potential POLA violation.)

If the behavior to modify the low-water mark in sendfile() is valuable enough to keep, then maybe we should add separate logic for this case which restores the behavior when the NOTE_LOWAT has been removed from the list? Or, perhaps, the right answer is to remove the hack and instead change the default low-water mark value?

[glebius]

sys/kern/uipc_socket.c

4520

First, we don't know if the hack is valuable at all. We can't name any particular application that we know would suffer if the sb_lowat modification is removed. It could be there are none, except benchmarks or tests. But we are trying to keep the hack in some form just for safety. A case of application that sometimes uses NOTE_LOWAT and sometimes doesn't of course is possible, but even less probable than just a mature application strongly depending on the hack. I'd rather take a risk of breaking such potential application, than lay a third layer of ifs to cover case when hack had been disabled but now needs to be enabled back.

Or, perhaps, the right answer is to remove the hack and instead change the default low-water mark value?

Just increase initial lowat for all TCP sockets? No, I'm afraid that's going to be a major POLA violation for all kinds of TCP programs that communicate in small chunks of data.