Doing some debugging I noticed that applications using rpc(3) would often
make lseek(2) on a totally bogus file descriptor, that looks more like a
pointer. So, what happens here is that xdrrec type xdr doesn't keep a
track of how many bytes were sent/received on the stream and tries to
obtain this number via lseek(2). Then it adds/subtracts the offset in the
internal buffer from the obtained number. This code originates from the
original Sun RPC import in 1994. However, it was not a working code even
if Solaris would support lseek(2) on a socket, because it was passing not
the file descriptor, but a pointer to opaque data from upper RPC layer.
It could be that previously (before import to FreeBSD) code was correct,
but the Solaris 8 documentation says that lseek(2) on socket isn't
supported [1]. Maybe supported on older Solaris?
Anyway, this lseek(2) never worked and xdr_getpos() would always fail on
xdrrec object, until 8f55a568f69c5 in 2008 it was slightly fixed to
tolerate failure of lseek(2) and return a correct value within the small
internal buffer for XDR_ENCODE mode and a an incorrect (negative to
unsigned) result for XDR_DECODE. It seems no consumer ever calls
xdr_getpos()/xdr_setpos() on this kind of descriptor when in XDR_DECODE
mode.
So, remove this lseek(2) and preserve operation within the small buffer
only. Supposedly fix the operation for XDR_DECODE mode. Note that there
is no use and no test coverage for the XDR_DECODE.
Note that xdr(3) manual page already documents limitations for
xdr_getpos() and xdr_setpos() for the stream type objects.
[1] https://docs.oracle.com/cd/E19109-01/tsolaris8/835-8003/6ruu1b0or/index.html