OpenSSL BIO classes provide an abstraction for dealing with I/O.
OpenSSL provides BIO classes for commonly used I/O primitives backed
by file descriptors, sockets, etc. as well as permitting consumers
of OpenSSL to define custom BIO classes.
One of the methods BIO classes implement is a control method invoked
by BIO_ctrl() for various ancilliary tasks somewhat analgous to
fcntl() and ioctl() on file descriptors. According to the BIO_ctrl(3)
manual page, control methods should return 0 for unknown control
requests.
KTLS support in OpenSSL adds new control requests. Two of those new
requests are queries to determine if KTLS is enabled for either
reading or writing. These control reuquest return 1 if KTLS is
enabled and 0 if it is not.
serf includes two custom BIO classes for wrapping I/O requests from
files and from a buffer in memory. These BIO classes both use a
custom control method. However, this custom control method was
returning 1 for unknown or unsupported control requests instead of 0.
As a result, OpenSSL with KTLS believed that these BIOs were using
KTLS and were thus adding headers and doing encryption/decryption in
the BIO. Correcting the return value removes this confusion.
PR: 253135
Reported by: Guido Falsi <mad@madpilot.net>
MFC after: 3 days
Sponsored by: Netflix