libfetch: Add read buffering
Needs ReviewPublic
Actions

Authored by des on Sat, Jun 27, 2:30 PM.

Details

Reviewers

Summary

Previously, we would read FTP control connection messages and HTTP
reponse headers one character at a time. Now, we read as much as will
fit in our buffer and look for a newline. If there is data left over,
it will be reused by the next fetch_getln() call. This also requires
the addition of a fetch_bufread() which takes the buffer into account,
otherwise the start of the HTTP response body will be stuck in the
buffer after we read the last line of the header.

This should noticeably improve HTTP performance, especially for small
transfers.

MFC after: 1 week

Diff Detail

Repository

rG FreeBSD src repository

Lint

Lint Skipped

Unit

Tests Skipped

Build Status

Buildable 74393
Build 71276: arc lint + arc unit

Event Timeline

des created this revision.Sat, Jun 27, 2:30 PM

Herald added a subscriber: imp. · View Herald TranscriptSat, Jun 27, 2:30 PM

des requested review of this revision.Sat, Jun 27, 2:30 PM

Harbormaster completed remote builds in B74328: Diff 180818.Sat, Jun 27, 2:31 PM

des planned changes to this revision.Mon, Jun 29, 1:09 PM

des retitled this revision from libfetch: Buffer the FTP control connection to libfetch: Add read buffering.Mon, Jun 29, 1:42 PM

des edited the summary of this revision. (Show Details)

fix http

Harbormaster completed remote builds in B74379: Diff 180929.Mon, Jun 29, 1:42 PM

des added a child revision: D57944: libfetch: Make fetch_ref an inline.Mon, Jun 29, 1:43 PM

fix chunk decoding

Harbormaster completed remote builds in B74390: Diff 180942.Mon, Jun 29, 7:08 PM

chunked encoding again

Harbormaster completed remote builds in B74393: Diff 180951.Mon, Jun 29, 8:57 PM

op added a subscriber: op.Wed, Jul 8, 11:28 AM

op added inline comments.

lib/libfetch/common.c
1425	wouldn't be better to use a circular buffer?
lib/libfetch/http.c
169	what was the problem with the original tolower? function call overhead?

des added inline comments.Wed, Jul 8, 11:53 AM

lib/libfetch/common.c
1425	A fully circular buffer wouldn't work since we use string operations on the contents. We can reduce the amount of copying we do if we change the `fetch_getln()` API a little, but that's not a priority. The objective was to stop reading FTP and HTTP responses one character at a time, and this achieves that.
lib/libfetch/http.c
169	It's much more than just a function that maps `A-Z` to `a-z`; it's locale-dependent.