packages: Compress in parallel
ClosedPublic
Actions

Authored by cperciva on Sun, Oct 12, 10:01 PM.

Details

Reviewers

ivy

Group Reviewers

pkgbase

Commits

rG8f5eb720e2e6: packages: Compress in parallel
rG3d8ec3273dbb: packages: Compress in parallel
rG16155cccc4bf: packages: Compress in parallel

Summary

The FreeBSD-base pkg repository is currrently dominated by a handful of
large packages: Out of a compressed repository size of ~960 MB, about
160 MB is taken up by the FreeBSD-src package, 128 MB (on amd64) is
used by the FreeBSD-kernel-generic-dbg package, and 91 MB is used by
the FreeBSD-src-sys package. Consequently, running 'make packages -jN'
provides less benefit than one might hope, as most of the packages
finish building quickly, ultimately leaving the FreeBSD-src package
building by itself for a couple minutes while all the other CPUs are
idle.

Pass -T${PKG_CTHREADS} to the 'pkg create' commands (with a default
of -T0) in order to instruct pkg's zstd compression to use multiple
threads.

Testing on an EC2 r7i.48xlarge instance with -j192, this reduces the
time taken by 'make packages' from 6m17s to 1m39s; package creation
time itself (excluding the initial process of installing into world
and kernel staging directories) dropped from 5m37s to 59s.

MFC after: 3 days
Sponsored by: https://www.patreon.com/cperciva

Diff Detail

Repository

rG FreeBSD src repository

Lint

Lint Not Applicable

Unit

Tests Not Applicable

Event Timeline

cperciva created this revision.Sun, Oct 12, 10:01 PM

Herald added subscribers: emaste, imp. · View Herald TranscriptSun, Oct 12, 10:01 PM

cperciva requested review of this revision.Sun, Oct 12, 10:01 PM

Harbormaster completed remote builds in B67732: Diff 164055.Sun, Oct 12, 10:01 PM

ivy added a subscriber: ivy.Mon, Oct 13, 5:09 AM

ivy added inline comments.

Makefile.inc1
1967	should this perhaps default to `${.MAKE.JOBS:U1}`? if `-j` is higher than 1, that won't be reliable since we also build packages in parallel (so you'll get up to N * N jobs instead of N jobs) but it at least keeps it to one thread in the -j1 case. i don't know if this is the right approach though: perhaps people expect multiple CPUs to be used where possible even if they don't do a parallel build.
2148	we could do with a new variable to avoid repeating `-f ${PKG_FORMAT} ${PKG_CLEVEL} -T${PKG_CTHREADS}` so many times.

cperciva added inline comments.Mon, Oct 13, 5:14 AM

Makefile.inc1
1967	I considered that, but most packages are small enough that they only use one CPU anyway. In fact on my 192-CPU system I think the largest package (FreeBSD-src) still only uses 8 threads.
2148	Yeah, I was considering condensing these into a `${PKG_CREATE_OPTS}` but didn't want to mix functional and non-functional changes.

ivy accepted this revision.Mon, Oct 13, 6:43 AM

ivy added inline comments.

Makefile.inc1
1967	i was thinking more about people with 4-8 CPUs who might use `make -j1` so it doesn't tie up all cores. but i don't feel very strongly about this, even in that case it's only a very small part of the overall build.