HomeFreeBSD

packages: Compress in parallel

Description

packages: Compress in parallel

The FreeBSD-base pkg repository is currrently dominated by a handful of
large packages: Out of a compressed repository size of ~960 MB, about
160 MB is taken up by the FreeBSD-src package, 128 MB (on amd64) is
used by the FreeBSD-kernel-generic-dbg package, and 91 MB is used by
the FreeBSD-src-sys package. Consequently, running 'make packages -jN'
provides less benefit than one might hope, as most of the packages
finish building quickly, ultimately leaving the FreeBSD-src package
building by itself for a couple minutes while all the other CPUs are
idle.

Pass -T${PKG_CTHREADS} to the 'pkg create' commands (with a default
of -T0) in order to instruct pkg's zstd compression to use multiple
threads.

Testing on an EC2 r7i.48xlarge instance with -j192, this reduces the
time taken by 'make packages' from 6m17s to 1m39s; package creation
time itself (excluding the initial process of installing into world
and kernel staging directories) dropped from 5m37s to 59s.

Approved by: re (cperciva)
Reviewed by: ivy
MFC after: 3 days
Sponsored by: https://www.patreon.com/cperciva
Differential Revision: https://reviews.freebsd.org/D53053

(cherry picked from commit 16155cccc4bf42a5a76806128e7ef637f9ae61fe)
(cherry picked from commit 3d8ec3273dbb9edf9080ded97e071edff59dd708)

Details

Provenance
cpercivaAuthored on Sun, Oct 12, 9:44 PM
Reviewer
ivy
Differential Revision
D53053: packages: Compress in parallel
Parents
rG77a00e86d64c: sys/rpc: UNIX auth: Do not log on bogus AUTH_SYS messages
Branches
Unknown
Tags
Unknown