HomeFreeBSD

packages: Compress in parallel

Description

packages: Compress in parallel

The FreeBSD-base pkg repository is currrently dominated by a handful of
large packages: Out of a compressed repository size of ~960 MB, about
160 MB is taken up by the FreeBSD-src package, 128 MB (on amd64) is
used by the FreeBSD-kernel-generic-dbg package, and 91 MB is used by
the FreeBSD-src-sys package. Consequently, running 'make packages -jN'
provides less benefit than one might hope, as most of the packages
finish building quickly, ultimately leaving the FreeBSD-src package
building by itself for a couple minutes while all the other CPUs are
idle.

Pass -T${PKG_CTHREADS} to the 'pkg create' commands (with a default
of -T0) in order to instruct pkg's zstd compression to use multiple
threads.

Testing on an EC2 r7i.48xlarge instance with -j192, this reduces the
time taken by 'make packages' from 6m17s to 1m39s; package creation
time itself (excluding the initial process of installing into world
and kernel staging directories) dropped from 5m37s to 59s.

Reviewed by: ivy
MFC after: 3 days
Sponsored by: https://www.patreon.com/cperciva
Differential Revision: https://reviews.freebsd.org/D53053

Details

Provenance
cpercivaAuthored on Oct 12 2025, 9:44 PM
Reviewer
ivy
Differential Revision
D53053: packages: Compress in parallel
Parents
rG4fc1e7546f12: RELNOTES: Fix a typo in the 5000d023a446 entry
Branches
Unknown
Tags
Unknown