Page MenuHomeFreeBSD

framework: add support for using sccache during port builds
Needs ReviewPublic

Authored by tcberner on Tue, Jul 20, 4:07 PM.

Details

Reviewers
None
Group Reviewers
portmgr
rust
desktop
Summary

A big issue with lang/rust and its users is that building it takes
about three times longer than people have patience for.

Simlilar to ccache there is sccache which can be used for rust.
As this is a rust program, this is unfortunately a chicken and
egg problem, as opposed to ccache, we cannot depend on the sscache
port.

The newly added bsd.sscache.mk supports the following three variables:

  • SSCACHE_SIZE how much space do you want to sacrifice for time
  • SSCACHE_DIR what space do you want to sacrifice
  • SCCACHE_BIN the sccache binary you provide

Default values:

  • SCCACHE_SIZE defaults to 16G
  • SCCACHE_BIN defaults to SCCACHE_DIR/sccache if the latter is set

This makes it quite easy to inject sccache into poudriere, by abusing
the current ccache infrastructure (obviously this can be improved).
For this to work properly, the ccache dirs in poudriere have to be
split by architecture/version (as we put the sccache binary there in
this example):

Assuming the ccache used by the 14-CURRENT amd64 jail called 140amd64 is /zcache/ccache/140-64

i.e. you have

	CCACHE_DIR=/zcache/ccache/140-64

in the /usr/local/etc/poudriere.d/140amd64-poudriere.conf

Create a place for the sccache:

	> mkdir -p /zcache/ccache/140-64/sccache

Copy the pre-compiled sccache binary:

	> cp /usr/local/bin/sccache /zcache/ccache/140-64/sccache

Configure poudriere

	> echo SCCACHE_DIR=/root/.ccache/sccache >> /usr/local/etc/poudriere.d/140amd64-make.conf

This change makes it possible to compile for example lang/rust and
graphics/librsvg2-rust in poudriere quite fast:

	> poudriere bulk -j140amd64 -pmain -tC lang/rust graphics/librsvg2-rust

i.e.

	[...]
	[00:19:04] [01] [00:18:56] Finished lang/rust | rust-1.53.0: Success
	[00:20:30] [01] [00:01:22] Finished graphics/librsvg2-rust | librsvg2-rust-2.50.3_4: Success
	[...]

while using around 1G of disk space.

Note: I have no clue of rust and its build tools, so I might have missed
something and broken another :)

As an example x11/wezterm here gives a speep improvement of around 40%:

  • with sccache: around 4.5 minutes
  • without sccache: around 7 minutes

Diff Detail

Repository
rP FreeBSD ports repository
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 40582
Build 37471: arc lint + arc unit

Event Timeline

tcberner retitled this revision from */*: add support for using sccache during port builds to framework: add support for using sccache during port builds.

Update commit message

  • fix cache size environment variable name
  • add post-build target to printout statistics of the cache
  • add SCCACHE_MAX_FRAME_LENGTH=104857600 to the environment [1]
  • remote the statistics print out
  • add sccache-start and -stop targets added to the CONFIGURE_ and STAGE_SEQ to ensure the sccache server gets shut down when the build stops.

[1] https://wiki.gentoo.org/wiki/Sccache#Initial_setup