Page MenuHomeFreeBSD

Upgrade devel/ccache to v4.1
Needs ReviewPublic

Authored by khng on Nov 27 2020, 10:39 AM.

Details

Reviewers
bdrewery
lwhsu
Group Reviewers
portmgr
Summary

This version introduces Zstd compression for the compiler cache.
Besides, the code is also rewritten in C++. C++11 is required to compile
ccache with version greater than or equal to 4.0.

ccache(1) is generated from adoc now.

Some related PRs upstream:

Diff Detail

Repository
rP FreeBSD ports repository
Lint
No Linters Available
Unit
No Unit Test Coverage
Build Status
Buildable 35725
Build 32614: arc lint + arc unit

Event Timeline

khng requested review of this revision.Nov 27 2020, 10:39 AM
  • Comment on "WARN: Makefile: "PATCH_SITES" has to appear earlier."
  • Fix a typo PATCH_SITE in comments

Thanks for working on this. I'll review+test soon.

  • Use upstream patches to replace PR #730

Do you try build ccache without installed ccache, cmake but with WITH_CCACHE_BUILD ?

I found an issue that libzstd wasn't statically linked. Now I am working on it.

  • Add ZSTD_SYSTEM_STATIC=On option to link against zstd static library
  • Add dependencies on perl5 when both DOCS and MEMCACHED options are on
  • Fix flipped MEMCACHED_CMAKE_ON/OFF
  • Update patches with "make makepatch"
  • Make patches from PR #750 extra

Rebased to the most recent ports tree.

  • Upgrade devel/ccache to v4.1
  • Comment on "WARN: Makefile: "PATCH_SITES" has to appear earlier."
  • Fix a typo PATCH_SITE in comments
  • Use upstream patches to replace PR #730
  • Add ZSTD_SYSTEM_STATIC=On option to link against zstd static library
  • Add dependencies on perl5 when both DOCS and MEMCACHED options are on
  • Fix flipped MEMCACHED_CMAKE_ON/OFF
  • Update patches with "make makepatch"
  • Make patches from PR #750 extra

This change should fix infinite recursion: https://github.com/rozhuk-im/freebsd-ports/commit/dfe7746fb71eb1df89474f527499a1f5a768f01c

Here https://github.com/rozhuk-im/freebsd-ports/commit/56f23ccb032fff4dcd265b433f1f13cdb98fc7ed
4.1 ccache-devel, port rewritten:

  • memcache removed
  • add do-test ans TESTS option
  • code compacted

Together both this commits allow normal bootstrap: build and install on fresh FreeBSD where no ports installed and ccache enabled before first port build.
IMHO better to move all memcache staff out from ccache / ccache-devel to ccache-memcached, this allow make port more compact and easy to support.

This change should fix infinite recursion: https://github.com/rozhuk-im/freebsd-ports/commit/dfe7746fb71eb1df89474f527499a1f5a768f01c

Here https://github.com/rozhuk-im/freebsd-ports/commit/56f23ccb032fff4dcd265b433f1f13cdb98fc7ed
4.1 ccache-devel, port rewritten:

  • memcache removed
  • add do-test ans TESTS option
  • code compacted

Together both this commits allow normal bootstrap: build and install on fresh FreeBSD where no ports installed and ccache enabled before first port build.
IMHO better to move all memcache staff out from ccache / ccache-devel to ccache-memcached, this allow make port more compact and easy to support.

I think the infinite loop somehow happens in a weird way on my side. The cmake build, somehow, managed to build an older devel/ccache (3.7.x) from my another tree, thus recursion didn't happen.

  • devel/ccache: Apply NO_CCACHE_DEPEND=yes on Mk/Scripts/do-depends.sh

This change should fix infinite recursion: https://github.com/rozhuk-im/freebsd-ports/commit/dfe7746fb71eb1df89474f527499a1f5a768f01c

My answer to the problem is to manipulate the .MAKEFLAGS target a bit, so that do-depends.sh will automatically inherit NO_CCACHE_DEPEND=yes on any possible depends-on builds.

IMHO,
.MAKEFLAGS: NO_CCACHE_DEPEND=yes
may stop work if some port define NOPRECIOUSMAKEVARS.

Mk/bsd.ccache.mk - all .MAKEFLAGS commented.
https://github.com/freebsd/freebsd-ports/commit/0033c2a4c4b4346e7ace80282f3a238e73434ec2
It is added as comment, but code uses MAKE_ENV, CONFIGURE_ENV to pass env vars to sub process.

Why to not fix NO_CCACHE_DEPEND, like in https://github.com/rozhuk-im/freebsd-ports/commit/dfe7746fb71eb1df89474f527499a1f5a768f01c ?

IMHO,
.MAKEFLAGS: NO_CCACHE_DEPEND=yes
may stop work if some port define NOPRECIOUSMAKEVARS.

NOPRECIOUSMAKEVARS will not stop variables not in _EXPORTED_VARS being exported to do-depends.sh

Why to not fix NO_CCACHE_DEPEND, like in https://github.com/rozhuk-im/freebsd-ports/commit/dfe7746fb71eb1df89474f527499a1f5a768f01c ?

I am not sure if it is really a good idea to simply let every dependencies inheriting NO_CCACHE_DEPEND. But if NO_CCACHE_DEPEND is only used by ccache-* then it seems fine to me.

If all ccache deps does not inherit NO_CCACHE_DEPEND then you may get infinite recursion, because some of dep may require ccache.
I already try bootstrap without this and get: ccache->cmake->ccache->cmake.....

Yes. And .MAKEFLAGS: NO_CCACHE_DEPEND=yes should be sufficient on its own.
If NO_CCACHE_DEPEND is only a thing in devel/ccache then factoring the
logic out to Mk/* is also fine.

  • Revert "devel/ccache: Apply NO_CCACHE_DEPEND=yes on Mk/Scripts/do-depends.sh"
  • Let NO_CCACHE_DEPEND propagate to the dependencies as well

I will move it to bsd.ccache.mk, but without CCACHE_ENV. Directly expanding CCACHE_ENV when invoking {SETENV} (which simply allows things inside sift through) in place seems to me doesn't match how the other flags work in do-depends.sh currently. At the end, this should still plays well with your proposed ports.

  • ZSTD_SYSTEM_STATIC=On could be replaced with explicit path to libzstd.a

Thanks for this. I split out ccache-memcached recently into its own port. It sucks that cmake and zstd are both dependencies now.
I also have local patches for src/ccache.cpp as well.
I will take what's here and adapt/credit where possible.

Mk/bsd.ccache.mk
36

I like the idea here but I believe this only works with direct port builds but will not with Poudriere. At least not without using the DEPENDS_ARGS hack which I think was mutually exclusive with FLAVORS support. I'll look into that.