Page MenuHomeFreeBSD
Differential D27393

Upgrade devel/ccache to v4.1
Needs ReviewPublic
Actions

Authored by khng on Nov 27 2020, 10:39 AM.
Tags
None
Referenced Files
F154348845: D27393.id.diff
Tue, Apr 28, 12:35 AM
Unknown Object (File)
Sat, Apr 25, 9:53 AM
Unknown Object (File)
Sat, Apr 25, 9:10 AM
Unknown Object (File)
Sat, Apr 25, 6:31 AM
Unknown Object (File)
Sat, Apr 25, 6:31 AM
Unknown Object (File)
Sat, Apr 25, 5:53 AM
Unknown Object (File)
Sat, Apr 25, 5:41 AM
Unknown Object (File)
Sat, Apr 25, 5:22 AM
View All Files
Subscribers
diizzy
fluffy
mat
rozhuk.im-gmail.com
tmunro
vishwin

Details

Reviewers
bdrewery
lwhsu
Group Reviewers
portmgr
Summary

This version introduces Zstd compression for the compiler cache.
Besides, the code is also rewritten in C++. C++11 is required to compile
ccache with version greater than or equal to 4.0.

ccache(1) is generated from adoc now.

Some related PRs upstream:

Diff Detail

Repository
rP FreeBSD ports repository
Lint
No Lint Coverage
Unit
No Test Coverage
Build Status
Buildable 35569
Build 32469: arc lint + arc unit

Event Timeline

khng created this revision.Nov 27 2020, 10:39 AM
Herald added a subscriber: mat. · View Herald TranscriptNov 27 2020, 10:39 AM
khng requested review of this revision.Nov 27 2020, 10:39 AM
Harbormaster completed remote builds in B35071: Diff 80054.Nov 27 2020, 10:39 AM
khng added a reviewer: lwhsu.Nov 27 2020, 10:40 AM
khng updated this revision to Diff 80055.Nov 27 2020, 10:49 AM
  • Comment on "WARN: Makefile: "PATCH_SITES" has to appear earlier."
Harbormaster completed remote builds in B35072: Diff 80055.Nov 27 2020, 10:49 AM
khng updated this revision to Diff 80056.Nov 27 2020, 10:50 AM
  • Fix a typo PATCH_SITE in comments
Harbormaster completed remote builds in B35073: Diff 80056.Nov 27 2020, 10:51 AM
bdrewery added a comment.Dec 1 2020, 11:46 PM

Thanks for working on this. I'll review+test soon.

vishwin added a subscriber: vishwin.Dec 2 2020, 1:35 AM
khng updated this revision to Diff 80692.Dec 14 2020, 3:45 PM
  • Use upstream patches to replace PR #730
Harbormaster completed remote builds in B35397: Diff 80692.Dec 14 2020, 3:45 PM
rozhuk.im-gmail.com added a subscriber: rozhuk.im-gmail.com.Dec 19 2020, 7:21 PM

Do you try build ccache without installed ccache, cmake but with WITH_CCACHE_BUILD ?

khng added a comment.Dec 20 2020, 3:22 PM

I found an issue that libzstd wasn't statically linked. Now I am working on it.

khng updated this revision to Diff 80977.Dec 20 2020, 4:34 PM
  • Add ZSTD_SYSTEM_STATIC=On option to link against zstd static library
  • Add dependencies on perl5 when both DOCS and MEMCACHED options are on
Harbormaster completed remote builds in B35556: Diff 80977.Dec 20 2020, 4:34 PM
khng edited the summary of this revision. (Show Details)Dec 20 2020, 4:34 PM
khng updated this revision to Diff 80978.Dec 20 2020, 4:41 PM
  • Fix flipped MEMCACHED_CMAKE_ON/OFF
Harbormaster completed remote builds in B35557: Diff 80978.Dec 20 2020, 4:41 PM
khng updated this revision to Diff 80979.Dec 20 2020, 4:48 PM
  • Update patches with "make makepatch"
Harbormaster completed remote builds in B35558: Diff 80979.Dec 20 2020, 4:48 PM
khng updated this revision to Diff 80980.Dec 20 2020, 5:01 PM
  • Make patches from PR #750 extra
Harbormaster completed remote builds in B35559: Diff 80980.Dec 20 2020, 5:02 PM
khng updated this revision to Diff 80981.Dec 20 2020, 5:09 PM

Rebased to the most recent ports tree.

  • Upgrade devel/ccache to v4.1
  • Comment on "WARN: Makefile: "PATCH_SITES" has to appear earlier."
  • Fix a typo PATCH_SITE in comments
  • Use upstream patches to replace PR #730
  • Add ZSTD_SYSTEM_STATIC=On option to link against zstd static library
  • Add dependencies on perl5 when both DOCS and MEMCACHED options are on
  • Fix flipped MEMCACHED_CMAKE_ON/OFF
  • Update patches with "make makepatch"
  • Make patches from PR #750 extra
Harbormaster completed remote builds in B35560: Diff 80981.Dec 20 2020, 5:09 PM
rozhuk.im-gmail.com added a comment.Dec 21 2020, 3:08 AM

This change should fix infinite recursion: https://github.com/rozhuk-im/freebsd-ports/commit/dfe7746fb71eb1df89474f527499a1f5a768f01c

Here https://github.com/rozhuk-im/freebsd-ports/commit/56f23ccb032fff4dcd265b433f1f13cdb98fc7ed
4.1 ccache-devel, port rewritten:

  • memcache removed
  • add do-test ans TESTS option
  • code compacted

Together both this commits allow normal bootstrap: build and install on fresh FreeBSD where no ports installed and ccache enabled before first port build.
IMHO better to move all memcache staff out from ccache / ccache-devel to ccache-memcached, this allow make port more compact and easy to support.

khng added a comment.EditedDec 21 2020, 9:29 AM
In D27393#619198, @rozhuk.im-gmail.com wrote:

This change should fix infinite recursion: https://github.com/rozhuk-im/freebsd-ports/commit/dfe7746fb71eb1df89474f527499a1f5a768f01c

Here https://github.com/rozhuk-im/freebsd-ports/commit/56f23ccb032fff4dcd265b433f1f13cdb98fc7ed
4.1 ccache-devel, port rewritten:

  • memcache removed
  • add do-test ans TESTS option
  • code compacted

Together both this commits allow normal bootstrap: build and install on fresh FreeBSD where no ports installed and ccache enabled before first port build.
IMHO better to move all memcache staff out from ccache / ccache-devel to ccache-memcached, this allow make port more compact and easy to support.

I think the infinite loop somehow happens in a weird way on my side. The cmake build, somehow, managed to build an older devel/ccache (3.7.x) from my another tree, thus recursion didn't happen.

khng updated this revision to Diff 80998.Dec 21 2020, 10:10 AM
  • devel/ccache: Apply NO_CCACHE_DEPEND=yes on Mk/Scripts/do-depends.sh
Harbormaster completed remote builds in B35569: Diff 80998.Dec 21 2020, 10:10 AM
khng added a comment.Dec 21 2020, 10:12 AM

This change should fix infinite recursion: https://github.com/rozhuk-im/freebsd-ports/commit/dfe7746fb71eb1df89474f527499a1f5a768f01c

My answer to the problem is to manipulate the .MAKEFLAGS target a bit, so that do-depends.sh will automatically inherit NO_CCACHE_DEPEND=yes on any possible depends-on builds.

rozhuk.im-gmail.com added a comment.Dec 22 2020, 12:54 AM

IMHO,
.MAKEFLAGS: NO_CCACHE_DEPEND=yes
may stop work if some port define NOPRECIOUSMAKEVARS.

Mk/bsd.ccache.mk - all .MAKEFLAGS commented.
https://github.com/freebsd/freebsd-ports/commit/0033c2a4c4b4346e7ace80282f3a238e73434ec2
It is added as comment, but code uses MAKE_ENV, CONFIGURE_ENV to pass env vars to sub process.

Why to not fix NO_CCACHE_DEPEND, like in https://github.com/rozhuk-im/freebsd-ports/commit/dfe7746fb71eb1df89474f527499a1f5a768f01c ?

khng added a comment.EditedDec 22 2020, 5:10 AM
In D27393#619474, @rozhuk.im-gmail.com wrote:

IMHO,
.MAKEFLAGS: NO_CCACHE_DEPEND=yes
may stop work if some port define NOPRECIOUSMAKEVARS.

NOPRECIOUSMAKEVARS will not stop variables not in _EXPORTED_VARS being exported to do-depends.sh

Why to not fix NO_CCACHE_DEPEND, like in https://github.com/rozhuk-im/freebsd-ports/commit/dfe7746fb71eb1df89474f527499a1f5a768f01c ?

I am not sure if it is really a good idea to simply let every dependencies inheriting NO_CCACHE_DEPEND. But if NO_CCACHE_DEPEND is only used by ccache-* then it seems fine to me.

rozhuk.im-gmail.com added a comment.Dec 23 2020, 1:53 AM

If all ccache deps does not inherit NO_CCACHE_DEPEND then you may get infinite recursion, because some of dep may require ccache.
I already try bootstrap without this and get: ccache->cmake->ccache->cmake.....

khng added a comment.Dec 23 2020, 1:59 AM

Yes. And .MAKEFLAGS: NO_CCACHE_DEPEND=yes should be sufficient on its own.
If NO_CCACHE_DEPEND is only a thing in devel/ccache then factoring the
logic out to Mk/* is also fine.

rozhuk.im-gmail.com added a comment.Dec 23 2020, 6:37 PM

Yes, please use Mk.
It will be useful for https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=252070 and https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=252088

khng updated this revision to Diff 81119.Dec 24 2020, 7:01 AM
  • Revert "devel/ccache: Apply NO_CCACHE_DEPEND=yes on Mk/Scripts/do-depends.sh"
  • Let NO_CCACHE_DEPEND propagate to the dependencies as well
Herald added a reviewer: portmgr. · View Herald TranscriptDec 24 2020, 7:01 AM
Harbormaster completed remote builds in B35662: Diff 81119.Dec 24 2020, 7:01 AM
khng added a comment.EditedDec 24 2020, 7:01 AM
In D27393#620127, @rozhuk.im-gmail.com wrote:

Yes, please use Mk.
It will be useful for https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=252070 and https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=252088

I will move it to bsd.ccache.mk, but without CCACHE_ENV. Directly expanding CCACHE_ENV when invoking {SETENV} (which simply allows things inside sift through) in place seems to me doesn't match how the other flags work in do-depends.sh currently. At the end, this should still plays well with your proposed ports.

khng updated this revision to Diff 81255.Dec 28 2020, 9:54 AM
  • ZSTD_SYSTEM_STATIC=On could be replaced with explicit path to libzstd.a
Harbormaster completed remote builds in B35725: Diff 81255.Dec 28 2020, 9:54 AM
bdrewery added a comment.Jul 12 2021, 5:55 PM

Thanks for this. I split out ccache-memcached recently into its own port. It sucks that cmake and zstd are both dependencies now.
I also have local patches for src/ccache.cpp as well.
I will take what's here and adapt/credit where possible.

Mk/bsd.ccache.mk
36 ↗(On Diff #81255)

I like the idea here but I believe this only works with direct port builds but will not with Poudriere. At least not without using the DEPENDS_ARGS hack which I think was mutually exclusive with FLAVORS support. I'll look into that.

tmunro added a subscriber: tmunro.Sep 2 2021, 2:31 AM
diizzy added a subscriber: diizzy.Oct 29 2021, 1:36 AM

Regarding libmemcached that might be worth looking at
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=258816

fluffy added a subscriber: fluffy.Nov 23 2021, 5:04 AM

Revision Contents
Changeset List

Diff 80998

View Options

devel/ccache/Makefile

Loading...
View Options

devel/ccache/distinfo

Loading...
View Options

devel/ccache/files/extra-patch-CMakeLists.txt

Loading...
View Options

devel/ccache/files/extra-patch-Makefile.in

Loading...
View Options

devel/ccache/files/extra-patch-cmake_Findzstd.cmake

Loading...
View Options

devel/ccache/files/extra-patch-src__ccache.c

Loading...
View Options

devel/ccache/files/patch-src__ccache.c

Loading...
View Options

devel/ccache/pkg-plist

Loading...