Upgrade to version 4.7.5 and take over maintainership from ports@.
There is something going on with optimizations on both the old and
new versions.
If you set USE_GCC=any the port will build way faster:
gcc: make 506.90s user 17.86s system 228% cpu 3:49.41 total
clang: make 3913.28s user 23.73s system 211% cpu 31:05.70 total
I added a GCC option for now.
Setting CFLAGS to -O0 will make clang go as fast as gcc.
Explanation still unknown.