lang/gcc fix build with poudriere bulk -t /testport
ClosedPublic

Authored by miwi on Apr 11 2017, 9:42 AM.

Details

Summary

The new qa script is a little bit more strict, so currently the build fails [1] when we run poudriere testport / bulk -t,
below patch fix 2 things, first it strips all so files, secondly fix invalid shebang for bin/aot-compile5.

[1]
http://pkg2.miwibox.org/data/11amd64-miwi-ports/2017-04-10_22h18m55s/logs/errors/gcc-5.4.0.log

Diff Detail

Repository
rP FreeBSD ports repository
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.
miwi created this revision.Apr 11 2017, 9:42 AM

First of all, thanks for doing this, Martin!

This is generally fine for lang/gcc, but also lang/gcc5, lang/gcc5-devel,
lang/gcc6 and lang/gcc6-devel, all of which should be able to use pretty
much the same patch. (I'll be happy to take care of lang/gcc7-devel
which does not feature Java any longer.)

Out of curiosity: Did you focus on lang/gcc since it's the most widely
used flavor, or did the others not show up as failures for some reason?

(I wrote "generally fine" since I've got a question/small ask or two.)

+INSTALL_TARGET= install-strip

Yes, but... I tried exactly this in January 2015 and ran into problems,
cf.
https://lists.freebsd.org/pipermail/freebsd-ports/2015-January/thread.html#9
7717

(This was still on my todo list, actually. :-/)

Do you have an idea why the very same thing I did two years ago now
appears to be working fine in your tests? Not an objection to the
change per se, in fact I'm happy about it, just a little nervous and
hoping to understand the difference.

Mind moving this line directly below the ALL_TARGET a bit lower in
the Makefile?

+JAVA_USES= python shebangfix
+SHEBANG_FILES= libjava/contrib/aot-compile.in

The qa script is now going to catch this? Cool.

Note, there is a .if ${PORT_OPTIONS:MJAVA} block a bit down in
the Makefile, and it would be nice to have everything JAVA related
there (so perhaps USES+=python shebangfix and SHEBANG_FILES= or +=
down in that section)?

Thanks, and Approved by: gerald (maintainer) [ideally with the two
changes suggested],

Gerald

gerald requested changes to this revision.Apr 11 2017, 12:28 PM
This revision now requires changes to proceed.Apr 11 2017, 12:28 PM

Bad news. I just gave the patch with the INSTALL_TARGET change a
spin (lang/gcc7-devel, but that should not make a difference) and
it still does fail as follows:

test -z 'strip' || strip /scratch/tmp/gerald/gcc7-devel-work/stage/scratch/tmp/gerald/gcc7-devel-prefix/libexec/gcc7/gcc/i386-portbld-freebsd10.3/7.0.1/install-tools/fixincl
strip: unable to copy file '/scratch/tmp/gerald/gcc7-devel-work/stage/scratch/tmp/gerald/gcc7-devel-prefix/libexec/gcc7/gcc/i386-portbld-freebsd10.3/7.0.1/install-tools/fixincl'; reason: Permission denied
gmake[3]: * [Makefile:191: install-strip] Error 1
gmake[3]: Leaving directory '/scratch/tmp/gerald/gcc7-devel-work/.build/fixincludes'
gmake[2]:
* [Makefile:3735: install-strip-fixincludes] Error 2
gmake[2]: Leaving directory '/scratch/tmp/gerald/gcc7-devel-work/.build'
gmake[1]: *** [Makefile:2477: install-strip] Error 2
gmake[1]: Leaving directory '/scratch/tmp/gerald/gcc7-devel-work/.build'

  • Error code 2

This is on ref10-i386.freebsd.org building as a regular user.

miwi updated this revision to Diff 27346.Apr 11 2017, 1:21 PM
  • move shebang to the java section as suggested
  • apply same patch to gcc5 ,6 ,-devel
miwi added a comment.Apr 11 2017, 1:25 PM

First of all, thanks for doing this, Martin!

This is generally fine for lang/gcc, but also lang/gcc5, lang/gcc5-devel,
lang/gcc6 and lang/gcc6-devel, all of which should be able to use pretty
much the same patch. (I'll be happy to take care of lang/gcc7-devel
which does not feature Java any longer.)

Out of curiosity: Did you focus on lang/gcc since it's the most widely
used flavor, or did the others not show up as failures for some reason?

My focus was only on lang/gcc, I hit it while testing other ports ;).

(I wrote "generally fine" since I've got a question/small ask or two.)

+INSTALL_TARGET= install-strip

Yes, but... I tried exactly this in January 2015 and ran into problems,
cf.
https://lists.freebsd.org/pipermail/freebsd-ports/2015-January/thread.html#9
7717

(This was still on my todo list, actually. :-/)

Do you have an idea why the very same thing I did two years ago now
appears to be working fine in your tests? Not an objection to the
change per se, in fact I'm happy about it, just a little nervous and
hoping to understand the difference.

Not entirely sure, but a lot things have change over the last few years.

Mind moving this line directly below the ALL_TARGET a bit lower in
the Makefile?

ups I missed that part, I will update the patch in few mins.

+JAVA_USES= python shebangfix
+SHEBANG_FILES= libjava/contrib/aot-compile.in

The qa script is now going to catch this? Cool.

Note, there is a .if ${PORT_OPTIONS:MJAVA} block a bit down in
the Makefile, and it would be nice to have everything JAVA related
there (so perhaps USES+=python shebangfix and SHEBANG_FILES= or +=
down in that section)?

done.

Thanks, and Approved by: gerald (maintainer) [ideally with the two
changes suggested],

Gerald

miwi added a comment.Apr 11 2017, 1:30 PM

Bad news. I just gave the patch with the INSTALL_TARGET change a
spin (lang/gcc7-devel, but that should not make a difference) and
it still does fail as follows:

test -z 'strip' || strip /scratch/tmp/gerald/gcc7-devel-work/stage/scratch/tmp/gerald/gcc7-devel-prefix/libexec/gcc7/gcc/i386-portbld-freebsd10.3/7.0.1/install-tools/fixincl
strip: unable to copy file '/scratch/tmp/gerald/gcc7-devel-work/stage/scratch/tmp/gerald/gcc7-devel-prefix/libexec/gcc7/gcc/i386-portbld-freebsd10.3/7.0.1/install-tools/fixincl'; reason: Permission denied
gmake[3]: * [Makefile:191: install-strip] Error 1
gmake[3]: Leaving directory '/scratch/tmp/gerald/gcc7-devel-work/.build/fixincludes'
gmake[2]:
* [Makefile:3735: install-strip-fixincludes] Error 2
gmake[2]: Leaving directory '/scratch/tmp/gerald/gcc7-devel-work/.build'
gmake[1]: *** [Makefile:2477: install-strip] Error 2
gmake[1]: Leaving directory '/scratch/tmp/gerald/gcc7-devel-work/.build'

  • Error code 2

    This is on ref10-i386.freebsd.org building as a regular user.

Interesting,
Here, is build log from our earlier testing. It worked fine.

https://people.freebsd.org/~tcberner/logs/gcc-5.4.0_1.log

miwi updated this revision to Diff 27400.Apr 13 2017, 10:36 AM
  • Fix target location
  • added gcc7-devel
  • Build logs ----------------

lang/gcc:
http://pkg2.miwibox.org/build.html?mastername=10amd64-miwi-ports&build=2017-04-12_03h23m47s
http://pkg2.miwibox.org/build.html?mastername=10i386-miwi-ports&build=2017-04-12_03h23m47s
http://pkg2.miwibox.org/build.html?mastername=11i386-miwi-ports&build=2017-04-12_03h35m14s
http://pkg2.miwibox.org/build.html?mastername=11amd64-miwi-ports&build=2017-04-12_03h35m53s
http://pkg2.miwibox.org/build.html?mastername=12amd64-miwi-ports&build=2017-04-11_15h35m54s
http://pkg2.miwibox.org/build.html?mastername=12i386-miwi-ports&build=2017-04-12_03h49m26s

lang/gcc5:
http://pkg2.miwibox.org/build.html?mastername=10amd64-miwi-ports&build=2017-04-12_04h07m33s
http://pkg2.miwibox.org/build.html?mastername=10i386-miwi-ports&build=2017-04-12_04h07m33s
http://pkg2.miwibox.org/build.html?mastername=11i386-miwi-ports&build=2017-04-12_04h49m09s
http://pkg2.miwibox.org/build.html?mastername=11amd64-miwi-ports&build=2017-04-12_04h50m23s
http://pkg2.miwibox.org/build.html?mastername=12amd64-miwi-ports&build=2017-04-12_05h29m02s
http://pkg2.miwibox.org/build.html?mastername=12i386-miwi-ports&build=2017-04-12_05h42m52s

lang/gcc5-devel
http://pkg2.miwibox.org/build.html?mastername=10amd64-miwi-ports&build=2017-04-13_00h57m15s
http://pkg2.miwibox.org/build.html?mastername=10i386-miwi-ports&build=2017-04-13_00h57m15s
http://pkg2.miwibox.org/build.html?mastername=11i386-miwi-ports&build=2017-04-13_01h32m48s
http://pkg2.miwibox.org/build.html?mastername=11amd64-miwi-ports&build=2017-04-13_01h34m54s
http://pkg2.miwibox.org/build.html?mastername=12i386-miwi-ports&build=2017-04-13_02h11m02s
http://pkg2.miwibox.org/build.html?mastername=12amd64-miwi-ports&build=2017-04-13_02h07m54s

lang/gcc6/
http://pkg2.miwibox.org/build.html?mastername=10amd64-miwi-ports&build=2017-04-12_07h34m35s
http://pkg2.miwibox.org/build.html?mastername=10i386-miwi-ports&build=2017-04-12_07h34m35s
http://pkg2.miwibox.org/build.html?mastername=11i386-miwi-ports&build=2017-04-12_08h12m46s
http://pkg2.miwibox.org/build.html?mastername=11amd64-miwi-ports&build=2017-04-12_08h18m01s
http://pkg2.miwibox.org/build.html?mastername=12amd64-miwi-ports&build=2017-04-12_08h54m57s
http://pkg2.miwibox.org/build.html?mastername=12i386-miwi-ports&build=2017-04-12_08h56m08s

lang/gcc6-devel
http://pkg2.miwibox.org/build.html?mastername=10amd64-miwi-ports&build=2017-04-12_11h44m17s
http://pkg2.miwibox.org/build.html?mastername=10i386-miwi-ports&build=2017-04-12_11h44m17s
http://pkg2.miwibox.org/build.html?mastername=11i386-miwi-ports&build=2017-04-12_12h22m19s
http://pkg2.miwibox.org/build.html?mastername=11amd64-miwi-ports&build=2017-04-12_12h26m47s
http://pkg2.miwibox.org/build.html?mastername=12amd64-miwi-ports&build=2017-04-12_13h04m53s
http://pkg2.miwibox.org/build.html?mastername=12i386-miwi-ports&build=2017-04-12_13h06m47s

lang/gcc7-devel
http://pkg2.miwibox.org/build.html?mastername=10amd64-miwi-ports&build=2017-04-13_02h58m59s
http://pkg2.miwibox.org/build.html?mastername=10i386-miwi-ports&build=2017-04-13_02h58m59s
http://pkg2.miwibox.org/build.html?mastername=11i386-miwi-ports&build=2017-04-13_03h45m50s
http://pkg2.miwibox.org/build.html?mastername=11amd64-miwi-ports&build=2017-04-13_03h49m36s
http://pkg2.miwibox.org/build.html?mastername=12i386-miwi-ports&build=2017-04-13_04h40m11s
http://pkg2.miwibox.org/build.html?mastername=12amd64-miwi-ports&build=2017-04-13_04h36m43s

A few updates from my side:

  1. Thanks for getting the shebang fixes in, Martin!
  1. A minor typo sneaked into the change for gcc7-devel/Makefile (line 1).
  1. I have made progress in creating a small reproducer script for the permission issue I ran into.

The shebangfix changes being committed earlier now are triggering
a false positive with portlint for "JAVA_USES=python shebangfix".

WARN: Makefile: [48]: possible direct use of command "python" found. 
use ${PYTHON_CMD} instead.

I filed https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=218672 for
this. No action required (and not a complain, the committed fix still
is the right approach), just FYI.

Attached a little reproducer on how to build lang/gcc7-devel with
the proposed patch to INSTALL_TARGET that exhibits the permission
error I reported earlier.

I looked into this a bit more:

The invocation that aborts the build is

test -z 'strip' || strip /scratch/tmp/gerald/work/stage/scratch/tmp/gerald/prefix/libexec/gcc7/gcc/i386-portbld-freebsd10.3/7.0.1/install-tools/fixincl
strip: unable to copy file '/scratch/tmp/gerald/work/stage/scratch/tmp/gerald/prefix/libexec/gcc7/gcc/i386-portbld-freebsd10.3/7.0.1/install-tools/fixincl'; reason: Permission denied
gmake[3]: *** [Makefile:191: install-strip] Error 1

The file in question does exist, and file reports

ELF 32-bit LSB executable, Intel 80386, version 1 (FreeBSD), dynamically 
linked, interpreter /libexec/ld-elf.so.1, for FreeBSD 10.3 (1003508), 
FreeBSD-style, stripped

The permissions of the file are -r-xr-xr-x.

emaste added a subscriber: emaste.Apr 18 2017, 1:26 PM

Digging into this some more I tested two potential workaround for the permission problem that Tijl suggested when I first looked into this myself two years ago.

Setting BINMODE=755 and setting STRIP=true in CONFIGURE_ARGS.

Both allow the build to succeed as a regular user (non-root), when INSTALL_TARGET=install-strip is set, but with the latter there are many files left that end up being not stripped, whereas with the BINMODE setting the list is down to one file.

This appears to be the case since various aspects of GCC do not use our install-* tools, but a script install-sh (which you can find at the root of the GCC source tree that uses cp, chmod, strip,... for compatibility reasons with many systems).

This is how a working patch looks like (with BINMODE setting in
addition to INSTALL_TARGET).

Are we okay with that?

Somehow Phabricator doesn't like me today, but in any case, here is what seems to do the job,
also as regular user, not just root:

I plan on applying this to lang/gcc6, lang/gcc7 and lang/gcc5 (in that order) unless there are
any objections.

Index: Makefile

  • Makefile (revision 453487)

+++ Makefile (working copy)
@@ -94,6 +94,10 @@
CONFIGURE_ARGS+=--with-build-config=bootstrap-debug
ALL_TARGET= bootstrap-lean
.endif
+INSTALL_TARGET= install-strip
+.if ${UID} != 0
+BINMODE= 755
+.endif
CONFIGURE_ARGS+=--disable-nls \

--enable-gnu-indirect-function \ 
--libdir=${TARGLIB} \
This revision was automatically updated to reflect the committed changes.

The shebang fixes have been addressed this spring for all my lang/gcc* ports, now the stripping
is also taken care of for lang/gcc7 and lang/gcc6 (the two production ports still in maintenance)
as well as lang/gcc5.