Page MenuHomeFreeBSD

Fix --sysroot for cross-toolchain GCC packages

Authored by jhb on Mar 8 2018, 6:28 PM.



By default, GCC assumes that a cross compiler should not have any valid
initial include or library paths aside from /usr/local/lib/gcc/<mumble>/.
This means that one cannot use --sysroot to point to a system root for
another architecture and have GCC automatically look for headers in
${sysroot}/usr/include, etc. Currently we workaround this in FreeBSD's
build system with explicit -isystem, -B, and -L directives. However,
this is cumbersome compared to clang (where a bare --sysroot DTRT)
especially when using the compiler to build other software (such as test
programs, etc.).

One can override GCC's assumption and force it to assume that it should
honor --sysroot by setting the '--with-sysroot' option to force GCC to
assume a specific system root. By setting this to '/', this means that
the cross-compiler will attempt to use the host's headers by default
if --sysroot is not specified, but if --sysroot is specified then
it is fully honored including for include paths and library paths. With
this change I can now cross-compile both C and C++ binaries simply by
using --sysroot without the need for -isystem, -B, or -L directives. Note
that the base/gcc and devel/riscv64-gcc ports both use --with-sysroot='/'

By default, GCC looks for headers in /usr/local/include (under the sysroot)
before /usr/include. To disable this and only look for headers in /usr/include,
patch gcc/ to not define LOCAL_INCLUDE_DIR.

Once -nostdinc is no longer required, the headers installed along with the
compiler are now used for compiling worlds and test programs. The "fixed"
headers in include-fixed are generally not helpful and are also derived from
the host's headers which might not match the target --sysroot, so just delete
them entirely. Even the stub limits.h headers GCC ships when using an empty
build sysroot are not helpful and need to be removed.

On a related note, I also fixed the name of the C++ include directory
option to configure. By my reading it is 'with-gxx-include-dir' rather
than 'with-gcc-include-dir'.

Test Plan
  • Tested with mips-gcc building some simple test C and C++ programs by just using --sysroot. To verify the include paths were correct I used -v and examined the list of include paths reported.
  • Am also doing ongoing testing of build worlds on MIPS with all of the -I/-L/-B flags removed from the GCC case and only using --sysroot.

Diff Detail

rP FreeBSD ports repository
Automatic diff as part of commit; lint not applicable.
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

In my testing, this causes a plist issue:

===> Checking for items in STAGEDIR missing from pkg-plist
Error: Orphaned: lib/gcc/%%GCC_TARGET%%/%%GCC_VERSION%%/include-fixed/libmilter/mfapi.h
Error: Orphaned: lib/gcc/%%GCC_TARGET%%/%%GCC_VERSION%%/include-fixed/netinet/ip_fil.h
Error: Orphaned: lib/gcc/%%GCC_TARGET%%/%%GCC_VERSION%%/include-fixed/netinet/ip_lookup.h
Error: Orphaned: lib/gcc/%%GCC_TARGET%%/%%GCC_VERSION%%/include-fixed/netinet/ip_nat.h
Error: Orphaned: lib/gcc/%%GCC_TARGET%%/%%GCC_VERSION%%/include-fixed/netinet/ip_proxy.h
Error: Orphaned: lib/gcc/%%GCC_TARGET%%/%%GCC_VERSION%%/include-fixed/netinet/ip_scan.h
Error: Orphaned: lib/gcc/%%GCC_TARGET%%/%%GCC_VERSION%%/include-fixed/netinet/ip_state.h
Error: Orphaned: lib/gcc/%%GCC_TARGET%%/%%GCC_VERSION%%/include-fixed/stddef.h
Error: Orphaned: lib/gcc/%%GCC_TARGET%%/%%GCC_VERSION%%/include-fixed/stdio.h
Error: Orphaned: lib/gcc/%%GCC_TARGET%%/%%GCC_VERSION%%/include-fixed/stdlib.h
Error: Orphaned: lib/gcc/%%GCC_TARGET%%/%%GCC_VERSION%%/include-fixed/sys/types.h
Error: Orphaned: lib/gcc/%%GCC_TARGET%%/%%GCC_VERSION%%/include-fixed/unistd.h
Error: Orphaned: lib/gcc/%%GCC_TARGET%%/%%GCC_VERSION%%/include-fixed/wchar.h

We generally don't want to use "fixed" includes with GCC anyway since I don't think any of its changes to headers are improvements. I'm not sure how others feel but I'd be happy if we just rm'd include-fixed in a post-install package and didn't include any of the headers. I wonder if we shouldn't do something like '--with-build-sysroot=/nonexistent' so that it doesn't try to use host headers though. I can try that and see what happens. Also, I see that for base/gcc we already use '--with-sysroot="/"' at least.

(Curiously my log file for mips-gcc doesn't show the same failure)

Deleting them sounds fine to me.

  • Use a hack to avoid using host headers for include-fixed.

    Create an empty build sysroot (containing only an empty /usr/include) in the work directory and use it as --build-sysroot. Trying to use /nonexistent failed to build with an error, but explicitly using an empty directory restores the previous behavior of only installing the GCC distribution limits headers and a README.

I think I've found a way to preserve the previous include-fixed contents (so fixing the PLIST warnings) without breaking the build. Hopefully creating a stub build_sysroot under ${WRKDIR} isn't too gross of a hack.

This revision is now accepted and ready to land.Mar 11 2018, 12:46 AM

One caveat is that the built GCC does look for headers in /usr/local/include
(under the sysroot) before /usr/include, but I think that is an existing
problem in the native GCC ports as well and whatever solution is used for
that would be applicable here.

And libs. See rS297435

One caveat is that the built GCC does look for headers in /usr/local/include
(under the sysroot) before /usr/include, but I think that is an existing
problem in the native GCC ports as well and whatever solution is used for
that would be applicable here.

And libs. See rS297435

So we'll still need -L, and -isystem, to continue working on the systems it works on now due to ports headers/libraries. Fixing that seems like a blocker for this commit to be worthwhile to only require --sysroot in the base build.

brooks added a subscriber: brooks.

We set sysroot to / in base so this seems like the right change.

I do wonder if we might want to specify a ${LOCALBASE}/sysroots hierarchy at some point and point these things to .../sysroots/<OS_REL>/<TARGET>.<TARGET_ARCH>.

I agree with the notion of killing off the include-fixed abomination, but that's probably best done separately.

To be clear, this improves the usefulness of the cross-compiler for cases outside of just buildworld. Previously to cross-compile _anything_ I had to set '-isystem', etc. since there were no built-in include paths. For example, I have a 'mips64_gcc' script in my ~/bin that looks like this:


mips-unknown-freebsd11.1-gcc --sysroot ${ROOTFS} -mabi=64 -nostdinc -isystem ${ROOTFS}/usr/include -B ${ROOTFS}/usr/lib -L${ROOTFS}/usr/lib -msoft-float "$@"

along with a matching mips64_g++, etc. Repeat for mips, mipsn32, mips64hf, etc. I can replace all of this now with just

mips-unknown-freebsd11.1-gcc --sysroot /home/john/work/qemu/mips64/rootfs -mabi=64 -msoft-float ...

I do think the fact that '/usr/local' is in the paths is probably fixable in the GCC build (and I think lang/gcc48, etc. are all similarly affected).

Hmm, I think the most expedient way to fix "/usr/local" is to just use sed to remove the line containing 'LOCAL_INCLUDE_DIR' from gcc/Makefile in a post-patch target. I'll try that and see what I get.

  • Remove LOCAL_INCLUDE_DIR from gcc/ This removes /usr/local/include from the default set of include paths.
This revision now requires review to proceed.Mar 12 2018, 10:48 PM

Here's what I get now:

> mips-unknown-freebsd11.1-gcc -v -c ~/empty.c --sysroot ~/work/qemu/mips/rootfs 
Using built-in specs.
Target: mips-unknown-freebsd11.1
Configured with: /wrkdirs/usr/ports/devel/mips-gcc/work/gcc-6.3.0/configure --target=mips-unknown-freebsd11.1 --disable-nls --enable-languages=c,c++ --without-headers --with-gmp=/usr/local --with-pkgversion='FreeBSD Ports Collection for mips' --with-system-zlib --with-gxx-include-dir=/usr/include/c++/v1/ --with-build-sysroot=/wrkdirs/usr/ports/devel/mips-gcc/work/build_sysroot --with-sysroot=/ --with-as=/usr/local/bin/mips-freebsd-as --with-ld=/usr/local/bin/mips-freebsd-ld --prefix=/usr/local --localstatedir=/var --mandir=/usr/local/man --infodir=/usr/local/info/ --build=amd64-unknown-freebsd11.1
Thread model: posix
gcc version 6.3.0 (FreeBSD Ports Collection for mips) 
COLLECT_GCC_OPTIONS='-v' '-c' '-EB' '-mabi=32' '-march=from-abi'
 /usr/local/libexec/gcc/mips-unknown-freebsd11.1/6.3.0/cc1 -quiet -v -isysroot /home/john/work/qemu/mips/rootfs /home/john/empty.c -meb -quiet -dumpbase empty.c -mabi=32 -march=from-abi -auxbase empty -version -o /tmp//ccX9rbaa.s
GNU C11 (FreeBSD Ports Collection for mips) version 6.3.0 (mips-unknown-freebsd11.1)
        compiled by GNU C version 4.2.1 Compatible FreeBSD Clang 5.0.0 (tags/RELEASE_500/final 312559), GMP version 6.1.2, MPFR version 3.1.6, MPC version 1.0.3, isl version none
GGC heuristics: --param ggc-min-expand=100 --param ggc-min-heapsize=131072
ignoring nonexistent directory "/usr/local/lib/gcc/mips-unknown-freebsd11.1/6.3.0/../../../../mips-unknown-freebsd11.1/include"
#include "..." search starts here:
#include <...> search starts here:
End of search list.
GNU C11 (FreeBSD Ports Collection for mips) version 6.3.0 (mips-unknown-freebsd11.1)
        compiled by GNU C version 4.2.1 Compatible FreeBSD Clang 5.0.0 (tags/RELEASE_500/final 312559), GMP version 6.1.2, MPFR version 3.1.6, MPC version 1.0.3, isl version none
GGC heuristics: --param ggc-min-expand=100 --param ggc-min-heapsize=131072
Compiler executable checksum: fe8d29a759fd21a0807614e2117cb49e
COLLECT_GCC_OPTIONS='-v' '-c' '-EB' '-mabi=32' '-march=from-abi'
 /usr/local/bin/mips-freebsd-as -v --traditional-format -EB -O1 -no-mdebug -mabi=32 -march=from-abi -KPIC -o empty.o /tmp//ccX9rbaa.s
GNU assembler version 2.28 (mips-freebsd) using BFD version (GNU Binutils) 2.28
COLLECT_GCC_OPTIONS='-v' '-c' '-EB' '-mabi=32' '-march=from-abi'

Note no '/usr/local/include' in the include paths and LIBRARY_PATH doesn't include /usr/local/lib either.

I'll try some world builds without the GCC hacks in Makefile.inc1 as well. I think I can even fix GCC 4.2.1 to DTRT so we can definitely remove the hacks.

  • Add a PORTREVISION bump.
  • Update pkg-plist for amd64 since it doesn't have a bunch of include-fixed from the host anymore.
jhb edited the test plan for this revision. (Show Details)
jhb added a subscriber: imp.
  • Remove include-fixed entirely. It's <limits.h> is too limited to be useful with a --sysroot.

Removing include-fixed seems like a good idea. It feels like a relic of the 90s when there were a zillion randomly divergent commercial unixes.

This revision is now accepted and ready to land.Mar 21 2018, 4:57 PM
  • Now that we are axeing all of include-fixed anyway, remove the build_sysroot hack as it isn't needed.
This revision now requires review to proceed.Mar 21 2018, 5:58 PM

Ok, I am finally happy with this and barring any objections in the next day or so will commit it. I have a similar patchset for riscv64-gcc I will upload in a bit. I have a pending change in D14780 to fix in-tree GCC 4.2.1 to work with only --sysroot. Once all three of those are in, we can remove the hacks to set the extra flags when building worlds with GCC. I haven't done a test build with a patched riscv64-gcc yet, but will do so soon. I have passed 'make tinderbox' with the change in D14780 with only --sysroot for GCC, and I've done builds of various mips ABIs using the patched mips-gcc built with these changes using only --sysroot.

  • Rebase on today's ports tree.
This revision was not accepted when it landed; it landed in state Needs Review.Mar 23 2018, 11:54 PM
This revision was automatically updated to reflect the committed changes.