Helper target native-xbuild for qemu-user package building jails

[imp]

Summary

Create the native-xtools target. This target creates only the cross
building toolchain for the host computer. This toolchain produces
TARGET_ARCH and assumes the rest of the system contains libraries for
the target. It is intended to be used in a."qemu-user jail" where all
the binaries would otherwise be the target architecture's to build
ports. However, emulation of the compilers is too slow, so we build
native binaries for that. Rather than use the xdev produced binaries,
with all their weird links and paths, these binaries use the native
paths. They will not work unless installed into the qemu-user jail.

[imp]

Makefile.inc1

1421

BMAKEENV is wrong here. It pulls in all the early bootstrap crap, including libegacy, which we're not creating.

1438

Clang tlbgen stuff is needed too.

[imp]

Fixed issues noted in review / testing. Clang seems to work now.

[sbruno]

Running a test build here: http://dirty.ysv.freebsd.org/build.html?mastername=11armv632-default&build=2014-08-12_04h59m42s

I literally copied /usr/obj/nxb-bin into the jail and adjusted make.conf

[sbruno]

This doesn't seem to generate a ranlib binary as the xdev target does. Should it?

[imp]

Yes. It should. Are there any others missing?
BTW, I also plan on experimenting with generating a /bin/sh, /bin/make and maybe /bin/awk and /bin/sed to help the build times.

[sbruno]

I am using this as a make.conf, after doing a direct mv of /usr/obj/arm.armv6/nxb-bin into the jail root:

CC=/nxb-bin/usr/bin/cc
CPP=/nxb-bin/usr/bin/cpp
CXX=/nxb-bin/usr/bin/c++
AS=/nxb-bin/usr/bin/as
NM=/nxb-bin/usr/bin/nm
LD=/nxb-bin/usr/bin/ld
OBJCOPY=/nxb-bin/usr/bin/objcopy
SIZE=/nxb-bin/usr/bin/size
STRIPBIN=/nxb-bin/usr/bin/st

[sbruno]

In D518#8, @imp wrote:

Yes. It should. Are there any others missing?
BTW, I also plan on experimenting with generating a /bin/sh, /bin/make and maybe /bin/awk and /bin/sed to help the build times.

make seems to be safe, as are sed and awk. I've done static builds of each to drop into the jail and they seem to work.

/bin/sh is a bit more problematic. There's some qemu foo that's in the way that Stacey knows about.

[sbruno]

http://chips.ysv.freebsd.org/build.html?mastername=11armv632-default&build=2014-08-12_17h14m28s

Moved over here as dirty.ysv seems to run 5x slower than chips for no good reason. :-)

[imp]

I don't suppose you've enabled process accounting to find out what the biggest users of CPU are during a build?

[sbruno]

In D518#13, @imp wrote:

I don't suppose you've enabled process accounting to find out what the biggest users of CPU are during a build?

now that I think about this ... I would have to generate a list of things from an AMD64 build, as process accounting on this QEMU enabled run is going to let me know that QEMU is running a lot. right?

[imp]

Update for newer files that we want.

[sbruno]

It looks like I am allowed to override quite a few tools in a ports build via make.conf

I've applied this to a head tree and build the native-xbuild target. copied the tree into the poudriere jail and modified make.conf with the following:

cat /usr/local/poudriere/jails/11armv632/etc/make.conf
CC=/nxb-bin/usr/bin/cc
CPP=/nxb-bin/usr/bin/cpp
CXX=/nxb-bin/usr/bin/c++
AS=/nxb-bin/usr/bin/as
NM=/nxb-bin/usr/bin/nm
LD=/nxb-bin/usr/bin/ld
OBJCOPY=/nxb-bin/usr/bin/objcopy
SIZE=/nxb-bin/usr/bin/size
STRIPBIN=/nxb-bin/usr/bin/strip
SED=/nxb-bin/usr/bin/sed
READELF=/nxb-bin/usr/bin/readelf
RANLIB=/nxb-bin/usr/bin/ranlib
YACC=/nxb-bin/usr/bin/yacc
NM=/nxb-bin/usr/bin/nm
MAKE=/nxb-bin/usr/bin/make
STRINGS=/nxb-bin/usr/bin/strings
AWK=/nxb-bin/usr/bin/awk
FLEX=/nxb-bin/usr/bin/flex

[sbruno]

Two thumbs up from me on this one. Current builds are flying along.

[imp]

This was committed with svn http://svnweb.freebsd.org/base?view=revision&revision=270155

[bdrewery]

I'm sure this is the perfect place for this!

Makefile.inc1

1449–1454

This stuff and everything else added after this commit really needs to have native libraries and includes built into a sysroot as well. The biggest problem is that 'native-xtools' uses host libraries/includes to generate its static binaries.

A few months ago I ran into the problem where building 'native-xtools' from poudriere on one of the build systems was wanting to link in libraries that were *not present* on the host since we were building a newer src tree than the host.

I worked around it by making poudriere always use 'native-xtools' from /usr/src to ensure the tree matches, but this could lead to subtle problems. For example (this wouldn't be a real issue but I can't think of a real one) suppose the qemu jail has a newer libc++ lib than the cross compiler we are building from the older /usr/src was built for.

Today I am running into the same issue with using host includes after rS289773 changed sys/sysctl.h. Now native-xtools can't build on my host since my /usr/include/sys/sysctl.h is too old.

Really native-xtools being just the toolchain that is slightly different than xdev seems fine to me.

Everything else should really just be 'make buildworld WITHOUT_TOOLCHAIN= WITHOUT_SHARED= MAKEOBJDIRPREFIX=${NXBSOMETHING}' with perhaps a SUBDIR_OVERRIDE list. This could be done directly from the 'native-xtools' target. Yes it incurs a lot of overhead but otherwise this target is very problematic to use.

[imp]

Native xtools was intended to build with host tools, and implicitly assumed host == target as far as FreeBSD version was concerned.
It isn't too surprising that it is failing when you cross thread.
We need to build it against a proper sysroot for the HOST's architecture.

[bdrewery]

In D518#85637, @imp wrote:

Native xtools was intended to build with host tools, and implicitly assumed host == target as far as FreeBSD version was concerned.
It isn't too surprising that it is failing when you cross thread.
We need to build it against a proper sysroot for the HOST's architecture.

Yes that's what I am saying. It doesn't do that now. It assumes you are building against a / that matches the src.