Initial changes to Makefile.inc1 to allow crossbuilding on Linux and Mac
AbandonedPublic
Actions

Authored by arichardson on Feb 2 2018, 5:15 PM.

Details

Reviewers

emaste
jhb
brooks
imp
bdrewery

Summary

When crossbuilding we can't use most of basic host commands since they have
slightly incompatible behaviour. Instead we need to build all tools during
the bootstrap-tools stage and then ensure that PATH only contains those
binariesinstead of the default host tools. For tools that are known to be
compatible with what the build systems expects (cp, env, etc.) we can
just add a symlink to $WORLDTMP instead of building them from the FreeBSD
sources.

In addition to this I also had to change all mtree invocations to include -W
when building on Linux/Mac since not all users and groups exist there.

Diff Detail

Lint

Lint Passed

Unit

No Test Coverage

Build Status

Buildable 14847
Build 14961: arc lint + arc unit

Event Timeline

arichardson created this revision.Feb 2 2018, 5:15 PM

emaste added inline comments.Feb 2 2018, 5:29 PM

Makefile.inc1
341	I'm afraid this variable might be confusing - it's likely to be interpreted as meaning we're crossbuilding a different FreeBSD arch. Maybe "CROSSBUILD_HOST"?
473–476	I think (later on) we should do this symlink thing even for FreeBSD on FreeBSD.
538	It's conventional to set BOOTSTRAPPING to 0 to mean "build everything" I think.
840–844	it looks like this should be committed as a NO_ROOT bugfix now?
845–846	this else/endif seems strange

I'd be inclined to break the ${FOO_MTREE} changes out and get them committed ASAP. Those look like a no-brainer-yes while the rest of this patch needs further study since my spidy sense is telling me there's some unintended side-effects still...

Makefile.inc1
373–374	I'd just set this to 0.
473–476	We used to do this. We relaxed it so more things would work without needing a full buildworld on a 'You are on your own if it fails' basis.
840–844	I'm wondering why we'd need this conditionally. Why not do it all the time at least for NO_ROOT?
953	Doesn't this kill it for the normal FreeBSD on FreeBSD case?
954	I'm of the opinion we should do this in the libegacy stuff. Is there a reason we can't do it there?
1250–1253	I think this now requires a full buildworld before these targets will work. I think that will break mergemaster and its ilk.
1281	This looks like a good patch that can be separated out.

I agree with @imp that pulling out the mtree bits makes sense.

Makefile.inc1
842	"thispass" ?
922	Capitalize "when". There are other similar instances.
929	This list clearly grew (or shrank?) organically and should be rewrapped to 80 columns and with breaks in sensible places.
947–949	It seems like there should be a comment indicating when this like should be updated. It's not clear to me how this particular list of directories is special.
953	It looks like the non-legacy part is now further down in `_build_tools`
954	If it would work that sounds like a good idea, though libegacy sounds like a weird name for a symlink creation tool.

Thanks for the feedback. I'll extract the mtree NO_ROOT changes to a separate patch.

Makefile.inc1
840–844	I'm happy to do it unconditionally, I just wasn't sure if it might break anything so I did it only for crossbuilding.
845–846	I originally had something there, will remove.
947–949	This list of directories is everything that I needed to build libegacy plus bootstrap-tools. I'm not sure if there is a better way of doing this without mtree. The full list of changes with lots of hacks that need cleaning up is here: https://github.com/arichardson/cheribsd/compare/baseline...arichardson:crossbuild-new
953	Building on FreeBSD worked fine for me.
954	By doing it in the libegacy stuff you mean moving this to tools/build/Makefile?
1250–1253	I can change it to only set PATH when crossbuilding.

Give me some time to look over this please. It conflicts majorly with some work I've been doing to build the toolchain *once* for make universe.

Makefile.inc1
373–374	Yup 0

This revision now requires changes to proceed.Feb 2 2018, 6:38 PM

imp added inline comments.Feb 2 2018, 6:57 PM

Makefile.inc1
954	We should have a general 'foreign OS cross building setup' thing. Otherwise as we expand this list we'll have bloat here. This file churns a lot, so having it in tools/build/<something> would keep it (a) isolated form the churn and (b) allow for easier new foreign OS environments w/o too much conflict.

arichardson updated this revision to Diff 38999.Feb 7 2018, 2:58 PM

arichardson marked 9 inline comments as done.

arichardson added parent revisions: D14185: Don't change directory owner to root when building with -DNO_ROOT, D14241: crossbuild: don't assume <bsd.compiler.mk> always exists.

Also bootstrap ar since the macos ar won't work

Harbormaster completed remote builds in B14846: Diff 39005.Feb 7 2018, 4:38 PM

Fix typo in last version of the diff

emaste added inline comments.Feb 8 2018, 2:44 AM

Makefile.inc1
345–349	As a longer term goal I would like to see all builds migrate to not being done as root.
627	It has been useful to have debug files for build tools; maybe this should be made optional, and set by default for cross builds? I'm fine with an argument for just adding `MK_DEBUG_FILES=no` and hacking it out if necessary to debug something.
670–677	I would suggest that changes like this, that are improvements not directly tied to cross building, be separated out.

I think we're not there yet...

Also, I'm growing to dislike 'crossbuild' because every single build is a cross build. This is building on a foreign OS, and we should update our comments / variables to reflect that.

Makefile
191	This is too specific. Maybe it works for 'brew' but doesn't work for 'ports' since stuff is installed elsewhere: which bmake /opt/local/bin/bmake
Makefile.inc1
345–349	We can't. We already BUILD w/o root at all. The problem is that installworld has to run as root when upgrading a system. There's no way around that.
538	We may need to expand this somewhat. Linux might need a different set than MacOS. And why does incompatible old tools on FreeBSD have anything to do with what we need to build on these other systems?
554	We have a mix of .if defined() .else .end and .if !defined() .else .end Pick one :)
584	Given the number of invasive changes so far, and some of my comments about things being 'oddly specific' I wonder if taking a step back and asking ourselves "Could we do this with .include Makefile.inc1.{linux,darwin,etc} instead?"
587	Why not? I hate 'for some reason' root causes, because maybe that means this papers over another problem.
627	Generally, debug files are only useful for final build targets, so I'd be fine with this :)
848	Is this built early enough that we can do this always, unconditionally?
930	delete-old* are only useful on installed hosts, so what's the use case for build on a foreign OS?

emaste added inline comments.Feb 8 2018, 6:36 PM

Makefile.inc1
345–349	`installworld` and `installkernel` need root of course, but there are a number of other targets that require root, e.g. the release targets. That does raise a good point here though: installworld and installkernel don't make sense to execute on a non-FreeBSD system, so which targets are you using in the crossbuild case that are affected by this change?
538	For FreeBSD this builds tools that are either incompatible or missing on the host, so the crossbuild is really just a special case of that.
627	That is generally true except when something fails with the bootstrap tools :) It does happen, but not often, and is likely not worth the build time and disk space to leave it on for everyone. This is why I'm fine with just hacking it out if necessary for debugging.

brooks added inline comments.Feb 8 2018, 7:06 PM

Makefile.inc1
345–349	Both targets are perfectly sensible on non-FreeBSD systems so long as you specify NO_ROOT. There's an argument that for the 99% of those targets that is running `install` we should have a separate target (run by `installworld` and `installkernel`) that generates a METALOG that is used to drive the actually installation of files.
584	This part should ideally be a .for loop to make the list easy to expand.
930	Not supporting `delete-old*` seems fine. Those targets only make sense when running as root and targeting the host.
1010	Does this need to survive? It's a bit weird to populate legacy with `lib lib/casper usr` and then add all the debug directories...

I agree the crossbuild name is not ideal. How about FOREIGN_OS_BUILD?

Makefile.inc1
345–349	Yes, I always run the install targets to get a directory that I can either use to build a disk image from the METALOG or as a NFS rootfs for the CHERI qemu.
538	Building every possible bootstrap tool from the sources ensures that it behaves as expected by the build system. So if there are old FreeBSD versions that won't work it probably also means that the versions on Linux/MacOS won't work.
587	I spent over two hours trying to see why it was still running `clang --version`. the hashed values looked sensible to me but it was not picking them up. After tons of debug prints in the makefile I just gave up and passed the values in the environment.
627	The reason I did this is so that I don't need a working objcopy (which won't exist on MacOS). I don't mind building with debug info, I just want to avoid the step that uses objcopy to create a ${PROG}.debug file from ${PROG} .full.
670–677	I think this is the same problem as above with bsd.compiler.mk caching not working. I just noticed it because I was running `bmake -dx`. I'll see if the same thing also happens with current HEAD on FreeBSD.
930	Should I change this to `!defined(NO_ROOT)`?
1010	I guess the the `mkdir lib lib/casper usr` above can be removed since it only installs to WORLDTMP/legacy so far.

jhb added inline comments.Feb 12 2018, 6:10 PM

Makefile.inc1
345–349	Eh, I use installworld, installkernel, and delete-old all the time in a cross environment to update a 'rootfs' tree that I then use 'makefs' to generate a bootable UFS image from. In my case it is using a FreeBSD/amd64 host to build a FreeBSD/mips bootable image to run in qemu, but I don't see why that same workflow wouldn't be sensible if I had a non-FreeBSD host and was still going to run a FreeBSD disk image in qemu, or build a FreeBSD disk image that I could copy onto the SD card that I eject and stick in a Pi, etc. This does assume you have a tool like makefs, but having a portable version of makefs doesn't seem that tall of an order.

I've been really busy lately and don't want to hold up anything but will try to look more seriously at this and others when I can.

emaste added inline comments.Mar 8 2018, 7:49 PM

Makefile.inc1
345–349	Ah, indeed.
345–349	Makefs should already be quite portable courtesy of NetBSD

emaste added inline comments.Mar 8 2018, 7:59 PM

Makefile
191	Or, should we just not set `PATH` on non-FreeBSD hosts, and expect that the user has an appropriate path?
Makefile.inc1
627	Nah, I think it's fine to make this change.