Page MenuHomeFreeBSD

Use Clang and LLD as the default toolchain for MIPS
ClosedPublic

Authored by arichardson on Jan 16 2020, 11:47 AM.

Details

Summary

Now that we have updated the in-tree version of LLVM to 10.0, we have all the
necessary LLVM changes to use Clang+LLD as the default toolchain for MIPS.

Relnotes: yes

Test Plan

Booted a clang + lld world MIPS64 in qemu.

Diff Detail

Repository
rS FreeBSD src repository
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

arichardson created this revision.Jan 16 2020, 11:47 AM
brooks accepted this revision.Jan 16 2020, 12:44 PM

This seems fine. I don't think there's much evidence that anyone currently cares about n32.

This revision is now accepted and ready to land.Jan 16 2020, 12:44 PM

I don't think there's much evidence that anyone currently cares about n32.

Should we work on an official deprecation?

@kevans can probably confirm that 32-bit mips is working

Makefile
495 ↗(On Diff #66828)

I suppose we're set up to use the same toolchain for all of mips, and with this change n32 is no longer part of universe. OK.

emaste accepted this revision.Jan 16 2020, 2:06 PM

I guess I could try to build and boot a n32 world, but I'm not sure which kernel I should be using to test it.

Makefile
495 ↗(On Diff #66828)

Yeah I couldn't see a way of using external toolchain only for a given TARGET_ARCH so I dropped n32 from the list instead.

kevans accepted this revision.Jan 16 2020, 2:21 PM

Yeah, so I succesfully built all of the ports that we use in freebsd-wifi-build with in-tree llvm (+ your patch that just got committed here) on o32; I test-booted MALTA a while ago just fine (with no substantial changes to llvm that would break it in the meantime) and I built a new image for one of my o32 dev units to test on real hardware, but haven't imaged one with it yet.

I guess I could try to build and boot a n32 world, but I'm not sure which kernel I should be using to test it.

Fair enough, I don't know either :)

I think we should commit this as is, and subsequently follow up on lists etc. to see if anyone is interested in n32. Either it gets tested (and fixed, if there's an issue) or if nobody cares about it we retire it before 13.

jhb added a comment.Jan 16 2020, 4:49 PM

I already have n32 listed as Tier 3 I think for 13 I think which means it could stay in the tree, but would be disconnected from universe. There's not a whole lot of n32 stuff to really remove. Most of it is is #if n64 || n32

arichardson planned changes to this revision.Jan 17 2020, 10:46 AM

mips64hf is currently broken because libc_nossp is built as soft-float (I think this is the lib32 version only since the error happens when building rtld-elf32):

ld: error: /home/alr48/obj/build/freebsd-universe-build/exports/users/alr48/sources/freebsd-universe/mips.mips64hf/libexec/rtld-elf32/rtld_libc.a(lstat.nossppico): floating point ABI '-msoft-float' is incompatible with target floa
ting point ABI '-mdouble-float'

It also seems like we need to cherry pick more LLVM fixes first:

mips OCTEON1 kernel failed, check _.mips.OCTEON1 for details
mips XLP64 kernel failed, check _.mips.XLP64 for details
mips XLPN32 kernel failed, check _.mips.XLPN32 for details
mips ERL kernel failed, check _.mips.ERL for details
sys/mips/cavium/asm_octeon.S
Assertion failed: (OffsetOp.isExpr() && "expected expression operand kind"), function expandMemInst, file /exports/users/alr48/sources/freebsd-universe/contrib/llvm-project/llvm/lib/Target/Mips/AsmParser/MipsAsmParser.cpp, line 3702.

I believe there were a bunch of octeon fixes in LLVM recently, so I'll check if one of those fixes the assert.

I also tried building n32 but that fails when linking the syscall objects. My guess is that SYS.h is missing the PIC n32 case and always generates position dependent assembly.

ld: error: can't create dynamic relocation R_MIPS_HI16 against symbol: __cerror in readonly segment; recompile object files with -fPIC or pass '-Wl,-z,notext' to allow text relocations in the output
>>> defined in cerror.pico
>>> referenced by shm_rename.S:4
>>>               shm_rename.pico:(__sys_shm_rename)

I also tried building n32 but that fails when linking the syscall objects. My guess is that SYS.h is missing the PIC n32 case and always generates position dependent assembly.

ld: error: can't create dynamic relocation R_MIPS_HI16 against symbol: __cerror in readonly segment; recompile object files with -fPIC or pass '-Wl,-z,notext' to allow text relocations in the output
>>> defined in cerror.pico
>>> referenced by shm_rename.S:4
>>>               shm_rename.pico:(__sys_shm_rename)

This appears to be fixed in LLVM 10 which expands la correctly for PIC n32:

00000028 err:
      28: 8f 99 00 00                  	lw	$25, 0($gp)
			00000028:  R_MIPS_CALL16	__cerror

We could either try to cherry-pick that fix or wait for llvm10 if anyone cares about n32.

emaste added a subscriber: dim.Jan 17 2020, 2:38 PM

I am fine with waiting for llvm 10 to arrive for n32.
@dim pointed out the branch has been created already, it's not that far away

imp added a comment.Jan 17 2020, 4:47 PM

I am fine with waiting for llvm 10 to arrive for n32.
@dim pointed out the branch has been created already, it's not that far away

That's fine. There's a plan to get there. If things are delayed, or people get too annoyed, we can turn off n32 for a bit...

jhb added a comment.Jan 17 2020, 5:16 PM

mips64hf is currently broken because libc_nossp is built as soft-float (I think this is the lib32 version only since the error happens when building rtld-elf32):

ld: error: /home/alr48/obj/build/freebsd-universe-build/exports/users/alr48/sources/freebsd-universe/mips.mips64hf/libexec/rtld-elf32/rtld_libc.a(lstat.nossppico): floating point ABI '-msoft-float' is incompatible with target floa
ting point ABI '-mdouble-float'

I almost fixed this when fixing the mips64el case. Probably what we should do is set the 32-bit MACHINE_ARCH to MACHINE_ARCH:s/64// or some such to preserve el and hf suffixes.

However, note that part of the real problem here is that rtld-elf32 is somewhat broken in that it lets CFLAGS from the non-compat MACHINE_ARCH "leak" through. This doesn't
happen for the lib32 stage which is why the lib32 stage was building as soft-float since the MACHINE was "mips" but does happen for rtld-elf32. That's probably a bug in the bsd.compat.mk thing that @brooks might want to look at.

In D23204#509367, @jhb wrote:

mips64hf is currently broken because libc_nossp is built as soft-float (I think this is the lib32 version only since the error happens when building rtld-elf32):

ld: error: /home/alr48/obj/build/freebsd-universe-build/exports/users/alr48/sources/freebsd-universe/mips.mips64hf/libexec/rtld-elf32/rtld_libc.a(lstat.nossppico): floating point ABI '-msoft-float' is incompatible with target floa
ting point ABI '-mdouble-float'

I almost fixed this when fixing the mips64el case. Probably what we should do is set the 32-bit MACHINE_ARCH to MACHINE_ARCH:s/64// or some such to preserve el and hf suffixes.

However, note that part of the real problem here is that rtld-elf32 is somewhat broken in that it lets CFLAGS from the non-compat MACHINE_ARCH "leak" through. This doesn't
happen for the lib32 stage which is why the lib32 stage was building as soft-float since the MACHINE was "mips" but does happen for rtld-elf32. That's probably a bug in the bsd.compat.mk thing that @brooks might want to look at.

@jhb Like this: D23229? I'm planning to commit that later today so it should unbreak that.

What's left as a blocker on this? It'd be nice to get at least mips32 and mips64 universe/CI builds OK again, since we still haven't worked out why the freebsd-gcc port is doing the wrong thing.

I can commit this now if we are okay with octeon kernels being broken until llvm 10 lands?

llvm 10 landing date seems to be tentatively scheduled for today-ish. \o/

emaste added inline comments.Mar 10 2020, 1:54 PM
Makefile
509 ↗(On Diff #66828)

Presumably this whole block disappears when rebasing.

share/mk/src.opts.mk
301 ↗(On Diff #66828)

With sparc64 out of the tree CLANG is always yes and only CLANG_BOOTSTRAP and CLANG_IS_CC remain here.

I'm currently on holiday but will update this once I'm back on Monday.

Rebase after LLVM 10 import

D24091, D24092 and D24093 are needed to build all universe kernels.
I have only tried booting MIPS n64 MALTA since I don't have a setup for o32/n32.

emaste accepted this revision.Mar 16 2020, 11:42 PM
emaste added inline comments.
Makefile
509 ↗(On Diff #66828)

I think it's probably better to remove this block altogether, because with no entries it's going to bitrot. That said I think this change is fine for now, we can leave it here for a bit.

share/mk/src.opts.mk
293 ↗(On Diff #69582)

You can move these to the main _DEFAULT_YES_OPTIONS block above.

This revision is now accepted and ready to land.Mar 16 2020, 11:42 PM
jhb accepted this revision.Mar 17 2020, 6:34 PM
jhb added inline comments.
Makefile
508 ↗(On Diff #69582)

I think it's fine to leave this block for now. I will probably at some point add some kind of USE_GCC=yes option to let one do a 'make tinderbox' using gcc instead of clang and would reuse some of these bits for that.

arichardson retitled this revision from Use clang and lld as the default toolchain for MIPS. to Use Clang and LLD as the default toolchain for MIPS.
arichardson edited the summary of this revision. (Show Details)

Rebased on latest HEAD.

Thanks for reviewing. I will commit this tomorrow morning (UK time) so that I can react to potential CI failures.

This revision now requires review to proceed.Mar 22 2020, 7:33 PM
emaste accepted this revision.Mar 22 2020, 8:10 PM
This revision is now accepted and ready to land.Mar 22 2020, 8:10 PM
This revision was automatically updated to reflect the committed changes.