Page MenuHomeFreeBSD

Use Clang and LLD as the default toolchain for MIPS
ClosedPublic

Authored by arichardson on Jan 16 2020, 11:47 AM.
Tags
None
Referenced Files
Unknown Object (File)
Sat, Mar 9, 10:34 AM
Unknown Object (File)
Sat, Mar 9, 10:34 AM
Unknown Object (File)
Sat, Mar 9, 10:34 AM
Unknown Object (File)
Sat, Mar 9, 10:34 AM
Unknown Object (File)
Sat, Mar 9, 10:34 AM
Unknown Object (File)
Sat, Mar 9, 9:50 AM
Unknown Object (File)
Feb 16 2024, 3:16 PM
Unknown Object (File)
Jan 18 2024, 9:40 AM
Subscribers

Details

Summary

Now that we have updated the in-tree version of LLVM to 10.0, we have all the
necessary LLVM changes to use Clang+LLD as the default toolchain for MIPS.

Relnotes: yes

Test Plan

Booted a clang + lld world MIPS64 in qemu.

Diff Detail

Lint
Lint Passed
Unit
No Test Coverage
Build Status
Buildable 28697
Build 26716: arc lint + arc unit

Event Timeline

This seems fine. I don't think there's much evidence that anyone currently cares about n32.

This revision is now accepted and ready to land.Jan 16 2020, 12:44 PM

I don't think there's much evidence that anyone currently cares about n32.

Should we work on an official deprecation?

@kevans can probably confirm that 32-bit mips is working

Makefile
495

I suppose we're set up to use the same toolchain for all of mips, and with this change n32 is no longer part of universe. OK.

I guess I could try to build and boot a n32 world, but I'm not sure which kernel I should be using to test it.

Makefile
495

Yeah I couldn't see a way of using external toolchain only for a given TARGET_ARCH so I dropped n32 from the list instead.

Yeah, so I succesfully built all of the ports that we use in freebsd-wifi-build with in-tree llvm (+ your patch that just got committed here) on o32; I test-booted MALTA a while ago just fine (with no substantial changes to llvm that would break it in the meantime) and I built a new image for one of my o32 dev units to test on real hardware, but haven't imaged one with it yet.

I guess I could try to build and boot a n32 world, but I'm not sure which kernel I should be using to test it.

Fair enough, I don't know either :)

I think we should commit this as is, and subsequently follow up on lists etc. to see if anyone is interested in n32. Either it gets tested (and fixed, if there's an issue) or if nobody cares about it we retire it before 13.

I already have n32 listed as Tier 3 I think for 13 I think which means it could stay in the tree, but would be disconnected from universe. There's not a whole lot of n32 stuff to really remove. Most of it is is #if n64 || n32

mips64hf is currently broken because libc_nossp is built as soft-float (I think this is the lib32 version only since the error happens when building rtld-elf32):

ld: error: /home/alr48/obj/build/freebsd-universe-build/exports/users/alr48/sources/freebsd-universe/mips.mips64hf/libexec/rtld-elf32/rtld_libc.a(lstat.nossppico): floating point ABI '-msoft-float' is incompatible with target floa
ting point ABI '-mdouble-float'

It also seems like we need to cherry pick more LLVM fixes first:

mips OCTEON1 kernel failed, check _.mips.OCTEON1 for details
mips XLP64 kernel failed, check _.mips.XLP64 for details
mips XLPN32 kernel failed, check _.mips.XLPN32 for details
mips ERL kernel failed, check _.mips.ERL for details
sys/mips/cavium/asm_octeon.S
Assertion failed: (OffsetOp.isExpr() && "expected expression operand kind"), function expandMemInst, file /exports/users/alr48/sources/freebsd-universe/contrib/llvm-project/llvm/lib/Target/Mips/AsmParser/MipsAsmParser.cpp, line 3702.

I believe there were a bunch of octeon fixes in LLVM recently, so I'll check if one of those fixes the assert.

I also tried building n32 but that fails when linking the syscall objects. My guess is that SYS.h is missing the PIC n32 case and always generates position dependent assembly.

ld: error: can't create dynamic relocation R_MIPS_HI16 against symbol: __cerror in readonly segment; recompile object files with -fPIC or pass '-Wl,-z,notext' to allow text relocations in the output
>>> defined in cerror.pico
>>> referenced by shm_rename.S:4
>>>               shm_rename.pico:(__sys_shm_rename)

I also tried building n32 but that fails when linking the syscall objects. My guess is that SYS.h is missing the PIC n32 case and always generates position dependent assembly.

ld: error: can't create dynamic relocation R_MIPS_HI16 against symbol: __cerror in readonly segment; recompile object files with -fPIC or pass '-Wl,-z,notext' to allow text relocations in the output
>>> defined in cerror.pico
>>> referenced by shm_rename.S:4
>>>               shm_rename.pico:(__sys_shm_rename)

This appears to be fixed in LLVM 10 which expands la correctly for PIC n32:

00000028 err:
      28: 8f 99 00 00                  	lw	$25, 0($gp)
			00000028:  R_MIPS_CALL16	__cerror

We could either try to cherry-pick that fix or wait for llvm10 if anyone cares about n32.

I am fine with waiting for llvm 10 to arrive for n32.
@dim pointed out the branch has been created already, it's not that far away

I am fine with waiting for llvm 10 to arrive for n32.
@dim pointed out the branch has been created already, it's not that far away

That's fine. There's a plan to get there. If things are delayed, or people get too annoyed, we can turn off n32 for a bit...

mips64hf is currently broken because libc_nossp is built as soft-float (I think this is the lib32 version only since the error happens when building rtld-elf32):

ld: error: /home/alr48/obj/build/freebsd-universe-build/exports/users/alr48/sources/freebsd-universe/mips.mips64hf/libexec/rtld-elf32/rtld_libc.a(lstat.nossppico): floating point ABI '-msoft-float' is incompatible with target floa
ting point ABI '-mdouble-float'

I almost fixed this when fixing the mips64el case. Probably what we should do is set the 32-bit MACHINE_ARCH to MACHINE_ARCH:s/64// or some such to preserve el and hf suffixes.

However, note that part of the real problem here is that rtld-elf32 is somewhat broken in that it lets CFLAGS from the non-compat MACHINE_ARCH "leak" through. This doesn't
happen for the lib32 stage which is why the lib32 stage was building as soft-float since the MACHINE was "mips" but does happen for rtld-elf32. That's probably a bug in the bsd.compat.mk thing that @brooks might want to look at.

In D23204#509367, @jhb wrote:

mips64hf is currently broken because libc_nossp is built as soft-float (I think this is the lib32 version only since the error happens when building rtld-elf32):

ld: error: /home/alr48/obj/build/freebsd-universe-build/exports/users/alr48/sources/freebsd-universe/mips.mips64hf/libexec/rtld-elf32/rtld_libc.a(lstat.nossppico): floating point ABI '-msoft-float' is incompatible with target floa
ting point ABI '-mdouble-float'

I almost fixed this when fixing the mips64el case. Probably what we should do is set the 32-bit MACHINE_ARCH to MACHINE_ARCH:s/64// or some such to preserve el and hf suffixes.

However, note that part of the real problem here is that rtld-elf32 is somewhat broken in that it lets CFLAGS from the non-compat MACHINE_ARCH "leak" through. This doesn't
happen for the lib32 stage which is why the lib32 stage was building as soft-float since the MACHINE was "mips" but does happen for rtld-elf32. That's probably a bug in the bsd.compat.mk thing that @brooks might want to look at.

@jhb Like this: D23229? I'm planning to commit that later today so it should unbreak that.

What's left as a blocker on this? It'd be nice to get at least mips32 and mips64 universe/CI builds OK again, since we still haven't worked out why the freebsd-gcc port is doing the wrong thing.

I can commit this now if we are okay with octeon kernels being broken until llvm 10 lands?

llvm 10 landing date seems to be tentatively scheduled for today-ish. \o/

Makefile
509

Presumably this whole block disappears when rebasing.

share/mk/src.opts.mk
301

With sparc64 out of the tree CLANG is always yes and only CLANG_BOOTSTRAP and CLANG_IS_CC remain here.

I'm currently on holiday but will update this once I'm back on Monday.

emaste added inline comments.
Makefile
509

I think it's probably better to remove this block altogether, because with no entries it's going to bitrot. That said I think this change is fine for now, we can leave it here for a bit.

share/mk/src.opts.mk
298–312

You can move these to the main _DEFAULT_YES_OPTIONS block above.

This revision is now accepted and ready to land.Mar 16 2020, 11:42 PM
jhb added inline comments.
Makefile
512

I think it's fine to leave this block for now. I will probably at some point add some kind of USE_GCC=yes option to let one do a 'make tinderbox' using gcc instead of clang and would reuse some of these bits for that.

arichardson retitled this revision from Use clang and lld as the default toolchain for MIPS. to Use Clang and LLD as the default toolchain for MIPS.
arichardson edited the summary of this revision. (Show Details)

Rebased on latest HEAD.

Thanks for reviewing. I will commit this tomorrow morning (UK time) so that I can react to potential CI failures.

This revision now requires review to proceed.Mar 22 2020, 7:33 PM
This revision is now accepted and ready to land.Mar 22 2020, 8:10 PM