Add WITH_LLVM_BINUTILS to install LLVM binutils instead of Elftoolchain

[arichardson]

Summary

When WITH_LLVM_BINUTILS is set, we will install the LLVM binutils as
ar/ranlib/nm/objcopy/etc. instead of the elftoolchain ones.
Having the LLVM binutils instead of the elftoolchain ones allows us to use
features such as LTO that depend on binutils that understand LLVM IR.
Another benefit will be an improved user-experience when compiling with
AddressSanitizer, since ASAN does not symbolize backtraces correctly if
addr2line is elftoolchain addr2line instead of llvm-symbolizer.
See https://lists.freebsd.org/archives/freebsd-toolchain/2021-July/000062.html
for more details.

This is currently off by default but will be turned on by default at some
point in the near future.

Depends on D31058 (to apply cleanly)

[jrtc27]

lib/Makefile

160–163

This seems a bit dated given MK_LLD and MK_LLDB exist

[arichardson]

lib/Makefile

160–163

Yeah I'm not sure how happy the build will be if you try -DWITH_LLD -DWITHOUT_CLANG.

[emaste]

Makefile.inc1

2350–2355

This part (clang-tblgen, lldb-tblgen) lgtm, and I think you can commit it first

usr.bin/Makefile

147

I'm not 100% sure about this one - I pulled strings out of WITH_/WITHOUT_TOOLCHAIN because it's often used outside of the context of a build toolchain, and ELF Tool Chain's strings will be much smaller.

[arichardson]

usr.bin/Makefile

147

Happy to drop strings from this change. Will update shortly.

[arichardson]

drop strings

[emaste]

LGTM, we might have a few things to iterate on with this in the tree

Makefile.inc1

720–738

Would we want to split out knobs for bootstrap and installed system? (If so, as a later step.)

usr.bin/Makefile

198–202

As an aside usr.bin/clang/ (now) seems like the wrong top-level directory name for the various LLVM components, but beyond the scope of this change.

[arichardson]

Makefile.inc1

720–738

I always build with an external toolchain, but I think adding MK_LLVM_BINUTILS_BOOTSTRAP sounds reasonable.

[emaste]

From hardenedbsd IRC, an issue that has come up in discussion before (and will iterate on via exp-runs):

14:47 < gehmehgeh> emaste: careful, though, when switching to llvm-ar: there is 
                   at least one "elftc" option it doesn't support. I forget 
                   which flag it was, but I've a section in my make.conf 
                   instructing the FreeBSD ports to use elftc's ar when 
                   building everything Haskell related-- exactly because their 
                   build script do something unsupported

[emaste]

From @mhorne via 4d0dc60f14019eab08f6d9dc656c9f9f1ebdde02, llvm-strip does not parse --
Now worked around in the base system, but we ought to support it in llvm-strip.

[arichardson]

In D31060#715946, @emaste wrote:

From @mhorne via 4d0dc60f14019eab08f6d9dc656c9f9f1ebdde02, llvm-strip does not parse --
Now worked around in the base system, but we ought to support it in llvm-strip.

Is there an upstream LLVM bugreport for this?

[emaste]

Is there an upstream LLVM bugreport for this?

I have not (yet) submitted one and did not find one after a quick look. I was planning to do a more holistic check of option parsing. Also, an exp-run with this change enabled should help find real-world issues.

[emaste]

It looks like llvm-strip (main) supports -- now:

llvm pkg

$ llvm-strip11 -o /dev/null -- /usr/bin/true
llvm-strip: error: unknown argument '--'

llvm-strip built from src today:

$ ~/src/llvm-project/build/bin/llvm-strip -o /dev/null -- /usr/bin/true
$ echo $?
0

[arichardson]

rebase