lang/rust: Update to 1.52.0

[tobik]

Summary

Scheduled to be released on 2021-04-06.

https://github.com/rust-lang/rust/blob/stable/RELEASES.md#version-1520-2021-05-06
https://internals.rust-lang.org/t/1-52-0-prerelease-testing/14628

Other changes:

• Added an update script for lang/rust similar to lang/rust-nightly
• Changed how USES=cargo generates DISTFILES and MASTER_SITES. See the associated commit message for more details. It's unrelated to the update but I've batched it together because of the exp-run
• Use binutils strip on ppc64 ELFv1; base strip complains about unsupported e_type

I've also pushed this here https://github.com/t6/freebsd-ports/tree/tobik/rust-1.52.0

Test Plan

ref11-i386 ok
ref12-amd64 ok
ref12-ppc64 ok-ish (strip complains about unsupported e_type?)
ref13-ppc64 ok
ref13-aarch64 ok
ref13-amd64 ok

I'll leave rust-1.52.0 installed in /scratch/tmp/tobik/local/usr/local on the reference machines when the build was ok in case you need to check something.

armv6 ok
armv7 ok
powerpc64le ok

Exp-run: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=255608

[tobik]

• Mk/Uses/cargo.mk: Refactor MASTER_SITES and DISTFILES construction
• Mk/Uses/cargo.mk: Simplify crate matches
• Use newer strip on ELFv1

[bdragon]

Note: I have the bootstrap side working for 32 bit powerpc, been holding off on submitting it though because I haven't successfully gotten a full lang/rust build out of it yet (problems with linking.) I did upstream the obligatory openssl-src-rs crate changes needed to bootstrap though at https://github.com/alexcrichton/openssl-src-rs/commit/9cc3301e6168237d2b466cb92d5535e0a6252c51 .

I stuck it at https://people.freebsd.org/~bdragon/rust_bootstrap_ppc.patch if you want to do a 1.51 bootstrap build. Otherwise, when I have it fully working, I'll sneak in another X -> X bootstrap like we did for ppc64le initially.

[tobik]

In D30099#675645, @bdragon wrote:

Note: I have the bootstrap side working for 32 bit powerpc, been holding off on submitting it though because I haven't successfully gotten a full lang/rust build out of it yet (problems with linking.)

That's nice. What problems specifically?

I stuck it at https://people.freebsd.org/~bdragon/rust_bootstrap_ppc.patch if you want to do a 1.51 bootstrap build.

I've applied it. Your patch has the prebuilt bootstrap in distinfo but I could not find it in LOCAL/bdragon. Can you upload them? I can copy them to LOCAL/tobik then.

Even if it isn't working we can at least hook it up. I've marked powerpc as broken in lang/rust for now.

I did upstream the obligatory openssl-src-rs crate changes needed to bootstrap though at https://github.com/alexcrichton/openssl-src-rs/commit/9cc3301e6168237d2b466cb92d5535e0a6252c51 .

Can you upstream the LLVM, libc, and powerpc-unknown-freebsd target bits too?

[tobik]

• Add 32-bit powerpc patches from bdragon

[bdragon]

In D30099#675678, @tobik wrote:

In D30099#675645, @bdragon wrote:

Note: I have the bootstrap side working for 32 bit powerpc, been holding off on submitting it though because I haven't successfully gotten a full lang/rust build out of it yet (problems with linking.)

That's nice. What problems specifically?

Out of range PLT relocs.

http://drop.rtk0.net/20210504.txt

jhibbits just clued me in that I wasn't enabling the longcall feature, so I'm going to retry with that.

I stuck it at https://people.freebsd.org/~bdragon/rust_bootstrap_ppc.patch if you want to do a 1.51 bootstrap build.

I've applied it. Your patch has the prebuilt bootstrap in distinfo but I could not find it in LOCAL/bdragon. Can you upload them? I can copy them to LOCAL/tobik then.

Yeah, I was going to upload them as soon as I got a lang/rust build to finish without bailing.

I am going to do a rebuild today though, jhibbits gave me some insight as to what's going wrong with the build currently, so I'm gonna do a rebootstrap with some slight targeting adjustments. (i.e. force -mlongcall.)

Even if it isn't working we can at least hook it up. I've marked powerpc as broken in lang/rust for now.

I did upstream the obligatory openssl-src-rs crate changes needed to bootstrap though at https://github.com/alexcrichton/openssl-src-rs/commit/9cc3301e6168237d2b466cb92d5535e0a6252c51 .

Can you upstream the LLVM, libc, and powerpc-unknown-freebsd target bits too?

Yeah, once I have things nailed down.

[bdragon]

In D30099#675684, @bdragon wrote:

In D30099#675678, @tobik wrote:

In D30099#675645, @bdragon wrote:

Note: I have the bootstrap side working for 32 bit powerpc, been holding off on submitting it though because I haven't successfully gotten a full lang/rust build out of it yet (problems with linking.)

That's nice. What problems specifically?

Out of range PLT relocs.

http://drop.rtk0.net/20210504.txt

It's looking like this problem was due to the rust build invoking the system compiler without the full FreeBSD version passed as the target. This meant that the system clang was defaulting to bss-plt, which wasn't able to set up branches due to the size of the binary.

I had encountered this same problem with rust-bootstrap and had fixed it there. I'm rebuilding the bootstrap with more minimal targeting changes to see if it works there without having to do stuff like patch the code model. If it works, I will tidy it up and submit that as the final 1.51.0 bootstrap.

Getting really close to having this fully up and running.

Most of the problems I've hit would have been avoided if we had per FreeBSD version rust target names. :-/

[bdragon]

Finalized boostraps for powerpc uploaded to my public_distfiles on freefall.

I'm still working on some fixes for the lang/rust build (plus I need to move my patches over to 1.52.0 anyway) but I have gotten an eventually successful native build of patched 1.51.0 out of this bootstrap, so I'm calling it good for now.

SHA256 (rust/2021-02-11/rustc-1.51.0-powerpc-unknown-freebsd.tar.xz) = a95f8bae7685cf512d096f75846181bb90e6f35dc0bc29d68f4574520e7488f6
SIZE (rust/2021-02-11/rustc-1.51.0-powerpc-unknown-freebsd.tar.xz) = 47820404
SHA256 (rust/2021-02-11/rust-std-1.51.0-powerpc-unknown-freebsd.tar.xz) = bc7fffac703c5bf3c90325110f17c507e39f2ae795ef2894ba58a9ad6bb87004
SIZE (rust/2021-02-11/rust-std-1.51.0-powerpc-unknown-freebsd.tar.xz) = 17509168
SHA256 (rust/2021-02-11/cargo-1.51.0-powerpc-unknown-freebsd.tar.xz) = 96b5935082775f6ad882f6b8b27e57abc2df32efbe1e5746f998226ef5f2b20e
SIZE (rust/2021-02-11/cargo-1.51.0-powerpc-unknown-freebsd.tar.xz) = 4947120

[tobik]

• Update distinfo with new powerpc bootstraps

[tobik]

• Add LibreSSL 3.3.x patch from PR 255652

[pkubaj]

On powerpc64, things look indeed ok, apart from one thing (a regression back from 1.51.0).
Firefox-esr and Thunderbird both hang during build (happens also on powerpc64le)
Attaching full build log:

powerpc64le-default.log4 MBDownload

I wouldn't call it a showstopper, since the main Firefox port builds and works, but it would be nice to get it working.

[pkubaj]

Nevermind, things seem to work on 1.52.0, powerpc64le also is ok.

[bdragon]

Final powerpc changes:

https://people.freebsd.org/~bdragon/rust-1.52-ppc-final-fixes.patch

I got a successful build with this for powerpc.

I've been thinking on this, and I think we should push for upstream to default to FreeBSD 13 ABIs on powerpc* platforms (ELFv2 / ppc32 secure-plt) so we can isolate the version specific patching in ports to only be needed for ELFv1. At least until they grow proper support for handling osversion.

[tobik]

• Apply final powerpc fixes from bdragon

[tobik]

• Use the right base for arc diff :-/

[diizzy]

Hi,

I apologize if this is just noise but adding @${ECHO_CMD} 'codegen-units=1' >> ${WRKSRC}/config.toml seems help a lot on my machines when it comes to memory usage during compilation, is this something we can consider adding by default?

Here's a patch to the current version in tree (1.51.0) for more context:
https://projects.pyret.net/files/public/freebsd/rust-memory-codegen-units.patch

Best regards,
Daniel

[diizzy]

A few things I've noted that other distros does that we might want to adopt? This is just a few differences I noticed while comparing our port to other distros, it's by no mean any attempt to hijack this patch.

Alpine,Arch,Gentoo,OpenSuse,Void uses LLVM from their package tree instead of bundled, perhaps that's worth looking into?

https://git.alpinelinux.org/aports/tree/community/rust/APKBUILD#n124
https://git.alpinelinux.org/aports/tree/community/rust/APKBUILD#n147
https://git.alpinelinux.org/aports/tree/community/rust/APKBUILD#n151
https://git.archlinux.org/svntogit/packages.git/tree/trunk/PKGBUILD?h=packages/rust#n56
https://build.opensuse.org/package/view_file/openSUSE:Factory/rust/rust.spec?expand=1 (L463)

Possibly related? https://github.com/void-linux/void-packages/blob/master/srcpkgs/rust/patches/00016-do-not-copy-libunwind.patch

Codegen-units defaults to 1 (Arch provides a bit more context)

https://git.alpinelinux.org/aports/tree/community/rust/APKBUILD#n157
https://git.alpinelinux.org/aports/tree/community/rust/APKBUILD#n158
https://git.archlinux.org/svntogit/packages.git/tree/trunk/PKGBUILD?h=packages/rust#n72
https://gitweb.gentoo.org/repo/gentoo.git/tree/dev-lang/rust/rust-1.52.0.ebuild#n317
https://github.com/void-linux/void-packages/blob/master/srcpkgs/rust/template#L243
https://github.com/void-linux/void-packages/blob/master/srcpkgs/rust/template#L244
https://build.opensuse.org/package/view_file/openSUSE:Factory/rust/rust.spec?expand=1

Disable parallel compiling, I guess this is to ease memory pressure
https://git.alpinelinux.org/aports/tree/community/rust/APKBUILD#n159
https://github.com/void-linux/void-packages/blob/master/srcpkgs/rust/template#L251

https://git.alpinelinux.org/aports/tree/community/rust/link-musl-dynamically.patch (adopt to our library?)
https://git.alpinelinux.org/aports/tree/community/rust/musl-fix-linux_musl_base.patch (same as above)
https://github.com/void-linux/void-packages/blob/master/srcpkgs/rust/patches/0009-Link-stage2-tools-dynamically-to-libstd.patch
https://gitweb.gentoo.org/repo/gentoo.git/tree/dev-lang/rust/rust-1.51.0-r2.ebuild#n323 (more static linking?)

Unbundle libssh2?
https://build.opensuse.org/package/view_file/openSUSE:Factory/rust/rust.spec?expand=1 (L507)

Thanks for keeping this up to date!

[tobik]

• Mk/Uses/cargo.mk: Fix with empty CARGO_CRATES
• lang/rust: Update to 1.52.1 (diff: https://github.com/rust-lang/rust/compare/1.52.0...1.52.1 i.e. no need to redo QA but was ok on ref13-amd64 anyway)

[pkubaj]

This comment was removed by pkubaj.

[tobik]

@bdragon lang/rust-bootstrap@powerpc failed to build. I tried to fix it in 0ade1fd7a6cd7eef17d19cf118bf1ee15ec1c653 but I'm wondering how you managed to build it? Are we missing a patch or something else?

http://package19.nyi.freebsd.org/data/122amd64-default-build-as-user/ab31b591e5a6/logs/powerpc-rust-bootstrap-1.52.1_1.log