lib{c,sys}: move auxargs more firmly into libsys

[brooks]

Summary

Continue to filter the public interface (elf_aux_info()), but entierly
relocate the private interfaces (_elf_aux_info(),
init_elf_aux_vector(), and elf_aux_vector) to libsys.

This ensures that rtld updates the correct (only) copy of
elf_aux_vector. After 968a18975adc9c2a619bb52aa2f009de99fc9e24
updates were confused and getosreldate was failing, causing
the system to fall back to compat compat12 syscalls in some cases.

[brooks]

This fixes __getosreldate in closefrom() and generally seems to work, but it adds a libsys dependency on -fsanitize=bounds users and breaks the MULTILIB option of lang/gcc*. I've also attempted a refactoring where I moved the auxargs bits back to libc, but then libsys was unable to resolve _elf_aux_info() so nothing worked. It might be that libsys needed a DT_NEEDED for libc, but that seems circular... Not sure how best to proceed here.

[emaste]

lib/libsys/Makefile.sys

59 ↗

(On Diff #134357)

Is it worth a comment here explaining auxv.o? libc_stubs.pico is obvious from context but auxv perhaps not

[kib]

In D43910#1001600, @brooks wrote:

This fixes __getosreldate in closefrom() and generally seems to work, but it adds a libsys dependency on -fsanitize=bounds users and breaks the MULTILIB option of lang/gcc*. I've also attempted a refactoring where I moved the auxargs bits back to libc, but then libsys was unable to resolve _elf_aux_info() so nothing worked. It might be that libsys needed a DT_NEEDED for libc, but that seems circular... Not sure how best to proceed here.

This does not solve the issue with libsys actually referencing symbols from libc, which perhaps add to the whole set of problems.

May be we should stop providing compat shims for future-compatibility in libsys, leaving it to libc? Then libsys does not need osreldate.

For now it is probably a good enough stop-gap.

share/mk/src.libnames.mk

397 ↗

(On Diff #134357)

I think we should also explicitly link libthr to libc and libsys.

[kib]

And more, I am thinking that embedding interposing machinery into libsys was also a mistake. The library should only export __sys_XXX versions of symbols, and all interposing occuring purely between libc and libthr.

[brooks]

In D43910#1001801, @kib wrote:

And more, I am thinking that embedding interposing machinery into libsys was also a mistake. The library should only export __sys_XXX versions of symbols, and all interposing occuring purely between libc and libthr.

That's an annoyingly a complicated endeavor. I believe we'd have to move to dynamic initialization. At least in early experiments I found that symbols used by static initializers aren't subject to filters and we ended up with libc symbols in the interposing table if it wasn't moved which broke everything.

I'm increasingly convinced that libthr should be merged into libc so we can eliminate all this complexity.

[brooks]

• Fix build with gcc
• Add comments to Makefile.sys
• rebase

[brooks]

I've concluded that gnu ld's -m32 support is broken. It finds libsys right away if you add -lsys to the command line and searches in a whole bunch of wrong places it the dependency is only recorded in libc. I don't have a handle on ways to debug that, but I think this is closer to the right way forward and I'd like to unbreak kernels without COMPAT12

share/mk/src.libnames.mk

397 ↗

(On Diff #134357)

As in adding LIBADD entries?

[kib]

In D43910#1001845, @brooks wrote:

In D43910#1001801, @kib wrote:

And more, I am thinking that embedding interposing machinery into libsys was also a mistake. The library should only export __sys_XXX versions of symbols, and all interposing occuring purely between libc and libthr.

That's an annoyingly a complicated endeavor. I believe we'd have to move to dynamic initialization. At least in early experiments I found that symbols used by static initializers aren't subject to filters and we ended up with libc symbols in the interposing table if it wasn't moved which broke everything.

Sorry, could you provide some example. I have trouble following this explanation.

I'm increasingly convinced that libthr should be merged into libc so we can eliminate all this complexity.

Yes, I agree, in principle. One of the reasons why I did not do that yet is because simply loading libthr causes some additional overhead, without starting the actual threading.

In D43910#1002002, @brooks wrote:

I've concluded that gnu ld's -m32 support is broken. It finds libsys right away if you add -lsys to the command line and searches in a whole bunch of wrong places it the dependency is only recorded in libc. I don't have a handle on ways to debug that, but I think this is closer to the right way forward and I'd like to unbreak kernels without COMPAT12

Does LIBADD help ultimately?

share/mk/src.libnames.mk

397 ↗

(On Diff #134357)

Yes, and ensuring that DT_NEEDED are in the right order -> libc then libsys

[brooks]

• explicitly link libthr to libc and libsys
• add runpath to lib32 libc and libthr to fix gcc -m32

[jrtc27]

lib/libc/Makefile

67

This needs an explanatory comment, though honestly I'd rather we just fix whatever's horrendously broken (untested?) in binutils...

[brooks]

I now understand more of what's going on in gnu ld. This code handles the DT_NEEDED libraries https://sourceware.org/git/?p=binutils-gdb.git;a=blob;f=ld/ldelf.c;h=04045acbf3dc56947edb15effff5818dd5b69fd9;hb=HEAD#l1091 and among other things We do not search using the -L arguments.. The problem is, ld doesn't know about -m32 mode at all so it just looks in the usual paths for i386. It appears that gcc attempts to communicate the -m32 paths by passing a LIBRARY_PATH environmental variable that includes /usr/lib/../lib32, but I can find no evidence that ld examines it. Tracing does suggest it looks in a i386 triple directory (something like /usr/lib/i386-unknown-freebsd15.0, but that's from memory) so creating a symlink to /usr/lib32 might be an alternative workaround to adding DT_RUNPATH entries as I've done in the latest update, but that seems annoying?

[brooks]

Since it's a bit annoying to extract on system with RUNPATH's in libc and libthr, the search of libsys.so.7 looks like:

82890 ld       NAMI  "/usr/lib/libsys.so.7"
82890 ld       NAMI  "/usr/lib/compat/libsys.so.7"
82890 ld       NAMI  "/usr/local/lib/libsys.so.7"
82890 ld       NAMI  "/usr/local/lib/compat/pkg/libsys.so.7"
82890 ld       NAMI  "/usr/local/lib/perl5/5.36/mach/CORE/libsys.so.7"
82890 ld       NAMI  "/usr/local/i386-portbld-freebsd15.0/lib/libsys.so.7"
82890 ld       NAMI  "/lib/libsys.so.7"
82890 ld       NAMI  "/usr/lib/libsys.so.7"
82890 ld       NAMI  "/usr/local/lib/libsys.so.7"
82890 ld       NAMI  "/lib/libsys.so.7"
82890 ld       NAMI  "/usr/lib/libsys.so.7"
82890 ld       NAMI  "/usr/lib/compat/libsys.so.7"
82890 ld       NAMI  "/usr/local/lib/libsys.so.7"
82890 ld       NAMI  "/usr/local/lib/compat/pkg/libsys.so.7"
82890 ld       NAMI  "/usr/local/lib/perl5/5.36/mach/CORE/libsys.so.7"
82890 ld       NAMI  "/usr/local/i386-portbld-freebsd15.0/lib/libsys.so.7"
82890 ld       NAMI  "/lib/libsys.so.7"
82890 ld       NAMI  "/usr/lib/libsys.so.7"
82890 ld       NAMI  "/usr/local/lib/libsys.so.7"

It does open and read part of /lib/libsys.so.7, but continues to search, presumably because the ABI doesn't match.

Also truss output of the ld invocation inside the gcc12 multilib build for gomp:

94440: execve("/usr/local/bin/ld",[ "/usr/local/bin/ld", "-plugin", "/wrkdirs/usr/ports/lang/gcc12/work/.build/./gcc/liblto_plugin.so", "-plugin-opt=/wrkdirs/usr/ports/lang/gcc12/work/.build/./gcc/lto-wrapper", "-plugin-opt=-fresolution=/tmp//ccJ84uJW.res", "-plugin-opt=-pass-through=-lgcc", "-plugin-opt=-pass-through=-lgcc_s", "-plugin-opt=-pass-through=-lpthread", "-plugin-opt=-pass-through=-lc", "-plugin-opt=-pass-through=-lgcc", "-plugin-opt=-pass-through=-lgcc_s", "--eh-frame-hdr", "-m", "elf_i386_fbsd", "-dynamic-linker", "/libexec/ld-elf.so.1", "/usr/lib/../lib32/crt1.o", "/usr/lib/../lib32/crti.o", "/wrkdirs/usr/ports/lang/gcc12/work/.build/./gcc/32/crtbegin.o", "-L/wrkdirs/usr/ports/lang/gcc12/work/.build/./gcc/32", "-L/usr/lib/../lib32", "-L/wrkdirs/usr/ports/lang/gcc12/work/.build/./gcc", "-L/usr/local/x86_64-portbld-freebsd15.0/bin", "-L/usr/local/x86_64-portbld-freebsd15.0/lib", "/tmp//ccwUgImk.o", "-lssp_nonshared", "-lgcc", "--push-state", "--as-needed", "-lgcc_s", "--pop-state", "-lpthread", "-lc", "-lgcc", "--push-state", "--as-needed", "-lgcc_s", "--pop-state", "/wrkdirs/usr/ports/lang/gcc12/work/.build/./gcc/32/crtend.o", "/usr/lib/../lib32/crtn.o" ],[ "LIBRARY_PATH=/wrkdirs/usr/ports/lang/gcc12/work/.build/./gcc/32/:/usr/lib/../lib32/:/wrkdirs/usr/ports/lang/gcc12/work/.build/./gcc/:/usr/local/x86_64-portbld-freebsd15.0/bin/:/usr/local/x86_64-portbld-freebsd15.0/lib/:/lib/:/usr/lib/", "COMPILER_PATH=/wrkdirs/usr/ports/lang/gcc12/work/.build/./gcc/:/usr/local/x86_64-portbld-freebsd15.0/bin/:/usr/local/x86_64-portbld-freebsd15.0/lib/", "COLLECT_GCC_OPTIONS='-B' '/wrkdirs/usr/ports/lang/gcc12/work/.build/./gcc/' '-B' '/usr/local/x86_64-portbld-freebsd15.0/bin/' '-B' '/usr/local/x86_64-portbld-freebsd15.0/lib/' '-isystem' '/usr/local/x86_64-portbld-freebsd15.0/include' '-isystem' '/usr/local/x86_64-portbld-freebsd15.0/sys-include' '-fno-checking' '-m32' '-g' '-O2' '-pipe' '-D' 'LIBICONV_PLUG' '-fstack-protector-strong' '-fno-strict-aliasing' '-pthread' '-mtune=generic' '-march=x86-64' '-dumpdir' 'a.'", "COLLECT_LTO_WRAPPER=/wrkdirs/usr/ports/lang/gcc12/work/.build/./gcc/lto-wrapper", "COLLECT_GCC=/wrkdirs/usr/ports/lang/gcc12/work/.build/./gcc/xgcc", "GCC_EXEC_PREFIX=/wrkdirs/usr/ports/lang/gcc12/work/.build/gcc/../lib/gcc12/gcc/", "LOGNAME=root", "LANG=C.UTF-8", "PAGER=less", "MAIL=/var/mail/root", "PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:/root/bin", "ENV=/root/.shrc", "PWD=/root", "TERM=tmux-256color", "USER=root", "HOME=/root", "SHELL=/bin/sh", "UNAME_r=15.0-CURRENT", "MM_CHARSET=UTF-8", "OSVERSION=1500014", "UNAME_v=FreeBSD 15.0-CURRENT 1500014", "BLOCKSIZE=K" ])

[kib]

In D43910#1002379, @brooks wrote:

Tracing does suggest it looks in a i386 triple directory (something like /usr/lib/i386-unknown-freebsd15.0, but that's from memory) so creating a symlink to /usr/lib32 might be an alternative workaround to adding DT_RUNPATH entries as I've done in the latest update, but that seems annoying?

Since it is versioned, I do not think that making such symlink is great. runpath/rpath are better in this regard.

[brooks]

In D43910#1002416, @kib wrote:

In D43910#1002379, @brooks wrote:

Tracing does suggest it looks in a i386 triple directory (something like /usr/lib/i386-unknown-freebsd15.0, but that's from memory) so creating a symlink to /usr/lib32 might be an alternative workaround to adding DT_RUNPATH entries as I've done in the latest update, but that seems annoying?

Since it is versioned, I do not think that making such symlink is great. runpath/rpath are better in this regard.

Yeah, the path that is search is also /usr/local/i386-portbld-freebsd15.0/lib/libsys.so.7 which isn't something we could make into a symlink in base so that won't work.

I'm going to file a bug against binutils once someone creates an account for me and reference that in the comments I'll add to libc and libthr.

[netchild]

Can the issues this patch is supposed to fix lead to compile issues? I see failing rust and cargo-c builds in the "libc" part. And I see issues with nslookup to a bind 9.18 which serves an internal domain.

If yes, would it be a good idea to back out the libsys changes and ask people to give the patchset a real world test and some ports exp run?

[vishwin]

In D43910#1002661, @netchild wrote:

Can the issues this patch is supposed to fix lead to compile issues? I see failing rust and cargo-c builds in the "libc" part. And I see issues with nslookup to a bind 9.18 which serves an internal domain.

If yes, would it be a good idea to back out the libsys changes and ask people to give the patchset a real world test and some ports exp run?

This revision doesn't result in any difference on top of 968a18975adc9c2a619bb52aa2f009de99fc9e24, certain programs are still crashing, timing out or refusing to run in the same manner.

[brooks]

In D43910#1002664, @vishwin wrote:

In D43910#1002661, @netchild wrote:

Can the issues this patch is supposed to fix lead to compile issues? I see failing rust and cargo-c builds in the "libc" part. And I see issues with nslookup to a bind 9.18 which serves an internal domain.

If yes, would it be a good idea to back out the libsys changes and ask people to give the patchset a real world test and some ports exp run?

This revision doesn't result in any difference on top of 968a18975adc9c2a619bb52aa2f009de99fc9e24, certain programs are still crashing, timing out or refusing to run in the same manner.

I'm not assigned or CC'd on any bug reports that I can see so I'm not sure how I'm supposed to be triaging these issues. Rust has definitely has built for me (it's usually the longest pole in retesting textproc/R-cran-xml2 due to its utterly terrible build parallelism). I'm building cargo-c now and can test other things if people tell me about them. I'm not going to do a revert based on "certain programs are still crashing".

[vishwin]

In D43910#1003335, @brooks wrote:

I'm not assigned or CC'd on any bug reports that I can see so I'm not sure how I'm supposed to be triaging these issues. Rust has definitely has built for me (it's usually the longest pole in retesting textproc/R-cran-xml2 due to its utterly terrible build parallelism). I'm building cargo-c now and can test other things if people tell me about them. I'm not going to do a revert based on "certain programs are still crashing".

I don't think this revision has anything to do with 968a18975adc9c2a619bb52aa2f009de99fc9e24 anyway, after testing this prior to committing.