Make p_vaddr % p_align == p_offset % p_align for TLS segments.

[kib]

Summary

See https://sourceware.org/bugzilla/show_bug.cgi?id=24606 for the test case.
See https://reviews.llvm.org/D64930 for the background and more discussion.

Also this fixes another bug in malloc_aligned() where total size of the allocated memory might be not enough to fir the aligned requested block after the initial pointer is incremented by the pointer size.

Test Plan

Would be nice to boot on non-x86 to see how it breaks things.

[fbsd-phab_maskray.me]

Local-Exec TP offsets are link-time constants, and thus contracts between ld and ld.so. It may be worth checking if rtld-elf computed Local-Exec TP offsets match lld.

https://reviews.llvm.org/D64906 series (EM_PPC, EM_PPC64, EM_AARCH64 and EM_386) were committed yesterday. See the changed ELF/InputSection.cpp static int64_t getTlsTpOffset(const Symbol &s) {. The lld formulae should match musl>1.1.22 and glibc. ARM/AArch64 are a bit tricky because there is an optional alignment padding after the 2 reserved words.

Quick instructions to build lld:

git clone [e-mail]:llvm/llvm-project.git
cd llvm-project
cmake -Hllvm -BRelease -GNinja -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_PROJECTS='lld'

# Delete `else if (Out::tlsPhdr && Out::tlsPhdr->firstSec == p->firstSec)` from lld/ELF/Writer.cpp around line 2244
# If sh_addralign(.tdata) < sh_addralign(.tbss), sometimes p_vaddr(PT_TLS)%p_align(PT_TLS)!=0

ninja -C Release lld
# lld is at Release/bin/ld.lld

a.c:

#include <stdio.h>
__thread int a __attribute__((aligned(4))) = 0xb612; // .tdata alignment, try a few numbers, e.g. 1, 2, 4, 8, 16, 32
__asm(".section .tbss,\"awT\"; .align 64"); // try 64, 128, 256, etc
int main() { printf("%p %x\n", &a, a); }

clang a.c -fuse-ld=path/to/ld.lld -o a
./a
# Experiment with a few different alignments

[kib]

In D21163#464395, @i_maskray.me wrote:

...
Experiment with a few different alignments

Is that what you mean ?

diff --git a/lld/ELF/Writer.cpp b/lld/ELF/Writer.cpp
index 9db4259be49..b8b154d1f32 100644
--- a/lld/ELF/Writer.cpp
+++ b/lld/ELF/Writer.cpp
@@ -2236,12 +2236,14 @@ template <class ELFT> void Writer<ELFT>::fixSectionAlignments() {
       // p_align for dynamic TLS blocks (PR/24606), FreeBSD rtld has the same
       // bug, musl (TLS Variant 1 architectures) before 1.1.23 handled TLS
       // blocks correctly. We need to keep the workaround for a while.
+#if 0
       else if (Out::tlsPhdr && Out::tlsPhdr->firstSec == p->firstSec)
         cmd->addrExpr = [] {
           return alignTo(script->getDot(), config->maxPageSize) +
                  alignTo(script->getDot() % config->maxPageSize,
                          Out::tlsPhdr->p_align);
         };
+#endif
       else
         cmd->addrExpr = [] {
           return alignTo(script->getDot(), config->maxPageSize) +

?

What should I infer from the './a' output ? I suspect there is some alignment that just happens, which makes the test not effective:

Program Headers:
  Type           Offset   VirtAddr           PhysAddr           FileSiz  MemSiz   Flg Align
...
  TLS            0x002000 0x0000000000202000 0x0000000000202000 0x000004 0x000040 R   0x40

orion% ./a                                                         ~/build/llvm
0x80022f900 b612

[fbsd-phab_maskray.me]

In D21163#465743, @kib wrote:

In D21163#464395, @i_maskray.me wrote:

...
Experiment with a few different alignments

Is that what you mean ?

diff --git a/lld/ELF/Writer.cpp b/lld/ELF/Writer.cpp
index 9db4259be49..b8b154d1f32 100644
--- a/lld/ELF/Writer.cpp
+++ b/lld/ELF/Writer.cpp
@@ -2236,12 +2236,14 @@ template <class ELFT> void Writer<ELFT>::fixSectionAlignments() {
       // p_align for dynamic TLS blocks (PR/24606), FreeBSD rtld has the same
       // bug, musl (TLS Variant 1 architectures) before 1.1.23 handled TLS
       // blocks correctly. We need to keep the workaround for a while.
+#if 0
       else if (Out::tlsPhdr && Out::tlsPhdr->firstSec == p->firstSec)
         cmd->addrExpr = [] {
           return alignTo(script->getDot(), config->maxPageSize) +
                  alignTo(script->getDot() % config->maxPageSize,
                          Out::tlsPhdr->p_align);
         };
+#endif
       else
         cmd->addrExpr = [] {
           return alignTo(script->getDot(), config->maxPageSize) +

?

Yes, you can get misaligned a PT_TLS segment with p_vaddr%p_align!=0 with a.c above. Or:

# lld/test/ELF/aarch64-tls-vaddr-align.s
add x0, x0, :tprel_lo12_nc:a

.section .tdata,"awT"
.byte 0

.section .tbss,"awT"
.p2align 8
a:
.quad 0

# lld/test/ELF/ppc64-tls-vaddr-align.s
ld 3, a@tprel(13)

.section .tdata,"awT"
.byte 0

.section .tbss,"awT"
.p2align 8
a:
.quad 0

Your patch looks good.

[bdragon]

Testing powerpc64.

libexec/rtld-elf/rtld.c

4822 ↗

(On Diff #60456)

missing param on Variant I.

[kib]

Compile fix for var I.

[bdragon]

Hmm, calculations on powerpc64 seem to be offset incorrectly in both cases with the patched lld. Gonna meditate on a disassembly for a bit.

[bdragon]

libexec/rtld-elf/rtld.c

4849 ↗

(On Diff #61768)

I *think* the problem I'm having is that this bit needs to compensate for the difference between the file offset and the first legal address according to the padding, so that it copies the preset variables at the correct offset, instead of copying them to the beginning of the TLS memory.

[bdragon]

libexec/rtld-elf/rtld.c

4849 ↗

(On Diff #61768)

Example, thinking out loud:

Broken:

header:
     TLS off    0x0000000000000cd0 vaddr 0x0000000010020cd0 paddr 0x0000000010020cd0 align 2**6
         filesz 0x0000000000000004 memsz 0x0000000000000030 flags r--

binary:
00000cc0  4e 80 04 20 00 00 00 00  00 02 02 50 4b ff ff c4  |N.. .......PK...|
00000cd0 [00 00 b6 12]00 00 00 00  ff ff ff ff ff ff ff ff  |................|
00000ce0  00 00 00 00 00 00 00 00  ff ff ff ff ff ff ff ff  |................|
00000cf0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|

Working:

header:
     TLS off    0x0000000000000c40 vaddr 0x0000000010010c40 paddr 0x0000000010010c40 align 2**6
         filesz 0x0000000000000004 memsz 0x0000000000000004 flags r--

binary:
00000c10  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
*
00000c40 [00 00 b6 12]00 00 00 00  ff ff ff ff ff ff ff ff  |................|
00000c50  00 00 00 00 00 00 00 00  ff ff ff ff ff ff ff ff  |................|
00000c60  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000c70  00 00 00 00 00 00 00 01  00 00 00 00 00 00 00 01  |................|

In the broken case, the binary has 0x10 offset built in to reference the thread variable, because it expected the init to start at the difference between off (0xcd0) and the base-according-to-alignment (0xcc0). But since it was copying to the start of the area instead of an offset into it, the variable is copied 0x10 too low.

So I guess another memset is needed to zero the bytes between the base and the offset here, and the memcpy should write starting after that.

[bdragon]

libexec/rtld-elf/rtld.c

4849 ↗

(On Diff #61768)

Something like this seems to work for my test case. (remote editing, sorry in advance about the patch copypasted from a terminal:)

Index: local-src/libexec/rtld-elf/rtld.c
===================================================================
--- local-src.orig/libexec/rtld-elf/rtld.c
Index: local-src/libexec/rtld-elf/rtld.c
===================================================================
--- local-src.orig/libexec/rtld-elf/rtld.c
+++ local-src/libexec/rtld-elf/rtld.c
@@ -1447,6 +1447,7 @@ digest_phdr(const Elf_Phdr *phdr, int ph
            obj->tlsalign = ph->p_align;
            obj->tlsinitsize = ph->p_filesz;
            obj->tlsinit = (void*)(ph->p_vaddr + obj->relocbase);
+           obj->tlspoffset = ph->p_offset;
            break;
 
        case PT_GNU_STACK:
@@ -4802,7 +4803,7 @@ allocate_tls(Obj_Entry *objs, void *oldt
     Elf_Addr addr;
     Elf_Addr i;
     size_t extra_size, maxalign, post_size, pre_size, tls_block_size;
-    size_t tls_init_align;
+    size_t tls_init_align, tls_init_offset;
 
     if (oldtcb != NULL && tcbsize == TLS_TCB_SIZE)
        return (oldtcb);
@@ -4843,13 +4844,17 @@ allocate_tls(Obj_Entry *objs, void *oldt
 
        for (obj = globallist_curr(objs); obj != NULL;
          obj = globallist_next(obj)) {
+           tls_init_offset = obj->tlspoffset & (obj->tlsalign - 1); 
+           if (tls_init_offset > 0)
+               memset((void*) addr, 0, tls_init_offset);
            if (obj->tlsoffset > 0) {
                addr = (Elf_Addr)tcb + obj->tlsoffset;
                if (obj->tlsinitsize > 0)
-                   memcpy((void*) addr, obj->tlsinit, obj->tlsinitsize);
+                   memcpy((void*)(addr + tls_init_offset), obj->tlsinit, obj->tlsinitsize);
+
                if (obj->tlssize > obj->tlsinitsize)
-                   memset((void*)(addr + obj->tlsinitsize), 0,
-                          obj->tlssize - obj->tlsinitsize);
+                   memset((void*)(addr + tls_init_offset + obj->tlsinitsize), 0,
+                          obj->tlssize - obj->tlsinitsize - tls_init_offset);
                dtv[obj->tlsindex + 1] = addr;
            }
        }

[bdragon]

libexec/rtld-elf/rtld.c

1449 ↗

(On Diff #61768)

As per the inline patch I just dumped into my last comment: Need to set obj->tlspoffset = ph->p_offset; here as well to handle an unaligned tls offset in the main program.

[kib]

Add change from bdragon.

[bz]

I've a patch with this + the LLVM changes running on arm64 for a bit:

! This is a backport of three llvm/lld changes:
! - https://reviews.llvm.org/rL367537
! - https://reviews.llvm.org/rL369343
! - https://reviews.llvm.org/rL369344
! without the test cases.
!
! In addition the rtld-elf changes are from:
! https://reviews.freebsd.org/D21163

in case that helps. I can also upload my local patch if needed for the LLVM changes to test more easily.

I wonder if there can be made progress on this committing it and importing the bits into FreeBSD?

[bdragon]

Putting myself on the review list becasue this has been part of my local patchset for a long time.

I have found a problem in this recently. There's something about building rust on powerpc64 that trips this up.

The stage1 compiler doesn't work properly because when a new thread is created, it causes the allocation of tls memory to damage list_fini, which then causes a crash while running destructors.

tlspoffset = 2228912 in libc seems really odd to me, I wouldn't have thought it would be more than 64k at most.

(gdb) cont
Continuing.

Watchpoint 2: list_fini.stqh_first->link.stqe_next->link.stqe_next

Old value = (struct Struct_Objlist_Entry *) 0x801092908
New value = (struct Struct_Objlist_Entry *) 0x500000000000000
allocate_tls (objs=0x801095c08, oldtcb=<optimized out>, tcbsize=<optimized out>, tcbalign=<optimized out>) at /usr/src/libexec/rtld-elf/rtld.c:4915
4915		   obj = globallist_next(obj)) {
(gdb) bt
#0  allocate_tls (objs=0x801095c08, oldtcb=<optimized out>, tcbsize=<optimized out>, tcbalign=<optimized out>) at /usr/src/libexec/rtld-elf/rtld.c:4915
#1  0x000000080105e140 in _rtld_allocate_tls (oldtls=0x0, tcbsize=16, tcbalign=16) at /usr/src/libexec/rtld-elf/rtld.c:5164
#2  0x0000000808156208 in _tcb_ctor (thread=0x808c92200, initial=<optimized out>) at /usr/src/lib/libthr/thread/thr_ctrdtr.c:45
#3  0x0000000808154f68 in _thr_alloc (curthread=0x801073000) at /usr/src/lib/libthr/thread/thr_list.c:173
#4  0x0000000808145748 in _pthread_create (thread=0x3fffffffffff9d28, attr=0x3fffffffffff9d30, start_routine=0x807bcdfe8 <std::sys::unix::thread::Thread::new::thread_start>, 
    arg=0x80107a330) at /usr/src/lib/libthr/thread/thr_create.c:80
#5  0x0000000807bcdcec in std::sys::unix::thread::Thread::new ()
   from /usr/obj/usr/ports/lang/rust/work/rustc-1.41.0-src/build/powerpc64-unknown-freebsd/stage1/lib/libstd-d8a6fc3ad86fb29d.so
#6  0x0000000802c94300 in std::thread::Builder::spawn ()
   from /usr/obj/usr/ports/lang/rust/work/rustc-1.41.0-src/build/powerpc64-unknown-freebsd/stage1/lib/librustc_driver-9120d4fb86875292.so
#7  0x0000000802c5bfd8 in rustc_interface::util::spawn_thread_pool ()
   from /usr/obj/usr/ports/lang/rust/work/rustc-1.41.0-src/build/powerpc64-unknown-freebsd/stage1/lib/librustc_driver-9120d4fb86875292.so
#8  0x0000000802c4d350 in rustc_driver::run_compiler ()
   from /usr/obj/usr/ports/lang/rust/work/rustc-1.41.0-src/build/powerpc64-unknown-freebsd/stage1/lib/librustc_driver-9120d4fb86875292.so
#9  0x0000000802c6c2f8 in <std::panic::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once ()
   from /usr/obj/usr/ports/lang/rust/work/rustc-1.41.0-src/build/powerpc64-unknown-freebsd/stage1/lib/librustc_driver-9120d4fb86875292.so
#10 0x0000000802ccb628 in std::panicking::try::do_call ()
   from /usr/obj/usr/ports/lang/rust/work/rustc-1.41.0-src/build/powerpc64-unknown-freebsd/stage1/lib/librustc_driver-9120d4fb86875292.so
#11 0x0000000807c174b8 in __rust_maybe_catch_panic () from /usr/obj/usr/ports/lang/rust/work/rustc-1.41.0-src/build/powerpc64-unknown-freebsd/stage1/lib/libstd-d8a6fc3ad86fb29d.so
#12 0x0000000802c53470 in rustc_driver::catch_fatal_errors ()
   from /usr/obj/usr/ports/lang/rust/work/rustc-1.41.0-src/build/powerpc64-unknown-freebsd/stage1/lib/librustc_driver-9120d4fb86875292.so
#13 0x0000000802c54570 in rustc_driver::main () from /usr/obj/usr/ports/lang/rust/work/rustc-1.41.0-src/build/powerpc64-unknown-freebsd/stage1/lib/librustc_driver-9120d4fb86875292.so
#14 0x0000000001020ff8 in rustc_binary::main ()
#15 0x0000000001020f44 in std::rt::lang_start::{{closure}} ()
#16 0x0000000807c14928 in std::sys_common::backtrace::__rust_begin_short_backtrace ()
   from /usr/obj/usr/ports/lang/rust/work/rustc-1.41.0-src/build/powerpc64-unknown-freebsd/stage1/lib/libstd-d8a6fc3ad86fb29d.so
#17 0x0000000807bd36d0 in std::panicking::try::do_call () from /usr/obj/usr/ports/lang/rust/work/rustc-1.41.0-src/build/powerpc64-unknown-freebsd/stage1/lib/libstd-d8a6fc3ad86fb29d.so
#18 0x0000000807c174b8 in __rust_maybe_catch_panic () from /usr/obj/usr/ports/lang/rust/work/rustc-1.41.0-src/build/powerpc64-unknown-freebsd/stage1/lib/libstd-d8a6fc3ad86fb29d.so
#19 0x0000000807be5e00 in std::rt::lang_start_internal () from /usr/obj/usr/ports/lang/rust/work/rustc-1.41.0-src/build/powerpc64-unknown-freebsd/stage1/lib/libstd-d8a6fc3ad86fb29d.so
#20 0x000000000102105c in main ()

(gdb) print *obj
$9 = {magic = 3578837114, version = 1, next = {tqe_next = 0x80109a008, tqe_prev = 0x801095818}, path = 0x801092228 "/lib/libc.so.7", origin_path = 0x0, refcount = 1, holdcount = 0, 
  dl_refcount = 0, mapbase = 0x807c95000 "\177ELF\002\002\001\t", mapsize = 3719168, vaddrbase = 0, relocbase = 0x807c95000 "\177ELF\002\002\001\t", dynamic = 0x807edd488, 
  entry = 0x807d29660 <__do_global_dtors_aux> "<L", phdr = 0x807c95040, phsize = 560, interp = 0x0, stack_flags = 6, tlsindex = 3, tlsinit = 0x807ed52b0, tlsinitsize = 6144, 
  tlssize = 6200, tlsoffset = 48, tlsalign = 16, tlspoffset = 2228912, 
  relro_page = 0x807ed5000 "K\377\364\260K\377\364\254K\377\364\250K\377\364\244K\377\364\240K\377\364\234K\377\364\230K\377\364\224K\377\364\220K\377\364\214K\377\364\210K\377\364\204K\377\364\200K\377\364|K\377\364xK\377\364tK\377\364pK\377\364lK\377\364hK\377\364dK\377\364`K\377\364\\K\377\364XK\377\364TK\377\364PK\377\364LK\377\364HK\377\364DK\377\364@K\377\364<K\377\364\070K\377\364\064K\377\364\060K\377\364,K\377\364(K\377\364$K\377\364 K\377\364\034K\377\364\030K\377\364\024K\377\364\020K\377\364\fK\377\364\bK\377\364\004K\377\364", relro_size = 49152, 
  pltgot = 0x80801eba0, rel = 0x0, relsize = 0, rela = 0x807cbf0b0, relasize = 94632, pltrel = 0x0, pltrelsize = 0, pltrela = 0x807cd6258, pltrelasize = 21072, symtab = 0x807c95270, 
  strtab = 0x807cb5458 "", strsize = 40019, glink = 34492662988, verneed = 0x0, verneednum = 0, verdef = 0x807ca9314, verdefnum = 9, versyms = 0x807ca7a68, buckets = 0x807caf1b0, 
  nbuckets = 3157, chains = 0x807cb2304, nchains = 3157, nbuckets_gnu = 787, symndx_gnu = 6, maskwords_bm_gnu = 1023, shift2_gnu = 26, dynsymcount = 3157, bloom_gnu = 0x807ca9420, 
  buckets_gnu = 0x807cab420, chain_zero_gnu = 0x807cac054, rpath = 0x0, runpath = 0x0, needed = 0x0, needed_filtees = 0x0, needed_aux_filtees = 0x0, names = {stqh_first = 0x8010980c8, 
    stqh_last = 0x801098108}, vertab = 0x801093408, vernum = 10, init = 34492662848, fini = 34492662912, preinit_array = 0, init_array = 34492765296, fini_array = 34492765312, 
  preinit_array_num = 0, init_array_num = 2, fini_array_num = 1, osrel = 0, fctl0 = 0, mainprog = false, rtld = false, relocated = true, ver_checked = true, textrel = false, 
  symbolic = false, bind_now = false, traced = false, jmpslots_done = false, init_done = true, tls_done = true, phdr_alloc = false, z_origin = false, z_nodelete = false, z_noopen = false, 
  z_loadfltr = false, z_interpose = false, z_nodeflib = false, z_global = false, static_tls = false, static_tls_copied = false, ref_nodel = false, init_scanned = false, 
  on_fini_list = true, dag_inited = false, filtees_loaded = false, irelative = false, gnu_ifunc = false, non_plt_gnu_ifunc = false, ifuncs_resolved = true, crt_no_init = false, 
  valid_hash_sysv = true, valid_hash_gnu = true, dlopened = false, marker = false, unholdfree = false, doomed = false, linkmap = {l_addr = 0x807c95000 "\177ELF\002\002\001\t", 
    l_name = 0x801092228 "/lib/libc.so.7", l_ld = 0x807edd488, l_next = 0x80109a248, l_prev = 0x801095a48}, dldags = {stqh_first = 0x0, stqh_last = 0x801095e70}, dagmembers = {
    stqh_first = 0x0, stqh_last = 0x801095e80}, dev = 5295521634524941288, ino = 2408, priv = 0x0}

(gdb) print *objs.next.tqe_next
$13 = {magic = 3578837114, version = 1, next = {tqe_next = 0x80109a408, tqe_prev = 0x801095c18}, 
  path = 0x801093208 "/usr/obj/usr/ports/lang/rust/work/rustc-1.41.0-src/build/powerpc64-unknown-freebsd/stage1/lib/libtest-219a57759081028f.so", 
  origin_path = 0x801099808 "/usr/obj/usr/ports/lang/rust/work/rustc-1.41.0-src/build/powerpc64-unknown-freebsd/stage1/lib", refcount = 1, holdcount = 0, dl_refcount = 0, 
  mapbase = 0x808021000 "\177ELF\002\002\001\t", mapsize = 868352, vaddrbase = 0, relocbase = 0x808021000 "\177ELF\002\002\001\t", dynamic = 0x8080e4150, 
  entry = 0x80806204c <__do_global_dtors_aux> "<L", phdr = 0x808021040, phsize = 504, interp = 0x0, stack_flags = 6, tlsindex = 0, tlsinit = 0x0, tlsinitsize = 0, tlssize = 0, 
  tlsoffset = 0, tlsalign = 0, tlspoffset = 0, 
  relro_page = 0x8080dd000 "K\377\374\200K\377\374|K\377\374xK\377\374tK\377\374pK\377\374lK\377\374hK\377\374dK\377\374`K\377\374\\K\377\374XK\377\374TK\377\374PK\377\374LK\377\374HK\377\374DK\377\374@K\377\374<K\377\374\070K\377\374\064K\377\374\060K\377\374,K\377\374(K\377\374$K\377\374 K\377\374\034K\377\374\030K\377\374\024K\377\374\020", relro_size = 32768, 
  pltgot = 0x8080e4648, rel = 0x0, relsize = 0, rela = 0x808029d10, relasize = 43536, pltrel = 0x0, pltrelsize = 0, pltrela = 0x808034720, pltrelasize = 5712, symtab = 0x808021238, 
  strtab = 0x808024410 "", strsize = 22784, glink = 34494794908, verneed = 0x808023510, verneednum = 4, verdef = 0x0, verdefnum = 0, versyms = 0x808023260, buckets = 0x808023958, 
  nbuckets = 343, chains = 0x808023eb4, nchains = 343, nbuckets_gnu = 34, symndx_gnu = 205, maskwords_bm_gnu = 31, shift2_gnu = 26, dynsymcount = 343, bloom_gnu = 0x8080235a0, 
  buckets_gnu = 0x8080236a0, chain_zero_gnu = 0x8080233f4, rpath = 0x0, 
  runpath = 0x801097188 "/usr/obj/usr/ports/lang/rust/work/rustc-1.41.0-src/build/powerpc64-unknown-freebsd/stage1/lib/../lib", needed = 0x801092248, needed_filtees = 0x0, 
  needed_aux_filtees = 0x0, names = {stqh_first = 0x801098148, stqh_last = 0x801098148}, vertab = 0x801093508, vernum = 6, init = 34494794768, fini = 34494794832, preinit_array = 0, 
  init_array = 0, fini_array = 0, preinit_array_num = 0, init_array_num = 0, fini_array_num = 0, osrel = 0, fctl0 = 0, mainprog = false, rtld = false, relocated = true, ver_checked = true, 
  textrel = false, symbolic = false, bind_now = true, traced = false, jmpslots_done = true, init_done = true, tls_done = true, phdr_alloc = false, z_origin = false, z_nodelete = false, 
  z_noopen = false, z_loadfltr = false, z_interpose = false, z_nodeflib = false, z_global = false, static_tls = false, static_tls_copied = false, ref_nodel = false, init_scanned = false, 
  on_fini_list = true, dag_inited = false, filtees_loaded = false, irelative = false, gnu_ifunc = false, non_plt_gnu_ifunc = false, ifuncs_resolved = true, crt_no_init = false, 
  valid_hash_sysv = true, valid_hash_gnu = true, dlopened = false, marker = false, unholdfree = false, doomed = false, linkmap = {l_addr = 0x808021000 "\177ELF\002\002\001\t", 
    l_name = 0x801093208 "/usr/obj/usr/ports/lang/rust/work/rustc-1.41.0-src/build/powerpc64-unknown-freebsd/stage1/lib/libtest-219a57759081028f.so", l_ld = 0x8080e4150, 
    l_next = 0x80109a648, l_prev = 0x801095e48}, dldags = {stqh_first = 0x0, stqh_last = 0x80109a270}, dagmembers = {stqh_first = 0x0, stqh_last = 0x80109a280}, dev = 14625451739019144795, 
  ino = 1257657, priv = 0x0}
(gdb) print *objs.next.tqe_next.next.tqe_next
$14 = {magic = 3578837114, version = 1, next = {tqe_next = 0x80109a808, tqe_prev = 0x80109a018}, path = 0x801098188 "/usr/lib/libexecinfo.so.1", origin_path = 0x0, refcount = 1, 
  holdcount = 0, dl_refcount = 0, mapbase = 0x8080f5000 "\177ELF\002\002\001\t", mapsize = 208896, vaddrbase = 0, relocbase = 0x8080f5000 "\177ELF\002\002\001\t", dynamic = 0x8081176a8, 
  entry = 0x808106304 <__do_global_dtors_aux> "<L", phdr = 0x8080f5040, phsize = 504, interp = 0x0, stack_flags = 6, tlsindex = 0, tlsinit = 0x0, tlsinitsize = 0, tlssize = 0, 
  tlsoffset = 0, tlsalign = 0, tlspoffset = 0, relro_page = 0x808117000 "xc", relro_size = 4096, pltgot = 0x808127878, rel = 0x0, relsize = 0, rela = 0x8080f5ae0, relasize = 168, 
  pltrel = 0x0, pltrelsize = 0, pltrela = 0x8080f5b88, pltrelasize = 768, symtab = 0x8080f5238, strtab = 0x8080f58a8 "", strsize = 564, glink = 34495034844, verneed = 0x8080f567c, 
  verneednum = 3, verdef = 0x0, verdefnum = 0, versyms = 0x8080f5628, buckets = 0x8080f5758, nbuckets = 42, chains = 0x8080f5800, nchains = 42, nbuckets_gnu = 2, symndx_gnu = 32, 
  maskwords_bm_gnu = 1, shift2_gnu = 26, dynsymcount = 42, bloom_gnu = 0x8080f5710, buckets_gnu = 0x8080f5720, chain_zero_gnu = 0x8080f56a8, rpath = 0x0, runpath = 0x0, 
  needed = 0x8010922e8, needed_filtees = 0x0, needed_aux_filtees = 0x0, names = {stqh_first = 0x8010981c8, stqh_last = 0x801098208}, vertab = 0x801093608, vernum = 7, init = 34495034704, 
  fini = 34495034768, preinit_array = 0, init_array = 0, fini_array = 0, preinit_array_num = 0, init_array_num = 0, fini_array_num = 0, osrel = 0, fctl0 = 0, mainprog = false, 
  rtld = false, relocated = true, ver_checked = true, textrel = false, symbolic = false, bind_now = false, traced = false, jmpslots_done = false, init_done = true, tls_done = true, 
  phdr_alloc = false, z_origin = false, z_nodelete = false, z_noopen = false, z_loadfltr = false, z_interpose = false, z_nodeflib = false, z_global = false, static_tls = false, 
  static_tls_copied = false, ref_nodel = false, init_scanned = false, on_fini_list = true, dag_inited = false, filtees_loaded = false, irelative = false, gnu_ifunc = false, 
  non_plt_gnu_ifunc = false, ifuncs_resolved = true, crt_no_init = false, valid_hash_sysv = true, valid_hash_gnu = true, dlopened = false, marker = false, unholdfree = false, 
  doomed = false, linkmap = {l_addr = 0x8080f5000 "\177ELF\002\002\001\t", l_name = 0x801098188 "/usr/lib/libexecinfo.so.1", l_ld = 0x8081176a8, l_next = 0x80109aa48, 
    l_prev = 0x80109a248}, dldags = {stqh_first = 0x0, stqh_last = 0x80109a670}, dagmembers = {stqh_first = 0x0, stqh_last = 0x80109a680}, dev = 5295521634524941288, ino = 15148, 
  priv = 0x0}
(gdb) print *objs.next.tqe_next.next.tqe_next.next.tqe_next
$15 = {magic = 3578837114, version = 1, next = {tqe_next = 0x80109ac08, tqe_prev = 0x80109a418}, path = 0x801092348 "/lib/libthr.so.3", origin_path = 0x0, refcount = 1, holdcount = 0, 
  dl_refcount = 0, mapbase = 0x808128000 "\177ELF\002\002\001\t", mapsize = 405504, vaddrbase = 0, relocbase = 0x808128000 "\177ELF\002\002\001\t", dynamic = 0x80816e088, 
  entry = 0x808143cf0 <__do_global_dtors_aux> "<L", phdr = 0x808128040, phsize = 504, interp = 0x0, stack_flags = 6, tlsindex = 0, tlsinit = 0x0, tlsinitsize = 0, tlssize = 0, 
  tlsoffset = 0, tlsalign = 0, tlspoffset = 0, 
  relro_page = 0x80816d000 "\353\241\377\350\353\201\377\340\353a\377\330\353A\377\320\353!\377\310\353\001\377\300\352\341\377\270\352\301\377\260\352\241\377\250\352\201\377\240\352a\377\230\352A\377\220\352!\377\210N\200", relro_size = 4096, pltgot = 0x80818a0d8, rel = 0x0, relsize = 0, rela = 0x80812e4b8, relasize = 4368, pltrel = 0x0, pltrelsize = 0, 
  pltrela = 0x80812f5c8, pltrelasize = 2760, symtab = 0x808128238, strtab = 0x80812c300 "", strsize = 8628, glink = 34495388220, verneed = 0x80812ad90, verneednum = 1, 
  verdef = 0x80812ac94, verdefnum = 9, versyms = 0x80812a950, buckets = 0x80812b5f8, nbuckets = 417, chains = 0x80812bc7c, nchains = 417, nbuckets_gnu = 76, symndx_gnu = 113, 
  maskwords_bm_gnu = 63, shift2_gnu = 26, dynsymcount = 417, bloom_gnu = 0x80812ae00, buckets_gnu = 0x80812b000, chain_zero_gnu = 0x80812af6c, rpath = 0x0, runpath = 0x0, 
  needed = 0x801092368, needed_filtees = 0x0, needed_aux_filtees = 0x0, names = {stqh_first = 0x801098248, stqh_last = 0x801098288}, vertab = 0x801094a08, vernum = 15, init = 34495388080, 
  fini = 34495388144, preinit_array = 0, init_array = 34495455360, fini_array = 0, preinit_array_num = 0, init_array_num = 1, fini_array_num = 0, osrel = 0, fctl0 = 0, mainprog = false, 
  rtld = false, relocated = true, ver_checked = true, textrel = false, symbolic = false, bind_now = false, traced = false, jmpslots_done = false, init_done = true, tls_done = true, 
  phdr_alloc = false, z_origin = false, z_nodelete = true, z_noopen = false, z_loadfltr = false, z_interpose = false, z_nodeflib = false, z_global = false, static_tls = false, 
  static_tls_copied = false, ref_nodel = false, init_scanned = false, on_fini_list = true, dag_inited = false, filtees_loaded = false, irelative = false, gnu_ifunc = false, 
  non_plt_gnu_ifunc = false, ifuncs_resolved = true, crt_no_init = false, valid_hash_sysv = true, valid_hash_gnu = true, dlopened = false, marker = false, unholdfree = false, 
  doomed = false, linkmap = {l_addr = 0x808128000 "\177ELF\002\002\001\t", l_name = 0x801092348 "/lib/libthr.so.3", l_ld = 0x80816e088, l_next = 0x80109ae48, l_prev = 0x80109a648}, 
  dldags = {stqh_first = 0x0, stqh_last = 0x80109aa70}, dagmembers = {stqh_first = 0x0, stqh_last = 0x80109aa80}, dev = 5295521634524941288, ino = 20250, priv = 0x0}
(gdb) print *objs.next.tqe_next.next.tqe_next.next.tqe_next.next.tqe_next
$16 = {magic = 3578837114, version = 1, next = {tqe_next = 0x80109c008, tqe_prev = 0x80109a818}, path = 0x801092388 "/lib/libm.so.5", origin_path = 0x0, refcount = 1, holdcount = 0, 
  dl_refcount = 0, mapbase = 0x80818b000 "\177ELF\002\002\001\t", mapsize = 348160, vaddrbase = 0, relocbase = 0x80818b000 "\177ELF\002\002\001\t", dynamic = 0x8081cf450, 
  entry = 0x8081a52d4 <__do_global_dtors_aux> "<L", phdr = 0x80818b040, phsize = 504, interp = 0x0, stack_flags = 6, tlsindex = 0, tlsinit = 0x0, tlsinitsize = 0, tlssize = 0, 
  tlsoffset = 0, tlsalign = 0, tlspoffset = 0, relro_page = 0x8081cf000 "\351\214\204x}\211\003\246N\200\004 \370A", relro_size = 4096, pltgot = 0x8081df7f8, rel = 0x0, relsize = 0, 
  rela = 0x80818eae0, relasize = 1080, pltrel = 0x0, pltrelsize = 0, pltrela = 0x80818ef18, pltrelasize = 2376, symtab = 0x80818b238, strtab = 0x80818e230 "", strsize = 2217, 
  glink = 34495787644, verneed = 0x80818d12c, verneednum = 1, verdef = 0x80818d030, verdefnum = 9, versyms = 0x80818cde0, buckets = 0x80818d8f8, nbuckets = 295, chains = 0x80818dd94, 
  nchains = 295, nbuckets_gnu = 70, symndx_gnu = 13, maskwords_bm_gnu = 63, shift2_gnu = 26, dynsymcount = 295, bloom_gnu = 0x80818d170, buckets_gnu = 0x80818d370, 
  chain_zero_gnu = 0x80818d454, rpath = 0x0, runpath = 0x0, needed = 0x8010923a8, needed_filtees = 0x0, needed_aux_filtees = 0x0, names = {stqh_first = 0x8010982c8, 
    stqh_last = 0x801098308}, vertab = 0x801094c08, vernum = 12, init = 34495787504, fini = 34495787568, preinit_array = 0, init_array = 0, fini_array = 0, preinit_array_num = 0, 
  init_array_num = 0, fini_array_num = 0, osrel = 0, fctl0 = 0, mainprog = false, rtld = false, relocated = true, ver_checked = true, textrel = false, symbolic = false, bind_now = false, 
  traced = false, jmpslots_done = false, init_done = true, tls_done = true, phdr_alloc = false, z_origin = false, z_nodelete = false, z_noopen = false, z_loadfltr = false, 
  z_interpose = false, z_nodeflib = false, z_global = false, static_tls = false, static_tls_copied = false, ref_nodel = false, init_scanned = false, on_fini_list = true, 
  dag_inited = false, filtees_loaded = false, irelative = false, gnu_ifunc = false, non_plt_gnu_ifunc = false, ifuncs_resolved = true, crt_no_init = false, valid_hash_sysv = true, 
  valid_hash_gnu = true, dlopened = false, marker = false, unholdfree = false, doomed = false, linkmap = {l_addr = 0x80818b000 "\177ELF\002\002\001\t", 
    l_name = 0x801092388 "/lib/libm.so.5", l_ld = 0x8081cf450, l_next = 0x80109c248, l_prev = 0x80109aa48}, dldags = {stqh_first = 0x0, stqh_last = 0x80109ae70}, dagmembers = {
    stqh_first = 0x0, stqh_last = 0x80109ae80}, dev = 5295521634524941288, ino = 3647, priv = 0x0}
(gdb) print *objs.next.tqe_next.next.tqe_next.next.tqe_next.next.tqe_next.next.tqe_next
$17 = {magic = 3578837114, version = 1, next = {tqe_next = 0x80109c408, tqe_prev = 0x80109ac18}, path = 0x8010923c8 "/usr/lib/libc++.so.1", origin_path = 0x0, refcount = 1, holdcount = 0, 
  dl_refcount = 0, mapbase = 0x8081e0000 "\177ELF\002\002\001\t", mapsize = 1142784, vaddrbase = 0, relocbase = 0x8081e0000warning: (Internal error: pc 0x8081e0000 in read in psymtab, but not in symtab.)

 "\177ELF\002\002\001\t", dynamic = 0x8082e3068, 
  entry = 0x80824819c <__do_global_dtors_aux> "<L", phdr = 0x8081e0040warning: (Internal error: pc 0x8081e0040 in read in psymtab, but not in symtab.)

, phsize = 504, interp = 0x0, stack_flags = 6, tlsindex = 0, tlsinit = 0x0, tlsinitsize = 0, tlssize = 0, 
  tlsoffset = 0, tlsalign = 0, tlspoffset = 0, relro_page = 0x8082de000 "}\211\003\246N\200\004 \370A", relro_size = 20480, pltgot = 0x8082f5f48, rel = 0x0, relsize = 0, 
  rela = 0x80820fd50, relasize = 49728, pltrel = 0x0, pltrelsize = 0, pltrela = 0x80821bf90, pltrelasize = 10416, symtab = 0x8081e0238, strtab = 0x8081f5bc0 "", strsize = 106894, 
  glink = 34496898012, verneed = 0x8081edfa0, verneednum = 3, verdef = 0x0, verdefnum = 0, versyms = 0x8081ece98, buckets = 0x8081f17a0, nbuckets = 2180, chains = 0x8081f39b0, 
  nchains = 2180, nbuckets_gnu = 501, symndx_gnu = 175, maskwords_bm_gnu = 511, shift2_gnu = 26, dynsymcount = 2180, bloom_gnu = 0x8081ee070, buckets_gnu = 0x8081ef070, 
  chain_zero_gnu = 0x8081ef588, rpath = 0x0, runpath = 0x0, needed = 0x8010923e8, needed_filtees = 0x0, needed_aux_filtees = 0x0, names = {stqh_first = 0x801098348, 
    stqh_last = 0x801098388}, vertab = 0x801094e08, vernum = 11, init = 34496897872, fini = 34496897936, preinit_array = 0, init_array = 34496983136, fini_array = 0, preinit_array_num = 0, 
  init_array_num = 1, fini_array_num = 0, osrel = 0, fctl0 = 0, mainprog = false, rtld = false, relocated = true, ver_checked = true, textrel = false, symbolic = false, bind_now = false, 
  traced = false, jmpslots_done = false, init_done = true, tls_done = true, phdr_alloc = false, z_origin = false, z_nodelete = false, z_noopen = false, z_loadfltr = false, 
  z_interpose = false, z_nodeflib = false, z_global = false, static_tls = false, static_tls_copied = false, ref_nodel = false, init_scanned = false, on_fini_list = true, 
  dag_inited = false, filtees_loaded = false, irelative = false, gnu_ifunc = false, non_plt_gnu_ifunc = false, ifuncs_resolved = true, crt_no_init = false, valid_hash_sysv = true, 
  valid_hash_gnu = true, dlopened = false, marker = false, unholdfree = false, doomed = false, linkmap = {l_addr = 0x8081e0000warning: (Internal error: pc 0x8081e0000 in read in psymtab, but not in symtab.)

 "\177ELF\002\002\001\t", 
    l_name = 0x8010923c8 "/usr/lib/libc++.so.1", l_ld = 0x8082e3068, l_next = 0x80109c648, l_prev = 0x80109ae48}, dldags = {stqh_first = 0x0, stqh_last = 0x80109c270}, dagmembers = {
    stqh_first = 0x0, stqh_last = 0x80109c280}, dev = 5295521634524941288, ino = 5234, priv = 0x0}
(gdb) print *objs.next.tqe_next.next.tqe_next.next.tqe_next.next.tqe_next.next.tqe_next.next.tqe_next
$18 = {magic = 3578837114, version = 1, next = {tqe_next = 0x80109c808, tqe_prev = 0x80109c018}, path = 0x801092448 "/lib/libcxxrt.so.1", origin_path = 0x0, refcount = 1, holdcount = 0, 
  dl_refcount = 0, mapbase = 0x8082f7000 "\177ELF\002\002\001\t", mapsize = 331776, vaddrbase = 0, relocbase = 0x8082f7000 "\177ELF\002\002\001\t", dynamic = 0x808332ae0, 
  entry = 0x808312e94 <__do_global_dtors_aux> "<L", phdr = 0x8082f7040, phsize = 504, interp = 0x0, stack_flags = 6, tlsindex = 0, tlsinit = 0x0, tlsinitsize = 0, tlssize = 0, 
  tlsoffset = 0, tlsalign = 0, tlspoffset = 0, relro_page = 0x808331000 "", relro_size = 8192, pltgot = 0x808346f48, rel = 0x0, relsize = 0, rela = 0x8082fca30, relasize = 9768, 
  pltrel = 0x0, pltrelsize = 0, pltrela = 0x8082ff058, pltrelasize = 1608, symtab = 0x8082f7238, strtab = 0x8082faddc "", strsize = 7246, glink = 34497240236, verneed = 0x8082f9968, 
  verneednum = 2, verdef = 0x8082f9834, verdefnum = 11, versyms = 0x8082f9548, buckets = 0x8082fa22c, nbuckets = 374, chains = 0x8082fa804, nchains = 374, nbuckets_gnu = 80, 
  symndx_gnu = 51, maskwords_bm_gnu = 63, shift2_gnu = 26, dynsymcount = 374, bloom_gnu = 0x8082f99d8, buckets_gnu = 0x8082f9bd8, chain_zero_gnu = 0x8082f9c4c, rpath = 0x0, runpath = 0x0, 
  needed = 0x801092468, needed_filtees = 0x0, needed_aux_filtees = 0x0, names = {stqh_first = 0x8010983c8, stqh_last = 0x801098408}, vertab = 0x80109d008, vernum = 16, init = 34497240096, 
  fini = 34497240160, preinit_array = 0, init_array = 0, fini_array = 0, preinit_array_num = 0, init_array_num = 0, fini_array_num = 0, osrel = 0, fctl0 = 0, mainprog = false, 
  rtld = false, relocated = true, ver_checked = true, textrel = false, symbolic = false, bind_now = false, traced = false, jmpslots_done = false, init_done = true, tls_done = true, 
  phdr_alloc = false, z_origin = false, z_nodelete = false, z_noopen = false, z_loadfltr = false, z_interpose = false, z_nodeflib = false, z_global = false, static_tls = false, 
  static_tls_copied = false, ref_nodel = false, init_scanned = false, on_fini_list = true, dag_inited = false, filtees_loaded = false, irelative = false, gnu_ifunc = false, 
  non_plt_gnu_ifunc = false, ifuncs_resolved = true, crt_no_init = false, valid_hash_sysv = true, valid_hash_gnu = true, dlopened = false, marker = false, unholdfree = false, 
  doomed = false, linkmap = {l_addr = 0x8082f7000 "\177ELF\002\002\001\t", l_name = 0x801092448 "/lib/libcxxrt.so.1", l_ld = 0x808332ae0, l_next = 0x80109ca48, l_prev = 0x80109c248}, 
  dldags = {stqh_first = 0x0, stqh_last = 0x80109c670}, dagmembers = {stqh_first = 0x0, stqh_last = 0x80109c680}, dev = 5295521634524941288, ino = 3581, priv = 0x0}
(gdb) print *objs.next.tqe_next.next.tqe_next.next.tqe_next.next.tqe_next.next.tqe_next.next.tqe_next.next.tqe_next
$19 = {magic = 3578837114, version = 1, next = {tqe_next = 0x80109cc08, tqe_prev = 0x80109c418}, path = 0x8010924a8 "/lib/libgcc_s.so.1", origin_path = 0x0, refcount = 1, holdcount = 0, 
  dl_refcount = 0, mapbase = 0x808348000 "\177ELF\002\002\001\t", mapsize = 303104, vaddrbase = 0, relocbase = 0x808348000 "\177ELF\002\002\001\t", dynamic = 0x808380968, 
  entry = 0x808360524 <__do_global_dtors_aux> "<L", phdr = 0x808348040, phsize = 504, interp = 0x0, stack_flags = 6, tlsindex = 0, tlsinit = 0x0, tlsinitsize = 0, tlssize = 0, 
  tlsoffset = 0, tlsalign = 0, tlspoffset = 0, relro_page = 0x808380000 "p\240", relro_size = 4096, pltgot = 0x8083913c0, rel = 0x0, relsize = 0, rela = 0x808349fc8, relasize = 3576, 
  pltrel = 0x0, pltrelsize = 0, pltrela = 0x80834adc0, pltrelasize = 840, symtab = 0x808348238, strtab = 0x808349878 "", strsize = 1866, glink = 34497561740, verneed = 0x8083490f0, 
  verneednum = 1, verdef = 0x808348fbc, verdefnum = 11, versyms = 0x808348eb0, buckets = 0x808349450, nbuckets = 133, chains = 0x808349664, nchains = 133, nbuckets_gnu = 27, 
  symndx_gnu = 22, maskwords_bm_gnu = 31, shift2_gnu = 26, dynsymcount = 133, bloom_gnu = 0x808349120, buckets_gnu = 0x808349220, chain_zero_gnu = 0x808349234, rpath = 0x0, runpath = 0x0, 
  needed = 0x8010924c8, needed_filtees = 0x0, needed_aux_filtees = 0x0, names = {stqh_first = 0x801098448, stqh_last = 0x801098488}, vertab = 0x80109d208, vernum = 13, init = 34497561600, 
  fini = 34497561664, preinit_array = 0, init_array = 0, fini_array = 0, preinit_array_num = 0, init_array_num = 0, fini_array_num = 0, osrel = 0, fctl0 = 0, mainprog = false, 
  rtld = false, relocated = true, ver_checked = true, textrel = false, symbolic = false, bind_now = false, traced = false, jmpslots_done = false, init_done = true, tls_done = true, 
  phdr_alloc = false, z_origin = false, z_nodelete = false, z_noopen = false, z_loadfltr = false, z_interpose = false, z_nodeflib = false, z_global = false, static_tls = false, 
  static_tls_copied = false, ref_nodel = false, init_scanned = false, on_fini_list = true, dag_inited = false, filtees_loaded = false, irelative = false, gnu_ifunc = false, 
  non_plt_gnu_ifunc = false, ifuncs_resolved = true, crt_no_init = false, valid_hash_sysv = true, valid_hash_gnu = true, dlopened = false, marker = false, unholdfree = false, 
  doomed = false, linkmap = {l_addr = 0x808348000 "\177ELF\002\002\001\t", l_name = 0x8010924a8 "/lib/libgcc_s.so.1", l_ld = 0x808380968, l_next = 0x80109ce48, l_prev = 0x80109c648}, 
  dldags = {stqh_first = 0x0, stqh_last = 0x80109ca70}, dagmembers = {stqh_first = 0x0, stqh_last = 0x80109ca80}, dev = 5295521634524941288, ino = 12097, priv = 0x0}
(gdb) print *objs.next.tqe_next.next.tqe_next.next.tqe_next.next.tqe_next.next.tqe_next.next.tqe_next.next.tqe_next.next.tqe_next
$20 = {magic = 3578837114, version = 1, next = {tqe_next = 0x0, tqe_prev = 0x80109c818}, path = 0x8010924e8 "/lib/libelf.so.2", origin_path = 0x0, refcount = 1, holdcount = 0, 
  dl_refcount = 0, mapbase = 0x808392000 "\177ELF\002\002\001\t", mapsize = 311296, vaddrbase = 0, relocbase = 0x808392000 "\177ELF\002\002\001\t", dynamic = 0x8083cd7d0, 
  entry = 0x8083a88fc <__do_global_dtors_aux> "<L", phdr = 0x808392040, phsize = 504, interp = 0x0, stack_flags = 6, tlsindex = 0, tlsinit = 0x0, tlsinitsize = 0, tlssize = 0, 
  tlsoffset = 0, tlsalign = 0, tlspoffset = 0, relro_page = 0x8083cd000 "=\202", relro_size = 4096, pltgot = 0x8083ddca8, rel = 0x0, relsize = 0, rela = 0x808393a20, relasize = 4152, 
  pltrel = 0x0, pltrelsize = 0, pltrela = 0x808394a58, pltrelasize = 912, symtab = 0x808392238, strtab = 0x80839347c "", strsize = 1438, glink = 34497876844, verneed = 0x808392dec, 
  verneednum = 1, verdef = 0x808392db4, verdefnum = 2, versyms = 0x808392cd0, buckets = 0x8083930f4, nbuckets = 113, chains = 0x8083932b8, nchains = 113, nbuckets_gnu = 22, 
  symndx_gnu = 24, maskwords_bm_gnu = 31, shift2_gnu = 26, dynsymcount = 113, bloom_gnu = 0x808392e30, buckets_gnu = 0x808392f30, chain_zero_gnu = 0x808392f28, rpath = 0x0, runpath = 0x0, 
  needed = 0x801092508, needed_filtees = 0x0, needed_aux_filtees = 0x0, names = {stqh_first = 0x8010984c8, stqh_last = 0x801098508}, vertab = 0x801097288, vernum = 5, init = 34497876704, 
  fini = 34497876768, preinit_array = 0, init_array = 0, fini_array = 0, preinit_array_num = 0, init_array_num = 0, fini_array_num = 0, osrel = 0, fctl0 = 0, mainprog = false, 
  rtld = false, relocated = true, ver_checked = true, textrel = false, symbolic = false, bind_now = false, traced = false, jmpslots_done = false, init_done = true, tls_done = true, 
  phdr_alloc = false, z_origin = false, z_nodelete = false, z_noopen = false, z_loadfltr = false, z_interpose = false, z_nodeflib = false, z_global = false, static_tls = false, 
  static_tls_copied = false, ref_nodel = false, init_scanned = false, on_fini_list = true, dag_inited = false, filtees_loaded = false, irelative = false, gnu_ifunc = false, 
  non_plt_gnu_ifunc = false, ifuncs_resolved = true, crt_no_init = false, valid_hash_sysv = true, valid_hash_gnu = true, dlopened = false, marker = false, unholdfree = false, 
  doomed = false, linkmap = {l_addr = 0x808392000 "\177ELF\002\002\001\t", l_name = 0x8010924e8 "/lib/libelf.so.2", l_ld = 0x8083cd7d0, l_next = 0x801090830 <obj_rtld+576>, 
    l_prev = 0x80109ca48}, dldags = {stqh_first = 0x0, stqh_last = 0x80109ce70}, dagmembers = {stqh_first = 0x0, stqh_last = 0x80109ce80}, dev = 5295521634524941288, ino = 13210, 
  priv = 0x0}

root@talos:/libexec # readelf -l /lib/libc.so.7

Elf file type is DYN (Shared object file)
Entry point 0x94660
There are 10 program headers, starting at offset 64

Program Headers:
  Type           Offset             VirtAddr           PhysAddr
                 FileSiz            MemSiz              Flg    Align
  PHDR           0x0000000000000040 0x0000000000000040 0x0000000000000040
                 0x0000000000000230 0x0000000000000230  R      0x8
  LOAD           0x0000000000000000 0x0000000000000000 0x0000000000000000
                 0x0000000000084654 0x0000000000084654  R      0x10000
  LOAD           0x0000000000084660 0x0000000000094660 0x0000000000094660
                 0x000000000019bc44 0x000000000019bc44  R E    0x10000
  LOAD           0x00000000002202b0 0x00000000002402b0 0x00000000002402b0
                 0x000000000000b830 0x000000000000b830  RW     0x10000
  LOAD           0x000000000022bae0 0x000000000025bae0 0x000000000025bae0
                 0x0000000000004080 0x000000000012fc40  RW     0x10000
  TLS            0x00000000002202b0 0x00000000002402b0 0x00000000002402b0
                 0x0000000000001800 0x0000000000001838  R      0x10
  DYNAMIC        0x0000000000228488 0x0000000000248488 0x0000000000248488
                 0x00000000000001a0 0x00000000000001a0  RW     0x8
  GNU_RELRO      0x00000000002202b0 0x00000000002402b0 0x00000000002402b0
                 0x000000000000b830 0x000000000000b830  R      0x1
  GNU_EH_FRAME   0x000000000005d450 0x000000000005d450 0x000000000005d450
                 0x00000000000075fc 0x00000000000075fc  R      0x4
  GNU_STACK      0x0000000000000000 0x0000000000000000 0x0000000000000000
                 0x0000000000000000 0x0000000000000000  RW     0

 Section to Segment mapping:
  Segment Sections...
   00     
   01     .dynsym .gnu.version .gnu.version_d .gnu.hash .hash .dynstr .rela.dyn .rela.plt .rodata .eh_frame_hdr .eh_frame 
   02     .text .init .fini .glink 
   03     .tdata .tbss .ctors .dtors .jcr .data.rel.ro .init_array .fini_array .dynamic .got .toc 
   04     .data .bss .plt .branch_lt 
   05     .tdata .tbss 
   06     .dynamic 
   07     .tdata .tbss .ctors .dtors .jcr .data.rel.ro .init_array .fini_array .dynamic .got .toc 
   08     .eh_frame_hdr 
   09

[bdragon]

I kinda wonder if what I'm running into is a corner case where the main program has no TLS section yet still creates threads.

Rust manually bootstraps pthread the first time a thread is created, this might be breaking some of our assumptions.

[bdragon]

libexec/rtld-elf/rtld.c

5026 ↗

(On Diff #61794)

This might be the bit I am hunting for.
I suspect this should be obj->tlspoffset & (obj->tlsalign - 1)

[bdragon]

libexec/rtld-elf/rtld.c

5026 ↗

(On Diff #61794)

Nope, it's not, as malloc_aligned already takes care of masking it off.

[kib]

Allocate enough slack to align the resulting pointer. Hopefully this fixes reported memory corruption.

[bdragon]

It did not.

I traced it to the point where tls_max_index gets overwritten:

Reading symbols from /usr/obj/usr/ports/lang/rust/work/rustc-1.41.0-src/build/powerpc64-unknown-freebsd/stage1/bin/rustc...
(gdb) break _rtld_start
Function "_rtld_start" not defined.
Make breakpoint pending on future shared library load? (y or [n]) y
Breakpoint 1 (_rtld_start) pending.
(gdb) run
Starting program: /usr/obj/usr/ports/lang/rust/work/rustc-1.41.0-src/build/powerpc64-unknown-freebsd/stage1/bin/rustc --crate-name ___ --print=file-names -Zexternal-macro-backtrace -Clink-args=-Wl,-rpath,/../lib -Wrust_2018_idioms -Wunused_lifetimes -Dwarnings -Zsave-analysis -Cprefer-dynamic --target powerpc64-unknown-freebsd --crate-type bin --crate-type rlib --crate-type dylib --crate-type cdylib --crate-type staticlib --crate-type proc-macro --print=sysroot --print=cfg --sysroot /usr/obj/usr/ports/lang/rust/work/rustc-1.41.0-src/build/powerpc64-unknown-freebsd/stage1 -C debug-assertions=n -Z force-unstable-if-unmarked

Breakpoint 1, _rtld_start () at /usr/src/libexec/rtld-elf/powerpc64/rtld_start.S:37
37	_ENTRY(_rtld_start)
(gdb) print tls_max_index
$1 = 1
(gdb) watch tls_max_index
Watchpoint 2: tls_max_index
(gdb) cont
Continuing.

Watchpoint 2: tls_max_index

Old value = 1
New value = 2
map_object (fd=<optimized out>, path=0x801093008 "/usr/obj/usr/ports/lang/rust/work/rustc-1.41.0-src/build/powerpc64-unknown-freebsd/stage1/lib/librustc_driver-9120d4fb86875292.so", 
    sb=0x3fffffffffffa688) at /usr/src/libexec/rtld-elf/map_object.c:318
318		obj->tlsinit = mapbase + phtls->p_vaddr;
(gdb) cont
Continuing.

Watchpoint 2: tls_max_index

Old value = 2
New value = 3
map_object (fd=<optimized out>, path=0x801092208 "/lib/libc.so.7", sb=0x3fffffffffffa688) at /usr/src/libexec/rtld-elf/map_object.c:318
318		obj->tlsinit = mapbase + phtls->p_vaddr;
(gdb) cont
Continuing.

Watchpoint 2: tls_max_index

Old value = 3
New value = 0
0x000000080106ea90 in memset (dst0=0x801090710 <tls_max_index>, c0=17368848, length=4) at /usr/src/lib/libc/string/memset.c:90
90				*dst++ = VAL;
(gdb) bt
#0  0x000000080106ea90 in memset (dst0=0x801090710 <tls_max_index>, c0=17368848, length=4) at /usr/src/lib/libc/string/memset.c:90
#1  0x000000080105dda4 in allocate_tls (objs=0x801095408, oldtcb=<optimized out>, tcbsize=<optimized out>, tcbalign=<optimized out>) at /usr/src/libexec/rtld-elf/rtld.c:4924
#2  0x000000080105730c in allocate_initial_tls (list=<optimized out>) at /usr/src/libexec/rtld-elf/powerpc64/reloc.c:715
#3  0x0000000801058d20 in _rtld (sp=<optimized out>, exit_proc=<optimized out>, objp=<optimized out>) at /usr/src/libexec/rtld-elf/rtld.c:775
#4  0x00000008010561d0 in _rtld_start () at /usr/src/libexec/rtld-elf/powerpc64/rtld_start.S:86
Backtrace stopped: frame did not save the PC
(gdb) frame 1
#1  0x000000080105dda4 in allocate_tls (objs=0x801095408, oldtcb=<optimized out>, tcbsize=<optimized out>, tcbalign=<optimized out>) at /usr/src/libexec/rtld-elf/rtld.c:4924
4924			 memset((void*) addr, 0, tls_init_offset);

} else {
    dtv = xcalloc(tls_max_index + 2, sizeof(Elf_Addr));
    tcb[0] = dtv;
    dtv[0] = tls_dtv_generation;
    dtv[1] = tls_max_index;

    for (obj = globallist_curr(objs); obj != NULL;
       obj = globallist_next(obj)) {
         tls_init_offset = obj->tlspoffset & (obj->tlsalign - 1);
         if (tls_init_offset > 0)
             memset((void*) addr, 0, tls_init_offset); <----- HERE
         if (obj->tlsoffset > 0) {
             addr = (Elf_Addr)tcb + obj->tlsoffset;
             if (obj->tlsinitsize > 0) {
                 memcpy((void *)(addr + tls_init_offset), obj->tlsinit,
                    obj->tlsinitsize);
             }
             if (obj->tlssize > obj->tlsinitsize) {
                 memset((void *)(addr + tls_init_offset + obj->tlsinitsize),
                    0, obj->tlssize - obj->tlsinitsize - tls_init_offset);
             }
             dtv[obj->tlsindex + 1] = addr;
         }
    }
}

I don't remember why my suggestion previously had that memset in the first place. Where are we initializing addr?

[bdragon]

Ahh, right below. So I guess we're using it unitialized, so it uses a random pointer from the stack.

[bdragon]

yep, I moved the setting of addr up a bit and I no longer crash.

[kib]

I see. I revamped the loop even more because there seems to be more. Please try this.

[kib]

Redo the initialization.

[bdragon]

OK, that appears to do the trick.

[bdragon]

I've now been running this for over a month on powerpcspe, powerpc, and powerpc64, with no additional issues.

[emaste]

This patch is running on my desktop without issue

[bz]

... and on my arm64 boards for 66 days or shorter. Updated to the latest version a few days ago. So far no issues either.

[bdragon]

Might be possibly getting crashes on this again with clang10. Still investigating.

[bdragon]

OK, with a pure lld10 ppc32 build, this is apparently causing crashes in threaded programs again. I'm not sure what changed.

[kib]

In D21163#532233, @bdragon wrote:

OK, with a pure lld10 ppc32 build, this is apparently causing crashes in threaded programs again. I'm not sure what changed.

Does crashing program have unaligned TLS segment ? Or some object loaded into the process with such TLS segment.

[bdragon]

I don't have good data on it yet, just giving a heads up that I'm hitting crashes again. Clang crashes on exit, which is very reminicient of past memory corruption issues here. I don't currently have a rust to test with on ppc32 to check that.

For the moment, I'm going to focus on getting the ppc32 transition to lld10 done, and then I'll circle back to this and get some backtraces and figure out why this is suddenly causing problems again.

[bdragon]

OK, I retested again and I can no longer reproduce the crashes I was having.

I am going to assume I had either tested an old version of the patch or there were some temporary compiler problems breaking it.