Just found one more issue but otherwise LGTM. Maybe better to consistently use the _Tthr prefix.

I can confirm that it works when I apply the following change on top of your patch:

diff --git a/lib/libthr/thread/thr_init.c b/lib/libthr/thread/thr_init.c
index d9bde8a37c9d..78146c2d6dcd 100644
--- a/lib/libthr/thread/thr_init.c
+++ b/lib/libthr/thread/thr_init.c
@@ -257,7 +257,7 @@ static pthread_func_t jmp_table[][2] = {
        [PJT_RWLOCK_TRYWRLOCK] = {DUAL_ENTRY(_Tthr_rwlock_trywrlock)},
        [PJT_RWLOCK_UNLOCK] = {DUAL_ENTRY(_Tthr_rwlock_unlock)},
        [PJT_RWLOCK_WRLOCK] = {DUAL_ENTRY(_Tthr_rwlock_wrlock)},
-       [PJT_SELF] = {DUAL_ENTRY(_thr_self)},
+       [PJT_SELF] = {DUAL_ENTRY(_Tthr_self)},
        [PJT_SETCANCELSTATE] = {DUAL_ENTRY(_thr_setcancelstate)},
        [PJT_SETCANCELTYPE] = {DUAL_ENTRY(_thr_setcanceltype)},
        [PJT_SETSPECIFIC] = {DUAL_ENTRY(_thr_setspecific)},
diff --git a/lib/libthr/thread/thr_private.h b/lib/libthr/thread/thr_private.h
index 90385d1ab31c..47b82009765d 100644
--- a/lib/libthr/thread/thr_private.h
+++ b/lib/libthr/thread/thr_private.h
@@ -1069,7 +1069,7 @@ int _thr_setspecific(pthread_key_t, const void *);
 void *_thr_getspecific(pthread_key_t);
 int _thr_setcancelstate(int, int *);
 int _thr_setcanceltype(int, int *);
-pthread_t _thr_self(void);
+pthread_t _Tthr_self(void);
 int _thr_rwlock_init(pthread_rwlock_t *, const pthread_rwlockattr_t *);
 int _thr_rwlock_destroy(pthread_rwlock_t *);
 int _Tthr_rwlock_rdlock(pthread_rwlock_t *);
diff --git a/lib/libthr/thread/thr_self.c b/lib/libthr/thread/thr_self.c
index 168d91721e0b..68bf5bc39312 100644
--- a/lib/libthr/thread/thr_self.c
+++ b/lib/libthr/thread/thr_self.c
@@ -38,11 +38,11 @@ __FBSDID("$FreeBSD$");

Some more information:

(lldb) register read
General Purpose Registers:
       rax = 0x00000000000001b0
       rbx = 0x0000000000000001
       rcx = 0x00007fffffffeb10
       rdx = 0x00000000002050fd
       rdi = 0x000000000031d2e8  __sF + 312
       rsi = 0x000000000031bec0  __xlocale_global_locale
       rbp = 0x00007fffffffe890
       rsp = 0x00007fffffffe878
        r8 = 0x00007fffffffeb10
        r9 = 0x00007fffffffeb10
       r10 = 0x00007fffffffeb10
       r11 = 0x0000000000000201
       r12 = 0x00000000002002e0
       r13 = 0x00000000002002e0
       r14 = 0x00007fffffffebd8
       r15 = 0x00007fffffffebe8
       rip = 0x00000000002fd4b8  test-c-first`__sys_thr_self + 8
    rflags = 0x0000000000000202
        cs = 0x0000000000000043

With -O0 at the time of the crash:

(lldb) p *__stdoutp
(FILE) $1 = {
  _p = 0x00000000000186e8 ""
  _r = 0
  _w = 0
  _flags = 2185
  _file = 1
  _bf = (_base = "pthread_create = 0x301e10\n\xa5\xa5\xa5\xa5\xa5\xa5\xa5\xa5\xa5\xa5\xa5\xa5\xa5\xa5\xa5\xa5\xa5\xa5\xa5\xa5\xa5\xa5\xa5\xa5\xa5\xa5\xa5\xa5\xa5\xa5\xa5)
  _lbfsize = -4096
  _cookie = 0x000000000031d2e8
  _close = 0x00000000002fe580 (test-c-first`__sclose at stdio.c:80)
  _read = 0x00000000002fe4c0 (test-c-first`__sread at stdio.c:56)
  _seek = 0x00000000002fe540 (test-c-first`__sseek at stdio.c:72)
  _write = 0x00000000002fe500 (test-c-first`__swrite at stdio.c:64)
  _ub = (_base = 0x0000000000000000, _size = 0)
  _up = 0x0000000000000000
  _ur = 0
  _ubuf = ""
  _nbuf = ""
  _lb = (_base = 0x0000000000000000, _size = 0)
  _blksize = 4096
  _offset = 0
  _fl_mutex = 0x0000000000000000
  _fl_owner = 0x0000000000000000
  _fl_count = 1
  _orientation = 0
  _mbstate = (__mbstate8 = "", _mbstateL = 0)
  _flags2 = 0
}
(lldb) p __stdoutp->_bf._base
(unsigned char *) $4 = 0x000000080033b000 "pthread_create = 0x301e10\n\xa5\xa5\xa5\xa5\xa5\xa5\xa5\xa5\xa5\xa5\xa5\xa5\xa5\xa5\xa5\xa5\xa5\xa5\xa5\xa5\xa5\xa5\xa5\xa5\xa.
(lldb) p __stdoutp->_bf._size
(int) $5 = 4096

Just to be clear this only happens if I actually start a thread and call pthread_join (using the program I posted here). The simple case from the PR works fine.

In D21088#458174, @kib wrote:

In D21088#457869, @arichardson wrote:

I am now getting a very strange error:

root@qemu-native-alr48:~ # clang -static -o test-thr-first -nodefaultlibs -lthr -lc test.c -fuse-ld=lld && ./test-thr-first
pthread_create = 0x2693d0
result = 0x1234
root@qemu-native-alr48:~ # clang -static -o test-c-first -nodefaultlibs -lc -lthr test.c -fuse-ld=lld && ./test-c-first
pthread_create = 0x2a58a0
Segmentation fault (core dumped)
root@qemu-native-alr48:~ # clang -static -o test-c-first -nodefaultlibs -lc -lthr test.c -fuse-ld=lld && lldb -- ./test-c-first
(lldb) target create "./test-c-first"
Current executable set to './test-c-first' (x86_64).
(lldb) r
Process 746 launching
Process 746 launched: '/root/test-c-first' (x86_64)
pthread_create = 0x2a58a0
Process 746 stopped
* thread #1, name = 'test-c-first', stop reason = signal SIGSEGV: invalid address (fault address: 0x186e8)
    frame #0: 0x00000000002a4b8d test-c-first`memcpy at memmove.S:306
(lldb) bt
* thread #1, name = 'test-c-first', stop reason = signal SIGSEGV: invalid address (fault address: 0x186e8)
  * frame #0: 0x00000000002a4b8d test-c-first`memcpy at memmove.S:306
    frame #1: 0x00000000002a2b53 test-c-first`__sfvwrite(fp=<unavailable>, uio=0x00007fffffffe888) at fvwrite.c:188:5
    frame #2: 0x0000000000285442 test-c-first`__vfprintf [inlined] __sprint(fp=<unavailable>, uio=<unavailable>, locale=<unavailable>) at vfprintf.c:166:8
    frame #3: 0x0000000000285436 test-c-first`__vfprintf [inlined] io_flush(iop=<unavailable>, locale=<unavailable>) at printfcommon.h:157
    frame #4: 0x000000000028541f test-c-first`__vfprintf(fp=0x00000000002b7ff8, locale=0x00000000002b6bf8, fmt0=<unavailable>, ap=0x00007fffffffeb10) at vfprintf.c:1033
    frame #5: 0x0000000000281ebd test-c-first`vfprintf_l(fp=0x00000000002b7ff8, locale=0x00000000002b6bf8, fmt0="result = %p\n", ap=0x00007fffffffeb10) at vfprintf.c:285:9
    frame #6: 0x00000000002a57ba test-c-first`printf(fmt=<unavailable>) at printf.c:57:8
    frame #7: 0x00000000002a5707 test-c-first`main + 119
    frame #8: 0x000000000022510f test-c-first`_start(ap=<unavailable>, cleanup=<unavailable>) at crt1.c:76:7

Indeed strange. I cannot reproduce it locally. For me it is just

x% ./test-c-first                     /usr/home/pooma/build/bsd/DEV/stuff/tests
pthread_create = 0x2a58b0
x%

Can you build libc with debugging info and lower optimization level (DEBUG_FLAGS=-g should be enough), and then print out the stdout FILE state ? I suspect there is some problem there.

I am now getting a very strange error:

root@qemu-native-alr48:~ # clang -static -o test-thr-first -nodefaultlibs -lthr -lc test.c -fuse-ld=lld && ./test-thr-first
pthread_create = 0x2693d0
result = 0x1234
root@qemu-native-alr48:~ # clang -static -o test-c-first -nodefaultlibs -lc -lthr test.c -fuse-ld=lld && ./test-c-first
pthread_create = 0x2a58a0
Segmentation fault (core dumped)
root@qemu-native-alr48:~ # clang -static -o test-c-first -nodefaultlibs -lc -lthr test.c -fuse-ld=lld && lldb -- ./test-c-first
(lldb) target create "./test-c-first"
Current executable set to './test-c-first' (x86_64).
(lldb) r
Process 746 launching
Process 746 launched: '/root/test-c-first' (x86_64)
pthread_create = 0x2a58a0
Process 746 stopped
* thread #1, name = 'test-c-first', stop reason = signal SIGSEGV: invalid address (fault address: 0x186e8)
    frame #0: 0x00000000002a4b8d test-c-first`memcpy at memmove.S:306
(lldb) bt
* thread #1, name = 'test-c-first', stop reason = signal SIGSEGV: invalid address (fault address: 0x186e8)
  * frame #0: 0x00000000002a4b8d test-c-first`memcpy at memmove.S:306
    frame #1: 0x00000000002a2b53 test-c-first`__sfvwrite(fp=<unavailable>, uio=0x00007fffffffe888) at fvwrite.c:188:5
    frame #2: 0x0000000000285442 test-c-first`__vfprintf [inlined] __sprint(fp=<unavailable>, uio=<unavailable>, locale=<unavailable>) at vfprintf.c:166:8
    frame #3: 0x0000000000285436 test-c-first`__vfprintf [inlined] io_flush(iop=<unavailable>, locale=<unavailable>) at printfcommon.h:157
    frame #4: 0x000000000028541f test-c-first`__vfprintf(fp=0x00000000002b7ff8, locale=0x00000000002b6bf8, fmt0=<unavailable>, ap=0x00007fffffffeb10) at vfprintf.c:1033
    frame #5: 0x0000000000281ebd test-c-first`vfprintf_l(fp=0x00000000002b7ff8, locale=0x00000000002b6bf8, fmt0="result = %p\n", ap=0x00007fffffffeb10) at vfprintf.c:285:9
    frame #6: 0x00000000002a57ba test-c-first`printf(fmt=<unavailable>) at printf.c:57:8
    frame #7: 0x00000000002a5707 test-c-first`main + 119
    frame #8: 0x000000000022510f test-c-first`_start(ap=<unavailable>, cleanup=<unavailable>) at crt1.c:76:7

Approach is definitely viable. However, if we need to update all functions a macro is probably a good idea.

Seems like it will still break for all other functions that aren't updated?

However, I just noticed that the following test case still results in an infinite loop/recursion even with this patch. I prints the address of pthread_create, but then freezes. Guess we need to do the same trick for even more functions.

#include <stdio.h>
#include <pthread.h>

Thanks! I can confirm that it works for the testcase from the PR if I apply the suggested fix.

I'm not sure why I made that change. I just remember that we had some compiler bugs in that function that went away when I added a printf. I think this was just one of the suspicious things that I saw so I made the change hoping that it would fix the miscompilation.

@bdrewery is this okay to commit now?

lld has been fixed.

In D20663#449410, @kib wrote:

In D20663#449383, @arichardson wrote:

In D20663#449349, @kib wrote:

In D20663#449347, @arichardson wrote:

In D20663#449299, @kib wrote:

What testing was done ? Did you boot amd64 multiuser with the patched rtld ?
libthr consumes some bits from rtld, mostly rtld_malloc.c.

libthr still builds fine (I did a full universe build).

Did you do any runtime testing of threaded apps ?

I did not try any threaded applications yet. Do you have any suggestions what would be a good test case?

Perhaps run libc and libthr tests from /usr/tests ...

For testing I booted MIPS64 multiuser with these changes and ran a few programs with rtld direct exec on amd64. I've also been running CheriBSD with these changes for a few weeks.

I believe that for such changes, a multiuser boot on amd64 (or i386) is required.

Okay, I'll report back once I've booted on amd64.

... on amd64.

Update comment in makefile.inc

In D20663#449349, @kib wrote:

In D20663#449347, @arichardson wrote:

In D20663#449299, @kib wrote:

What testing was done ? Did you boot amd64 multiuser with the patched rtld ?
libthr consumes some bits from rtld, mostly rtld_malloc.c.

libthr still builds fine (I did a full universe build).

Did you do any runtime testing of threaded apps ?

In D20663#449299, @kib wrote:

What testing was done ? Did you boot amd64 multiuser with the patched rtld ?
libthr consumes some bits from rtld, mostly rtld_malloc.c.

fix a style issue and expand comment on how we avoid the use of __libc_interposing

Remove the unncessary errlst.h header

Fix build for non-x86 and non-MIPS64 architectures

Extract the required .o files from libc_nossp_pic
Size reduction is now 22% instead of 18%

In D20663#448177, @kib wrote:

In D20663#448176, @arichardson wrote:

Okay, I'll try extracting the required files from libc_nossp_pic.a using ar and link against those files. Is that what you were suggesting? I would prefer building all the .o files, but I think there might be some per-architecture assembly files that need to be included and adding that logic to the makefile is slightly annoying.

Ok for me, but I wonder how would it work. Basically, to avoid the interposing code, you need to use _xxx or __sys_xxx symbols directly, right ? Then if you do, wouldn't simply linking with libc_nossp_pic.a enough ?

In D20663#448175, @kib wrote:

In D20663#448150, @arichardson wrote:

In D20663#448138, @kib wrote:

I always wanted to remove -lc (or any currently used variant of libc) from the rtld linking. Ideally, we would just compile required libc sources second time for rtld, inside libexec/rtld-elf. I am sure that this would be a lot of work, but I am not sure that it would be much more work than continuing the approach of patching libc.a to cut off unneeded pieces.

There are not actually that many files from libc that are used in rtld. Here is the -Wl,--trace output:
...
I think if we linked against a libsyscalls (D14609) we only really need to build the string-related files (plus a few more which seem to mostly come from getenv()) again.

If there are not that many files, I would prefer to directly add them to the build (or just add .o files from libc nossp pic) instead of adding hackery to cut libc internal mechanisms. Mostly because that adding interposable symbol to libc becomes even more tricky, with no clear indication to the author that he forget or not aware about rtld.
I know about libsyscall, but the existing review seems to be not reviewable, and a way to get that in is to re-do it from scratch.

• Address feedback
• Only enable link error logic with RTLD_VERIFY_LIBC_DEPENDENCIES=1 since it otherwise builds many unnecessary .o files (this can go away when we stop linking against libc).

In D20663#448138, @kib wrote:

I always wanted to remove -lc (or any currently used variant of libc) from the rtld linking. Ideally, we would just compile required libc sources second time for rtld, inside libexec/rtld-elf. I am sure that this would be a lot of work, but I am not sure that it would be much more work than continuing the approach of patching libc.a to cut off unneeded pieces.

In D20533#443680, @emaste wrote:

Perhaps we should put a comment at the beginning of this file explaining that it's not a perfect match for style(9) and providing an example or two of how it can be used?
I had a much less comprehensive .clang-format and used it like so:

clang-format file.c > formatted.c
kdiff3 file.c formatted.c &

and then just edited file.c to manually apply appropriate changes.
The .clang-format here is much better than what I had and gets quite close. Some changes it made and some comments on those changes below, but my overall comment: this is indeed already useful as is for work I have in progress.

 struct glyph {
-       TAILQ_ENTRY(glyph)       g_list;
-       SLIST_ENTRY(glyph)       g_hash;
-       uint8_t                 *g_data;
-       unsigned int             g_index;
+       TAILQ_ENTRY(glyph) g_list;
+       SLIST_ENTRY(glyph) g_hash;
+       uint8_t *g_data;
+       unsigned int g_index;
 };

We have a mix of lining up and not lining up variable names in the tree; examples in style(9) have them lined up.

In D18291#438265, @emaste wrote:

In D18291#438263, @arichardson wrote:

In D18291#438254, @jhb wrote:

I'm pretty sure we use 4byte alignment (.balign 4) to ensure its aligned to one instruction.

My argument for just removing it is that if we're linking in an object file with an .init or .fini containing part of an instruction we're going to fail at startup (or shutdown) anyway so it doesn't matter. In cases where the objects are well-formed the alignment directive is redundant, and other architectures (except sparc64) don't specify it so MIPS ought not as well.

I agree, dropping it should be completely fine since the individual files should implicitly have alignment 4. If not then the compiler is doing something wrong.

In D18291#438254, @jhb wrote:

Did you test this? In CheriBSD we are definitely using the equivalent of .align 16. MIPS instructions are only 4-bytes, so instruction alignment as described in your latest comment would seem to imply that '.align 4' was redundant?

LGTM.

Lgtm but I would wait for @bdrewery or @emaste to also review.

In D16868#415223, @bdrewery wrote:

In D16868#415222, @bdrewery wrote:

In D16868#415167, @bdrewery wrote:

In D16868#415150, @bdrewery wrote:

Has this turned up any bad pipelines in Makefiles?

Ah right there is that dd one from a while back.
Now that pipefail is in we could do this a lot simpler like tools/install.sh works.
I don't think it needs to be a MK_ option.
It doesn't need to install.
Maybe something like this could work in tools/sh.sh that we just set with __MAKE_SHELL=?

#! /bin/sh
set -o pipefail 2>/dev/null || :
exec "$@"

Hmm set -o pipefail || : doesn't actually ignore the error.

# sh -ec 'set -o pipefail || :; echo test'
set: Illegal option -o pipefail

:(

This works and avoids a fork and error.

x=$(set -o); case $x in *pipefail*) set -o pipefail ;; esac

I'm trying to avoid adding fork/exec/grep overhead to a sh_pipefail.sh script since it will be executed a lot.

Ping?

Also adjust make_dtbo.sh

It worked afterward I made this change but I'm not 100% certain that it was because of this patch (not using META_MODE).

Rebased and fixed typo in info message

ping?

Looks like we also need a newer version of libcasper.h in -legacy to build on 11

Avoid warning message if /usr/include/capsicum_helpers.h doesn't exist

In D16842#381595, @jhb wrote:

In D16842#381593, @bdrewery wrote:

In D16842#359095, @jhb wrote:

Agreed that as a followup we should define a single variable in src/Makefile.inc1 to hold 'freebsd12.0'. Will be useful since we are about to bump it to 13 in a few weeks.

It's called TARGET_TRIPLE

I feel like TARGET_TRIPLE alone doesn't work. For example for mips I have to use 'mips-unknown-freebsd12.0' for lib32 when TARGET_TRIPLE is 'mips64-unknown-freebsd12.0'.

In D16842#381591, @bdrewery wrote:

I don't get emails for phabricator emails.
I do get and respond to IRC usually instantly.
Please get my feedback for further build changes.