- Queries
- All Stories
- Search
- Advanced Search
- Transactions
- Transaction Logs
Advanced Search
Nov 20 2019
Nov 19 2019
Nov 17 2019
Nov 15 2019
In D17002#489675, @jhb wrote:Does this work with GCC? Seems no:
> mips-unknown-freebsd12.1-gcc -print-resource-dir mips-unknown-freebsd12.1-gcc: error: unrecognized command line option '-print-resource-dir'; did you mean '-print-search-dirs'? mips-unknown-freebsd12.1-gcc: fatal error: no input files compilation terminated.
This change is required to build with an unmodified upstream LLVM since the default upstream clang builtin headers are incompatible with the ones in /usr/bin/include (see my earlier comments in this review).
ping
Disable all flags except -l/-u when bootstrapping
Oct 10 2019
In D21971#480086, @brooks wrote:In D21971#480075, @arichardson wrote:I wish the C standard library had some kind of dynamic array so that this is not reimplement in every program...
I wonder if we could use something like the utarray.h macros from uthash?
Could we import the five headers that https://github.com/troydhanson/uthash into contrib and use utarray instead? I know the library is not that efficient since it uses strdup() for all strings that you insert but at least it means not reinventing the wheel.
We already have two copies of uthash.h in the source tree and elftoolchain also includes with utarray.h.We certainly could use utarray instead, though I'm not super convinced by its readability.
(Importing uthash.h into contrib is on my todo list since I'd like to merge my changes to use it instead of bdb for hashes in init and a couple other places.)
I wish the C standard library had some kind of dynamic array so that this is not reimplement in every program...
Oct 2 2019
Sep 26 2019
If prefer to just spell out the assembly rather than just the magic cp* macros.
Apparently they place a $gp reload after every call: Additionally, it causes the assembler to emit:
Sep 25 2019
Looks good. Although I'm not sure if we want this to be conditional on clang/lld since modern GCC+bfd should work (and maybe even clang+new bfd).
Sep 18 2019
In D21699#473438, @arichardson wrote:
Looks good to me but I'd wait for @jhb to review it too.
I guess this happens because some files have more static globals than can be accessed in the immediate. In general lld should be able to use the multi-GOT trick but in that case it won't work.
If we really care about performance we could track down the individual files that need mxgot and only add it there but asking it globally should be fine.
Sep 17 2019
I believe that splitting the operation into two asm blocks with C code between will break at -O0 since any variable access goes via the stack. This might cause the sc to fail.
Sep 11 2019
Aug 29 2019
LGTM but I also haven't tested it. I don't know how useful the profile stuff is. If it's not implemented for MIPS64 anyway we could just not build it?
Aug 15 2019
Jul 30 2019
Just found one more issue but otherwise LGTM. Maybe better to consistently use the _Tthr prefix.
I can confirm that it works when I apply the following change on top of your patch:
diff --git a/lib/libthr/thread/thr_init.c b/lib/libthr/thread/thr_init.c index d9bde8a37c9d..78146c2d6dcd 100644 --- a/lib/libthr/thread/thr_init.c +++ b/lib/libthr/thread/thr_init.c @@ -257,7 +257,7 @@ static pthread_func_t jmp_table[][2] = { [PJT_RWLOCK_TRYWRLOCK] = {DUAL_ENTRY(_Tthr_rwlock_trywrlock)}, [PJT_RWLOCK_UNLOCK] = {DUAL_ENTRY(_Tthr_rwlock_unlock)}, [PJT_RWLOCK_WRLOCK] = {DUAL_ENTRY(_Tthr_rwlock_wrlock)}, - [PJT_SELF] = {DUAL_ENTRY(_thr_self)}, + [PJT_SELF] = {DUAL_ENTRY(_Tthr_self)}, [PJT_SETCANCELSTATE] = {DUAL_ENTRY(_thr_setcancelstate)}, [PJT_SETCANCELTYPE] = {DUAL_ENTRY(_thr_setcanceltype)}, [PJT_SETSPECIFIC] = {DUAL_ENTRY(_thr_setspecific)}, diff --git a/lib/libthr/thread/thr_private.h b/lib/libthr/thread/thr_private.h index 90385d1ab31c..47b82009765d 100644 --- a/lib/libthr/thread/thr_private.h +++ b/lib/libthr/thread/thr_private.h @@ -1069,7 +1069,7 @@ int _thr_setspecific(pthread_key_t, const void *); void *_thr_getspecific(pthread_key_t); int _thr_setcancelstate(int, int *); int _thr_setcanceltype(int, int *); -pthread_t _thr_self(void); +pthread_t _Tthr_self(void); int _thr_rwlock_init(pthread_rwlock_t *, const pthread_rwlockattr_t *); int _thr_rwlock_destroy(pthread_rwlock_t *); int _Tthr_rwlock_rdlock(pthread_rwlock_t *); diff --git a/lib/libthr/thread/thr_self.c b/lib/libthr/thread/thr_self.c index 168d91721e0b..68bf5bc39312 100644 --- a/lib/libthr/thread/thr_self.c +++ b/lib/libthr/thread/thr_self.c @@ -38,11 +38,11 @@ __FBSDID("$FreeBSD$");
Some more information:
(lldb) register read General Purpose Registers: rax = 0x00000000000001b0 rbx = 0x0000000000000001 rcx = 0x00007fffffffeb10 rdx = 0x00000000002050fd rdi = 0x000000000031d2e8 __sF + 312 rsi = 0x000000000031bec0 __xlocale_global_locale rbp = 0x00007fffffffe890 rsp = 0x00007fffffffe878 r8 = 0x00007fffffffeb10 r9 = 0x00007fffffffeb10 r10 = 0x00007fffffffeb10 r11 = 0x0000000000000201 r12 = 0x00000000002002e0 r13 = 0x00000000002002e0 r14 = 0x00007fffffffebd8 r15 = 0x00007fffffffebe8 rip = 0x00000000002fd4b8 test-c-first`__sys_thr_self + 8 rflags = 0x0000000000000202 cs = 0x0000000000000043
With -O0 at the time of the crash:
(lldb) p *__stdoutp (FILE) $1 = { _p = 0x00000000000186e8 "" _r = 0 _w = 0 _flags = 2185 _file = 1 _bf = (_base = "pthread_create = 0x301e10\n\xa5\xa5\xa5\xa5\xa5\xa5\xa5\xa5\xa5\xa5\xa5\xa5\xa5\xa5\xa5\xa5\xa5\xa5\xa5\xa5\xa5\xa5\xa5\xa5\xa5\xa5\xa5\xa5\xa5\xa5\xa5) _lbfsize = -4096 _cookie = 0x000000000031d2e8 _close = 0x00000000002fe580 (test-c-first`__sclose at stdio.c:80) _read = 0x00000000002fe4c0 (test-c-first`__sread at stdio.c:56) _seek = 0x00000000002fe540 (test-c-first`__sseek at stdio.c:72) _write = 0x00000000002fe500 (test-c-first`__swrite at stdio.c:64) _ub = (_base = 0x0000000000000000, _size = 0) _up = 0x0000000000000000 _ur = 0 _ubuf = "" _nbuf = "" _lb = (_base = 0x0000000000000000, _size = 0) _blksize = 4096 _offset = 0 _fl_mutex = 0x0000000000000000 _fl_owner = 0x0000000000000000 _fl_count = 1 _orientation = 0 _mbstate = (__mbstate8 = "", _mbstateL = 0) _flags2 = 0 } (lldb) p __stdoutp->_bf._base (unsigned char *) $4 = 0x000000080033b000 "pthread_create = 0x301e10\n\xa5\xa5\xa5\xa5\xa5\xa5\xa5\xa5\xa5\xa5\xa5\xa5\xa5\xa5\xa5\xa5\xa5\xa5\xa5\xa5\xa5\xa5\xa5\xa5\xa. (lldb) p __stdoutp->_bf._size (int) $5 = 4096
Jul 29 2019
Just to be clear this only happens if I actually start a thread and call pthread_join (using the program I posted here). The simple case from the PR works fine.
In D21088#458174, @kib wrote:In D21088#457869, @arichardson wrote:I am now getting a very strange error:
root@qemu-native-alr48:~ # clang -static -o test-thr-first -nodefaultlibs -lthr -lc test.c -fuse-ld=lld && ./test-thr-first pthread_create = 0x2693d0 result = 0x1234 root@qemu-native-alr48:~ # clang -static -o test-c-first -nodefaultlibs -lc -lthr test.c -fuse-ld=lld && ./test-c-first pthread_create = 0x2a58a0 Segmentation fault (core dumped) root@qemu-native-alr48:~ # clang -static -o test-c-first -nodefaultlibs -lc -lthr test.c -fuse-ld=lld && lldb -- ./test-c-first (lldb) target create "./test-c-first" Current executable set to './test-c-first' (x86_64). (lldb) r Process 746 launching Process 746 launched: '/root/test-c-first' (x86_64) pthread_create = 0x2a58a0 Process 746 stopped * thread #1, name = 'test-c-first', stop reason = signal SIGSEGV: invalid address (fault address: 0x186e8) frame #0: 0x00000000002a4b8d test-c-first`memcpy at memmove.S:306 (lldb) bt * thread #1, name = 'test-c-first', stop reason = signal SIGSEGV: invalid address (fault address: 0x186e8) * frame #0: 0x00000000002a4b8d test-c-first`memcpy at memmove.S:306 frame #1: 0x00000000002a2b53 test-c-first`__sfvwrite(fp=<unavailable>, uio=0x00007fffffffe888) at fvwrite.c:188:5 frame #2: 0x0000000000285442 test-c-first`__vfprintf [inlined] __sprint(fp=<unavailable>, uio=<unavailable>, locale=<unavailable>) at vfprintf.c:166:8 frame #3: 0x0000000000285436 test-c-first`__vfprintf [inlined] io_flush(iop=<unavailable>, locale=<unavailable>) at printfcommon.h:157 frame #4: 0x000000000028541f test-c-first`__vfprintf(fp=0x00000000002b7ff8, locale=0x00000000002b6bf8, fmt0=<unavailable>, ap=0x00007fffffffeb10) at vfprintf.c:1033 frame #5: 0x0000000000281ebd test-c-first`vfprintf_l(fp=0x00000000002b7ff8, locale=0x00000000002b6bf8, fmt0="result = %p\n", ap=0x00007fffffffeb10) at vfprintf.c:285:9 frame #6: 0x00000000002a57ba test-c-first`printf(fmt=<unavailable>) at printf.c:57:8 frame #7: 0x00000000002a5707 test-c-first`main + 119 frame #8: 0x000000000022510f test-c-first`_start(ap=<unavailable>, cleanup=<unavailable>) at crt1.c:76:7Indeed strange. I cannot reproduce it locally. For me it is just
x% ./test-c-first /usr/home/pooma/build/bsd/DEV/stuff/tests pthread_create = 0x2a58b0 x%Can you build libc with debugging info and lower optimization level (DEBUG_FLAGS=-g should be enough), and then print out the stdout FILE state ? I suspect there is some problem there.
Jul 28 2019
I am now getting a very strange error:
root@qemu-native-alr48:~ # clang -static -o test-thr-first -nodefaultlibs -lthr -lc test.c -fuse-ld=lld && ./test-thr-first pthread_create = 0x2693d0 result = 0x1234 root@qemu-native-alr48:~ # clang -static -o test-c-first -nodefaultlibs -lc -lthr test.c -fuse-ld=lld && ./test-c-first pthread_create = 0x2a58a0 Segmentation fault (core dumped) root@qemu-native-alr48:~ # clang -static -o test-c-first -nodefaultlibs -lc -lthr test.c -fuse-ld=lld && lldb -- ./test-c-first (lldb) target create "./test-c-first" Current executable set to './test-c-first' (x86_64). (lldb) r Process 746 launching Process 746 launched: '/root/test-c-first' (x86_64) pthread_create = 0x2a58a0 Process 746 stopped * thread #1, name = 'test-c-first', stop reason = signal SIGSEGV: invalid address (fault address: 0x186e8) frame #0: 0x00000000002a4b8d test-c-first`memcpy at memmove.S:306 (lldb) bt * thread #1, name = 'test-c-first', stop reason = signal SIGSEGV: invalid address (fault address: 0x186e8) * frame #0: 0x00000000002a4b8d test-c-first`memcpy at memmove.S:306 frame #1: 0x00000000002a2b53 test-c-first`__sfvwrite(fp=<unavailable>, uio=0x00007fffffffe888) at fvwrite.c:188:5 frame #2: 0x0000000000285442 test-c-first`__vfprintf [inlined] __sprint(fp=<unavailable>, uio=<unavailable>, locale=<unavailable>) at vfprintf.c:166:8 frame #3: 0x0000000000285436 test-c-first`__vfprintf [inlined] io_flush(iop=<unavailable>, locale=<unavailable>) at printfcommon.h:157 frame #4: 0x000000000028541f test-c-first`__vfprintf(fp=0x00000000002b7ff8, locale=0x00000000002b6bf8, fmt0=<unavailable>, ap=0x00007fffffffeb10) at vfprintf.c:1033 frame #5: 0x0000000000281ebd test-c-first`vfprintf_l(fp=0x00000000002b7ff8, locale=0x00000000002b6bf8, fmt0="result = %p\n", ap=0x00007fffffffeb10) at vfprintf.c:285:9 frame #6: 0x00000000002a57ba test-c-first`printf(fmt=<unavailable>) at printf.c:57:8 frame #7: 0x00000000002a5707 test-c-first`main + 119 frame #8: 0x000000000022510f test-c-first`_start(ap=<unavailable>, cleanup=<unavailable>) at crt1.c:76:7
Jul 27 2019
Approach is definitely viable. However, if we need to update all functions a macro is probably a good idea.
Seems like it will still break for all other functions that aren't updated?
However, I just noticed that the following test case still results in an infinite loop/recursion even with this patch. I prints the address of pthread_create, but then freezes. Guess we need to do the same trick for even more functions.
#include <stdio.h> #include <pthread.h>
Thanks! I can confirm that it works for the testcase from the PR if I apply the suggested fix.
Jul 19 2019
I'm not sure why I made that change. I just remember that we had some compiler bugs in that function that went away when I added a printf. I think this was just one of the suspicious things that I saw so I made the change hoping that it would fix the miscompilation.
Jul 2 2019
Jul 1 2019
@bdrewery is this okay to commit now?
lld has been fixed.
Jun 30 2019
Jun 28 2019
In D20663#449410, @kib wrote:In D20663#449383, @arichardson wrote:In D20663#449349, @kib wrote:In D20663#449347, @arichardson wrote:In D20663#449299, @kib wrote:What testing was done ? Did you boot amd64 multiuser with the patched rtld ?
libthr consumes some bits from rtld, mostly rtld_malloc.c.
libthr still builds fine (I did a full universe build).
Did you do any runtime testing of threaded apps ?
I did not try any threaded applications yet. Do you have any suggestions what would be a good test case?
Perhaps run libc and libthr tests from /usr/tests ...
For testing I booted MIPS64 multiuser with these changes and ran a few programs with rtld direct exec on amd64. I've also been running CheriBSD with these changes for a few weeks.
I believe that for such changes, a multiuser boot on amd64 (or i386) is required.
Okay, I'll report back once I've booted on amd64.
... on amd64.
Update comment in makefile.inc
Jun 26 2019
In D20663#449349, @kib wrote:In D20663#449347, @arichardson wrote:In D20663#449299, @kib wrote:What testing was done ? Did you boot amd64 multiuser with the patched rtld ?
libthr consumes some bits from rtld, mostly rtld_malloc.c.
libthr still builds fine (I did a full universe build).
Did you do any runtime testing of threaded apps ?
In D20663#449299, @kib wrote:What testing was done ? Did you boot amd64 multiuser with the patched rtld ?
libthr consumes some bits from rtld, mostly rtld_malloc.c.
fix a style issue and expand comment on how we avoid the use of __libc_interposing
Remove the unncessary errlst.h header
Fix build for non-x86 and non-MIPS64 architectures
Jun 23 2019
Extract the required .o files from libc_nossp_pic
Size reduction is now 22% instead of 18%
In D20663#448177, @kib wrote:In D20663#448176, @arichardson wrote:Okay, I'll try extracting the required files from libc_nossp_pic.a using ar and link against those files. Is that what you were suggesting? I would prefer building all the .o files, but I think there might be some per-architecture assembly files that need to be included and adding that logic to the makefile is slightly annoying.
Ok for me, but I wonder how would it work. Basically, to avoid the interposing code, you need to use _xxx or __sys_xxx symbols directly, right ? Then if you do, wouldn't simply linking with libc_nossp_pic.a enough ?
In D20663#448175, @kib wrote:In D20663#448150, @arichardson wrote:In D20663#448138, @kib wrote:I always wanted to remove -lc (or any currently used variant of libc) from the rtld linking. Ideally, we would just compile required libc sources second time for rtld, inside libexec/rtld-elf. I am sure that this would be a lot of work, but I am not sure that it would be much more work than continuing the approach of patching libc.a to cut off unneeded pieces.
There are not actually that many files from libc that are used in rtld. Here is the -Wl,--trace output:
...
I think if we linked against a libsyscalls (D14609) we only really need to build the string-related files (plus a few more which seem to mostly come from getenv()) again.If there are not that many files, I would prefer to directly add them to the build (or just add .o files from libc nossp pic) instead of adding hackery to cut libc internal mechanisms. Mostly because that adding interposable symbol to libc becomes even more tricky, with no clear indication to the author that he forget or not aware about rtld.
I know about libsyscall, but the existing review seems to be not reviewable, and a way to get that in is to re-do it from scratch.
- Address feedback
- Only enable link error logic with RTLD_VERIFY_LIBC_DEPENDENCIES=1 since it otherwise builds many unnecessary .o files (this can go away when we stop linking against libc).
In D20663#448138, @kib wrote:I always wanted to remove -lc (or any currently used variant of libc) from the rtld linking. Ideally, we would just compile required libc sources second time for rtld, inside libexec/rtld-elf. I am sure that this would be a lot of work, but I am not sure that it would be much more work than continuing the approach of patching libc.a to cut off unneeded pieces.
Jun 16 2019
Jun 7 2019
In D20533#443680, @emaste wrote:Perhaps we should put a comment at the beginning of this file explaining that it's not a perfect match for style(9) and providing an example or two of how it can be used?
I had a much less comprehensive .clang-format and used it like so:
clang-format file.c > formatted.c kdiff3 file.c formatted.c &and then just edited file.c to manually apply appropriate changes.
The .clang-format here is much better than what I had and gets quite close. Some changes it made and some comments on those changes below, but my overall comment: this is indeed already useful as is for work I have in progress.
struct glyph { - TAILQ_ENTRY(glyph) g_list; - SLIST_ENTRY(glyph) g_hash; - uint8_t *g_data; - unsigned int g_index; + TAILQ_ENTRY(glyph) g_list; + SLIST_ENTRY(glyph) g_hash; + uint8_t *g_data; + unsigned int g_index; };We have a mix of lining up and not lining up variable names in the tree; examples in style(9) have them lined up.
Jun 6 2019
May 20 2019
In D18291#438265, @emaste wrote:In D18291#438263, @arichardson wrote:In D18291#438254, @jhb wrote:I'm pretty sure we use 4byte alignment (.balign 4) to ensure its aligned to one instruction.
My argument for just removing it is that if we're linking in an object file with an .init or .fini containing part of an instruction we're going to fail at startup (or shutdown) anyway so it doesn't matter. In cases where the objects are well-formed the alignment directive is redundant, and other architectures (except sparc64) don't specify it so MIPS ought not as well.
I agree, dropping it should be completely fine since the individual files should implicitly have alignment 4. If not then the compiler is doing something wrong.
In D18291#438254, @jhb wrote:Did you test this? In CheriBSD we are definitely using the equivalent of .align 16. MIPS instructions are only 4-bytes, so instruction alignment as described in your latest comment would seem to imply that '.align 4' was redundant?
Apr 9 2019
Apr 8 2019
Mar 23 2019
Feb 28 2019
In D16868#415223, @bdrewery wrote:In D16868#415222, @bdrewery wrote:In D16868#415167, @bdrewery wrote:In D16868#415150, @bdrewery wrote:Has this turned up any bad pipelines in Makefiles?
Ah right there is that dd one from a while back.
Now that pipefail is in we could do this a lot simpler like tools/install.sh works.
I don't think it needs to be a MK_ option.
It doesn't need to install.
Maybe something like this could work in tools/sh.sh that we just set with __MAKE_SHELL=?#! /bin/sh set -o pipefail 2>/dev/null || : exec "$@"Hmm set -o pipefail || : doesn't actually ignore the error.
# sh -ec 'set -o pipefail || :; echo test' set: Illegal option -o pipefail:(
This works and avoids a fork and error.
x=$(set -o); case $x in *pipefail*) set -o pipefail ;; esacI'm trying to avoid adding fork/exec/grep overhead to a sh_pipefail.sh script since it will be executed a lot.