jemalloc 5.3.0 was released in 2022 but has not been imported into FreeBSD base system yet. The new version of jemalloc includes new features and bug fixes which are listed on release note.
Details
Details
Diff Detail
Diff Detail
- Repository
- rG FreeBSD src repository
- Lint
Lint Passed - Unit
No Test Coverage - Build Status
Buildable 54744 Build 51633: arc lint + arc unit
Event Timeline
There are a very large number of changes, so older changes are hidden. Show Older Changes
Comment Actions
I did some test on 32-bit FreeBSD and I found out that when clock_gettime is used on 64bit machine but compiled in 32bit, it sets tv_sec (time_t) as -1. On 32bit FreeBSD, clock_gettime works as expected.
Comment Actions
that seems like it's a bug. On powerpc and amrv7 tv_sec should be the same. On i386, tv_sec is only32-bit, but we still have a dozen years left until it's inadequate. Can you finle a bug, if you haven't already, with a way to reproduce this?
Comment Actions
Here is a bug report which includes an explanation and a way to reproduce the problem.
Comment Actions
I still get error
cd /usr/src; _PARALLEL_SUBDIR_OK=1 time env MACHINE_ARCH=amd64 MACHINE=amd64 CPUTYPE= BUILD_TOOLS_META=.NOMETA CC="cc -target x86_64-unknown-freebsd14.0 --sysroot=/usr/obj/usr/src/amd64.amd64/tmp -B/usr/obj/usr/src/amd64.amd64/tmp/usr/bin" CXX="c++ -target x86_64-unknown-freebsd14.0 --sysroot=/usr/obj/usr/src/amd64.amd64/tmp -B/usr/obj/usr/src/amd64.amd64/tmp/usr/bin" CPP="cpp -target x86_64-unknown-freebsd14.0 --sysroot=/usr/obj/usr/src/amd64.amd64/tmp -B/usr/obj/usr/src/amd64.amd64/tmp/usr/bin" AS="as" AR="ar" ELFCTL="elfctl" LD="ld" LLVM_LINK="" NM=nm OBJCOPY="objcopy" RANLIB=ranlib STRINGS= SIZE="size" STRIPBIN="strip" INSTALL="install -U" PATH=/usr/obj/usr/src/amd64.amd64/tmp/bin:/usr/obj/usr/src/amd64.amd64/tmp/usr/sbin:/usr/obj/usr/src/amd64.amd64/tmp/usr/bin:/usr/obj/usr/src/amd64.amd64/tmp/legacy/usr/sbin:/usr/obj/usr/src/amd64.amd64/tmp/legacy/usr/bin:/usr/obj/usr/src/amd64.amd64/tmp/legacy/bin:/usr/obj/usr/src/amd64.amd64/tmp/legacy/usr/libexec::/sbin:/bin:/usr/sbin:/usr/bin SYSROOT=/usr/obj/usr/src/amd64.amd64/tmp make -f Makefile.inc1 BWPHASE=everything DESTDIR=/usr/obj/usr/src/amd64.amd64/tmp all (cd /usr/src/lib/csu/tests/dynamic && DEPENDFILE=.depend.init_test NO_SUBDIR=1 make -f /usr/src/lib/csu/tests/dynamic/Makefile _RECURSING_PROGS=t PROG=init_test ) Building /usr/obj/usr/src/amd64.amd64/lib/csu/tests/dynamic/init_test.o Building /usr/obj/usr/src/amd64.amd64/lib/csu/tests/dynamic/init_test.full ld: error: undefined reference due to --no-allow-shlib-undefined: _pthread_getname_np >>> referenced by /usr/obj/usr/src/amd64.amd64/tmp/lib/libc.so.7 cc: error: linker command failed with exit code 1 (use -v to see invocation) *** Error code 1 Stop.
Why doesn't the build system accept _pthread_getname_np but accepts other _pthread_*s like _pthread_join? I already tried adding _pthread_getname_np to FBSDprivate_1.0 in lib/libthr/pthread.map and lib/libc/gen/Symbol.map but it doesn't work either.
Comment Actions
It seems extremely late in the release to be doing a malloc update.
| lib/libc/stdlib/jemalloc/Makefile.inc | ||
|---|---|---|
| 17 ↗ | (On Diff #125955) | This changes the C standard used for all of libc. It can't be part of this commit. |
| lib/libc/stdlib/jemalloc/Makefile.inc | ||
|---|---|---|
| 17 ↗ | (On Diff #125955) | jemalloc 5.3.0 uses gnu11 for build. Without CSTD= gnu11 flag, the build generates these errors: In file included from jemalloc_cache_bin.c:2:
In file included from /usr/src/contrib/jemalloc/include/jemalloc/internal/jemalloc_internal_includes.h:52:
In file included from /usr/src/contrib/jemalloc/include/jemalloc/internal/arena_structs.h:4:
In file included from /usr/src/contrib/jemalloc/include/jemalloc/internal/arena_stats.h:8:
In file included from /usr/src/contrib/jemalloc/include/jemalloc/internal/pa.h:11:
In file included from /usr/src/contrib/jemalloc/include/jemalloc/internal/pac.h:6:
/usr/src/contrib/jemalloc/include/jemalloc/internal/san_bump.h:12:25: error: redefinition of typedef 'ehooks_t' is a C11 feature [-Werror,-Wtypedef-redefinition]
typedef struct ehooks_s ehooks_t;
^
/usr/src/contrib/jemalloc/include/jemalloc/internal/ehooks.h:21:25: note: previous definition is here
typedef struct ehooks_s ehooks_t;
^
In file included from jemalloc_cache_bin.c:2:
In file included from /usr/src/contrib/jemalloc/include/jemalloc/internal/jemalloc_internal_includes.h:52:
In file included from /usr/src/contrib/jemalloc/include/jemalloc/internal/arena_structs.h:4:
In file included from /usr/src/contrib/jemalloc/include/jemalloc/internal/arena_stats.h:8:
In file included from /usr/src/contrib/jemalloc/include/jemalloc/internal/pa.h:11:
/usr/src/contrib/jemalloc/include/jemalloc/internal/pac.h:77:22: error: redefinition of typedef 'pac_t' is a C11 feature [-Werror,-Wtypedef-redefinition]
typedef struct pac_s pac_t;
^
/usr/src/contrib/jemalloc/include/jemalloc/internal/san_bump.h:13:22: note: previous definition is here
typedef struct pac_s pac_t;Therefore, I think that bumping CSTD for libc to c11 is a necessary step to adopt jemalloc 5.3.0. I tried cleanworld, buildworld, installworld and rebooted, and luckily, the system boots and works fine without any crash or error messages. In this case, should I open a new revision regarding to changing CSTD to gnu11 for libc? |
| lib/libc/stdlib/jemalloc/Makefile.inc | ||
|---|---|---|
| 17 ↗ | (On Diff #125955) | If we're going to change the C standard for libc we can't just rely on a reboot smoke test. We need to examine any new warnings generated and solicit input about that change on its own. It's probably a good thing to do, but we can't sneak it in as a part of another change. FWIW should be trivial to add ifdef/define/endif guards around the above typedefs to address the errors above as a workaround. |
| lib/libc/stdlib/jemalloc/Makefile.inc | ||
|---|---|---|
| 17 ↗ | (On Diff #125955) | .for src in ${JEMALLOCSRCS}
CFLAGS.${src}+=-std=gnu11
.endforshould work, FWIW, if needed |
Comment Actions
Created separate revision for pthread_getname_np and _pthread_getname_np. This revision depends on D41461
Comment Actions
Parent revision D41461 is closed. Now we can test this on local machines on -CURRENT.
Comment Actions
I've been testing this on main and stable/13 (with some tweaks). If there are any specific tests I can do that would be helpful let me know
Comment Actions
In the release notes, there are some new features added in jemalloc 5.3.0. For example,
- Make the behavior of realloc(ptr, 0) configurable with opt.zero_realloc
- Add the thread.idle mallctl which hints that the calling thread will be idle for a nontrivial period of time.
- Add mallctl interfaces:
- opt.zero_realloc (@davidtgoldblatt)
- opt.cache_oblivious (@interwq)
- opt.prof_leak_error (@yunxuo)
- opt.stats_interval (@interwq)
- opt.stats_interval_opts (@interwq)
- opt.tcache_max (@interwq)
- opt.trust_madvise (@azat)
- prof.prefix (@zhxchen17)
- stats.zero_reallocs (@davidtgoldblatt)
- thread.idle (@davidtgoldblatt)
- thread.peak.{read,reset} (@davidtgoldblatt)
and so on.
By the way, because this is not a major version bump (5.x.x), will this version bump be abled to be MFCed to stable/14?
Comment Actions
It would be really nice if we can test this against with the tests under /usr/tests . I know it is still very tricky to run it. If you are interested, please check the scripts at https://github.com/freebsd/freebsd-ci and let me know (and/or -testing mailing list) if you have any questions.
By the way, because this is not a major version bump (5.x.x), will this version bump be abled to be MFCed to stable/14?
Theoretically, yes, but it would be nice to check the compatibility thoroughly.
Comment Actions
I think it might make sense to split the switch to the jemalloc memalign() into a separate commit.
| lib/libc/gen/memalign.c | ||
|---|---|---|
| 38 ↗ | (On Diff #127096) | Does jemalloc's implementation handle this compatibly?.. |
| lib/libc/gen/memalign.c | ||
|---|---|---|
| 38 ↗ | (On Diff #127096) | While glibc implementation passes to malloc when alignment is 0, jemalloc implementation and illumos implementaion sets errno EINVAL and return NULL. However, from glibc man page:
glibc's implementation for align=0 was added through PR 269688, but it is not documented in glibc. Whereas, the glibc man page says it "must be a power of two". This function is also obsolete according to glibc man page and undocumented in FreeBSD man page. Therefore, I think that there isn't enough reason for maintaining compatability despite the fact that it is obsolete and undocumented, and it is okay to adopt jemalloc/illumos way to set errno to EINVAL and return NULL |
Comment Actions
@vangyzen: are you interested in reviewing this? as you did the update jemalloc to 5.2.1.
Comment Actions
Thank you for asking, but I didn't actually do the last update. @jasone did it, but had to revert it. I simply un-reverted his work.
Comment Actions
How does jemalloc 5.3.0 handle when alignment is 0
jemalloc follows the illumos way of handling alignment when it is zero, whereas the current FreeBSD and glibc returns memory space using malloc(). If we are okay with this compatability issue, this can be landed to 15-CURRENT branch. I think 12-STABLE, 13-STABLE, 14-STABLE should stick with jemalloc 5.2.1 to keep the compatability with FreeBSD's memalign.
Any ideas?
Comment Actions
IMO the only reason to support memalign is to be compatible. We should do what glibc does regardless of its official documentation.
Comment Actions
I totally agree with that point, but I'm not sure how we can implement memalign exception and maintain it in future releases of jemalloc. This might cause confusion to some users who are already using jemalloc in other projects or systems because they know FreeBSD uses jemalloc, but we're making an exception here.
If we are going make this exception, what would be the best way to implement it? Should I patch je_memalign() thorugh FREEBSD-diffs?
Comment Actions
Would you like me to keep memalign() by using the original version in memalign.c or by adding if branch in jemalloc.c 's je_memalign?
I'll comment on diff to show what I mean
| contrib/jemalloc/include/jemalloc/jemalloc.h | ||
|---|---|---|
| 32 |
I prefer this option. The trivial implementation is fine and reduces the burden on other malloc implementations. | |
Comment Actions
Thanks, I just saw it.
I think this patch generally looks good. I think we still need to do more building (for all supported arches) and testing (at least for tier 1 arches) to import.
I will try to squeeze time to work on it. If you are interested in running tests in FreeBSD's test suite with this patch, please let me know.
Comment Actions
Build Success
minsoochoo in jemalloc at freebsd on jemalloc took 34.8s ➜ make -j24 tinderbox [Creating objdir /usr/obj/usr/src/freebsd/jemalloc/amd64.amd64...] --- tinderbox --- --- universe_prologue --- -------------------------------------------------------------- >>> make universe started on Tue Apr 9 18:20:49 EDT 2024 -------------------------------------------------------------- --- universe_amd64_prologue --- --- universe_arm_prologue --- --- universe_arm64_prologue --- --- universe_i386_prologue --- --- universe_powerpc_prologue --- --- universe_riscv_prologue --- --- universe_amd64_prologue --- >> amd64 started on Tue Apr 9 18:20:49 EDT 2024 --- universe_arm64_prologue --- >> arm64 started on Tue Apr 9 18:20:49 EDT 2024 --- universe_amd64_amd64 --- --- universe_arm_prologue --- >> arm started on Tue Apr 9 18:20:49 EDT 2024 --- universe_arm64_aarch64 --- --- universe_arm_armv7 --- --- universe_powerpc_prologue --- >> powerpc started on Tue Apr 9 18:20:49 EDT 2024 --- universe_riscv_prologue --- >> riscv started on Tue Apr 9 18:20:49 EDT 2024 --- universe_i386_prologue --- >> i386 started on Tue Apr 9 18:20:49 EDT 2024 --- universe_powerpc_powerpc --- --- universe_powerpc_powerpc64 --- --- universe_powerpc_powerpc64le --- --- universe_riscv_riscv64 --- --- universe_i386_i386 --- >> arm64.aarch64 buildworld started on Tue Apr 9 18:20:49 EDT 2024 >> arm.armv7 buildworld started on Tue Apr 9 18:20:49 EDT 2024 >> amd64.amd64 buildworld started on Tue Apr 9 18:20:49 EDT 2024 >> powerpc.powerpc64 buildworld started on Tue Apr 9 18:20:49 EDT 2024 >> powerpc.powerpc buildworld started on Tue Apr 9 18:20:49 EDT 2024 >> powerpc.powerpc64le buildworld started on Tue Apr 9 18:20:49 EDT 2024 >> i386.i386 buildworld started on Tue Apr 9 18:20:49 EDT 2024 >> riscv.riscv64 buildworld started on Tue Apr 9 18:20:49 EDT 2024 --- universe_powerpc_powerpc --- >> powerpc.powerpc buildworld completed on Tue Apr 9 18:34:13 EDT 2024 --- universe_arm_armv7 --- >> arm.armv7 buildworld completed on Tue Apr 9 18:35:32 EDT 2024 --- universe_arm_kernels --- --- universe_kernels_prologue --- >> arm kernels started on Tue Apr 9 18:35:38 EDT 2024 --- universe_kernconf_arm_ARMADA38X --- >> arm.armv7 ARMADA38X kernel started on Tue Apr 9 18:35:38 EDT 2024 --- universe_kernconf_arm_GENERIC --- >> arm.armv7 GENERIC kernel started on Tue Apr 9 18:35:42 EDT 2024 --- universe_kernconf_arm_GENERIC-NODEBUG --- >> arm.armv7 GENERIC-NODEBUG kernel started on Tue Apr 9 18:35:43 EDT 2024 --- universe_kernconf_arm_LINT --- >> arm.armv7 LINT kernel started on Tue Apr 9 18:35:44 EDT 2024 --- universe_kernconf_arm_TEGRA124 --- >> arm.armv7 TEGRA124 kernel started on Tue Apr 9 18:35:45 EDT 2024 --- universe_kernconf_arm_ZEDBOARD --- >> arm.armv7 ZEDBOARD kernel started on Tue Apr 9 18:36:02 EDT 2024 --- universe_riscv_riscv64 --- >> riscv.riscv64 buildworld completed on Tue Apr 9 18:37:54 EDT 2024 --- universe_riscv_kernels --- --- universe_kernels_prologue --- >> riscv kernels started on Tue Apr 9 18:38:06 EDT 2024 --- universe_kernconf_riscv_GENERIC --- >> riscv.riscv64 GENERIC kernel started on Tue Apr 9 18:38:06 EDT 2024 --- universe_kernconf_riscv_GENERIC-NODEBUG --- >> riscv.riscv64 GENERIC-NODEBUG kernel started on Tue Apr 9 18:38:08 EDT 2024 --- universe_kernconf_riscv_LINT --- >> riscv.riscv64 LINT kernel started on Tue Apr 9 18:38:13 EDT 2024 --- universe_powerpc_powerpc64le --- >> powerpc.powerpc64le buildworld completed on Tue Apr 9 18:38:35 EDT 2024 --- universe_i386_i386 --- >> i386.i386 buildworld completed on Tue Apr 9 18:39:14 EDT 2024 --- universe_i386_kernels --- --- universe_kernels_prologue --- >> i386 kernels started on Tue Apr 9 18:39:36 EDT 2024 --- universe_kernconf_i386_GENERIC --- >> i386.i386 GENERIC kernel started on Tue Apr 9 18:39:36 EDT 2024 --- universe_kernconf_i386_GENERIC-NODEBUG --- >> i386.i386 GENERIC-NODEBUG kernel started on Tue Apr 9 18:39:36 EDT 2024 --- universe_kernconf_i386_LINT --- >> i386.i386 LINT kernel started on Tue Apr 9 18:39:39 EDT 2024 --- universe_kernconf_i386_MINIMAL --- >> i386.i386 MINIMAL kernel started on Tue Apr 9 18:39:40 EDT 2024 --- universe_kernconf_i386_PAE --- >> i386.i386 PAE kernel started on Tue Apr 9 18:39:42 EDT 2024 --- universe_arm_kernels --- >> arm.armv7 ZEDBOARD kernel completed on Tue Apr 9 18:45:38 EDT 2024 --- universe_kernconf_arm_TEGRA124 --- >> arm.armv7 TEGRA124 kernel completed on Tue Apr 9 18:45:46 EDT 2024 --- universe_kernconf_arm_ARMADA38X --- >> arm.armv7 ARMADA38X kernel completed on Tue Apr 9 18:46:23 EDT 2024 --- universe_kernconf_arm_GENERIC --- >> arm.armv7 GENERIC kernel completed on Tue Apr 9 18:46:37 EDT 2024 --- universe_kernconf_arm_GENERIC-NODEBUG --- >> arm.armv7 GENERIC-NODEBUG kernel completed on Tue Apr 9 18:46:53 EDT 2024 --- universe_kernconf_arm_LINT --- >> arm.armv7 LINT kernel completed on Tue Apr 9 18:48:44 EDT 2024 --- universe_kernels --- >> arm kernels completed on Tue Apr 9 18:48:44 EDT 2024 --- universe_arm_done --- >> arm completed on Tue Apr 9 18:48:49 EDT 2024 --- universe_riscv_kernels --- --- universe_kernconf_riscv_GENERIC --- >> riscv.riscv64 GENERIC kernel completed on Tue Apr 9 18:49:18 EDT 2024 --- universe_kernconf_riscv_GENERIC-NODEBUG --- >> riscv.riscv64 GENERIC-NODEBUG kernel completed on Tue Apr 9 18:49:27 EDT 2024 --- universe_powerpc_powerpc64 --- >> powerpc.powerpc64 buildworld completed on Tue Apr 9 18:50:07 EDT 2024 --- universe_powerpc_kernels --- --- universe_kernels_prologue --- >> powerpc kernels started on Tue Apr 9 18:50:11 EDT 2024 --- universe_kernconf_powerpc_GENERIC --- >> powerpc.powerpc GENERIC kernel started on Tue Apr 9 18:50:11 EDT 2024 --- universe_kernconf_powerpc_GENERIC-NODEBUG --- >> powerpc.powerpc GENERIC-NODEBUG kernel started on Tue Apr 9 18:50:12 EDT 2024 --- universe_kernconf_powerpc_LINT --- >> powerpc.powerpc LINT kernel started on Tue Apr 9 18:50:14 EDT 2024 --- universe_kernconf_powerpc_MPC85XX --- >> powerpc.powerpc MPC85XX kernel started on Tue Apr 9 18:50:17 EDT 2024 --- universe_kernconf_powerpc_GENERIC64 --- >> powerpc.powerpc64 GENERIC64 kernel started on Tue Apr 9 18:50:21 EDT 2024 --- universe_kernconf_powerpc_GENERIC64-NODEBUG --- >> powerpc.powerpc64 GENERIC64-NODEBUG kernel started on Tue Apr 9 18:50:31 EDT 2024 --- universe_kernconf_powerpc_LINT64 --- >> powerpc.powerpc64 LINT64 kernel started on Tue Apr 9 18:50:47 EDT 2024 --- universe_kernconf_powerpc_QORIQ64 --- >> powerpc.powerpc64 QORIQ64 kernel started on Tue Apr 9 18:50:59 EDT 2024 --- universe_kernconf_powerpc_GENERIC64LE --- >> powerpc.powerpc64le GENERIC64LE kernel started on Tue Apr 9 18:51:02 EDT 2024 --- universe_kernconf_powerpc_GENERIC64LE-NODEBUG --- >> powerpc.powerpc64le GENERIC64LE-NODEBUG kernel started on Tue Apr 9 18:51:25 EDT 2024 --- universe_i386_kernels --- --- universe_kernconf_i386_MINIMAL --- >> i386.i386 MINIMAL kernel completed on Tue Apr 9 18:51:52 EDT 2024 --- universe_powerpc_kernels --- --- universe_kernconf_powerpc_LINT64LE --- >> powerpc.powerpc64le LINT64LE kernel started on Tue Apr 9 18:54:01 EDT 2024 --- universe_riscv_kernels --- --- universe_kernconf_riscv_LINT --- >> riscv.riscv64 LINT kernel completed on Tue Apr 9 18:54:19 EDT 2024 --- universe_kernels --- >> riscv kernels completed on Tue Apr 9 18:54:19 EDT 2024 --- universe_riscv_done --- >> riscv completed on Tue Apr 9 18:54:28 EDT 2024 --- universe_i386_kernels --- --- universe_kernconf_i386_PAE --- >> i386.i386 PAE kernel completed on Tue Apr 9 18:55:15 EDT 2024 --- universe_kernconf_i386_GENERIC --- >> i386.i386 GENERIC kernel completed on Tue Apr 9 18:55:15 EDT 2024 --- universe_kernconf_i386_GENERIC-NODEBUG --- >> i386.i386 GENERIC-NODEBUG kernel completed on Tue Apr 9 18:55:33 EDT 2024 --- universe_arm64_aarch64 --- >> arm64.aarch64 buildworld completed on Tue Apr 9 18:56:03 EDT 2024 --- universe_arm64_kernels --- --- universe_kernels_prologue --- >> arm64 kernels started on Tue Apr 9 18:56:08 EDT 2024 --- universe_kernconf_arm64_GENERIC --- >> arm64.aarch64 GENERIC kernel started on Tue Apr 9 18:56:08 EDT 2024 --- universe_kernconf_arm64_GENERIC-KASAN --- >> arm64.aarch64 GENERIC-KASAN kernel started on Tue Apr 9 18:56:09 EDT 2024 --- universe_kernconf_arm64_GENERIC-KMSAN --- >> arm64.aarch64 GENERIC-KMSAN kernel started on Tue Apr 9 18:56:15 EDT 2024 --- universe_kernconf_arm64_GENERIC-NODEBUG --- >> arm64.aarch64 GENERIC-NODEBUG kernel started on Tue Apr 9 18:56:18 EDT 2024 --- universe_kernconf_arm64_GENERIC-UP --- >> arm64.aarch64 GENERIC-UP kernel started on Tue Apr 9 18:56:21 EDT 2024 --- universe_kernconf_arm64_LINT --- >> arm64.aarch64 LINT kernel started on Tue Apr 9 18:56:37 EDT 2024 --- universe_kernconf_arm64_LINT-ACPI --- >> arm64.aarch64 LINT-ACPI kernel started on Tue Apr 9 18:56:44 EDT 2024 --- universe_kernconf_arm64_LINT-FDT --- >> arm64.aarch64 LINT-FDT kernel started on Tue Apr 9 18:57:38 EDT 2024 --- universe_i386_kernels --- --- universe_kernconf_i386_LINT --- >> i386.i386 LINT kernel completed on Tue Apr 9 18:57:53 EDT 2024 --- universe_kernels --- >> i386 kernels completed on Tue Apr 9 18:57:53 EDT 2024 --- universe_i386_done --- >> i386 completed on Tue Apr 9 18:58:06 EDT 2024 --- universe_amd64_amd64 --- >> amd64.amd64 buildworld completed on Tue Apr 9 18:58:49 EDT 2024 --- universe_amd64_kernels --- --- universe_kernels_prologue --- >> amd64 kernels started on Tue Apr 9 18:58:54 EDT 2024 --- universe_kernconf_amd64_FIRECRACKER --- >> amd64.amd64 FIRECRACKER kernel started on Tue Apr 9 18:58:54 EDT 2024 --- universe_kernconf_amd64_GENERIC --- >> amd64.amd64 GENERIC kernel started on Tue Apr 9 18:58:57 EDT 2024 --- universe_kernconf_amd64_GENERIC-KASAN --- >> amd64.amd64 GENERIC-KASAN kernel started on Tue Apr 9 18:59:00 EDT 2024 --- universe_kernconf_amd64_GENERIC-KCSAN --- >> amd64.amd64 GENERIC-KCSAN kernel started on Tue Apr 9 18:59:07 EDT 2024 --- universe_kernconf_amd64_GENERIC-KMSAN --- >> amd64.amd64 GENERIC-KMSAN kernel started on Tue Apr 9 18:59:14 EDT 2024 --- universe_powerpc_kernels --- --- universe_kernconf_powerpc_MPC85XX --- >> powerpc.powerpc MPC85XX kernel completed on Tue Apr 9 19:02:05 EDT 2024 --- universe_kernconf_powerpc_GENERIC --- >> powerpc.powerpc GENERIC kernel completed on Tue Apr 9 19:02:52 EDT 2024 --- universe_amd64_kernels --- --- universe_kernconf_amd64_GENERIC-NODEBUG --- >> amd64.amd64 GENERIC-NODEBUG kernel started on Tue Apr 9 19:02:56 EDT 2024 --- universe_powerpc_kernels --- --- universe_kernconf_powerpc_GENERIC-NODEBUG --- >> powerpc.powerpc GENERIC-NODEBUG kernel completed on Tue Apr 9 19:03:00 EDT 2024 --- universe_amd64_kernels --- --- universe_kernconf_amd64_LINT --- >> amd64.amd64 LINT kernel started on Tue Apr 9 19:03:52 EDT 2024 --- universe_powerpc_kernels --- --- universe_kernconf_powerpc_QORIQ64 --- >> powerpc.powerpc64 QORIQ64 kernel completed on Tue Apr 9 19:04:22 EDT 2024 --- universe_amd64_kernels --- --- universe_kernconf_amd64_LINT-NOINET --- >> amd64.amd64 LINT-NOINET kernel started on Tue Apr 9 19:04:23 EDT 2024 --- universe_kernconf_amd64_LINT-NOINET6 --- >> amd64.amd64 LINT-NOINET6 kernel started on Tue Apr 9 19:04:54 EDT 2024 --- universe_powerpc_kernels --- --- universe_kernconf_powerpc_LINT --- >> powerpc.powerpc LINT kernel completed on Tue Apr 9 19:05:04 EDT 2024 --- universe_amd64_kernels --- --- universe_kernconf_amd64_LINT-NOIP --- >> amd64.amd64 LINT-NOIP kernel started on Tue Apr 9 19:06:11 EDT 2024 --- universe_powerpc_kernels --- --- universe_kernconf_powerpc_GENERIC64 --- >> powerpc.powerpc64 GENERIC64 kernel completed on Tue Apr 9 19:06:41 EDT 2024 --- universe_kernconf_powerpc_LINT64 --- >> powerpc.powerpc64 LINT64 kernel completed on Tue Apr 9 19:07:08 EDT 2024 --- universe_amd64_kernels --- --- universe_kernconf_amd64_MINIMAL --- >> amd64.amd64 MINIMAL kernel started on Tue Apr 9 19:07:18 EDT 2024 --- universe_kernconf_amd64_MINIMALUP --- >> amd64.amd64 MINIMALUP kernel started on Tue Apr 9 19:07:25 EDT 2024 --- universe_powerpc_kernels --- --- universe_kernconf_powerpc_GENERIC64-NODEBUG --- >> powerpc.powerpc64 GENERIC64-NODEBUG kernel completed on Tue Apr 9 19:07:37 EDT 2024 --- universe_kernconf_powerpc_GENERIC64LE --- >> powerpc.powerpc64le GENERIC64LE kernel completed on Tue Apr 9 19:07:55 EDT 2024 --- universe_kernconf_powerpc_GENERIC64LE-NODEBUG --- >> powerpc.powerpc64le GENERIC64LE-NODEBUG kernel completed on Tue Apr 9 19:08:17 EDT 2024 --- universe_kernconf_powerpc_LINT64LE --- >> powerpc.powerpc64le LINT64LE kernel completed on Tue Apr 9 19:11:20 EDT 2024 --- universe_kernels --- >> powerpc kernels completed on Tue Apr 9 19:11:20 EDT 2024 --- universe_powerpc_done --- >> powerpc completed on Tue Apr 9 19:11:21 EDT 2024 --- universe_arm64_kernels --- --- universe_kernconf_arm64_GENERIC-KASAN --- >> arm64.aarch64 GENERIC-KASAN kernel completed on Tue Apr 9 19:16:01 EDT 2024 --- universe_kernconf_arm64_GENERIC-UP --- >> arm64.aarch64 GENERIC-UP kernel completed on Tue Apr 9 19:16:13 EDT 2024 --- universe_kernconf_arm64_GENERIC --- >> arm64.aarch64 GENERIC kernel completed on Tue Apr 9 19:16:16 EDT 2024 --- universe_kernconf_arm64_GENERIC-NODEBUG --- >> arm64.aarch64 GENERIC-NODEBUG kernel completed on Tue Apr 9 19:17:06 EDT 2024 --- universe_amd64_kernels --- --- universe_kernconf_amd64_FIRECRACKER --- >> amd64.amd64 FIRECRACKER kernel completed on Tue Apr 9 19:18:53 EDT 2024 --- universe_kernconf_amd64_MINIMALUP --- >> amd64.amd64 MINIMALUP kernel completed on Tue Apr 9 19:20:53 EDT 2024 --- universe_kernconf_amd64_MINIMAL --- >> amd64.amd64 MINIMAL kernel completed on Tue Apr 9 19:21:02 EDT 2024 --- universe_arm64_kernels --- --- universe_kernconf_arm64_LINT-ACPI --- >> arm64.aarch64 LINT-ACPI kernel completed on Tue Apr 9 19:21:28 EDT 2024 --- universe_amd64_kernels --- --- universe_kernconf_amd64_GENERIC --- >> amd64.amd64 GENERIC kernel completed on Tue Apr 9 19:21:34 EDT 2024 --- universe_arm64_kernels --- --- universe_kernconf_arm64_LINT --- >> arm64.aarch64 LINT kernel completed on Tue Apr 9 19:21:52 EDT 2024 --- universe_amd64_kernels --- --- universe_kernconf_amd64_GENERIC-KCSAN --- >> amd64.amd64 GENERIC-KCSAN kernel completed on Tue Apr 9 19:22:10 EDT 2024 --- universe_kernconf_amd64_GENERIC-KASAN --- >> amd64.amd64 GENERIC-KASAN kernel completed on Tue Apr 9 19:22:14 EDT 2024 --- universe_arm64_kernels --- --- universe_kernconf_arm64_LINT-FDT --- >> arm64.aarch64 LINT-FDT kernel completed on Tue Apr 9 19:22:15 EDT 2024 --- universe_amd64_kernels --- --- universe_kernconf_amd64_GENERIC-NODEBUG --- >> amd64.amd64 GENERIC-NODEBUG kernel completed on Tue Apr 9 19:22:38 EDT 2024 --- universe_arm64_kernels --- --- universe_kernconf_arm64_GENERIC-KMSAN --- >> arm64.aarch64 GENERIC-KMSAN kernel completed on Tue Apr 9 19:22:42 EDT 2024 --- universe_kernels --- >> arm64 kernels completed on Tue Apr 9 19:22:42 EDT 2024 --- universe_arm64_done --- >> arm64 completed on Tue Apr 9 19:22:42 EDT 2024 --- universe_amd64_kernels --- --- universe_kernconf_amd64_LINT-NOINET --- >> amd64.amd64 LINT-NOINET kernel completed on Tue Apr 9 19:23:07 EDT 2024 --- universe_kernconf_amd64_LINT-NOIP --- >> amd64.amd64 LINT-NOIP kernel completed on Tue Apr 9 19:23:15 EDT 2024 --- universe_kernconf_amd64_LINT-NOINET6 --- >> amd64.amd64 LINT-NOINET6 kernel completed on Tue Apr 9 19:23:19 EDT 2024 --- universe_kernconf_amd64_LINT --- >> amd64.amd64 LINT kernel completed on Tue Apr 9 19:23:20 EDT 2024 --- universe_kernconf_amd64_GENERIC-KMSAN --- >> amd64.amd64 GENERIC-KMSAN kernel completed on Tue Apr 9 19:23:32 EDT 2024 --- universe_kernels --- >> amd64 kernels completed on Tue Apr 9 19:23:32 EDT 2024 --- universe_amd64_done --- >> amd64 completed on Tue Apr 9 19:23:32 EDT 2024 --- universe_epilogue --- -------------------------------------------------------------- >>> make universe completed on Tue Apr 9 19:23:32 EDT 2024 (started Tue Apr 9 18:20:49 EDT 2024) --------------------------------------------------------------
Comment Actions
I can test on AMD64 (currently using it) and ARM64 and i386. I might be able to do BE PPC64.
Comment Actions
So I've been trying to recreate the steps here and finding the path kinda tricky.
I plan on landing the pull request, kinda. Since I want to do it as a merge commit from a vendor branch, I gotta create a vendor branch and that's being super tricky.
And all the other changes, I'm doing separately. I want a clean as possible merge. There's 4 conflicts.
But the hard part is that jemalloc has gone generate this file happy and so we have to do something approximately like:
- Do vendor branch import
- Copy files from upstream
- apply patches to the .in files
- autoconf.sh <lots of args>
- commit and tag
- Then do the merge
- and also you gotta generate the namespace.h files by hand here
- and then selectively trim
- and resolve the merge conflicts
- and also things like isthreaded and a few other I'm still sorting through
Before xz, I'd have just landed this since we didn't have a good vendor import, but post xz, I think we need to be more careful. Which also means, I think, some pre-import un-thinning of what we have in contrib/jemalloc. Plus I have much angst about this deviating so much from a typical vendor import, though the upstream release rate is slow enough I'm not super worried (though there's been a fair number of changes since the release).
Though all in all, I think once it's sorted in head, we can have it in 14.3...
All this to "quickly" verify that this is good code (I'm 99.99% sure it is from the diffs I've run, but I've not chased down all the diffs just yet).
| lib/libc/stdlib/jemalloc/Makefile.inc | ||
|---|---|---|
| 17 ↗ | (On Diff #125955) | The above won't work, since we add the JEMALLOCSRCS to MISRCS after mangling the name diff --git a/lib/libc/stdlib/malloc/jemalloc/Makefile.inc b/lib/libc/stdlib/malloc/jemalloc/Makefile.inc index d916d69f471f..b804490835b7 100644 --- a/lib/libc/stdlib/malloc/jemalloc/Makefile.inc +++ b/lib/libc/stdlib/malloc/jemalloc/Makefile.inc @@ -12,6 +12,7 @@ CFLAGS+=-D_USE_LG_VADDR_WIDE .for src in ${JEMALLOCSRCS} MISRCS+=jemalloc_${src} CLEANFILES+=jemalloc_${src} +CFLAGS.jemalloc_${src}=-std=gnu11 jemalloc_${src}: ${SRCTOP}/contrib/jemalloc/src/${src} .NOMETA ln -sf ${.ALLSRC} ${.TARGET} .endfor is what's needed. And why gnu11? Does anybody know why upstream chose gnu11 instead of c11? |
| lib/libc/stdlib/jemalloc/Makefile.inc | ||
|---|---|---|
| 17 ↗ | (On Diff #125955) | I'm sorely tempted to just do this bit right here ASAP, btw. It's not all of libc, and I think it's constrained enough to be fine. |
| lib/libc/stdlib/jemalloc/Makefile.inc | ||
|---|---|---|
| 17 ↗ | (On Diff #125955) | Also, there's some other new c11 features, related to anonymous unions and/or flexible arrays, that also cause problems if you try to fix the re-decl with ugly CPP logic. Once you specify C11, life is almost good. /usr/local/bin/x86_64-unknown-freebsd15.0-ld: _malloc_message: undefined version: FBSD_1.0 /usr/local/bin/x86_64-unknown-freebsd15.0-ld: _malloc_options: undefined version: FBSD_1.0 /usr/local/bin/x86_64-unknown-freebsd15.0-ld: _malloc_first_thread: undefined version: FBSDprivate_1.0 /usr/local/bin/x86_64-unknown-freebsd15.0-ld: failed to set dynamic section sizes: bad value so more diffs against the pull request to see what I've missed. Maybe these are the old, ancient symbols that Brooks killed? |
| lib/libc/stdlib/jemalloc/Makefile.inc | ||
|---|---|---|
| 17 ↗ | (On Diff #125955) |
The undefined version bits are obsolete things we patch back in for symbol compatibility. We went way overboard exposing non-standard symbols in public namespaces and now we pay the price forever (or until libc.so.8). These should certainly all be relegated being compat symbols as I did with some others in 5680cf6dc6e25cffa3930e9cb06f6982fcb80209. I'm not sure what the cause of the last bit is. |
| contrib/jemalloc/include/jemalloc/internal/hash.h | ||
|---|---|---|
| 202 | Using JEMALLOC_FALLTHROUGH was introduced in 7bcf5319c1106b761bcb2a4facfeccf1b9db758d. But the upstream jemalloc and 5.3.0 tag use break instead. (https://github.com/jemalloc/jemalloc/blob/54eaed1d8b56b1aa528be3bdd1877e59c56fa90c/include/jemalloc/internal/hash.h#L202). I'm sticking with break because I get compilation error fallthrough annotation does not directly precede switch label. So if there is no reason that we must use the macro, I would prefer to stick with the upstream. | |
| lib/libc/stdlib/jemalloc/Makefile.inc | ||
|---|---|---|
| 17 ↗ | (On Diff #125955) | I'm reverting gnu11 part. After rebasing main and building again, build succeeds without having gnu11 option. |
| lib/libc/stdlib/jemalloc/Makefile.inc | ||
|---|---|---|
| 17 ↗ | (On Diff #125955) | I updated main to have the quoted change for you to remove gnu11 here. After the diff reduction changes, I've figured out what changes we need in contrib/jemalloc, and when in the import process we need to do the config, etc to avoid the FreeBSD-diff file and reduce what we have to check into our tree. There's also some diffs I need to sort out with out FreeBSD includes too. I may need to fix upstream detection of a few symbols as well, since I have to change config still... Thanks for rebasing. I'm using this as a reference to make sure I can reproduce what you've done. When I push that work, I'll credit you as the author since I'm just verifying there's nothing unexpected and also trying to find FreeBSD changes we no longer need... Brooks' comments are next up for me.... Thanks for all the hard work to get to this point... It's certainly making the updating of the vendor branch to current project standards way easier. |
| contrib/jemalloc/include/jemalloc/internal/hash.h | ||
|---|---|---|
| 202 | I must have read upstream vs the change incorrectly. It sure looked like upstream used JEMALLOC_FALLTHROUGH in the diffs I ran, but I've reverted so there's no conflict here now. | |