Page MenuHomeFreeBSD
Differential D23874

Make jemalloc(3) default to retain:true.
ClosedPublic
Actions

Authored by trasz on Feb 28 2020, 2:20 PM.
Tags
None
Referenced Files
Unknown Object (File)
Mon, Nov 18, 6:54 PM
Unknown Object (File)
Fri, Nov 1, 2:47 PM
Unknown Object (File)
Thu, Oct 31, 7:25 AM
Unknown Object (File)
Oct 10 2024, 12:23 PM
Unknown Object (File)
Oct 5 2024, 2:12 AM
Unknown Object (File)
Oct 3 2024, 9:53 AM
Unknown Object (File)
Oct 2 2024, 11:54 PM
Unknown Object (File)
Oct 2 2024, 6:19 PM
View All 92 Files
Subscribers
imp
mjg
val_packett.cool

Details

Reviewers
markj
alc
Commits
rS359481: Make jemalloc(3) default to retain:true on 64-bit platforms,
Summary

Make jemalloc(3) default to retain:true, like it does on Linux
and OSX. This results in significantly fewer calls to mmap(2).
For example, here's the diff of truss output for id(1):

--- bez	2020-02-27 18:41:46.374698000 +0000
+++ z	2020-02-27 18:41:57.547764000 +0000
@@ -26,55 +26,24 @@ mprotect(0x800424000,36864,PROT_READ)		 = 0 (0x0)
 sysarch(AMD64_SET_FSBASE,0x7fffffffda20)	 = 0 (0x0)
 mprotect(0x800424000,36864,PROT_READ|PROT_WRITE) = 0 (0x0)
 mprotect(0x800424000,36864,PROT_READ)		 = 0 (0x0)
-readlink("/etc/malloc.conf",0x7fffffffd130,1024) ERR#2 'No such file or directory'
+readlink("/etc/malloc.conf","retain:true,junk:false",1024) = 22 (0x16)
 issetugid()					 = 0 (0x0)
 __sysctl("vm.overcommit",2,0x7fffffffd0dc,0x7fffffffd0d0,0x0,0) = 0 (0x0)
 mmap(0x0,2097152,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON|MAP_ALIGNED(21),-1,0x0) = 34368126976 (0x800800000)
-mmap(0x0,32768,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON|MAP_ALIGNED(12),-1,0x0) = 34366410752 (0x80065d000)
-mmap(0x0,4194304,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON|MAP_ALIGNED(21),-1,0x0) = 34370224128 (0x800a00000)
-mmap(0x0,4096,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON|MAP_ALIGNED(12),-1,0x0) = 34362114048 (0x800244000)
+mmap(0x0,2097152,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON|MAP_ALIGNED(12),-1,0x0) = 34370224128 (0x800a00000)
+mmap(0x0,4194304,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON|MAP_ALIGNED(21),-1,0x0) = 34372321280 (0x800c00000)
 mprotect(0x204000,4096,PROT_READ)		 = 0 (0x0)
 getuid()					 = 1000 (0x3e8)
-mmap(0x0,4096,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON|MAP_ALIGNED(12),-1,0x0) = 34366443520 (0x800665000)
-mmap(0x0,4096,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON|MAP_ALIGNED(12),-1,0x0) = 34366447616 (0x800666000)
-mmap(0x0,4096,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON|MAP_ALIGNED(12),-1,0x0) = 34366451712 (0x800667000)
 fstatat(AT_FDCWD,"/etc/nsswitch.conf",{ mode=-rw-r--r-- ,inode=2167112,size=338,blksize=32768 },0x0) = 0 (0x0)
 open("/etc/nsswitch.conf",O_RDONLY|O_CLOEXEC,0666) = 3 (0x3)
-mmap(0x0,28672,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON|MAP_ALIGNED(12),-1,0x0) = 34366455808 (0x800668000)
-mmap(0x0,28672,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON|MAP_ALIGNED(12),-1,0x0) = 34366484480 (0x80066f000)
-mmap(0x0,20480,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON|MAP_ALIGNED(12),-1,0x0) = 34366513152 (0x800676000)
-mmap(0x0,20480,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON|MAP_ALIGNED(12),-1,0x0) = 34366533632 (0x80067b000)
 fstat(3,{ mode=-rw-r--r-- ,inode=2167112,size=338,blksize=32768 }) = 0 (0x0)
-mmap(0x0,36864,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON|MAP_ALIGNED(12),-1,0x0) = 34366554112 (0x800680000)
 read(3,"#\n# nsswitch.conf(5) - name ser"...,32768) = 338 (0x152)
-read(3,0x800680180,32768)			 = 0 (0x0)
-mmap(0x0,4096,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON|MAP_ALIGNED(12),-1,0x0) = 34366590976 (0x800689000)
-mmap(0x0,20480,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON|MAP_ALIGNED(12),-1,0x0) = 34366595072 (0x80068a000)
-mmap(0x0,12288,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON|MAP_ALIGNED(12),-1,0x0) = 34366615552 (0x80068f000)
-mmap(0x0,4096,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON|MAP_ALIGNED(12),-1,0x0) = 34366627840 (0x800692000)
-mmap(0x0,12288,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON|MAP_ALIGNED(12),-1,0x0) = 34366631936 (0x800693000)
+read(3,0x800a24180,32768)			 = 0 (0x0)
 close(3)					 = 0 (0x0)
 geteuid()					 = 1000 (0x3e8)
-mmap(0x0,12288,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON|MAP_ALIGNED(12),-1,0x0) = 34366644224 (0x800696000)
 open("/etc/pwd.db",O_RDONLY|O_CLOEXEC,00)	 = 3 (0x3)
 fstat(3,{ mode=-rw-r--r-- ,inode=2167182,size=40960,blksize=32768 }) = 0 (0x0)
 read(3,"\0\^F\^Ua\0\0\0\^B\0\0\^P\M-a\0"...,260) = 260 (0x104)
-mmap(0x0,4096,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON|MAP_ALIGNED(12),-1,0x0) = 34366656512 (0x800699000)
-mmap(0x0,4096,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON|MAP_ALIGNED(12),-1,0x0) = 34366660608 (0x80069a000)
-mmap(0x0,4096,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON|MAP_ALIGNED(12),-1,0x0) = 34366664704 (0x80069b000)
-mmap(0x0,4096,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON|MAP_ALIGNED(12),-1,0x0) = 34366668800 (0x80069c000)
-mmap(0x0,4096,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON|MAP_ALIGNED(12),-1,0x0) = 34366672896 (0x80069d000)
-mmap(0x0,12288,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON|MAP_ALIGNED(12),-1,0x0) = 34366676992 (0x80069e000)
-mmap(0x0,4096,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON|MAP_ALIGNED(12),-1,0x0) = 34366689280 (0x8006a1000)
-mmap(0x0,4096,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON|MAP_ALIGNED(12),-1,0x0) = 34366693376 (0x8006a2000)
-mmap(0x0,4096,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON|MAP_ALIGNED(12),-1,0x0) = 34366697472 (0x8006a3000)
-mmap(0x0,4096,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON|MAP_ALIGNED(12),-1,0x0) = 34366701568 (0x8006a4000)
-mmap(0x0,4096,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON|MAP_ALIGNED(12),-1,0x0) = 34366705664 (0x8006a5000)
-mmap(0x0,4096,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON|MAP_ALIGNED(12),-1,0x0) = 34366709760 (0x8006a6000)
-mmap(0x0,4096,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON|MAP_ALIGNED(12),-1,0x0) = 34366713856 (0x8006a7000)
-mmap(0x0,4096,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON|MAP_ALIGNED(12),-1,0x0) = 34366717952 (0x8006a8000)
-mmap(0x0,4096,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON|MAP_ALIGNED(12),-1,0x0) = 34366722048 (0x8006a9000)
-mmap(0x0,4096,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON|MAP_ALIGNED(12),-1,0x0) = 34366726144 (0x8006aa000)
 pread(3,"\0*\^O\M-x\^O\M-w\^O\M-r\^O\M-&"...,4096,0x6000) = 4096 (0x1000)
 pread(3,"\0&\^O\M-{\^O\M->\^O\M-9\^Oz\^Ou"...,4096,0x4000) = 4096 (0x1000)
 pread(3,"\0.\^O\M-{\^O\M-/\^O\M-*\^OU\^OP"...,4096,0x5000) = 4096 (0x1000)
@@ -85,7 +54,7 @@ pread(3,"\0 \^O\M-{\^O\M->\^O\M-9\^Ol\^Of"...,4096,0x2
 pread(3,"\0"\^O\M-|\^O\M-0\^O\M-,\^Oq\^Ol"...,4096,0x3000) = 4096 (0x1000)
 close(3)					 = 0 (0x0)
 __sysctl("kern.ngroups",2,0x7fffffffe3cc,0x7fffffffe3d0,0x0,0) = 0 (0x0)
-getgroups(0x400,0x8006a3000)			 = 4 (0x4)
+getgroups(0x400,0x800a47000)			 = 4 (0x4)
 fstat(1,{ mode=crw--w---- ,inode=715,size=0,blksize=4096 }) = 0 (0x0)
 ioctl(1,TIOCGETA,0x7fffffffdda0)		 = 0 (0x0)
 open("/etc/group",O_RDONLY|O_CLOEXEC,0666)	 = 3 (0x3)

Diff Detail

Repository
rS FreeBSD src repository - subversion
Lint
No Lint Coverage
Unit
No Test Coverage
Build Status
Buildable 29810
Build 27638: arc lint + arc unit

Event Timeline

trasz created this revision.Feb 28 2020, 2:20 PM
Herald added a subscriber: imp. · View Herald TranscriptFeb 28 2020, 2:20 PM
Harbormaster completed remote builds in B29669: Diff 68939.Feb 28 2020, 2:20 PM
trasz edited the summary of this revision. (Show Details)Feb 28 2020, 2:22 PM
trasz added a reviewer: alc.
markj added a comment.Feb 28 2020, 3:14 PM

I ran some buildkernels on a 32-thread system. We get a small reduction in system CPU time and improved superpage usage. https://reviews.freebsd.org/P368

My suspicion is that we have a similar behaviour to Linux wrt "common sequences of mmap()/munmap() calls will cause virtual memory map holes." In particular, each mapping created by jemalloc will advance the cursor in vm_map_find().

contrib/jemalloc/include/jemalloc/internal/jemalloc_internal_defs.h
218 ↗(On Diff #68939)

Should this be enabled on 32-bit platforms as well? On 64-bit platforms I don't see much of a downside to enabling this, though I'm not sure if our virtual memory allocator is susceptible to the same problems that were seen on Linux. On 32-bit platforms it seems this option would make it easy to run out of virtual address space.

trasz updated this revision to Diff 69230.Mar 5 2020, 5:20 PM

Rework to limit to 64-bit platforms.

Harbormaster completed remote builds in B29810: Diff 69230.Mar 5 2020, 5:20 PM
markj accepted this revision.Mar 5 2020, 7:24 PM

Seems reasonable to me.

This revision is now accepted and ready to land.Mar 5 2020, 7:24 PM
trasz added a comment.Mar 6 2020, 10:39 PM

(Tinderboxed.)

val_packett.cool added a subscriber: val_packett.cool.Mar 18 2020, 7:03 PM
mjg added a subscriber: mjg.Mar 25 2020, 12:16 AM

anything stopping this from going in?

trasz added a comment.Mar 30 2020, 6:33 PM

Kind of - I've noticed that it increases the number of page faults with small binaries, like id(1), and I'm wondering why.

markj added a comment.Mar 30 2020, 6:43 PM
In D23874#533005, @trasz wrote:

Kind of - I've noticed that it increases the number of page faults with small binaries, like id(1), and I'm wondering why.

I can't think of any reason, but I also can't measure any difference. For me, ktrace -tf reports 102 page faults for id(1) with or without retain:true.

mjg added a comment.EditedMar 30 2020, 6:57 PM

Maybe there is a difference depending on MALLOC_PRODUCTION? Either way, looks like a net win.

Which also reminds me that someone(tm) should take a look at importing newer jemalloc. There was an attempt some time ago but it got backed out due to problems with mips(?).

Closed by commit rS359481: Make jemalloc(3) default to retain:true on 64-bit platforms, (authored by trasz). · Explain WhyMar 31 2020, 1:48 PM
This revision was automatically updated to reflect the committed changes.
trasz added a commit: rS359481: Make jemalloc(3) default to retain:true on 64-bit platforms,.

Revision Contents
Changeset List

PathSize
contrib/
jemalloc/
include/
jemalloc/
4 lines

Diff 69230

View Options

contrib/jemalloc/include/jemalloc/jemalloc_FreeBSD.h

Loading...