Allow ZFS ARC min / max to be tuned at runtime
ClosedPublic

Authored by smh on Apr 11 2016, 9:09 AM.

Details

Summary

Allow ZFS ARC min / max to be tuned at runtime

Test Plan
  • Decrease vfs.zfs.arc_max testing threshold values
  • Increase vfs.zfs.arc_max testing threshold values
  • Decrease vfs.zfs.arc_min testing threshold values
  • Increase vfs.zfs.arc_min testing threshold values

In each of the cases confirm ARC values adjust and memory usage follows.

Diff Detail

Repository
rS FreeBSD src repository
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.
smh retitled this revision from to Allow ZFS ARC min / max to be tuned at runtime.Apr 11 2016, 9:09 AM
smh updated this object.
smh edited the test plan for this revision. (Show Details)
smh edited the test plan for this revision. (Show Details)Apr 11 2016, 9:11 AM
smh added reviewers: delphij, avg.
smh updated this revision to Diff 15069.Apr 11 2016, 9:12 AM

Remove redundent check on <= 0

bapt added a subscriber: bapt.Jun 28 2016, 11:44 AM
emaste added a subscriber: emaste.Jun 28 2016, 1:28 PM
allanjude added inline comments.
sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c
938 ↗(On Diff #15069)

It might be worth calling arc_kmem_reap_now() or kmem_reap() if the user is lowing the arc_max (as they obviously want memory back)

This cleans up and usually results in lowering the amount of 'wired' memory.

I am going to quickly experiment with that and get back to you.

smh added inline comments.Jun 28 2016, 3:14 PM
sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c
938 ↗(On Diff #15069)

When I tested it cleaned up quickly without that, so not sure its actually needed.

allanjude added inline comments.Jun 28 2016, 6:59 PM
sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c
938 ↗(On Diff #15069)

In my case it did not:

Frequently read files in ARC:

before:

Mem: 90M Active, 25M Inact, 61G Wired, 64G Free
ARC: 22G Total, 21G MFU, 555M MRU, 32K Anon, 173M Header, 67M Other

So I lowered the arc_max:
vfs.zfs.arc_max: 24000000000 -> 16000000000

Mem: 90M Active, 25M Inact, 61G Wired, 64G Free
ARC: 15G Total, 14G MFU, 555M MRU, 32K Anon, 169M Header, 49M Other

So the ARC shrunk, but no additional memory was freed

However if I run some sysctl's I added:
debug.zfs_arc_reap: 0 -> 0

Mem: 125M Active, 26M Inact, 38G Wired, 87G Free
ARC: 15G Total, 14G MFU, 555M MRU, 32K Anon, 169M Header, 50M Other

A great deal of the 'Wired' memory moves to free.

This does even more:
debug.zfs_kmem_reap: 0 -> 0

Mem: 90M Active, 25M Inact, 17G Wired, 107G Free
ARC: 15G Total, 14G MFU, 555M MRU, 32K Anon, 169M Header, 50M Other

It is not clear to me why in arc_kmem_reap_now() we only call kmem_reap() in the i386 case.

So I think we should either make lowering the arc_max call this, or, make it a separate sysctl similar to the temporary one I created. It is not instantaneous like changing arc_max is, so maybe separate makes more sense.

I just thought I'd mention this here, since usually the point of lowering arc_max is to free memory for another purpose.

allanjude accepted this revision.Jun 28 2016, 11:28 PM

Tested on two of my machines. Works as expected.

Would love to see this in 11.0-RELEASE

This revision is now accepted and ready to land.Jun 28 2016, 11:28 PM
This revision was automatically updated to reflect the committed changes.