Allow ZFS ARC min / max to be tuned at runtime

[smh]

Summary

Allow ZFS ARC min / max to be tuned at runtime

Test Plan

• Decrease vfs.zfs.arc_max testing threshold values
• Increase vfs.zfs.arc_max testing threshold values
• Decrease vfs.zfs.arc_min testing threshold values
• Increase vfs.zfs.arc_min testing threshold values

In each of the cases confirm ARC values adjust and memory usage follows.

[smh]

Remove redundent check on <= 0

[allanjude]

sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c

938 ↗

(On Diff #15069)

It might be worth calling arc_kmem_reap_now() or kmem_reap() if the user is lowing the arc_max (as they obviously want memory back)

This cleans up and usually results in lowering the amount of 'wired' memory.

I am going to quickly experiment with that and get back to you.

[smh]

sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c

938 ↗

(On Diff #15069)

When I tested it cleaned up quickly without that, so not sure its actually needed.

[allanjude]

sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c

938 ↗

(On Diff #15069)

In my case it did not:

Frequently read files in ARC:

before:

Mem: 90M Active, 25M Inact, 61G Wired, 64G Free
ARC: 22G Total, 21G MFU, 555M MRU, 32K Anon, 173M Header, 67M Other

So I lowered the arc_max:
vfs.zfs.arc_max: 24000000000 -> 16000000000

Mem: 90M Active, 25M Inact, 61G Wired, 64G Free
ARC: 15G Total, 14G MFU, 555M MRU, 32K Anon, 169M Header, 49M Other

So the ARC shrunk, but no additional memory was freed

However if I run some sysctl's I added:
debug.zfs_arc_reap: 0 -> 0

Mem: 125M Active, 26M Inact, 38G Wired, 87G Free
ARC: 15G Total, 14G MFU, 555M MRU, 32K Anon, 169M Header, 50M Other

A great deal of the 'Wired' memory moves to free.

This does even more:
debug.zfs_kmem_reap: 0 -> 0

Mem: 90M Active, 25M Inact, 17G Wired, 107G Free
ARC: 15G Total, 14G MFU, 555M MRU, 32K Anon, 169M Header, 50M Other

It is not clear to me why in arc_kmem_reap_now() we only call kmem_reap() in the i386 case.

So I think we should either make lowering the arc_max call this, or, make it a separate sysctl similar to the temporary one I created. It is not instantaneous like changing arc_max is, so maybe separate makes more sense.

I just thought I'd mention this here, since usually the point of lowering arc_max is to free memory for another purpose.

[allanjude]

Tested on two of my machines. Works as expected.

Would love to see this in 11.0-RELEASE