Add a tunable to set the pgcache zone's maxcache

[gallatin]

Summary

We currently set the maxcache to 0 on the pgcache uma zone. Under a heavy Netflix ktls encrypted web workload, this leads to lock contention on the vm page free queue from vm_page_zone_import / vm_page_zone_release as the buckets are frequently drained. This change adds a tunable to allow the administrator to specify the maxache.

On a Netflix 100G cache (16 core / 32t Xeon E5-v4 2697A), we get to about 80Gb/s with CPU maxed contending for the vm page free lock when maxcache is 0. When it is set to 0x60000 we reach 95Gb/s with idle CPU.

[jeff]

We have run into issues with the auto-tuning here. It all needs to do better. We should be able to set max cache based on total memory. I'm not opposed to this patch as an intermediate step though.

[markj]

In D22112#483262, @jeff wrote:

We have run into issues with the auto-tuning here. It all needs to do better. We should be able to set max cache based on total memory. I'm not opposed to this patch as an intermediate step though.

Indeed, I noted this when we discussed it on slack. As a start, uma_reclaim() needs to start reclaiming from cache zones in at least this case. (It probably doesn't make much sense to reclaim cached bufs, for instance.)