Make dirty queues a per-domain property

[jeff]

Summary

We found two bugs with recent buffer cache changes. Firstly, if all of the dirty buffers were on a single domain we still may not trip the limit to start buf_daemon. Secondly, softdep has an issue where it can hold thousands of locked bufs while waiting on a few delayed writes to happen. This second one is really hard to solve gracefully. For the first, we move all dirty queues and limits into per-domain structures while keeping a single buf daemon for now. This should also help with high bandwidth write situations. For the second, a comment and a bd_speedup() if we can't free clean buffers.

I also fixed per-domain sysctls by leaving the appearance of globals and doing the division before assignment to individual queues.

I added some significant debug info to show bufqueues which was quite enlightening and should continue to be so if we have other leaks.

I added a small optimization to bufspace_adjust() that reduces cache contention on the hot bufspace variable.

This has already gotten the thumbs up from pho.

[jeff]

vfs_bio.c

104 ↗

(On Diff #40329)

I had to move all of this or really scatter the function definitions. Sorry it makes the diff harder to read.

376 ↗

(On Diff #40329)

Same technique as vm

1483 ↗

(On Diff #40329)

I suppose if I made the bqempty per-domain we could simplify this whole function into an array lookup again. However, there is no concurrency value in making per-domain bqempty and it would complicate other code.