Add "Logical Block Mode" to geom multipath

[allanjude]

Summary

Solaris multipath has two modes, round-robin and "Logical Block" mode
http://docs.oracle.com/cd/E19253-01/820-1931/6ndinfi78/index.html

Add "Logical Block" support to our existing Active/Active and Active/Read modes

It divides the devices up into stripes of 2^x bytes, where x is configurable
All sectors in the same stripe are always accessed via the same path (if it is up).
This can improve performance by hitting the same controller for sequential reads, and taking advantage of read-ahead and cache.

Test Plan

dvl@ has volunteered to do benchmarks on his new rig
I also have some new multipath hardware coming in the next week or two

[allanjude]

Fix some compiler warnings

[bapt]

I added @araujo in subscriber as he is (or planned) to work on the same thing

[araujo]

In D3343#67914, @bapt wrote:

I added @araujo in subscriber as he is (or planned) to work on the same thing

Yes, I was working on that, but not so far like @allanjude. Feel free to add me as reviewer if you wish.

[allanjude]

In D3343#67916, @araujo wrote:

In D3343#67914, @bapt wrote:

I added @araujo in subscriber as he is (or planned) to work on the same thing

Yes, I was working on that, but not so far like @allanjude. Feel free to add me as reviewer if you wish.

I wrote this up quickly yesterday afternoon. Other than the metadata version bump, it is fairly straight forward, accept a 'logicalblock' argument, 0 = disabled, other values are used to bit-shift the request offset, then modulus by the number of paths to determine which disk to use to service the request (only applies to active/active or active/read mode, active/passive always just uses the active disk).

Currently, I have the acceptable range for logicalblock as 9 - 31, I suppose we could allow higher values as well though.

[wblock]

sbin/geom/class/multipath/gmultipath.8

142

If "option" is used, it needs to be "The .Fl -L option". Otherwise, leave out the word "option".
Also, comma splice. Not sure about the capitalization of logical block, either, although the other sections do it. Suggestion:

.Fl L
enables Logical Block mode.
The device is divided into stripes of

144

"served via the same path" is kind of vague. It's hard to see how the path ties in to the size of the requests, but the sentence implies it. Maybe this should be two separate sentences.

145

How? You mean because this ensures that requests are going to the same device?

181

I count three instances of this same text. Is that Phabricator helping us again?

[wollman]

For what it's worth, in all my testing -A and -R modes have been a small but significant lose, so I'll be interested to see if this mode fixes that.

[allanjude]

Testing to make sure I can still post an update after moving Phabricator

[gbe]

LGTM, but the comments of @wblock should be addressed first.