Nice improvements, I learned some interesting details while following these improvements :-)

LGTM

Few nits

LGTM

LGM, much cleaner than it was.

Wow that's a lot of changes.

Convert timestamp to ticks to ensure that comparisons are correct for
hz != 1000.

Lawrence is now happy with this in is current form, so just a prod to see if I get any comments for or against from transport members / gnn?

Extract common logic to tcp_autorcvbuf as suggested by lstewart.

Some style nits but otherwise LGM

Thanks for the feedback Lawrence, based on that I've updated to use just the SRTT check, added the fastpath version and generally cleaned up.

Corrected dtrace autoresize mbuf parameter definition.

Updated dtrace:

dtrace
#!/usr/sbin/dtrace -s

Eliminated timestamp specific code path resulting in a simplified yet still effective single SRTT path.

LGTM

Massive amounts of changes in this so impossible to see if everything is good, however a number of style related bits highlighted, some of which are regressions.

book -> bool

In D9829#203014, @jhb wrote:

I agree with shortening the line a bit so it fits in 80 cols. If you are worried about duplicating "ARC:" you could perhaps just use a whitespace prefix so you end up with something like:

Mem: 754M Active, 836M Inact, 75M Laundry, 3761M Wired, 10G Free
ARC: 2476M Total, 914M MFU, 1302M MRU, 1696K Anon, 140M Header, 118M Other
     1899M Compressed, 6065M Uncompressed, 2.45:1 Ratio, 318M Overhead
Swap: 3072M Total, 3072M Free

Some may find it useful if it was bit shorter, currently its 82 chars in the example.

Indeed, do you have any bandwidth to do proper testing to prove either way or have you already done this?

LGTM

LGTM

LGTM

Just a few seemingly redundant assignments to error vars, sorry didn't spot them before

Disabled reset receive buffer auto scaling when not in bulk receive mode, which gives an extra 20% performance increase, bring it closer to Linux.

For reference fastpath version of this hasn't been done, which need also need the same changes to:

tcp_stacks/fastpath.c

In D9611#198637, @avg wrote:

Essentially, L2ARC periodically writes to every block on the device and that allows to physically move the data around.
That's exactly the reason why I think that TRIM is not needed for L2ARC.
TRIM is useful when we don't need data in some area, but we are not going to overwrite that area, so we need to a way to tell the storage system that it can reuse the physical cells without worrying about any data in them. But if we overwrite that area anyway, then the storage system is automatically aware that the data in those physical cells is obsolete. It's free to choose either those same cells or any different cells for the new data according to the wear leveling algorithms, but that's beside the point.

In D9611#198616, @avg wrote:

My understanding of how L2ARC writing works is this. The code maintains a "hand" (like a clock hand) that points to a disk offset. At the regular intervals a certain space in front of the hand is freed by discarding L2 headers that point to that space and then new buffers are written to that space. Then the hand is moved forward by the appropriate amount.
There is also some freeing of L2 headers when ARC headers are freed, etc. In any case, after some uptime almost the whole cache disk is usually filled with data. And the hand inevitably moves forward. So, every block gets written over sooner or later. I do not see how TRIM helps in that case.
The only scenario where it makes difference, in my humble opinion, is the scenario that @mav described: a worn out disk where the "cheese holes" behind the hand can make some difference for writing new blocks at the hand. But I think that that's too marginal to be important.

In D9611#198582, @mav wrote:

In D9611#198579, @smh wrote:

In scenario #1 the performance of TRIM is also generally good, mitigating the need to avoid doing it.

Is there such thing as good TRIM performance? On my new Samsung 950 NVMe I had to disable TRIM as unusable. Though yes, NVMe driver still probably needs aggregation of TRIM requests to get better numbers.

In scenario #1 the performance of TRIM is also generally good, mitigating the need to avoid doing it.

This can could excessive slow down as the capacity of the disk is reached, however it could be argued that a better mitigation for L2ARC devices would be to use an under-provisioned slice to ensure the SSD controller always has space to work.

Couple of little style nits and I'd like to understand why ENAMETOOLONG error gets turned into success in a few places.

Wouldn't it be better to have 0 (the default) be "use the PhyNum field as a fallback to the mapping logic" as that way at least the device would initialise when mpX_mapping_get_sas_id fails?

Thanks for attacking this Andriy its a big task which really needed some attention.

Apart from the comment this looks good in principle, however I think we need to better understand why retrying the probe works as it feels like where just hiding the real error with this.

This looks reasonable however as you say its potentially racey with multiple renames happening.