Past 80 comments and I'm losing focus. Submitting what I have, will continue later.

These are a few things I spotted, but I'm not sure I caught everything and I'm on the wrong side of entirely sober. :)

Updated diff incorporating suggestions by @ceri

In D31707#715385, @ceri wrote:

Running out of battery so submitting what I have reviewed. Generally speaking I think the shorter text works against the intent of this being a guide, I'm afraid.

Running out of battery so submitting what I have reviewed. Generally speaking I think the shorter text works against the intent of this being a guide, I'm afraid.

This one uses git diff --no-prefix -U1000

Can you switch this diff to one with context? It makes it easier to make suggestions. :)

I always have to ask sjg. Maybe add him to the review?

This looks good to me. I recall that Makefile.depend will be generated by whoever needs it and does not need to be committed if you aren't sure how to do it, but I don't see that in the committer's guide so I'm not sure where that idea came from. Hopefully someone else has input on that aspect.

atomic.S patch by mav

I am in for the assembler code patch. I will prepare an updated version.

Actually my original version was written in assembler to be added into atomic.S without using the compiler atomics: https://people.freebsd.org/~mav/zzz3S.patch . I think it would be more consistent with original code. But since it is hopefully temporary, I am fine with any approach.

I am definitely going to do that. But in the meantime we need a buildable stable/13 for i386. The solution I am proposing does (temporarily) place an additional file into cddl/lib/libspl what is outside of sys/contrib/openzfs.

Could you ask him to merge the dependency instead of creating unneeded divergence, unless there is a good cause?

I assume Brian just forgot to merge that commit or intentionally didn't merge that commit and did not inspect your commit for changes in that direction. The i386 build should be broken for Linux as well.

Aaa... While the patch may be OK, I am surprised that my e76373de7b7384bb6e5c6fd5e04f15b54df20fb7 was merged into zfs-2.1-release before fef8bd41fc178d7212957b611c9bc81fe10cb63e. Merge in proper order would not cause this problem in first place. Do you have a clue why it was not done that way?

Thank you for catching this. The version strings look reasonable to me.

I've been able to keep a NO_CLEAN tree going for several years; it does break occasionally but we have a standard, if ugly, way of addressing the resulting dependency issues when they occur.

I will apply your patch, but it's a bit silly to call that "breaking world". -There's no guarantee that -DNO_CLEAN or -DNO_KERNELCLEAN will work across major changes.

Yes.

@neel_neelc.org was this a -DNO_CLEAN build?

looks good to me.

This is the error I get without the patch:

This patch fixes the build issue complaining about:

@neel_neelc.org can you tell me a bit more about your environment, make universe builds for all of us and only the powerpcspe tinderbox is currently broken because of these changes. I'm happy to commit these changes I just want to know what it's fixing.

In D19094#443654, @avg wrote:

Sorry, I myself went missing for a long while.
Yes, I can commit this change.

Do you want anything specific to appear in a commit message?
Like any additional attributions, etc?

Sorry, I myself went missing for a long while.
Yes, I can commit this change.

In D19094#418092, @avg wrote:

In D19094#417188, @slw_zxy.spb.ru wrote:

No more replays?

Unfortunately, no.
I think that we can commit your proposed change. If George comes up with a different and better solution later on, there should be no problem switching to it.

In D19094#417188, @slw_zxy.spb.ru wrote:

No more replays?

In D19094#412736, @avg wrote:

In D19094#411932, @slw_zxy.spb.ru wrote:

Do you succesefull contact George?

I've just got a reply from George.
He agrees with your analysis, but needs some more time to think about how to address the issue.
Let's wait a bit more.
Thanks!

In D19094#411932, @slw_zxy.spb.ru wrote:

Do you succesefull contact George?

In D19094#410508, @avg wrote:

Let me try to contact George again.

Let me try to contact George again.

In D19094#410239, @mav wrote:

In D19094#410192, @slw_zxy.spb.ru wrote:

Don't sure about calling remove_reference() from arc_hdr_alloc_pabd() (or from parallel tasks), but see at ARC MFU/MRU size calculation in arc_change_state() called from arc_access() and !GHOST_STATE(state) case in arc_get_data_impl() called from arc_hdr_alloc_pabd().
I mean interchange this lines can cause problems for this accountings.

I am not sure what accounting problem you are talking about,

In D19094#410192, @slw_zxy.spb.ru wrote:

Don't sure about calling remove_reference() from arc_hdr_alloc_pabd() (or from parallel tasks), but see at ARC MFU/MRU size calculation in arc_change_state() called from arc_access() and !GHOST_STATE(state) case in arc_get_data_impl() called from arc_hdr_alloc_pabd().
I mean interchange this lines can cause problems for this accountings.

In D19094#410062, @mav wrote:

While I see the problem you are fixing, the fix looks ugly to me, that is why I would look for something nicer.

In D19094#410076, @lev wrote:

Huh? As far as I can see, problem not remove_reference(), but arc_adapt() called too late, with header in wrong state (promoted from something-ghost to live LRU), which brraks main idea of ARC adaptation.

In D19094#410062, @mav wrote:

While I see the problem you are fixing, the fix looks ugly to me, that is why I would look for something nicer. I agree that according to logic of remove_reference() dropping last reference for header in ghost state is a failure, but how can remove_reference() be called before the arc_access() just on following line? I would guess from description telling about the case of prefetch read it should happen no sooner then we actually initiate the I/O, which is done much later then those two lines. So while I agree it is somewhat odd to have buffer for header in ghost state, is that a criminal.

While I see the problem you are fixing, the fix looks ugly to me, that is why I would look for something nicer. I agree that according to logic of remove_reference() dropping last reference for header in ghost state is a failure, but how can remove_reference() be called before the arc_access() just on following line? I would guess from description telling about the case of prefetch read it should happen no sooner then we actually initiate the I/O, which is done much later then those two lines. So while I agree it is somewhat odd to have buffer for header in ghost state, is that a criminal.

Thank you for the additions! Just my $0.05 worth of nit picks post commit.

Update the diff based on Allan's comments. I've decided to remove the newsyslog compression example. This might be something better suited for the handbook. In it's place, I've added Allan's zfs snapshot range deletion examples. I've also changed the date in the custom property example to a more prominent date. ;-)

Thanks for your feedback, @allanjude. I'll work on an updated patch that incorporates it.

Adding more reviewers for more pairs of eyes for eventual approval.

Update with suggestions by Dru.

These look great!

8. ARC don't rised, memory pressure does not arise, page daemon not activated.

The ARC is not growing after 8, but the ARC hit rate is too low. Why is it not growing? Is it because the free_memory < (arc_c >> arc_no_grow_shift) condition is true, or is there some other reason?

In D7538#361198, @slw_zxy.spb.ru wrote:

In D7538#361135, @markj wrote:

No, lots of memory in mbuf cache independ of memory pressure/page daemon in my workload:

1. net workload raise
2. mbuf consumption rise
3. free memory reduction
4. ARC reacts to free memory reduction by ARC reduction
5. net workload reduction started
6. mbuf released to zone cache
7. free memory don't raised
8. ARC don't rised, memory pressure does not arise, page daemon not activated.

In D7538#361135, @markj wrote:

In D7538#359546, @slw_zxy.spb.ru wrote:

To be clear, I'm just stating that r332365 changed zfs_arc_free_target to be equal vm_cnt.v_free_target. It looks to me that this is equivalent to the change you made to arc_available_memory(EXCLUDE_ZONE_CACHE), where v_free_target is referenced directly.

No.

arc_available_memory(EXCLUDE_ZONE_CACHE) check conditions for memory pressure, check how many free memory see by OS (and kmem cache not counted for this).

Yes, which is exactly what the computation freemem - zfs_arc_free_target is. If you expand these definitions, it is vm_cnt.v_free_count - vm_cnt.v_free_target, where v_free_count does not include UMA caches. When v_free_count < v_free_target, the system is under memory pressure, and the page daemon attempts to free pages until v_free_count >= v_free_target. In -CURRENT, you can think of needfree as being the same as v_free_target - v_free_target when this difference is positive. In stable branches this is not quite true.

In D7538#359546, @slw_zxy.spb.ru wrote:

To be clear, I'm just stating that r332365 changed zfs_arc_free_target to be equal vm_cnt.v_free_target. It looks to me that this is equivalent to the change you made to arc_available_memory(EXCLUDE_ZONE_CACHE), where v_free_target is referenced directly.

No.

arc_available_memory(EXCLUDE_ZONE_CACHE) check conditions for memory pressure, check how many free memory see by OS (and kmem cache not counted for this).

To be clear, I'm just stating that r332365 changed zfs_arc_free_target to be equal vm_cnt.v_free_target. It looks to me that this is equivalent to the change you made to arc_available_memory(EXCLUDE_ZONE_CACHE), where v_free_target is referenced directly.

In D7538#359161, @slw_zxy.spb.ru wrote:

In D7538#358922, @markj wrote:

• The FMR_NEEDFREE target in -CURRENT is now the same as in this patch (though I didn't realize that when I made the change). See r332365.

Are you sure? A im still see 'r = FMR_LOTSFREE;' at line 4624

r332365: Discussed. Not very impotant. I am don't touch this is patch, may be zfs_arc_free_target is badly nameed (this target used as zfs_arc_min_free_target and I am use for my setup v_free_targetx1.4).
Any way, this is sysctl and can be tuned.

In D7538#358922, @markj wrote:

Sorry that this review has stalled lately. I would like to compare this patch to what's in -CURRENT, which has evolved a fair bit since the patch was updated. Once that picture is more clear, we can focus on stable/11.