I feel that I made the API overly complicated because I was trying to unify an API for administration and for programming. This example is more complex than necessary if you are simply trying to change the set for the process. A non-anonymous or numeric set is only required if you wish to refer to it later. Think of it more like a process group where you want to be able to apply that constraint to multiple things at once. Normal programs simply exist within their current process group and only in specific circumstances do you create a new one. The numbered sets are this 'group'. They exist in the middle of the hierarchy with the root set above them giving an absolute limit on available CPUs that may be from jail or the actual system. Below them exists anonymous sets for programs that have constrained themselves to a subset of the numbered set.

hilarious, thank you.

We should set reference bits or similar so that the LRU is updated lazily.

I approve of the approach.

First off, lots of good discussion here. I think this is the start of a good approach and I support committing it disabled.

Have you thought about whether there are any side-effects in swap behavior from using the same object? Might we run into clustering behavior?

I think there may be a bug in this. busy_sleep may not properly wait for an exclusive lock if it is sleeping to zero a page while it is sbusy and the caller requested an sbusy after zeroing. Currently no caller does this so it is not going to cause problems. There are simply too many flags and it complicates things but I do not see an easy way to drop many.

One other thing; I have submitted patches to drm to address these changes. I will have to bump FreeBSD_version and fix one extra case in drm-legacy because it is not written in a way that it can use vm_page_busy_acquire().

I agree with mark. CK_LIST is just a copy of queue.h with ck barriers added. We want stronger assertions to tie to the smr interface. We have wide arm64 platforms available to us for testing now. I think the risk of incorrect barriers is minimal if we are conservative. The only real question is how often to require acquire load.

I am ok with this but eventually it likely should not be shared busy.

Would you not prefer to use an invalid value rather than a bit to hold the free state? i.e. VHOLD_NO_SMR would be something like VHOLD_DEAD 0xffffffff. Some of your asserts would have to be adjusted. The first ref should switch dead with 1.

I don't mind adding accessors after the fact. It will help us continue to make them more natural and less obtrusive if we can make this work with them.

So In general the naming scheme is:
vn_* or v* in older code for vnode specific routines.
vop_ for routines specific to ops.
vfs_ for global or mount.

Overall this is nice and a better first example of smr than radix.

also please put a comment in the pwd structure describing how the synchronization works. I believe I understand but am not 100%.

This is a good opportunity to push for idiomatic access.

Review feedback.

Upload the correct diff

Sometimes if you keep re-arranging pieces they shuffle into a more compact
representation.

Fully handle sleep fail/nowait cases. Simplify a few lockless functions by
introducing another helper. Handle the object NULL assignment in a single
place with a compiler barrier.

I forgot to submit these comments

If there are no objections I will commit this version.

Name my SMR variant GUS for Global Unbounded Sequences.