Provide a generalized way to retry IPIs with allowing CPUs to mark completion

[mjg]

Summary

This is moved away from sleepable rm locks.

vfs patch is here as a demo, will be committed separately. This eliminates an atomic from vfs busy/unbusy and vfs_cache_root. This is not ported over to rms locks instead because of granularity -- right now it's all or nothing, but for example suspending writes should not prevent root vnode lookup from working (and later on it wont).

[jeff]

sys/kern/kern_rmlock.c

1031

Why did this barrier go?

[mjg]

sys/kern/kern_rmlock.c

1031

It is spurious in that smp_rendezvous_cpus issues it internally, but I can add it back if only for clarity.

[mjg]

sys/kern/kern_rmlock.c

1031

To that point, note all smp_rendezvous_cpus consumers depend on said barrier being there.

[jeff]

sys/kern/kern_rmlock.c

1031

I would just not delete the whole comment block. I wouldn't add fences that are not necessary.

[mjg]

sys/kern/kern_rmlock.c

1031

If a comment should exist, it should be placed above smp_rendezvous_cpus and state that it provides appropriate fencing.

1. release fence on the calling CPU before issuing any IPIs
2. acquire fence on targeted CPUs so that they see the update
3. release fence after all work is complete so that targeted CPUs publish everything
4. acquire fence on the calling CPU so that it sees said updates

Note the routine has comments to that effect internally.

[mjg]

• add a comment about dependence on smp_rendezvous fences
• drop a spurious acq

[jeff]

I don't object but I haven't entirely understood it.