Oh, I should have noted that I either wrote the code
or cribbed it from Linux sources that carry the same
dual license as the OFED sources.

In D58851#1352085, @kib wrote:

In D58851#1351887, @rmacklem wrote:

In D58851#1351758, @kib wrote:

I looked more carefully at the issue of the interaction between vm_fault_quickXXX() and map_remove_YYY(). It could be implemented using the observation that when pmap_extract_and_hold() succeeds but the resulting page is busy, we should fall back to the vm_fault_hold() path for that page. This requires some kind of interlock between the busy state and wiring.

I even started implementing it, but then I considered what would be the consequence of not implementing. Basically, to get advantage of that situation, userspace needs to construct some situation. For instance, it needs to map a file shared, and then do read(2) from pipe into the mapping. Similarly, for physio. IMO if userspace is so much dedicated to cause this inconsistency, we might allow it. The VM code is simpler, and user is causing only self-inflicted damage. I do not see a problem with it.

On the other hand, the pmap_remove_all/write() calls make the regular file io with read(2) or write(2), or normal userspace writes through the mapping, atomic with regard to the NFS over RDMA io RPCs. I believe it is good enough.

Well, if this interface is not used, the NFS server does a..
VOP_READ() with a shared vnode lock
or
VOP_WRITE() with either a shared or exclusively locked vnode

(depending on whether the fs sets MNTK_SHARED_WRITES)

No range locking.

So, ideally, this interface would provide the same semantics.
If not, someone, someday, will go "when I mount rdma this weird.. happens".
I don't see that when I do a tcp mount.

What I discussed above is not about the range locking. It is about some very specific user buffer configurations that probably cannot be created by real applications, and cannot be utilized for any useful work. Also, I believe that the issue is not limited to the proposed interface, but also exists e.g. for buffered reads. My point is that it does not corrupt the kernel state, and is irrelevant practically for correctness.

In D58851#1351758, @kib wrote:

I looked more carefully at the issue of the interaction between vm_fault_quickXXX() and map_remove_YYY(). It could be implemented using the observation that when pmap_extract_and_hold() succeeds but the resulting page is busy, we should fall back to the vm_fault_hold() path for that page. This requires some kind of interlock between the busy state and wiring.

I even started implementing it, but then I considered what would be the consequence of not implementing. Basically, to get advantage of that situation, userspace needs to construct some situation. For instance, it needs to map a file shared, and then do read(2) from pipe into the mapping. Similarly, for physio. IMO if userspace is so much dedicated to cause this inconsistency, we might allow it. The VM code is simpler, and user is causing only self-inflicted damage. I do not see a problem with it.

On the other hand, the pmap_remove_all/write() calls make the regular file io with read(2) or write(2), or normal userspace writes through the mapping, atomic with regard to the NFS over RDMA io RPCs. I believe it is good enough.

In D58851#1351260, @kib wrote:

IOMMU has nothing to do with this function. It does not even look into the bus address space.

Oops, I confused this with what I need for the client, which is your _bus_dmamap_load_ma[] that maps pages to
configuous dma space.

If it isn't hard to do, it would be nice if vm_object_take_pages_for_io()
had a flag option that indicated if it should return failure if there is
no IOMMU hardware. (Or is there an easy way for the caller to check
before calling this function?)

In D58181#1334837, @fuz wrote:

This would have been nicer if the flag was named "noucache" with inverse
semantics as with nodump. (so "ucache" to turn it off)
nodontcache is a weird double negation.

Maybe this can still be revised?

In the NFS game there are two things...

• The RFCs. They don't define exactly when an NFS server should return ESTALE, so I don't think either the current behaviour or the behaviour with this patch is incorrect.
• Linux. It has become the "defacto standard" for NFS. (Solaris once was the "defacto standard" but??)

So, if this patch results in behaviour consistent with
Linux, I think it is reasonable to do.

I also updated the comment in kgss_load().

In D57681#1322466, @kib wrote:

In D57681#1322456, @rmacklem wrote:

This looks ok to me, although I'll admit I am not
conversant with UFS semantics, so I'll leave that
to others.

All existing callers pass tsa == NULL, so the intent is that the semantic does not change.
It is the question whether your case is served when tsa != NULL is provided.

I guess it does. I don't know UFS, but it does seem "inefficient" to first do
UFS_INODE_SET_FLAG_SHARED(), which locks/unlocks the vnode interlock and
then call ufs_itimes(), which locks the vnode interlock again and calls ufs_itimes_locked().

This looks ok to me, although I'll admit I am not
conversant with UFS semantics, so I'll leave that
to others.

In D57681#1322365, @kib wrote:

What is not clear to me, should ufs_itimes() only use the passed timespec for atime update, or it shall be used for all times as I did in the initial version of the patch.
Perhaps this is the question to Rick, since I believe the only caller with tsa != NULL would be nfs server.

Made changes as suggested by kib@.

Changed return value to EBUSY when the nfsd
are running and got rid of the locking around
reading the list head, as suggested by markj@.

Add vfs_netexport_reset() as suggested by markj@.
Also, add KASSERT()s to check ne_ref > 0.

In D57555#1320278, @glebius wrote:

Mark, again thanks a lot for fixing my bug that I was too slow (extremely slow!) to fix. I feel shame for that. Rick, very sorry for delaying looking into the problem for two months.

In D57554#1320166, @kib wrote:

Do you need a kernel thread and code to create the placeholder files? Could it be done in userspace?

Updated with changes recommended by markj@
and kib@. Mostly removal of unnecessary locking
calls, plus a couple of important typo fixes.

Added vfs_netexport_acquire() as suggested by markj@
and renamed vfs_netexport_free() to vfs_netexport_release(),
since that seemed more consistent.

This looks fine to me and seemed to behave when I
tested with lotsa printf()s.

Ok, so I finally think I remember how this works;-)
Your patch tests out well, but I think it needs something like the following:

• sys/rpc/rpcsec_tls/rpctls_impl.c.yyy 2026-06-14 09:08:45.910463000 -0700

+++ sys/rpc/rpcsec_tls/rpctls_impl.c 2026-06-14 09:01:06.722976000 -0700
@@ -298,8 +298,18 @@ printf("rpctls_connect soref so=%p\n", so);

	stat = rpctlscd_connect_2(&arg, &res, rpctls_connect_handle);
	if (stat == RPC_SUCCESS)
		*reterr = res.reterr;

• else

+ else {
+ struct ct_data *ct = (struct ct_data *)newclient->cl_private;
+
+printf("clnt at rpctls_rpc_failed so=%p\n", so);

		rpctls_rpc_failed(&ups, so);

+ /*
+ * Since rpctls_rpc_failed() will do a soclose() if the
+ * daemon is not running, set ct_tlsstate to avoid doing
+ * a close in clnt_vc_destroy().
+ */
+ ct->ct_tlsstate = RPCTLS_INHANDSHAKE;
+ }

In D57555#1319373, @rmacklem wrote:

Ok, so now I see what you've done. You are now taking the
soref() when it is inserted in the RB tree instead of when it
is taken out.

I suspect the "else" case in rpctls_rpc_failed() needs a
sorele() along with the soshutdown(), to get rid of the
reference.

Now that I am testing it, this does not seem to be needed.
(You get to the "else" case if rpc.tlsservd is running on the
server and you use your little tls_trigger.c test against it.)

Once I've put it through some testing, I'll update the patch.

Ok, so now I see what you've done. You are now taking the
soref() when it is inserted in the RB tree instead of when it
is taken out.