I strongly do not like this, esp the proliferation of PMAP_HAS_DMAP into MD code.

Since you access the buffer pages one by one, you can use sf_bufs to get the mapping. On dmap arches, this would give the same optimization as you did manually coding PHYS_TO_DMAP(). But also the same code path would be used on !dmap platforms.

This is an impressive performance improvement and one well worth adding to the system.

If it is possible to achieve your goal in the way outlined by kib that would certainly be preferable.

Is there any concern that this new use will add pressure to the sf_buf system on non-direct map platforms?

In D53424#1221598, @gallatin wrote:

Is there any concern that this new use will add pressure to the sf_buf system on non-direct map platforms?

No. sf_bufs were designed specifically to provide transient efficient remapping of arbitrary page. On non-dmap arches, there is a limited (like 1k) number of slots available for sf_buf mappings, and in the worst case the requester would wait until other thread free its sf_buf, which should not take long.

Using cpu-private sf_bufs together with sched_pin() would eliminate the need to broadcast invalidation IPI for non-dmap case. It costs nothing on dmap.

This change stalled, and I feel guilty. So I did the modification that I suggested, namely switch the code to use sfbufs instead of (mis)using DMAP in the MI code. Please see P709.