I think it's fine to leave this as "sets". I assume you changed it because the function does not necessarily set both. Because it's entirely dependent on the function's arguments I think it's still reasonable to say sets. To me "may set" implies that vm_map_protect will itself decide if one/both will be set.

Perhaps "are not allowed to"?

As is (and before your change) it seems unclear to me whether this is something the caller must take care not to violate, or that the function disallows.

Maybe something like
"Also, any in-progress wiring operation on the map affecting the specified range will cause .Nm to sleep, waiting for completion"

"The map does not allow ..." I think is clearer

Drop If as above, "Both new protection and..."
Or maybe Both VM_MAP_PROTECT_SET_PROT and VM_MAP_PROTECT_SET_MAXPROT were ..."

"The operation"

I would change "is declined" to "will fail" and "the condition" to "this condition".

"a shared mapping"

"of a read-only file from read-only to read-write"

s/for backing/to back/

"... protection updates were requested"

"but the specified"