This is based on the observation by Yoshihiro Ota expressed in D22256, but exploits that observation differently.
Break swap_pager_meta_ctl into two functions, swap_pager_meta_lookup and swap_pager_meta_remove. Use one or the other in place of swap_pager_meta_ctl depending on whether the POP flag bit was set by the caller.
For the instance of swap_pager_meta_remove left in swap_pager_copy, replace it with an implementation that copies from vm_pager_meta_free_all to find the next nonempty source swap block, rather than testing every source swap block for emptiness.