This allows me to do more unlocked cases in fault and tmpfs. The idea is to check whether you're the first to dirty a swap object and free the space if so. This could be optimized with another page flag that says whether or not there is any swap space allocated.
Another approach would be to simply look for dirty pages with allocated swap space in the active/inactive queues. It makes me a little uncomfortable that every place that calls vm_page_dirty() needs to know if it should clear swap or not. I can't convince myself that are gaps but there are a lot of calls to dirty().