vfs: trylock in vfs_msync and refactor the func

• use LK_NOWAIT instead of calling VOP_ISLOCKED before deciding to lock
• evaluate flags before looping over vnodes

Reviewed by: kib
Tested by: pho (in a larger patch, previous version)
Differential Revision: https://reviews.freebsd.org/D23035