vfs mount: Consistently use ENODEV internally for an invalid fstype

Change vfs_byname_kld to always return an error value of ENODEV to
indicate an unsupported fstype leaving ENOENT to indicate errors such
as a missing mount point or invalid path. This allows nmount(2) to
better distinguish these cases and avoid treating a missing device
node as an invalid fstype after commit 6e8272f317b8.

While here, change mount(2) to return EINVAL instead of ENODEV for an
invalid fstype to match nmount(2).

PR: 274600
Reviewed by: pstef, markj
Differential Revision: https://reviews.freebsd.org/D42327