newfs_msdos: rework error handling for eventual use in makefs

Return -1 on errors from mkfs_msdos() instead of err()/errx(), to
allow different consumers to handle errors as appropriate.

Obtained from: NetBSD
Sponsored by: The FreeBSD Foundation