Don't leak fd on sectorsize malloc failure

Also, call endfsent after calling getfsent (i.e. when not explicitly called
with a swap device) for code cleanliness

CID: 1354785
Differential Revision: https://reviews.freebsd.org/D6014
X-MFC with: r298076
Reported by: Coverity
Reviewed by: cem
Sponsored by: EMC / Isilon Storage Division