fsx: bounds check the inputs

In particular, don't allow the user to specify a file size that can't be
expressed as an int, since fsx's random-number generator only has a 32
bit range.

(cherry picked from commit 3f8ca7a22ed917a3e3a4ad78538d9f468d6d3bd8)