fsx: bounds check the inputs

In particular, don't allow the user to specify a file size that can't be
expressed as an int, since fsx's random-number generator only has a 32
bit range.

MFC after: 2 weeks