Make sure fmemopen succeeds in :test_append_binary_pos before calling ftell
on the FILE object

This fixes potential null pointer dereferences on failure

CID: 1254952
MFC after: 2 weeks
Reported by: Coverity
Sponsored by: EMC / Isilon Storage Division