Fix compiler warnings with open_to_operation.c

Other sidenotes:

• Remove unused variables with main(..)
• Convert errx/exit with -1 to errx/exit with 1
• Fix a bogus test in try_directory_open (expected_errno == expected_errno -> errno == expected_errno) [*]
• Fix some warnings related to discarded qualifiers
• Remove a bogus else-statement at the end of check_mmap_exec(..) in the successful case. mmap(2), POSIX, Linux, etc all don't state what the behavior is when mixing O_WRONLY + PROT_EXEC, so assume success for now to get the test program to pass again.

PR: 201286 [*]
MFC after: 1 week
Submitted by: David Binderman <dcb314@hotmail.com>
Sponsored by: EMC / Isilon Storage Division