tools: test: iconv: fix open_2 to not segfault

Record error condition when iconv_open() fails rather than leaving a
bogus iconv_t that iconv_close() can later choke on; this is one failure
mode.

If we opened MAX_LIMIT files with success, we need to rewind one so that
we don't iconv_close() one past the end of cd; this is the second
failure mode.

Sponsored by: Klara, Inc.

(cherry picked from commit 814bd1ed438f7dfc5bedcb1f3e772a46fe7026bb)