ar: detect and error out on 32-bit symbol table overflow

BSD ar currently does not support the /SYM64/ 64-bit symbol table, and
previously truncated to 32-bits, silently producing corrupted archives
larger than 4GB.

Note that this is only a partial fix; additional checks will come.

PR: 234454
Reported by: Aijaz Baig, imp
MFC after: 2 weeks
Sponsored by: The FreeBSD Foundation