ar: detect and error out on 32-bit symbol table overflow

BSD ar currently does not support the /SYM64/ 64-bit symbol table, and
previously truncated to 32-bits, silently producing corrupted archives
larger than 4GB.

This is another overflow case in addtion to r342575.

PR: 234454
Reported by: Aijaz Baig, imp
MFC after: 2 weeks
MFC with: r342575
Sponsored by: The FreeBSD Foundation