regex(3): Handle invalid {} constructs consistently and adjust tests

Currently, regex(3) exhibits the following wrong behavior as demonstrated
with sed:

• echo "a{1,2,3}b" | sed -r "s/{/_/" (1)
• echo "a{1,2,3}b" | sed "s/\}/_/" (2)
• echo "a{1,2,3}b" | sed -r "s/{}/_/" (3)

Cases (1) and (3) should throw errors but they actually succeed, and (2)
throws an error when it should match the literal '}'. The correct behavior
was decided by comparing to the behavior with the equivalent BRE (1)(3) or
ERE (2) and consulting POSIX, along with some reasonable evaluation.

Tests were also adjusted/added accordingly.

PR: 166861
Reviewed by: emaste, ngie, pfg
Approved by: emaste (mentor)
MFC after: never
Differential Revision: https://reviews.freebsd.org/D10315