libbe(3): Impose dataset length restrictions on boot env name validation

Previously, we only validated names for character restrictions. This is
helpful, but we should've also checked length restrictions- dataset names
must be restricted to MAXNAMELEN.

While here, move validation before doing a bunch of concatenations and fix
error handling in be_rename. It was previously setting the error state based
on return value from a libzfs function, which is wrong: libzfs errors don't
necessarily match cleanly to libbe errors. This would cause the assertion in
be_error to hit when the error was printed.