getdirentries: Return ENOTDIR if not a directory.

This is both more logical and more useful than EINVAL.

While here, also check for VBAD and return EBADF in that case. This can
happen if the underlying filesystem got forcibly unmounted after the
directory was opened. Previously, this would also have returned EINVAL,
which wasn't right but wasn't wrong either; however, ENOTDIR would not
be appropriate.

MFC after: never
Sponsored by: Klara, Inc.
Reviewed by: kevans, kib
Differential Revision: https://reviews.freebsd.org/D51209