vfs_vnops.c: Fix the named attribute check for open

[rmacklem]

Summary

In vn_open_cred(), the correct check for O_NAMEDATTR
was done when O_CREAT was specified, but the file already
exists. (Added by commit 2ec2ba7e232d, which will be listed
as a Fixes: in the commit log message.)

This correct check was not copied to the case where O_CREAT
has not been specified.

This patch fixes this.

Test Plan

Tested with a system where ZFS is patched to handle
named attributes.

[rmacklem]

I now realize that the check for non-named attribute is in
ZFS's VOP_LOOKUP(), so this patch is not really needed,
but I think it is a good "safety belt" to put in.

[kib]

sys/kern/vfs_vnops.c.sav

377 ↗

(On Diff #153892)

Can we introduce a helper to calculate error for both pathes, and avoid code duplication?

[rmacklem]

Added a helper function, as suggested by kib@.