Avoid acquiring the object lock if color is already set. It can not be
unset until the object is recycled so this check is stable. Now that we
can acquire the ref without a lock it is not necessary to group these
operations and we can avoid it entirely in many cases.

Reviewed by: kib, markj
Differential Revision: https://reviews.freebsd.org/D22565