Rather than just protecting the removal from the list, also protect
the check even if it means we may lock and unlock without doing any
work, plugging possible race condition in a rarely exercised path of
if_detach_internal().

Sponsored by: The FreeBSD Foundation