ifnet: Defer detaching address family dependent data

While diagnosing PR 279653 and PR 285129, I observed that thread may
write to freed memory but the system does not crash. This hides the
real problem. A clear NULL pointer derefence is much better than writing
to freed memory.

PR: 279653
PR: 285129
Reviewed by: glebius
MFC after: 3 weeks
Differential Revision: https://reviews.freebsd.org/D49444