Remove adapters from t4_list earlier during detach.

This ensures the clip task won't race with t4_destroy_clip_table.

While here, make some mutex destroys unconditional since attach always
initializes them.

Reviewed by: np
MFC after: 1 week
Sponsored by: Chelsio Communications
Differential Revision: https://reviews.freebsd.org/D21952