Don't destroy device twice. This can lead to kernel panic on reboot/shutdown.
This is not an MFC, because the code in HEAD has other changes and there is
no such problem there.

Reported and tested by: ken
Approved by: re (hrs)