Detach all children before beginning to tear down the hardware, instead of
doing it last. Also, remove the local tracking of whether usb's busdma
memory allocation got done, because it's safe to call the free_all
function even if it wasn't.