This fixes some weird behavior triggered by nvidia-drm.ko: some DRM
cleanup functions will be called multiple times, leading to a double
free. drm_mode_config_cleanup will be called twice, causing ida_destroy
to be called twice. Although calling the cleanup twice doesn't seem
very clean, on Linux this seems to be permissable as it handles it
just fine.
In order to preserve this behavior this change checks if the objects
have already been destroyed and bails if so. This fixes the panic seen
when unloading the nvidia-drm driver.