The problem happens when a device is 1. removed, 2. removed and added back,
and 3. removed and a different one is added.
The case 2 applies to PR 219829 case.
One can reproduce the problem by creating, using, and destroying wlan or
tap devises as these are the easy one to add and remove on the fly.