Currently linux_destroy_dev() waits for the reference count on the linux cdev to drain, and each open file hold the reference. Practically it means that destroy_dev() is blocked until all userspace processes that have the cdev open, exit. FreeBSD devfs does not have such problem, because device refcount only prevents freeing of the cdev memory, and separate 'active methods' counter blocks destroy_dev() until all threads leave the cdevsw methods. After that, attempts to enter cdevsw methods are refused with an error.
Implement somewhat similar mechanism for LinuxKPI cdevs. Lower cdev refcount to only mean hold on the cdev memory. Add sirefs count to track both number of threads inside the cdev methods, and for single-bit indicator that cdev is being destroyed. In the later case, the call is redirected to the dummy cdev.
Add the mlx5_ib_disassociate_ucontext() method for mlx5 ib driver, which basically enables call to linux_destroy_dev() on the mlx5ib driver unload, which now uses the updated linux_destroy_dev() infrastructure.
Add a partial implementation for zap_vma_ptes() when possible (basically, when vma is backed by a managed object).
Sponsored by: Mellanox Technologies