The patch adds a facility to detect stray callouts after driver unload. Instead of crashes on callwheel list iteration, hopefully the checker would detect a callout for which the driver was already unloaded and provide the diagnostic, which should be enough to find the offender.
How to use:
1. Compile kernel with the option CALLOUT_DEBUG_DRAIN. This changes KBI, all modules must be built with the option.
2. Ensure that there is only one device for attach. Disable all other devices, including sibling PCI functions, with `devctl disable <pci name>`.
3. Load modules, `kldload mlx5en`. Then print the modules load address with `kldstat`, save this info.
4. Put some load.
5. Unload the driver with `kldunload mlx5en`. If a callout was left armed, kernel panics and prints addresses of callout callback, and for Linuxkpi callouts, additional address of the actual timer of work function that would be called.
6. Using kldstat base addresses of modules, and addresses of callout or timer functions, the offender should be identified.
The patch works by copying the callout and timer function addresses from the callwheel callout entry to the _previous_ entry in callwheel list. In other word, until callout exec operates on valid callouts, there is always the data about next, possible invalid, callout.
Sponsored by: Mellanox Technologies