Change Details

The patch adds a facility to detect stray callouts after driver unload. Instead of crashes on callwheel list iteration, hopefully the checker would detect a callout for which the driver was already unloaded and provide the diagnostic, which should be enough to find the offender. How to use: 1. Compile kernel with the option CALLOUT_DEBUG_DRAIN. This changes KBI, all modules must be built with the option. 2. Ensure that there is only one device for attach. Disable all other devices, including sibling PCI functions, with `devctl disable <pci name>`. 3. Load modules, `kldload mlx5en`. Then print the modules load address with `kldstat`, save this info. 4. Put some load. 5. Unload the driver with `kldunload mlx5en`. If a callout was left armed, kernel panics and prints addresses of callout callback, and for Linuxkpi callouts, additional address of the actual timer of work function that would be called. 6. Using kldstat base addresses of modules, and addresses of callout or timer functions, the offender should be identified. The patch works by copying the callout and timer function addresses from the callwheel callout entry to the _previous_ entry in callwheel list. In other word, until callout exec operates on valid callouts, there is always the data about next, possible invalid, callout. Sponsored by: Mellanox Technologies

The patch adds a facility to detect stray callouts after driver unload. Instead of crashes on callwheel list iteration, hopefully the checker would detect a callout for which the driver was already unloaded and provide the diagnostic, which should be enough to find the offender. How to use: 1. Compile kernel with the option CALLOUT_DEBUG_DRAIN. This changes KBI, all modules must be built with the option. 2. Ensure that there is only one device for attach. Disable all other devices, including sibling PCI functions, with `devctl disable <pci name>`. 3. Load modules, `kldload mlx5en`. Then print the modules load address with `kldstat`, save this info. 4. Put some load. 5. Unload the driver with `kldunload mlx5en`. If a callout was left armed, kernel panics and prints addresses of callout callback, and for Linuxkpi callouts, additional address of the actual timer of work function that would be called. 6. Using kldstat base addresses of modules, and addresses of callout or timer functions, the offender should be identified. The patch works by copying the callout and timer function addresses from the callwheel callout entry to the _previous_ entry in callwheel list. In other word, until callout exec operates on valid callouts, there is always the data about next, possible invalid, callout. Example of the test panic dump: ``` root@:/ # sysctl debug.callout_check_drain_test=1 debug.callout_check_drain_test: 0CALLOUT CHECK DRAIN 0xffffffff803dc2e0 0xfeedf00d 0xffffffff803dc2e0 func 0xffffffff803dc2e0 at callout_check_drain_test_func+0 arg1 0xffffffff803dc2e0 at callout_check_drain_test_func+0 #0 0xffffffff803da240 at callout_reset_sbt_on_arg1+0x480 #1 0xffffffff803dc2a2 at callout_check_drain_test+0xb2 #2 0xffffffff803cddfc at sysctl_root_handler_locked+0x9c #3 0xffffffff803cd261 at sysctl_root+0x1c1 #4 0xffffffff803cd8db at userland_sysctl+0x13b #5 0xffffffff803cd75f at sys___sysctl+0x5f #6 0xffffffff806751ab at amd64_syscall+0x18b #7 0xffffffff8064d6ae at fast_syscall_common+0xf8 panic: callout check drain ``` Sponsored by: Mellanox Technologies

The patch adds a facility to detect stray callouts after driver unload. Instead of crashes on callwheel list iteration, hopefully the checker would detect a callout for which the driver was already unloaded and provide the diagnostic, which should be enough to find the offender. How to use: 1. Compile kernel with the option CALLOUT_DEBUG_DRAIN. This changes KBI, all modules must be built with the option. 2. Ensure that there is only one device for attach. Disable all other devices, including sibling PCI functions, with `devctl disable <pci name>`. 3. Load modules, `kldload mlx5en`. Then print the modules load address with `kldstat`, save this info. 4. Put some load. 5. Unload the driver with `kldunload mlx5en`. If a callout was left armed, kernel panics and prints addresses of callout callback, and for Linuxkpi callouts, additional address of the actual timer of work function that would be called. 6. Using kldstat base addresses of modules, and addresses of callout or timer functions, the offender should be identified. The patch works by copying the callout and timer function addresses from the callwheel callout entry to the _previous_ entry in callwheel list. In other word, until callout exec operates on valid callouts, there is always the data about next, possible invalid, callout. Example of the test panic dump: ``` root@:/ # sysctl debug.callout_check_drain_test=1 debug.callout_check_drain_test: 0CALLOUT CHECK DRAIN 0xffffffff803dc2e0 0xfeedf00d 0xffffffff803dc2e0 func 0xffffffff803dc2e0 at callout_check_drain_test_func+0 arg1 0xffffffff803dc2e0 at callout_check_drain_test_func+0 #0 0xffffffff803da240 at callout_reset_sbt_on_arg1+0x480 #1 0xffffffff803dc2a2 at callout_check_drain_test+0xb2 #2 0xffffffff803cddfc at sysctl_root_handler_locked+0x9c #3 0xffffffff803cd261 at sysctl_root+0x1c1 #4 0xffffffff803cd8db at userland_sysctl+0x13b #5 0xffffffff803cd75f at sys___sysctl+0x5f #6 0xffffffff806751ab at amd64_syscall+0x18b #7 0xffffffff8064d6ae at fast_syscall_common+0xf8 panic: callout check drain ``` Sponsored by: Mellanox Technologies