Looks fine to me.

In D6928#148146, @ralsaadi_swin.edu.au wrote:

Should I remove DN_BH_WLOCK()/DN_BH_WUNLOCK() and use atomic_add_int() instead?
Is the patch reasonable now?

In D6928#148012, @ralsaadi_swin.edu.au wrote:

Sorry if my thought is incorrect but just to explain my idea of using DN_BH_WLOCK():
1- ref_count (namely pie_desc.ref_count) is updated only by PIE module and checked (read access) by unload_dn_sched() in ip_dummynet.c.
2- ref_count in unload_dn_sched() is accessed with DN_BH_WLOCK()
3- ref_count in PIE module is updated when DN_BH_WLOCK() is held (either explicitly by calling DN_BH_WLOCK() in PIE module or implicitly by Dummynet somewhere before call PIE functions.
So, is the lock I added still isn't sufficient?
Anyway, using atomic_add_int() is much easier than dealing with deadlocks ;-)

Regarding the "a small race that remains", I agree with you, this race could happen. However, changing Dummynet lock type is out of my knowledge.

In D6928#147905, @ralsaadi_swin.edu.au wrote:

If PIE ref_count is decremented by the user thread after callout_reset_sbt(), how we guarantee that pie_callout_cleanup() finished its execution?

Are similar fixes needed elsewhere in the AQM code?

By switching to callout_drain() and removing the cleanup code from the callout, it should be possible to further simplify the code by getting rid of CALLOUT_RETURNUNLOCKED and removing the mtx_unlock() calls from the callout.

In D6928#145364, @hselasky wrote:

It would be better to associate the callout with a mutex and use callout_async_drain() to catch the final teardown.

That looks correct to me.