epoch: Queue drain callbacks from each CPU's callback task

[rcm]

Summary

epoch_drain_callbacks() bound the calling thread to each CPU in turn
to queue a callback on that CPU's record. Callers are often user
threads, such as kldunload or ifconfig destroy, and ULE does not
preempt a CPU-bound thread for a migrating thread of no better
priority, so each bind could wait out a time slice. On an 8-CPU VM
with six busy loops running, destroying an epair interface took 1.5 s
and kldunload pflow 0.8 s.

Instead, set a request flag on each record and enqueue that CPU's
callback task, which queues the callback before it polls the record.
The task runs on the CPU's softirq gtaskqueue thread at PI_SOFT, which
preempts a busy user thread, so the request is served at once; the
caller no longer migrates. Both operations now take under 10 ms, idle
or under load.

Sponsored by: Rubicon Communications, LLC ("Netgate")

[markj]

Nice change.

sys/kern/subr_epoch.c

994

I think you need a barrier between these two loops, to ensure that the store to e_drain_count is visible to a thread which observed er_drain_req != 0. In particular, epoch_call_task() may be running concurrently already, so the gtaskqueue mutex doesn't provide synchronization. Or am I missing something?

In other words, I think the atomic_store_int() below needs to be atomic_store_rel_int(), and the atomic_load_int() in epoch_call_task() needs to be atomic_load_acq_int().

[rcm]

epoch_drain_callbacks() now sets the flag with atomic_store_rel_int() and epoch_call_task() reads it with atomic_load_acq_int().

[rcm]

sys/kern/subr_epoch.c

994

Ah yep, thanks!

[markj]

This approach works mostly by virtue of the fact that the gtaskqueue threads have high scheduling priority compared to user threads, so they're able to preempt a busy worker. That's not really clear from the commit log message.