The two-pass drain of 7bf11a2f0c9a assumes that once the first drain
callback has run, every callback pending at the time of the call has at
least been dispatched. ck_epoch's ring breaks that: a record's callbacks
sit in four buckets indexed by the global epoch, a poll at epoch G
dispatches bucket G-2 and bucket G-1 only on the CAS path, and the global
epoch is advanced by every other record's poll and by epoch_wait(). When
it moves by two or more between polls of one record, a bucket is passed
over for a full turn of the ring while younger buckets, including the
drain markers of both passes, are dispatched. epoch_drain_callbacks()
then returns with callbacks pending, and epoch_free() frees the record
with them in it.
A randomized stress hits this in about 1 % of drains on an 8 vCPU VM
and 3 % on a 4-core Netgate 4200.
Instead of a marker, have the callback task record which of its record's
buckets are non-empty when it sees the request and report the drain done
once each of them has been dispatched. A bucket only empties by the
task's own poll, which takes all of it and runs it before reporting, so
one pass suffices. A record with nothing pending finishes at the first
task run.
Fixes: 7bf11a2f0c9a ("epoch: Fix epoch_drain_callbacks()")
MFC after: 1 month
Sponsored by: Rubicon Communications, LLC ("Netgate")