Page MenuHomeFreeBSD

opencrypto: Fix assignment of crypto completions to worker threads
ClosedPublic

Authored by markj on Jan 14 2021, 4:35 PM.

Details

Summary

Since r336439 we simply take the session pointer value mod the number of
worker threads (ncpu by default). This does not work very well.
Instead, maintain an incrementing counter with a unique value per
session, and use that to distribute work to completion threads.

We could alternately hash the pointer in some way, but a counter gives
consistent distributions across reboots. It also plays more nicely with
GELI, which creates ncpu sessions per GEOM. It would be preferable to
dynamically re-assign sessions to worker threads periodically based on
load, but we're not there yet. In the meantime, using more than one
worker helps IPSec throughput.

Diff Detail

Repository
R10 FreeBSD src repository
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

👍

sys/opencrypto/crypto.c
948

Any reason not to use either a native word size or just 32-bit everywhere? The bias should be pretty minimal for even a few doubling of core counts on bigger machines.

This revision is now accepted and ready to land.Jan 14 2021, 5:37 PM
sys/opencrypto/crypto.c
948

Not really, I did it this way to be defensive in case something starts assuming the value is a unique session ID. There's no problem using atomic_fetchadd_64() in machine-independent code.

markmi_dsl-only.net added inline comments.
sys/opencrypto/crypto.c
949

32-bit powerpc and powerpcspe do not have atomic_fetchadd_64 implementations. The stable/12 ci.freebsd.org results are "undefined reference to `atomic_fetchadd_64'" failures.