Page MenuHomeFreeBSD

ktls: Defer creation of threads and zones until first use.
ClosedPublic

Authored by jhb on Wed, Oct 13, 7:43 PM.

Details

Summary

Run ktls_init() when the first KTLS session is created rather than
unconditionally during boot. This avoids creating unused threads and
resources on systems which do not use KTLS.

Sponsored by: Chelsio Communications

Diff Detail

Repository
R10 FreeBSD src repository
Lint
Automatic diff as part of commit; lint not applicable.
Unit
Automatic diff as part of commit; unit tests not applicable.

Event Timeline

jhb requested review of this revision.Wed, Oct 13, 7:43 PM
markj added inline comments.
sys/kern/uipc_ktls.c
427

I know you didn't add this, but this comment doesn't make sense to me. If there an empty domain (meaning that there is a memory domain with no CPUs), we fall back to pinning each thread to a CPU.

This revision is now accepted and ready to land.Thu, Oct 14, 6:03 PM
sys/kern/uipc_ktls.c
427

I think the comment is about the code in ktls_get_cpu() where we use the value of this variable to decide if we choose a KTLS thread in a connection's NUMA domain vs choosing any KTLS thread.

It is also true that by moving this check earlier before starting worker threads we are now consistent in how we bind kthreads in the case we failed back to 1.

2584

I wonder if this should just use sched_bind() instead? It doesn't fail. I guess 'cpuset -g -t <tid>' wouldn't show the binding, but that's the only difference I think?

sys/kern/uipc_ktls.c
2584

sched_bind() does have the advantage that it binds immediately. cpuset_setthread() does not, it just (eventually) sets TDF_NEEDSCHED, so we'd have to wait for the next context switch. But I don't see why sched_bind() is really preferable, given that we use cpuset_setthread() for binding to domains as well.

I'm not a fan of this added complexity, but it looks like it was done in a very careful way. I really appreciate it being done when creating the session, rather than upon enqueue.

I definitely tried to minimize the runtime overhead, so create session is definitely the better place than something per-op. It's also only feasible to handle failure in the create session context.

FWIW, when I boot a RISC-V GENERIC kernel in QEMU KTLS is one of the few "unused" kprocs that stands out along with the "soaiod" kprocs (which I also just fixed to be created on demand instead).