ktls: Defer creation of threads and zones until first use.

[jhb]

Summary

Run ktls_init() when the first KTLS session is created rather than
unconditionally during boot. This avoids creating unused threads and
resources on systems which do not use KTLS.

Sponsored by: Chelsio Communications

[markj]

sys/kern/uipc_ktls.c

423

I know you didn't add this, but this comment doesn't make sense to me. If there an empty domain (meaning that there is a memory domain with no CPUs), we fall back to pinning each thread to a CPU.

[jhb]

sys/kern/uipc_ktls.c

423

I think the comment is about the code in ktls_get_cpu() where we use the value of this variable to decide if we choose a KTLS thread in a connection's NUMA domain vs choosing any KTLS thread.

It is also true that by moving this check earlier before starting worker threads we are now consistent in how we bind kthreads in the case we failed back to 1.

2474

I wonder if this should just use sched_bind() instead? It doesn't fail. I guess 'cpuset -g -t <tid>' wouldn't show the binding, but that's the only difference I think?

[markj]

sys/kern/uipc_ktls.c

2474

sched_bind() does have the advantage that it binds immediately. cpuset_setthread() does not, it just (eventually) sets TDF_NEEDSCHED, so we'd have to wait for the next context switch. But I don't see why sched_bind() is really preferable, given that we use cpuset_setthread() for binding to domains as well.

[gallatin]

I'm not a fan of this added complexity, but it looks like it was done in a very careful way. I really appreciate it being done when creating the session, rather than upon enqueue.

[jhb]

I definitely tried to minimize the runtime overhead, so create session is definitely the better place than something per-op. It's also only feasible to handle failure in the create session context.

FWIW, when I boot a RISC-V GENERIC kernel in QEMU KTLS is one of the few "unused" kprocs that stands out along with the "soaiod" kprocs (which I also just fixed to be created on demand instead).