allow epochs to compose

[mmacy]

Summary

• add tracker argument to preemptible epochs
• make epoch read path inline in kernel
• change in_epoch to take an epoch as argument

Test Plan

pho + llnw canary

[imp]

sys/conf/kern.post.mk

188 ↗

(On Diff #44650)

Will we be able to use epochs in modules?

[mmacy]

sys/conf/kern.post.mk

188 ↗

(On Diff #44650)

Yes. See the _KBI functions in epoch.h. Much like locks and atomics there's no provision for in-tree modules to use the inline variants. I hope to address that at some point.

[mmacy]

• fix rack / LINT build

[mmacy]

• fix cast
• add dumponfail option to in_epoch

[mmacy]

• use dynamically generated thread_global to eliminate proc.h dependencies

[mmacy]

sys/conf/kern.post.mk

188 ↗

(On Diff #44650)

Modules wanting inline atomics, locks, and epoch sections can define KLD_BASE. So far I've only done it for rack and fastpath but it may make sense for ZFS and the 100G drivers.

[mmacy]

• rack fixes
• eliminate redundant acquisitions

[mmacy]

• eliminate ti_locked branchfest in do_segment, rack, and fastpath
• replace all INP_INFO_{RLOCK, RUNLOCK} with explicit epoch_tracker usage
• restore TIMEWAIT checks at close

[mmacy]

• don't drop reference in lock del as we're no longer adding a ref prior
• rebase
• cleanups

[mmacy]

• rebase to inline critical

[mmacy]

rebase

[kbowling]

We've tested this in synthetic stress testing and production load at LLNW. This is a wide reaching change and has higher chance for fallout, but I think getting additional public testing in HEAD will be the most prudent use of time to stabilize the KPI and implementation for 12.0-RELEASE.

@jtl for NF there are some minor mechanical changes that will be needed for your out of tree stacks.